One 、 brief introduction
How do we make a game ?
   Not long ago in Chengdu TGC2016 The show , We developed a 《 Naruto Tour 》 The somatosensory game , It mainly simulates the mobile game chapter 《 Nine tails hit  》, Four generations of users , Against nine tails , Attracted a large number of players to participate in . On the surface , This game is no different from other somatosensory experiences , actually , It's always running in the browser Chrome Next , in other words , We just need to master the front-end technology , Can be developed based on Kinect Web somatosensory games .
 
Two 、 Realization principle
What is the realization idea ?
   Use H5 Based on Development Kinect The somatosensory game , In fact, the working principle is very simple , from Kinect Collect player and environment data , Like human bones , In some way , So that the browser can access the data .
1、 Collect data
  Kinect There are three shots , The middle lens is like a normal camera , Get color images . The left and right cameras get the depth data through infrared . We use Microsoft's SDK To read the following types of data :
  • Color data : Color images ;
  • Depth data : Color try information ;
  • Human bone data : Based on the above data, through calculation , Get human bone data .
2、 Make it accessible to the browser Kinect data
The framework I've tried and learned about , It's basically based on socket Let the browser process communicate with the server , Data transfer :
  • Kinect-HTML5  use C# Set up the server , Color data 、 Try data 、 Bone data are available ;
  • ZigFu  Support H5、U3D、Flash Development ,API Relatively complete , It looks like a charge ;
  • DepthJS  Provide data access in the form of browser plug-ins ;
  • Node-Kinect2  With Nodejs Build the server , The data provided is relatively complete , There are many examples .
I finally chose Node-Kinect2, Although there is no documentation , But there are many examples , Use what front-end engineers are familiar with Nodejs, In addition, the author's feedback is relatively fast .
  • Kinect: Capture player data , Like depth images 、 Color images, etc ;
  • Node-Kinect2: from Kinect Obtain corresponding data , And secondary processing ;
  • browser : monitor node Apply the specified interface , Get player data and complete game development .
 
3、 ... and 、 preparation
You have to buy one first Kinect ah
1、 System requirements :
It's a hard requirement , I used to waste too much time in an environment that didn't meet the requirements .
  • USB3.0
  • Support DX11 The graphics card
  • win8 And above
  • Support Web Sockets Browser
  • Of course Kinect v2 Sensors are indispensable
 
2、 Environment building process :
  1. Connected to the Kinect v2
  2. install  KinectSDK-v2.0
  3. install Nodejs
  4. install Node-Kinect2
npm install kinect2

  

Four 、 Examples demonstrate
Give me an example !
As shown in the figure below , We show you how to get human bones , And identify the middle of the spine and gestures :

1、 Server side
establish web The server , And send the bone data to the browser , The code is as follows :
var Kinect2 = require('../../lib/kinect2'),
express = require('express'),
app = express(),
server = require('http').createServer(app),
io = require('socket.io').listen(server); var kinect = new Kinect2();
// open kinect
if(kinect.open()) {
// monitor 8000 port
server.listen(8000);
// Specify that the request points to the root
app.get('/', function(req, res) {
res.sendFile(__dirname + '/public/index.html');
});
// Send the bone data to the browser
kinect.on('bodyFrame', function(bodyFrame){
io.sockets.emit('bodyFrame', bodyFrame);
});
// Start reading bone data
kinect.openBodyReader();
}

  

2、 Browser side
Get bone data from browser side , And use canvas Draw it out , The key codes are as follows :

var socket = io.connect('/');
var ctx = canvas.getContext('2d');
socket.on('bodyFrame', function(bodyFrame){
ctx.clearRect(0, 0, canvas.width, canvas.height);
var index = 0;
// Traverse all the bone data
bodyFrame.bodies.forEach(function(body){
if(body.tracked) {
for(var jointType in body.joints) {
var joint = body.joints[jointType];
ctx.fillStyle = colors[index];
// If the bone node is the midpoint of the spine
if(jointType == 1) {
ctx.fillStyle = colors[2];
}
ctx.fillRect(joint.depthX * 512, joint.depthY * 424, 10, 10);
}
// Recognize left and right hand gestures
updateHandState(body.leftHandState, body.joints[7]);
updateHandState(body.rightHandState, body.joints[11]);
index++;
}
});
});

  

Simple lines of code , We have finished the player's skeleton capture , There are certain javascript Basic students should be easy to understand , But what we don't understand is what data we can get ? How to get ? What are the names of the bone nodes ? and node-kienct2 There's no documentation to tell us that .

5、 ... and 、 Developing documents
 
Node-Kinect2 No documentation was provided , I'll sort out my test summary document as follows :
1、 The type of data that the server can provide ;
kinect.on('bodyFrame', function(bodyFrame){}); // What other data types are there ?

  

bodyFrame Bone data
infraredFrame Infrared data
longExposureInfraredFrame similar infraredFrame, It seems more accurate , Optimized data
rawDepthFrame Unprocessed depth of field data
depthFrame Depth of field data
colorFrame Color images
multiSourceFrame All the data
audio Audio data , Not tested
2、 Bone node type
body.joints[11] // joints What does it include ?

  

Node type JointType The name of the node
0 spineBase The base of the spine
1 spineMid  The middle of the spine
2 neck neck
3 head Head
4 shoulderLeft Left shoulder
5 elbowLeft Left elbow
6 wristLeft Left wrist
7 handLeft The left palm
8 shoulderRight Right shoulder
9 elbowRight Right elbow
10 wristRight Right wrist
11 handRight Right palm
12 hipLeft Left fart
13 kneeLeft Left knee
14 ankleLeft Left ankle
15 footLeft His left foot
16 hipRight Right fart
17 kneeRight Right knee
18 ankleRight Right ankle
19 footRight Right foot
20 spineShoulder The lower cervical spine
21 handTipLeft The left finger ( There is nothing small in the food )
22 thumbLeft Left thumb
23 handTipRight Right finger
24 thumbRight Right thumb
 
3、 gesture , According to the test, the identification is not very accurate , It can be used in the case of low precision requirement
0 unknown Can't identify
1 notTracked Could not detect
2 open Palm
3 closed His fist
4 lasso Scissors hand , And merge the middle index finger
 
4、 Bone data
body [object] {
  bodyIndex [number]: Indexes , allow 6 people
  joints [array]: Bone nodes , Contains coordinate information , Color information
  leftHandState [number]: Left hand gestures
  rightHandState [number]: Right hand gesture
  tracked [boolean]: Whether to capture
  trackingId
 
5、kinect object
on Monitor data
open open Kinect
close close
openBodyReader Reading bone data
open**Reader Similar to the above method , Read other types of data
6、 ... and 、 Practical summary
Fire shadow somatosensory game experience summary
Next , Let me summarize TGC2016《 Naruto Tour 》 Some problems encountered in the development of somatosensory games .
1、 Before I explain , We need to understand the process of the game first .

1.1、 Start the game with gestures

1.2、 Players incarnate four generations , Run left and right to avoid nine tail attack

1.3、 Gesture “ Aoyi ”, Trigger four generations of big moves

1.4、 Users scan the QR code to get their own live photos

 
2、 Server side
The game needs player bone data ( Move 、 gesture ), Color image data ( Trigger a picture with a certain gesture ), So we need to send these two parts of data to the client . It is worth noting that , The volume of color image data is too large , It needs to be compressed .
var emitColorFrame = false;
io.sockets.on('connection', function (socket){
socket.on('startColorFrame', function(data){
emitColorFrame = true;
});
});
kinect.on('multiSourceFrame', function(frame){ // Send player bone data
io.sockets.emit('bodyFrame', frame.body); // Players take pictures
if(emitColorFrame) {
var compression = 1;
var origWidth = 1920;
var origHeight = 1080;
var origLength = 4 * origWidth * origHeight;
var compressedWidth = origWidth / compression;
var compressedHeight = origHeight / compression;
var resizedLength = 4 * compressedWidth * compressedHeight;
var resizedBuffer = new Buffer(resizedLength);
// ...
// The photo data is too big , Compression is needed to improve transmission performance
zlib.deflate(resizedBuffer, function(err, result){
if(!err) {
var buffer = result.toString('base64');
io.sockets.emit('colorFrame', buffer);
}
});
emitColorFrame = false;
}
});
kinect.openMultiSourceReader({
frameTypes: Kinect2.FrameType.body | Kinect2.FrameType.color
});

  

3、 client
The client business logic is complex , Let's extract the key steps to explain .
3.1、 When users take pictures , Because the processing data is relatively large , To prevent the page from getting stuck , We need to use web worker

(function(){
importScripts('pako.inflate.min.js'); var imageData;
function init() {
addEventListener('message', function (event) {
switch (event.data.message) {
case "setImageData":
imageData = event.data.imageData;
break;
case "processImageData":
processImageData(event.data.imageBuffer);
break;
}
});
}
function processImageData(compressedData) {
var imageBuffer = pako.inflate(atob(compressedData));
var pixelArray = imageData.data;
var newPixelData = new Uint8Array(imageBuffer);
var imageDataSize = imageData.data.length;
for (var i = 0; i < imageDataSize; i++) {
imageData.data[i] = newPixelData[i];
}
for(var x = 0; x < 1920; x++) {
for(var y = 0; y < 1080; y++) {
var idx = (x + y * 1920) * 4;
var r = imageData.data[idx + 0];
var g = imageData.data[idx + 1];
var b = imageData.data[idx + 2];
}
}
self.postMessage({ "message": "imageReady", "imageData": imageData });
}
init();
})();

  

3.2、 After the projector , If the rendering area is large , White screen will appear , Need to turn off browser hardware acceleration .

3.3、 The scene was dark , Other players interfere , In the process of tracking the player's trajectory , There may be jitters , We need to remove the interference data .( When a large displacement suddenly occurs , You need to remove the data )

var tracks = this.tracks;
var len = tracks.length; // Data filtering
if(tracks[len-1] !== window.undefined) {
if(Math.abs(n - tracks[len-1]) > 0.2) {
return;
}
}
this.tracks.push(n);

  

3.4、 When the player stands , It's just a little bit of left and right shaking , We think players are standing up .

//  Retain 5 Data 
if(this.tracks.length > 5) {
this.tracks.shift();
} else {
return;
} // Total displacement
var dis = 0;
for(var i = 1; i < this.tracks.length; i++) {
dis += this.tracks[i] - this.tracks[i-1];
}
if(Math.abs(dis) < 0.01) {
this.stand();
} else {
if(this.tracks[4] > this.tracks[3]) {
this.turnRight();
} else {
this.turnLeft();
}
this.run();
}

  

7、 ... and 、 expectation
 
1、 Use HTML5 Development Kinect Body feeling game , Lower the technical threshold , Front end engineers can easily develop somatosensory games ;
2、 A lot of frameworks can be applied , For example, use JQuery、CreateJS、Three.js( Three different rendering methods );
3、 Infinite imagination space , Think about the combination of somatosensory games webAR, combination webAudio、 Combined with mobile devices , It's so digging …… It's exciting to think about it, isn't it !

Use HTML5 Development Kinect More articles on somatosensory games

  1. OpenNI combination Unity3D Kinect Developing somatosensory games ( turn )

    OpenNI combination Unity3D Kinect Developing somatosensory games ( turn ) Landlord # more Published on :2012-07-17 16:42     1. Download and install Unity3D( The current version is 3.4)2. download OpenN ...

  2. 【CityHunter】 be based on LBS Of AR The concept of somatosensory game design

    I am not a member of the game industry at present , I don't know , The ecosystem of the game industry , I don't know , What's the so-called “ China's market environment ”. So I dare not express my opinion in this respect , But I want to put in a word here , That's what I think , Within the industry , There are still different levels between people ...

  3. Kinect Outsourcing team ( Long term commitment to Microsoft Kinect Somatosensory outsourcing , There's a big Kinect Case study )

    Undertaking Kinect Somatosensory enterprise project . Game outsourcing Beijing company . Professional team , Members are professional WPF Product company frontline developers , Experience in large scale product development : Provide excellent after-sales service , To guarantee the quality of products , Lightweight products can provide standard and clear source code , There is business ...

  4. Kinect outsource - Just look for Beijing moving point Feiyang software ( Long term commitment to Microsoft Kinect Somatosensory outsourcing , There's a big Kinect Case study )

    Undertaking Kinect Somatosensory enterprise project . Game outsourcing   There are plenty of cases to demonstrate , We can sign a formal contract , draw a bill . We are a company in Beijing . Professional team , Members are professional WPF Product company frontline developers , Experience in large scale product development : Provide excellent after-sales service ...

  5. [ Body feeling game ] 1、MPU6050 Data acquisition, transmission and visualization

    I've been studying somatosensory games recently , So far, we have implemented a system based on 51 Single chip microcomputer MPU6050 Data collection . Use Bluetooth module to transmit data to PC , And make use of C# Self made high speed serial data acquisition software , And display the data visually through the self-made line drawing module ...

  6. Use unity3d and tensorflow Realize the body sense game based on attitude estimation

    Use unity3d and tensorflow Realize the body sense game based on attitude estimation Preface Do attitude recognition before , I dream that I can make a sense of body game by myself , However, it was later discovered that too young. But dreams still have to be , What if it does . Taking the advantage of p ...

  7. Make Kinect Somatosensory control car tutorial &amp;lt; One &amp;gt;

    Reprint please indicate the source :http://blog.csdn.net/lxk7280                                        Kinect Somatosensory control car        Kine ...

  8. scratch Programming somatosensory games

    There are many kinds of somatosensory games , The most common ones are cameras and voice control , Today we're going to use scratch Write a series of somatosensory games !!! Isn't it exciting ? First of all, let's edit the camera class : No.1 Punch the ghost You can hit the ghost with a wave of your head ! The specific procedures are as follows ...

  9. C# Developing somatosensory games Kinect Applied knowledge

    Kinect The first is a XBox 360 External somatosensory devices , Capture motion perception wirelessly . from PrimeSense Provide Range Camera technology , Similar products like Nintendo Wii.Play Station Move, Must let ...

Random recommendation

  1. eclipse Boot failure ( Can't find jvm)

    Launched today eclipse It's a mistake The solution found online is in eclipse Add Java Of JVM The path of , The method is as follows : Right click eclipse Shortcut -> attribute In the goal   If only  D: ...

  2. java Programming idea -java IO System

    One . Input and output Programming language I/O The abstract concept of flow is often used in class libraries , It represents any data source object capable of producing data or any receiver object capable of receiving data ." flow " Shield the actual I/O Details of processing data in the device . J ...

  3. picurl

    from lxml import etreeimport requestsdef getHtml(html): novelcontent = requests.get(html).content re ...

  4. Qt Of QTabelWidget

    QTableWidget To sum up the usage of   http://blog.csdn.net/zb872676223/article/details/39959061 [Qt] stay QTableWidget Add QCh ...

  5. C Language sorting algorithm

    (1)“ Bubbling ” Bubble method is familiar to everyone . The principle is from a[0] Start , Compare it with the following elements in turn , if a[0]>a[i], Then exchange them , All the way to a[n]. Same reason a[1],a[2],...a[n-1] Handle , namely ...

  6. springMVC+MyBatis+Spring Integrate (2)

    mybatis And Spring Integration of . 1. Import Spring and Springmvc My bag pom <project xmlns="http://maven.apache.org/POM ...

  7. WiFi And WLAN The difference between

    When many people go to a certain place, the first thing they want to look for is wireless network , Sometimes I ask people around me : Here you are WiFi Do you ? Or say : Here you are WLAN Do you ? that WiFi and WLAN What's the difference ? simply ,WiFi It's Wi Fi (wireless fid ...

  8. ( 5、 ... and )backbone - DEMO - The use of address book transformation requirejs

    DEMO The introduction is DEMO The extension of the address book , Use requirejs Modular integration In general • model file model/contact.js define(function (){ // user cont ...

  9. Apache Lucene

    1.Lucene  - Full text search engine Apache Lucene It's based on Java Full text search engine , It can be used easily for Java Software add � The function of full text search engine . Lucene The most important job is to index every word of the file , ...

  10. use CSS3 Media Query Technology adapts to Android Flat screen resolution and screen pixel density

    use HTML5 In the development of mobile applications to meet a variety of needs Android Resolution and tablet density of the screen , It's a very troublesome process , The ultimate solution is to use css media query, Match the same time resolution and screen pixel density . on ...