Kinect v2 Introduction and Tutorial

Introduction to Kinect v2
Tsukasa Sugiura
@UnaNancyOwen

Self-Introduction
Tsukasa Sugiura
Microsoft MVP for Kinect for Windows
 @UnaNancyOwen
 http://UnaNancyOwen.com
 t.sugiura0204@gmail.com
(July 2014 - June 2015)

Agenda
 What is Kinect v2
 Specifications
 New Features
 Demo
 Tutorial

Disclaimer
“This is preliminary software and/or hardware and APIs
are preliminary and subject to change.”
（Kinect for Windows v2は暫定的なものであり、ソフトウェア、ハード
ウェア、およびAPIは製品版で変更される可能性があります。）

Kinect for Windows v1 Sensor
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
RGB CAMERA

MULTI-ARRAY MIC
3D DEPTH SENSOR
( IR Camera + IR Emitters )
RGB CAMERA

Image by iFixit
IR EMITTERS
IR CAMERA

Specifications
Kinect for Windows v1 Kinect for Windows v2
Color 640×480 @ 30fps 1920×1080 @ 30fps
Depth 320×240 @ 30fps 512×424 @ 30fps
Sensor Structured Light
(PrimeSense Light Coding)
Time of Flight
(ToF)
Range 0.8~4.0 m 0.5~4.5 m
Angle of View
Horizontal / Vertical
57 / 43 degree 70 / 60 degree
Microphone Array ◯ ◯

Specifications
BodyIndex 6 people 6 people
Body 2 people 6 people
Joint 20 joint/people 25 joint/people
Hand State Open / Closed Open / Closed / Lasso
Gesture ☓ ◯
Face ◯
Speech / Beamforming ◯ ◯

Basic Features
 Color
 1920×1080@30fps / 15fps (Lighting Condition)
 RGBA, YUV, BGRA, Bayer, YUY2

Basic Features
 Depth
 512×424@30fps
 500~4500[mm]
 ToF (Time of Flight)

Basic Features
 Infrared / LongExposureInfrared
 512×424@30fps
 16bit (higher 8 bits)

Basic Features
 BodyIndex
 512×424@30fps
 6 people
 Body Area : 0~5, Other Area : 255 (5 < Index)
255
0 1

Basic Features
 Body
 6 people
 25 joint / people (Add Tip, Thumb, Neck)
 Orientation (Quaternion)
 Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)

Basic Features
 Audio
 Beamforming (+/-50 degree)
 Speaker Estimation
 Speech Recognition

Application Features
 Gesture
 Gesture Recognition using Machine Learning
 Discrete (detected true/false), Continuous (progress 0.0f~1.0f)
 Learning Classifier Tool “Visual Gesture Builder”
Video by http://youtu.be/-XYoblrnDpg

 Face
 Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)
 Activity, Appearance, Expression
 Activity … Eye Closed, Mouth Open / Moved, Looking Away
 Appearance … Wearing Glasses
 Expression … Happy

 HDFace
 For Creating 3D Face Model
 Points (1347), Triangles (2340), Hair Color, Skin Color
 Fitting Face Model

 Other
 Kinect Fusion (3D Shape Reconstruction)
 Controls (Assist in implementation of NUI)

System / Software Requirements
OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU * DirectX 11 supported
USB * USB 3.0 (Intel or Renesas Host Controller)
Compiler * Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

Connection
PCPC

Example Multiple Connection
PCPC PC
Hub
Server

System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kinect SDK
Application

Tutorial
 Basic Flow of Programming (C++)
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
 Source independent to each Data
(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
 Doesn’t depend on each other Source
(e.g. Doesn't need to Depth Source when retrieve Body Data)

Tutorial
 Sensor
// Sensor
IKinectSensor* pSensor;
HRESULT hResult = S_OK;
hResult = GetDefaultKinectSensor( &pSensor );
if( FAILED( hResult ) ){
std::cerr << "Error : GetDefaultKinectSensor" << std::endl;
return -1;
}
hResult = pSensor->Open();
std::cerr << "Error : IKinectSensor::Open()" << std::endl;
return -1;
}
 IKinectSensor

Tutorial
 Source
// Source
I***FrameSource* pSource;
hResult = pSensor->get_***FrameSource( &pSource );
std::cerr << "Error : IKinectSensor::get_FrameSource()" << std::endl;
return -1;
}
 IColorFrameSource
 IDepthFrameSource
 IInfraredFrameSource
 IBodyIndexFrameSource
 IBodyFrameSource

Tutorial
 Reader
// Reader
I***FrameReader* pReader;
hResult = pSource->OpenReader( &pReader );
std::cerr << "Error : IFrameSource::OpenReader()" << std::endl;
return -1;
}
 IColorFrameReader
 IDepthFrameReader
 IInfraredFrameReader
 IBodyIndexFrameReader
 IBodyFrameReader

Tutorial
 Frame
while( 1 ){
// Frame
I***Frame* pFrame = nullptr;
hResult = pReader->AcquireLatestFrame( &pFrame );
if( SUCCEEDED( hResult ) ){
/* Data */
}
SafeRelease( pFrame );
}
 IColorFrame
 IDepthFrame
 IInfraredFrame
 IBodyIndexFrame
 IBodyFrame

Tutorial
 Data (Depth, Infrared, BodyIndex)
while( 1 ){
// Frame
IDepthFrame* pDepthFrame = nullptr;
hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame );
unsigned int bufferSize = 0;
unsigned short* pBuffer = nullptr;
hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );
/* Processing*/
}
}
SafeRelease( pDepthFrame );
}
 Depth, Infrared … unsigned short
 BodyIndex … unsigned char

Tutorial
 Data (Color)
while( 1 ){
// Frame
IColorFrame* pColorFrame = nullptr;
hResult = pColorReader->AcquireLatestFrame( &pColorFrame );
unsigned int bufferSize = 0;
unsigned char* pBuffer = nullptr;
hResult = pColorFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );
/* Processing*/
}
}
SafeRelease( pColorFrame );
}
 Color … unsigned char
 Default Color Image Format … YUY2

Tutorial
 Data (Color with Converted Format)
unsigned int buffeerSize = width * height * channel * sizeof(unsigned char);
unsigned char* pBuffer = new unsigned char[bufferSize];
while( 1 ){
// Frame
IColorFrame* pColorFrame = nullptr;
hResult = pReader->AcquireLatestFrame( &pColorFrame );
hResult = pColorFrame->CopyConvertedFrameDataToArray( bufferSize, pBuffer,
ColorImageFormat::ColorImageFormat_Bgra );
/* Processing */
}
}
SafeRelease( pColorFrame );
}
delete[] pBuffer;

Tutorial
while( 1 ){
IBodyFrame* pFrame = nullptr;
hResult = pBodyReader->AcquireLatestFrame( &pBodyFrame );
IBody* pBody[BODY_COUNT] = { 0 };
hResult = pBodyFrame->GetAndRefreshBodyData( BODY_COUNT, pBody );
/* Processing */
}
for( int count = 0; count < BODY_COUNT; count++ ){
SafeRelease( pBody[count] );
}
}
SafeRelease( pBodyFrame );
}
 Data (Body)

Tutorial
for( int count = 0; count < BODY_COUNT; count++ ){
BOOLEAN bTracked = false;
hResult = pBody[count]->get_IsTracked( &bTracked );
if( SUCCEEDED( hResult ) && bTracked ){
Joint joint[JointType::JointType_Count];
hResult = pBody[count]->GetJoints( JointType::JointType_Count, joint );
for( int type = 0; type < JointType::JointType_Count; type++ ){
if( joint[type].TrackingState != TrackingState::TrackingState_NotTracked ){
CameraSpacePoint cameraSpacePoint = joint[type].Position;
cameraSpacePoint.x; // x (+/- 1.0f)
cameraSpacePoint.y; // y (+/- 1.0f)
cameraSpacePoint.z; // z (500~4500[mm])
}
}
}
}
}
 Data (Joint)

Tutorial
 Coordinate System
 ColorSpace (Coordinate System of the Color Image)
… Color
 DepthSpace (Coordinate System of the Depth Data)
… Depth, Infrared, BodyIndex
 CameraSpace (Coordinate System with the origin located the Depth Sensor)
… Body (Joint)

Tutorial
 Coordinate Mapper
// Coordinate Mapper
ICoordinateMapper* pCoordinateMapper;
hResult = pSensor->get_CoordinateMapper( &pCoordinateMapper );
std::cerr << "Error : IKinectSensor::get_CoordinateMapper()" << std::endl;
return -1;
}
 ICoordinateMapper::Map○○○FrameTo△△△Space()
… Mapping Coordinate System in the Frame
 ICoordinateMapper::Map○○○PointsTo△△△Space()
… Mapping Coordinate System in the Array of Points
 ICoordinateMapper::Map○○○PointTo△△△Space()
… Mapping Coordinate System in the Point

Reference
 Sample Program
 Kinect2Sample | GitHub
https://github.com/UnaNancyOwen/Kinect2Sample
 Article Series
 Introduction to Kinect for Windows v2 | Build Insider
http://www.buildinsider.net/small/kinectv2cpp
 Blog
 Kinect | Summary?Blog
http://unanancyowen.com/?cat=3

Kinect v2 Introduction and Tutorial

More Related Content

What's hot (20)

Similar to Kinect v2 Introduction and Tutorial (20)

More from Tsukasa Sugiura (6)

Recently uploaded (20)

Kinect v2 Introduction and Tutorial