Jancke kinect programming

1. Programming with the Kinect for Windows SDK

2. What we’ll cover  Kinect Sensor  Using Cameras  Understanding Depth Data  Skeletal Tracking  Audio

3. 3D DEPTH SENSORS RGB CAMERA MULTI-ARRAY MIC MOTORIZED TILT

4. SDK Architecture Applications Video Components Audio Components Windows Core Audio 5 3 NUI API and Speech APIs DMO codec for mic array 4 2 Device Device Video stream control Audio stream control User Mode setup access WinUSB device stack WinUSB camera stack USBAudio audio stack Kernel Mode Kernel-mode drivers for Kinect for Windows USB Hub Hardware 1 Motor Cameras Audio mic array Kinect sensor Kinect for Windows User-created Windows SDK components components

9. Using Cameras Demos

10. Understanding Depth Data  ImageFrame.Image.Bits  Array of bytes - public byte[] Bits;  Array – Starts at top left of image – Moves left to right, then top to bottom – Represents distance for pixel in millimeters

11. Calculating Distance  2 bytes per pixel (16 bits)  Depth – Distance per pixel – Bitshift second byte by 8 – Distance (0,0) = (int)(Bits[0] | Bits[1] << 8); – VB (int)(CInt(Bits(0)) Or CInt(Bits(1)) << 8);  DepthAndPlayer Index – Includes Player index – Bitshift by 3 first byte (player index), 5 second byte – Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5); – VB:(int)(CInt(Bits(0)) >> 3 Or CInt(Bits(1)) << 5);

12. Depth Reference  Distance Range: 850 mm to 4000 mm range  Depth value 0 means unknown – Shadows, low reflectivity, and high reflectivity among the few reasons  Player Index – 0 – No player – 1 – Skeleton 0 – 2 – Skeleton 1 – …

13. Demos

14. Skeleton Data

15. Joints  Maximum two players tracked at once – Six player proposals  Each player with set of <x, y, z> joints in meters  Each joint has associated state – Tracked, Not tracked, or Inferred  Inferred - Occluded, clipped, or low confidence joints  Not Tracked - Rare, but your code must check for this state

16. Skeletal Tracking

17. Demos

18. Audio Going Inside of Kinect  Four microphone array with hardware-based audio processing – Multichannel echo cancellation (MEC) – Sound position tracking – Other digital signal processing (noise suppression and reduction)

19. Audio  Kinect as a microphone  Kinect for Speech Recognition

20. Multi-modal Feedback  Engagement model – Mic indicator for speech-enabled menus – Keyword to engage  Feedback and confirmation (both passive and active)

21. Speech Recognition  Kinect Grammar available to download  Grammar – What we are listening for – Code – GrammarBuilder, Choices – Speech Recognition Grammar Specification (SRGS)  C:Program Files (x86)Microsoft Speech Platform SDKSamplesSample Grammars

22. Grammar  --> <rule id="Confirmation_YesNo" scope="public"> <rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes </example> <example> no </example> <example> yes please </example> <one-of> <one-of> <item> <item> yes </item> <ruleref uri="#Confirmation_Yes" /> <item> yeah </item> </item> <item> yep </item> <item> <item> ok </item> <ruleref uri="#Confirmation_No" /> </one-of> </item> <item repeat="0-1"> please </item> </one-of> <tag> out = rules.latest() </tag> <tag> out._value = "Yes";</tag> </rule> </rule>

23. Demos

24. Resources  Kinect Programming Walkthroughs – http://research.microsoft.com/kinectsdk/  Coding4Fun Kinect Toolkit – http://c4fkinect.codeplex.com  Kinect SDK Quickstarts – http://channel9.msdn.com/series/KinectSDKQuickstarts

Jancke kinect programming

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Jancke kinect programming (20)

Recently uploaded (20)

Jancke kinect programming