Building Applications with the Microsoft Kinect SDK
9 likes2,225 views
The document provides an overview of building applications using the Microsoft Kinect SDK, focusing on Natural User Interface (NUI) concepts, system requirements, and development tips for creating custom Kinect applications. It covers essential features such as camera functionality, depth data, skeletal tracking, and audio capabilities, along with relevant programming resources and quickstart guides. The presentation highlights the potential applications of Kinect technology in various fields and encourages developers to explore its capabilities.
![KINECT FOR WINDOWS
GRAMMAR
<!-- Confirmation_Yes._value: string ["Yes"]
-->
<rule id="Confirmation_Yes" scope="public">
<example> yes </example>
<example> yes please </example>
<one-of>
<item> yes </item>
<item> yeah </item>
<item> yep </item>
<item> ok </item>
</one-of>
<item repeat="0-1"> please </item>
<tag> out._value = "Yes";</tag>
var grammar = new Choices();
grammar.Add(“yes please");
grammar.Add(“yes");
grammar.Add(“yeah");
grammar.Add(“ok");](https://image.slidesharecdn.com/kinectallnuitomefinal-120422172559-phpapp01/85/Building-Applications-with-the-Microsoft-Kinect-SDK-46-320.jpg)
Building Applications with the Microsoft Kinect SDK
- 1. KINECT FOR WINDOWS It’s All NUI to me: Building Apps with the Microsoft Kinect SDK David Silverlight Founder and Chief Software Architect TheSilverlightGroup.com IsYourWebMasterDead.com
- 2. KINECT FOR WINDOWS Agenda • NUI – What? • Demo’s – See what all the hullabaloo is about • Getting Started/Kinect Explorer/QuickStarts • Building Your Own Kinect apps. • (Video, Depth, Skeleton, Audio) • Resources
- 3. KINECT FOR WINDOWS Kinect: Setter of World Records!
- 4. KINECT FOR WINDOWS And Some Disasters
- 5. KINECT FOR WINDOWS It’s in the headlines
- 6. KINECT FOR WINDOWS What is NUI? • Natural User Interface • Common term used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users
- 7. KINECT FOR WINDOWS What is NUI? • An Invisible Interface. • The Content is the Interface. • Removing the Proxy • Direct Manipulation • Gestural interfaces
- 8. KINECT FOR WINDOWS Seeing is believing – Kinect Development Allows you to capture your movements and opens up a whole new world to animators on a tight budget. 12 Best Kinect Hacks Personal Data Mirror Catch up with the latest headlines, weather and health all whilst brushing your teeth Real Time Motion Capture with XNA Sample showing just what people are doing with the Kinect
- 9. KINECT FOR WINDOWS Fitnect Virtual Vacation Now another reason to never leave the house. Interactive Fitting room showing how Kinect and business can connect. Seeing is believing – Kinect Development
- 10. KINECT FOR WINDOWS What can you do with Kinect on a PC or Mac ?
- 11. KINECT FOR WINDOWS RGB CAMERA MULTI-ARRAY MIC MOTORIZED TILT 3D DEPTH SENSORS
- 12. KINECT FOR WINDOWS Getting Started: Building Your Own Apps • Designed to get you started • Assumes you have some programming experience • Managed API - Concepts work in both VB/C#
- 13. KINECT FOR WINDOWS Kinect USB cable
- 14. KINECT FOR WINDOWS System Requirements • You must run applications that are built by using the Kinect for Windows SDK in a native Windows environment. You cannot run applications in a virtual machine, because the Microsoft Kinect drivers and this SDK must be installed on the computer that is running the application. • Supported Operating Systems and Architectures • Windows 7 (x86 or x64) • Hardware Requirements • Computer with a dual-core, 2.66-GHz or faster processor • Windows 7–compatible graphics card that supports Microsoft® DirectX® 9.0c capabilities • 2 GB of RAM • Kinect for Xbox 360® sensor—retail edition, which includes special USB/power cabling
- 15. KINECT FOR WINDOWS System Requirements • Software Requirements • Microsoft Visual Studio® 2010 Express or other Visual Studio 2010 edition • Microsoft .NET Framework 4.0 (installed with Visual Studio 2010) • For C++ SkeletalViewer samples: • DirectX Software Development Kit, June 2010 or later version • DirectX End-User Runtime Web Installer • For Speech sample (x86 only): • Microsoft Speech Platform - Server Runtime, version 10.2 (x86 edition) • Microsoft Speech Platform - Software Development Kit, version 10.2 (x86 edition) • Kinect for Windows Runtime Language Pack, version 0.9 (acoustic model from Microsoft Speech Platform for the Kinect for Windows SDK Beta)
- 16. KINECT FOR WINDOWS It all starts here: http://www.microsoft.com/en-us/kinectforwindows/ Developing your own Kinect aps
- 17. KINECT FOR WINDOWS Kinect for Windows SDK Quickstarts • http://channel9.msdn.com/Series/KinectQuickstart - All are great, IMHO • The Kinect Explorer is new from the Beta • Kinect for Windows SDK Browser. • Installing and Using the Kinect Sensor • Setting up your development environment • Skeletal Tracking fundamentals • Camera Fundamentals • Working with Depth Data • Audio Fundamentals
- 18. KINECT FOR WINDOWS SETTING UP YOUR DEVELOPMENT ENVIRONMENT
- 19. KINECT FOR WINDOWS Setting up your Development Environment Add our references to: - The Microsoft.Samples.Kinect.WpfViewers Project - The Coding4Fun.Wpf.dll Reference the assembly: • using Microsoft.Kinect;
- 20. KINECT FOR WINDOWS KINECT API BASICS • Manage Kinect state • Connected • Enable Color, Depth, Skeleton • Start Kinect • Get Data • Events - AllFramesReady • Polling – OpenNextFrame
- 21. KINECT FOR WINDOWS RESOURCES • Install Kinect Explorer • KinectWpfViewers • Coding4Fun Toolkit • Skeletal scaling
- 22. KINECT FOR WINDOWS SETTING UP YOUR DEV ENVIRONMENT DEMO
- 23. KINECT FOR WINDOWS Setting the Camera Tilt Angle CAMERA FUNDAMENTALS
- 24. KINECT FOR WINDOWS Setting the Camera Tilt Angle • Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function. • To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation: • kinectSensorChooser.Kinect.ElevationAngle = Convert.ToInt16(slider1.Value);
- 26. KINECT FOR WINDOWS RESOLUTIONS • Color • 12 FPS: 1280X960 RGB • 15 FPS: Raw YUV 640x480 • 30 FPS: 640x480 • Depth • 30 FPS: 80x60, 320x240, 640x480
- 27. KINECT FOR WINDOWS Camera Data
- 28. KINECT FOR WINDOWS CAMERA DATA • Events return ImageFrame • PixelDataLength • FrameNumber • Timestamp • Dimensions: Height, Width • Use AllFramesReady event to synchronize
- 29. KINECT FOR WINDOWS CAMERA FUNDAMENTALS DEMO
- 30. KINECT FOR WINDOWS WORKING WITH DEPTH DATA
- 31. KINECT FOR WINDOWS DEPTH DATA • Returns the distance and player for every pixel • Ex: 320x240 = 76,800 pixels • Distance • Distance in mm from Kinect ex: 2,000mm (6.56 feet) • Player • 1-6 players
- 32. KINECT FOR WINDOWS MODE Mode Depth & Player Center Hip Joint Other 19 Joints Default Yes Yes Yes Near Yes Yes No, for v1.0
- 33. KINECT FOR WINDOWS DISTANCES 1.3’ 2.6’ 9.8’ 13.1’ 26.2’ .4 .8 3 4 8 Default Mode Near Mode Feet Meters
- 34. KINECT FOR WINDOWS FORMULAS • Distance Formula int depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth; • Player Formula int player = depthPoint & DepthImageFrame.PlayerIndexBitmask;
- 35. KINECT FOR WINDOWS WORKING WITH DEPTH DATA DEMO
- 37. KINECT FOR WINDOWS Joints • Maximum two players tracked at once • Six player proposals • Each player with set of <x, y, z> joints in meters • Each joint has associated state • Tracked, Not tracked, or Inferred • Inferred - Occluded, clipped, or low confidence joints • Not Tracked - Rare, but your code must check for this state
- 38. KINECT FOR WINDOWS Skeleton Data
- 39. KINECT FOR WINDOWS Skeleton API
- 40. KINECT FOR WINDOWS SKELETAL TRACKING DEMO
- 42. KINECT FOR WINDOWS QUICKSTART SERIES • Installing and Using the Kinect Sensor • Setting up your Development Environment • Camera Fundamentals • Working with Depth Data • Skeletal Tracking Fundamentals • Audio Fundamentals
- 43. KINECT FOR WINDOWS SOUND POSITION • Sound Source Angle – the angle and confidence level of where audio is coming from • Beam Angle – The angle used to record audio that you can set as a “directional microphone” -50 0 +50
- 44. KINECT FOR WINDOWS RECORDING • KinectAudioSource audio buffer • Use Echo Cancellation to not record PC sound
- 45. KINECT FOR WINDOWS SPEECH RECOGNITION • Grammar – What we are listening for • Code – GrammarBuilder, Choices • Note: Ensure AutomaticGainControl = false
- 46. KINECT FOR WINDOWS GRAMMAR <!-- Confirmation_Yes._value: string ["Yes"] --> <rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes please </example> <one-of> <item> yes </item> <item> yeah </item> <item> yep </item> <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag> var grammar = new Choices(); grammar.Add(“yes please"); grammar.Add(“yes"); grammar.Add(“yeah"); grammar.Add(“ok");
- 47. KINECT FOR WINDOWS MULTI-MODAL FEEDBACK • Engagement model – Mic indicator for speech-enabled menus – Keyword to engage • Feedback and confirmation
- 48. KINECT FOR WINDOWS Audio Data
- 50. KINECT FOR WINDOWS The Future of Kinect • Better close up
- 51. KINECT FOR WINDOWS Gesture Libraries
- 52. KINECT FOR WINDOWS Gesture Libraries
- 53. KINECT FOR WINDOWS Kinect version of Siri (other CodePlex)
- 55. KINECT FOR WINDOWS Resources • Coding 4 Fun - Coding4Fun Kinect Toolkit • OpenSource Projects • SilverlightKinect
- 56. KINECT FOR WINDOWS What we have learned • Examples of the amazing things that can be done with the Kinect • How amazing the Kinect Device is, equivalent to $200,000 in capabilities • How to create our own Kinect applications, Video, Skeletal and Audio tracking. • The future of Kinect
- 57. KINECT FOR WINDOWS Thank You • David Silverlight • David@TheSilverlightGroup.com