Project Gesture & RealSense: gestures in a simple way!!
Download as PPTX, PDF1 like930 views
The document presents Project Prague, an SDK by Microsoft for developers to incorporate gesture-based controls into applications, utilizing Intel RealSense cameras. It outlines technical requirements, gesture detection capabilities, and the architecture for registering and detecting gestures. The initiative aims to enhance human-computer interaction through an experimental gesture detection service, with potential future developments indicated.
![Requirements
Category Recommended Minimal
CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores
Free RAM 2GB or more 1GB
Operating System Windows 10 with Creator Update installed Windows 10
Brand Gesture Detection Range Experience
Intel® RealSense™ SR300 camera 20-60 [cm] Best
Intel® RealSense™ F200 camera 20-60 [cm] Good
Kinect for Windows v2 60-110 [cm] Good
http://aka.ms/gestures/download](https://image.slidesharecdn.com/projectpraguerealsense-gesturesinasimpleway-180320173039/85/Project-Gesture-RealSense-gestures-in-a-simple-way-8-320.jpg)
![Project Gesture Web Site
https://labs.cognitive.microsoft.com/en-us/project-gesture
Project Gesture documentation
https://docs.microsoft.com/en-us/gestures/
Twitter/mail account
@ProjectPrague / gestures@microsoft.com
Project Gesture GitHub
https://github.com/Microsoft/Gestures-Samples
Toward Realistic Hands Gesture Interface: Keeping it Simple for
Developers and Machines [PDF]
https://aka.ms/TRHGI
Sample GitHub
https://github.com/massimobonanni/ProjectPragueSamples
Project Prague & RealSense: il potere nelle mani!!! (dotNet{podcast}
meet{cast})
http://bit.ly/ProjectPragueMeetCast
Reference](https://image.slidesharecdn.com/projectpraguerealsense-gesturesinasimpleway-180320173039/85/Project-Gesture-RealSense-gestures-in-a-simple-way-28-320.jpg)
Project Gesture & RealSense: gestures in a simple way!!
- 1. Sviluppo di applicazioni per Mobile, Web ed IoT Cross-Platform & Open Source #dotnetday2018
- 2. M assimo Bonanni Paranormal Developer, with the head in the Cloud and all the REST in microservices! massimo.bonanni@tiscali.it @massimobonanni Project Prague & RealSense: gestures in a simple way!!
- 4. “… the idea of gesture input will become mainstream – both when working far away from the screen (TV, big meeting room) and when working at your desk (PC).” Microsoft’s Focused Dreams to Survive and Thrive, Bill Gates, January 2016
- 6. What is Project Prague? Is an easy-to-use SDK that helps developers to incorporate gesture- based controls into their apps. Use a machine learning algorithm called DFE (Discriminative Ferns Ensemble) to obtain the hand skeleton. The gesture detection is implemented using a straight forward heuristic, with very few manually tuned parameters. Gesture
- 7. What do I need? Intel® RealSense™ SR300 https://aka.ms/IntelRS300 Intel® RealSense™ F200
- 8. Requirements Category Recommended Minimal CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores Free RAM 2GB or more 1GB Operating System Windows 10 with Creator Update installed Windows 10 Brand Gesture Detection Range Experience Intel® RealSense™ SR300 camera 20-60 [cm] Best Intel® RealSense™ F200 camera 20-60 [cm] Good Kinect for Windows v2 60-110 [cm] Good http://aka.ms/gestures/download
- 10. Palm Pose Finger Poses Fingertip Placement Relations + + Fingertip Distance Relations+ HandPose = Direction: {left, right, up, down, backward, forward} Flexion: {opened, openedStretched, folded, foldedTucked} RelativePlacement: {left, right, above, below, behind, inFront} RelativeDistance: {notTouching, touching} Direction: {left, right, up, down, backward, forward} Orientation: {left, right, up, down, backward, forward} Hand Pose A hand pose refers to a snapshot of the hand at a given moment and contains a complete description of the state of the palm and the fingers in that snapshot
- 11. Pinch pose Hand Pose palm facing {forward}, orientation {up} {index, thumb} are {touching} {middle, ring, pinky} are {above} {index, thumb} {index} is {above} the {thumb} {left} hand {thumb, index, middle, ring, pinky} are {opened}
- 12. <svg> <path d="M 100 350 q 50 -50 100 0 q 50 -50 100 0 q 50 -50 100"/> </svg> HandMotion = MotionScript Upward, Downward, Right,Left, ClockwiseArcRightUpward, ClockwiseArcRightDownward, ClockwiseArcLeftUpward, ClockwiseArcLeftDownward, CounterClockwiseArcLeftDownward, CounterClockwiseArcLeftUpward, CounterClockwiseArcRightDownward, CounterClockwiseArcRightUpward, For example, the SVG curve above can be simply described by: ClockwiseArcRightUpward, ClockwiseArcRightDownward, ClockwiseArcRightUpward, ClockwiseArcRightDownward, ClockwiseArcRightUpward, ClockwiseArcRightDownward, DiagonalLeftUpward, DiagonalRightDownward, DiagonalRightUpward, DiagonalLeftDownward, Hand Motion As you move your hand, your fingertips trace a curve through space: this curve is a "hand motion"
- 13. A Gesture is a finite state machine that its states are either Pose objects, Motion objects or other Gesture object Basic templates: A gesture should be Un/Registered with respect to the application context. Only registered gestures trigger ( ). Gesture Pose Motion Pose Pose Pose Gesture = Gesture Pose Gesture Motion
- 14. hand facing {forward}, orientation {up} {index, thumb} are {touching} {index} is {above} {thumb} {left, right} hand {thumb, index} are {opened} Pinch Retract NotPinchingSlingshot = {index, thumb} are {opened} {left, right} hand {index, thumb} are {notTouching} {index, thumb} are {opened} {left, right} hand {index, thumb} are {notTouching} move {backward} {left, right} hand NotPinching Slingshot Example
- 16. Gestures Service The Gestures Service provides gesture detection as a local service for client applications. After you install Project Prague, the Gestures Service will be launched every time your machine starts.
- 18. Using Gesture Service The detection is performed in the Microsoft.Gestures.Service.exe process that acts as a service for gesture detection. You have to use the GesturesServiceEndpoint class in order to communicate with the service.
- 19. Create a gesture
- 20. Detect a gesture You need to hook up the event indicating gesture detection to the appropriate handler: You can detect when the single pose is triggered by the user:
- 21. Declare a Gesture as XAML
- 22. Skeleton tracking Project Gesture also provides API for the skeleton tracking. For each hand: Position of the fingertips Fingers direction Palm position Palm orientation
- 24. V1 Released July ‘17 Runtime ◦ Supported sensors: Intel RealSense SR300 & F200, Kinect v2 ◦ Gestures support injected into ◦ Windows Shell, PowerPoint, Skype, YouTube and Photos App ◦ Wolfly Unity Game ◦ Camera Gesture-triggered overlays (can be used in skype) Docs ◦ Great Overview + tutorials Samples @Github ◦ .NET ◦ Unity
- 26. Key takeaways Human Interaction is the new challenge. Project Gesture may be one of the solution. In the roadmap will be different kind of camera (e.g. IR camera) Keep in mind: Project Gesture is an experimental project. If we’ll use it, it will become a supported project.
- 28. Project Gesture Web Site https://labs.cognitive.microsoft.com/en-us/project-gesture Project Gesture documentation https://docs.microsoft.com/en-us/gestures/ Twitter/mail account @ProjectPrague / gestures@microsoft.com Project Gesture GitHub https://github.com/Microsoft/Gestures-Samples Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines [PDF] https://aka.ms/TRHGI Sample GitHub https://github.com/massimobonanni/ProjectPragueSamples Project Prague & RealSense: il potere nelle mani!!! (dotNet{podcast} meet{cast}) http://bit.ly/ProjectPragueMeetCast Reference
- 29. GRAZIE A TUTTI VOI … #DOTNETDAY2018 33