Gesture Based Computing
6 likes2,304 views
The document discusses gesture-based computing, highlighting the differences between touchscreen and freeform gestural interfaces that detect physical movements without traditional pointing devices. It emphasizes the principles of direct manipulation and Fitts' law, noting the importance of target size and proximity for effective gesture recognition. Additionally, it explores challenges with complex gestures and the limitations of current gesture vocabularies, suggesting that less complicated gestures are more user-friendly.
Gesture Based Computing
- 1. GESTURE BASED COMPUTING Paul Coulton Banksy
- 2. GESTURE BASED COMPUTING Paul Coulton O’Sullivan and Igoe Banksy
- 3. GESTURAL INTERFACES Allows physical movements to be detected by a digital system Dan Saffer without the aid of a traditional pointing device.
- 4. A wave, a head nod, a touch, a toe tap, or even a raised eyebrow could be a gesture GESTURAL INTERFACES Allows physical movements to be detected by a digital system Dan Saffer without the aid of a traditional pointing device.
- 5. TANGIBLE INTERFACES Providing a physical form of digital information and facilitates the direct manipulation of the associated bits
- 6. Text Jesper Juul MIMETIC INTERFACES Mimetic interfaces require players to perform actions that closely resemble the physical activity required in reality.
- 7. NATURAL USER INTERFACES “natural refers to the user’s behaviour and feeling during the experience rather than being the product of some organic process” Wigdor and Wixon and indeed suggest a natural experience “is NOT best achieved through mimicry”
- 8. TYPES OF GESTURAL INTERFACE Most gestural interfaces can be categorised as either touchscreen or free form. Touchscreens require the user to be touching device directly whereas freeform systems do not. Dan Saffer
- 9. DIRECT VS INDIRECT MANIPULATION
- 10. Direct manipulation features a natural representation of task objects and actions promoting the notion of people performing a task themselves (directly) not through an intermediary. DIRECT VS INDIRECT MANIPULATION
- 11. MICE VS FINGERS Dan Saffer
- 12. A cursor is often unnecessary since the user is not constantly pointing at something. Finger moves from point to point whereas mouse makes a trail. MICE VS FINGERS Dan Saffer
- 13. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. MICE VS FINGERS Dan Saffer
- 14. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but should be used with caution. A threshold has to be set during which two touch events at same location count a double click. MICE VS FINGERS Dan Saffer
- 15. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to go which two touch events at same away from direct manipulation location count a double click. philosophy MICE VS FINGERS Dan Saffer
- 16. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to go which two touch events at same away from direct manipulation location count a double click. philosophy Drop down menus generally don’t work very well for same reason as right click menus combined with limitations over hover. MICE VS FINGERS Dan Saffer
- 17. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to go which two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is now Drop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingers MICE VS FINGERS Dan Saffer
- 18. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to go which two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is now Drop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingers As humans have a limited number of fingers we have a limit on selecting multiple items. Normally this means some form of select mode is used. MICE VS FINGERS Dan Saffer
- 19. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to go which two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is now Drop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingers As humans have a limited number Its hard to do an Undo gesture of fingers we have a limit on once a gesture is done therefore selecting multiple items. Normally its better to have an easy way to this means some form of select cancel or directly undo the mode is used. action. MICE VS FINGERS Dan Saffer
- 20. FITTS LAW
- 21. Fitts's law is a model of human movement relating to pointing that predicts that the time required to rapidly move to a target area is a function of the distance to the target and the size of the target. FITTS LAW
- 22. Put simply a large object closer to the user is easier to point to than a large one far away. Fitts's law is a model of human movement relating to pointing that predicts that the time required to rapidly move to a target area is a function of the distance to the target and the size of the target. FITTS LAW
- 23. FITTS LAW
- 24. This law is equally applicable to gestures. Visual targets should be designed to be close to the user to avoid reaching across the interface. Objects to be manipulated should be large enough to accomodate human finger FITTS LAW
- 25. FINGER TIPS A general guide for the size acceptable targets ideally should be no smaller than the smallest average finger pad, typically a 1cm (0.4 inch) diameter is used.
- 26. PPI What 1cm translates to in pixels depends on the pixel density or Pixels Per Inch (PPI). Once you have PPI simply times by 0.4 to get number of pixels of your touch point
- 27. Model Diagonal Pixels PPI iPhone 3GS 3.5” 320X480 163 iPhone 4 3.5” 640x960 326 iPad, iPad2 9.7” 1024x768 132 Google Nexus 1 3.7” 480x800 252 Motorola Droid X 4.3” 854x480 228 Nokia N8 3.5” 640x360 209 Nexus S 4.0” 480x800 235 EXAMPLE PPI
- 28. Dan Saffer ICEBERG TIPS These are controls that have larger targets than what is visible. The implication is you need more space between objects.
- 29. Dan Saffer OK ICEBERG TIPS These are controls that have larger targets than what is visible. The implication is you need more space between objects.
- 30. ADAPTION The keyboard on the iPhone actually uses some of the smallest targets at 5mm (0.2 inches). It uses adaptive targets to get over this limitation. Dan Saffer
- 31. Dan Saffer ADAPTIVE TARGETS These are created algorithmically by guessing the next item the user will touch and increasing the touch target appropriately.
- 32. Dan Saffer 1st 2nd ADAPTIVE TARGETS These are created algorithmically by guessing the next item the user will touch and increasing the touch target appropriately.
- 33. Select Tap Drag Flick Pinch Spread Slide Left to Right Slide Up and Down TOUCHSCREEN PATTERNS
- 34. CURSE OF TWO FINGERED ZOOM!
- 35. The more complicated the gesture the fewer people will use it! CURSE OF TWO FINGERED ZOOM!
- 36. USER GENERATED
- 38. Spatial Gesture Models 3D Model Based Appearance Based Deformable 2D Skeletal Volumetric Image Sequences Templates NURBS Primitives Super Quadratics SPATIAL GESTURE MODELS
- 39. Pointing Pointing Wiimote EXISTING FREE FORM GESTURE PATTERNS
- 40. Fukuda’s Proximity Wave AirSwitch Automatic Light Door EXISTING FREE FORM GESTURE PATTERNS
- 41. Hands Inside Rotate Nokia Dyson Air N93 Blade EXISTING FREE FORM GESTURE PATTERNS
- 42. Step Shake Dance SE W910i Mat EXISTING FREE FORM GESTURE PATTERNS
- 43. Clap Tilt Wii Balance Board EXISTING FREE FORM GESTURE PATTERNS
- 44. Moving a Cursor or Avatar Select Swith On/Off Slide Tap Tap Head Tilt Stare Flick Turn Head Left/Right Point Stomp Lean Torso Left/Right Hand Gun Wave Point Clap Snap Confirmation Cancel Nod Shake No Smile Frown Okay Thumbs Down Thumbs Up Stop Nose Tap MATCHING GESTURE TO Dan Saffer ACTIVITY
- 46. KINECT Xbox game controller proving a natural user interface using gestures and spoken commands
- 47. Infrared laser projector combined with a Sensor's microphone array enables monochrome CMOS sensor captures acoustic source localisation and ambient video data in 3D under any ambient light noise suppression conditions KINECT
- 48. KINECT SDK
- 49. SDK includes: 1. Raw sensor streams: Access to low-level streams from the depth sensor, colour camera sensor, and four-element microphone array. 2. Skeletal tracking: The capability to track the skeleton image of one or two people moving within the field of view for gesture-driven applications. KINECT SDK
- 50. SDK includes: 1. Raw sensor streams: Access to low-level streams from the depth sensor, colour camera sensor, and four-element microphone array. 2. Skeletal tracking: The capability to track the skeleton image of one or two people moving within the field of view for gesture-driven applications. 3. Advanced audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. 4. Sample code and Documentation. KINECT SDK
- 51. CONCLUSIONS • Currently gestures can be classified predominantly as touch or free form • The finger is not a direct replacement for the mouse. • The more complicated the gesture the fewer the number of people who will use it successfully. • If gestures are not obvious to the user the need to be clearly communicated • As yet our vocabulary of free form gestures is limited and needs much more development. • Free form gestures should make the user embarrassed.