Natural User Interfaces
Download as PPTX, PDF1 like1,206 views
The document discusses advancements in natural user interfaces (NUI) and the evolution from traditional user interfaces, emphasizing the shift towards smart computing and ambient intelligence. It explores various input methods such as gesture recognition, facial recognition, and multi-touch technologies, while addressing design considerations and usability challenges. Additionally, it highlights the significance of creating collaborative, user-centered experiences through the integration of diverse technologies.
![Costs
Cost Buy Link
Kinect 1 100€ [???]
Kinect 2 150€ http://goo.gl/rskPuD
Real Sense 99$ http://goo.gl/G67TVy
Leap Motion 90€ http://goo.gl/zyVXZZ
Myo 199$ https://goo.gl/ubv6wV
EyeX 99€ http://goo.gl/oGD3Ds](https://image.slidesharecdn.com/nui-151111150638-lva1-app6892/85/Natural-User-Interfaces-54-320.jpg)
Natural User Interfaces
- 1. Luigi Oliveto POLIMI Natural User Interfaces
- 2. Luigi Oliveto Master of Science at Politecnico di Milano Researcher at Politecnico di Milano IT Consultant email: luigi.oliveto@gmail.com twitter: @LuigiOliveto linkedin: https://it.linkedin.com/in/luigioliveto Nice to Meet You
- 3. Lots of words… Ambient Intelligence Internet of Things Pervasive Computing Physical Computing Ubiquitous computing Augmented reality Human-centered computing Smart device
- 4. • No more desktop-centered computation, but distributed computation(“ubiquitous”) • Objects become more “intelligent” and “smart” • New information’s model • New possibility of interaction with information • Machines fit the human environment instead of forcing humans to enter theirs … One concept
- 5. Interface Evolution Command Line Interface CLI Graphical User Interface GUI NUI Natural User Interface
- 6. Natural User Interface Computer Vision Facial Recognition Spatial Recognition Augmented Reality Gesture Sensing Audio Recognition Voice Comma nd Natural Speech Ambien t Noise Touch Single Touch Multi- Touch Pen Input Sensors Geospatial Sensing Accelerometers Biometrics Ambient Light Brain Waves Mind control Mood Recogniti on
- 9. megawatts86
- 11. mrkathika
- 12. Shopping…
- 13. pin add
- 14. What’s happened?
- 15. Natural…experience! “Natural interaction is defined in terms of experience: people naturally communicate through gestures, expressions, movements, and discover the world by looking around and manipulating physical stuff.” Alessandro Valli https://www.linkedin.com/in/alessandrovalli
- 17. What is changed? AFTERBEFORE • Single device • Collaborative experience • More points • Multiple devices • Individual experience • One click
- 18. • Sensible to touch surface (touch screen) Multi-Touch Hardware RESISTIVE CAPACITIVE
- 19. • Camera-based technology: Laser Line Plan Frustrated Total Internal Reflection Diffused illumination Pixel Sense Multi-Touch Hardware (2)
- 20. Multi-Touch Devices Smartphone Tablet Monitor
- 21. Multi-Touch Devices (2) Tangible Table / Wall
- 22. • Microsoft "Surface SDK" and "Windows Presentation Foundation" include API, documentation and tool to develop multi-touch apps on Windows 7 and Surface • "Cocoa Touch" is a library to develop software for iPhone, iPod Touch, e iPad. "Cocoa Touch" is included in iPhone SDK. • Android SDK include tool, emulator, debugger and library to develop App for Android • Gestureworks (by Ideum) is an interesting Flash multi-touch. The Gestureworks software allows to develop multiuser and multi-touch-enabled applications with Adobe Flash. Multi-Touch Software
- 24. • Nails • Gloves • Dirty fingers • Gestures (are they so easy?) • Accuracy Problems: Input
- 25. Gorilla arm problem Problems: Accessibility
- 26. • Touch-based applications introduce new important constraints in design of interface and interaction: • Target’s dimensions must be fit to fingers’ dimension (min 10mm) Problems: Usability
- 27. Labels’ position: hands and fingers can hide information Problems: Usability (2)
- 28. Gesture’s design: some actions can hide part of information, too. Problems: Usability (3)
- 29. Iceberg Tips: create a wider invisible area Adaptive Targets: device tries to guess next button pressed by user and zoom it Hell Tips & Tricks
- 30. • Don’t assume that people will know that they can touch a screen. • Create an “attract state” that demonstrates interactivity while nobody is using the device • Make touchable things look touchable • Design for fingers • Make sure hands don’t cover up information necessary for interaction • Don’t rely on traditional mouse-based interactions, such as hover & double click • Use consistent and familiar gestures Tips & Tricks (2)
- 31. The Power of Microsoft® PixelSense™ https://www.youtube.com/watch?v=58dsqozft3k Samsung SUR40 with Microsoft® PixelSense™ https://www.youtube.com/watch?v=kmOku92MlQc Microsoft Surface wine-tasting demo http://www.youtube.com/watch?v=Y3KzprGxpZU&feature=related Patient Consultation Interface Surface Application http://www.youtube.com/watch?v=Qf0WxOo3O4g&feature=related Videos
- 32. Microsoft Surface Application- Barclay's https://www.youtube.com/watch?v=cBF5BI5H7vs Playing with Microsoft Surface http://www.youtube.com/watch?v=SUfRSZppUYs&feature=related Touch2Much - Microsoft Surface Museum/Gallery Application http://www.youtube.com/watch?v=DDrCq9632YY AR.Drone Quadrotor Flight via Microsoft Surface http://www.youtube.com/watch?v=x1bbT8M6uRs Video (2)
- 34. • Camera • Monitor • Microphone QuiQui’s Giant Bounce Game for children (4 - 9 years) Game’s paradigm: story telling with animated characters The actions of child activate specific behaviors of avatar
- 35. • The EyeToy is a color digital camera device, similar to a webcam, for the PlayStation 2. • The technology uses computer vision and Gesture recognition to process images taken by the camera. • This allows players to interact with games using motion, color detection and also sound, through its built-in microphone. • Limited success due to the low precision Sony Eye Toy
- 36. • The console was released on November 19, 2006. About eight days after, 600,000 Wii’s were reported to be sold. • It has revolutionized game play and has impacted society: anyone can play! Nintendo Wii
- 37. • The Wii remote, or “Wiimote”, interacts with a sensor bar by using accelerometers, infrared LED’s, and triangulation. • In general, a player’s Wiimote movements would determine their character’s actions. A gamer would have to move in order to play. Wii Technologies
- 38. • Wii and wiimote comunicate by Bluetooth • TED 2008: Johnny Lee show how is possible connect wiimote with a normal pc and use them in innovative application: – interactive whiteboard – 3D head tracking – finger tracking • Many others researcher start to use wiimote in academia projects: http://hackaday.com Wii Hacks
- 39. • It is a motion sensing input device by Microsoft for the Xbox 360/XBOX ONE console. • It enables users to control and interact with the Xbox without the need to touch a game controller, through a natural user interface using gestures and spoken commands. Microsoft Kinect
- 40. Immersive user experience Kinect’s magic “Any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke)
- 41. Provided Data
- 42. Cursors (hands tracking) Target an object Avatars (body tracking) Interaction with virtual space • Depend by the tasks • Important aspect in design of UI Interaction metaphors
- 43. The shadow/mirror effect Shadow Effect: • I see the back of my avatar • Problems with Z movements Mirror Effect: • I see the front of my avatar • Problem with mapping left/right movements
- 44. User Interaction Game mindset ≠ UI mindset Challenging = fun Challenging = easy and effective
- 45. Gesture semantically fits user task Abstract Meaningful
- 46. Intel Real Sense
- 47. 4 basic types of input Categories of Input Capabilities Features Hands • Hand and Finger Tracking • Gesture Recognition • 22-point Hand and Finger Tracking • 9 static and dynamic mid-air gestures Face • Face Detection and Tracking • Multiple Face Detection and tracking • 78-point Landmark Detection (facial features) • Emotion Recognition (7 emotions, coming post-Beta) • Pulse Estimation • Face Recognition (Coming post-beta) Speech • Speech Recognition • Command and Control • Dictation • Text to Speech Environment • Segmentation • 3D Scanning • Augmented Reality • Background Removal • 3D Object / Face / Room Scanning (Coming post-beta) • 2D/3D Object Tracking • Scene Perception (coming post-beta)
- 49. Leap Motion - Field of View 150° - Long Side 120° - Short Side Max 60 cm above the controller Max 60 cm wide on each side
- 52. Power Comes from the Sum • Any single technology on its own – can create good experiences • The sum: This is where the magic is • Tons of opportunities ahead
- 53. Some selection criteria… 1° 2° 3° 4° 5° 2 or more users Kinect Intel Leap - - Full body interaction Kinect - - - - Hand Gesture Recognition Myo Intel Leap Kinect - Accuracy Leap Intel Kinect Myo - Voice command Intel Kinect - - - Face Tracking Tobii Intel Kinect - - Commecial use Kinect Intel Tobii Leap Myo Compatibility Leap MYO Tobii Intel Kinect
- 54. Costs Cost Buy Link Kinect 1 100€ [???] Kinect 2 150€ http://goo.gl/rskPuD Real Sense 99$ http://goo.gl/G67TVy Leap Motion 90€ http://goo.gl/zyVXZZ Myo 199$ https://goo.gl/ubv6wV EyeX 99€ http://goo.gl/oGD3Ds
- 55. Leap, Real Sense, Kinect ranges 2,5 cm 60 cm 2 m 4 m
- 57. Capture Volumes The user is performing a hand gesture outside of the capture volume. The camera will not see this gesture
- 58. Evaluate different settings and environment
- 59. Sensor with Camera use IR light and Sunlight can blind the camera!!! • Check exposition during all day • Verify that there isn’t direct light on the camera These devices aren’t a Rugged devices: • Check temperatures (+3/33°) • Check humidity Indoor/Outdoor
- 60. Comfortable positions Your users are not GORILLAS!!!
- 61. User posture may affect design of a gesture
- 63. Feedback, feedback, feedback,… View of user: • User Viewport • User Overlay
- 64. … where actions performed for some other purpose or unconscious signs are interpreted in order to influence/improve/facilitate the actors' future interaction or day-to-day life (from Alan Dix) • The interaction is not purposeful from the person side, but it is designed “to happen” • It “happens” in relation to signs which are not done for that (body temperature, unconscious reactions such as blink rate, or unconscious aspects of activities such as typing rate, vocabulary shifts (e.g. modal verbs), actions done for other purposes, … • It is designed for people acting Manage Incidental Interaction
- 65. Luigi Oliveto POLIMI Natural User Interfaces