Mixed reality the second generation is all about ux
- 1. The Second Generation is all about UX
- 3. Cameron Vetter is a technologist with 20 years of experience using Microsoft tools and technologies to develop software. Cameron has experience in many roles including Development, Architecture, Infrastructure, Management, and Leadership roles. Cameron recently received a Microsoft MVP award for his evangelism work around Deep Learning in Azure. Cameron has worked for some of the largest companies in the world and for small local companies getting a breadth of experience helping him understand the needs of different size businesses and different Industries. He helps clients develop Technical Strategies and helps clients Architect, Design, and Develop software focusing on Deep Learning / AI, Cloud Architecture, Microservices, Mixed Reality, and Azure.
- 4. Introduction Review of Generation 1 What is Generation 2? HoloLens 2’s UXD Story Tools and Challenges What should I do to Prepare? Question & Answer
- 7. / Microsoft / • Shipped March 2016 • Around 25,000 sold • Limited Target Market
- 8. / Magic Leap / • Shipped August 2018 • Limited Sales, Likely < 10,000 • Limited Target Market
- 9. / Microsoft / • Shipped October 2017 • Around 500,000+ sold • Consumer Target Market • Products from Samsung, HP, Dell
- 10. HoloLens • Intelligent Factory • Elevator Repair • Architecture • Construction • Instructions • Medical • Interior Design Magic Leap https://www.magicleap.com/experiences
- 13. State of the Market 02 / Seaborn / Why should I care about MR? /
- 14. / Microsoft / • Shipping Soon • Improves FOV • Adds Eye Tracking • Better Weight Distribution • Limited Enterprise Target Market
- 15. / Magic Leap / • Shipping Soon • Similar technology plus LTE • Sold Exclusively through AT&T • Consumer Target Market
- 16. / Microsoft / • Shipped September 2017 • Sold Out • Enterprise Target Market
- 18. / The Hero of Our Story / • Research Intern – Xbox One Kinect Skeletal Tracking • Joined HoloLens Team in 2015 • 20 patents already!
- 19. Iron Man Minority Report / From the Movies /
- 20. HoloLens 2 Remote Assist HoloLens 2 Buttons Demo / To Reality /
- 21. Air Tap and Bloom gestures. Air Tap can be hard to teach. Gesturing is fatiguing. Spoken Commands, also low Fatigue. Annoying and slow for the user. Targeting with head, low fatigue.
- 22. / Shipping A Vision One Prototype at a Time / • Cubes follow the hand, making it feel interactive. • Feedback when grabbed, with color changes and sound • Discovered FOV is a huge factor • Video went viral inside Microsoft
- 23. / Shipping A Vision One Prototype at a Time / • Experimenting with Interaction technique for position, size, and orientation. • Discovered Users require feedback about hand state. • Discovered hand tracking is hard when hands get close together. • Hardware team improved hand tracking to solve problem.
- 24. / Shipping A Vision One Prototype at a Time / • Near input is manipulating holograms directly • Far input is manipulating indirectly • Research focused on if user would understand what is happening
- 25. / Shipping A Vision One Prototype at a Time / • Discovered users like to mimic the real world interaction • Knob worked poorly, too easy to end up in an awkward position with hand occluded
- 26. / Shipping A Vision One Prototype at a Time / • 3 things worked well: Push Button, Pinch Slider, Flick Switch • Focused on these simple controls that worked well
- 27. / Shipping A Vision One Prototype at a Time / • Rotating an object is limited by the motion of your wrist • Pushing a cube around is fun but very imprecise
- 28. / Shipping A Vision One Prototype at a Time / • Rotating the slate by nudging feels very satisfying • Adding constraints to motion made it easier to manipulate objects in a control manner
- 29. / Shipping A Vision One Prototype at a Time / • Focus was scrolling, panning and selections • Keyboard was an after thought • Keyboard was inaccurate due to calibration errors • Hardware team improved calibration • Discovered the need to amplify Visual and Audio feedback x10
- 30. / Shipping A Vision One Prototype at a Time / • Some users use one hand, others two • Some tend towards direct actions, others indirect • If this does not mirror a real world activity, interaction points are required • Different shape handles encourage different gestures
- 31. Enhanced microphone array for work class speech recognition. Enhanced 3D sound allows the user to hear which object the sound is coming from. Eye Tracking, Fully articulated hand tracking, semantic labeling, spatial audio, and JBL filter all require too much computer to do on a CPU. Eye tracking allows the HoloLens to understand where the user is looking and whole their eyes are tracking across the scene.
- 32. / Press Content With Your Finger / • Buttons should only be press-able from the front. • When performing collision tests, perform ray cast from previous to current point to allow fast movements to work. • Visual feedback when a user hovers over a button. • Visual + Audio feedback on button pressed or activated.
- 33. / Use Touchscreen Interactions for 2D Content / • Surface should only be touch-able from front. • When scrolling, allow for hand to go beyond the bounds of the scrollable area by .25 meters. • Project a cursor onto the 2D surface so user knows where their finger is. • Draw a targeting aid on finger tip to help with pressing small targets.
- 34. / Use a single point of interaction per hand / • Single AR interactions do not have haptics or physical services making it easy for hands to penetrate surfaces, expect that. • Place a single collidable on the index fingertip to reduce accidental touches from other fingers.
- 35. / Prefer Bounding Box for Precise Manipulations / • For precise manipulation, use affordances on corners and edges, like in the shell. • When precision is unimportant use the rotation of the wrist or palm to rotate objects. • Consider allowing inspection mode, where a person can use wrist rotation to inspect an object, but one they let go it snaps back.
- 36. / Physics and Constraints / • Adding physics behaviors such as momentum will delight your customers, but comes at the cost of reduced precision. • Add constraints to your rigid bodies to allow for further precision.
- 41. I’ve found that it’s hard for 2D UX/product designers to transfer their thinking over to designing immersive products. Most carry over their 2D thinking / best practices and want to stick 2D screens all over a multidimensional experience. Thus negating the point of the design -- Suzanne Border (BadVR)
- 42. / Think Like a Director / • Consider setting up a dedicated space with props • Simulate happy case and worse case scenarios
- 43. / Why would you look at that? / • How do you make sure the user looks where they need to look? • If they aren’t looking there, How do you know? • What will you do if they weren’t?
- 44. / Why? / • Display Resolution is limiting • UI elements will looks pixelated • High Levels of aliasing • Avoid using big text blocks
- 45. / Steps in the Right Direction / • Design Text to be viewed at a specific distance • DMM = Distance independent millimeter Example: Desgn of the same dialog for three viewing distances. Each appears the same to the user although they are different distances away. Using DMM to Design Text Dialogs
- 46. / Wave your arms in the air for the next 10 minutes / • Minority Report style UI is exhausting. • Julia’s research completely ignores this huge issue.
- 47. / Good Luck / • Can’t just take a screenshot • 2D Videos are confusing • Sharing an Experience between devices is very complex HoloLens Spectator View
- 49. / When? Where? How? / • 3 to 5 years • 3rd Generation will start the shift • 4th Generation will finish it – MR Contact Lenses • Once they are Ubiquitous expect Smart Phones to disappear
- 50. / When? Where? How? / • We’re Shifting Now. • Pokémon Go started the shift. • Google Maps will finish it. • Limited uses for AR beyond games. • Market will be cannibalized by MR Shift
- 51. When will users being to expect MR? Most users will have used some form of xR within a year, expect expectations to begin shifting very soon.
- 52. • User experience design research is happening now that will be the foundation for UXD in MR • This is a unique opportunity to make your mark on the future Be like Julia Schwarz!