Comp4010 Lecture12 Research Directions
1 like1,718 views
The document discusses various applications and advancements in virtual and augmented reality, highlighting educational tools like Google Expeditions and training solutions using virtual patients. It emphasizes the importance of immersive experiences and effective interaction techniques, as well as the evolution of display technologies and user interfaces. Key challenges such as addressing simulator sickness and developing compelling content are also addressed in the context of ensuring a rich user experience.
Comp4010 Lecture12 Research Directions
- 1. RESEARCH DIRECTIONS COMP 4010 Lecture Twelve Mark Billinghurst October 26th 2021 mark.billinghurst@unisa.edu.au
- 3. Virtual Reality Applications • Ideal applications for VR should: • Be strongly visual, have 3D spatial elements • Benefit from first person immersion • Benefit from 3D manipulation/navigation • Support Autonomy, Interaction and Presence (AIP Cube) • Etc..
- 4. Google Expeditions • Goal: Provide low-cost educational VR experience • Based on Google Cardboard VR platform • Different roles: • Guide— person leading an expedition on a tablet • Explorer— person following an expedition on a phone. • Usage • Used by millions of students • Over 1000 educational experiences developed • Royal Collection Trust, American Museum of Natural History, etc.
- 5. Key Findings • Low-cost VR/mobile VR can provide a valuable educational experience • Visit different locations, different times, etc. • Teach interaction key • Acting as guide, providing educational context • VR requires more work • Address simulator sickness, ergonomic issues, etc. • Immersion/Presence creates learning • Immersion creates memorable educational experience
- 6. Virtual Patients • Problem • Many doctors have poor doctor/patient skills • Have limited opportunity during training to learn skills • Solution • Virtual patients that doctors can communicate with naturally • Artificial agents with speech understanding
- 7. Key Findings • Virtual Humans can replace actors in training • interaction skills used with a virtual human translate to the interaction skills used with a real human • Students feel a strong sense of co-presence • Having character respond to speech and gesture increases immersion • VR is capable of creating realistic characters • Life size, intelligent backend, speech recognition • Skills learnt transfer to real world
- 8. • Wide Area Tracking • Computer vision, lights/reflective balls • > 120 cameras for 300 m2 space • Backpack VR system • Haptic feedback, wireless HMD • Real Props • Tracked objects, walls Tracking cameras Backpack system Large Scale VR Gaming
- 9. Key Findings • Wide area tracking possible • vision based systems can create large scale wide areas tracking, fast enough for game play • Shared gameplay improves experience • Focus on collaborative experiences, using avatar representations and roll division • Haptic feedback significantly increases presence • Use of physical props (objects, walls) • Content is king • Systems need compelling content/game place
- 10. • Goal: Extremely natural 3D painting/sculpting • User Interface • Two handed interface designed for two controllers (Vive, Rift) • Brush in dominant hand, tool palette in non-dominant • Typical drawing functionality – color, brush width, undo/redo, etc.. • Content sharing • Created content can be exported/shared in 2D/3D formats Tilt Brush
- 11. Key Findings • Use familiar tools • Tilt brush interface has familiar sculpting/painting tools – e.g. brush size, colour pallet, etc • Use intuitive interface • Two handed tools with natural metaphor – one hand for pallet/menu, one hand for painting/sculpting • Provide Magical experience • Provide experience not possible in real world, e.g. changing body scale, painting in 3D, etc. • Create a community • Provide ways for people to share content
- 12. Facebook Spaces • Collaborative VR environment • VR meeting and interaction space (up to 4 people) • Focus on communication • Speech and gesture based • https://www.facebook.com/spaces
- 13. Facebook Workrooms • Designed to support meetings • Meeting seating • Shared blackboard • Limited movement • Support for real devices • Keyboard/mouse input • Calibrated computer screen • Private space/public space • Rich communication cues • Lip sync • Natural gestures
- 14. Key Findings • Minimal social cues okay • Even simple avatars can provide rich social experience • Create shared social context • Important to place users in same shared Virtual Reality environment/shared social context • Audio is key • Provide low latency audio, spatial audio cues • Create a reason for communicating • Why should people want to connect? Create shared activity/reason for people to conference
- 16. My First VR Experience - 1990 • Silicon Graphics Reality Engine • 500,000 polygons/second • VPL Eyephone HMD • 320 x 240 resolution • Magnetic tracking • Glove input • Expensive - $250,000+
- 17. 1999 – Shared Space Demo • Face to face collaborative AR like Star Wars concept
- 18. CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec 1998: SGI O2 2008: Nokia N95 CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec By 2008 phones had the same hardware as used in Shared Space demo
- 19. 2005: Mobile AR version of Shared Space • AR Tennis • Shared AR content • Two user game • Audio + haptic feedback • Bluetooth networking
- 20. VR and AR Today • Large growing market • > $25 Billion USD in 2020 • Hundreds of millions of users • Many available devices • HMD, phones, tablets, HUDs • Robust developer tools • Vuforia, MRTK, Unity, etc • Large number of applications • > 150K developers, > 100K apps • Strong research/business communities • ISMAR, IEEE VR, AWE conferences, AugmentedReality.org, etc
- 23. “.. the technologies that will significantly affect our lives over the next 10 years have been around for a decade. The future is with us ... The trick is learning how to spot it” October 2004 Bill Buxton
- 24. Key Technologies for AR/VR Systems • Display • Stimulate visual, hearing/touch sense • Tracking • Changing viewpoint, registered content • Interaction • Supporting user input
- 25. Grand Challenges in AR Billinghurst, M. (2021). Grand Challenges for Augmented Reality. Frontiers in Virtual Reality, 2, 12.
- 26. DISPLAY RESEARCH
- 27. • Past • Bulky Head mounted displays • Current • Handheld, lightweight head mounted • Future • Projected AR • Wide FOV see through • Retinal displays • Contact lens Evolution in AR/VR Displays
- 28. Current Smart Glasses: North Focals (2020) • https://www.bynorth.com/ (acquired by Google in 2020) • Socially acceptable smart glasses • $599 USD, small field of view • Ring input device
- 29. Wide FOV See-Through Displays • Waveguide techniques • Wider FOV • Thin see through • Socially acceptable • Pinlight Displays • LCD panel + point light sources • 110 degree FOV • UNC/Nvidia Lumus DK40 Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014 Emerging Technologies (p. 20). ACM.
- 31. Kura Gallium Glasses (2020) • https://www.kura.tech/ • Pinpoint wide field of view effect - 150° FOV; • Resolution in the range 4K-8K; • Very transparent screen
- 32. LightField Displays • Addressing the vergence accommodation conflict • Generates simulated views from many directions
- 33. LightField Display • Multiple views and lenses • 3D viewing without glasses Multi-user support
- 34. LightField Displays in HMDs • Use microlens array to generate many images
- 35. Display Results
- 36. Nvidia Prototype • 1 cm thick lightfield display • 146x78 pixel resolution, 29x16 degree field of view Lanman, D., & Luebke, D. (2013). Near-eye light field displays. ACM Transactions on Graphics (TOG), 32(6), 1-10.
- 38. Virtual Retinal Display (1993 - ) • Use scanner to scan images into the eye
- 39. System Diagram High speed horizontal and vertical scanners key element
- 40. From Prototype to Reality Tom Furness (1993) Microvision Nomad (2002)
- 41. Current Microvision Scanning Technology https://www.youtube.com/watch?v=OvHN_bypoNE
- 42. Pros and Cons • Advantages • Very bright - outdoor use • Low power • Infinite focus • True 3D display • High spatial/temporal resolution • Overcome cornea damage • Challenges • Stereo rendering • Small exit pupil (need eye tracking) • Wearability Small exit pupil Typical HUD Retinal Display
- 43. Continuing the Dream.. • Other companies have tried to bring retinal displays to market • MagicLeap, Avegant, EyeWay, etc.. MagicLeap EyeWay
- 46. Contact Lens Displays • Contact Lens only • Unobtrusive • Significant technical challenges • Power, data, resolution • Babak Parviz (2008) • MojoVision • Mojo Lens • Prototype smart contact lens • https://www.mojo.vision/ http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/
- 50. Optical See-Through Occlusive Displays
- 51. Prototype Implementation Kiyokawa, Kiyoshi, Yoshinori Kurata, and Hiroyuki Ohno. "An optical see-through display for mutual occlusion with a real-time stereovision system." Computers & Graphics 25.5 (2001): 765-779.
- 52. Results
- 53. More Recent – Occlusion Leak Compensation • Change occlusion mask to larger than AR object • Compensate with additional graphics Itoh, Yuta, Takumi Hamasaki, and Maki Sugimoto. "Occlusion leak compensation for optical see-through displays using a single- layer transmissive spatial light modulator." IEEE transactions on visualization and computer graphics 23.11 (2017): 2463-2473.
- 54. Details • asdf
- 56. Literature Survey Macedo, M. C. D. F., & Apolinario, A. L. (2021). Occlusion Handling in Augmented Reality: Past, Present and Future. IEEE Transactions on Visualization and Computer Graphics. Number of papers Example systems
- 57. Peripheral Displays • Use second display in HMD to add peripheral view Nakano, Kizashi, et al. "Head-Mounted Display with Increased Downward Field of View Improves Presence and Sense of Self-Location." IEEE Transactions on Visualization and Computer Graphics (2021).