Research Directions in Heads-Up Computing
1 like697 views
The document discusses research directions in heads-up computing, emphasizing a shift from device-centered to human-centric interaction paradigms that enhance daily activities. It highlights various research areas, including remote communication, cross reality systems, novel interfaces, and ethical considerations related to social acceptance and privacy. Additionally, it outlines the development of prototypes and tools to facilitate seamless integration of augmented and virtual reality in collaborative environments.
Research Directions in Heads-Up Computing
- 1. RESEARCH DIRECTIONS IN HEADS-UP COMPUTING Mark Billinghurst mark.billinghurst@unisa.edu.au Workshop on Heads-up Computing Ubicomp 2024 October 5th 2024
- 2. 1967 – IBM 1401 – half of the computers in the world, $10,000/month to run
- 4. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand 2010 - Head
- 5. Heads-Up Computing A new interaction paradigm that offers seamless computing support for humans' daily activities. Zhao, S., Tan, F., & Fennedy, K. (2023). Heads-Up Computing Moving Beyond the Device-Centered Paradigm. Communications of the ACM, 66(9), 56-63.
- 6. Moving from device centric interactions to human centric
- 8. Devices Becoming Available • Socially acceptable form factor, all day use, enhancing everyday activities Meta Ray-ban Vuzix z100 Xreal Air 2
- 9. Meta Orion
- 10. Many Possible Research Directions •Collaborative System •Novel Interfaces •Cross Reality Systems •Hybrid Interfaces •Prototyping Tools •Social issues
- 12. A wide variety of communication cues used. Speech Paralinguistic Para-verbals Prosodics Intonation Audio Gaze Gesture Face Expression Body Position Visual Object Manipulation Writing/Drawing Spatial Relationship Object Presence Environmental Face to Face Communication
- 13. Smart Glass for Remote Collaboration • Camera + Processing + AR Display + Connectivity • First person Ego-Vision Collaboration
- 14. AR View Remote Expert View
- 15. Empathy Glasses • Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression + + Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
- 16. Remote Collaboration • Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
- 18. Shared Sphere – 360 Video Sharing Shared Live 360 Video Host User Guest User Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a live panorama. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications (pp. 1-4).
- 20. 3D Live Scene Capture • Use cluster of RGBD sensors • Fuse together 3D point cloud
- 21. Scene Capture and Sharing Scene Reconstruction Remote Expert Local Worker
- 22. AR View Remote Expert View
- 23. View Sharing Evolution • Increased immersion • Improved scene understanding • Better collaboration 2D 360 3D
- 24. NOVEL INTERFACES
- 25. Interface Challenge • Need to support • Natural user interaction • Rich input expression • In a way that • Is socially acceptable • Ergonomic, reliable • Can be used all day • Uses few devices
- 26. Avoiding Gorilla Arm • Design interface to reduce mid-air gestures
- 27. Multi-Scale Gesture • Combine different gesture types • In-air gestures – natural but imprecise • Micro-gesture – fine scale gestures • Gross motion + fine tuning interaction Ens, B., Quigley, A., Yeo, H. S., Irani, P., Piumsomboon, T., & Billinghurst, M. (2018). Counterpoint: Exploring Mixed-Scale Gesture Interaction for AR Applications. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (p. LBW120). ACM.
- 29. Gaze and Pinch Interaction Pfeuffer, K., Mayer, B., Mardanbegi, D., & Gellersen, H. (2017, October). Gaze+ pinch interaction in virtual reality. In Proceedings of the 5th symposium on spatial user interaction (pp. 99-108).
- 31. Physiological Sensor Input • Using physiological sensors for implicit input • Systems that recognize user intent/activity • EEG • Measuring brain activity • EMG • Measuring muscle activity
- 33. Meta EMG Demo • https://www.youtube.com/watch?v=WmxLiXAo9ko
- 34. Sensor Enhanced VR HMDs Eye tracking, heart rate, pupillometry, and face camera HP Omnicept Project Galea EEG, EEG, EMG, EDA, PPG, EOG, eye gaze, etc.
- 35. Multiple Physiological Sensors into HMD • Incorporate range of sensors on HMD faceplate and over head • EMG – muscle movement • EOG – Eye movement • EEG – Brain activity • EDA, PPG – Heart rate
- 37. • Measure physiological cues • Brain activity • Heart rate • Eye gaze • Show user state • Cognitive load • Attention Showing Cognitive Load in Collaboration
- 38. Demo
- 40. Transitioning Along the RV Continuum Reality Virtuality Augmented Reality Augmented Virtuality Mixed Reality Using the same device to experience Reality, AR and VR
- 41. Why Transition Along the RV Continuum ? • Combining the best of AR/VR • Augmented Reality is preferred for: • augmenting real objects • co-located collaboration • Virtual Reality is preferred for: • experiencing world immersively • sharing views, remote collaboration
- 42. Collaboration Between Points on the RV Continuum Reality Virtuality Augmented Reality Augmented Virtuality Collaboration between people in reality, AR and VR Mixed Reality
- 43. Why Collaboration Across the RV Continuum ? • Enabling everyone to connect • Shared experience, shared understanding • Using the ideal interface for the task • E.g. AR for interacting in real world, VR for providing remote help
- 44. The MagicBook (2001) Reality Virtuality Augmented Reality (AR) Augmented Virtuality (AV) Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The MagicBook: a transitional AR interface. Computers & Graphics, 25(5), 745-753.
- 45. MagicBook Demo
- 46. Features • Seamless transition from Reality to Virtuality • Reliance on real decreases as virtual increases • Supports egocentric and exocentric views • User can pick appropriate view • Independent Views • Privacy, role division, scalability • Collaboration on multiple levels: • Physical Object, AR Object, Immersive Virtual Space • Egocentric + exocentric collaboration • multiple multi-scale users
- 47. Apple Vision Pro (2024) • Transitioning from AR to VR • Spatial Computing – interface seamlessly blending with real world
- 48. Cross Reality (CR) An emerging technology focusing on the concurrent usage of, or the transition between multiple systems at different points on the reality-virtuality continuum (RVC). Simeone, Adalberto L., Mohamed Khamis, Augusto Esteves, Florian Daiber, Matjaž Kljun, Klen Čopič Pucihar, Poika Isokoski, and Jan Gugenheimer. "International workshop on cross-reality (xr) interaction." In Companion Proceedings of the 2020 Conference on Interactive Surfaces and Spaces, pp. 111-114. 2020.
- 49. HYBRID SYSTEMS
- 50. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand 2010 - Head
- 51. Computers Don’t Disappear Room Desk Lap Hand Head 1960’s 1980’s 2010’s
- 52. Current Generation of Smart Glasses • Trend towards lightweight thin displays • Offload processing onto second device • High bandwidth connectivity to cloud services
- 54. Hybrid Interfaces AR HMDs tethered to handheld devices/phones ○New opportunity for interaction/display ○Wide field of view of HMD and Precise input of HHD
- 56. Previous Work Our previous work • Use touch input on phone to interact with AR HMD content • Use tablet to provide 2D view of 3D AR conferencing space Bleeker 2013 Budhiraja 2013
- 58. Example: Secondsight A prototyping cross-device interface • Enables an AR HMD to "extend" the screen of a smartphone Key Features • Can simulate a range of HMD Field of View • Enables World-fixed or Device-fixed content placement • Supports touch screen input, free-hand gestures, head-pose selection Reichherzer, C., Fraser, J., Rompapas, D. C., & Billinghurst, M. (2021, May). Secondsight: A framework for cross-device augmented reality interfaces. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-6).
- 59. Content Placement © 2021 SIGGRAPH. All Rights Reserved.
- 60. Input
- 61. Map application © 2021 SIGGRAPH. All Rights Reserved.
- 63. Challenges with Prototyping HUC Systems • Cross platform support • Need for programming skills • Building collaborative systems • Need to build multiple different interfaces • Connecting multiple devices/components • Difficult to prototype hardware/display systems
- 64. Second Sight Implementation Hardware • Meta2 AR Glasses (82o FOV) • Samsung Galaxy S8 phone • OptiTrack motion capture system Software • Unity game engine • Mirror networking library
- 65. Google - The Cross-device Toolkit - XDTK • Open-source toolkit to enable communication between Android devices and Unity • Handles • Device discovery/communication • Sensor data streaming • ARCore pose information • https://github.com/google/xdtk Gonzalez, E. J., Patel, K., Ahuja, K., & Gonzalez-Franco, M. (2024, March). XDTK: A Cross-Device Toolkit for Input & Interaction in XR. In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 467-470). IEEE.
- 66. VRception Toolkit • First multi-user and multi-environment rapid-prototyping toolkit for non-experts • Designed for rapid prototyping of CR systems • Supports prototyping in VR or in Unity3D Gruenefeld, U., et al. (2022, April). Vrception: Rapid prototyping of cross-reality systems in virtual reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-15).
- 68. SOCIAL ISSUES
- 69. Social Acceptance • People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses • 20% of mobile AR browser users experience social issues • Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)
- 70. Ethical Issues • Persuasive Technology • Affecting emotions • Behavior modification • Privacy Concerns • Facial recognition • Space capture • Personal data • Safety Concerns • Sim sickness, Distraction • Long term effects Pase, S. (2012). Ethical considerations in augmented reality applications. In Proceedings of the International Conference on e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE) (p. 1). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).
- 73. CONCLUSIONS
- 74. Summary • Heads-Up Computing – new interaction paradigm • Moving from device focused to human focused • Important research areas • Remote communication, Cross Reality, Novel Interface • Prototyping Tools, Privacy and Security • New research opportunities emerging • XR + AI + Sensing + ??