![Telepresence
• Requires a sensor like Kinect for capturing colors and depth data from
cameras.
• Captured data (e.g., arrays of Vector3[] / Color[]) can be transmitted
over the network via sockets.
• Requires a PC to act as a server.
• A full implementation is available here:
Kowalski, M.; Naruniec, J.; Daniluk, M.: "LiveScan3D: A Fast and
Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors". in
3D Vision (3DV), 2015 International Conference on, Lyon, France, 2015
https://github.com/MarekKowalski/LiveScan3D
• Further improvements could include audio streaming, usage of other
protocols (e.g. WebRTC)
• Streaming point
clouds to AR/VR
devices](https://image.slidesharecdn.com/gettingstartedwithunityandar-vrforthe-211129173833/85/Getting-started-with-Unity-and-AR-VR-for-the-NET-developer-Reloaded-39-320.jpg)
Getting started with Unity and AR/VR for the .NET developer... Reloaded!
- 1. Getting started with Unity and AR/VR for the .NET developer… Reloaded! Davide Zordan B: https://davide.dev E: mail@davide.dev T: @DavideZordan
- 2. Davide Zordan Senior software engineer MCSD, MCSA, MS, MCP Microsoft MVP (2009-2014)
- 3. Agenda Introduction to Mixed Reality VR interactions and locomotion using Unity and SteamVR Enabling Oculus Quest passthrough Telepresence using points cloud Hand-tracking
- 4. What is Mixed Reality?
- 5. The state of things as they “actually exist” through our human senses without any technology. Artificially created sensory experiences of people, environments and objects, which can include sight, touch, hearing, and smell. PHYSICAL REALITY VIRTUAL REALITY
- 6. MIXED REALITY VIRTUAL REALITY PHYSICAL REALITY
- 7. PHYSICAL REALITY DIGITAL REALITY MIXED REALITY (MR) Windows Mixed Reality Holographic Devices (see-through) / Magic Leap / Nreal Light / RealWare HMT-1 / LYNX-R / Meta Project Aria, Project Cambria / Varjo Oculus Quest 2 / HTC VIVE - Cosmos / Valve Index / Windows Mixed Reality Immersive Devices (Opaque) / Varjo AR TODAY VR TODAY Physical - Digital Reality Spectrum *The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, "A Taxonomy of Mixed Reality Visual Displays."
- 8. How can we design for AR/VR?
- 9. Designing great VR experiences Place illusion Plausibility illusion Embodiment illusion Source: https://www.coursera.org/lecture/introduction-virtual-reality/introduction-to-plausibility-illusion-psi-K5PGj
- 11. A new way to interact
- 12. Gaze Gesture Voice Motion control World coordinates Spatial audio Spatial mapping Hand-tracking Eye-tracking
- 15. DEMO VR locomotion and custom interactions
- 17. 1 – Using the SteamVR controllers bindings
- 20. 1 – Creating a VR headset manager
- 21. 2 – Initialising device-specific settings
- 22. Adding vocal commands: • Unity KeywordRecognizer • Natural Language Understanding … available online at https://davide.dev/blog
- 23. DEMO Basic selection and interactions
- 25. Performance tips
- 26. 1 - Mark not moving objects as static
- 27. 2 – Reduce the number of draw calls - baked lights
- 28. 3 – Reduce the number of draw calls • Combine meshes and materials • Some Unity plugins available in the asset store can help
- 29. 4 – Avoid not performant code at runtime
- 30. 5 – Reduce max polys count in scenes • Mobile devices: ~200,000 • Desktop VR: ~1,000,000 • Do not draw polys that cannot be seen by the user
- 31. 6 – Optimise garbage collection • Try to avoid the GC while a scene is running • Unity now allows incremental garbage collection
- 32. 7 – Use object pooling • Try to not instantiate objects at runtime but reuse instanced using object pooling https://github.com/davidezordan/PerformanceBounceback/blob/master/PerformanceBounceback
- 33. Using Oculus Quest passthrough APIs
- 35. What are passthrough APIs? https://developer.oculus.com/experimental/passthrough-api/ • Passthrough APIs must be enabled using experimental features using adb: adb shell setprop debug.oculus.experimentalEnabled 1 • Must be enabled in the OVR Manager script of the OVRCameraRig: • Require additional passthrough script: • Enable development of pass-through experiences using Quest • New SDKs recently announced (Voice, Hands manipulations)
- 37. Collaboration in AR/VR Telepresence using point clouds
- 38. Collaboration • Facebook Horizon Worlds Social experience https://www.oculus.com/facebook-horizon/ • Facebook Horizon Workrooms Collaboration experience for meetings https://about.fb.com/news/2021/08/introducing-horizon- workrooms-remote-collaboration-reimagined/ • Microsoft Mesh Virtual collaboration / Holoportation https://www.microsoft.com/en-us/mesh Source: https://techcommunity.microsoft.com/t5/mixed-reality-blog/microsoft-mesh-app-august-2021-update-new-features/ba-p/2746856
- 39. Telepresence • Requires a sensor like Kinect for capturing colors and depth data from cameras. • Captured data (e.g., arrays of Vector3[] / Color[]) can be transmitted over the network via sockets. • Requires a PC to act as a server. • A full implementation is available here: Kowalski, M.; Naruniec, J.; Daniluk, M.: "LiveScan3D: A Fast and Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors". in 3D Vision (3DV), 2015 International Conference on, Lyon, France, 2015 https://github.com/MarekKowalski/LiveScan3D • Further improvements could include audio streaming, usage of other protocols (e.g. WebRTC) • Streaming point clouds to AR/VR devices
- 40. LiveScan 3D project: https://github.com/MarekKowalski/LiveScan3D
- 41. Adapted from LiveScan 3D HoloLens project: https://github.com/MarekKowalski/LiveScan3D-Hololens
- 42. Adapted from LiveScan 3D HoloLens project: https://github.com/MarekKowalski/LiveScan3D-Hololens
- 45. Mixed Reality Toolkit for Unity • https://aka.ms/MRTK MRTK extensions for Oculus Quest • https://github.com/HoloLabInc/MRTKExtensionForOculu sQuest • https://github.com/provencher/MRTK-Quest
- 46. Interactions SDK for Oculus Quest (available 2022) Source: Facebook Connect 2021 - https://www.facebookconnect.com/en-gb/
- 47. DEMO Hand tracking using Oculus Quest and Mixed Reality Toolkit
- 49. Enhancing the experience using Mixed Reality services
- 56. Microsoft Azure mixed reality services: • Remote rendering • Spatial anchors • Object anchors
- 57. Summary What is Mixed Reality Why Mixed Reality How to develop for AR/VR VR Interactions / Locomotion Performance considerations Using Oculus passthrough mode Telepresence Developing for AR Mixed Reality Services
- 58. Key takeaways: Check performance early and often to avoid motion sickness Design properly the experience for maximising the user immersion Use cloud services to enhance the user experience (speech, vision, anchors) Collaboration using AR/VR is an important topic which requires research
- 59. “Any sufficiently advanced technology is indistinguishable from magic” Arthur C. Clark, Profiles of the future, 1961
- 60. References AR / VR / Mixed Reality samples https://github.com/davidezordan SteamVR Unity plugin https://github.com/ValveSoftware/steamvr_unity_plugin Mixed Reality Toolkit https://aka.ms/MRTK Unity XR Interaction Toolkit https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@0.9/manual/index.html Unity XR Interaction Toolkit examples https://github.com/Unity-Technologies/XR-Interaction-Toolkit-Examples Goldman Sachs VR/AR report https://www.goldmansachs.com/insights/pages/virtual-and-augmented-reality-report.html
- 61. References Coursera “Introduction to Virtual Reality” https://www.coursera.org/lecture/introduction-virtual-reality/introduction-to-plausibility-illusion-psi-K5PGj Cognitive Services https://azure.microsoft.com/en-us/services/cognitive-services Oculus VR performance optimization guide https://developer.oculus.com/documentation/pcsdk/latest/concepts/dg-performance-opt-guide Mixed Reality Academy and Azure Services tutorial https://docs.microsoft.com/en-us/windows/mixed-reality/tutorials Unity: the most common mistakes to avoid https://unity3d.com/how-to/unity-common-mistakes-to-avoid Microsoft Ignite https://8gportalvhdsf9v440s15hrt.blob.core.windows.net/ignite2017/session-presentations/BRK2404.PPTX
- 62. References Oculus passthrough APIs https://developer.oculus.com/experimental/passthrough-api/ Kowalski, M.; Naruniec, J.; Daniluk, M.: "LiveScan3D: A Fast and Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors". in 3D Vision (3DV), 2015 International Conference on, Lyon, France, 2015 https://github.com/MarekKowalski/LiveScan3D Displaying a Point Cloud Using Scene Depth https://developer.apple.com/documentation/arkit/environmental_analysis/displaying_a_point_cloud_usin g_scene_depth iPad LiDAR Depth Sample for Unity https://github.com/TakashiYoshinaga/iPad-LiDAR-Depth-Sample-for-Unity Prof. Mark Billinghurst - COMP 4010: Virtual Reality lectures – Introduction to XR https://www.slideshare.net/marknb00/comp-4010-2010-lecture1introduction-to-xr
- 63. References Prof. Mark Billinghurst - COMP 4010: Virtual Reality lectures – Research directions https://www.slideshare.net/marknb00/comp4010-lecture12-research-directions
- 64. Thanks Davide Zordan B: https://davide.dev E: mail@davide.dev T: @DavideZordan