Michael Hoffman (Object Theory): Mixed Reality Use Cases and Challenges
- 1. Mixed Reality Use Cases and Challenges Michael Hoffman Co-founder / Partner Object Theory michael@objecttheory.com
- 2. As a trusted advisor, Object Theory helps F1000 clients strategically identify, design and build XR solutions that provide real business value utilizing today’s XR devices
- 3. Mixed Reality Benefits • Hands-free – Increase information access efficiency while performing tasks • Shared 3D context – Accurately convey focus & intent to reduce misunderstandings • Stereoscopic vision – Improve comprehension of subject matter • True scale – Accurately experience & evaluate spatial relationships and appearance • Spatial mapping – Reduce mistakes by proximity of information to objects/environment • 3D sound – Increase sense of presence, and accurately direct attention to focal points • Data visualization – Improve comprehension of complex geospatial data sets • High fidelity 3D objects – • Reduce need for expensive 3D print & paint • Reduce dependency on expensive equipment and supplies • Safely train for dangerous tasks • 3D environments – Leverage spatial memory to improve retention & navigation Quantifiable business value from these unique features & benefits:
- 4. Mixed Reality Use Cases Remote Collaboration Product Design Annotated Environment Premium Selling Soft Skills Training Hard Skills Training Space Planning Situation Planning Experience Invisible Data Medical Imaging Remote Expert Infinite Screen Education Expert Coaching Automated Guidance Art and Creativity Entertainment
- 5. Remote Collaboration The Business Challenge When dispersed teams collaborate on 3D problems, it’s difficult to establish an accurate shared context and convey intent with existing 2D tools such as screen sharing, which can result in costly mistakes. Mixed Reality Solution Collaborators feel a sense of presence, plus virtual geospatial annotations and markers can accurately convey intent to establish a more accurate shared context resulting in more informed decisions and fewer mistakes due to miscommunication. Why Mixed Reality? • Gaze indicates focus and intent • 3D Spatial sound provides cues and sense of presence • Stereoscopic vision improves comprehension • Geospatial annotations provide critical context • Ability to see content while conversing/collaborating Potential Challenges • 3D audio chat requires reliable internet connection
- 6. Product Design The Business Challenge When designing products, it is difficult to understand how all of its components interplay and how it will appear as a final product. Furthermore, efficiently collaborating between different design disciplines to find the best outcomes is challenging using existing 2D means. Mixed Reality Solution Mixed Reality makes it possible for design collaborators to establish an accurate shared context of a product under design for efficient, accurate decision making and faster iteration cycle times. Why Mixed Reality? • Stereoscopic vision • Accurate, true scale visualization of product • Ability to see content while conversing/collaborating • Efficient ability to try alternative designs Potential Challenges • Color fidelity not as good as calibrated monitors • Large model geometries
- 7. Annotated Environment (BIM / IoT) The Business Challenge BIM data and environmental sensor data is collected by the terabytes, yet most of that data is never utilized. Only a fraction of the potential value is extracted. Mixed Reality Solution With ML/AI + MR, the collected data can be analyzed for patterns an then the insights can be presented geosnatially right where an issues need addressing. Further, the relationships between the metadata and the physical plant are much easier to glean from observation. Why Mixed Reality? • Virtual on Physical • Ability to see data while performing tasks • Stereoscopic vision for better insights Potential Challenges • Not recommended yet for outdoor use cases • Need to use markers when navigating large facilities.
- 8. Premium Selling The Business Challenge It is difficult to sell expensive products, services and experiences that are hard to put in front of a client or hard to explain in detail. Mixed Reality Solution Telling the rich story of an offering in 3D makes it more personal, relatable, understandable and memorable Why Mixed Reality? • Stereoscopic vision to better visualize • Alignment to physical to reveal hidden features • Spatial story telling is more compelling and memorable Potential Challenges • It takes 5 - 15 minutes to acclimate to mixed reality
- 9. Soft skills training The Business Challenge It is expensive to train soft skills that require understanding nuance and adept reaction to varying external stimuli and behaviors. Mixed Reality Solution The combination of AI+MR reduces training cost by making it possible to recreate realistic interactive soft skill scenarios, automate the AI/ML assessment of performance and generation of precise feedback. Why Mixed Reality? • Stereoscopic vision for more realistic scenarios • Alignment to physical further increases realism • Gaze indicates trainee focus and intent • Positional info, camera and mics feed data to cognitive services • Hands free enables using actual soft skill devices Potential Challenges • No significant challenges
- 10. Hard skills training The Business Challenge It is expensive to train skills to a large workforce that require learning a wide variety of steps and operations, particularly in high-turnover environments. Mixed Reality Solution Creation of self-guided, interactive skills training tutorials substantially reduces the labor cost of repeatedly training the same skill. Why Mixed Reality? • Hands free • Stereoscopic vision for better, faster comprehension • Alignment to physical enables practicing on real equipment • Information close to where you need it Potential Challenges • Creating 3D interactive content for each skill can be expensive
- 11. Space Planning The Business Challenge Experiencing the feel and function of a space under design is a critical step in the design process to reduce mistakes, yet is so costly to achieve that it’s rarely done. Mixed Reality Solution Mixed Reality makes it possible to efficiently experience and even modify a non-existent room design at true scale. Why Mixed Reality? • True size and scale • Stereoscopic vision • Ability to see content while conversing/collaborating • Align virtual changes to physical environment Potential Challenges • True scale immersive scenarios can benefit from wider FOV, particularly for new users
- 12. Situation Planning The Business Challenge Planning and monitoring complex geospatial scenarios demands comprehensive understanding of complex information across a dispersed team of experts. It is difficult to provide all relevant information to create a comprehensive and accurate shared context. Mixed Reality Solution Mixed reality enables effective collaboration across dispersed teams while interacting with a wide variety of geospatially accurate 3D information and supplementary 2D information. Why Mixed Reality? • Geospatially accurate information • Stereoscopic vision • Ability to see content while conversing/collaborating Potential Challenges • Content creation can be expensive • Geometry complexity • WiFi security (for Defense use cases)
- 13. Experience Invisible Data The Business Challenge Data analysis of 3D data sets and simulations of geospatial information on 2D screens makes it difficult to comprehend the full extent of insights that can be gained. Mixed Reality Solution With Mixed Reality, simulations and visualizations of collected sensor data or simulations can be played and animated geospatially and can be aligned with physical objects related to the data. Why Mixed Reality? • Stereoscopic vision • 3D in 3D • Align virtual to physical • Ability to see content while conversing/collaborating • Experience the inaccessible Potential Challenges • No major challenges
- 14. Medical Imaging The Business Challenge Surgeons are constantly evaluating 3D problems on 2D screens. Missing critical insights and longer assessment times can lead to worse outcomes for patients. Mixed Reality Solution Hands-free access to all relevant 2D and 3D information exactly where it’s needed enables more accurate and efficient assessment for better outcomes. Why Mixed Reality? • Hands free • Stereoscopic vision for better, faster insights • Alignment to physical reduces errors • Information proximity without physical interference • Instant access to vast amounts of 2D & 3D information Potential Challenges • Accurate alignment to body below 1-2mm difficult • Regulatory requirements for operating room • Weight and comfort for long-term use
- 15. Remote Expert The Business Challenge Flying subject matter experts onsite to support field operations is expensive and requires too many experts on staff for peak load. Mixed Reality Solution A smaller team of centralized experts can effectively assist many field technicians as needed at lower travel and labor cost. Why Mixed Reality? • Hands free enables performing tasks while being assisted • Geospatial indicators/annotations increases accuracy • Stereoscopic vision increases comprehension Potential Challenges • Requires reliable internet connection for audio & video • Safety compliance requirements • Outdoor use • Safety compliance requirements • WiFi security (for Defense use cases)
- 16. Infinite Screen The Business Challenge Many tasks require analyzing huge amounts of 2D and 3D information. We are constrained by the realistic number of pixels we can surround ourselves with. Mixed Reality Solution Mixed Reality enables augmenting existing screen real estate with a nearly infinite amount of additional 2D panels as well as 3D content for a vast increase in available information for analysis and decision making. Why Mixed Reality? • Can interact with physical and virtual world simultaneously • Not constrained by limits of 2D monitors Potential Challenges • Today’s devices don’t yet run the sophisticated apps professionals depend on.
- 17. Education The Business Challenge It is difficult and expensive to teach concepts and techniques that consume expensive supplies, depend on expensive equipment, or can be dangerous. Mixed Reality Solution For at least the early stages of the learning process, simulating concepts and techniques is a less dangerous and expensive alternative, and can reduce dependency on access to expensive equipment. Why Mixed Reality? • Hands free • Stereoscopic vision for better, faster comprehension • Alignment to physical enables practicing with some or all physical equipment, or even props. • Information close to where you need it • Instant access to vast amounts of 2D & 3D information • Ability to see content while conversing/collaborating Potential Challenges • Creating a large variety of interactive 3D content can be
- 18. Expert Coaching The Business Challenge Improving a fine motor skill such as the perfect golf swing is extremely difficult and requires deep expertise. Mixed Reality Solution IoT sensors plus mixed reality enables accurate capture and visualization of a wide variety of information for more accurate assessment and improvement of technique. Why Mixed Reality? • Hands free • Alignment to physical to see relationships and variance • 3D visualization for accurate assessment and comparison of 3D movements Potential Challenges • Outdoor use not recommended
- 19. Automated Guidance The Business Challenge Utilizing only physical props, providing all useful information to guide an audience through an experience creates traffic bottlenecks, unwanted visual clutter and limits storytelling possibilities. Mixed Reality Solution Mixed reality increases expressiveness and efficiency while eliminating bottlenecks. Any amount of interactive 2D, 3D and spatial audio content can be situated geospatially around points of interest to convey useful information in compelling ways that maximize expressiveness, comprehension and memory. Why Mixed Reality? • Stereoscopic vision for better, faster comprehension • Information proximity & alignment increases context • Hands free • Instant access to vast amounts of 2D & 3D information Potential Challenges • Outdoor use not recommended
- 20. Art and Creativity The Business Challenge Monetizing entertainment is becoming increasingly difficult. Theater and sports arena attendance continues to drop. Mixed Reality Solution Mixed Reality adds interactivity and extends the average time spent consuming entertainment. This increases the opportunities to monetize it. Why Mixed Reality? • Can augment existing means of consuming entertainment. • Can move and interact with environment while experiencing content. Potential Challenges • MR not yet a ubiquitous consumer device.
- 21. Entertainment The Business Challenge Monetizing entertainment is becoming increasingly difficult. Theater and sports arena attendance continues to drop. Mixed Reality Solution Mixed Reality adds interactivity and extends the average time spent consuming entertainment. This increases the opportunities to monetize it. Why Mixed Reality? • Can augment existing means of consuming entertainment. • Can move and interact with environment while experiencing content. Potential Challenges • MR not yet a ubiquitous consumer device.
- 22. Addressing Today’s Challenges • Safety: 3rd Party HoloLens hard hats (Trimble) • Typing: Device assisted authentication • Placement/Location: • Fiducial markers • Manual logical alignment • Global anchor sharing • Attention: 3D audio, narrative and visual cues • Static 3D Content: • Expert manual optimization • Level-of-detail (LODs) • Smart use of geometry culling • FOV: • Start small for context, then enlarge • Attention directors and audio cues • Dynamic 3D Content: Simplygon automation • Material fidelity: Hand optimized shaders • Input: • Hand controllers and clickers • Smart phones as companion input devices • All day: • USB battery packs • Device swap • Spectator View for professional videos / photos • Spatial mapping / tracking: How Object Theory addresses today’s MR challenges:
- 23. With thoughtful experience design, diligent optimization and smart engineering, today’s mixed reality devices deliver real business value for a wide variety of use cases.
- 24. Thank you! Michael Hoffman Co-founder / Partner Object Theory michael@objecttheory.com linkedin.com/in/mthoffman @objecttheory