"How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD
5 likes1,518 views
The document discusses the advancements and challenges in virtual reality (VR) technology, particularly focusing on the role of computer vision (CV) in enhancing user experience. It outlines the current state of VR, its applications across various fields, and emphasizes the need for improvements in rendering, latency, and interactive capabilities. Additionally, it highlights the evolving ecosystem of VR developers and the potential business opportunities that arise from this technology's growth.
"How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD
- 1. Copyright © 2016 AMD 1 How Computer Vision is Accelerating the Future of Virtual Reality Allen Rush May 2, 2016
- 2. Copyright © 2016 AMD 2 • Virtual Reality — current view • Computer Vision support in VR • Future of VR with Advanced CV • Challenges — Technical, Business Agenda
- 3. Copyright © 2016 AMD 3 VR Has Come a Long way • Experience primarily thru close display and optics • Immersion thru control of local view & positional info • Factors: resolution, optics accommodation, vergence, frame rate, dynamic range… • 2016: ~2M users, 200,000 developers • >260 startups + key players
- 4. Copyright © 2016 AMD 4 VR Experiences — Imagination to Immersion Training and Simulation Gaming Big Data Visualization Entertainment Virtual Social World Remote Presence MedicineEducation
- 5. Photorealism in VR Full Presence Progression of REAL-TIME Visual Experiences Basic 2D Rendering Physically Based Rendering Basic 3D Rendering Immersive 2D Displays Shaders VR Level of Presence Today
- 6. Copyright © 2016 AMD 6 VR + Camera, CV Basic Functions Viewport Display vs Human FOV
- 7. Copyright © 2016 AMD 7 Functions and processes to produce VR: Latency challenges
- 8. Copyright © 2016 AMD 8 • Elementary: Feedback for eye tracking, positional information (X,Y,Z, pitch, roll, yaw, x,y,z motion- 9 DOF), extremity detection (hands, feet, etc.) • Fundamental: IR tracking, positioning, calibration • Complex: deep learning training for user’s eyes expressions, motion prediction • Sensory fusion: combining visual, tactile, accelerometers to model dynamic environment Computer Vision in VR
- 9. Copyright © 2016 AMD 9 • “Rendering of a virtual human that can purposefully interact with a real person — for example, through speech recognition, the generation of meaningful sentences, facial expression, emotion, skin color and tone, and muscle and joint movements — is still beyond the capabilities of real-time computer graphics and artificial intelligence,” write neuroscientist Maria V. Sanchez-Vives and computer scientist Mel Slater • Key improvements in: latency, resolution, FOV, color, dynamic range, focus management • Key additions: eye tracking, surround vision capture, position accuracy, object detection and tracking, 3D view angle • Key Assumption: Speeds and feeds will improve, optics will improve, additions such as eye tracking and immersive capture will develop as natural components of VR platform • Emergence of VR/CV development community — SDKs, app development, user groups, metrics, reviewers, etc. Virtual Reality — Quickly evolving and improving, but has a way to go…
- 10. Copyright © 2016 AMD 10 Example: Leap Motion “Virtual Hands” • Start of Visual Projection + Visual Feedback • Next Step: Complex rendering, recognition, depth, tracking, etc.
- 11. Copyright © 2016 AMD 11 • Application developers: rich collection of topics to choose from • Military (long history), medical, games (obvious), cinema (new), social media, education, immersive work environment • Content creation: CGI, cameras, video composite VR Application Space — Diverse and Innovative VR Touch UIHTC Vive Prototype Demo VR Hockey
- 12. Copyright © 2016 AMD 12 • “The seat at the 50-yard line in the first or second row, that might go for around $1,000 a ticket, but what you could also do is sell that same seat for $10 a ticket to 100,000 people.” – Max Cohen, Oculus • Seeing vs. experiencing: immersive and sensory • Collaborative: Immersive telepresence • CV: eye tracking, face reconstruction, expression projection, etc. Advanced VR Applications: Extending Basic Ideas to Make Compelling Experiences Virtual conferencing
- 13. Copyright © 2016 AMD 13 What Does Advanced VR Look Like? Gaze detection, learning and prediction Automotive safety: obstacles, cues, dictionary detection Once baseline VR is acceptable, many new apps, features, ideas, and functions can be developed
- 14. Copyright © 2016 AMD 14 • New solutions to old ideas • Low latency rendering: creative management of pipeline rendering (ex: LiquidVR) • Foveated rendering • Sync of sensor, detection, overlay • New Tools • Stitching, mixing content, workflow management What Does it Take? Videostitch panoramic workflow Mix in: CV detection, recognition, learning, tracking and tagging
- 15. VR Vision To achieve the vision of full presence you need Scalable CPUs, GPUs, Accelerators Sound Other Senses Sight Sensory Integration
- 16. Copyright © 2016 AMD 16 Business Opportunities and Ecosystem
- 17. Copyright © 2016 AMD 17 • Content Developers • Obvious: game developers. 3D games natural transition to 3D VR • VR cinema. 360o capture, reformat, render, projection to VR display • UI: mouse/keyboard replacement • Captured content mixed with graphics content • CV overlays: detection, object tagging, tracking • Platform SDKs: standard methods for development • Testing, metrics, distribution, support • Optimization: compilers, libraries, porting to new platforms Ecosystem
- 18. Copyright © 2016 AMD 18 • Game titles: sell vs. rent vs. service? • Collaborative: time on station? Server access • 1+1+1=4 (cardboard + smartphone + camera = VR/CV) • Embedded: system integration + services Business Models
- 19. Copyright © 2016 AMD 19 Challenges Going Forward
- 20. Copyright © 2016 AMD 20 • Improve render and display experience • Limit of human vision, based on foveal cone spacing • 2x210degx150@ 1pix/arc-min-> ~226Mpix • At 90FPS ->~20Gpix/sec rendering speed: settle for much less • Need good optics, lens shading, CA, etc. • Render complexity • 1k-10k+(!) Ops/Pix ->100’s to1000’s GOPs on GPU Technical and Design Challenges — Display and Graphics
- 21. Copyright © 2016 AMD 21 Technical Challenges — Camera and CV • Example: Hand and Eye Tracking • Detail capture, frame rate, tracking, position, prediction • New research • Deep learning to customize and train
- 22. Copyright © 2016 AMD 22 • Content Development: titles, stories, 360/180 surround, volume audio… • Distribution: Gb of content- how to efficiently transport • How to test, evaluate, use feedback in content creation • Who can make money? Challenges — Ecosystem and Business Environment
- 23. Copyright © 2016 AMD 23 • VR — picking up as a business! • Improvements in historical limitations • Cost effective solutions • Improvements in performance • CV — accelerates emergence of wide variety of applications • Common attributes — detection, tracking • Unique attributes — accuracy, categories (medical vs. games, etc.) • Ecosystem — SDK, user groups, content creators, reviewers, • Business — new paradigms in entertainment, efficiency, capabilities Summary
- 24. Copyright © 2016 AMD 24 • Alger, Mike, “Visual Design Methods for Virtual Reality” Sept 2015 • MacVey, Matthew “Exploring Future Reality”, NYC Media Lab, 2016 References