Augmented Reality: Connecting physical and digital worlds
Download as PPTX, PDF1 like899 views
The document discusses the nexus of physical and digital worlds through augmented reality (AR), emphasizing the importance of creating immersive experiences that blend both realms. It outlines the evolution and potential of MPEG technologies related to AR, highlighting tools and standards that facilitate real-time object recognition, interaction, and content delivery. The document concludes by inviting collaboration in further AR development, stressing that many opportunities for innovation remain unexplored.
Augmented Reality: Connecting physical and digital worlds
- 1. CONNECTING PHYSICAL AND DIGITAL WORLDS MARIUS PREDA, MPEG 3DG CHAIR, INSTITUT MINES TELECOM www.slideshare.net/MariusPreda
- 2. From centuries, we are building two different types of worlds Physical Informational Immersion, from a human user point of view, is making abstraction of the Physical world (remove all possible connection to it) and have stimuli only from the Informational one Immersion, presence, stimuli, perception, … stories music films knowledge
- 3. In which world our users are? The stimuli from physical worlds are familiar or at very low intensities
- 4. We intentionally block our interfaces with the physical world Improving immersion feeling
- 5. Strategies for blocking the interfaces with the Physical world Improving immersion feeling
- 6. Strategies for extending the space of the Informational world stimuli Improving immersion feeling
- 7. How far we’ve gone with this strategy? Relatively far … Improving immersion feeling
- 8. Probably too far … Improving immersion feeling How far?
- 9. It’s the time to connect our worlds
- 10. Why now? • For the first time, smartphones enable “everyone” to access Informational world while sensing also the Physical world • They change our view of the environment • Enhance the rich history and meaningfulness of the real world with powerful digital elements
- 11. Physical & Informational Augment The Reality An invitation: become architects of a new kind of bridges
- 12. In AR, almost everything remains to be invented Las Vegas This place some years ago
- 13. In AR, almost everything remains to be invented
- 14. • Marketing • Tourism • Culture • Education • Serious and less serious games • … • Hardware manufacturers • Software vendors In AR, almost everything remains to be invented
- 15. We (and few others) already started and invite you to be part of The AR Revolution In AR, almost everything remains to be invented
- 16. Tidy City
- 17. Portal Hunt
- 18. Elements
- 19. ARQuiz
- 20. Augmented Books
- 22. TECHNOLOGY INSIGHTS MPEG-V AND MPEG ARAF
- 23. MPEG technologies related to AR MPEG-1/2 (AV content) 1992/4 VRML 1997 • Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video • Part 2 - Visual: - 3D Mesh compression - Face animation 1998 • Part 2 – Visual - Body animation 1999 MPEG-4 v.1 MPEG-4 v.2 First form of broadcast signal augmentation
- 24. MPEG technologies related to AR MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4A rich set of 3D Graphics representation and compression tools
- 25. MPEG technologies related to AR MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4 2009 • Part 25 - Compression of third-party XML (X3D, COLLADA) MPEG-4 2004 • Part 16 - X3D Interactive Profile MPEG-4
- 26. MPEG technologies related to AR MPEG-V - Media Context and Control 2011 • 2nd Edition: - GPS - Biosensors - 3D Camera 201x • Compression of video + depth 201x MPEG-V - 3D Video • 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds • Feature-point based descriptors for image recognition 201x CDVS MPEG-U – Advanced User Interface 2012 A rich set of sensors and actuators - 3D Audio MPEG-H
- 27. MPEG technologies related to AR MPEG-V – Media Context and Control
- 28. Actuators Light Flash Heating Cooling Wind Vibration Sprayer Scent Fog Color correction Initialize color correction parameter Rigid body motion Tactile Kinesthetic Global position command Sensors Light Ambient noise Temperature Humidity Distance Atmospheric pressure Position Velocity Acceleration Orientation Angular velocity Angular acceleration Force Torque Pressure Motion Intelligent camera type Multi Interaction point Gaze tracking Wind Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate Electrograph EEG , ECG, EMG, EOG , GSR Weather Facial expression Facial morphology Facial expression characteristics Geomagnetic Global position Altitude Bend Gas MPEG technologies related to AR MPEG-V – Media Context and Control
- 29. • All AR-related data is available from MPEG standards • Real time composition of synthetic and natural objects • Access to • Remotely/locally stored BIFS/compressed 2D/3D mesh objects • Streamed real-time BIFS/compressed 2D/3D mesh objects • Inherent object scalability (e.g. for streaming) • User interaction & server generated scene changes • Physical context • Captured by a broad range of standard sensors • Affected by a broad range of standard actuators Main features of MPEG AR technologies
- 30. MPEG vision on AR: the MPEG AR Browser Point to a URL – no need to download new applications for each context The browser • Retrieves scenario from the internet • Starts video acquisition • Tracks objects • Recognizes objects from visual signatures • Recovers camera pose • Gets streamed 3D graphics • Composes new scenes • Gets inputs from various sensors • Offers optimal AR experience by constantly adapting interaction possibilities and objects from a remote server. Industry • Maximize number of customers through MPEG-compliant authoring tools and browsers • No need to develop a new application for each use case and device platform
- 31. MPEG vision on AR MPEG-4 + MPEG-7 + MPEG-21 + + MPEG-U+ MPEG-V = ARAF MPEG Player Compression Authoring Tool Produce Download
- 32. Technology insights Modeling the content chain – Creation – Consumption – The pivot : a representation formalism
- 33. Technology insights: the chain ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools
- 34. Technology insights: Infrastructure ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools
- 35. The ARAF browser ARAF Browser • Full scene graph representation (2D and 3D) • Static and dynamic objects and scenes • Programmable behavior (JS) • Access to sensors and cameras • Access to remote resources and services (http request and web sockets) • Support for user interaction • Support for compressed resources
- 36. The ARAF browser ARAF Browser ARAF Browser JavaScript Engine MPEG Totem Services WebSockets Server Communication service WebSockets Client ARAF Browser SC3DMC Decoder BBA Decoder ARAF File BIFS Stream SC3DMC Stream BBA Stream BIFS Decoder MPEG-4 Scene IndexedFaceSet SBVCAnimation Camera Image data Reference Images Image data Image data Image data AFAR Browser Reference Signal Detection Detected Images
- 37. MyMultimediaWorld.com • Convert media assets in a common format • Describe, index, organize, search digital assets • Compressed assets, ready for transmission A multimedia repository and processing platform supporting both creation and consumption Media & Service Servers
- 38. Technology insights: the formalism ARAF Player Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools
- 39. ARAF Description Language ARAF : an XML representation Player Compression Authoring Tool Produce Download ARAF
- 40. A set of scene graph nodes defined in MPEG-4 • Audio, image, video, graphics, programming, communication, user interactivity, animation • Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, Cam eraCalibration, AugmentedRegion Connection to sensors defined in MPEG-V • Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic, Altitude, Local camera(s) Compressed media • Image, sound, video, 2D/3D graphics Augmented Reality Application Format ARAF
- 41. Technology insights: creation ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools
- 42. Authoring ARAF • Specific tools • Generic tools – TOTEM.Designer – TOTEM.Scout Auth. Tools
- 43. Specific Authoring Tools • Scenario driven approach – Tidy City, Portal Hunt, ARQuiz, AR Book, Treasure Hunt Auth. Tools
- 44. Towards a generic Designer • Create templates for app. objects • Fill structure with data • Create GUI • Create application logic • Export data
- 45. On site designer
- 46. Conclusions • AR – the next digital revolution • For making revolutions, one needs weapons: – MPEG technologies represent an efficient and powerful set of weapons • Current MPEG activities – Improve and enrich the weapons – Package them in consistent and easy to use tools