Hearing Voices: An Overview of Augmented Reality Audio
- 1. Hearing voices AUGMENTED REALITY AUDIO CAN CHANGE THE WAY PEOPLE EXPERIENCE THE WORLD, BOTH AT HOME AND IN TRAVEL Tim Haynes Principal Storyteller Sabre Labs
- 2. What is augmented reality (AR) audio? • A layer of digital audio information that enriches the physical world • Usually associated with playback of location-based (geotagged) digital audio information • Includes the hardware and software necessary to play layers of geotagged content in a meaningful way • Can also include digital augmentation of aural senses to affect a user’s perception of an environment
- 3. Partial graveyard of failed AR audio platforms
- 4. Causes for the graveyard Discoverability — Lack of standard name or buzzword • 2011, “dynamic spatial audio” & “ambient spatial audio” were buzzwords for the next big thing with smartphones • Also: location-based audio, geotagged audio, locative audio, augmented reality audio, location-aware audio… Failure of the “why” — Startups failed to find a broad and compelling use case for AR audio, focusing on niches Lack of partnership — Most AR audio apps are startups and not integrated into, partnered with or promoted by an existing, successful platform and user base
- 5. Continuing challenge: poor mobile interfaces Content playback • Many apps (both past and present) were never optimized for mobile, even in-app • If it’s hard to find or to play content, an app will fail Content creation • Most apps have very poor interfaces for adding location-specific content, especially mobile creation • The best creation interfaces have tended to be focused on art and documentary uses in niche locations
- 6. Usability premise: Audio makes the real world more interesting This is Massachusetts Ave. in Cambridge, MA. On the next slide, try different clips of music and see how it changes the way the scene feels. I recommend trying Yann Tiersen, Amelie Soundtrack, “La valse d’Amelie, version original,” followed by some classic jazz and then something hip hop or alt-country.
- 8. How is AR audio being used? • Art — theater and conceptual art installations • Documentary— sharing and cataloguing stories • Gaming — AR and MR possibilities are endless • Navigation — associating sounds with places • Notifications — directions, alerts, messages, etc. • Social — sharing experiences, both real and virtual • Tours — cities, historical sites, museums
- 9. Placing sounds in space as interactive art Art 2010 – Audio Graffiti project, Connecticut 2010 – Scapes project*, Decordova Sculpture Park, Lincoln, MA *created w/Roundware
- 10. Blended audio tours of art spaces Art Voices: FAMSF* (2015) An “immersive soundscape” with multiple voices and music centered on 21 sculptures; users can personalize with a mix of curators, art experts and fellow patrons *created w/Roundware
- 11. Ghosts of poets in Harvard Yard Art re~verse (2015) commissioned for Harvard Litfest, “consists of a continuous layer of location- sensitive music punctuated with recorded voices of poets who have visited Harvard in the past and given readings of their work.” *created w/Roundware
- 12. High Street Stories Christchurch, NZ, aftermath of 2011 earthquakes Documentary
- 13. Tributaries* Tyneside, Northern England, WWI recordings Documentary *created w/Roundware
- 14. Listening to the voices I forgot where I was and was taken somewhere else.” —KERRY BOSSONS PLAY STORE REVIEW OF TRIBUTARIES “ Documentary
- 15. Location-based gaming is on the rise Gaming Geocaching Zombies, Run! Pokemon GO
- 16. Incorporating AR audio into gaming Gaming • Several ambitious / experimental smartphone games over the past few years have used AR audio as their main interface: • Inception and Dimensions produced location-based sound and music and use the phone’s microphone to generate a unique soundscape • Papa Sangre allowed players to navigate a virtual horror soundscape, attempting to gather lost souls and escape enemies • Meltdown never made it out of prototype, but was an Alien-esque game in which you hunted aliens on foot using only your ears • None of these AR audio games are still around (AFAIK) • Immersion in AR games can benefit tremendously from location-based audio
- 17. Using AR audio to navigate Navigation • Turn-by-turn directions are a common feature of virtually all GPS-based mapping apps, but that’s the most basic use • Just as the sounds of a city help tell you where you are, an AR audio layer of sound effects and music can convey additional spatial or location information • Art installations are examples—sound effects and music are tied to places in a exhibition and volume and directionality are directly affected by your location
- 18. MIT Media Labs, Loco-Radio project Navigation Music for navigation and discovery
- 19. Using AR audio to give people important context Notifications • At it’s most basic, having earbuds in can provide private notification of messages or alerts • More importantly, location-aware alerts can inform people of: • Points of interest • Transportation options • Potential hazards • On the commerce side—location alerts can market promotions, sales, and/or experience availability
- 20. Discoverable content for the visually impaired Notifications Ariadne GPS (2012) Featured as part of Apple Keynote 2012; create personalized maps with alerts and access audio information through touch about maps in any area. Allows users to understand an area they are unable to see with their eyes
- 21. Social elements of AR audio overlap other uses Social • Startups have attempted to use location-based audio with chat, friend proximity and other social services • Art and documentary installations connect visitors with other participants over time • Gaming uses have the potential to be deeply social • Synchronized tour groups are inherently social as shared experiences
- 22. People want to feel an authentic connection with... what makes a place different from all the other places they could visit in the world.” —ANDREW MASON DETOUR FOUNDER AND CEO “ Tours
- 23. Audio tours are increasingly important to travel Tours • Plethora of apps for “traditional” audio tours: Izi, Pocket Guide, Rick Steves, UniGuide, etc… • High adoption by Millennial and Z generations • Detour is the current AR audio darling, one-upping traditional tour approaches and incorporating limited use of AR audio elements
- 24. +
- 25. Detour NPR-style tours of San Francisco +9 other cities • Current tours are professionally produced, taking 3-6 months each • Participants need to follow tour directions • Music and speech run separately to account for different movement rates • Groups can sync tours to share in the experience • “Descript” turns editing voice to word processing Tours
- 26. Audio tours can be cinematic experiences that get you closer to the feeling of walking in another person’s shoes than any other medium can.” —ANDREW MASON DETOUR FOUNDER AND CEO “ Tours
- 27. Touring doesn’t have to be proscribed Tours • I experimented with various platforms for AR audio content: • Audio Walk — Assign audio files to geographic locations; all features currently appear broken • Echoes — Limited content (mostly European), significant login issues and lack of support for content creation • GeoTourist — Basic tours, decent discoverability; open-to-public, Web-based content creation tool (no mobile creation) • HearUsHere — Handful of tours, poor discoverability of nearby content, and content creation done only by company • Roundware – really excellent open-source, but it would take a team effort to create a viable experience • SonicMaps — Terrible interface; lots of broken links; counter- intuitive content creation
- 28. What are the advantages of AR audio? • Constant context • Adds information without monopolizing attention • Fully adaptive to individual pace and schedule • No fiddling with a device – you interface by moving • Aural illusion of directionality and distance • The brain is really good at filtering sounds, allowing audio AR to target different layers of your attention
- 29. How does AR audio apply to travel?
- 30. AR audio opportunities for travel: Art • Theater performances could overlap w/games, potentially allowing users to take on roles asynchronous w/other users or create new kinds of improve performances • Art installations enrich content and add a participatory element to art cultural experiences – either standalone AR audio or integrated w/other art – potentially driving additional traffic to specific locations
- 31. AR audio opportunities for travel: Documentary • Needs a shared social platform for documentary content for viability • If not a standardized creation app, at least a site or app that aggregates, curates and maps niche documentary apps to increase discoverability • Documentary AR audio allows people to experience locations from new and diverse perspectives, enriching the value of traveling—and spending additional time—in a particular place
- 32. AR audio opportunities for travel: Gaming • Huge opportunities for AR audio gaming experiences, both standalone and with AR visuals (more common), as well as integrated into deeper merged reality experiences • Location-based digital souvenirs are likely to be a huge part of future travel, largely driven by younger generations; if not the impetus for travel, definitely part of the anticipated travel experience • When AR audio and visuals combine, “gamification” of everyday life is likely to become more common; lessons learned now by experimenting will help shape future uses
- 33. AR audio opportunities for travel: Navigation • AR audio is increasingly essential to travel, both for direction and for discoverability • Layering a location-based soundtrack could be a powerful wayfinding method for large buildings or city areas, tying locations or areas to certain music or soundscapes • Layered with gaming elements, audio navigation can potentially drive people to visit new places • In the near future, may start to include “superpowers” like near-real time translation of voices and music
- 34. AR audio opportunities for travel: Notifications • Being able to layer directions with other sounds makes travel more seamless (vs. current method of most audio apps/streams demanding exclusivity (e.g. Google Maps stops your music to give you directions) • Audio alerts can help maintain hands-free, eyeballs-free travel through both indoor and outdoor spaces • Listening to messages (and advertisements) can be layered as part of background audio, helping with multitasking while moving
- 35. AR audio opportunities for travel: Social • To date, efforts to make AR audio social have largely failed, in part due to the challenge of most AR audio apps doing too little to be worth using; none have been better than existing social options • Airbnb Tours may be the app that creates critical mass to encourage AR audio experiences, and the leisure nature of much of Airbnb’s business may help give AR audio an inherent social utility of experiencing AR audio in the real world together • A simpler route that may emerge is for a photo-first platform like Instagram to make it easy to integrate audio clips with geotagged images, changing how people catalogue both everyday and travel experiences
- 36. AR audio opportunities for travel: Tours • AR audio elements need to be integrated into existing city- discoverability platforms, from mainstream services like TripAdvisor & Rough Guides to startups like Dojo and Peek • Tour needs to take guidance from both Detour and from prototypes like Loco-radio, taking users and usage possibilities into account in soundscape design • When Detour opens its platform and Descript audio editing, it may radically accelerate tour content development (could take off the same way podcasts have) • Integration of AR audio content into Google Maps (or similar) would be a game changer, normalizing ad hoc tour design
- 37. Recommendations to experiment w/ AR audio
- 38. Considerations for implementation • Rate of user’s movement (walking, biking, driving, etc.) • Sensory immersion level (e.g. walking=high, driving=low) • Scaling for distance and directionality of sound • Foreground vs. background layers (speech vs. music or noise) • Overlapping layers of sound • Appropriate duration of audio elements (affected by rate of movement and density of audio elements) Recommendations
- 39. Geotourist Top option for basic, location-based tour creation Recommendations
- 40. Roundware Top development platform for sophisticated AR audio • Roundtable is an open-source platform being used for many of the most creative examples of AR audio • Ideal for creating and experimenting with “niche” experiences • Allows real-time, geotagged collection of audio from smartphones • Filters and plays back collective, continuous audio streams • Creates seamless, non-linear, location-sensitive layer of audio mixed on the fly based on user input • Uses Python for server, Swift for iOS app, Java for Android app • “Soon” to be expanded to include video, photos and text Recommendations
- 41. Possible inflection points for AR audio taking off • Airbnb promoting Detour, then Detour making both creation and distribution tools open to the public • Apple, Google or Microsoft making AR audio an integral part of their existing map software • Shift to wireless headphones may be a minor, but notable step towards always-on AR audio • Visual AR (Hololens, etc.) becoming widespread, and bringing AR audio along with it (several years out) Recommendations
- 42. Thank you for reading. I HOPE THIS HAS BEEN HELPFUL. IF YOU HAVE QUESTIONS OR WISH TO DISCUSS TRAVEL IMPLICATIONS OF AUGMENTED REALITY OR OF OTHER TECHS, PLEASE REACH OUT. timothy.haynes@sabre.com