The Decade of Augmented Reality

The beginning of the year is the time to look forward and try to predict what will be happening during the year. Even if this year almost everything is different due to the pandemic we had a virtual CES (with a few relevant announcements on AR - great summary by Terry Schussler here) and in addition to all the webinars, virtual conferences and fireside chats we also saw a number of excellent blog posts about the AR trends for 2021. Just to name a few: Skarredghost, Bas Gezelle and Tom Emrich's reality check of his own 2020 predictions (watch out for his predictions for 2021 which are yet to be released) are excellent reads and very insightful. In summary everyone in the industry agrees that AR is on a growth path and I would not argue against that. Though will 2021 be “the year of Augmented Reality”?

For no other technology with an equally transformational impact as AR there was a single year where a breakthrough happened. Or can you tell when was “the year of the internet” or “the year of the cloud”? Technologies unfold their impact at different speeds in different areas and the same is true for AR. Let’s therefore look at a few dimensions of AR technology and their status:

AR vs. VR

There is a lot of discussion on if and how AR and VR are going to converge. As see-through AR solutions (AR glasses like Hololens which let you see the real world directly in combination with the virtual image) have a lot of limitations (more on this below). One solution is pass-through AR, i.e. take a camera input and render the virtual objects on top. This is what is done in mobile AR (i.e. AR on mobile phones and tablets). This is also a path for VR headsets with external cameras. This enables you to use some of the advantages of VR optical systems in an AR experience, especially the larger field of view, but also solving the “how-to-show-a-black-pixel-in-AR-problem”. On the other hand this is not a solution for outdoor, mobile AR usage so i believe this will be mostly a solution for specific enterprise use cases. The other way around, i.e. using AR glasses for VR is technically simple, you just need to put some cover in front of the lenses to block the real world's light. Compared to VR headsets the experience is limited though but may have its merit in specific scenarios. If ultimately AR and VR will fully converge is an open question. 

Mobile vs. Headworn

Today the main consumption devices for consumer AR are by far mobile devices (i.e. a smartphone or a tablet) rather than headworn devices (i.e. glasses like MagicLeap, Hololens or nreal light). It’s not prime time yet for headworn devices as the weight, size, optical quality, performance, price point and a few other parameters are still far away from what marketing videos are pretending. Technically there are two major challenges: different than smartphones which in the early 2010s converged to bigger form factors above 4.5” glasses have to be as light and consume as little volume as possible. The other even bigger problem for see-through AR is optics: it is very hard (or even impossible) to combine a large field of view, a high image quality, a good combination of natural light from the real world and the virtual screen, thin lenses (with prescription) and a short distance to the eyes. Read Karl Guttags blog to understand why. That means every-day, fashionable AR glasses are yet out for quite some years and I would be more than surprised if Facebook or Apple would have solved this already - the upcoming glasses from Apple will certainly be a smart approach but not yet the “full solution” the industry is looking for. 

In the short-term headworn AR will be very relevant for enterprise applications like logistics, maintenance and quality assurance where headworn solutions are already deployed. But the consumer mass market for AR experiences can only be addressed through mobile AR. 

Tethered vs. Un-tethered

The first VR solutions were all driven by PCs, tethered through a cable, simply due to the fact that the computing power needed to render the high resolution and frame rates required was not possible on the device. Still today high-quality VR can be done only through very capable PCs while mobile VR headsets like the Oculus Quest conquer the market as those systems are much more affordable and easy to maintain - at the cost of visual quality. The path forward to deliver high-quality on VR headsets is to make the link wireless, first to a local PC and eventually to an edge cloud (using solutions like NVIDIA CloudXR or Holo-Light’s ISAR to stream the high-quality visuals to the device). 

In headworn AR the journey was a bit different: while the first true AR headsets like the Hololens or MagicLeap were “all-in-one” devices recently smartphone tethered devices like the nreal light came to market - mainly to achieve lower price points and lower weight at the cost of having a cable attached to the glasses. This will evolve also to wireless tethering eventually before also the smartphone as a driver for this type of glasses will be replaced with a virtual and much more powerful computer in the edge cloud. 

Single user vs. Shared

Most of today's AR experiences are consumed by a single user. Of course you can share the videos created, but you can hardly experience virtual objects through AR by people in the same location (either at the same time or at different times). Just think about a simple example: you want to create a virtual flower in your living room. You would not  only want to record a video of that, but you would want your family members to see it through their devices (which may be on different platforms) . Also, when you would leave the room and come back you would want to see it at the same location as before. Or when you moved the flower (virtually) and your wife came back she should see it at the new location. This would create the level of immersion we all want to achieve, moreover only this capability will unlock a huge range of use cases. As of today this is not easily possible, though. There are solutions like Google Earth Cloud Anchors or Azure Object Anchors,but they are not yet easy to implement, nor available across all platforms. While this will be solved technically over the next few years the fundamental question of data privacy is probably the harder nut to crack: who should get access to the data (which will be essentially a digital twin of your house, a shop floor or an office space)? It is yet unclear if the large internet giants will be able to just copy and paste the approach they successfully did with the personal data of consumers - they are trying, of course. There is an opportunity though for other players to build alternatives if this is done quickly, with high quality and scale. 

Quality

To unlock the full range of augmented reality use cases, the quality of the experience is decisive, especially when it comes to a general “all-day” AR interface. Some of the current quality constraints are related to devices issues discussed above. 

Another aspect are the user interfaces, which need to be rethought from scratch. Today a common approach in AR and VR is to just recreate the physical interfaces we had before like a virtual keyboard or a pointer or a virtual screen. However those are not easy to use and carry on their limitations into the world of AR, sometimes even amplified. AR requires new paradigms of interaction, away from a declarative interaction, where the user explicitly expresses his input through a touch or another trigger, towards a contextual interaction where depending on the users condition and the physical context (location, elements in the room, etc.) an interaction or experience is automatically happening (think about it as instead of opening an app the app opens itself when you need it). This requires a lot of data and a strong AI and a lot more to enable this. 

Sensor quality and visual quality are obviously other key factors , which I will save for another time. The bottom line is, we are just at the beginning but things will only get better from here. 

What’s next?

We are just at the beginning of the AR revolution and there are tons of opportunities not only for the big internet giants, who are trying to capture this market, but also and especially for new entrants, new coalitions but also startups and literally anyone with good ideas. In 2021 with the latest generation of smartphones with better sensors (like the LiDAR scanner in the iPhone 12 Pro, SW advancements in the areas of AI, real world sensing and understanding, powerful infrastructure like the edge cloud and fast network access via fiber, WiFi6 or 5G, the fundamental enablers for good quality mobile AR are becoming more and more ubiquitous. In the consumer space the market focus is on entertainment and retail (“Try-on / Try-out” features). These are predominantly single user experiences, so solutions which are able to deliver true shared AR experiences will have the opportunity to stand out. Especially the combination of mobile AR and multiplayer fuels opportunities for hyperlocal mobile AR gaming (with your family during the pandemic and with your friends hopefully soon).  

However, there is still a long way to go to achieve the dream of everyone engaged with augmented reality, unleashing the full power of AR and enable a huge variety of use cases: to have unobtrusive interfaces in the form of glasses that are lightweight, comfortable, fashionable, widely socially accepted, safe to wear all-day and affordable. These interfaces will provide the ability to experience and manipulate persistent virtual objects in the context of the real world and this can be done jointly with any other person. 

We are well on the way to get there. It won’t be a short way and it won’t be a straight one. It will last a decade. Though, it’ll be this decade and we can step by step enable more use cases that will improve the lives of all of us - in the real world!