Michael Abrash's "What VR could, should, and almost certainly will be within 2 years"

Hi! I’m Michael Abrash, and I’m part of the team working on virtual reality at Valve.
I have a nice talk prepared for you today, with a lot of meat and a look at an exciting future – and I'll
get to it in a minute, but first I thought I'd sum up the whole thing in 59 words. I’m sure someone out
there will be counting, so I’m just going to read this to make sure I get it right:
Compelling consumer-priced VR hardware is coming, probably within two years
It’s for real this time – we’ve built prototypes, and it’s pretty incredible
Our technology should work for consumer products
VR will be best and will evolve most rapidly on the PC
Steam will support it well
And we think it’s possible that it could transform the entire entertainment industry
If you come away today with nothing but those 59 words, your time will have been well spent.
Still, there’s a lot more to be said on the subject, so let’s look at the future of VR in more detail.

My personal path to working on virtual reality has been a long and winding one, and without the good
fortune to encounter a couple of visionaries 20 years ago, I might never have taken the first step on
that path. Back then, I understood networking – after all, I worked at Microsoft, and used the network
constantly – and I had seen demos of 3D rendering on workstations and had written a simple 3D
package myself, so I knew about realtime 3D too. And I had read and watched a ton of science fiction,
so I was familiar with the concept of virtual worlds. Fascinating as the idea was, though, and even
given my knowledge of 3D and networking, virtual worlds still felt more like fiction than like something
to think about seriously.
Then I read Neal Stephenson's Snow Crash, and it all started to come together. When I thought about
what it would take to build the Metaverse, I estimated that I had at least an idea of how to implement
maybe 80% of it with existing technology. I was too optimistic – I'm still waiting for my first VR
swordfight – but still, Snow Crash made me realize that networked 3D virtual worlds were ripe to
happen.

Later, I got together with John Carmack, and he talked for two hours straight about how he was going
to make it possible to create virtual 3D worlds running on persistent servers, with the ability to link
servers together and for anyone to build their own worlds, and how those virtual worlds would accrete
to form cyberspace, and suddenly I saw the shape of things to come, and wanted very much to be a
part of it. I jumped in to work with John, Quake happened, and today, tens of millions of people
immerse themselves in networked 3D on a regular basis.
And I wouldn't have been a part of it without Neal and John pointing the way to the future.

I’m not a visionary, but I hope to do much the same for you today with a part of the Metaverse that
hasn’t yet happened, virtual reality. I’m sure you’re all familiar with the Oculus Rift DK1, and I’d guess
that most of you think it’s just an interesting curiosity at this point. That’s a reasonable take right now,
but we think you should pay close attention, because VR is likely to have a big impact much sooner
than you think – and Valve’s working hard to make that happen.

Two years ago, we identified virtual reality as a promising way to enable compelling new experiences.
Our research and development since then has led to prototype head-mounted displays – like this one
– and a set of demos that together create a powerful sense of what’s known as presence.

I’ll talk much more about presence later, but, briefly, it’s the sense of being someplace else while in
virtual reality; many people feel as if they’ve been teleported. Presence is an incredibly powerful
sensation, and it’s unique to VR; there’s no way to create it in any other medium. Most people find it to
be kind of magical, and we think that once people have experienced presence, they’ll want it badly.

The same hardware that enables presence is also highly effective at reducing motion sickness. Our
demos show that given the right hardware it’s entirely possible to create astonishing VR experiences –
experiences that could only happen in VR – that don’t make people sick. And we strongly believe that
it’s feasible to use the same technology to ship a consumer version within two years.

The obvious question is: who might actually ship this class of VR hardware within two years? Oculus
is clearly the leading candidate. They’ve made VR a hot topic again, and they’ve already shipped
more than 50,000 Rift DK1s. The DK1 is not good enough to enable strong presence for most people,
but Oculus’ new version of the Rift, Crystal Cove, offers higher resolution, lower latency, low
persistence, and translation, all of which are key elements of presence, as we’ll see later, so it’s a big
step in the right direction.
Valve’s goal is to enable great VR for the PC, so we’ve shared what we’ve learned through our R&D
with Oculus. We’ve showed them our prototypes and demos, we’ve explained how our hardware
works, and we’ve provided them with feedback on their hardware designs. By showing them a
prototype with low persistence, we convinced Oculus of its importance, and the lack of blur in Crystal
Cove is a direct result of that. We collaborated with Oculus on tracking as well. We’re continuing to
work with them to improve tracking, displays, lenses, and calibration, and we’re excited about where
they’re headed. If Oculus executes well, and so far they seem to be, I think they could well deliver
strong presence on the PC within the next two years; we hope that happens, because it would be a
huge boost for PC VR.

Once hardware that supports presence ships, we think it has the potential to cause a sea change in
the entertainment industry. Not only could VR rapidly evolve into a major platform, but it could actually
tip the balance of the entire industry from traditional media toward computer entertainment. You see,
for latency and bandwidth reasons, presence can only happen with a head-mounted display
connected to a device capable of heavy-duty 3D rendering, so there’s no way that TV, movies,
streaming, or anything that lacks lots of local compute power is up to the task. A corollary is that the
PC – Linux, Windows, and OSX – is going to be the best place for VR, because that’s where the most
FLOPs are.
The bottom line is that we think VR has tremendous potential as a gaming platform and has an
excellent shot at taking off soon, so we’re working to enable great VR for the Steam community. First,
we’re building VR support into Steam, as you’ll learn from Joe Ludwig’s talk. Second, we’re driving VR
hardware forward on the PC by continuing to do R&D and by collaborating with Oculus. Finally, we’re
prototyping software to figure out how to produce great VR experiences, and we’ll share what we find
with you as we learn it.
We have no current plans to ship VR hardware ourselves, but that could change in the future. Right
now, we’re just figuring out what’s fun about VR, and as we learn more we’ll do whatever’s needed to
enable VR on the PC and Steam.

It should be pretty obvious by this point why we think you should keep an eye on VR, but of course
you should develop your own opinions about its potential. The best way to do that is to try our demos.
Unfortunately, the demos currently take half an hour and require a dedicated room, and there are only
two such rooms at Dev Days, so we haven’t been able to give most of you demos here. What I can
offer is this: any Steam partner who’s in Seattle in the next few months and wants to try the demos
can send me email at mabrash@valvesoftware.com, and we'll schedule a day and time. Once you’ve
tried the demos, I think you’ll agree that the near-term future of VR is brighter than almost anyone
imagines.
The key to that bright future is presence, so it’s important that you understand what it is, why it
matters, and what it takes to enable it. Before we can discuss presence itself, though, we need to lay
some groundwork.

At Valve, we’ve spent a lot of time investigating the factors that affect how people experience virtual
reality. Most critically, we’ve learned that the key to making the experience unique and compelling is
convincing perceptual systems that operate at a low level - well below conscious awareness - that
they’re perceiving reality.
Since by definition you’re not directly aware of the relevant systems, it’s hard to understand what they
are and why they matter. To illustrate, I’ll briefly describe an example that doesn’t involve VR but that
does clearly demonstrate low-level processing.

The example involves the scene you see on this slide, which contains a number of cues that suggest
a considerable distance to the farther sphere, including lighting, perspective, texture convergence, and
the relationship between the spheres and their shadows. The user’s task is to adjust the size of the
nearer sphere, using the arrow keys, until the 2D circles formed on the screen by the two spheres look
to be the same size.
What unfailingly happens is that the user makes the front circle larger than the back circle. (You can
try it for yourself at the URL shown.) In fact, take a moment now and look up the alley. Which circle
seems bigger? To me, the back circle is clearly the bigger one. In fact, though, the front circle is nearly
20% larger.
This may not work for you, since seeing the illusion at a distance is not ideal, but if you had a similar
result, that’s not surprising. Research using functional MRI imaging shows that even in the earliest
layer of the visual cortex, long before conscious awareness comes into play, the sphere that appears
farther away projects to a larger area in the brain even if it occupies the same area on the retina. The
visual system has already judged the sizes before the data reaches conscious awareness.
There are many similar processors in your visual system, all contributing to your perceptions of the
world in ways you’re not directly conscious of. Here’s another one:

Take a look at the center piece of each ‘X’. One’s gray, and the other’s dark yellow, right? Now let’s
mask out the background, without changing the colors of the center pieces, and see what happens.

It turns out that due to the need for consistent vision across a wide range of illumination conditions,
there’s a huge contextual element involved in low-level color processing. Consciously, you’re not even
aware of this, just as you’re not aware of the processing that alters the perceived size of the circles in
the previous example, but collectively, signals like these from low-level processing form your sense of
the world. If virtual reality provides inputs that stimulate the low-level processors properly, you’ll feel
like you’re actually someplace; if it doesn’t, you’ll feel disconnected from the scene in some
indefinable way.
So far this hasn’t been directly relevant to VR, so let me give another example that’s closer to home.
This is a bit of a spoiler for one of our demos, so if you don't want to hear it, cover your ears. I’ll wait a
moment while you decide.

We have a demo where you're standing on a ledge, looking down at a substantial drop. Here’s the
scene; the stone texture is a diving board-like ledge far above the floor of a box room that’s textured
with outdated web pages. Yes, I know it doesn’t look like much of anything here, but that just
illustrates how different VR can be from staring at a screen. Looking at this on a screen (even when
it’s not warped) doesn’t do anything for me, but whenever I stand on that ledge in VR, my knees lock
up, just like they did when I was on top of the Empire State building. Even though I know for certain
that I'm in a demo room, wearing a head-mounted display, looking at imagery of the inside of a badly
textured box, my body reacts as if I'm at the edge of a cliff. What's more, that effect doesn't fade with
time or repetition. The inputs are convincing enough that my body knows, at a level below
consciousness, that it's not in the demo room; it’s someplace else, standing next to a drop.

This feeling of being someplace real when you’re in VR is well known to researchers, and is referred
to as “presence,” and it’s presence that most distinguishes VR from 3D on a screen. Presence is
distinct from immersion, which merely means that you feel surrounded by the image of the virtual
world; presence means that you feel like you’re in the virtual world.
Trying to describe presence is bound to come up short – you can only really understand it by
experiencing it – but I’ll give it a shot. Presence is when, even though you know you’re in a demo
room and there’s nothing really there, you can’t help reaching out to try to touch a cube; when you
automatically duck your head to avoid a pipe dangling from the ceiling; when you feel uneasy because
there’s a huge block hanging over you; when you’re unwilling to step off a ledge. It’s taking off the
head-mounted display and being disoriented to find the real world there. It’s more than just looking at
someplace interesting; it’s flipping the switch that makes you believe, deep in your lizard brain, that
you are someplace interesting. Presence is one of the most powerful experiences you can have
outside reality, precisely because it operates by engaging you along many of the same channels as
reality. For many people, presence is simply magic.

Different people experience varying degrees of presence in response to our demos; clearly there are
significant variations within the population. Responses have strengthened overall as we’ve improved
the experience, so we expect presence to become steadily more powerful as VR technology evolves.

Presence is hard to quantify, but our demos have shown that it is a very real and compelling
phenomenon, one that hooks far deeper into the perceptual system than anything that’s come before,
and it’s why we’re so excited about the future of VR. It’s our belief that great VR will be built on
presence, because it engages you at a deeper, more visceral level than any other form of
entertainment, and can only be experienced in VR. Consequently, we think that building hardware
that’s capable of delivering a strong sense of presence is the key to VR’s success.
So, what does it take to create a sense of presence? Hard-won experience from a lot of R&D and
prototyping has taught us that all of the following aspects have to be good enough before a strong
sense of presence emerges:

I’ll talk about each of these in more detail in a minute, but first I’d like to note that these elements also
reduce motion sickness. While the causes of motion sickness are not well understood, there are good
reasons to anticipate that many of the same factors that affect presence should affect motion sickness
as well. For example, flawed tracking causes a mismatch between what your eyes see and what your
vestibular system reports, and those sorts of conflicts are thought to be key to motion sickness.
Let’s take a few minutes to look at each of the elements and see why they matter.

A wide field of view is obviously required so that you feel immersed, but also provides peripheral
visual cues that are critical for motion, balance, and situational awareness. Presence starts to work
somewhere around an 80 degree field of view, and improves significantly at least out to 110 degrees,
which is the widest we’ve tested.

Resolution is a particular issue with VR because the wide field of view spreads out and magnifies the
pixels; the per-degree pixel density of a 1K x 1K, 110-degree VR display is roughly one-seventh that
of a big-screen TV, and about one-tenth that of the eye itself. In fact, it’s actually lower pixel density
than the original Quake running at 320x200

which as you can see is pretty low density. We’ve found that 1080p seems to be enough for presence.
We expect that 1440p, or better yet 2160p would be huge steps up, but won’t know until we can get
appropriately sized panels at those resolutions.

Persistence is the length of time each pixel remains lit. It’s not that important for TVs, monitors, or
mobile, but it’s uniquely important for VR, due to the much faster motion of the eyes relative to the
head-mounted display. This is especially true due to a low-level visual system called VOR, which
allows the eyes to remain steady during rapid head motion. The longer pixels persist, the farther the
pixel images smear across the retina when the eye is moving, and the blurrier the scene becomes.

Here’s an example of the sort of smear that results from persistence. The image on the left is with the
head held still, and the image on the right is a simulation of what happens with a leisurely 120 degrees
per second head turn rate. On a 60 Hz full-persistence display, that results in two degrees of smearing
across the retina per frame, which as you can see reduces detail considerably.
We’ve found that persistence of 3 ms or less is required for presence with a 1K x 1K, 110-degree
head-mounted display. Shorter persistences will be required at higher pixel densities.

Given a wide field of view, once persistence is lowered, refresh rate has to increase; at 60 Hz, low
persistence images flicker badly. In order to address this, we built the fastest low-persistence headmounted display we could; it runs at 95 Hz, and that successfully eliminates visible flicker. A
somewhat lower refresh rate may be adequate, but we haven’t done the experiments yet.
It’s worth noting that VR quality suffers noticeably when rendering doesn’t keep up with frame rate,
and that it’s going to be a challenge to maintain 95 Hz stereo rendering, especially as resolutions
climb.

Our prototype uses global display, where all pixels are illuminated simultaneously. This avoids the
compression, stretching, and tilting problems that can occur with the more standard rolling display,
where pixels are illuminated in a scanned sequence over the course of a frame. It may be that the
artifacts of rolling display can be largely corrected by adjusting the frame buffer during each frame to
account for eye motion, but that’s not yet proven; also, while head motion can often be used as a
proxy for eye motion, without eye tracking there will always be failure cases, and low-latency headmounted eye tracking is not a solved problem. So right now global display is the only approach known
to work.

It seems like the selection of the lens or lenses and the design of the optical path should be simple.
We have to use a cellphone panel with magnifying lenses, because there’s currently no reasonable
way to get a wide enough field of view with a microdisplay or waveguide. Furthermore, due to the
demands of field of view, weight, and industrial design, there can only be one or maybe two lenses per
eye. How hard could it be to optimize such a simple system?
The problem is that there’s no way that just one or two lenses can produce ideal VR viewing; there are
many different types of aberration and distortion, so something like this:

would be required to get everything exactly right. For reference, the two big lenses on the left are
more than a foot in diameter, and the whole assembly would weigh more than five pounds – not
particularly practical for a head-mounted display.
With a limitation of just one or two lenses, fixing one problem often means making another one worse,
especially with the requirement for a wide field of view. This is further complicated by the many
possible focal lengths, sizes, and viewing distances.

So lens design becomes a process of searching for the best set of tradeoffs in a huge space, made
even more complicated because there’s no mechanical way to evaluate the resulting visual quality.
The only test that matters is actually using the lens in VR, which is time-consuming, subjective, and
varies from person to person. We’ve developed lenses that work well enough to allow presence, but
there’s lots of room for improvement.

Optical calibration seems like a minor point, but has proven to be critical and hugely challenging,
because the human visual system is astonishingly sensitive to slight errors, especially when motion
and straight lines are involved. A scene that looks perfect when viewed statically can ripple horribly
when you swivel your head from side to side. This destroys presence, and can induce motion sickness
almost instantaneously.
It’s also impossible to figure out what else needs improving until optical calibration is dead-on. As one
example, until we got calibration right, we didn’t really notice small tracking glitches, because the
image wasn’t any more stable than the tracking. Once calibration was good enough, the glitches
jumped out at us. The bottom line is that a highly accurate process for characterizing the lenses and
correcting the rendered image is absolutely essential.

At this point, we have the ingredients we need to produce the visual quality required for presence.
However, that alone is not sufficient. The image has to be presented in such a way that the perceptual
system accepts it, and that requires rock-solid head tracking that reports translation - position in x, y,
and z - as well as orientation. We’ve found that we can get presence with tracking accuracy of a
millimeter in position and a quarter-degree in orientation, maintained over a volume no smaller than a
meter and a half on a side.
Achieving this is much harder than it sounds; there is no currently-available consumer system that
comes close. Here’s what we’re using for our demos:

It works, but you can see that it isn’t exactly consumer-friendly. We have a couple of promising
tracking systems in development that don’t involve wallpapering your house, and while they may or
may not pan out, they’re close enough to make us confident that a consumer-grade tracking system is
feasible in the near future.

The virtual image needs to be in the right place at the right time, which means that latency has to be
very low, as measured from the time head motion occurs to the time the last photon is emitted as a
result of that motion. We’ve found that latency of 20 ms, combined with good prediction, works well,
and it’s possible that latency up to 25 ms may be adequate. Past that, the virtual world no longer
seems nailed in place, and the human perceptual system is no longer convinced that it’s looking at
reality.

I want to emphasize that presence is not a property of any one of the elements I’ve discussed; it’s a
property that emerges when all of the elements are good enough. If the optics aren’t calibrated
perfectly, then the scene will warp as you turn your head no matter how good everything else is.
Likewise, no amount of fidelity will convince your visual system that a virtual scene is real if latency is
too high. Presence can’t be induced if even one of the key elements is subpar.
Okay, due to time limitations, this has been a very fast overview of the elements of presence. If you’d
like to know more, there’s lots more detail on my Valve blog.

So – how does hardware that could plausibly appear in a consumer product in the next couple of
years measure up with respect to the key elements of presence? As it turns out, things look
surprisingly good. It’s my opinion, based on Valve’s R&D, prototypes, and projections, that a
consumer head-mounted display could be built to the following spec in 2015:

This head-mounted display would support a powerful sense of presence and would have an excellent
shot at widespread adoption. VR can certainly get much better yet down the road, but that’ll require
time and major hardware R&D. In contrast, we believe everything on this slide is doable with relatively
minor tweaks of existing technology; no breakthroughs or miracles are needed, just solid engineering.

This specification is based on direct experience. Our prototype head-mounted displays have all the
characteristics I’ve specified, and enable a strong sense of presence. They’re also built from modified
commodity parts that point the way to cost-effective manufacturing at scale.
That doesn’t mean it’ll be trivial to build and ship consumer VR hardware. We’ve created an R&D
prototype, not a product. We think that the technology we’ve used will be transferrable to consumer
head-mounted displays, but lots of additional work will be required to turn that into a shipping product.
I’m confident that that work will happen soon, now that we’ve demonstrated what’s possible, and if that
happens, it should be feasible to have these head-mounted displays on the market within a couple of
years.

Beyond manufacturing issues, there’s still a lot to be solved and improved. For one thing, presence
would benefit from every one of the key elements getting better than what’s in our prototypes. We
could literally use up to 100 times as many pixels, and a wider field of view, lower latency, and all the
rest would also improve the experience; the optics in particular are far from optimal. Also, getting peruser lens positioning right is a challenge. As I mentioned, we think we’re close on head tracking, but
we don’t have a shippable solution yet, and then there’s eye tracking, which could greatly enhance
presence but is nowhere near solved. Going to a wireless connection and eliminating the tether would
be a big plus. And while we believe that it’s possible to modify existing display panels to support low
persistence, global display, and low latency, that remains to be proven, and will require the close
cooperation of a display manufacturer.

Then there’s all the stuff that needs to be figured out in non-visual VR areas. 3D audio, haptics, body
tracking, and input are going to be huge positives for presence, and they’re bigger and harder
problems than head-mounted displays. In particular, VR input and its interaction with game design is
at the core of the experience, and almost completely unexplored. We’re researching those areas, but
it’s going to take many years and the combined efforts of the whole game industry to fully explore
them.

And then there’s software.
Once the hardware's built, it won't really matter until software provides great, unique VR experiences,
and those have yet to be created. In addition to the question of how games will interact with input,
rules about how players can move around a virtual space without getting motion sick or losing
presence have yet to be figured out. We’ve found that traditional FPS movement is far from optimal
and tends to cause motion sickness, so VR may be best with slow movement and a lot of up-close
interaction, in which case we’ll have to learn how to create fun games around that. In the end, a whole
new VR gameplay vocabulary will need to be developed, just as was the case with FPSes.
Multiplayer VR is also going to be very different – and more compelling – than on a screen. Sharing a
virtual space that feels real with other people has the potential to be the most powerful online social
experience yet, but raises many questions. Remember when deathmatch and mods and team games
on persistent servers evolved? This is going to be like that, but much more so.
Then there’s content. For example, no one knows yet which art styles work in VR. Detailed scenes
that look great on a screen can look like cheesy stage sets in VR – and simple scenes can seem
startlingly real. Normal maps don’t look good, and textures sometimes do and sometimes don’t. So
we’re going to have to come up with a whole new visual vocabulary for VR too.

This is where you come in. Platform shifts enable software to create breakthrough experiences – think
Myst, Quake, Wii Sports, Angry Birds. VR has the potential to be one of the biggest platform shifts
ever, and you can be on the leading edge of that. That’s all the more true because we’ve learned that
great VR requires dedicated experiences designed specifically for virtual reality; immersion can make
ports to VR interesting, but great VR really requires custom software. Much as was the case with the
first 3D games, pretty much everything about the VR gaming experience remains to be invented. I’m
sure you saw some of the adoring press on the 20th anniversary of Doom; the first game that nails the
VR experience is likewise going to be fondly remembered for a long, long time.

When that first great VR game does show up, it’s likely to be on the PC, for several reasons. First, VR
hardware is going to evolve rapidly on the PC, as you can already see with the Rift, while the
consoles, if they even support VR, will remain static for years. Second, there are going to be far more
hardware and software developers figuring out how to do awesome VR on the PC than on the
consoles, and they’ll be much freer to experiment. Third, VR needs as much processing power as it
can get – remember, we’re talking about stereo rendering at 95 Hz – and high end PCs are already
much more powerful than consoles, with the gap due to increase for years to come. And because of
power and heat constraints, PCs will always be far more powerful than mobile. Finally, having figured
out how to bring presence to consumer VR, and having added VR support to Steam, Valve is going to
continue to help drive VR on the PC and Steam forward, so that all of us can deliver new, compelling
experiences. All of which means that the PC and Steam are going to be at the heart of VR for the
foreseeable future.

Today's key takeaway is simple, but it has powerful implications: a great VR system at a consumer
price in 2015 is more than just possible – it’s sitting there waiting to happen. And it will happen, if not
in 2015, then soon after.
Virtual reality on the PC over the next few years may be as exciting as anything that’s ever happened
in games. We’re sharing what we’ve learned with you, and we’ll continue to do so. There’s a huge
amount to be learned and figured out about VR, and we certainly can’t figure it all out by ourselves; I
hope that as you dive into VR, you’ll make it a two-way exchange, so together we can make VR one of
the great entertainment revolutions.

There’s a lot more to talk about, and I look forward to continuing the conversation with you at the Q&A
and on my blog.
Thank you.

Michael Abrash's "What VR could, should, and almost certainly will be within 2 years"

More Related Content

Similar to Michael Abrash's "What VR could, should, and almost certainly will be within 2 years" (20)

Recently uploaded (20)

Michael Abrash's "What VR could, should, and almost certainly will be within 2 years"