2014 COSC 426 Lecture 2: Augmented Reality Technology

COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 23rd 2014
Lecture 2: AR Technology
mark.billinghurst@hitlabnz.org

Augmented Reality Definition
  Defining Characteristics [Azuma 97]
  Combines Real and Virtual Images
-  Both can be seen at the same time
  Interactive in real-time
-  Virtual content can be interacted with
  Registered in 3D
-  Virtual objects appear fixed in space

What is not Augmented Reality?
  Location-based services
  Barcode detection (QR-codes)
  Augmenting still images
  Special effects in movies
  …
  … but they can be combined with AR!

AR vs VR
  Virtual Reality: Replaces Reality
  Scene Generation: requires realistic images
  Display Device: fully immersive, wide FOV
  Tracking and Sensing: low accuracy is okay
  Augmented Reality: Enhances Reality
  Scene Generation: minimal rendering okay
  Display Device: non-immersive, small FOV
  Tracking and Sensing: high accuracy needed

Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

2014 COSC 426 Lecture 2: Augmented Reality Technology

A Brief History of AR (1)
  1960’s: Sutherland / Sproull’s
first HMD system was see-
through

History Summary
  1960’s – 80’s: Early Experimentation
  1980’s – 90’s: Basic Research
  Tracking, displays
  1995 – 2005: Tools/Applications
  Interaction, usability, theory
  2005 - : Commercial Applications
  Games, Medical, Industry

2008 - Browser Based AR
  Flash + camera + 3D graphics
  High impact
  High marketing value
  Large potential install base
  1.6 Billion web users
  Ease of development
  Lots of developers, mature tools
  Low cost of entry
  Browser, web camera

2005 - Mobile Phone AR
  Mobile Phones
  camera
  processor
  display
  AR on Mobile Phones
  Simple graphics
  Optimized computer vision
  Collaborative Interaction

2009 - Outdoor Information Overlay
  Mobile phone based
  Tag real world locations
  GPS + Compass input
  Overlay graphics data on live video
  Applications
  Travel guide, Advertising, etc
  Wikitude, Layar, Junaio, etc..
  Android based, Public API released

AR Today
 Key Technologies Available
-  Robust tracking (Computer Vision, GPS/sensors)
-  Display (Handheld, HMDs)
-  Input Devices (Kinect, etc)
-  Developer tools (Qualcomm, Metaio, ARTW)
 Commercial Business Growing
-  Gaming, GPS/Mobile, Online Advertisement
•  >$5 Billion USD by 2016 (Markets andMarkets)
•  >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

Applications
  Medicine
  Manufacturing
  Information overlay
  Architecture
  Museum
  Marketing
  Gaming

Applications: medical
  “X-ray vision” for surgeons
  Aid visualization, minimally-invasive operations.
Training. MRI, CT data.
  Ultrasound project, UNC Chapel Hill.
Courtesy
UNC
Chapel
Hill

Medical AR Trials
  Sauer et al. 2000 at Siemens
Corporate Research, NJ
  Stereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a
Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,
MICCAI 2002

Assembly and maintenance
© 1993 S. Feiner, B. MacIntyre, &
D. Seligmann, Columbia University
© 1996 S. Feiner, B. MacIntyre, &
A. Webster, Columbia University

PS3 - Eye of Judgment (2007)
  Computer Vision Tracking
  Card based battle game
  Collaborative AR
  October 24th 2007

AR Books – Markerless Tracking

AR Annotations
Columbia
University
HRL
© 1993 S. Feiner, B. MacIntyre,
M. Haupt, & E. Solomon,
Columbia University
© 1997 S. Feiner, B. MacIntyre,
T. Höllerer, & A. Webster,
Columbia University

Interactive Museum Experiences
  BlackMagic
  Virtual America’s Cup
  410,000 people in six months
  MagicPlanet
  TeManawa science museum
  Virtual Astronomy
  Collaborative AR experience
  ARVolcano
  Interactive AR kiosk
  Scienceworks museum, Melbourne

Digital Binocular Station
http://www.DigitalBinocularStation.com/

Museum Archeology
  LifePlus (2002-2004)
  Natural feature tracking
  Virtual characters
  Mobile AR system
  Archeoguide (2000-2002)
  Cultural heritage on-site guide
  Hybrid tracking
  Virtual overlay

Sales and Marketing
  Connect with brands and branded objects
  Location Based Experiences
  Lynx Angels
  Web based
  Rayban glasses
  Mobile
  Ford Ka campaign
  Print based
  Red Bull Magazine

Summary
  AR technology can be used to develop a wide
range of applications
  Promising application areas include
  Games
  Education
  Engineering
  Medicine
  Museums
  Etc..

“The product is no longer
the basis of value.The
experience is.”
Venkat Ramaswamy
The Future of Competition.

experiences
services
products
components
Value
Sony CSL © 2004
Gilmore + Pine: Experience Economy
Function
Emotion

Good Experience Design
  Reactrix
  Top down projection
  Camera based input
  Reactive Graphics
  No instructions
  No training

Improve the experience of picking up rubbish?

World’s Deepest Rubbish Bin
  The Fun Theory – http://www.funtheory.com

Improve the experience of walking up stairs?

Musical Stairs
  The Fun Theory – http://www.funtheory.com

Apple: The Value of Good Design
  Good Experience Design Dominates Markets
iPod Sales 2002-2007

SideTalking
  http://www.sidetalkin.com

Interaction Design
“Designing interactive products to support people in their
everyday and working lives”
Preece, J., (2002). Interaction Design
  Design of User Experience with Technology
  Higher in the value chain than product design

  Interaction Design involves answering three questions:
  What do you do? - How do you affect the world?
  What do you feel? – What do you sense of the world?
  What do you know? – What do you learn?

Interaction Design is All About You
  Users should be
involved throughout
the Design Process
  Consider all the needs
of the user

experiences
applications
tools
components
Building Compelling AR Experiences
Tracking, Display
Authoring
Interaction
Usability

Summary
  In order to build AR applications you need to
focus on the user experience
  Great user experience is based on
  Low level AR component technology
  Authoring tools
  Application/Interaction design
  User experience texting

experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Display, Tracking

Core Technologies
 Combining Real and Virtual Images
•  Display technologies
 Interactive in Real-Time
•  Input and interactive technologies
 Registered in 3D
•  Viewpoint tracking technologies
Display
Processing
Input Tracking

AR Displays
e.g. window
reflections
Virtual Images
seen off windows
e.g. Reach-In
Projection CRT Display
using beamsplitter
Not Head-Mounted
e.g. Shared Space
Magic Book
Liquid Crystal
Displays LCDs
Head-Mounted
Display (HMD)
Primarily Indoor
Environments
e.g. WLVA
and IVRD
Cathode Ray Tube (CRT)
or Virtual Retinal Display (VRD)
Many Military Applications
& Assistive Technologies
Head-Mounted
Display (HMD)
e.g. Head-Up
Display (HUD)
Projection Display
Navigational Aids in Cars
Military Airborne Applications
Not Head Mounted
(e.g. vehicle mounted)
Primarily Outdoor
(Daylight) Environments
AR
Visual Displays

Display Technologies
 Types (Bimber/Raskar 2003)
 Head attached
•  Head mounted display/projector
 Body attached
•  Handheld display/projector
 Spatial
•  Spatially aligned projector/monitor

Head Mounted Displays (HMD)
-  Display and Optics mounted on Head
-  May or may not fully occlude real world
-  Provide full-color images
-  Considerations
•  Cumbersome to wear
•  Brightness
•  Low power consumption
•  Resolution limited
•  Cost is high?

Key Properties of HMD
  Field of View
  Human eye 95 degrees horizontal, 60/70 degrees vertical
  Resolution
  > 320x240 pixel
  Refresh Rate
  Focus
  Fixed/manual
  Power
  Size

Types of Head Mounted Displays
The image cannot be
displayed. Your
computer may not have
enough memory to open
the image, or the image
may have been
corrupted. Restart your
computer, and then open
the ﬁle again. If the red x
still appears, you may
have to delete the image
and then insert it again.
The
ima
ge
can
not
be
dis
play
ed.
You
r
co
mp
uter
Occluded
See-thru
Multiplexed

Immersive VR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
head position/orientation
to network
Display!
Driver
Non see-thru!
Image source
& optics
virtual
object
Virtual
World

See-thru AR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
to network
Display!
Driver
see-thru!
combiner
Virtual Image
superimposed!
over real world
object
real world
Image source

Optical see-through head-mounted display
Virtual images
from monitors
Real
World
Optical
Combiners

Epson Moverio BT-200
▪  Stereo see-through display ($700)
▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪  Android Powered, separate controller
▪  VGA camera, GPS, gyro, accelerometer

View Through Optical See-Through HMD

The Virtual Retinal Display
  Image scanned onto retina
  Commercialized through Microvision
  Nomad System - www.mvis.com

Strengths of optical see-through AR
  Simpler (cheaper)
  Direct view of real world
  Full resolution, no time delay (for real world)
  Safety
  Lower distortion
  No eye displacement
  but COASTAR video see-through avoids this

Video AR Architecture
Head!
Tracker
Host!
Processor
Graphics!
renderer
Digital!
Mixer
Frame!
Buffer
to network
Display!
Driver
Non see-thru!
Image source
& optics
Head-mounted
camera aligned to
display optics
Video!
Processor
Video image
of real world
Virtual image
inset into
video of real
world

Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video

Vuzix Wrap 1200DXAR
▪  Stereo video see-through display ($1500)
■ Twin 852 x 480 LCD displays, 35 deg. FOV
■ Stereo VGA cameras
■ 3 DOF head tracking

View Through a Video See-Through HMD

Strengths of Video See-Through AR
  True occlusion
  Virtual images can block view of real world
  Digitized image of real world
  Flexibility in composition
  Matchable time delays
  More registration, calibration strategies
  Wide FOV is easier to support

Optical vs. Video AR Summary
  Both have proponents
  Video is more popular today?
  Likely because lack of available optical products
  Depends on application?
  Manufacturing: optical is cheaper
  Medical: video for calibration strategies

Eye multiplexed AR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
to network
Display!
Driver
Virtual Image
inset into!
real world scene
real world
Opaque!
Image source

Virtual Image ‘inset’ into real

Vuzix M-100
▪  Monocular multiplexed display ($1000)
■ 852 x 480 LCD display, 15 deg. FOV
■ 5 MP camera, HD video
■ GPS, gyro, accelerometer

Display Types
▪  Curved Mirror
▪  off-axis projection
▪  curved mirrors in front of eye
▪  high distortion, small eye-box
▪  Waveguide
▪  use internal reflection
▪  unobstructed view of world
▪  large eye-box

See-through thin displays
▪  Waveguide techniques for thin see-through displays
▪  Wider FOV, enable AR applications
▪  Social acceptability
Opinvent Ora

Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Holographic
Hologram diffracts light
Limited FOV
Colour bleeding
Diffractive
Slanted gratings
Total internal reflection
Costly, small FOV

Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Clear-Vu Reflective
Several reflective elements
Thinner light guide
Large FOV, eye-box
Reflective
Simple reflective elements
Lower cost
Size is function of FOV

2014 COSC 426 Lecture 2: Augmented Reality Technology

More Related Content

What's hot (20)

Similar to 2014 COSC 426 Lecture 2: Augmented Reality Technology (20)

More from Mark Billinghurst (20)

Recently uploaded (20)

2014 COSC 426 Lecture 2: Augmented Reality Technology