SlideShare a Scribd company logo

COSC 426 Lect 2. - AR Technology

Mark Billinghurst, profile picture
Mark Billinghurst

This document summarizes key points from Lecture 2 on augmented reality technology. It discusses: 1. The definition of augmented reality, including its key characteristics of combining real and virtual images in real-time and having the virtual content registered in 3D space. 2. An overview of different AR display technologies, including optical see-through displays, video see-through displays, and spatial/projected AR. 3. The importance of tracking technologies for registering virtual objects in 3D space, including different tracking methods like optical, magnetic, ultrasonic, and inertial tracking. 4. How marker-based optical tracking works, including fiducial detection, rectangle fitting, and coordinate system establishment to determine an

TechnologyBusiness
Related topics:
Insights on Augmented Reality
1 of 138
Downloaded 155 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
Lecture 2: AR T h l L 2 Technology Mark Billinghurst mark.billinghurst@hitlabnz.org July 2011 COSC 426: Augmented Reality
Key Points from Lecture 1
Augmented Reality Definition Defining Characteristics [Azuma 97] Combines Real and Virtual Images g - Both can be seen at the same time Interactive in real-time - Virtual content can be interacted with Registered in 3D g - Virtual objects appear fixed in space
What is not Augmented Reality? Location-based services Barcode detection (QR-codes) B d d i (QR d ) Augmenting still images g g g Special effects in movies … … but they can be combined with AR!
Milgram’s Reality-Virtuality Continuum Mixed Reality Real Augmented Augmented Virtual Environment Reality (AR) Virtuality (AV) Environment Reality - Virtuality (RV) Continuum
Metaverse
AR History Summary 1960’s – 80’s: Early Experimentation 1980 s 90 s: 1980’s – 90’s: Basic Research Tracking, displays 1995 – 2005: Tools/Applications Interaction, usability, theory y y 2005 - : Commercial Applications Games, M di l Industry G Medical, I d
Applications Medicine Manufacturing Information overlay Architecture Museum Marketing Gaming
Interaction Design Process
Interaction Design is All About You Users should be involved throughout the Design Process Co s de all the eeds Consider a t e needs of the user
Building Compelling AR Experiences B ildi C lli E i experiences Usability applications Interaction tools Authoring components Tracking, Display
AR Technology
Building Compelling AR Experiences experiences applications tools components Display, Tracking Sony CSL © 2004
AR Technology Key Technologies Display p y Tracking Display Tracking Input Processing Processing Input
AR Displays
AR Displays AR Visual Displays Primarily Indoor Primarily Outdoor Environments (Daylight) Environments Not Head-Mounted Head-Mounted Head-Mounted Not Head Mounted Display (HMD) Display (HMD) (e.g. vehicle mounted) Virtual Images Projection CRT Display Liquid Crystal Cathode Ray Tube (CRT) Projection Display or Virtual Retinal Display (VRD) p y( ) Navigational Aids in Cars g seen off windows using beamsplitter Displays LCDs Many Military Applications Military Airborne Applications & Assistive Technologies e.g. window e.g. Reach-In e.g. Shared Space e.g. WLVA e.g. Head-Up reflections Magic Book and IVRD Display (HUD)
Head Mounted Displays
Head Mounted Displays (HMD) - Display and Optics mounted on Head - May or may not fully occlude real world - Provide full-color images - Considerations • Cumbersome to wear • Brightness • Low power consumption • Resolution limited • Cost is high? g
Types of Head Mounted Displays Occluded See-thru Multiplexed
Immersive VR Architecture Virtual World head position/orientation Head Non see-thru Tracker Image source & optics Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer virtual to network object Display Driver
See-thru AR Architecture head position/orientation Head see-thru Tracker combiner real world Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer to network Virtual Image Display superimposed Driver over real world object Image source g
Optical see-through head-mounted display Virtual images from monitors Real World Optical Combiners
Optical See-Through HMD
Optical see-through HMDs Virtual Vision VCAP Sony Glasstron
DigiLens Compact HOE C Solid state optics Switchable Bragg Grating Stacked SBG Fast switching Ultra compact www.digilens.com
The Virtual Retinal Display Image scanned onto retina age sca e o to et a Commercialized through Microvision Nomad System - www.mvis.com
Strengths of optical AR Simpler (cheaper) Direct view of real world Di i f l ld Full resolution, no time delay (for real world) Safety Lower distortion No eye displacement but COASTAR video see-through avoids this
Video AR Architecture Head-mounted camera aligned to head position/orientation display optics Video image Head Tracker of real world Video Processor Host Graphics Digital Frame Processor P renderer d Mixer Mi Buffer to network Display Driver Virtual image inset into Non see-thru video of real Image source world & optics
Video see-through HMD Video cameras Video Graphics Monitors Combiner
Video See-Through HMD
Video see-through HMD MR Laboratory’s COAS A HMD ’ COASTAR (Co-Optical Axis See-Through Augmented Reality) Parallax-free Parallax free video see through HMD see-through
TriVisio www.trivisio.com p Stereo video input PAL resolution cameras 2 x SVGA displays 30 degree FOV User adjustable convergence $6,000 USD
Vuzix Display www.vuzix.com Wrap 920 $350 USD Twin 640 x 480 LCD displays 31 degree diagonal field of view Weighs less than three ounces
Strengths of Video AR True occlusion Kiyokawa optical display that supports occlusion y p p y pp Digitized image of real world Flexibility Fl b l in composition Matchable time delays More registration, calibration strategies Wide FOV is easier to support
Optical vs. Video AR Summary Both have proponents Video is more popular today? Likely because lack of available optical products y p p Depends on application? Manufacturing: optical i cheaper M f i i l is h Medical: video for calibration strategies
Eye multiplexed AR Architecture head position/orientation Head Tracker real world Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer to network Display Virtual Image Driver inset into real world scene Opaque Image source
Virtual Image ‘inset’ into real
Virtual Vision Personal Eyewear
Virtual image inset into real world
Spatial/Projected AR
Spatial Augmented Reality Project onto irregular surfaces Geometric Registration Projector blending, High dynamic range Book: Bimber, Rasker “Spatial Augmented Reality” p g y
Projector-based AR User (possibly head-tracked) Projector Examples: Real objects Raskar, Raskar MIT Media Lab with retroreflective Inami, Tachi Lab, U. Tokyo covering
Example of projector-based AR Ramesh Raskar, UNC, MERL , ,
Example of projector-based AR Ramesh Raskar, UNC Chapel Hill
The I/O Bulb Projector + C P j Camera John Underkoffler, Hiroshi Ishii MIT Media Lab
Head Mounted Projector Head Mounted Projector J Jannick Rolland ( (UCF) ) Retro-reflective Material Potentially portable
Head Mounted Projector NVIS P 50 HMPD P-50 1280x1024/eye Stereoscopic Stere sc ic 50 degree FOV www.nvis.com i
HMD vs. HMPD Head Mounted Display Head Mounted Projected Display
Pico Projectors Microvision - www.mvis.com 3M, Samsung, Phili etc 3M S Philips, t
MIT Sixth Sense Body worn camera and projector p p y p j http://www.pranavmistry.com/projects/sixthsense/
Other AR Displays
Video Monitor AR Video Stereo cameras Monitor g glasses Video Graphics Combiner
Virtual Showcase Mirrors on a projection table Head H d tracked stereo k d Up to 4 users Merges graphic and real objects M hi d l bj Exhibit/museum applications Fraunhofer Institute (2001) Bimber, Frohlich
Augmented Paleontology Bimber et. al. IEEE Computer Sept. 2002
Alternate Displays LCD Panel Laptop PDA
Handheld Displays Mobile Phones Camera Display Input
Other Types of AR Display Audio spatial sound p ambient audio Tactile T til physical sensation Haptic virtual touch
Haptic Input AR Haptic Workbench CSIRO 2003 – Adcock et al et. al.
Phantom Sensable Technologies (www.sensable.com) 6 DOF Force Feedback Device
AR Haptic Interface Phantom, ARToolKit, Magellan g
AR Tracking and Registration
Tracking Locating the users viewpoint g p Position (x,y,z) Orientation (r p y) (r,p,y) Registration Positioning virtual object wrt real world
Tracking Requirements Head Stabilized Body Stabilized World Stabilized Augmented Reality Information Display World Stabilized Body Stabilized Increasing Tracking Requirements Head Stabilized
Tracking Technologies • Mechanical • g Electromagnetic • Optical • Acoustic • Inertial d dead I ti l and d d reckoning k i • GPS • Hybrid
AR Tracking Taxonomy AR TRACKING Indoor Outdoor Environment Environment Limited Range Extended Range Low Accuracy & High Accuracy Not Robust & Robust Low Accuracy High Accuracy Many Fiducials Not Hybridized Hybrid Tracking at 15-60 Hz & High Speed in space/time GPS or GPS and Hybrid H b id but b t Camera or C Camera and C d Tracking no GPS Compass Compass e.g. e g AR Toolkit e.g. e g IVRD e.g. e g HiBall e.g. e g WLVA e.g. e g BARS
Tracking Types Magnetic Inertial Ultrasonic Optical Mechanical Tracker T k Tracker T k Tracker T k Tracker T k Tracker Specialized Marker-Based Marker Based Markerless Tracking Tracking Tracking Edge-Based Template-Based Interest Point g Tracking g Tracking g Tracking
Tracking Systems Mechanical Tracker Magnetic Tracker Ultrasonic Tracker Inertial Tracker Vision (Optical Tracking) Specialized (Infrared, Retro-Reflective) Monocular (DVCam, Webcam) M l (DVC W b )
Mechanical Tracker Idea: Id mechanical arms with joint sensors h l h Microscribe ++: high accuracy haptic feedback accuracy, -- : cumbersome, expensive
Magnetic Tracker Idea: difference between a magnetic transmitter and a receiver Flock of Birds (Ascension) ++: 6DOF robust 6DOF, b -- : wired, sensible to metal, noisy, expensive y p
Inertial Tracker I lT k Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) IS300 (Intersense) Wii Remote ++: no transmitter, cheap small high frequency wireless transmitter cheap, small, frequency, -- : drift, hysteris only 3DOF
Ultrasonics Tracker Idea: Time of Flight or Phase Coherence Sound Waves Phase-Coherence Ultrasonic Logitech IS600 ++: Small, Cheap -- : 3DOF, Line of Sight, Low resolution, Affected Environment Conditon (pressure, temperature)
Global Positioning System (GPS) Created by US in 1978 Currently 29 satellites Satellites send position + time GPS Receiver positioning p g 4 satellites need to be visible Differential time of arrival Triangulation Accuracy y 5-30m+, blocked by weather, buildings etc
Problems with GPS Takes time to get satellite fix Satellites moving around Earths atmosphere affects signal p g Assumes consistent speed (the speed of light). Delay depends where you are on Earth Weather effects Signal reflection Multi-path reflection off buildings Signal blocking Trees, buildings, mountains T b ildi t i Satellites send out bad data Misreport their own position
Accurate to < 5cm close to base station (22m/100 km) Expensive - $20-40,000 USD
Optical Tracking
Optical Tracker Idea: Image Processing and Computer Vision g g p Specialized Infrared, Retro-Reflective, Stereoscopic f f S ART Hi-Ball Monocular Based Vision Tracking g
Outside-In vs. Inside-Out Tracking
Optical Tracking Technologies Scalable active trackers InterSense IS-900, 3rd Tech HiBall 3rd Tech, I 3 d T h Inc. Passive optical computer vision Line of sight, may require landmarks sight Can be brittle. Computer vision i computationally-intensive C i i is i ll i i
HiBall Tracking System (3rd Tech) Inside-Out Tracker O $50K USD Scalable over large area Fast d t (2000H ) F t update (2000Hz) Latency Less than 1 ms. Accurate Position 0.4mm RMS 0 4mm Orientation 0.02° RMS
COSC 426 Lect 2. - AR Technology
Starting simple: Marker tracking Has been done for more than 10 years Several open source solutions exist S l l i i Fairly simple to implement y p p Standard computer vision methods A rectangular marker provides 4 corner points l k id i Enough for pose estimation!
Marker Based Tracking: ARToolKit http://artoolkit.sourceforge.net/
Coordinate Systems C d S
Marker T k M k Tracking – O Overview
Marker Tracking – Fiducial Detection Threshold the whole image to black and white Search scanline by scanline for edges (white to black) Follow edge until either Back to starting pixel Image border Check for size Reject fiducials early that are too small (or too large)
Marker Tracking – Rectangle Fitting Start with an arbitrary point “x” on the contour S ih bi i “ ” h The point with maximum distance must be a corner c0 Create a diagonal through the center C d l h h h Find points c1 & c2 with maximum distance left and right of diag. New diagonal from c1 to c2 Find point c3 right of diagonal with maximum distance
Marker Tracking – Pattern checking Calculate homography using the 4 corner points “Direct Linear Transform” algorithm Maps normalized coordinates to marker coordinates p (simple perspective projection, no camera model) Extract pattern by sampling Check pattern Id (implicit encoding) Template ( T l (normalized cross correlation) li d l i )
Marker Tracking – Corner refinement Refine corner coordinates Critical for high quality tracking Remember: 4 points is the bare minimum! So these 4 points should better be accurate… Detect sub-pixel coordinates E.g. Harris corner detector g - Specialized methods can be faster and more accurate Strongly reduces jitter! gy j Undistort corner coordinates Remove radial distortion from lens R di l di t ti f l
Marker M k tracking – P k Pose estimation Calculates marker position and rotation C relative to the camera Initial estimation directly from homography Very fast, but coarse fast Jitters a lot… Refinement via G R fi i Gauss-Newton iteration N i i 6 parameters (3 for position, 3 for rotation) to refine At each iteration we optimize on the reprojection error
Coordinates for Marker Tracking
Coordinates for Marker Tracking •Camera Observed Screen Marker Ideal Screen Screen •Ideal ScreenCamera •Marker Observed •Perspective model (barrel shape) •Goal •Correspondence of •Nonlinear function 4 vertices •Obtained fromTranslation •Rotation & Camera Calibration •Obtained fromT processing R t ti •Real time image l ti Camera Calibration
From Marker To Camera F M k T C Rotation & Translation TCM : 4x4 transformation matrix from marker coord. to camera coord.
Tracking challenges in ARToolKit Occlusion Unfocused camera, Dark/unevenly lit Jittering (image by M. Fiala) motion blur scene, vignetting (Photoshop illustration) Image noise False positives and inter-marker confusion (e.g. poor lens, block coding / compression, neon tube) (image by M. Fiala)
Tracking, Tracking, Tracking
Other Marker Tracking Libraries arTag T http://www.artag.net/ ARToolKitPlus [Discontinued] http://studierstube.icg.tu- graz.ac.at/handheld_ar/artoolkitplus.php graz ac at/handheld ar/artoolkitplus php stbTracker http://studierstube.icg.tu- htt // t di t b i t graz.ac.at/handheld_ar/stbtracker.php MXRToolKit http://sourceforge.net/projects/mxrtoolkit/
Markerless Tracking
COSC 426 Lect 2. - AR Technology
Markerless Tracking No more Markers! Markerless Tracking Magnetic T k M i Tracker Inertial I i l Ultrasonic Ul i Optical O i l Tracker Tracker Tracker Specialized Marker-Based Markerless Tracking Tracking Tracking Edge-Based Template-Based Interest Point Tracking Tracking Tracking
Natural feature tracking Tracking from features of the surrounding environment Corners, edges, blobs, ... Generally more diffi l than marker tracking G ll difficult h k ki Markers are designed for their purpose g p p The natural environment is not… Less well established methods well-established Usually much slower than marker tracking
Natural Feature Tracking Features Points Use Natural Cues of Real Elements Contours Edges Surface Texture Interest Points Model or Model-Free ++: no visual pollution Surfaces
Texture Tracking
Tracking by d T k b detection This is what most trackers do do… Camera Image g Targets are detected every frame Keypoint detection yp Popular because tracking and detection Descriptor creation are solved simultaneously and matching d t hi Outlier Removal Pose estimation and refinement Pose
Natural feature tracking – What is a keypoint? It depends on the detector you use! For high performance use the FAST corner detector Apply A l FAST t all pixels of your i to ll i l f image Obtain a set of keypoints for your image - R d Reduce the amount of corners using non-maximum suppression h f Describe the keypoints E. Rosten and T. Drummond (May 2006). "Machine learning for high‐speed corner detection". 
Corner keypoint
Natural feature tracking – Descriptors Again depends on your choice of a descriptor! Can use SIFT Estimate the d i E i h dominant keypoint k i orientation using gradients Compensate for C m ensate f r detected orientation Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,  Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 
NFT – D b Database creation Offline step p Searching for corners in a static image For robustness look at corners on multiple scales Some corners are more descriptive at larger or smaller scales We d ’t k W don’t know how far users will be from our image h f ill b f i Build a database file with all descriptors and their position on the original i ii h i i l image
NFT – R l Real-time tracking k Search for keypoints Camera Image in the video image Create the d C t th descriptorsi t Keypoint detection Match the descriptors from the Descriptor creation p live video against those and matching in the database Outlier Removal O tli R l Brute force is not an option Need the speed-up of special Pose estimation data structures and refinement - E.g., we use multiple spill trees Pose
NFT – O l removal Outlier l Cascade of removal techniques Start with cheapest, finish with most expensive… First simple geometric tests - E.g., line tests • Select 2 points to form a line • Check all other points being on correct side of line Then, homography-based tests
NFT – P Pose refinement f Pose from homography makes good starting point Based on Gauss-Newton iteration Try to minimize the re-projection error of the keypoints Part of tracking pipeline that mostly benefits from floating point usage Can still be implemented effectively in fixed point Typically 2-4 iterations are enough…
NFT – R l Real-time tracking k Search for keypoints Camera Image in the video image p Create the descriptors Keypoint detection Match the descriptors from the live video against those Descriptor creation and matching in the database Remove the keypoints that Outlier Removal are outliers Pose estimation Use h U the remaining k i i keypoints i and refinement to calculate the pose Pose of the camera f h
NFT – R l Results Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,  Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 
Edge Based Tracking RAPiD [Drummond et al. 02] Initialization, Control Points, Pose Prediction (Global Method)
Line Based Tracking Visual Servoing [Comport et al. 2004]
Model Based Tracking OpenTL - www.opentl.org General purpose library for model based visual tracking
OpenTL Features
COSC 426 Lect 2. - AR Technology
Visual Modalities Used For Tracking
The Tracking Pipeline
Marker vs. natural feature tracking Marker tracking Usually requires no database to be stored Markers can be an eye-catcher Tracking is less demanding g g The environment must be instrumented with markers Markers usually work only when fully in view y y y Natural feature tracking A database of keypoints must be stored/downloaded Natural feature targets might catch the attention less Natural f t N t l feature targets are potentially everywhere t t t ti ll h Natural feature targets work also if partially in view
Hybrid Tracking
Example: Outdoor Hybrid Tracking Combines computer vision - natural feature tracking inertial gyroscope sensors Both correct for each other Inertial gyro - provides frame to frame prediction of camera orientation Computer vision - correct for gyro drift
Outdoor AR Tracking System You, Neumann, Azuma outdoor AR system (1999)
Robust Outdoor Tracking Hybrid T ki H b id Tracking Computer Vision, GPS, inertial Going Out Reitmayer & Drummond (Univ. Cambridge)
Handheld Display
Registration
The Registration Problem Virtual and Real must stay properly aligned If not: t Breaks the illusion that the two coexist Prevents acceptance of many serious applications
Sources of registration errors Static errors S Optical distortions Mechanical misalignments Tracker errors Incorrect viewing parameters Dynamic errors System delays (largest source of error) - 1 ms d l = 1/3 mm registration error delay i t ti
Reducing static errors Distortion compensation Manual adjustments View-based or direct measurements [Azuma94] [Caudell92] [Janin93] etc. Camera calibration (video) [ARGOS94] [Bajura93] [Tuceryan95] etc.
View Based Calibration (Azuma 94)
Dynamic errors Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint Scene Display Simulation 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms Total Delay = 50 + 2 + 33 + 17 = 102 ms 1 ms delay = 1/3 mm = 33mm error
Reducing dynamic errors (1) Reduce system lag [Olano95] [Wloka95a] [Regan SIGGRAPH99] Reduce apparent lag Image deflection [Burbidge89] [Regan94] [So92] [Kijima [Kiji ISMR 2001] Image warping [Mark 3DI 97]
Reducing System Lag Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint Scene Display Simulation Faster Tracker Faster CPU Faster GPU Faster Display
Reducing Apparent Lag Virtual Display Virtual Display x,y,z y Physical Ph i l r,p,y Physical Display Display (640x480) (640x480) Tracking 1280 x 960 Update 1280 x 960 Last known position Latest position Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint p Scene Display p y Simulation
Reducing dynamic errors (2) Match input streams (video) Delay video of real world to match system lag Predictive Tracking [Azuma94] [Emura94] Inertial sensors helpful Azuma / Bishop 1994 u a s op 99
Predictive Tracking Position Now Past Future Time Can predict up to 80 ms in future (Holloway)
Predictive Tracking (Azuma 94)
Wrap-up Tracking and Registration are key problems Registration error Measures against static error Measures against dynamic error M i d i AR typically requires multiple tracking technologies yp y q p g g Research Areas: Hybrid Markerless Techniques, Deformable Surface, Mobile, Outdoors Surface Mobile
More Information • M k Billi h t Mark Billinghurst – mark.billinghurst@hitlabnz.org • Websites – www.hitlabnz.org hi l b

More Related Content

PDF
NIAR_VRC_2010
fftoledo
 
PDF
Computer Assisted Manufacturing
vrt-medialab
 
PDF
COSC 426 Lect. 8: AR Research Directions
Mark Billinghurst
 
PDF
Probabilistic generative models for machine vision
zukun
 
PDF
Scd 2020 r
sanzen enterprises
 
PDF
Scd 2020.2040 e(revised3.31)
sanzen enterprises
 
PDF
ARM Techcon Keynote 2012: Sensor Integration and Improved User Experiences at...
HSA Foundation
 
ODP
Alternativa3D
Eugenia Startseva
 
NIAR_VRC_2010
fftoledo
 
Computer Assisted Manufacturing
vrt-medialab
 
COSC 426 Lect. 8: AR Research Directions
Mark Billinghurst
 
Probabilistic generative models for machine vision
zukun
 
Scd 2020 r
sanzen enterprises
 
Scd 2020.2040 e(revised3.31)
sanzen enterprises
 
ARM Techcon Keynote 2012: Sensor Integration and Improved User Experiences at...
HSA Foundation
 
Alternativa3D
Eugenia Startseva
 

What's hot (15)

PPT
Raskar Graphics Interface May05 Web
Camera Culture Group, MIT Media Lab
 
PDF
Scd 2080 r
sanzen enterprises
 
PDF
Camera-based sensory substitution and augmented reality for the blind (ACIVS ...
Peter Meijer
 
ODP
Alternativa3D
Eugenia Startseva
 
PPT
Sccg Many Projects Layout03
Camera Culture Group, MIT Media Lab
 
ODP
Alternativa3D_en
Eugenia Startseva
 
PDF
Agile2011-Agile Education by Object Game
Tsuyoshi Ushio
 
PDF
Augmented Reality: Beyond the Hype
Paul Coulton
 
PPTX
セミナー資料(20OCT2011)Power Point
Tsuyoshi Horigome
 
PPT
Object modeling in robotic perception
MoniqueO Opris
 
PDF
Providing new customer_experience(final)
Tsukasa Makino
 
PDF
Good code-isnt-enough
Skills Matter
 
PDF
Future of Mobile Augmented Reality (Zenitum's View Point)
DoubleMe, Inc.
 
PPTX
Wired Ovita E Tech
WIRED staff
 
PDF
Computer generated holography as a generic display technology
Pritam Bhansali
 
Raskar Graphics Interface May05 Web
Camera Culture Group, MIT Media Lab
 
Scd 2080 r
sanzen enterprises
 
Camera-based sensory substitution and augmented reality for the blind (ACIVS ...
Peter Meijer
 
Alternativa3D
Eugenia Startseva
 
Sccg Many Projects Layout03
Camera Culture Group, MIT Media Lab
 
Alternativa3D_en
Eugenia Startseva
 
Agile2011-Agile Education by Object Game
Tsuyoshi Ushio
 
Augmented Reality: Beyond the Hype
Paul Coulton
 
セミナー資料(20OCT2011)Power Point
Tsuyoshi Horigome
 
Object modeling in robotic perception
MoniqueO Opris
 
Providing new customer_experience(final)
Tsukasa Makino
 
Good code-isnt-enough
Skills Matter
 
Future of Mobile Augmented Reality (Zenitum's View Point)
DoubleMe, Inc.
 
Wired Ovita E Tech
WIRED staff
 
Computer generated holography as a generic display technology
Pritam Bhansali
 
Ad

Similar to COSC 426 Lect 2. - AR Technology (20)

PDF
426 lecture2: AR Technology
Mark Billinghurst
 
PDF
2013 426 Lecture 2: Augmented Reality Technology
Mark Billinghurst
 
PPT
Augmented reality
Brijesh Kothari
 
PPT
Augmented Reality
Kaviyaraj R
 
PDF
2014 COSC 426 Lecture 2: Augmented Reality Technology
Mark Billinghurst
 
PDF
Augmented Reality
Connected-Blog
 
PDF
COMP 4010 - Lecture 8 AR Technology
Mark Billinghurst
 
PDF
AR(mobas2012)s
Woontack Woo
 
PPT
Augmented reality
kaviyaraj
 
PPTX
Simulating Presence: BIM to Virtual Reality
Jason Halaby
 
PDF
426 lecture6a osgART Development
Mark Billinghurst
 
PDF
SVR2011 Keynote
Mark Billinghurst
 
PDF
Augmented Reality
Michael Bartl
 
PPTX
Augmented Reality
Ajay Sankar
 
PDF
Droidcon2013 miracast final2
Droidcon Berlin
 
PDF
Tomer kimhi mobmodcon-nov2015-integrating new camera hardware
Ron Munitz
 
PDF
Aurea Vision & Aurea Consulting: technology proposal
Anna Doroshenko
 
PDF
COSC 426 Lect. 5 - Mobile AR
Mark Billinghurst
 
PPTX
Augmented Reality for Architects
Gaurav Kudalkar
 
PDF
Nizar Romdhane, ARM
White Nights Conference
 
426 lecture2: AR Technology
Mark Billinghurst
 
2013 426 Lecture 2: Augmented Reality Technology
Mark Billinghurst
 
Augmented reality
Brijesh Kothari
 
Augmented Reality
Kaviyaraj R
 
2014 COSC 426 Lecture 2: Augmented Reality Technology
Mark Billinghurst
 
Augmented Reality
Connected-Blog
 
COMP 4010 - Lecture 8 AR Technology
Mark Billinghurst
 
AR(mobas2012)s
Woontack Woo
 
Augmented reality
kaviyaraj
 
Simulating Presence: BIM to Virtual Reality
Jason Halaby
 
426 lecture6a osgART Development
Mark Billinghurst
 
SVR2011 Keynote
Mark Billinghurst
 
Augmented Reality
Michael Bartl
 
Augmented Reality
Ajay Sankar
 
Droidcon2013 miracast final2
Droidcon Berlin
 
Tomer kimhi mobmodcon-nov2015-integrating new camera hardware
Ron Munitz
 
Aurea Vision & Aurea Consulting: technology proposal
Anna Doroshenko
 
COSC 426 Lect. 5 - Mobile AR
Mark Billinghurst
 
Augmented Reality for Architects
Gaurav Kudalkar
 
Nizar Romdhane, ARM
White Nights Conference
 
Ad

More from Mark Billinghurst (20)

PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 6 - AI for Prototyping and Research Directi...
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 4 - High Level Prototyping.
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 3 - Video and Paper Prototyping
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.
Mark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 1 Introduction to Prototyping
Mark Billinghurst
 
PDF
Research Directions in Heads-Up Computing
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction ...
Mark Billinghurst
 
PDF
IVE 2024 Short Course Lecture15 - Measuring Cybersickness
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture14 - Evaluation
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture12 - OpenVibe Tutorial
Mark Billinghurst
 
PDF
IVE 2024 Short Course Lecture10 - Multimodal Emotion Recognition in Conversat...
Mark Billinghurst
 
PDF
IVE 2024 Short Course Lecture 9 - Empathic Computing in VR
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture 8 - Electroencephalography (EEG) Basics
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture16- Cognixion Axon-R
Mark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture 2 - Fundamentals of Perception
Mark Billinghurst
 
PDF
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 6 - AI for Prototyping and Research Directi...
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 4 - High Level Prototyping.
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 3 - Video and Paper Prototyping
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.
Mark Billinghurst
 
Rapid Prototyping for XR: Lecture 1 Introduction to Prototyping
Mark Billinghurst
 
Research Directions in Heads-Up Computing
Mark Billinghurst
 
IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
Mark Billinghurst
 
IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction ...
Mark Billinghurst
 
IVE 2024 Short Course Lecture15 - Measuring Cybersickness
Mark Billinghurst
 
IVE 2024 Short Course - Lecture14 - Evaluation
Mark Billinghurst
 
IVE 2024 Short Course - Lecture12 - OpenVibe Tutorial
Mark Billinghurst
 
IVE 2024 Short Course Lecture10 - Multimodal Emotion Recognition in Conversat...
Mark Billinghurst
 
IVE 2024 Short Course Lecture 9 - Empathic Computing in VR
Mark Billinghurst
 
IVE 2024 Short Course - Lecture 8 - Electroencephalography (EEG) Basics
Mark Billinghurst
 
IVE 2024 Short Course - Lecture16- Cognixion Axon-R
Mark Billinghurst
 
IVE 2024 Short Course - Lecture 2 - Fundamentals of Perception
Mark Billinghurst
 
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 

Recently uploaded (20)

PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
PDF
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
Chapter 5: Probability Theory and Statistics
RandyFin
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
A Presentation on Touch Screen Technology
memembruh336
 
PDF
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
PPTX
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
August Patch Tuesday
Ivanti
 
Approach and Philosophy of On baking technology
30anthuong17
 
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Chapter 5: Probability Theory and Statistics
RandyFin
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
A Presentation on Touch Screen Technology
memembruh336
 
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
August Patch Tuesday
Ivanti
 

COSC 426 Lect 2. - AR Technology

  • 1. Lecture 2: AR T h l L 2 Technology Mark Billinghurst mark.billinghurst@hitlabnz.org July 2011 COSC 426: Augmented Reality
  • 2. Key Points from Lecture 1
  • 3. Augmented Reality Definition Defining Characteristics [Azuma 97] Combines Real and Virtual Images g - Both can be seen at the same time Interactive in real-time - Virtual content can be interacted with Registered in 3D g - Virtual objects appear fixed in space
  • 4. What is not Augmented Reality? Location-based services Barcode detection (QR-codes) B d d i (QR d ) Augmenting still images g g g Special effects in movies … … but they can be combined with AR!
  • 5. Milgram’s Reality-Virtuality Continuum Mixed Reality Real Augmented Augmented Virtual Environment Reality (AR) Virtuality (AV) Environment Reality - Virtuality (RV) Continuum
  • 7. AR History Summary 1960’s – 80’s: Early Experimentation 1980 s 90 s: 1980’s – 90’s: Basic Research Tracking, displays 1995 – 2005: Tools/Applications Interaction, usability, theory y y 2005 - : Commercial Applications Games, M di l Industry G Medical, I d
  • 10. Interaction Design is All About You Users should be involved throughout the Design Process Co s de all the eeds Consider a t e needs of the user
  • 11. Building Compelling AR Experiences B ildi C lli E i experiences Usability applications Interaction tools Authoring components Tracking, Display
  • 13. Building Compelling AR Experiences experiences applications tools components Display, Tracking Sony CSL © 2004
  • 14. AR Technology Key Technologies Display p y Tracking Display Tracking Input Processing Processing Input
  • 16. AR Displays AR Visual Displays Primarily Indoor Primarily Outdoor Environments (Daylight) Environments Not Head-Mounted Head-Mounted Head-Mounted Not Head Mounted Display (HMD) Display (HMD) (e.g. vehicle mounted) Virtual Images Projection CRT Display Liquid Crystal Cathode Ray Tube (CRT) Projection Display or Virtual Retinal Display (VRD) p y( ) Navigational Aids in Cars g seen off windows using beamsplitter Displays LCDs Many Military Applications Military Airborne Applications & Assistive Technologies e.g. window e.g. Reach-In e.g. Shared Space e.g. WLVA e.g. Head-Up reflections Magic Book and IVRD Display (HUD)
  • 18. Head Mounted Displays (HMD) - Display and Optics mounted on Head - May or may not fully occlude real world - Provide full-color images - Considerations • Cumbersome to wear • Brightness • Low power consumption • Resolution limited • Cost is high? g
  • 19. Types of Head Mounted Displays Occluded See-thru Multiplexed
  • 20. Immersive VR Architecture Virtual World head position/orientation Head Non see-thru Tracker Image source & optics Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer virtual to network object Display Driver
  • 21. See-thru AR Architecture head position/orientation Head see-thru Tracker combiner real world Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer to network Virtual Image Display superimposed Driver over real world object Image source g
  • 22. Optical see-through head-mounted display Virtual images from monitors Real World Optical Combiners
  • 24. Optical see-through HMDs Virtual Vision VCAP Sony Glasstron
  • 25. DigiLens Compact HOE C Solid state optics Switchable Bragg Grating Stacked SBG Fast switching Ultra compact www.digilens.com
  • 26. The Virtual Retinal Display Image scanned onto retina age sca e o to et a Commercialized through Microvision Nomad System - www.mvis.com
  • 27. Strengths of optical AR Simpler (cheaper) Direct view of real world Di i f l ld Full resolution, no time delay (for real world) Safety Lower distortion No eye displacement but COASTAR video see-through avoids this
  • 28. Video AR Architecture Head-mounted camera aligned to head position/orientation display optics Video image Head Tracker of real world Video Processor Host Graphics Digital Frame Processor P renderer d Mixer Mi Buffer to network Display Driver Virtual image inset into Non see-thru video of real Image source world & optics
  • 29. Video see-through HMD Video cameras Video Graphics Monitors Combiner
  • 31. Video see-through HMD MR Laboratory’s COAS A HMD ’ COASTAR (Co-Optical Axis See-Through Augmented Reality) Parallax-free Parallax free video see through HMD see-through
  • 32. TriVisio www.trivisio.com p Stereo video input PAL resolution cameras 2 x SVGA displays 30 degree FOV User adjustable convergence $6,000 USD
  • 33. Vuzix Display www.vuzix.com Wrap 920 $350 USD Twin 640 x 480 LCD displays 31 degree diagonal field of view Weighs less than three ounces
  • 34. Strengths of Video AR True occlusion Kiyokawa optical display that supports occlusion y p p y pp Digitized image of real world Flexibility Fl b l in composition Matchable time delays More registration, calibration strategies Wide FOV is easier to support
  • 35. Optical vs. Video AR Summary Both have proponents Video is more popular today? Likely because lack of available optical products y p p Depends on application? Manufacturing: optical i cheaper M f i i l is h Medical: video for calibration strategies
  • 36. Eye multiplexed AR Architecture head position/orientation Head Tracker real world Host Data Base Rendering Frame Processor P Model M d l Engine E i Buffer to network Display Virtual Image Driver inset into real world scene Opaque Image source
  • 39. Virtual image inset into real world
  • 41. Spatial Augmented Reality Project onto irregular surfaces Geometric Registration Projector blending, High dynamic range Book: Bimber, Rasker “Spatial Augmented Reality” p g y
  • 42. Projector-based AR User (possibly head-tracked) Projector Examples: Real objects Raskar, Raskar MIT Media Lab with retroreflective Inami, Tachi Lab, U. Tokyo covering
  • 43. Example of projector-based AR Ramesh Raskar, UNC, MERL , ,
  • 44. Example of projector-based AR Ramesh Raskar, UNC Chapel Hill
  • 45. The I/O Bulb Projector + C P j Camera John Underkoffler, Hiroshi Ishii MIT Media Lab
  • 46. Head Mounted Projector Head Mounted Projector J Jannick Rolland ( (UCF) ) Retro-reflective Material Potentially portable
  • 47. Head Mounted Projector NVIS P 50 HMPD P-50 1280x1024/eye Stereoscopic Stere sc ic 50 degree FOV www.nvis.com i
  • 48. HMD vs. HMPD Head Mounted Display Head Mounted Projected Display
  • 49. Pico Projectors Microvision - www.mvis.com 3M, Samsung, Phili etc 3M S Philips, t
  • 50. MIT Sixth Sense Body worn camera and projector p p y p j http://www.pranavmistry.com/projects/sixthsense/
  • 52. Video Monitor AR Video Stereo cameras Monitor g glasses Video Graphics Combiner
  • 53. Virtual Showcase Mirrors on a projection table Head H d tracked stereo k d Up to 4 users Merges graphic and real objects M hi d l bj Exhibit/museum applications Fraunhofer Institute (2001) Bimber, Frohlich
  • 54. Augmented Paleontology Bimber et. al. IEEE Computer Sept. 2002
  • 56. Handheld Displays Mobile Phones Camera Display Input
  • 57. Other Types of AR Display Audio spatial sound p ambient audio Tactile T til physical sensation Haptic virtual touch
  • 58. Haptic Input AR Haptic Workbench CSIRO 2003 – Adcock et al et. al.
  • 60. AR Haptic Interface Phantom, ARToolKit, Magellan g
  • 61. AR Tracking and Registration
  • 62. Tracking Locating the users viewpoint g p Position (x,y,z) Orientation (r p y) (r,p,y) Registration Positioning virtual object wrt real world
  • 63. Tracking Requirements Head Stabilized Body Stabilized World Stabilized Augmented Reality Information Display World Stabilized Body Stabilized Increasing Tracking Requirements Head Stabilized
  • 64. Tracking Technologies • Mechanical • g Electromagnetic • Optical • Acoustic • Inertial d dead I ti l and d d reckoning k i • GPS • Hybrid
  • 65. AR Tracking Taxonomy AR TRACKING Indoor Outdoor Environment Environment Limited Range Extended Range Low Accuracy & High Accuracy Not Robust & Robust Low Accuracy High Accuracy Many Fiducials Not Hybridized Hybrid Tracking at 15-60 Hz & High Speed in space/time GPS or GPS and Hybrid H b id but b t Camera or C Camera and C d Tracking no GPS Compass Compass e.g. e g AR Toolkit e.g. e g IVRD e.g. e g HiBall e.g. e g WLVA e.g. e g BARS
  • 66. Tracking Types Magnetic Inertial Ultrasonic Optical Mechanical Tracker T k Tracker T k Tracker T k Tracker T k Tracker Specialized Marker-Based Marker Based Markerless Tracking Tracking Tracking Edge-Based Template-Based Interest Point g Tracking g Tracking g Tracking
  • 67. Tracking Systems Mechanical Tracker Magnetic Tracker Ultrasonic Tracker Inertial Tracker Vision (Optical Tracking) Specialized (Infrared, Retro-Reflective) Monocular (DVCam, Webcam) M l (DVC W b )
  • 68. Mechanical Tracker Idea: Id mechanical arms with joint sensors h l h Microscribe ++: high accuracy haptic feedback accuracy, -- : cumbersome, expensive
  • 69. Magnetic Tracker Idea: difference between a magnetic transmitter and a receiver Flock of Birds (Ascension) ++: 6DOF robust 6DOF, b -- : wired, sensible to metal, noisy, expensive y p
  • 70. Inertial Tracker I lT k Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) IS300 (Intersense) Wii Remote ++: no transmitter, cheap small high frequency wireless transmitter cheap, small, frequency, -- : drift, hysteris only 3DOF
  • 71. Ultrasonics Tracker Idea: Time of Flight or Phase Coherence Sound Waves Phase-Coherence Ultrasonic Logitech IS600 ++: Small, Cheap -- : 3DOF, Line of Sight, Low resolution, Affected Environment Conditon (pressure, temperature)
  • 72. Global Positioning System (GPS) Created by US in 1978 Currently 29 satellites Satellites send position + time GPS Receiver positioning p g 4 satellites need to be visible Differential time of arrival Triangulation Accuracy y 5-30m+, blocked by weather, buildings etc
  • 73. Problems with GPS Takes time to get satellite fix Satellites moving around Earths atmosphere affects signal p g Assumes consistent speed (the speed of light). Delay depends where you are on Earth Weather effects Signal reflection Multi-path reflection off buildings Signal blocking Trees, buildings, mountains T b ildi t i Satellites send out bad data Misreport their own position
  • 74. Accurate to < 5cm close to base station (22m/100 km) Expensive - $20-40,000 USD
  • 76. Optical Tracker Idea: Image Processing and Computer Vision g g p Specialized Infrared, Retro-Reflective, Stereoscopic f f S ART Hi-Ball Monocular Based Vision Tracking g
  • 78. Optical Tracking Technologies Scalable active trackers InterSense IS-900, 3rd Tech HiBall 3rd Tech, I 3 d T h Inc. Passive optical computer vision Line of sight, may require landmarks sight Can be brittle. Computer vision i computationally-intensive C i i is i ll i i
  • 79. HiBall Tracking System (3rd Tech) Inside-Out Tracker O $50K USD Scalable over large area Fast d t (2000H ) F t update (2000Hz) Latency Less than 1 ms. Accurate Position 0.4mm RMS 0 4mm Orientation 0.02° RMS
  • 81. Starting simple: Marker tracking Has been done for more than 10 years Several open source solutions exist S l l i i Fairly simple to implement y p p Standard computer vision methods A rectangular marker provides 4 corner points l k id i Enough for pose estimation!
  • 82. Marker Based Tracking: ARToolKit http://artoolkit.sourceforge.net/
  • 84. Marker T k M k Tracking – O Overview
  • 85. Marker Tracking – Fiducial Detection Threshold the whole image to black and white Search scanline by scanline for edges (white to black) Follow edge until either Back to starting pixel Image border Check for size Reject fiducials early that are too small (or too large)
  • 86. Marker Tracking – Rectangle Fitting Start with an arbitrary point “x” on the contour S ih bi i “ ” h The point with maximum distance must be a corner c0 Create a diagonal through the center C d l h h h Find points c1 & c2 with maximum distance left and right of diag. New diagonal from c1 to c2 Find point c3 right of diagonal with maximum distance
  • 87. Marker Tracking – Pattern checking Calculate homography using the 4 corner points “Direct Linear Transform” algorithm Maps normalized coordinates to marker coordinates p (simple perspective projection, no camera model) Extract pattern by sampling Check pattern Id (implicit encoding) Template ( T l (normalized cross correlation) li d l i )
  • 88. Marker Tracking – Corner refinement Refine corner coordinates Critical for high quality tracking Remember: 4 points is the bare minimum! So these 4 points should better be accurate… Detect sub-pixel coordinates E.g. Harris corner detector g - Specialized methods can be faster and more accurate Strongly reduces jitter! gy j Undistort corner coordinates Remove radial distortion from lens R di l di t ti f l
  • 89. Marker M k tracking – P k Pose estimation Calculates marker position and rotation C relative to the camera Initial estimation directly from homography Very fast, but coarse fast Jitters a lot… Refinement via G R fi i Gauss-Newton iteration N i i 6 parameters (3 for position, 3 for rotation) to refine At each iteration we optimize on the reprojection error
  • 91. Coordinates for Marker Tracking •Camera Observed Screen Marker Ideal Screen Screen •Ideal ScreenCamera •Marker Observed •Perspective model (barrel shape) •Goal •Correspondence of •Nonlinear function 4 vertices •Obtained fromTranslation •Rotation & Camera Calibration •Obtained fromT processing R t ti •Real time image l ti Camera Calibration
  • 92. From Marker To Camera F M k T C Rotation & Translation TCM : 4x4 transformation matrix from marker coord. to camera coord.
  • 93. Tracking challenges in ARToolKit Occlusion Unfocused camera, Dark/unevenly lit Jittering (image by M. Fiala) motion blur scene, vignetting (Photoshop illustration) Image noise False positives and inter-marker confusion (e.g. poor lens, block coding / compression, neon tube) (image by M. Fiala)
  • 95. Other Marker Tracking Libraries arTag T http://www.artag.net/ ARToolKitPlus [Discontinued] http://studierstube.icg.tu- graz.ac.at/handheld_ar/artoolkitplus.php graz ac at/handheld ar/artoolkitplus php stbTracker http://studierstube.icg.tu- htt // t di t b i t graz.ac.at/handheld_ar/stbtracker.php MXRToolKit http://sourceforge.net/projects/mxrtoolkit/
  • 98. Markerless Tracking No more Markers! Markerless Tracking Magnetic T k M i Tracker Inertial I i l Ultrasonic Ul i Optical O i l Tracker Tracker Tracker Specialized Marker-Based Markerless Tracking Tracking Tracking Edge-Based Template-Based Interest Point Tracking Tracking Tracking
  • 99. Natural feature tracking Tracking from features of the surrounding environment Corners, edges, blobs, ... Generally more diffi l than marker tracking G ll difficult h k ki Markers are designed for their purpose g p p The natural environment is not… Less well established methods well-established Usually much slower than marker tracking
  • 100. Natural Feature Tracking Features Points Use Natural Cues of Real Elements Contours Edges Surface Texture Interest Points Model or Model-Free ++: no visual pollution Surfaces
  • 102. Tracking by d T k b detection This is what most trackers do do… Camera Image g Targets are detected every frame Keypoint detection yp Popular because tracking and detection Descriptor creation are solved simultaneously and matching d t hi Outlier Removal Pose estimation and refinement Pose
  • 103. Natural feature tracking – What is a keypoint? It depends on the detector you use! For high performance use the FAST corner detector Apply A l FAST t all pixels of your i to ll i l f image Obtain a set of keypoints for your image - R d Reduce the amount of corners using non-maximum suppression h f Describe the keypoints E. Rosten and T. Drummond (May 2006). "Machine learning for high‐speed corner detection". 
  • 105. Natural feature tracking – Descriptors Again depends on your choice of a descriptor! Can use SIFT Estimate the d i E i h dominant keypoint k i orientation using gradients Compensate for C m ensate f r detected orientation Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,  Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 
  • 106. NFT – D b Database creation Offline step p Searching for corners in a static image For robustness look at corners on multiple scales Some corners are more descriptive at larger or smaller scales We d ’t k W don’t know how far users will be from our image h f ill b f i Build a database file with all descriptors and their position on the original i ii h i i l image
  • 107. NFT – R l Real-time tracking k Search for keypoints Camera Image in the video image Create the d C t th descriptorsi t Keypoint detection Match the descriptors from the Descriptor creation p live video against those and matching in the database Outlier Removal O tli R l Brute force is not an option Need the speed-up of special Pose estimation data structures and refinement - E.g., we use multiple spill trees Pose
  • 108. NFT – O l removal Outlier l Cascade of removal techniques Start with cheapest, finish with most expensive… First simple geometric tests - E.g., line tests • Select 2 points to form a line • Check all other points being on correct side of line Then, homography-based tests
  • 109. NFT – P Pose refinement f Pose from homography makes good starting point Based on Gauss-Newton iteration Try to minimize the re-projection error of the keypoints Part of tracking pipeline that mostly benefits from floating point usage Can still be implemented effectively in fixed point Typically 2-4 iterations are enough…
  • 110. NFT – R l Real-time tracking k Search for keypoints Camera Image in the video image p Create the descriptors Keypoint detection Match the descriptors from the live video against those Descriptor creation and matching in the database Remove the keypoints that Outlier Removal are outliers Pose estimation Use h U the remaining k i i keypoints i and refinement to calculate the pose Pose of the camera f h
  • 111. NFT – R l Results Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,  Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 
  • 112. Edge Based Tracking RAPiD [Drummond et al. 02] Initialization, Control Points, Pose Prediction (Global Method)
  • 113. Line Based Tracking Visual Servoing [Comport et al. 2004]
  • 114. Model Based Tracking OpenTL - www.opentl.org General purpose library for model based visual tracking
  • 117. Visual Modalities Used For Tracking
  • 119. Marker vs. natural feature tracking Marker tracking Usually requires no database to be stored Markers can be an eye-catcher Tracking is less demanding g g The environment must be instrumented with markers Markers usually work only when fully in view y y y Natural feature tracking A database of keypoints must be stored/downloaded Natural feature targets might catch the attention less Natural f t N t l feature targets are potentially everywhere t t t ti ll h Natural feature targets work also if partially in view
  • 121. Example: Outdoor Hybrid Tracking Combines computer vision - natural feature tracking inertial gyroscope sensors Both correct for each other Inertial gyro - provides frame to frame prediction of camera orientation Computer vision - correct for gyro drift
  • 122. Outdoor AR Tracking System You, Neumann, Azuma outdoor AR system (1999)
  • 123. Robust Outdoor Tracking Hybrid T ki H b id Tracking Computer Vision, GPS, inertial Going Out Reitmayer & Drummond (Univ. Cambridge)
  • 126. The Registration Problem Virtual and Real must stay properly aligned If not: t Breaks the illusion that the two coexist Prevents acceptance of many serious applications
  • 127. Sources of registration errors Static errors S Optical distortions Mechanical misalignments Tracker errors Incorrect viewing parameters Dynamic errors System delays (largest source of error) - 1 ms d l = 1/3 mm registration error delay i t ti
  • 128. Reducing static errors Distortion compensation Manual adjustments View-based or direct measurements [Azuma94] [Caudell92] [Janin93] etc. Camera calibration (video) [ARGOS94] [Bajura93] [Tuceryan95] etc.
  • 129. View Based Calibration (Azuma 94)
  • 130. Dynamic errors Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint Scene Display Simulation 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms Total Delay = 50 + 2 + 33 + 17 = 102 ms 1 ms delay = 1/3 mm = 33mm error
  • 131. Reducing dynamic errors (1) Reduce system lag [Olano95] [Wloka95a] [Regan SIGGRAPH99] Reduce apparent lag Image deflection [Burbidge89] [Regan94] [So92] [Kijima [Kiji ISMR 2001] Image warping [Mark 3DI 97]
  • 132. Reducing System Lag Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint Scene Display Simulation Faster Tracker Faster CPU Faster GPU Faster Display
  • 133. Reducing Apparent Lag Virtual Display Virtual Display x,y,z y Physical Ph i l r,p,y Physical Display Display (640x480) (640x480) Tracking 1280 x 960 Update 1280 x 960 Last known position Latest position Application Loop x,y,z Tracking r,p,y Calculate Render Draw to Viewpoint p Scene Display p y Simulation
  • 134. Reducing dynamic errors (2) Match input streams (video) Delay video of real world to match system lag Predictive Tracking [Azuma94] [Emura94] Inertial sensors helpful Azuma / Bishop 1994 u a s op 99
  • 135. Predictive Tracking Position Now Past Future Time Can predict up to 80 ms in future (Holloway)
  • 137. Wrap-up Tracking and Registration are key problems Registration error Measures against static error Measures against dynamic error M i d i AR typically requires multiple tracking technologies yp y q p g g Research Areas: Hybrid Markerless Techniques, Deformable Surface, Mobile, Outdoors Surface Mobile
  • 138. More Information • M k Billi h t Mark Billinghurst – mark.billinghurst@hitlabnz.org • Websites – www.hitlabnz.org hi l b