Stereo and 3D Displays - Matt Hirsch
5 likes3,402 views
This document discusses various types of 3D displays. It begins with an overview of depth cues that can be presented to the human visual system, both monocular cues like size and occlusion, as well as binocular cues like retinal disparity and convergence. The document then presents a taxonomy of 3D display technologies, categorizing them as either glasses-bound or unencumbered designs. Specific display types are described in more detail, including head-mounted displays, spatial and temporal multiplexing, parallax barriers, integral imaging, and volumetric and holographic displays. Multi-view rendering techniques for generating stereoscopic images are also covered, such as using OpenGL for anaglyph generation and off-axis perspective projection.
![Additional Monocular Depth Cues
motion parallax [Hermann von Helmholtz, 1866]
accommodation
What is missing?](https://image.slidesharecdn.com/rameshclass2013-131003065906-phpapp01/85/Stereo-and-3D-Displays-Matt-Hirsch-4-320.jpg)
![Binocular Depth Cues
retinal disparity [Charles Wheatstone, 1838]
convergence
“It being thus established that the mind perceives an object of three dimensions by means of the
two dissimilar pictures projected by it on the two retinae, the following question occurs: What
would be the visual effect of simultaneously presenting to each eye, instead of the object itself, its
projection on a plane surface as it appears to that eye?”
Binocular Depth Cues](https://image.slidesharecdn.com/rameshclass2013-131003065906-phpapp01/85/Stereo-and-3D-Displays-Matt-Hirsch-5-320.jpg)
![Taxonomy of 3D Displays:
Dynamic Holograms (Holovideo)
Unencumbered
Automultiscopic
Parallax-based
(2D display with light-directing elements)
Volumetric
(directly illuminate points within a volume)
Holographic
(reconstructs wavefront using 2D element)
Parallax Barriers
(uniform array of 1D slits or 2D pinhole arrays)
Integral Imaging
(lenticular sheets or fly’s eye lenslet arrays)
Multi-planar
(time-sequential projection onto swept surfaces)
Transparent Substrates
(intersecting laser beams, fog layers, etc.)
Static
(holographic films)
Dynamic
(holovideo)
Tay et al.
[Nature, 2008]
MIT Media Lab Spatial Imaging Group
[Holovideo, 1989 – present]](https://image.slidesharecdn.com/rameshclass2013-131003065906-phpapp01/85/Stereo-and-3D-Displays-Matt-Hirsch-19-320.jpg)
![% read in images
ImL = imread('l.png');
ImR = imread('r.png');
% define "half color" matrices (see slides)
L = [.299 0 0
.587 0 0
.114 0 0];
R = [0 0 0
0 1 0
0 0 1];
% create a pixel x color array
ImL1d = double(reshape(ImL,prod(size(ImL(:,:,1))),3));
ImR1d = double(reshape(ImR,prod(size(ImR(:,:,1))),3));
% perform per pixel color permutation
ImL1d = ImL1d*L;
ImR1d = ImR1d*R;
Anaglyphic Model Viewer:
Making an Anaglyph Image in MATLAB
% convert back to 2d x color image
ImL = uint8(reshape(ImL1d,size(ImL)));
ImR = uint8(reshape(ImR1d,size(ImR)));
% create output
Iout = ImL + ImR;
anaglyph.m](https://image.slidesharecdn.com/rameshclass2013-131003065906-phpapp01/85/Stereo-and-3D-Displays-Matt-Hirsch-27-320.jpg)
Stereo and 3D Displays - Matt Hirsch
- 1. ➢Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology Stereo and 3D Displays
- 2. Monocular Depth Cues Supported by Conventional Displays relative and familiar size perspective and occlusion texture gradient, shading and lighting, atmospheric effects Limitations of Conventional Displays
- 3. Limitations of Conventional DisplaysLimitations of Conventional Displays Monocular Depth Cues with Conventional Displays relative and familiar size perspective and occlusion texture gradient, shading and lighting, atmospheric effects
- 4. Additional Monocular Depth Cues motion parallax [Hermann von Helmholtz, 1866] accommodation What is missing?
- 5. Binocular Depth Cues retinal disparity [Charles Wheatstone, 1838] convergence “It being thus established that the mind perceives an object of three dimensions by means of the two dissimilar pictures projected by it on the two retinae, the following question occurs: What would be the visual effect of simultaneously presenting to each eye, instead of the object itself, its projection on a plane surface as it appears to that eye?” Binocular Depth Cues
- 6. American Civil War-era stereoscopic photos • Available from the US library of congress •http://www.loc.gov/pictures/search - Search for “stereographs civil war prints” • Lincoln in 3D • Selection of stereographs converted to red- cyan anaglyph images •John J. Richter: ISBN 978-0811872317 Interesting Historical Example
- 7. l The HVS can ignore conflicting or missing depth cues l Understand depth in 2D (monocular) video l Perceive shape in “noise” Ponzo Illusion: © Walt Anthony 2006magiceye.com Conflicting Cues
- 8. Stereo and 3D Displays Introduction: History and Physiology ➢Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology
- 9. Taxonomy of 3D Displays: Glasses-bound vs. Unencumbered Designs Glasses-bound Stereoscopic Immersive (blocks direct-viewing of real world) See-through (superimposes synthetic images onto real world) Head-mounted (eyepiece-objective and microdisplay) Multiplexed (stereo pair with same display surface) Spatially-multiplexed (field-concurrent) (color filters, polarizers, autostereograms, etc.) Temporally-multiplexed (field-sequential) (LCD shutter glasses) Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Multi-planar (time-sequential projection onto swept surfaces) Transparent Substrates (intersecting laser beams, fog layers, etc.) Static (holographic films) Dynamic (holovideo) Taxonomy adapted from Hong Hua
- 10. Taxonomy of 3D Displays: Immersive Head-mounted Displays (HMDs) Glasses-bound Stereoscopic Immersive (blocks direct-viewing of real world) Head-mounted (eyepiece-objective and microdisplay) Multiplexed (stereo pair with same display surface)
- 11. Taxonomy of 3D Displays: See-through Head-mounted Displays (HMDs) Glasses-bound Stereoscopic Immersive (blocks direct-viewing of real world) See-through (superimposes synthetic images onto real world) Head-mounted (eyepiece-objective and microdisplay) Multiplexed (stereo pair with same display surface)
- 12. Taxonomy of 3D Displays: Spatial Multiplexing (e.g., Anaglyphs) Glasses-bound Stereoscopic Immersive (blocks direct-viewing of real world) See-through (superimposes synthetic images onto real world) Head-mounted (eyepiece-objective and microdisplay) Multiplexed (stereo pair with same display surface) Spatially-multiplexed (field-concurrent) (color filters, polarizers, etc.)
- 13. Taxonomy of 3D Displays: Temporal Multiplexing (e.g., Shutter Glasses) Glasses-bound Stereoscopic Immersive (blocks direct-viewing of real world) See-through (superimposes synthetic images onto real world) Head-mounted (eyepiece-objective and microdisplay) Multiplexed (stereo pair with same display surface) Spatially-multiplexed (field-concurrent) (color filters, polarizers, autostereograms, etc.) Temporally-multiplexed (field-sequential) (LCD shutter glasses)
- 14. Taxonomy of 3D Displays: Parallax Barriers Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) NewSight MV-42AD3 42'' (1920x1080, 1x8 views)
- 15. Taxonomy of 3D Displays: Integral Imaging Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Alioscopy 3DHD 42'' (1920x1200, 1x8 views)
- 16. Taxonomy of 3D Displays: Multi-planar Volumetric Displays Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Multi-planar (time-sequential projection onto swept surfaces)
- 17. Taxonomy of 3D Displays: Transparent-substrate Volumetric Displays Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Multi-planar (time-sequential projection onto swept surfaces) Transparent Substrates (intersecting laser beams, fog layers, etc.)
- 18. Taxonomy of 3D Displays: Static Holograms Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Multi-planar (time-sequential projection onto swept surfaces) Transparent Substrates (intersecting laser beams, fog layers, etc.) Static (holographic films) capture reconstruction
- 19. Taxonomy of 3D Displays: Dynamic Holograms (Holovideo) Unencumbered Automultiscopic Parallax-based (2D display with light-directing elements) Volumetric (directly illuminate points within a volume) Holographic (reconstructs wavefront using 2D element) Parallax Barriers (uniform array of 1D slits or 2D pinhole arrays) Integral Imaging (lenticular sheets or fly’s eye lenslet arrays) Multi-planar (time-sequential projection onto swept surfaces) Transparent Substrates (intersecting laser beams, fog layers, etc.) Static (holographic films) Dynamic (holovideo) Tay et al. [Nature, 2008] MIT Media Lab Spatial Imaging Group [Holovideo, 1989 – present]
- 20. Stereo and 3D Displays Introduction: History and Physiology Display Taxonomy ➢Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology
- 21. Overview: GLSL: Programmable Pipeline Fixed Function Pipeline Simple 1-Slide Explanation! Drawing APIDrawing API Process VerticesProcess Vertices Process PixelsProcess Pixels FramebufferFramebuffer Programmable Pipeline Vertex ProgramVertex Program Fragment ProgramFragment Program
- 22. l Some graphics cards have support for stereo 3D (Not on mobile) l Double buffered stereo = Quad buffered void display(void) { glDrawBuffer(GL_BACK_LEFT); <Draw left eye here> glDrawBuffer(GL_BACK_RIGHT); <Draw right eye here> glutSwapBuffers(); } int main(int argc, char **argv) { glutInit(&argc, argv); glutInitDisplayMode( GLUT_DOUBLE | GLUT_RGB | GLUT_STEREO); glutCreateWindow("stereo example"); glutDisplayFunc(display); glutMainLoop(); return 0; } Anaglyphic Model Viewer: Stereo 3D in OpenGL
- 23. Overview: Multi-View Rendering in OpenGL OpenGL Draw Calls Render Standard Pipeline Output Multi-View Pipeline Loop Over Views Backbuffer Framebuffer Object Array Render View Change Camera Screen:Memory:
- 24. Overview: Multi-View Interlacing using GLSL Shaders Framebuffer Object Array Framebuffer Object Array View 1 View 2 View 3 GLSL Program Translate views appropriately for output device Translate views appropriately for output device BackbufferBackbuffer Anaglyph Glasses Anaglyph Glasses LenticularLenticular Shown in this course… The model can apply to many others
- 25. Multi-View Rendering in OpenGL: Off-Axis Perspective Projection with glFrustum() Output
- 26. Anaglyphic Model Viewer: Anaglyph Compositing Algorithms LL RR 3x3 Color Transform Matrix Pair3x3 Color Transform Matrix Pair Full Color Half Color Optimized L= R= 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 L= R= 0.299 0 0 0.587 0 0 0.114 0 0 0 0 0 0 1 0 0 0 1 L= R= 0 0 0 0.7 0 0 0.3 0 0 0 0 0 0 1 0 0 0 1 = Source: http://3dtv.at/Knowhow/AnaglyphComparison_en.aspx
- 27. % read in images ImL = imread('l.png'); ImR = imread('r.png'); % define "half color" matrices (see slides) L = [.299 0 0 .587 0 0 .114 0 0]; R = [0 0 0 0 1 0 0 0 1]; % create a pixel x color array ImL1d = double(reshape(ImL,prod(size(ImL(:,:,1))),3)); ImR1d = double(reshape(ImR,prod(size(ImR(:,:,1))),3)); % perform per pixel color permutation ImL1d = ImL1d*L; ImR1d = ImR1d*R; Anaglyphic Model Viewer: Making an Anaglyph Image in MATLAB % convert back to 2d x color image ImL = uint8(reshape(ImL1d,size(ImL))); ImR = uint8(reshape(ImR1d,size(ImR))); % create output Iout = ImL + ImR; anaglyph.m
- 29. Stereo and 3D Displays Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL ➢Designing Content for Glasses-free 3D Displays Emerging Technology
- 30. • Stereo cameras (commercial and improvised) are common Source Data Stereo Cameras
- 31. • Many researchers/hobbyists have built their own solutions to capture light fields • The PointGrey ProFusion is one of the few commercially available PointGrey ProFusion Stanford Source Data Light Field Cameras MIT
- 32. Focal Plane Example in Anaglyph Viewer Screen Virtual Object Placing objects farm from the plane of focus is uncomfortable Displays with limited DOF: objects further from screen plane are blurred Rendering Tips Accommodation & Object Placement
- 33. Focal Plane Screen Kirshnan, V. V., Stark, L. A heuristic model for the human vergence eye movement system, IEEE Trans. BioMed, 1977. Limit distance of virtual object to viewer Limit rate of change in scene distance <1 m/s for distant objects Rendering Tips Comfortable Vergence
- 34. Off-axis parallel projection Rotate and translate – Toe-in • Puts ‘infinity’ at axis of rotation •Requires user to focus beyond infinity •Some find diverged eyes uncomfortable Disadvantages of toe-in • Distortion between views •Camera distance to most objects change •Off axis objects will have different perspective projection Rendering Tips Camera Model Choice
- 35. Avoid cases that cause a view to differ greatly from its neighbor Left Right Pillar pointing at viewer Left Right Clipped by edge of screen Also watch out for • Far objects clipped by near object • Edges of hallways, tunnels, tubes, etc Also watch out for • Don’t exit in front of screen plane • More comfortable behind screen • Avatar does a good job with this Rendering Tips Clipping and Degenerate Cases
- 36. Warzone 2100: GL Game Conversion
- 37. Stereo and 3D Displays Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays ➢Emerging Technology
- 38. Tensor Display
- 39. Tensor Display
- 40. Stereo and 3D Displays Resources SIGGRAPH 2010/2011 Course: BYO3D http://web.media.mit.edu/~mhirsch/byo3d/index.html Long-form slides Code and examples display blocks Display Blocks blog http://displayblocks.org Tutorials Building blocks explained Gordon Wetzstein gordonw@media.mit.edu Matt Hirsch mhirsch@media.mit.edu