Introduction to Light Fields
Download as PPTX, PDF9 likes6,266 views
This document provides an introduction to light fields and their applications. It discusses the plenoptic function, which describes the set of all light rays that can be observed. Light fields can be parameterized in different ways, including using position and angle or using two planes. Applications of light fields include digital image refocusing, 3D displays, camera arrays, and controlling camera or object motion. Modern implementations include the Lytro light field camera, which uses a microlens array in front of the image sensor. Other applications discussed include using light fields to reduce lens glare and developing thin, depth-sensing LCD displays.
![The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-2-320.jpg)
![The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-3-320.jpg)
![Application: Digital Image Refocusing
[Ng 2005]
Kodak 16-megapixel sensor
125μ square-sided microlenses](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-10-320.jpg)
![Application: 3D Displays
Parallax Panoramagram
[Kanolt 1918]
3DTV with Integral Imaging
[Okano et al. 1999]
MERL 3DTV
[Matusik and Pfister 2004]](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-11-320.jpg)
![Static Camera Arrays
Stanford Multi-Camera Array
125 cameras using custom hardware
[Wilburn et al. 2002, Wilburn et al. 2005]
Distributed Light Field Camera
64 cameras with distributed rendering
[Yang et al. 2002]](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-13-320.jpg)
![Controlled Camera or Object Motion
Stanford Spherical Gantry
[Levoy and Hanrahan 1996]
Relighting with 4D Incident Light Fields
[Masselus et al. 2003]](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-16-320.jpg)
![Uncontrolled Camera or Object Motion
Unstructured Lumigraph Rendering
[Gortler et al. 1996; Buehler et al. 2001]](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-17-320.jpg)
![Parallax Barriers (Pinhole Arrays)
[Ives 1903]
sensor
barrier
Spatially-multiplexed light field capture using masks (i.e., barriers):
• Cause severe attenuation long exposures or lower SNR
• Impose fixed trade-off between spatial and angular resolution
(unless implemented with programmable masks, e.g. LCDs)](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-19-320.jpg)
![Integral Imaging (“Fly’s Eye” Lenslets)
[Lippmann 1908]
sensor
lenslet
f
Spatially-multiplexed light field capture using lenslets:
• Impose fixed trade-off between spatial and angular resolution](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-20-320.jpg)
![Light Field Photograph (Decoded)
[The (New) Stanford Light Field Archive]
lookingup
looking to the right
Sample Image](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-22-320.jpg)
![Modern, Digital Implementations
Digital Light Field Photography
• Hand-held plenoptic camera [Ng et al. 2005]
• Heterodyne light field camera [Veeraraghavan et al. 2007]](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-24-320.jpg)
![Lens Glare Reduction
[Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]
Glare/Flare due to camera lenses reduces contrast](https://image.slidesharecdn.com/3introlightfields-130924104409-phpapp01/85/Introduction-to-Light-Fields-37-320.jpg)
Introduction to Light Fields
- 1. Introduction to Light Fields
- 2. The 5D Plenoptic Function Q: What is the set of all things that one can ever see? A: The Plenoptic Function [Adelson and Bergen 1991] (from plenus, complete or full, and optic) P(q, f, l, t)
- 3. The 5D Plenoptic Function Q: What is the set of all things that one can ever see? A: The Plenoptic Function [Adelson and Bergen 1991] (from plenus, complete or full, and optic) P(q, f, l, t)
- 4. Position-Angle Parameterization 2D position 2D direction s q
- 5. Two-Plane Parameterization 2D position 2D position s u
- 6. Two-Plane Parameterization – camera array 2D position 2D position s u
- 7. Two-Plane Parameterization – SLR camera 2D position 2D position s u
- 8. Two-Plane Parameterization – cell phone s u
- 9. Two-Plane Parameterization – cell phone (pelican) s u
- 10. Application: Digital Image Refocusing [Ng 2005] Kodak 16-megapixel sensor 125μ square-sided microlenses
- 11. Application: 3D Displays Parallax Panoramagram [Kanolt 1918] 3DTV with Integral Imaging [Okano et al. 1999] MERL 3DTV [Matusik and Pfister 2004]
- 12. Multiple Sensors
- 13. Static Camera Arrays Stanford Multi-Camera Array 125 cameras using custom hardware [Wilburn et al. 2002, Wilburn et al. 2005] Distributed Light Field Camera 64 cameras with distributed rendering [Yang et al. 2002]
- 14. Refocus
- 16. Controlled Camera or Object Motion Stanford Spherical Gantry [Levoy and Hanrahan 1996] Relighting with 4D Incident Light Fields [Masselus et al. 2003]
- 17. Uncontrolled Camera or Object Motion Unstructured Lumigraph Rendering [Gortler et al. 1996; Buehler et al. 2001]
- 19. Parallax Barriers (Pinhole Arrays) [Ives 1903] sensor barrier Spatially-multiplexed light field capture using masks (i.e., barriers): • Cause severe attenuation long exposures or lower SNR • Impose fixed trade-off between spatial and angular resolution (unless implemented with programmable masks, e.g. LCDs)
- 20. Integral Imaging (“Fly’s Eye” Lenslets) [Lippmann 1908] sensor lenslet f Spatially-multiplexed light field capture using lenslets: • Impose fixed trade-off between spatial and angular resolution
- 21. Light Field Photograph (Sensor)
- 22. Light Field Photograph (Decoded) [The (New) Stanford Light Field Archive] lookingup looking to the right Sample Image
- 23. DEMO lenstoys: light field camera
- 24. Modern, Digital Implementations Digital Light Field Photography • Hand-held plenoptic camera [Ng et al. 2005] • Heterodyne light field camera [Veeraraghavan et al. 2007]
- 25. Marc Levoy Light Field = Array of (Virtual) Cameras Slide by Marc Levoy
- 26. Marc Levoy Sub-aperture Virtual Camera = Sub-aperture View Light Field = Array of (Virtual) Cameras Slide by Marc Levoy
- 27. Marc Levoy Sub-aperture Virtual Camera = Sub-aperture View Light Field = Array of (Virtual) Cameras Slide by Marc Levoy
- 28. Marc Levoy
- 30. The Lytro Light Field Camera
- 33. Lytro: microlens array in camera Main lens Microlens array (MLA) Digital sensor Lens is not really thin, but can be treated as so. 330 × 330 hex array, 13.9 micron pitch Occluders are not required. 14 Mpixel, square cropped to 11 Mpixels Choose pixel in same location behind each microlens
- 34. Microlens array (MLA) Main lens Digital sensor # of sub-apertures = # of pixels behind each microlens (10 × 10) # of pixels per sub-aperture image = # of microlenses (~ 120,000) All rays pass through a “sub-aperture” Sub-aperture captures on camera view Choose pixel in same location behind each microlens So why put the microlens array inside the camera?
- 35. Lytro camera has unusual shape 8x f/2 lens Light field sensor Battery 43-343 mm equivalent
- 36. Other Applications of Light Fields
- 37. Lens Glare Reduction [Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008] Glare/Flare due to camera lenses reduces contrast
- 38. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare Glare Reduced Image After removing outliersConventional Photo
- 39. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Enhancing Glare Glare Enhanced ImageConventional Photo
- 40. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan
- 41. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan
- 42. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Glare due to Lens Inter-Reflections a Sensor b
- 43. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Effects of Glare on Image • Hard to model, Low Frequency in 2D • But reflection glare is outlier in 4D ray-space Angular Variation at pixel a Lens Inter-reflections a Sensor b
- 44. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Key Idea • Lens Glare manifests as low frequency in 2D Image • But Glare is highly view dependent – manifests as outliers in 4D ray-space • Reducing Glare == Remove outliers among rays a Sensor b
- 45. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare using a Light Field Camera
- 46. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Single Shot Light Field Cameras Using Mask, this paper Using Lenslets, Ng et al. 2005 Mask Adelson and Wang, 1992, Ng et al. 2005 Kanolt 1933, Veeraraghavan et al. 2007
- 47. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Captured Photo: LED off
- 48. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Captured Photo: LED On
- 49. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Each Disk: Angular Samples at that Spatial Location No Glare
- 50. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan With Glare
- 51. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan x y u v
- 52. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Sequence of Sub-Aperture Views Average of all the Light Field views One of the Light Field views Low Res Traditional Camera Photo Glare Reduced Image
- 53. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Key Idea • Reducing Glare == Remove outlier among angular samples a Sensor b
- 54. Light-sensitive Displays Depth Cameras Multi-touch Interaction Multi-touch Interaction with Thin Displays
- 55. BiDi Screen: Thin, Depth-sensing LCDs • Seamless transition from multi-touch to gesture • Thin form factor (LCD)