introduction to camera, the operation of camera
0 likes22 views
The document discusses various aspects of camera technology and image projection, tracing its historical development from the camera obscura to modern digital sensors. It explains the principles of projection, the use of homogeneous coordinates, and the impact of lens imperfections on image quality. Additionally, it covers the evolution of photographic techniques and highlights significant milestones in the history of photography.
![Homogeneous Coordinates
z
x
y
[x,y,w]
λ[x,y,w]
Two homogeneous coordinates are
equivalent if they are proportional
to each other. Not = !
𝑢
𝑣
𝑤
≡
𝑢′
𝑣′
𝑤′
𝑢
𝑣
𝑤
= 𝜆
𝑢′
𝑣′
𝑤′
𝜆 ≠ 0
Triple /
Equivalent
Double /
Equals](https://image.slidesharecdn.com/02cameras-241109044538-36ee51f3/85/introduction-to-camera-the-operation-of-camera-27-320.jpg)
![Benefits of Homogeneous Coords
What’s the intersection?
0x + 1y - 2 = 0
1x + 0y - 1 = 0
[0,1,-2] x [1,0,-1] = [-1,-2,-1]
Converting back (divide by -1)
(1,2)](https://image.slidesharecdn.com/02cameras-241109044538-36ee51f3/85/introduction-to-camera-the-operation-of-camera-30-320.jpg)
![Recap: Homogeneous Coords
Line y=2
0x + 1y - 2 = 0
Line x=1
1x + 0y - 1 = 0
[0,1,-2] x [1,0,-1] = [-1,-2,-1]
Converting back (divide by -1)
(1,2)
𝑎1, 𝑏1, 𝑐1 = (0,1, −2)
𝑎2, 𝑏2, 𝑐2 = (1,0, −1)
Intersection: l1 x l2](https://image.slidesharecdn.com/02cameras-241109044538-36ee51f3/85/introduction-to-camera-the-operation-of-camera-34-320.jpg)
![Benefits of Homogeneous Coords
0x + 1y - 1 = 0
0x + 1y - 2 = 0
Intersection of y=2, y=1
[0,1,-2] x [0,1,-1] = [1,0,0]
0x + 1y - 3 = 0
Does it lie on y=3? Intuitively?
[0,1,-3]T[1,0,0] = 0](https://image.slidesharecdn.com/02cameras-241109044538-36ee51f3/85/introduction-to-camera-the-operation-of-camera-35-320.jpg)
![Typical Perspective Model
𝑷 ≡
𝑓 0 𝑢0
0 𝑓 𝑣0
0 0 1
𝑹3𝑥3 𝒕3𝑥1 𝑿4𝑥1
Intrinsic
Matrix K
Extrinsic
Matrix [R,t]
𝑷 ≡ 𝑲 𝑹, 𝒕 𝑿 ≡ 𝑴3𝑥4𝑿4𝑥1](https://image.slidesharecdn.com/02cameras-241109044538-36ee51f3/85/introduction-to-camera-the-operation-of-camera-44-320.jpg)
introduction to camera, the operation of camera
- 2. Let’s Take a Picture! Slide inspired by S. Seitz; image from Michigan Engineering Photosensitive Material Idea 1: Just use film Result: Junk
- 3. Let’s Take a Picture! Slide inspired by S. Seitz; image from Michigan Engineering Photosensitive Material Idea 2: add a barrier
- 4. Let’s Take a Picture! Slide inspired by S. Seitz; image from Michigan Engineering Photosensitive Material Idea 2: add a barrier
- 5. Let’s Take a Picture! Slide inspired by S. Seitz; image from Michigan Engineering Photosensitive Material Film captures all the rays going through a point (a pencil of rays). Result: good in theory!
- 6. Camera Obscura • Basic principle known to Mozi (470-390 BCE), Aristotle (384-322 BCE) • Drawing aid for artists: described by Leonardo da Vinci (1452-1519) Gemma Frisius, 1558 Source: A. Efros
- 7. Camera Obscura Abelardo Morell, Camera Obscura Image of Manhattan View Looking South in Large Room, 1996 http://www.abelardomorell.net/project/camera-obscura/ From Grand Images Through a Tiny Opening, Photo District News, February 2005
- 9. Projection O P How do we find the projection P of a point X? Form visual ray from X to camera center and intersect it with camera plane X Source: L Lazebnik
- 10. Projection P Both X and X’ project to P. Which appears in the image? Are there points for which projection is undefined? X’ X Source: L Lazebnik O
- 11. Quick Aside: Remember This? θ θ a b c d 𝑎 𝑏 = 𝑑 𝑐 𝑎 = 𝑏𝑑 𝑐
- 12. Projection Equations O P X (x,y,z) Coordinate system: O is origin, XY in image, Z sticks out. XY is image plane, Z is optical axis. z x y f (x,y,z) projects to (fx/z,fy/z) via similar triangles Source: L Lazebnik
- 13. Some Facts About Projection The projection of any 3D parallel lines converge at a vanishing point List of properties from M. Hebert 3D lines project to 2D lines Distant objects are smaller
- 14. Some Facts About Projection Let’s try some fake images
- 15. Some Facts About Projection Slide by Steve Seitz
- 16. Some Facts About Projection Slide by Steve Seitz
- 17. Some Facts About Projection Illusion Credit: RN Shepard, Mind Sights: Original Visual Illusions, Ambiguities, and other Anomalies
- 18. What’s Lost? Inspired by D. Hoiem slide Is she shorter or further away? Are the orange lines we see parallel / perpendicular / neither to the red line? 1 2 3 4
- 19. What’s Lost? Adapted from D. Hoiem slide Is she shorter or further away? Are the orange lines we see parallel / perpendicular / neither to the red line? 1 2 3 4
- 20. What’s Lost? Be careful of drawing conclusions: • Projection of 3D line is 2D line; NOT 2D line is 3D line. • Can you think of a counter-example (a 2D line that is not a 3D line)? • Projections of parallel 3D lines converge at VP; NOT any pair of lines that converge are parallel in 3D. • Can you think of a counter-example?
- 21. Do You Always Get Perspective?
- 22. Do You Always Get Perspective? 𝒇𝒚 𝒛𝟏 𝒇𝒚 𝒛𝟐 𝒇𝒚 𝒛 𝒇𝒚 𝒛 Y location of blue and red dots in image:
- 23. Do You Always Get Perspective? When plane is fronto-parallel (parallel to camera plane), everything is: • scaled by f/z • otherwise is preserved.
- 24. What’s This Useful For? Things looking different when viewed from different angles seems like a nuisance. It’s also a cue. Why?
- 25. Projection Equation P X f z x y (x,y,z) → (fx/z,fy/z) I promised you linear algebra: is this linear? Nope: division by z is non-linear (and risks division by 0) Adapted from S. Seitz slide O
- 26. Homogeneous Coordinates (2D) Adapted from M. Hebert slide Trick: add a dimension! This also clears up lots of nasty special cases What if w = 0? Physical Point 𝑥 𝑦 Homogeneous Point 𝑢 𝑣 𝑤 Concat w=1 Divide by w 𝑢/𝑤 𝑣/𝑤 Physical Point
- 27. Homogeneous Coordinates z x y [x,y,w] λ[x,y,w] Two homogeneous coordinates are equivalent if they are proportional to each other. Not = ! 𝑢 𝑣 𝑤 ≡ 𝑢′ 𝑣′ 𝑤′ 𝑢 𝑣 𝑤 = 𝜆 𝑢′ 𝑣′ 𝑤′ 𝜆 ≠ 0 Triple / Equivalent Double / Equals
- 28. Benefits of Homogeneous Coords General equation of 2D line: 𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0 Homogeneous Coordinates 𝒍𝑇 𝒑 = 0, 𝒍 = 𝑎 𝑏 𝑐 , 𝒑 = 𝑥 𝑦 1 Slide from M. Hebert
- 29. Benefits of Homogeneous Coords • Lines (3D) and points (2D → 3D) are now the same dimension. • Use the cross (x) and dot product for: • Intersection of lines l and m: l x m • Line through two points p and q: p x q • Point p on line l: lTp • Parallel lines, vertical lines become easy (compared to y=mx+b)
- 30. Benefits of Homogeneous Coords What’s the intersection? 0x + 1y - 2 = 0 1x + 0y - 1 = 0 [0,1,-2] x [1,0,-1] = [-1,-2,-1] Converting back (divide by -1) (1,2)
- 33. Recap: Homogeneous Coords 𝑢, 𝑣, 𝑤 = (1,2,1) Append 1 Divide by w 𝑥, 𝑦 = (1,2) 0x + 1y - 2 = 0 Line of y=2 in ax+by+c=0: 𝑎, 𝑏, 𝑐 = 0,1, −2 0,1, −2 𝑇 1,2,1 = 0 𝑎, 𝑏, 𝑐 𝑇 𝑢, 𝑣, 𝑤 = Point-on-line test: lTp 𝑥, 𝑦 = (1,2)
- 34. Recap: Homogeneous Coords Line y=2 0x + 1y - 2 = 0 Line x=1 1x + 0y - 1 = 0 [0,1,-2] x [1,0,-1] = [-1,-2,-1] Converting back (divide by -1) (1,2) 𝑎1, 𝑏1, 𝑐1 = (0,1, −2) 𝑎2, 𝑏2, 𝑐2 = (1,0, −1) Intersection: l1 x l2
- 35. Benefits of Homogeneous Coords 0x + 1y - 1 = 0 0x + 1y - 2 = 0 Intersection of y=2, y=1 [0,1,-2] x [0,1,-1] = [1,0,0] 0x + 1y - 3 = 0 Does it lie on y=3? Intuitively? [0,1,-3]T[1,0,0] = 0
- 36. Benefits of Homogeneous Coords Translation is now linear / matrix-multiply Rigid body transforms (rot + trans) now linear 𝑢′ 𝑣′ 𝑤′ = 𝑟11 𝑟12 𝑡𝑥 𝑟21 𝑟22 𝑡𝑦 0 0 1 𝑢 𝑣 𝑤 𝑢′ 𝑣′ 𝑤′ = 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 𝑢 𝑣 1 = 𝑢 + 𝑡𝑥 𝑣 + 𝑡𝑦 1 𝑢′ 𝑣′ 𝑤′ = 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 𝑢 𝑣 𝑤 = 𝑢 + 𝑤𝑡𝑥 𝑣 + 𝑤𝑡𝑦 𝑤 If w = 1 Generically
- 37. 3D Homogeneous Coordinates Same story: add a coordinate, things are equivalent if they’re proportional 𝑢 𝑣 𝑤 𝑡 𝑥 𝑦 𝑧 𝑢/𝑡 𝑣/𝑡 𝑤/𝑡
- 38. Projection Matrix Projection (fx/z, fy/z) is matrix multiplication Slide inspired from L. Lazebnik 𝑓𝑥 𝑓𝑦 𝑧 ≡ 𝑓 0 0 0 0 𝑓 0 0 0 0 1 0 𝑥 𝑦 𝑧 1 → 𝑓𝑥/𝑧 𝑓𝑦/𝑧 O f dis
- 39. Projection Matrix Projection (fx/z, fy/z) is matrix multiplication Slide inspired from L. Lazebnik 𝑓𝑥 𝑓𝑦 𝑧 ≡ 𝑓 0 0 0 0 𝑓 0 0 0 0 1 0 𝑥 𝑦 𝑧 1 → 𝑓𝑥/𝑧 𝑓𝑦/𝑧 O f
- 40. Why ≡ ≠ = O P X’ X 𝑓𝑥 𝑓𝑦 𝑧 ≡ 𝑓𝑥′ 𝑓𝑦 𝑧′ ′ YES 𝑓𝑥 𝑓𝑦 𝑧 = 𝑓𝑥′ 𝑓𝑦 𝑧′ ′ NO Project X and X’ to the image and compare them
- 41. Typical Perspective Model 𝑷 ≡ 𝑓 0 𝑢0 0 𝑓 𝑣0 0 0 1 𝑹3𝑥3 𝒕3𝑥1 𝑿4𝑥1 P: 2D homogeneous point (3D) X: 3d homogeneous point (4D)
- 42. Typical Perspective Model 𝑷 ≡ 𝑓 0 𝑢0 0 𝑓 𝑣0 0 0 1 𝑹3𝑥3 𝒕3𝑥1 𝑿4𝑥1 t: translation between world system and camera R: rotation between world system and camera
- 43. Typical Perspective Model 𝑷 ≡ 𝑓 0 𝑢0 0 𝑓 𝑣0 0 0 1 𝑹3𝑥3 𝒕3𝑥1 𝑿4𝑥1 f focal length u0,v0: principal point (image coords of camera origin on retina)
- 44. Typical Perspective Model 𝑷 ≡ 𝑓 0 𝑢0 0 𝑓 𝑣0 0 0 1 𝑹3𝑥3 𝒕3𝑥1 𝑿4𝑥1 Intrinsic Matrix K Extrinsic Matrix [R,t] 𝑷 ≡ 𝑲 𝑹, 𝒕 𝑿 ≡ 𝑴3𝑥4𝑿4𝑥1
- 45. Other Cameras – Orthographic Orthographic Camera (z infinite) 𝑷 = 1 0 0 0 1 0 0 0 0 𝑿3𝑥1 Image Credit: Wikipedia
- 46. Other Cameras – Orthographic 𝑷 = 1 0 0 0 1 0 0 0 0 𝑥 𝑦 𝑧 Why does this make things easy and why is this popular in old games?
- 47. The Big Issue Slide inspired by S. Seitz; image from Michigan Engineering Photosensitive Material Film captures all the rays going through a point (a pencil of rays). How big is a point?
- 48. Math vs. Reality • Math: Any point projects to one point • Reality (as pointed out by the class) • Don’t image points behind the camera / objects • Don’t have an infinite amount of sensor material • Other issues • Light is limited • Spooky stuff happens with infinitely small holes
- 49. Limitations of Pinhole Model Ideal Pinhole -1 point generates 1 image -Low-light levels Finite Pinhole -1 point generates region -Blurry. Why is it blurry? Slide inspired by M. Hebert
- 50. Limitations of Pinhole Model Slide Credit: S. Seitz
- 51. Adding a Lens • A lens focuses light onto the film • Thin lens model: rays passing through the center are not deviated (pinhole projection model still holds) Slide Credit: S. Seitz
- 52. Adding a Lens • All rays parallel to the optical axis pass through the focal point focal point f Slide Credit: S. Seitz
- 53. What’s The Catch? “circle of confusion” Slide Credit: S. Seitz • There’s a distance where objects are “in focus” • Other points project to a “circle of confusion”
- 54. Thin Lens Formula object image plane lens Diagram credit: F. Durand We care about images that are in focus. When is this true? Discuss with your neighbor. When two paths from a point hit the same image location.
- 55. Thin Lens Formula f D D′ object image plane lens Diagram credit: F. Durand Let’s derive the relationship between object distance D, image plane distance D’, and focal length f. y y′
- 56. Thin Lens Formula f D D′ object image plane lens Diagram credit: F. Durand One set of similar triangles: y y′ 𝑦′ 𝐷′ − 𝑓 = 𝑦 𝑓 𝑦′ 𝑦 = 𝐷′ − 𝑓 𝑓
- 57. Thin Lens Formula f D D′ object image plane lens Diagram credit: F. Durand y y′ 𝑦′ 𝐷′ = 𝑦 𝐷 Another set of similar triangles: 𝑦′ 𝑦 = 𝐷′ 𝐷
- 58. Thin Lens Formula f D D′ object image plane lens Diagram credit: F. Durand y y′ Set them equal: 𝐷′ 𝐷 = 𝐷 − 𝑓 𝑓 1 𝐷 + 1 𝐷′ = 1 𝑓
- 59. Thin Lens Formula Diagram credit: F. Durand f D D′ object image plane lens 1 𝐷 + 1 𝐷′ = 1 𝑓 Suppose I want to take a picture of a lion with D big? Which of D, D’, f are fixed? How do we take pictures of things at different distances?
- 60. Depth of Field http://www.cambridgeincolour.com/tutorials/depth-of-field.htm Slide Credit: A. Efros
- 61. Controlling Depth of Field Changing the aperture size affects depth of field A smaller aperture increases the range in which the object is approximately in focus Diagram: Wikipedia
- 62. Controlling Depth of Field Diagram: Wikipedia If a smaller aperture makes everything focused, why don’t we just always use it?
- 63. Varying the Aperture Slide Credit: A. Efros, Photo: Philip Greenspun Large aperture = small DOF Small aperture = large DOF
- 65. Field of View (FOV) tan-1 is monotonic increasing. How can I get the FOV bigger? Photo. Material 𝜙 = tan−1 𝑑 2𝑓 𝜙 𝑓 𝑑
- 66. Field of View Slide Credit: A. Efros
- 67. Field of View Slide Credit: A. Efros
- 68. Field of View and Focal Length Large FOV, small f Camera close to car Small FOV, large f Camera far from the car Slide Credit: A. Efros, F. Durand
- 69. Field of View and Focal Length standard wide-angle telephoto Slide Credit: F. Durand
- 70. Dolly Zoom Change f and distance at the same time Video Credit: Goodfellas 1990
- 71. More Bad News! • First a pinhole… • Then a thin lens model…. Slide: L. Lazebnik
- 72. Lens Flaws: Radial Distortion Photo: Mark Fiala, U. Alberta Lens imperfections cause distortions as a function of distance from optical axis Less common these days in consumer devices
- 73. Radial Distortion Correction Photo. Material r f z Ideal 𝑦′ = 𝑓 𝑦 𝑧 y' y Distorted 𝑦′ = (1 + 𝑘1𝑟2 + ⋯ ) 𝑦 𝑧
- 74. Photo. Material Vignetting Slide inspired by L. Lazebnik Slide What happens to the light between the black and red lines?
- 75. Vignetting Photo credit: Wikipedia (https://en.wikipedia.org/wiki/Vignetting)
- 76. Lens Flaws: Spherical Abberation Lenses don’t focus light perfectly! Rays farther from the optical axis focus closer Slide: L. Lazebnik
- 77. Lens Flaws: Chromatic Abberation Lens refraction index is a function of the wavelength. Colors “fringe” or bleed Image credits: L. Lazebnik, Wikipedia
- 78. Lens Flaws: Chromatic Abberation Researchers tried teaching a network about objects by forcing it to assemble jigsaws. Slide Credit: C. Doersch
- 79. From Photon to Photo • Each cell in a sensor array is a light-sensitive diode that converts photons to electrons • Dominant in the past: Charge Coupled Device (CCD) • Dominant now: Complementary Metal Oxide Semiconductor (CMOS) Slide Credit: L. Lazebnik, Photo Credit: Wikipedia, Stefano Meroli
- 80. From Photon to Photo Rolling Shutter: pixels read in sequence Can get global reading, but $$$
- 81. Preview of What’s Next Demosaicing: Estimation of missing components from neighboring values Bayer grid Human Luminance Sensitivity Function Slide Credit: S. Seitz
- 82. Historic milestones • Pinhole model: Mozi (470-390 BCE), Aristotle (384-322 BCE) • Principles of optics (including lenses): Alhacen (965-1039 CE) • Camera obscura: Leonardo da Vinci (1452-1519), Johann Zahn (1631-1707) • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes (1839) • Photographic film (Eastman, 1889) • Cinema (Lumière Brothers, 1895) • Color Photography (Lumière Brothers, 1908) • Television (Baird, Farnsworth, Zworykin, 1920s) • First consumer camera with CCD Sony Mavica (1981) • First fully digital camera: Kodak DCS100 (1990) Niepce, “La Table Servie,” 1822 Alhacen’s notes Old television camera Slide Credit: S. Lazebnik
- 83. First digitally scanned photograph • 1957, 176x176 pixels Slide Credit: http://listverse.com/history/top-10-incredible-early-firsts-in-photography/
- 84. Historic Milestone Sergey Prokudin-Gorskii (1863-1944) Photographs of the Russian empire (1909-1916) Slide Credit: S. Maji
- 85. Historic Milestone Slide Credit: S. Maji
- 86. Future Milestone Your job in homework 1: Make the left look like the right.