Introduction to Binocular Stereo in Computer Vision
- 1. Binocular Stereo CS5670: Computer Vision Single image stereogram, https://en.wikipedia.org/wiki/Autostereogram What is this?
- 2. Announcements • Project 3 due tomorrow, Friday, March 18 at 8pm (code), Monday, March 21 at 8pm (artifact) • Project 4 (Stereo) to be released on Tuesday, March 22, due Friday, April 1, by 8pm – To be done in groups of two
- 3. “Mark Twain at Pool Table", no date, UCR Museum of Photography
- 5. • An object point will project to some point in our image • That image point corresponds to a ray in the world • Two rays intersect at a single point, so if we want to localize points in 3D we need 2 eyes Stereo Vision as Localizing Points in 3D
- 6. Stereo • Given two images from different viewpoints – How can we compute the depth of each point in the image? – Based on how much each pixel moves between the two images
- 7. epipolar lines Epipolar geometry (x1, y1) (x2, y1) x2 - x1 = the disparity of pixel (x1, y1) Two images captured by a purely horizontal translating camera (rectified stereo pair)
- 8. Disparity = inverse depth http://stereo.nypl.org/view/41729 (Or, hold a finger in front of your face and wink each eye in succession.)
- 9. Your basic stereo matching algorithm • Match Pixels in Conjugate Epipolar Lines – Assume brightness constancy – This is a challenging problem – Hundreds of approaches • A good survey and evaluation: http://www.middlebury.edu/stereo/
- 10. Your basic stereo matching algorithm For each epipolar line For each pixel in the left image • compare with every pixel on same epipolar line in right image • pick pixel with minimum match cost Improvement: match windows
- 11. Stereo matching based on SSD SSD dmin d Best matching disparity
- 12. Window size – Smaller window + more detail - more noise – Larger window + less noise - less detail W = 3 W = 20 Better results with adaptive window • T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment, ICRA 1991. • D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. IJCV, July 1998 Effect of window size
- 13. Stereo results – Data from University of Tsukuba – Similar results on other images without ground truth Ground truth Scene
- 14. Results with window search Window-based matching (best window size) Ground truth
- 15. Better methods exist... Graph cuts-based method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision 1999. Ground truth For the latest and greatest: http://www.middlebury.edu/stereo/
- 16. Stereo as energy minimization • What defines a good stereo correspondence? 1. Match quality • Want each pixel to find a good match in the other image 2. Smoothness • If two pixels are adjacent, they should (usually) move about the same amount
- 17. Stereo as energy minimization • Find disparity map d that minimizes an energy function • Simple pixel / window matching SSD distance between windows I(x, y) and J(x + d(x,y), y) =
- 18. Stereo as energy minimization y = 141 C(x, y, d); the disparity space image (DSI) x d
- 19. Stereo as energy minimization y = 141 x d Simple pixel / window matching: choose the minimum of each column in the DSI independently:
- 20. Greedy selection of best match
- 21. Stereo as energy minimization • Better objective function { { match cost smoothness cost Want each pixel to find a good match in the other image Adjacent pixels should (usually) move about the same amount
- 22. Stereo as energy minimization match cost: smoothness cost: 4-connected neighborhood 8-connected neighborhood : set of neighboring pixels
- 23. Smoothness cost “Potts model” L1 distance How do we choose V?
- 24. Smoothness cost • If λ = infinity, then we only consider smoothness • Optimal solution is a surface of constant depth/disparity – Fronto-parallel surface • In practice, want to balance data term with smoothness term
- 25. Dynamic programming • Can minimize this independently per scanline using dynamic programming (DP)
- 26. Dynamic programming • Finds “smooth”, low-cost path through DPI from left to right • Visiting a node incurs its data cost, switching disparities from one column to the next also incurs a (smoothness) cost y = 141 x d
- 28. Dynamic programming • Can we apply this trick in 2D as well? • No: the shortest path trick only works to find a 1D path Slide credit: D. Huttenlocher
- 29. Stereo as a minimization problem • The 2D problem has many local minima – Gradient descent doesn’t work well • And a large search space – n x m image w/ k disparities has knm possible solutions – Finding the global minimum is NP-hard in general • Good approximations exist (e.g., graph cuts algorithms)
- 30. Questions?
- 31. Depth from disparity f x x’ baseline z C C’ X f
- 32. Real-time stereo • Used for robot navigation (and other tasks) – Several real-time stereo techniques have been developed (most based on simple discrete search) Nomad robot searches for meteorites in Antartica
- 33. • Camera calibration errors • Poor image resolution • Occlusions • Violations of brightness constancy (specular reflections) • Large motions • Low-contrast image regions Stereo reconstruction pipeline • Steps – Calibrate cameras – Rectify images – Compute disparity – Estimate depth What will cause errors?
- 34. Active stereo with structured light • Project “structured” light patterns onto the object – simplifies the correspondence problem – basis for active depth sensors, such as Kinect and iPhone X (using camera 2 camera 1 projector camera 1 projector Li Zhang’s one-shot stereo
- 35. Active stereo with structured light https://ios.gadgethacks.com/news/watch-iphone-xs-30k-ir-dots-scan-your-face-0180944/
- 36. Laser scanning • Optical triangulation – Project a single stripe of laser light – Scan it across the surface of the object – This is a very precise version of structured light scanning Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/
- 37. Laser scanned models The Digital Michelangelo Project, Levoy et al.
- 38. Laser scanned models The Digital Michelangelo Project, Levoy et al.
- 39. Laser scanned models The Digital Michelangelo Project, Levoy et al.
- 40. Laser scanned models The Digital Michelangelo Project, Levoy et al.
- 41. 3D Photography on your Desk http://www.vision.caltech.edu/bouguetj/ICCV98/
- 42. Questions?