Lecture21
- 1. Robert Collins CSE486, Penn State Lecture 21: Stereo Reconstruction
- 2. Robert Collins CSE486, Penn State Steps to General Stereo find 8 or more initial point matches (somehow)
- 3. Robert Collins CSE486, Penn State Steps to General Stereo Compute F matrix using 8-point algorithm -0.00310695 -0.0025646 2.96584 F= -0.028094 -0.00771621 56.3813 13.1905 -29.2007 -9999.79
- 4. Robert Collins CSE486, Penn State Steps to General Stereo Infer epipolar geometry (epipoles, epipolar lines) from F. -0.00310695 -0.0025646 2.96584 F= -0.028094 -0.00771621 56.3813 13.1905 -29.2007 -9999.79
- 5. Robert Collins CSE486, Penn State Steps to General Stereo Rectify images to get simple scanline stereo pair. rectify
- 6. Robert Collins CSE486, Penn State Steps to General Stereo Compute disparity map (correspondence matching) sparse interpolated
- 7. Robert Collins CSE486, Penn State Steps to General Stereo Compute 3D surface geometry from disparity map disparity map in pixel coords Views of texture mapped depth surface
- 8. Robert Collins CSE486, Penn State Reconstruction Even though we have a dense disparity map, when talking about recovering 3D scene structure, we will consider it to just be a set of point matches. u point match: (u,v) in left image v matches (u-d,v) in right image disparity(u,v)=d
- 9. Robert Collins CSE486, Penn State Stereo Reconstruction Given point correspondences, how to compute 3D point positions using triangulation. Results depend on how calibrated the system is: 1) Intrinsic and extrinsic parameters known Can compute metric 3D geometry 2) Only intrinsic parameters known Unknown scale factor 3) Neither intrinsic nor extrinsic known Recover structure up to an unknown projective transformation of the scene
- 10. Robert Collins CSE486, Penn State Fully Calibrated Stereo Known intrinsics -- can compute viewing rays in camera coordinate system Know extrinsics -- know how rays from both cameras are positioned in 3D space Reconstruction: triangulation of viewing rays
- 11. Robert Collins CSE486, Penn State Calibrated Triangulation P Pl RTPr pl pr Ol el er Or T ideally, P is the point of intersection of two 3D rays: ray through Ol with direction Pl ray through Or with direction RTPr
- 12. Robert Collins CSE486, Penn State Triangulation with Noise Unfortunately, these rays typically don’t intersect due to noise in point locations and calibration params P? RTPr Pl Ol Or T
- 13. Robert Collins CSE486, Penn State Triangulation with Noise Unfortunately, these rays typically don’t intersect due to noise in point locations and calibration params P RTPr Pl Ol Or T Solution: Choose P as the “pseudo-intersection point”. This is point that minimizes the sum of squared distance to both rays. (The SSD is 0 if the rays exactly intersect)
- 14. Robert Collins CSE486, Penn State Solution from T&V Book P is midpoint of the segment perpendicular to Pl and RTPr Let w = Pl x RTPr (this is perpendicular to both) w P RTPr Pl Ol Or T Introducing three unknown scale factors a,b,c we note we can write down the equation of a “circuit”
- 15. Robert Collins CSE486, Penn State Solution from T&V Book Writing vector “circuit diagram” with unknowns a,b,c a Pl + c (Pl X RTPr) - b RTPr = T c w - b RTPr a Pl Ol Or T note: this is three linear equations in three unknowns a,b,c => can solve for a,b,c
- 16. Robert Collins CSE486, Penn State Solution from T&V Book After finding a,b,c , solve for midpoint of line segment between points Ol + a Pl and Ol + T + b RT Pr b RTPr a Pl Ol Or T
- 17. Robert Collins CSE486, Penn State Alternate Solution I prefer an alternate solution based on using least squares to solve for an unknown point P that minimizes SSD to viewing rays. Why? it generalizes readily to N cameras.
- 18. Robert Collins CSE486, Penn State Stereo Reconstruction Given point correspondences, how to compute 3D point positions using triangulation. Results depend on how calibrated the system is: 1) Intrinsic and extrinsic parameters known Can compute metric 3D geometry 2) Only intrinsic parameters known Unknown scale factor 3) Neither intrinsic nor extrinsic known Recover structure up to an unknown projective transformation of the scene
- 19. Robert Collins CSE486, Penn State Only Intrinsic Params Known General outline of solution (see book for details) Use knowledge that E = R S to solve for R and T, then use previous triangulation method. Note: since E is only defined up to a scale factor, we can only determine the direction of T, not its length. So... 3D reconstruction will have an unknown scale.
- 20. Robert Collins CSE486, Penn State Only Intrinsic Params Known Using E to solve for extrinsic params R and T E = R S where elements of S are functions of T Then ET E = ST RT R S = ST S (because RTR = I) Thus ET E is only a function of T. Solve for elements of T assuming it is a unit vector. After determining T, plug back into E = R S to determine R.
- 21. Robert Collins CSE486, Penn State Only Intrinsic Params Known Unfortunately, four different solutions for (R,T) are possible (due to choice of sign of E, and choice of sign of T when solving for it). However, only one choice will give consistent solutions when used for triangulation, where consistent means reconstructed points are in front of the cameras (positive Z coordinates). So, check all four solutions, choose the correct one, and you are done.
- 22. Robert Collins CSE486, Penn State Stereo Reconstruction Given point correspondences, how to compute 3D point positions using triangulation. Results depend on how calibrated the system is: 1) Intrinsic and extrinsic parameters known Can compute metric 3D geometry 2) Only intrinsic parameters known Unknown scale factor 3) Neither intrinsic nor extrinsic known Recover structure up to an unknown projective transformation of the scene
- 23. Robert Collins Stereo when “Nothing” is Known CSE486, Penn State What if we don’t known intrinsic nor extrinsic params? (we just look at two pictures of the scene with no prior information) Can we recover any 3D information? It wasn’t clear that you could, but then in 1992... •Faugeras “What can be seen in three dimensions from an uncalibrated stereo rig”, ECCV 1992 •Hartley et.al., “Stereo from Uncalibrated Cameras”, CVPR 1992 •Mohr et.al., “Relative 3D Reconstruction using Multiple Uncalibrated Images, LIFIA technical report, 1992
- 24. Robert Collins Stereo when “Nothing” is Known CSE486, Penn State Of course, we don’t really know “nothing”. We know point correspondences, and because of that, we can compute the fundamental matrix F Sketch of solution: use knowledge of F and use 5 points in the scene to define an arbitrary projective coordinate system. These points will have coordinates: (1 0 0 0) (0 1 0 0) (0 0 1 0) (0 0 0 1) (1 1 1 1) Result: You can recover 3D locations of other points with respect to that projective coordinate system.
- 25. Robert Collins Stereo when “Nothing” is Known CSE486, Penn State Result: You can recover 3D locations of other points with respect to that projective coordinate system. Why would that be practical? Often, you can then use other information to determine how your arbitrary projective coordinate system relates to the “real” Euclidean scene coordinates system. e.g. use prior knowledge of lengths and angles of some items in the world (like a house)