Disparity Estimation Using A Color Segmentation V3
- 1. Disparity Estimation using a Color Segmentation Xin Wang Barcelona, 3rd Sep. 2009
- 2. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 3. Task and Problem Identification Stereo Correspondence Problem Given an element in the left image, we search for the corresponding element in the right image. Disparity is the displacement for pixel positions between two corresponding points in these images. Epipolar Rectification allows simplifying the search problem from 2D to 1D.
- 4. Task and Problem Identification Challenges Textureless Region : Matching in an untextured area becomes ambiguous, since there are many potential matching points of very similar intensity patterns. Depth Discontinuity Region : Pixels whose neighbouring disparities differ by more than a given value, usually appear at the boundaries of different scene objects. Occluded Region : Zones that are occluded in the matching image.
- 5. Task and Problem Identification Quality Measure : Reconstruction : To reconstruct the reference image in a back-ward way, is using reference image’s disparity map and the other view image for reconstruction. Reconstruction Image PSNR (Peak Signal to Noise Ration) : , where L (ori) R (ori) Disparity (Left) Disparity (Right)
- 6. Task and Problem Identification Occlusion mask is made from performing L-R check to groundtruth. PSNR is Calculated in non-occluded region.
- 7. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 8. Algorithmic Overview Block Diagram of Algorithm Initial Disparity Estimation Color Segmentation to extract the label Polynomial Model based Disparity Representation in Each Color Segment (use label information) Disparity Improvement By Region Merging Iterations Final Disparity Map
- 9. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 10. Initial Disparity Estimation Window-based Local Matching : Matching Score : is a quantitative similarity measurement for two different pixels. It is the criterion for the solution of correspondence problem. Winner-take-all : disparity associated with the minimum cost value is selected at each pixel.
- 11. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 12. Segmentation-based Approach Approach: Segmentation-based approaches divide one or sometimes both images into non-overlapping regions of homogeneous color. Assumptions: 1, Inside a segment of homogeneous colour the disparity values are expected to follow some particular smooth disparity model (constant disparity, planar model, etc) 2, Disparity discontinuities are assumed to coincide with the boundaries of those color regions.
- 13. Segmentation-based Approach Polynomial Models for Each Segment Order 0 Polynomial Model : Order 1 Polynomial Model : Order 2 Polynomial Model : (planar parallel to the camera front plane) (planar model) (parabolic curved surface)
- 14. Segmentation-based Approach Model Parameters Estimation We use initial disparity as input data Unknown Parameters: Order 0 : 1 parameter - > 1 point Order 1 : 3 parameters -> 3 points Order 2 : 6 parameters -> 6 points Proposed Approaches for Model Fitting: Least Squares Random Sample and Consensus (RANSAC)
- 15. Segmentation-based Approach Least Squares Over-determined Problem : The number of points in each region is much higher than the number of equations needed to compute the analytical solution. Best Fit for Observed Data : Sum of squared residuals has its least value. (but sensitive to outliers) min( )
- 16. Segmentation-based Approach RANSAC Non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability. Assumption : A set of data is that the data consists of inliers and the data's contribution follows a certain set of model parameters, while the outliers do not fit the model. Algorithm: Hypothesize-and-Test framework HYPOTHESIZE: Minimal Sample Set (MSS) is randomly selected from initial input data. TEST: Check the entire dataset with the estimated model from MSS. Data are consistent with the model are called consensus set (CS).
- 17. Segmentation-based Approach RANSAC in Polynomial Model Fitting Polynomial Order 0 : Median filter Polynomial Order 1 and 2: 1. Select MSS : Randomly select 3 or 6 points from initial disparity points 2. Fit the Model by Solving Linear Equations System: 3. Select Inliers to Form the CS: Distance calculation -> D = d i – f (x i ,y i ,P) if below the distance threshold 4. Choose the best CS to Output Fitted Model: CS with most inliers
- 18. Segmentation-based Approach Comparison between RANSAC & LS LS : Analytical solution, non-iterative solution. Less computation cost. Very sensitive to outliers. RANSAC: Robust to outliers, non-deterministic algorithm. Accuracy will increase with increasing the iterations and strictness in thresholds. More computation cost and the model may subject to noise. PSNR Comparison Test: (Teddy 559 regions, Cones 553 regions, Venus 398 regions)
- 19. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 20. Disparity Refinement via Merging Problems : In some regions, due to unproper-segmentation, occlusion, noise, lack of enough inliers, may result in incorrect model estimation. Solution : Merge adjacent “good regions” to re-fit the model. For lack of inliers region : enlarge the area , includes more inliers For occlusion area : get the support from neighbouring good region For small region : large distance separated points, more easy to fit a correct model
- 21. Disparity Refinement via Merging Merging Block Diagram :
- 22. Disparity Refinement via Merging Region Similarity Measure Criteria Similarity in Model’s Parameters Number of inliers PSNR of Reconstructed Region (Euclidean Distance Based)
- 23. Disparity Refinement via Merging Implementation for similarity measure: Distance similarity in region’s parameters + PSNR criterion Distance similarity matrix will be calculated: Due to distance similarity based on region’s parameters can not always be true, so I use PSNR criterion for actual merging step to secure merging process will always improve the disparity quality
- 24. Outline for the presentation Task and Problem Identification Algorithmic Overview Segmentation-based Stereo Matching Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging Experiment Result and Analysis Conclusions and Future Work Questions
- 25. Experiment Results Polynomial Model Fitting Compare with initial disparity, usually no improvement Analysis : RANSAC works not well, due to initial color segmentation fails to meet the two assumptions, and some regions lack of enough inliers. Parameters : Teddy (559 regions) , Cones (553 regions), Venus (398 regions) RANSAC iterations:200, inlier threshold:0.99 , distance threshold:0.5
- 26. Experiment Results Mixed Order Polynomial Model Fitting Program runs on a 3GHz processor
- 27. Experiment Results Mixed Order Polynomial Model Fitting Red: Order 0, Green: Order 1, Blue: Order 2, Black: No improvement
- 28. Experiment Results Polynomial order 2 Model Region Merging Teddy (559 regions) , Cones (553 regions), Venus (398 regions)
- 29. Experiment Results Polynomial order 2 Teddy case
- 30. Experiment Results Polynomial order 1 model results: Reconstruction using initial Disparity Reconstruction using merged disparity, 160 rounds
- 31. Conclusions and Future Work Conclusions: LS is more sensitive to outliers, RANSAC is more robust Sometimes RANSAC will fail to estimate correct parameters, due to the segmented region’s size, noise etc. Small regions may not have enough inliers, region merging could help Bad estimated regions could get support from neighbouring regions in merging process Future work More accurate and computationally efficient region similarity measure should be further studied Try to do merging for all the possibilities for adjacent regions. Convert Matlab code to C++ version
- 32. Acknowledgement Prof. Josep Ramon Morros i Rubio Albert (Technical support) All Professors and friends in TSC who supported me before.
- 33. Questions Thank you very much ! Questions?
- 34. Experiment Results Example of Region Merging Improvement (Teddy, order2) A, improvement point B, how label changes C, disparity refinement
- 35. Experiment Results Drop point in the evolution curve (214 & 216region)
- 36. Experiment Results Order 1 Merging -> Teddy
- 37. Experiment Results Poor Modeled Region in order2 Teddy
- 38. Disparity & Depth Disparity is proportional inversely to the depth
- 39. Experiment Results Local Matching How the local window size affect local matching Too small window can not catch enough intensity variation to give the correct disparity in less-textured regions, too large window can not capture well small details Teddy Image Cones Image Venus Image
- 40. Matching Score Matching Score: Matching measures such as SSD and SAD are strictly assuming the constant color con-straint. So if the computed area fails to meet this constraint, SSD and SAD measures may not be robust. While the other matching scores like gradient-based measure is more robust to changes in camera gain. So we could expect to combine the sum of absolute differences and a gradient based measure together to define the total matching score in order to increase the robustness. The weight w which balance the CSAD and CGRAD portions. It is determined by maxi-mizing the number of reliable corresponding pixel pairs that are filtered by performing cross-checking test.