Contour-Constrained Superpixels for Image and Video Processing
1 like468 views
The document presents contour-constrained superpixels (CCS) as an efficient method for image and video processing, emphasizing its hierarchical refinement and contour adherence. It discusses the algorithm's capabilities in superpixel labeling, temporal consistency for video applications, and experimental results showing superior performance compared to traditional methods. CCS can significantly enhance applications such as object segmentation and saliency detection in computer vision tasks.
Contour-Constrained Superpixels for Image and Video Processing
- 1. Contour-Constrained Superpixels for Image and Video Processing Se-Ho Lee, Won-Dong Jang, and Chang-Su Kim Korea University CVPR 2017
- 2. Overview • Introduction • Contour-constrained superpixels (CCS) • Superpixel labeling • Contour constraint • Hierarchical superpixel refinement • Temporal superpixels • Temporal superpixel labeling • Merging, splitting, and relabeling • Experimental results • Conclusions
- 3. • Superpixel • Over-segment input images into a set of meaningful regions • Can reduce the number of image primitives or units greatly • Superpixels should adhere to object contours Introduction
- 4. Superpixel • Applications • Preprocessing technique for many vision applications • ex) Saliency detection and image segmentation
- 5. Superpixel • Conventional methods • K-means based methods • Assign each pixel to the nearest cluster and update the cluster center iteratively • SLIC (TPAMI’12), LSC (CVPR’15), and MSLIC (CVPR’16) • Coarse-to-fine methods • Change the superpixel label of boundary regions iteratively based on the cost function • From block-level to pixel-level • SEEDS (IJCV’15) and Yao et al. (CVPR’15)
- 6. Contour-Constrained Superpixels • Contributions • Introduction of the contour constraint • Extension of the proposed algorithm for video processing • Remarkable performance achievement and improvement of many computer vision algorithms
- 7. Contour-Constrained Superpixels • Coarse-to-fine refinement • From block-level to pixel-level • Hierarchical block structure
- 8. Contour-Constrained Superpixels • Superpixel labeling • Hierarchical block structure • From block to pixel levels • Divide only inhomogeneous regions • Can maintain relatively regular and compact shape
- 9. Contour-Constrained Superpixels • Superpixel labeling • Change the label of region (block or pixel) 𝑅𝑖 from 𝑙 𝑅𝑖 to 𝑙 𝑅𝑗 of neighboring region 𝑅𝑗 ∈ 𝒩𝑅 𝑖 • Change the label of region by minimizing the cost function • 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗 • To preserve the topology of superpixels • Check the boundary region 𝑅𝑖 is simple point Initial Level 1 Level 2 Level 3 Level 4
- 10. Superpixel Labeling • Feature distance from superpixel centroid • 𝐸D 𝑖, 𝑗 = 𝐜 𝑅𝑖 − 𝐜 𝑆𝑙 𝑅 𝑗 2 + 𝐩 𝑅𝑖 − 𝐩 𝑆𝑙 𝑅 𝑗 2 • Color distance term makes the color of each superpixel to be regular • Spatial distance term imposes the superpixels to be distributed compactly Color distance Spatial distance
- 11. Superpixel Labeling • Boundary length cost • 𝐸L 𝑖, 𝑗 = 𝜆 𝑅𝑖, 𝑙 𝑅𝑗 − 𝜆 𝑅𝑖, 𝑙 𝑅𝑖 • 𝜆 𝑅𝑖, 𝑘 : total number of boundary regions when the superpixel label of 𝑅𝑖 is 𝑘 • Minimize the boundary length of superpixels • Make the superpixels distributed compactly
- 12. Superpixel Labeling • Inter-region color cost • 𝐸I 𝑖, 𝑗 = max 0, 𝐜 𝑅𝑖 − 𝐜 𝑅𝑗 2 − 𝜅 𝑆𝑙 𝑅 𝑗 • Neighboring regions with dissimilar color information → Assign different superpixel labels • Internal difference • Measure the amount of texture information of superpixels • 𝜅 𝑆𝑙 𝑅 𝑖 = max 𝑅 𝑚,𝑅 𝑛∈𝑆𝑙 𝑅 𝑖 , 𝑅 𝑛∈𝒩 𝑅 𝑚 𝐜 𝑅 𝑚 − 𝐜 𝑅 𝑛 2 • Maximum color difference between neighboring regions within 𝑆𝑙 𝑅 𝑖
- 13. Superpixel Labeling • Contour constraint 𝐸C 𝑖, 𝑗 • Amplify the cost function when there is an object contour between two regions • Use holistically-nested edge detection (HED)* • Deep learning based edge detection method 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗 * S. Xie and Z. Tu, Holistically-nested edge detection. In ICCV, pages 1395-1403, 2015.
- 14. Contour Constraint • Contour pattern set extraction • Use 200 training images in the BSDS500 dataset • Denote patch centered at each contour pixel as a contour pattern • Select top 1,000 frequently occurring patterns
- 15. Contour Constraint • Contour pattern set • Only consider the patterns which divide the patch into two regions • 1,000 patterns cover 90.5% of the patches in the training dataset
- 16. Contour Constraint • Contour pattern matching • Get thin binary contour map • Perform non-maximum suppression and then do thresholding • Compute Hamming distances from the contour patterns and select the best matching pattern
- 17. Contour Constraint • Contour probability 𝜙 𝑢, 𝑣 • Difficult to estimate the existence of object contours only considering the thin contour map • Measure the proportion of patches whose matching patterns separate 𝑢 from 𝑣 • Consider patches containing both pixels 𝑢 and 𝑣
- 18. Contour Constraint • Determining contour constraint 𝐸C 𝑖, 𝑗 • Contour probability 𝜓 𝑅𝑖, 𝑅𝑗 between regions • Find the maximum contour probability between the pixels in 𝑅𝑖 and 𝑅𝑗 • 𝜓 𝑅𝑖, 𝑅𝑗 = max 𝑢∈𝑅 𝑖,𝑣∈𝑅 𝑗 𝜙 𝑢, 𝑣 • 𝐸C 𝑖, 𝑗 = exp 𝛽 × 𝜓 𝑅𝑖, 𝑅𝑗
- 19. Hierarchical Superpixel Refinement • Hierarchical block structure • Divide only inhomogeneous regions into four blocks • Dissimilarity function • 𝜃 𝑅𝑖 = max 𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢 𝐜 𝑢 − 𝐜 𝑣 2 + exp 𝛽 × max 𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢 𝜙 𝑢, 𝑣 • Divide the region 𝑅𝑖 if 𝜃 𝑅𝑖 > 𝜏div
- 21. Temporal Superpixels • Superpixels for videos • Get temporally consistent superpixels Input SLIC CCS
- 22. Temporal Superpixels • Initialization • To make temporally consistent superpixels • Estimate optical flows from 𝐼 𝑡−1 to 𝐼 𝑡 • Transfer the label of each superpixel by employing the average optical flow of the superpixel • Do not assign any labels to occluded or disoccluded pixels • Occluded pixel: a pixel mapped from multiple superpixels • Disoccluded pixel: a pixel mapped from no superpixel
- 23. Temporal Superpixels • Temporal superpixel labeling • Performed at the pixel level only • Based on the energy function • 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸T 𝑖, 𝑗 • 𝐸T 𝑖, 𝑗 : temporal contour constraint
- 24. Temporal Superpixel Labeling • Temporal Contour constraint • Make superpixels temporally consistent • Make superpixels compatible with image contours • 𝐸T 𝑖, 𝑗, 𝑡 = 𝐸C 𝑖, 𝑗, 𝑡 × 𝜌 𝑖, 𝑗, 𝑡 • Relaxation factor 𝜌 𝑖, 𝑗, 𝑡 • 𝜌 𝑖, 𝑗, 𝑡 = ൞ 1 1+exp −𝜁×ℎ 𝑅𝑖 𝑡 , if 𝑙 𝑅𝑗 𝑡 ∈ ℒ 𝑖 𝑡 1, otherwise • ℒ 𝑖 𝑡 is the set of labels that are mapped to 𝑅𝑖 𝑡 from 𝐼 𝑡−1 • Relax 𝐸T 𝑖, 𝑗, 𝑡 when 𝑅𝑖 𝑡 does not contain object contour
- 25. Merging, Splitting, and Relabeling • Merging and splitting • Prevent irregular superpixel size • When Τ𝐴 𝑘 𝑡 ҧ𝐴 is larger than 𝜏spl → split • Consider the biggest eigenvector of the spatial distribution • When Τ𝐴 𝑘 𝑡 ҧ𝐴 is smaller than 𝜏mer → merge • Merge to the nearest superpixel
- 26. Merging, Splitting, and Relabeling • Relabeling • Avoid incorrect labeling • Because of occlusion or illumination variation • Define color consistency 𝐶 𝑘 • 𝐶 𝑘 = 𝐜1:𝑡−1 𝑆 𝑘 − 𝐜 𝑡 𝑆 𝑘 2 • If 𝐶 𝑘 is larger than 𝜏rel → relabel
- 28. Experimental Results • Superpixel results • On BSDS500 dataset • CCS-wo-CC denotes proposed algorithm without the contour constraint ASA ↑ BR ↑ UE ↓
- 29. Experimental Results • Temporal superpixel results • On SegTrack dataset SA2D ↑ BR2D ↑ UE2D ↓ SA3D ↑ BR3D ↑ UE3D ↓
- 30. Experimental Results • Visual comparison of superpixel methods Input SEEDS LSC Proposed
- 31. Experimental Results • Visual comparison of temporal superpixel methods TCS TSP Proposed
- 32. Experimental Results • Run-times comparison • Superpixel algorithms • Temporal superpixel algorithms Turbopixels SLIC ERS LSC SEEDS MSLIC CCS Times(s) 8.09 0.26 1.52 0.34 0.06 0.36 0.97 sGBH SLIC TCS TSP CCS Times(s) 5.71 0.08 7.83 2.39 1.70
- 33. Experimental Results • Application to video object segmentation • To superpixel-based video object segmentation method* • Use CCS instead of SLIC as a preprocessing • Intersection over union (IoU) is increased from 0.532 to 0.571 * W.-D. Jang and C.-S. Kim, Semi-supervised video object segmentation using multiple random walkers. In BMVC, 2016.
- 34. Experimental Results • Application to video saliency detection • Postprocessing • Average the saliency values of the pixels in all frames, constituting each superpixel • Applied to HS* and DHSNet** * Q. Yan, L. Xu, J. Shi, and J. Jia, Hierarchical saliency detection. In CVPR, 2013. ** N. Liu and J. Han, DHSNet: Deep hierarchical saliency network for salient object detection. In CVPR, 2016.
- 36. Conclusions • Contour-constrained superpixels (CCS) • Perform hierarchical refinement from block to pixel levels • Based on the contour constraint • Temporal superpixels • CCS algorithm for video processing • Use optical flow to obtain temporally consistent superpixels • Experimental results • CCS outperforms the state-of-the-art superpixel methods • Can be applied to object segmentation and saliency detection
- 37. Thank You Q & A