Optical Flow with Semantic Segmentation and Localized Layers
Download as PPTX, PDF0 likes91 views
This document summarizes a paper on optical flow estimation using semantic segmentation. The paper proposes: 1. Using semantic segmentation to provide object boundaries and class types to inform motion models. 2. Modeling motion with localized layers instead of globally, to better represent complex scene motions and boundaries. 3. Defining three object classes (things, planes, stuff) with different motion models and optimizing a cost function combining data, motion, time, layer, and space terms. Experiments on YouTube videos and KITTI 2015 data show improved optical flow over methods without semantic guidance. The code has been released to experiment further integrating segmentation and optical flow.
![Proposed Method
1. Segmentation with DeepLab [1] and objects class matching
2. Compute Initial flow field with Discrete Flow [2].
3. Initialization and Optimization
4. Composition of the Flow Field
1. L. Chen et. al. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR,
abs/1412.7062, 2014.
2. M. Menze et. al. Discrete optimization for optical flow. In German Conference on Pattern Recognition (GCPR), volume
9358, pages 16–28. Springer International Publishing, 2015.](https://image.slidesharecdn.com/cmp717-opticalflowwithsemanticsegmentation-200507120954/85/Optical-Flow-with-Semantic-Segmentation-and-Localized-Layers-10-320.jpg)
Optical Flow with Semantic Segmentation and Localized Layers
- 1. Arif Akar Seval Çapraz Presenters Optical Flow with Semantic Segmentation and Localized Layers Laura Sevilla-Lara, Deqing Sun, Varun Jampani, Michael J. Black Max Plank Institute for Intelligent Systems, Harvard University, Nvidia Corporation
- 2. Outline ● Introduction ○ Problem Statement, Key Assumptions ● Semantic Optical Flow ○ Semantic Segmentation, Localized Layers, Model and Methods, ● Experiments ○ Natural Youtube Videos, KITTI 2015 ● Conclusion ● Project Stage
- 3. Q: What is the aim of Optical Flow Research? ● We are interested in finding the movement of scene objects from time-varying images (videos). ● Lots of uses ○ Track object behavior ○ Correct for camera jitter (stabilization) ○ Align images (mosaics) ○ 3D shape reconstruction ○ Human Action Recognition Slide Credit: S. Narasimhan
- 4. Problem Statement – Optical Flow ● How to estimate pixel motion from image H to image I? • Find pixel correspondences • Given a pixel in H, look for nearby pixels of the same color in I
- 5. Key Assumptions ○ Color Constancy: A point in H looks “the same” in image I ■ For grayscale images, this is brightness constancy ○ Small Motion: Points do not move very far ■ Reduce the resolution to solve problems due to this assumption -> Use Pyramid Representation!
- 6. Semantic Optical Flow What can be improved with existing Optical Flow approaches? ○ Generic, spatially homogenous assumptions about spatial structure of the flow ○ Different objects move differently ○ Handling complex scene motion ○ Handling discontinuities at object boundaries
- 7. 1. Use Semantic Segmentation ○ Provide information on object boundaries ○ Object class type determine movement type ■ Things, Planes and Stuff ○ Provide information on relative local depth orderings Semantic Optical Flow
- 8. 2. Localized Layer Models ○ To handle complex scene motions and motion boundaries ○ Not globally modeled, localized layers ○ Better foreground- background representation Semantic Optical Flow
- 9. a. Initial Segmentation b. Resulting Segmentation c. Discrete Flow d. Resulting Semantic Flow Semantic Optical Flow
- 10. Proposed Method 1. Segmentation with DeepLab [1] and objects class matching 2. Compute Initial flow field with Discrete Flow [2]. 3. Initialization and Optimization 4. Composition of the Flow Field 1. L. Chen et. al. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062, 2014. 2. M. Menze et. al. Discrete optimization for optical flow. In German Conference on Pattern Recognition (GCPR), volume 9358, pages 16–28. Springer International Publishing, 2015.
- 11. Model and Methods ● Three classes of Objects: 1. Things ○ Defined spatial extent, rigid or non-rigid, move independently, typically foreground 2. Planes ○ Broad spatial extent, roughly planar, typically background 3. Stuff ○ Buildings, vegetation, unknown classes assigned
- 12. Model and Methods ● Three classes of Objects: 1. Motion of Things ○ Modeled as affine transformation + smooth deformation from affine 2. Motion of Planes ○ Modeled as homographies, use RANSAC to estimate homography parameters hi 3. Motion of Stuff ○ No specific motion model, set each region to initial flow
- 15. ● Data Term: imposes appearance constancy when pixels are visible at the same layer Models and Methods
- 16. ● Motion Term: encodes two assumptions;1. neighbor pixels should move together if they belong to same layer k. 2. pixels from each layers should share a global motion model where changes over time and depends on object class. Models and Methods
- 17. ● Time Term: encourages corresponding pixels over time to have the same layer label. Models and Methods
- 18. ● Layer Term: is a coupling term that enforces similarity between layer segmentation and semantic segmentation Models and Methods
- 19. ● Space Term: encourages spatial contiguity of layer segmentation. Models and Methods
- 20. Experiments 1. Natural Youtube Videos (containing objects of Pascal-VOC classes) ○ No ground truth for quantitative analysis ○ Only qualitative results provided
- 21. 2. KITTI 2015 Overall percentage of outliers compared Experiments
- 22. Experiments 2. KITTI 2015 – Online Competition Top results among all monocular methods. http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=flow
- 23. Conclusion ● Using semantic segmentation improves optical flow estimation ● Different motion models defined and focused on motion of things ● A key insight is that a detected object region is likely to contain at most two motions and the object is likely to be in front
- 24. Project Stage ● Code of the paper was released (Matlab), DEMO script provided ● We modified the code so that we are able to run the code for whole KITTI 2015 data set (~4.5 hours) ● We experiment on code to see how motion models can be improved ● ….. ● ….. ● Ultimate Goal: Segmentation can help OF, can OF help segmentation too? These two processes can be integrated together and can both converge to outstanding results.