Deep Video Object Segmentation - Xavier Giro - UPC Barcelona 2019
2 likes2,700 views
This lecture discusses techniques for video object segmentation including online learning on individual frames, mask propagation between frames, flow propagation, and using recurrent neural networks (RNNs). It introduces datasets like DAVIS and YouTube-VOS and benchmarks models on tasks like one-shot and unsupervised video object segmentation. RNN approaches are discussed that process frames sequentially using spatial RNNs on objects and spatiotemporal RNNs. The RVOS model is highlighted which uses an end-to-end spatiotemporal RNN for one-shot and unsupervised video object segmentation.
![@DocXavi
[http://pagines.uab.cat/mcv/]
Module 6 - Day 8 - Lecture 2
Deep Video
Object Segmentation
28th March 2019
Xavier Giro-i-Nieto
xavier.giro@upc.edu
Associate Professor
Universitat Politècnica de
Catalunya](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-1-320.jpg)
![Online learning (frame-based)
13
A neural network is fine-tuned with the provided mask for the first frame (online
learning). Each frame is processed separately
#OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.
"One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé]](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-13-320.jpg)
![Online learning (frame-based)
14
...but results are still very convincing.
#OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.
"One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé]
Frame-based processing introduces temporal inconsistencies...](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-14-320.jpg)
![15
What are the limitations of online learning (OL) ?
#OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.
"One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé]
Online learning (frame-based)
Qualitative evolution of the fine tuning:
Results at 10 seconds and 1 minute per sequence.](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-15-320.jpg)
![16
How is it possible to fine-tune a ConvNet with just a single frame ?
#OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.
"One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé]
Online learning (frame-based)](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-16-320.jpg)
![19
#MaskTrack Perazzi, Federico, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung.
"Learning video object segmentation from static images." CVPR 2017. [talk]
Mask Propagation
The ConvNet is trained to refine the previous mask to the current frame.](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-19-320.jpg)
![RNN
36
Tokmakov, Pavel, Karteek Alahari, and Cordelia Schmid. "Learning video object segmentation with visual memory." ICCV
2017. [talk]
Limitations
● Each instance is trained and segmented independently
● Optical flow depends on a network trained for another task: model is not end-to-end trainable](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-36-320.jpg)
![51
Deep Learning courses @ UPC TelecomBCN:
● MSc course [2017] [2018]
● BSc course [2018] [2019]
● 1st edition (2016)
● 2nd edition (2017)
● 3rd edition (2018)
● 4th edition (2019)
● 1st edition (2017)
● 2nd edition (2018)
● 3rd edition - NLP (2019)
Next edition: Autumn 2019 Registration open for 2019Registration open for 2019](https://image.slidesharecdn.com/mcv-m62019d8l2videoobjectsegmentation-190329071506/85/Deep-Video-Object-Segmentation-Xavier-Giro-UPC-Barcelona-2019-51-320.jpg)
Deep Video Object Segmentation - Xavier Giro - UPC Barcelona 2019
- 1. @DocXavi [http://pagines.uab.cat/mcv/] Module 6 - Day 8 - Lecture 2 Deep Video Object Segmentation 28th March 2019 Xavier Giro-i-Nieto xavier.giro@upc.edu Associate Professor Universitat Politècnica de Catalunya
- 3. Outline 3 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 4. Video Object Segmentation (VOS) Semi-supervised (“One-shot”) video object segmentation Unsupervised (“zero-shot”) video object segmentation VS
- 5. #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. Semi- supervised VOS One-shot VOS
- 6. #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. Un supervised VOS Zero-shot VOS
- 7. Outline 7 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 8. Datasets and Benchmarks 8 Perazzi, Federico, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. "A benchmark dataset and evaluation methodology for video object segmentation." CVPR 2016. DAVIS-2017 ● 90 training videos (train+val) ● 30 testing videos (test-dev set)
- 9. Datasets and Benchmarks 9 #DAVIS Perazzi, Federico, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. "A benchmark dataset and evaluation methodology for video object segmentation." CVPR 2016.
- 10. Datasets and Benchmarks 10 #YouTube-VOS Xu, Ning, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. "Youtube-vos: Sequence-to-sequence video object segmentation." ECCV 2018.
- 11. Datasets and Benchmarks 11 #YouTube-VOS Xu, Ning, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. "YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark." arXiv preprint arXiv:1809.03327 (2018).
- 12. Outline 12 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 13. Online learning (frame-based) 13 A neural network is fine-tuned with the provided mask for the first frame (online learning). Each frame is processed separately #OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. "One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé]
- 14. Online learning (frame-based) 14 ...but results are still very convincing. #OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. "One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé] Frame-based processing introduces temporal inconsistencies...
- 15. 15 What are the limitations of online learning (OL) ? #OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. "One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé] Online learning (frame-based) Qualitative evolution of the fine tuning: Results at 10 seconds and 1 minute per sequence.
- 16. 16 How is it possible to fine-tune a ConvNet with just a single frame ? #OSVOS Caelles, Sergi, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. "One-shot video object segmentation." CVPR 2017. [Talk by Laura Leal-Taixé] Online learning (frame-based)
- 17. Outline 17 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 19. 19 #MaskTrack Perazzi, Federico, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung. "Learning video object segmentation from static images." CVPR 2017. [talk] Mask Propagation The ConvNet is trained to refine the previous mask to the current frame.
- 20. Outline 20 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 21. 21 Jain, Suyog Dutt, Bo Xiong, and Kristen Grauman. "Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos." CVPR 2017. Flow Propagation
- 22. 22 Jain, Suyog Dutt, Bo Xiong, and Kristen Grauman. "Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos." CVPR 2017.
- 23. Outline 23 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 24. 24 #MaskRNN Hu, Yuan-Ting, Jia-Bin Huang, and Alexander Schwing. "MaskRNN: Instance level video object segmentation." NIPS 2017. Mask + Flow Propagation The masks of the N objects in the previous frame are warped with the optical flow. Each mask is fed separately into another NN that detects & segments instances.
- 25. 25 #MaskRNN Hu, Yuan-Ting, Jia-Bin Huang, and Alexander Schwing. "MaskRNN: Instance level video object segmentation." NIPS 2017. Mask + Flow Propagation Where is the RNN in the MaskRNN architecture ?
- 26. 26 Mask from previous frame is warped & concatenated with optical flow in two set ups: Two streams One stream Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 27. 27 Two streams One stream How could these architectures deal with multiple objects in a single pass ? Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 28. 28 Multiple object tracking is handled by adding more mask channels. Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 29. 29 Two streams One stream Which architecture do you think it will perform better ? Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 30. 30 Two streams One stream Which architecture do you think it will perform better ? Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 31. 31 #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019. Two streams One stream Which architecture would you use ? Mask + Flow Propagation
- 32. 32 Which architecture would you use ? “The lighter one stream network performs as well as a network with two streams. We will thus use the one stream architecture” “One stream network is more affordable to train and allows to easily add extra input channels, e.g. providing additional semantic information about objects.” One stream Mask + Flow Propagation #LucidTracker Khoreva, Anna, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. "Lucid data dreaming for object tracking." IJCV 2019.
- 33. Outline 33 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 35. RNN (ConvLSTM) 35 Limitations ● Each instance is trained and segmented independently ● Designed only for one-shot video object segmentation. #S2S Xu, Ning, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. "Youtube-vos: Sequence-to-sequence video object segmentation." ECCV 2018.
- 36. RNN 36 Tokmakov, Pavel, Karteek Alahari, and Cordelia Schmid. "Learning video object segmentation with visual memory." ICCV 2017. [talk] Limitations ● Each instance is trained and segmented independently ● Optical flow depends on a network trained for another task: model is not end-to-end trainable
- 37. RNN (Spatial + Temporal) 37 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. time (frame sequence) space (object sequence)
- 38. RNN (Spatial) 38 space (object sequence) Previous work #RSIS Salvador, Amaia, Miriam Bellver, Victor Campos, Manel Baradad, Ferran Marques, Jordi Torres, and Xavier Giro-i-Nieto. "Recurrent neural networks for semantic instance segmentation." arXiv preprint arXiv:1712.00617 (2017).
- 39. RNN (Spatial) 39 #RSIS Salvador, Amaia, Miriam Bellver, Victor Campos, Manel Baradad, Ferran Marques, Jordi Torres, and Xavier Giro-i-Nieto. "Recurrent neural networks for semantic instance segmentation." arXiv preprint arXiv:1712.00617 (2017). Previous work
- 40. RNN (Spatial + Temporal) 40 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. time (frame sequence) space (object sequence)
- 41. RNN (Spatial + Temporal) 41 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019.
- 42. RNN (Spatial + Temporal) 42 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. One-shot Quality vs Inference Time for the Semi-supervised (one-shot) task Speed values measured on a GPU K80 (*) and P100 (♱), otherwise obtained from YouTube-VOS paper..
- 43. RNN (Spatial + Temporal) 43 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. Why are techniques using online learning (OL) much slower than those that don’t ?
- 44. RNN (Spatial + Temporal) 44 #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019. RVOS can naturally solve both the semi-supervised (one-shot) & unsupervised (zero-shot) tasks: CNN time CNN One-shot RVOS CNN CNN CNN Zero-shot RVOS time
- 45. RNN (Spatial + Temporal) 45 CNN CNN CNN Zero-shot RVOS time In zero-shot RVOS, masks were not propagated because of their low quality (also for the first frame). How could this limitation be addressed ? Seen clases Unseen classes Jseen Junseen Fseen Funseen Semi-supervised 63.6 45.5 67.2 51.0 Unsupervised 44.7 21.2 45.0 23.9 MSc thesis
- 46. RNN (Spatial + Temporal) 46 An alternative semi-supervised signal is a language description of the object to segment, instead of a binary mask. How could RVOS be adapted to it ? MSc thesis Khoreva, Anna, Anna Rohrbach, and Bernt Schiele. "Video object segmentation with language referring expressions." ACCV 2018.
- 47. RNN (Spatial + Temporal) 47 An alternative task would be an interactive set up in which the user draws scribbles over the object to segment. How could RVOS be adapted to it ? MSc thesis “The interactive scenario assumes the user gives iterative refinement inputs to the algorithm, in our case in the form of a scribble, to segment the object of interest. Methods have to produce the segmentation mask for that object in all the frames of a video sequence taking into account all the user interactions.”
- 48. RNN (Spatial + Temporal) 48 We released the RVOS PyTorch source code today, so feel free to play with it (maybe even for your final M6 project deliverable ?). #RVOS Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019.
- 49. Outline 49 ● Motivation ● Datasets & Benchmarks ● Online learning (Frame-based) ● Mask propagation ● Flow Propagation ● RNN
- 50. 50 Questions ?
- 51. 51 Deep Learning courses @ UPC TelecomBCN: ● MSc course [2017] [2018] ● BSc course [2018] [2019] ● 1st edition (2016) ● 2nd edition (2017) ● 3rd edition (2018) ● 4th edition (2019) ● 1st edition (2017) ● 2nd edition (2018) ● 3rd edition - NLP (2019) Next edition: Autumn 2019 Registration open for 2019Registration open for 2019
- 52. 52 Deep Learning for Professionals @ UPC School Next edition starts November 2019. Sign up here.