Image segmentation with deep learning
- 1. Image Segmentation with Deep Learning Antonio Rueda-Toicen and Imran Kocabiyik Berlin Computer Vision Group December 2020
- 3. Agenda ● Image segmentation ■ Semantic segmentation ● Fully convolutional networks, U-net ■ Instance segmentation ● Mask R-CNN ■ Panoptic segmentation ● Feature Pyramid Networks ○ Public datasets ■ COCO ■ Google Open Images ○ Implementations: Detectron2, Fast.ai
- 4. Classification, detection, and segmentation Classification refers to image-wide labels Detection refers to localization of bounding boxes with labels Segmentation refers to pixel-wise localization of the labels
- 5. Goals of supervised image segmentation Given an input image we wish to obtain: 1. A class label associated to each individual pixel in the image. This is also called pixel-wise localization. 3. The probability score associated with each class label
- 6. Applications of image segmentation Link
- 7. Applications of image segmentation http://withoutbg.com/
- 8. Applications of image segmentation
- 9. Applications of image segmentation
- 10. Applications of image segmentation
- 11. Applications of image segmentation https://www.segmentive.ai/
- 12. Segmentation as pixel-wise localization
- 13. Instance segmentation requires object detection
- 14. Panoptic segmentation https://arxiv.org/pdf/1801.00868.pdf Explore it in the detectron2 inference notebook
- 15. “Fully Convolutional” networks draw segmentation masks All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most classifiers, we use the local info of the pixel neighborhood
- 16. What is a convolution filter? https://setosa.io/ev/image-kernels/
- 17. What is a convolution filter? https://setosa.io/ev/image-kernels/
- 18. What is a convolution filter? Convolution of 3x3 and stride = 1 without padding Effect: the output loses one pixel on each dimension
- 19. What is a convolution filter? Convolution of 3x3 and stride = 1 with zero padding Effect: the output preserves original image size
- 20. What is a convolution filter? Convolution of 3x3 and stride = 2 with zero padding Effect: the output is downsampled to about half its size
- 21. “Fully Convolutional” networks draw segmentation masks All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most classifiers, we use the local info of the pixel neighborhood
- 22. U-net for semantic segmentation All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most classifiers, we need this fully convolutional architecture to label images pixel by pixel preserving their local info
- 23. U-net for semantic segmentation All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most classifiers, we need this fully convolutional architecture to label images pixel by pixel preserving their local info
- 24. Image pyramids
- 25. Image Pyramids in Feature Proposal Networks (FPNs)
- 26. Convolutional networks implement “pyramids” The deeper we go into the network, the more semantic value is compressed in lower x,y dimensions
- 27. Resnets
- 28. Nearest neighbor interpolation Resnets in feature pyramid networks
- 29. 1x1 convolution Resnets in feature pyramid networks
- 31. Image Pyramids in Feature Proposal Networks (FPNs)
- 32. Mask R-CNN
- 34. The Google Open Images Dataset
- 40. Detectron2 detectron2/MODEL_ZOO.md at master · facebookresearch/detectron2 · GitHub Inference (Colab notebook) Training (Colab notebook) Generating validation set plots
- 41. Panoptic segmentation with feature pyramid network (FPN-50)
- 45. Objective Example Case: Image Matting
- 46. ⊕ Using a Unet Example Case: Image Matting
- 47. Matting algorithm: Example Case: Image Matting
- 48. instance segment Using trimap or instance segments? Example Case: Image Matting ⊕ or ⊕ ?
- 49. Results Example Case: Image Matting Photo: Ayo Ogunseinde https://unsplash.com/photos/THIs-cpyebg
- 50. Results Example Case: Image Matting Photo: Eugen Proskouriakov https://unsplash.com/photos/C-gvAA8q3Tc
- 51. Results Example Case: Image Matting Photo: Mathieu Renier https://unsplash.com/photos/4WBvCqeMaDE
- 52. Results Example Case: Image Matting Photo: Gulyás Bianka https://unsplash.com/photos/3WOh54znPGU
- 53. For more examples: withoutbg.com Example Case: Image Matting
- 54. Which things should be kept in this picture? Kid, ball, 2 dogs, 9 people? Example Case: Image Matting Photo: Treddy Chen https://unsplash.com/photos/UdQWvefOXJk
- 55. Issue: When there is more than one person in the image... Example Case: Image Matting
- 56. Review questions - How do we compute the confusion matrix for a segmentation mask? How do we compute it for a bounding box? - Can we use the Intersection over Union equation to evaluate the quality of a segmentation mask? - What’s the recall of a classifier that only outputs ‘1’ (positive class)? - What’s the precision of a classifier that outputs a single true positive, with all its other predictions being equal to ‘0’ (negative class)? - Why does precision go down when recall increases? - Does the F1 measure weigh precision and recall equally? - What’s the appeal of using Detectron2? Do we need to write a Pytorch model to use it for inference or training?
- 57. Google Colab Notebooks ● Unet in FastAI 2 ● Mask R-CNN and Panoptic Segmentation with Detectron 2
- 58. - How does panoptic segmentation combine instance and semantic segmentation? Which method produces the ‘stuff’? Which method produces the ‘things’? - Is semantic segmentation more computationally costly than instance segmentation? Why? - Is panoptic segmentation more computationally costly than instance segmentation? Why? Review questions
- 59. References ● Stanford’s cs231n lecture on Object Detection and Segmentation ● PyImageSearch tutorial on Mask R-CNN