DeepLab V3+: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Download as PPTX, PDF1 like2,465 views
The document details the DeepLabV3+ model, an advanced encoder-decoder architecture for semantic image segmentation utilizing atrous separable convolution. It highlights the evolution of DeepLab models, the significance of atrous convolution in capturing contextual information at multiple scales, and introduces depth-wise separable convolution as a computationally efficient alternative. The architecture integrates an encoder-decoder structure, where the encoder extracts semantic information and the decoder enhances resolution for improved image quality.
DeepLab V3+: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
- 1. DeepLabV3+ Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
- 2. Background ▪ DeepLabV3+ is the latest version of the DeepLab models. ▪ DeepLab V1: Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. ICLR 2015. ▪ DeepLab V2: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. TPAMI 2017. ▪ DeepLab V3: Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv 2017. ▪ DeepLab V3+: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv 2018.
- 3. Semantic Segmentation ▪ Classifying all pixels in an image into classes. ▪ Classification at the pixel level. ▪ Does not have to separate different instances of the same class. ▪ Has important applications in Medical Imaging.
- 4. Current Results on Pascal VOC 2012
- 5. Motivation and Key Concepts ▪ Use Atrous Convolution and Separable Convolutions to reduce computation. ▪ Combine Atrous Spatial Pyramid Pooling Modules and Encoder-Decoder Structures. ▪ ASPPs capture contextual information at multiple scales by pooling features at different resolutions. ▪ Encoder-Decoders can obtain sharp object boundaries.
- 8. Convolution (Cross-Correlation) for 1 Channel Convolution with Zero-Padding Display with Convolution Kernel Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed
- 9. Other Convolutions (Cross-Correlations) Strided Convolution with Padding Atrous (Dilated) Convolution with r=2 Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed
- 10. Atrous Convolution ▪ à trous is French for “with holes” ▪ Atrous Convolution is also known as Dilated Convolution. ▪ Atrous Convolution with r=1 is the same as ordinary Convolution ▪ The image on the left shows 1D atrous convolution
- 11. Receptive Field of Atrous Convolutions ▪ Left: r=1, Middle: r=2, Right: r=4 ▪ Atrous Convolution has a larger receptive field than normal convolution with the same number of parameters.
- 12. Depth-wise Separable Convolution ▪ A special case of Grouped Convolution. ▪ Separate the convolution operation along the depth (channel) dimension. ▪ It can refer to both (depth -> point) and (point -> depth). ▪ It only has meaning in multi-channel convolutions (cross-correlations).
- 13. Review: Multi-Channel 2D Convolution
- 14. Exact Shapes and Terminology ▪ Filter: A collection of 𝑪𝒊𝒏 Kernels of shape (𝑲 𝑯, 𝑲 𝑾) concatenated channel-wise. ▪ Input Tensor Shape: 𝑵, 𝑪𝒊𝒏, 𝑯, 𝑾 𝑜𝑟 (𝑵, 𝑯, 𝑾, 𝑪𝒊𝒏) ▪ Filters are 3D, Kernels are 2D, All filters are concatenated to a single 4D array in 2D CNNs. (𝑵: 𝐵𝑎𝑡𝑐ℎ 𝑁𝑢𝑚𝑏𝑒𝑟, 𝑯: 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝐻𝑒𝑖𝑔ℎ𝑡, 𝑾: 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝑊𝑖𝑑𝑡ℎ, 𝑪𝒊𝒏: #𝐼𝑛𝑝𝑢𝑡 𝐶ℎ𝑎𝑛𝑛𝑒𝑙𝑠, 𝑪 𝒐𝒖𝒕: #𝑂𝑢𝑡𝑝𝑢𝑡 𝐶ℎ𝑎𝑛𝑛𝑒𝑙𝑠, 𝑲 𝑯: 𝐾𝑒𝑟𝑛𝑒𝑙 𝐻𝑒𝑖𝑔ℎ𝑡, 𝑲 𝑾: 𝐾𝑒𝑟𝑛𝑒𝑙 𝑊𝑖𝑑𝑡ℎ)
- 15. Step 1: Convolution on Input Tensor Channels
- 16. Step 2: Summation along Input Channel Dimension
- 17. Step 3: Add Bias Term ▪ Each kernel of a filter iterates only 1 channel of the input tensor. ▪ The number of filters is 𝐶 𝑜𝑢𝑡. Each filter generates one output channel. ▪ Each 2D kernel is different from all other kernels in the 3D filter. Key Points
- 18. Normal Convolution ▪ Top: Input Tensor ▪ Middle: Filter ▪ Bottom: Output Tensor
- 19. Depth-wise Separable Convolution ▪ Replace Step 2. ▪ Instead of summation, use point-wise convolution (1x1 convolution). ▪ There is now only one (𝑪𝒊𝒏, 𝑲 𝑯, 𝑲 𝑾) filter. ▪ The number of 1x1 filters is 𝑪 𝒐𝒖𝒕. ▪ Bias is usually included only at the end of both convolution operations. ▪ Usually refers to depth-wise convolution -> point-wise convolution. ▪ Xception uses point-wise convolution -> depth-wise convolution.
- 21. Characteristics ▪ Depth-wise Separable Convolution can be used as a drop-in replacement for ordinary convolution in DCNNs. ▪ The number of parameters is reduced significantly (sparse representation). ▪ The number of flops is reduced by several orders of magnitude (computationally efficient). ▪ There is no significant drop in performance (performance may even improve). ▪ Wall-clock time reduction is less dramatic due to GPU memory access patterns.
- 22. Example: Flop Comparison (Padding O, Bias X) Ordinary Convolution ▪ 𝐻 ∗ 𝑊 ∗ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡 ▪ For a 256x256x3 image with 128 filters with kernel size of 3x3, the number of flops would be 256 ∗ 256 ∗ 3 ∗ 3 ∗ 3 ∗ 128 = 226,492,416 ▪ There is an 8-fold reduction in the number of flops. Depth-wise Separable Convolution ▪ 𝐻 ∗ 𝑊 ∗ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 + 𝐻 ∗ 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡 ▪ Left: Depth Conv, Right: Point Conv ▪ For a 256x256x3 image with 128 filters and a 3x3 kernel size, the number of flops would be 256 ∗ 256 ∗ 3 ∗ 3 ∗ 3 + 256 ∗ 256 ∗ 3 ∗ 128 = 1,769,472 + 25,165,824 = 26,935,296
- 23. Example: Parameter Comparison (Excluding Bias Term) Ordinary Convolution ▪ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡 ▪ For a 256x256x3 image with 128 filters and 3x3 kernel size, the number of weights would be 3 ∗ 3 ∗ 3 ∗ 128 = 3,456 Depth-wise Separable Convolution ▪ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 + 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡 ▪ For a 256x256x3 image with 128 filters and 3x3 kernel size, the number of flops would be 3 ∗ 3 ∗ 3 + 3 ∗ 128 = 411 ▪ There is also an 8-fold reduction in parameter numbers.
- 24. Atrous Depth-wise Separable Convolution
- 26. Encoder-Decoder Structures ▪ The Encoder reduces the spatial sizes of feature maps, while extracting higher- level semantic information. ▪ The Decoder gradually recovers the spatial information. ▪ UNETs are a classical example of encoder-decoder structures. ▪ In DeepLabV3+, DeepLabV3 is used as the encoder.
- 28. Decoder Layer Structure 1. Apply 4-fold bilinear up-sampling on the ASPP outputs. 2. Apply 1x1 Convolution with reduced filter number on a intermediate feature layer. 3. Concatenate ASPP outputs with intermediate features. 4. Apply two 3x3 Convolutions. 5. Apply 4-fold bilinear up-sampling. Purpose & Implementation ▪ The ASPP is poor at capturing fine details. ▪ The decoder is used to improve the resolution of the image. ▪ The intermediate layer has 1x1 convolutions to reduce channel number.
- 29. ASPP: Atrous Spatial Pyramid Pooling
- 30. The ASPP Layer ▪ Encodes multi-scale contextual information through multiple rates. ▪ Concatenate all extracted features and an up-sampled global average pooling layer channel-wise. ▪ Use Atrous Depth-wise separable convolutions for multiple channels. ▪ Bad at capturing sharp object boundaries.
- 31. Modified Aligned Xception Network ▪ Xception: Extreme Inception Network. ▪ Backbone network for DeepLabV3+ ▪ Uses residual blocks and separable convolutions.
- 32. Explanation of Xception ▪ Takes the “Inception Hypothesis”, which states that cross-channel correlations and spatial correlations are sufficiently decoupled that it is preferable not to map them jointly, to the extreme. ▪ The extensive use of separable convolutions and atrous convolutions allows the model to fit in GPU memory despite the huge number of layers. ▪ Originally applied point-wise convolution before depth-wise convolution. ▪ Invented by François Chollet.
- 34. The End