MobileNet V3
2 likes1,060 views
The document summarizes improvements made in MobileNetV3 models, including using complementary search techniques to find efficient building blocks, modifying nonlinearities like h-swish to be more efficient, and improving expensive layers through techniques like removing unnecessary projections. It also describes experiments that showed MobileNetV3 models achieving better performance versus V1/V2 models on tasks like image classification, object detection, and semantic segmentation while maintaining high efficiency for mobile applications.
MobileNet V3
- 2. Introduction This paper describes the approach we took to develop MobileNetV3 Large and Small models 1. complementary search techniques 2. new efficient versions of nonlinearities practical for the mobile setting 3. new efficient network design 4. a new efficient segmentation decoder MobileNet V3
- 3. Recap: MobileNet V1 Depth-wise separable Convolution Depthwise Convolution Pointwise Convolution • Standard Convolution cost: ℎ𝑖 × 𝑤𝑖 × 𝑑𝑗 × 𝑘 × 𝑘 × 𝑑𝑖 • Deptwise seperable convolution cost ℎ𝑖 × 𝑤𝑖 × 𝑑𝑗 × (𝑘2 + 𝑑𝑖) • Reduction in computation 1 𝑑 𝑗 + 1 𝑘2
- 4. Recap: MobileNet V2 Linear Bottlenecks • If the manifold of interest remains non-zero volume after ReLU transformation, it corresponds to a linear transformation. • ReLU is capable of preserving complete information about the input manifold, but only if the input manifold lies in a low-dimensional subspace of the input space. • we can capture this by inserting linear bottleneck layers into the convolutional blocks.
- 5. Recap: MobileNet V2 Inverted Residual Residual Inverted Residual wide → narrow → wide narrow → wide → narrow ℎ × 𝑤 × 𝑡 × 𝑑′ × 𝑑′ + ℎ × 𝑤 × 𝑡 × 𝑑′ × 𝑘 × 𝑘 + ℎ × 𝑤 × 𝑑′′ × 𝑡 × 𝑑′′ = ℎ × 𝑤 × 𝑑′ × 𝑡 × (𝑑′ + 𝑘2 + 𝑑′′) Block of size ℎ × 𝑤, expansion factor 𝑡, kernel size 𝑘, input channel 𝑑′, output channel 𝑑′′
- 10. Efficient Mobile Building Block we use a combination of these layers as building blocks in order to build the most effective models. upgraded with modified swish nonlinearities → h-swish Combination of layers 1. MobileNetV1: depthwise separable convolutions 2. MobileNetV2: linear bottleneck, inverted residual structure 3. MnasNet: lightweight attention modules based on squeeze and excitation into the bottleneck structure MobileNetV3
- 11. Network Search Platform-Aware NAS for Block-wise Search 𝜔 = −0.17 vs the original 𝜔 = −0.07 We observe that accuracy changes much more dramatically with latency for small models 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝐴𝐶𝐶(𝑚) × 𝐿𝐴𝑇(𝑚) 𝑇 𝜔 𝑚
- 12. Network Search NetAdapt for Layer-wise Search 1. Starts with a seed network architecture found by platform-aware NAS. 2. For each step: (a) Generate a set of new proposals. Each proposal represents a modification of an architecture that generates at least δ reduction in latency compared to the previous step. (b) For each proposal we use the pre-trained model from the previous step and populate the new proposed architecture, truncating and randomly initializing missing weights as appropriate. Finetune each proposal for T steps to get a coarse estimate of the accuracy. (c) Selected best proposal according to some metric. 3. Iterate previous step until target latency is reached.
- 13. Network Search Proposal 1. Reduce the size of any expansion layer; 2. Reduce bottleneck in all blocks that share the same bottleneck size - to maintain residual connections 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 ∆Acc |∆latency| → maximize tan Evaluation
- 14. Network Improvements Redesigning Expensive Layers We redesign the computionally-expensive layers at the beginning and the end of the network. These modifications are outside of the scope of the current search space. 1. Current models based on MobileNetV2’s inverted bottleneck structure and variants use 1x1 convolution. This layer is critically important in order to have rich features for prediction. However, this comes at a cost of extra latency. → Move this layer (1) past the final average pooling 2. The previous bottleneck projection layer is no longer needed to reduce computation. → Remove the projection and filtering layers in the previous bottleneck layer
- 15. Network Improvements Redesigning Expensive Layers
- 16. Network Improvements Redesigning Expensive Layers We experimented with reducing the number of filters and using different nonlinearities to try and reduce redundancy. We were able to reduce the number of filters to 16 while maintaining the same accuracy as 32 filters using either ReLU or swish.
- 17. Network Improvements Nonlinearities While this nonlinearity improves accuracy, it comes with non-zero cost in embedded environments as the sigmoid function is much more expensive to compute on mobile devices.
- 18. Network Improvements Nonlinearities First, optimized implementations of ReLU6 are available on virtually all software and hardware frameworks. Second, in quantized mode, it eliminates potential numerical precision loss caused by different implementations of the approximate sigmoid. Finally, in practice, h-swish can be implemented as a piece-wise function to reduce the number of memory accesses driving the latency cost down substantially
- 19. Network Improvements Nonlinearities The cost of applying nonlinearity decreases as we go deeper into the network, since each layer activation memory typically halves every time the resolution drops. Thus in our architectures we only use h-swish at the second half of the model.
- 20. Network Improvements Large squeeze-and-excite we replace them all to fixed to be 1/4 of the number of channels in expansion layer.
- 27. Experiments Results - Semantic Segmentation R-ASPP
- 28. Experiments Results - Semantic Segmentation R-ASPP