Architecture Design for Deep Neural Networks III

Neural Architecture Search:
The Next Half Generation of Machine Learning
Speaker: Lingxi Xie (谢凌曦)
Noah’s Ark Lab, Huawei Inc. (华为诺亚方舟实验室)
Slides available at my homepage (TALKS)

Take-Home Messages
 Neural architecture search (NAS) is the future
 Deep learning makes feature learning automatic
 NAS makes deep learning automatic
 The future is approaching faster than we used to think!
 2017: NAS appears
 2018: NAS becomes approachable
 2019 and 2020: NAS will be mature and a standard technique

Outline
 Introduction
 Framework
 RepresentativeWork
 Our New Progress
 Future Directions

Introduction: Neural Architecture Search
 NeuralArchitecture Search (NAS)
 Instead of manually designing neural network architecture (e.g., AlexNet,VGGNet,
GoogLeNet, ResNet, DenseNet, etc.), exploring the possibility of discovering
unexplored architecture with automatic algorithms
 Why is NAS important?
 A step from manual model design to automatic model design (analogy: deep learning
vs. conventional approaches)
 Able to develop data-specific models
[Krizhevsky, 2012] A. Krizhevsky et al., ImageNetClassification with Deep Convolutional Neural Networks, NIPS,
2012.
[Simonyan, 2015] K. Simonyan et al.,Very Deep Convolutional Networks for Large-scale Image Recognition, ICLR,
2015.
[Szegedy, 2015] C. Szegedy et al., Going Deeper withConvolutions, CVPR, 2015.
[He, 2016] K. He et al., Deep Residual Learning for Image Recognition,CVPR, 2016.
[Huang, 2017] G. Huang et al., Densely Connected Convolutional Networks, CVPR, 2017.

Introduction: Examples and Comparison
 Model comparison: ResNet,GeNet, NASNet and ENASNet
[He, 2016] K. He et al., Deep Residual Learning for Image Recognition, CVPR, 2016.
[Xie, 2017] L. Xie et al., Genetic CNN, ICCV, 2017.
[Zoph, 2018] B. Zoph et al., LearningTransferable Architectures for Scalable Image Recognition, CVPR, 2018.
[Pham, 2018] H. Pham et al., Efficient Neural Architecture Search via Parameter Sharing, ICML, 2018.

Outline
 Introduction
 Framework
 RepresentativeWork
 Our New Progress
 RelatedApplications
 Future Directions

Framework:Trial and Update
 Almost all NAS algorithms are based on the “trial and update” framework
 Starting with a set of initial architectures (e.g., manually defined) as individuals
 Assuming that better architectures can be obtained by slight modification
 Applying different operations on the existing architectures
 Preserving the high-quality individuals and updating the individual pool
 Iterating till the end
 Three fundamental requirements
 The building blocks: defining the search space (dimensionality, complexity, etc.)
 The representation: defining the transition between individuals
 The evaluation method: determining if a generated individual is of high quality

Framework: Building Blocks
 Building blocks are like basic genes for these individuals
 Some examples here
 Genetic CNN: only 3 × 3 convolution is allowed to be searched (followed by default BN
and ReLU operations), 3 × 3 pooling is fixed
 NASNet: 13 operations shown below
 PNASNet: 8 operations, removing those
never-used ones from NASNet
 ENASNet: 6 operations
 DARTS: 8 operations
[Liu, 2018] C. Liu et al., Progressive NeuralArchitecture Search, ECCV, 2018.
[Liu, 2019] H. Liu et al., DARTS: Differentiable Architecture Search, ICLR, 2019.

Framework: Search
 Finding new individuals that have potentials to work better
 Heuristic search in the large space
 Two mainly applied methods: the genetic algorithm and reinforcement learning
 Both are heuristic algorithms applied to the scenarios of a large search space and
limited ability to explore every single element in the space
 A fundamental assumption: both of these heuristic algorithms can preserve good genes
and based on which discover possible improvements
 Also, it is possible to integrate architecture search to network optimization
 These algorithms are often much faster
[Real, 2017] E. Real et al., Large-Scale Evolution of ImageClassifiers, ICML, 2017.

Framework: Evaluation
 Evaluation aims at determining which individuals are good and to be preserved
 Conventionally, this was often done by training a network from scratch
 This is extremely time-consuming, so researchers often train NAS on a small dataset
like CIFAR and then transfer the found architecture to larger datasets like ImageNet
 Even in this way, the training process is really slow: Genetic-CNN requires 17 GPU-days
for a single training process, and NAS-RL requires more than 20,000 GPU-days
 Efficient methods were proposed later
 Ideas include parameter sharing (without the need of re-training everything for each
new individual) and using a differentiable architecture (joint optimization)
 Now, an efficient search process on CIFAR can be reduced to a few GPU-hours, though
training the searched architecture on ImageNet is still time-consuming
[Zoph, 2017] B. Zoph et al., Neural Architecture Search with Reinforcement Learning, ICLR, 2017.

RepresentativeWork on NAS
 Evolution-based approaches
 Reinforcement-learning-based approaches
 Towards one-shot approaches
 Applications

Genetic CNN
 Only considering the connection between basic building blocks
 Encoding each network into a fixed-length binary string
 Standard operators:
mutation, crossover,
and selection
 Limited by computation
 Relatively low accuracy

Genetic CNN
 CIFAR10 experiments
 3 stages, 𝐾1, 𝐾2, 𝐾3 = 3,4,5 , 𝐿 = 19
 𝑁 = 20 (individuals), 𝐿 = 50 (rounds)
Gen # Max % Min % Avg % Med % St-D %
0 75.96 71.81 74.39 74.53 0.91
1 75.96 73.93 75.01 75.17 0.57
2 75.96 73.95 75.32 75.48 0.57
5 76.24 72.60 75.32 75.65 0.89
10 76.72 73.92 75.68 75.80 0.88
20 76.83 74.91 76.45 76.79 0.61
50 77.06 75.84 76.58 76.81 0.55
Figure: the
impact of
initialization
is ignorable
after a
sufficient
number of
rounds
Figure: (a)
parent(s)
with higher
recognition
accuracy are
more likely
to generate
child(ren)
with higher
quality

Genetic CNN
 Generalizing the best learned structures to other tasks
 The small datasets with deeper networks
0
1
2 3
4
0
1
2 3
4
5
0
1
2 3
4 5
6
0
1
2 3
4
0
1
2 3
4
5
0
1
2 3
4 5
6
Code: 1-01
Code: 1-01
Chain-Shaped
Networks
✓ AlexNet
✓ VGGNet
Code: 0-01-100
Code: 1-01-100
Code: 0-11-
101-0001
Code: 0-11-
101-0001
Multiple-Path
Networks
✓ GoogLeNet
Highway
Networks
✓ Deep ResNet
Network SVHN CF10 CF100
GeNet #1, after Gen. #0 2.25 8.18 31.46
GeNet #1, after Gen. #5 2.15 7.67 30.17
GeNet #1, after Gen. #20 2.05 7.36 29.63
GeNet #1, after Gen. #50 1.99 7.19 29.03
GeNet #2, after Gen. #50 1.97 7.10 29.05
Network ILSVRC2012, 1/5 Depth
19-layerVGGNet 28.7 9.9 19
GeNet #1, after Gen. #50 28.12 9.95 22
GeNet #2, after Gen. #50 27.87 9.74 22

Large-Scale Evolution of Image Classifiers
 Modifying the individuals with a pre-defined set of
operations, shown in the right part
 Larger networks work better
 Much larger computational overhead is used: 250
computers for hundreds of hours
 Take-home message: NAS requires careful design
and large computational costs
[Real, 2017] E. Real et al., Large-Scale Evolution of Image Classifiers, ICML, 2017.

Large-Scale Evolution of Image Classifiers
 The search progress
[Real, 2017] E. Real et al., Large-Scale Evolution of Image Classifiers, ICML, 2017.

NAS with Reinforcement Learning
 Using reinforcement learning (RL)
to search over the large space
 The entire structure is generated by
an RL algorithm or an agent
 The validation accuracy serves as
feedback to train the agent’s policy
 Computational overhead is high
 800 GPUs for 28 days (CIFAR)
 No ImageNet experiments
 Superior accuracy to manually-
designed network architectures
[Zoph, 2017] B. Zoph et al., Neural Architecture Search with Reinforcement Learning, ICLR, 2017.

NAS Network
 Instead of the previous work that searched everything, this work only searched for
a limited number of basic building blocks
 The remaining part is mostly the same
 Computational overhead is still high
 Good ImageNet performance

Progressive NAS
 Instead of searching over the entire network (containing a few blocks), this work
added one block each time (progressive search)
 The best combinations are recorded for the next-stage search
 The efficiency of search is higher
 The remaining part is mostly the same
 Computational overhead is still high
 100 GPUs for 1.5 days (CIFAR)
 Better ImageNet performance

Regularized Evolution
 Regularized evolution: assigning “aged”
individuals with a higher probability to be
eliminated
 Evolution works equally well or better than
RL algorithms
 Take-home message: evolutional algorithms
play an important role especially when the
computational budget is limited; also, the
conventional evolutional algorithms need to
be modified so as to fit the NAS task
[Real, 2019] E. Real et al., Regularized Evolution for ImageClassifierArchitecture Search, AAAI, 2019.

Efficient NAS by NetworkTransformation
 Instead of training a new individual from scratch, this work reused the weights of
a prior network (expected to be similar to the current network), so that the
current training is more efficient
 Net2Net is used for initialization
 Operations: wider and deeper
 Much more efficient
[Chen, 2015] T. Chen et al., Net2Net:Accelerating Learning via KnowledgeTransfer, ICLR, 2015.
[Cai, 2018] H. Cai et al., Efficient Architecture Search by NetworkTransformation, AAAI, 2018.

Efficient NAS via Parameter Sharing
 Instead of modifying network initialization, this work
goes one step forward by sharing parameters among
all generated networks
 Each training stage is much shorter
 1 GPU for 0.45 days (CIFAR)

DifferentiableArchitecture Search
 With a fixed number of intermediate blocks, the operator applied to each state is
unknown in the beginning
 During the training process, the operator is formulated as a mixture model
 The learning goal is the mixture
coefficients (differentiable)
 In the end of training, the most
likely operator is kept, and the
entire network is trained again
 1 GPU for 4 days (CIFAR)
 Reasonable ImageNet results
(in the mobile setting)

DifferentiableArchitecture Search
 The best cell changes over time

Proxyless NAS
 The first NAS work that is directly optimized on ImageNet (ILSVRC2012)
 Learning weight parameters and binarized architectures simultaneously
 Close to Differentiable NAS
 Efficient
 1 GPU
for 8
days
 Reason-
able
perfor-
mance
(mobile)
[Cai, 2019] H. Cai et al., ProxylessNAS: Direct Neural Architecture Search onTargetTask and Hardware, ICLR, 2019.

Probabilistic NAS
 A new way to train a super-network
 Sampling sub-networks from a distribution
 Also able to perform proxyless architecture search
 Efficiency brought by flexible control of search time on each sub-network
 1 GPU for
0.2 days
 Accuracy
is a little bit
weak on
ImageNet
[Noy, 2019] F.P. Casale et al., Probabilistic Neural Architecture Search, arXiv preprint: 1902.05116, 2019.

Single-PathOne-Shot NAS
 Main idea: balancing the sampling probability of each path in one-shot search
 With the benefit of decoupling operations on each edge
 Bridging the gap between search and evaluation
 Modified search space
 Blocks based on ShuffleNet-v2
 Evolution-based search algorithm
 Channel number search
 Latency and FLOPs constraints
 Improved accuracy on single-shot
NAS
[Guo, 2019] Z. Guo et al., Single Path One-Shot Neural Architecture Search with Uniform Sampling, arXiv preprint:
1904.00420, 2019.

Architecture Search,Anneal and Prune
 Another effort to deal with the decoupling
issue of DARTS
 Decreasing the temperature term in computing
the probability added to each edge
 Pruning edges with low weights
 Gradually turning the architecture to one-path
 Efficiency brought by pruning
 1 GPU for 0.2 days
 Accuracy is still a little bit weak on ImageNet
[Noy, 2019] A. Noy et al., ASAP:Architecture Search, Anneal and Prune, arXiv preprint: 1904.04123, 2019.

RandomlyWired Neural Networks
 A more diverse set of connectivity patterns
 Connecting NAS and randomly wired neural networks
 An important insight: when the search
space is large enough, randomly wired
networks are almost as effective as
carefully searched architectures
 This does not reflect that NAS is useless,
but reveals that the current NAS methods
are not effective enough
[Xie, 2019] S. Xie et al., Exploring RandomlyWired Neural Networks for Image Recognition, arXiv preprint:
1904.01569, 2019.

Auto-Deeplab
 A hierarchical architecture search space
 With both network-level and cell-level structures being investigated
 Differentiable search method (in order to accelerate)
 Similar performance to Deeplab-v3 (without pre-training)
[Liu, 2019] C. Liu et al., Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation,
CVPR, 2019.

NAS-FPN
 Searching for the feature pyramid network
 Reinforcement-learning-based search
 Good performance on MS-COCO
 Improving mobile detection accuracy by 2% AP
compared to SSDLite on MobileNetV2
 Achieving 48.3% AP, surpassing state-of-the-arts
[Ghaisi, 2019] G. Ghaisi et al., NAS-FPN: Learning Scalable Feature PyramidArchitecturefor Object Detection,
CVPR, 2019.

Auto-ReID
 A search space with part-aware module
 Using both ranking and classification loss
 Differentiable search
 State-0f-the-art performance on ReID
[Quan, 2019] R. Quan et al., Auto-ReID:Searching for a Part-aware ConvNet for Person Re-Identification, arXiv
preprint: 1903.09776, 2019.

GraphNAS
 A search space containing components of
GNN layers
 RL-based search algorithm
 A modified parameter sharing scheme
 Surpassing manually designed GNN architectures
[Gao, 2019] Y.Gao et al., GraphNAS:Graph NeuralArchitecture Search with ReinforcementLearning, arXiv
preprint: 1904.09981, 2019.

V-NAS
 Medical image segmentation
 Volumetric convolution required
 Searching volumetric convolution
 2D conv, 3D conv and P3D conv
 Differentiable search algorithm
 Outperforming state-of-the-arts
[Zhu, 2019] Z. Zhu et al.,V-NAS: NeuralArchitecture Search forVolumetric Medical Image Segmentation, arXiv
preprint: 1906.02817, 2019.

AutoAugment
 Learning hyper-parameters
 Search Space: Shear-X/Y,Translate-X/Y,
Rotate, AutoContrast, Invert, etc.
 Reinforcement-learning-based search
 Impressive performance on a few standard
image classification benchmarks
 Transferring to other tasks, e.g., NAS-FPN
[Cubuk, 2019] E. Cubuk et al., AutoAugment: Learning AugmentationStrategies from Data, CVPR, 2019.

MoreWork forYour Reference
 https://github.com/markdtw/awesome-architecture-search

P-DARTS: Overview
 We start with the drawbacks of DARTS
 There is a depth gap between search and evaluation
 The search process is not stable: multiple runs, different results
 The search process is not likely to transfer: only able to work on CIFAR10
 We proposed a new approach named Progressive DARTS
 A multi-stage search progress which gradually increases the search depth
 Two useful techniques: search space approximation and search space regularization
 We obtained nice results
 SOTA accuracy by the searched networks on CIFAR10/CIFAR100 and ImageNet
 Search cost as small as 0.3 GPU-days (one single GPU, 7 hours)
[Chen, 2019] X.Chen et al., Progressive Differentiable Architecture Search: Bridging the DepthGap between
Search and Evaluation, submitted, 2019.

P-DARTS: Motivation
 The depth gap and why it is important
DARTS: CIFAR10 test error 2.83%
8
cells
20
cells
search evaluation
P-DARTS: CIFAR10 test error 2.55%
5
cells
20
cells
search evaluation
11
cells
17
cells

P-DARTS: Search Space Approximation
 The progressive way of increasing search depth

P-DARTS: Search Space Regularization
 Problem: the strange behavior of skip-connect
 Searching on a deep network leads to many skip-connect operations (poor results)
 Reasons?
 On the one hand, skip-connect often leads to fastest gradient descent
 On the other hand, skip-connect does not have parameters and so leads to bad results
 Solution: regularization
 Adding a Dropout after each skip-connect, dedaying the rate during search
 Preserving a fixed number of skip-connect after the entire search
 Results
Dropout on skip-c Testing Error, 2 SC Testing Error, 3 SC Testing Error, 4 SC
with Dropout 2.93% 3.28% 3.51%
without Dropout 2.69% 2.84% 2.97%

P-DARTS: Performance on CIFAR10/100
 CIFAR10 andCIFAR100 (a useful enhancement:Cutout)
[DeVries, 2017] T. DeVries et al., Improved Regularization of Convolutional Neural Networks with Cutout, arXiv
1708.04552, 2017.

P-DARTS: Performance on ImageNet
 ImageNet (ILSVRC2012) under the Mobile Setting

P-DARTS: Searched Cells
 Searched architectures (verification of depth gap!)

P-DARTS: Summary
 The depth gap needs to be solved
 Different properties of networks with different depths
 Depth is still the key issue in deep learning
 Our approach
 State-of-the-art results on both CIFAR10/100 and ImageNet
 Search cost as small as 0.3 GPU-days
 Future directions
 Directly searching on ImageNet
 There are many unsolved issues on NAS!

PC-DARTS:A More Powerful Approach
 We still build our approach upon DARTS
 We proposed a new approach named Partially-Connected DARTS
 An alternative approach to deal with the over-fitting issue of DARTS
 Using partial channel connection as regularization
 This method is even more stable, which can be directly searched on ImageNet
 We obtained nice results
 SOTA accuracy by the searched networks on ImageNet
 Search cost as small as 0.06 GPU-days (one single GPU, 1.5 hours) on CIFAR10/100, or 4
GPU-days (8 GPUs, 11.5 hours) on ImageNet
[Xu, 2019] Y. Xu et al., PC-DARTS: Partial Channel Connections for Memory-Efficient Differentiable Architecture
Search, submitted, 2019.

PC-DARTS: Illustration
 Partial channel connection and edge normalization

PC-DARTS: Performance on ImageNet
 ImageNet (ILSVRC2012) under the Mobile Setting

PC-DARTS: Summary
 Regularization is still a big issue
 Partial channel connection in order to prevent over-fitting
 Edge normalization in order to make partial channel connection work more stable
 Our approach
 State-of-the-art results on ImageNet
 Search cost as small as 0.06 GPU-days
 Future directions
 Searching on a larger number of classes
 There are many unsolved issues on NAS!

Conclusions
 NAS is a promising and important trend for machine learning in the future
 NAS vs. fixed architectures as deep learning vs. conventional handcrafted features
 Two important factors of NAS to be determined
 Basic building blocks: fixed or learnable
 The way of exploring the search space: genetic algorithm, reinforcement learning, or
joint optimization
 The importance of computational power is reduced, but still significant

RelatedApplications
 The searched architectures were verified effective for transfer learning tasks
 NASNet outperformed ResNet101 in object detection by 4%
 Take-home message: stronger architectures are often transferrable
 The ability of NAS in other vision tasks
 Preliminary success in semantic segmentation, object detection, etc.
[Chen, 2018] L. Chen et al., Searching for Efficient Multi-ScaleArchitectures for Dense Image Prediction, NIPS,
2018.
[Liu, 2019] C. Liu et al., Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation,
CVPR, 2019.
[Ghiasi, 2019] G. Ghiasi et al., NAS-FPN: Learning Scalable Feature PyramidArchitecture for Object Detection,
CVPR, 2019.

Future Directions
 Currently, the search space is constrained by the limited types of building blocks
 It is not guaranteed that the current building blocks are optimal
 It remains to explore the possibility of searching into the building blocks
 Currently, the searched architectures are not friendly to hardware
 Which leads to dramatically slow speed in network training
 Currently, the searched architectures are task-specific
 This may not be a problem, but an ideal vision system should be generalized
 Currently, the searching process is not yet stable
 We desire a framework as generalized as regular deep networks

Thanks
 Questions, please?
 Contact me for collaboration and internship ☺

Architecture Design for Deep Neural Networks III

More Related Content

What's hot (20)

Similar to Architecture Design for Deep Neural Networks III (20)

More from Wanjin Yu (16)

Recently uploaded (20)

Architecture Design for Deep Neural Networks III