Review : Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
0 likes121 views
The document presents a semi-supervised transfer learning framework that enhances consistency regularization by incorporating adaptive knowledge consistency (AKC) and adaptive representation consistency (ARC). These methods utilize labeled and unlabeled data effectively, demonstrating competitive results against state-of-the-art semi-supervised learning techniques across various benchmarks. The proposed framework aims to improve transfer learning by addressing the domain gap and optimizing representation learning.
![Introduction
Transfer Learning
• The powerful pre-trained model
1) excellent transferability
2) generalization capacity
• Zhou et al.
1) the benefit of SSL are smaller when trained from a pre-trained model
2) combining SSL and transfer learning can solve the domain gap
[Zhou et al, When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets, arXiv 2018]](https://image.slidesharecdn.com/adaptiveconsistencyregularizationforsemi-supervisedtransferlearningreviewcdm-210307084235/85/Review-Adaptive-Consistency-Regularization-for-Semi-Supervised-Transfer-Learning-3-320.jpg)
Review : Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
- 1. Adaptive Consistency Regularization for Semi-Supervised Transfer Learning Abuduweili et al. (CVPR 2021) Dongmin Choi Yonsei University Translational Artificial Intelligence Lab
- 2. Introduction Semi-Supervised Learning (SSL) • Effectively leveraging both labeled and unlabeled data • Three main approaches: 1) consistency based regularization 2) entropy minimization 3) pseudo label
- 3. Introduction Transfer Learning • The powerful pre-trained model 1) excellent transferability 2) generalization capacity • Zhou et al. 1) the benefit of SSL are smaller when trained from a pre-trained model 2) combining SSL and transfer learning can solve the domain gap [Zhou et al, When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets, arXiv 2018]
- 4. Introduction A Semi-Supervised Transfer Learning Framework • Extend consistency regularization in SSL to adapt the inductive transfer learning • Two essential components: 1) Adaptive Knowledge Consistency (AKC) - transfer knowledge from the pre-trained model 2) Adaptive Representation Consistency (ARC) - utilize unlabeled examples to adjust the representation
- 5. Related Work Domain Adaptation • Tackle the sample selection bias btw the training and test data • Generate domain invariant representation over the training set • 내용 추가 필요
- 6. Related Work Semi-Supervised Learning • Consistency based regularization - hypothesis : the decision boundary should not pass through high- density areas → two close inputs are expected to have the same label [Engelen et al, A survey on semi-supervised learning, Machine Learning 2020
- 7. Related Work Semi-Supervised Learning • П-model [Laine, Temporal Ensembling for Semi-Supervised Learning, ICLR 2017 Targets can be noisy prior network evaluations
- 8. Related Work Semi-Supervised Learning • Mean Teacher [Tarvainen, Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, NIPS 2017 Averages model weights instead of label predictions
- 9. Related Work Semi-Supervised Learning • FixMatch [Sohn et al., FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, NeurIPS 2020 Consistency regularization + Pseudo labeling
- 11. The Proposed Framework 𝑫𝒕 𝒍 𝑫𝒕 𝒖 𝑭𝜽𝟎 𝑭𝜽 𝑮𝜽𝟎 𝑮𝜽 𝜃∗ , 𝜙∗ = arg min 𝜃,𝜙 ∑𝑖=1 𝑛 𝐿CE 𝜃, 𝜙; 𝑥𝑙 𝑖 + 𝑅 𝜃
- 12. The Proposed Framework 𝑅𝐾 = 1 𝐵𝑙 + 𝐵𝑢 𝑥𝑖∈𝐿∪𝑈 𝑤K 𝑖 KL 𝐹𝜃0 𝑥𝑖 , 𝐹𝜃 𝑥𝑖 1. Adaptive Knowledge Consistency (AKC)
- 13. The Proposed Framework 𝑅𝐾 = 1 𝐵𝑙 + 𝐵𝑢 𝑥𝑖∈𝐿∪𝑈 𝒘𝐊 𝒊 KL 𝐹𝜃0 𝑥𝑖 , 𝐹𝜃 𝑥𝑖 1. Adaptive Knowledge Consistency (AKC) Sample importance 𝒘𝐊 𝒊 = 𝐈 𝐇 𝐩𝒔 𝒊 ≤ 𝝐𝐊 - An entropy function H p𝑠 𝑖 = − ∑𝑗=1 𝐶𝑠 p𝑠,𝑗 𝑖 log p𝑠,𝑗 𝑖 - I : a hard entropy-gate function (calculated entropy → binary sample importance)
- 14. The Proposed Framework 2. Adaptive Representation Consistency (ARC) Maximum Mean Discrepancies (MMD) to measure the distance (Let’s skip the details!)
- 15. The Proposed Framework Summarization of the Framework 𝐿 𝜃, 𝜙 = 1 𝑛 𝑖=1 𝑛 𝐿CE 𝜃, 𝜙; 𝑥𝑙 𝑖 + 𝜆S𝐿S 𝑥𝑢 𝑖 + 𝜆K𝑅K 𝑥𝑙 𝑖 , 𝑥𝑢 𝑖 + 𝜆R𝐿R 𝑥𝑙 𝑖 , 𝑥𝑢 𝑖 1 2 3 4 1 2 3 4
- 20. Experiments The actual sample selected ratio in ARC and AKC Near 0.9 - exclude hard samples
- 21. Experiments In Fully Supervised Transfer Learning
- 22. Conclusion Two regularization methods : AKC and ARC • Competitive among S.O.T.A SSL methods • Best performance among several baseline methods on various transfer learning benchmarks • Can be used for more general transfer learning and (semi-) supervised learning frameworks
- 23. Thank you