[MICCAI 2021] MT-UDA: Towards unsupervised cross-modality medical image segmentation with limited source labels
0 likes47 views
The document discusses a novel approach called MT-UDA for unsupervised cross-modality medical image segmentation, particularly under the challenge of limited source labels. It highlights the integration of semi-supervised learning with domain adaptation to improve segmentation performance, emphasizing the importance of both semantic and structural knowledge transfer. Experimental results demonstrate the effectiveness of this approach in generating reliable segmentation masks with fewer false positives, even with reduced labeled data.
![Existing Work – Unsupervised Domain Adaptation
- Image adaptation – Cycle GAN (2017)
- Feature adaptation – ADDA (2017)
- Sequentially combine two adaptive strategies – CyCADA (2018)
- Synergistic Image and Feature Adaptation – SIFA (2019, 2020)
Cardiac CT Cardiac MR
Domain Gap
CycleGAN [1]
ADDA [2]
Cycada [3]
SIFA [4]
[1] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." ICCV 2017
[2] Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.
[3] Hoffman, Judy, et al. "Cycada: Cycle-consistent adversarial domain adaptation.“ ICML 2018
[4] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020
Source Domain Target Domain](https://image.slidesharecdn.com/mt-udamiccai2021-220901163232-97bf56d1/85/MICCAI-2021-MT-UDA-Towards-unsupervised-cross-modality-medical-image-segmentation-with-limited-source-labels-5-320.jpg)
![Results (1) – Quantitative Comparison
- Degraded performance on target domain when using 4 labeled source domain scans
- MT and UA-MT can help improve the segmentation performance on target domain
- Demonstrate the feasibility of integrating SSL into UDA for label-efficient UDA
suffix −4 or −16
# labelled source scans
used for training
[1] Dou, Qi, et al. "Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss." IJCAI 2018
[2] Chen, Cheng, et al. "Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation." AAAI 2019
[3] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020
[4] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.“NIPS 2017
[5] Yu, Lequan, et al. "Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation.“ MICCAI 2019](https://image.slidesharecdn.com/mt-udamiccai2021-220901163232-97bf56d1/85/MICCAI-2021-MT-UDA-Towards-unsupervised-cross-modality-medical-image-segmentation-with-limited-source-labels-13-320.jpg)
[MICCAI 2021] MT-UDA: Towards unsupervised cross-modality medical image segmentation with limited source labels
- 1. MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels Ziyuan Zhao1,2, Kaixin Xu2, Shumeng Li1,2, Zeng Zeng2, Cuntai Guan1 Presenter: Zhao Ziyuan (G2104205L) 1 Nanyang Technological University 2 Institute for Infocomm Research, A*STAR Th-S2: Image Segmentation + Domain Adaptation # 678
- 2. Introduction – Medical Image Segmentation - Segmentation in medical physics plays a crucial role in medical image analysis (MedIA) - For instance, left atrium (LA) segmentation can provide a pre-operative assessment of its anatomy, which is essential for treating various cardiovascular diseases, such as atrial fibrillation - Deep learning has been widely used for medical image segmentation Left Atrial Cavity 3D LA Visualization 2018 Atrial Segmentation Challenge. https://atriaseg2018.cardiacatlas.org/
- 3. Challenge (1) – Label Scarcity - DCNNs are data-hungry and require large amounts of well-annotated data. - Annotating medical images is laborious, expensive, and requires human expertise → Label Scarcity Time & money consuming Knowledge-driven Labor-intensive Image DCNNs Segmentation map
- 4. Challenge (2) – Domain Shift - In real-world clinical scenarios, medical images are acquired with different physical principles and modalities, e.g., MRI & CT→ different visual appearance & distribution (Domain Gap) - DCNNs suffer from severe performance degradation when domain shift (e.g., CT → MRI) Cardiac Label Cardiac CT Source Domain Annotate DCNNs Transfer Cardiac MR Feed Domain Gap Target Domain
- 5. Existing Work – Unsupervised Domain Adaptation - Image adaptation – Cycle GAN (2017) - Feature adaptation – ADDA (2017) - Sequentially combine two adaptive strategies – CyCADA (2018) - Synergistic Image and Feature Adaptation – SIFA (2019, 2020) Cardiac CT Cardiac MR Domain Gap CycleGAN [1] ADDA [2] Cycada [3] SIFA [4] [1] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." ICCV 2017 [2] Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017. [3] Hoffman, Judy, et al. "Cycada: Cycle-consistent adversarial domain adaptation.“ ICML 2018 [4] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020 Source Domain Target Domain
- 6. Problem – Source Label Scarcity - Despite the success of adversarial learning in UDA, these methods heavily rely on abundant source labels. - Become sub-optimal when only limited source labels are available in clinical deployment. - Motivates us to study a challenging UDA scenario – source label scarcity Source Label Scarcity Cardiac CT Cardiac MR Domain Gap Source Domain Target Domain Less Annotations Lower Performance
- 7. Motivation – SSL + UDA - Image-level adaptation → generates a lot of synthetic images with abundant information, which can be leveraged for semi-supervised learning - Appearance consistency → synthetic and real images from the same domain maintain a similar visual appearance - Structural consistency → transformed images should have the same structural information as the original ones Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021 SSL Labeled data Performance Number of labeled data DCNNs Source Domain Target Domain Style Transfer
- 8. Method – MT-UDA - Investigate the feasibility of integrating SSL into UDA under source label scarcity - Develop a label-efficient UDA framework based on mean teacher (MT) to explore the knowledge from both domains Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 9. Method (1)– Dual Cycle Alignment Module - Generate synthetic samples for two domains using generative adversarial networks - Synthesize source-like domain images and target-like domain images via adversarial learning Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 10. Method (2)– Semantic Knowledge Transfer - Appearance consistency → synthetic and real images from the same domain maintain a similar visual appearance - Employ the mean teacher (MT) model to distill the intra-domain semantic knowledge by forcing the prediction consistency Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 11. Method (3)– Structural Knowledge Transfer - Structural consistency → transformed images should have the same structural information as the original ones - Propose a teacher model for keeping structural consistency between predictions of source images and corresponding synthetic target images Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 12. Experiments - Dataset - Multi-Modality Whole Heart Segmentation (MM-WHS) 2017 dataset - Unpaired 20 MR and 20 CT volumes with ground truth masks - Data preprocessing - MR → source domain, CT → target domain - SIFA setting (-16) → 16: 4 random split for train / val - Our setting (-4)→ 4 MR volumes are labelled for UDA under source label scarcity - Images were cropped into the size of 256 x 256 - Implementation details - Test on fake MR images generated from CT - Backbone: U-Net - Supervised loss: Dice + Cross-entropy - Total loss: Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 13. Results (1) – Quantitative Comparison - Degraded performance on target domain when using 4 labeled source domain scans - MT and UA-MT can help improve the segmentation performance on target domain - Demonstrate the feasibility of integrating SSL into UDA for label-efficient UDA suffix −4 or −16 # labelled source scans used for training [1] Dou, Qi, et al. "Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss." IJCAI 2018 [2] Chen, Cheng, et al. "Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation." AAAI 2019 [3] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020 [4] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.“NIPS 2017 [5] Yu, Lequan, et al. "Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation.“ MICCAI 2019
- 14. Results (2) – Qualitative Comparison - It is observed that our method can generate more reliable masks with fewer false positives Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 15. Results (3) – Ablation Study - Remove one of the teacher models, separately - W/o semantic knowledge transfer (MT-UDA-NS) - W/o structural knowledge transfer (MTUDA-NT) - Replace structural consistency loss with MSE loss (MT-UDA-NS-MSE) Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
- 16. Conclusions - Study a practical, challenging, and different UDA setting from the past, where only limited source labels are accessible → Source Label Scarcity - Investigate the feasibility of integrating SSL into UDA under source label scarcity - Propose a label-efficient UDA framework for cross-modality medical image segmentation - Leverage intra-domain semantic knowledge and exploit inter-domain structural knowledge concurrently, thereby mitigating both the domain discrepancy and source label scarcity. Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021