CycleMorph: Cycle consistent unsupervised deformable image registration
- 1. CycleMorph: Cycle consistent unsupervised deformable image registration Medical Image Analysis Boah Kim, Dong Hwan Kim, Seong Ho Park, Jieun Kim, June-Goo Lee, Jong Chul Ye
- 2. CycleMorph 2 Introduction Image Registration MRI Abdominal CT Deforming data into one coordinate system • subjects / time / modalities / ... Du, Juan, et al. "Intensity-based robust similarity for multimodal image registration." International Journal of Computer Mathematics 83.1 (2006): 49-57. Fundamental task to analyze data • Tumor volumetry studies • Multimodal information fusion • Therapy planning PET
- 3. CycleMorph 3 Background Classical iterative method Deep-learning-based method Supervised learning Method Unsupervised learning method Image Registration Methods
- 4. CycleMorph 4 • Deformation field A vector field of all displacement vectors for all coordinate in images Background Classical Iterative Method Deep-learning-based method • Spatial transformer Grid sampling to warp moving image into fixed image 𝑥𝑥 𝑦𝑦 Moving image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Spatial transformer
- 5. CycleMorph 5 Background 𝑳𝑳 𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝜙𝜙 = 𝑳𝑳𝒔𝒔𝒔𝒔𝒔𝒔 𝑇𝑇 𝑥𝑥; 𝜙𝜙 , 𝑦𝑦 + 𝑳𝑳𝒓𝒓𝒓𝒓𝒓𝒓(𝜙𝜙) Deep-learning-based method 𝑥𝑥 𝑦𝑦 Moving image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Cost function • 𝑳𝑳𝒔𝒔𝒔𝒔𝒔𝒔: Similarity cost function • 𝑳𝑳𝒓𝒓𝒓𝒓𝒓𝒓: Regularization function Classical Iterative Method Spatial transformer
- 6. CycleMorph 6 Background Advantages Pitfalls • Preserve topology between two different images • sufficient iteration / parameter tuning • Require substantial time, extensive computation Deep-learning-based method 𝑥𝑥 𝑦𝑦 Moving image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Classical Iterative Method Spatial transformer
- 7. CycleMorph 7 Background Classical iterative method Deep-learning-based Method 𝑥𝑥 𝑦𝑦 Moving image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Deep neural network Supervised learning Unsupervised learning Spatial transformer
- 8. CycleMorph 8 Background Require the ground-truth registration fields Cao. et al. “Non-rigid Brain MRI Registration Using Two-Stage Deep Perceptive Networks.” ISMRM 2018 Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 9. CycleMorph 9 Background Limitation • Difficult to obtain the real ground-truth in practice • Depend on the quality of the ground-truth registration fields Advantages • No parameter tuning for the inference • Applicable to various image domains Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 10. CycleMorph 10 Background Does not require any ground-truth label Balakrishnan. et al. “An unsupervised learning model for deformable medical image registration,.” CVPR 2018l • Spatial transformer network = Field generator + Spatial transformer • To provide deformable registration without labels for registration fields • Pitfalls: Potential for the degeneracy of mapping on large deformable volumes ex) liver CT scans Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 11. CycleMorph 11 Proposed Method Cycle Consistency Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint (2017). Motivation model: cycleGAN Horse 𝑭𝑭 𝑹𝑹 Zebra • To adopt cyclic constraint in network training → Improve topology preservation (less degeneracy)
- 13. CycleMorph 13 CycleMorph min GX,GY 𝑳𝑳(𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌) 𝑳𝑳 𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌 = 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋 + 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑌𝑌, 𝑋𝑋, 𝐺𝐺𝑌𝑌 +𝛼𝛼𝑳𝑳𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐(𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌) + 𝛽𝛽𝑳𝑳𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖(𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌) Loss Function
- 14. CycleMorph 14 Loss Function CycleMorph 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟(𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋) = − 𝑇𝑇 𝑋𝑋, 𝜙𝜙𝑋𝑋𝑋𝑋 ⨂𝑌𝑌 + 𝜆𝜆∑ 𝛻𝛻𝜙𝜙𝑋𝑋𝑋𝑋 2 • Registration loss Similarity metric Regularization Local normalized cross-correlation
- 15. CycleMorph 15 CycleMorph 𝑳𝑳𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐(𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌) = 𝑇𝑇 � 𝑌𝑌, � 𝜙𝜙𝑌𝑌𝑌𝑌 − 𝑋𝑋 1 + 𝑇𝑇 � 𝑋𝑋, � 𝜙𝜙𝑋𝑋𝑋𝑋 − 𝑌𝑌 1 Loss Function • Cycle loss: to retain the topology between moving and deformed images
- 16. CycleMorph 16 CycleMorph 𝑳𝑳𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑋𝑋, 𝑌𝑌, 𝐺𝐺𝑋𝑋, 𝐺𝐺𝑌𝑌 = − 𝑇𝑇(𝑌𝑌, 𝐺𝐺𝑋𝑋(𝑌𝑌, 𝑌𝑌) ⨂𝑌𝑌) − 𝑇𝑇(𝑋𝑋, 𝐺𝐺𝑌𝑌(𝑋𝑋, 𝑋𝑋) ⨂𝑋𝑋) Loss Function • Identity loss: to prevent the network from deforming fixed points
- 17. CycleMorph 17 CycleMorph Multiscale image registration • To solve a problem of limitation on GPU memory • CycleMorph can be applied to full-sized images / local patches • Training stage: (global registration) + (local registration)
- 18. CycleMorph 18 CycleMorph Multiscale image registration • Test stage: Deformation only once for the moving image - Get global deformation fields 𝜙𝜙𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔 - Get and fuse patch fields 𝜙𝜙𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 - Obtain local deformation field 𝜙𝜙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 at the fine scale - Compose 𝜙𝜙𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔° 𝜙𝜙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
- 19. CycleMorph 19 Dataset • Moving image = T1 scans from OASIS-3 dataset • Fixed image = Atlas volume Experiment 1 Application to Brain MRI Registration (3D) Moving image Registration Fixed image (atlas)
- 20. CycleMorph 20 Results Experiment 1 Application to Brain MRI Registration (3D)
- 21. CycleMorph 21 Results Experiment 1 Application to Brain MRI Registration (3D)
- 22. CycleMorph 22 Experiment 2 Application to Liver CT Registration (3D) Dataset • Multiphase abdominal CT images (from Asan Medical Center) • Registration images before and after the injection of contrast agents
- 23. CycleMorph 23 Experiment 2 Application to Liver CT Registration (3D) Results (multiscale registration)
- 24. CycleMorph 24 Target registration error & Tumor size measurement Experiment 2 Application to Liver CT Registration (3D) Effect of cycle consistency : less folding problem
- 25. CycleMorph 25 Experiment 3 Application to Face Registration (2D) Dataset • RaFD dataset: 8 types x 3 eye orientation of emotion faces per person • Use all pairs of face images gazing both the same and different directions
- 26. CycleMorph 26 Experiment 3 Application to Face Registration (2D) Results
- 27. CycleMorph 27 Experiment 3 Application to Face Registration (2D) Results
- 28. CycleMorph 28 Experiment 3 Application to Face Registration (2D) Results Quantitative evaluation Study on the consistency
- 29. CycleMorph 29 Medical Image Analysis Thank you.