Unsupervised Deformable Image Registration Using Cycle-Consistent CNN
- 1. Unsupervised Deformable Image Registration Using Cycle-Consistent CNN MICCAI 2019 Boah Kim, Jieun Kim, June-Goo Lee, Dong Hwan Kim, Seong Ho Park, and Jong Chul Ye
- 2. Image registration 2 Introduction Medical Image Registration MRI Abdominal CT Deforming data into one coordinate system • subjects / time / modalities / ... Du, Juan, et al. "Intensity-based robust similarity for multimodal image registration." International Journal of Computer Mathematics 83.1 (2006): 49-57. Fundamental task to analyze data • Tumor volumetry studies • Multimodal information fusion • Therapy planning PET
- 3. Image registration 3 Background Classical iterative method Deep-learning-based method Supervised learning Method Unsupervised learning method Medical Image Registration Methods
- 4. Image registration 4 • Deformation field A vector field of all displacement vectors for all coordinate in images Background Classical Iterative Method Deep-learning-based method • Transformer Grid sampling to warp moving image into fixed image 𝑥𝑥 𝑦𝑦 Floating image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Transformer
- 5. Image registration 5 Background 𝑳𝑳 𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝜙𝜙 = 𝑳𝑳𝒔𝒔𝒔𝒔𝒔𝒔 𝑇𝑇 𝑥𝑥; 𝜙𝜙 , 𝑦𝑦 + 𝑳𝑳𝒓𝒓𝒓𝒓𝒓𝒓(𝜙𝜙) Deep-learning-based method 𝑥𝑥 𝑦𝑦 Floating image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Transformer Cost function • 𝑳𝑳𝒔𝒔𝒔𝒔𝒔𝒔: Similarity cost function • 𝑳𝑳𝒓𝒓𝒓𝒓𝒓𝒓: Regularization function Classical Iterative Method
- 6. Image registration 6 Background Advantages Pitfalls • Preserve topology between two different images • sufficient iteration / parameter tuning • Require substantial time, extensive computation Deep-learning-based method 𝑥𝑥 𝑦𝑦 Floating image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Transformer Classical Iterative Method
- 7. Image registration 7 Background Classical iterative method Deep-learning-based Method 𝑥𝑥 𝑦𝑦 Floating image Fixed image 𝜙𝜙 Deformation field 𝑇𝑇(𝑥𝑥; 𝜙𝜙) Deformed image 𝑇𝑇 Transformer Deep neural network Supervised learning Unsupervised learning
- 8. Image registration 8 Background Require the ground-truth registration fields Cao. et al. “Non-rigid Brain MRI Registration Using Two-Stage Deep Perceptive Networks.” ISMRM 2018 Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 9. Image registration 9 Background Limitation • Difficult to obtain the real ground-truth in practice • Depend on the quality of the ground-truth registration fields Advantages • No parameter tuning for the inference • Applicable to various image domains Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 10. Image registration 10 Background Does not require any ground-truth label Balakrishnan. et al. “An unsupervised learning model for deformable medical image registration,.” CVPR 2018l • Spatial transformer network = Deformation field generator + Transformer • To provide deformable registration without labels for registration fields • Pitfalls: Potential for the degeneracy of mapping on large deformable volumes ex) liver CT scans Supervised learning Unsupervised learning Classical iterative method Deep-learning-based Method
- 11. Image registration 11 Proposed Method Cycle Consistency Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint (2017). Motivation model: cycleGAN Horse 𝑭𝑭 𝑹𝑹 Zebra • To adopt cyclic constraint in network training → Improve topology preservation (less degeneracy)
- 12. Image registration 12 Proposed Method Overall Framework Cycle consistency Cycle consistency
- 13. Image registration 13 Proposed Method min GA,GB ∑𝑥𝑥∈𝐴𝐴 ∑𝑦𝑦∈𝐵𝐵 𝑳𝑳(𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴, 𝐺𝐺𝐵𝐵) 𝑳𝑳 𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴, 𝐺𝐺𝐵𝐵 = 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴 + 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑦𝑦, 𝑥𝑥, 𝑇𝑇, 𝐺𝐺𝐵𝐵 +𝛼𝛼𝑳𝑳𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐(𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴, 𝐺𝐺𝐵𝐵) + 𝛽𝛽𝑳𝑳𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖(𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴, 𝐺𝐺𝐵𝐵) Loss Function
- 14. Image registration 14 Loss Function Proposed Method 𝑳𝑳𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 = − 𝑇𝑇(𝐴𝐴, 𝜙𝜙𝐴𝐴𝐴𝐴⨂𝐵𝐵 + 𝜆𝜆 𝜙𝜙𝐴𝐴𝐴𝐴 2 • Registration loss Similarity metric Regularization 𝒙𝒙 𝒚𝒚 − 𝑇𝑇(𝐵𝐵, 𝜙𝜙𝐵𝐵𝐵𝐵⨂𝐴𝐴 + 𝜆𝜆 𝜙𝜙𝐵𝐵𝐵𝐵 2
- 15. Image registration 15 Proposed Method 𝑳𝑳𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 = 𝑇𝑇 � 𝐵𝐵, � 𝜙𝜙𝐵𝐵𝐵𝐵 − 𝐴𝐴 1 + 𝑇𝑇 ̂ 𝐴𝐴, � 𝜙𝜙𝐴𝐴𝐴𝐴 − 𝐵𝐵 1 Loss Function 𝑨𝑨 𝑩𝑩 • Cycle loss � 𝑩𝑩 � 𝑨𝑨
- 16. Image registration 16 Proposed Method 𝑳𝑳𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥, 𝑦𝑦, 𝑇𝑇, 𝐺𝐺𝐴𝐴, 𝐺𝐺𝐵𝐵 = − 𝑇𝑇(𝐴𝐴, 𝐺𝐺𝐴𝐴𝐴𝐴(𝐴𝐴, 𝐴𝐴) ⨂𝐴𝐴 − 𝑇𝑇(𝐵𝐵, 𝐺𝐺𝐵𝐵𝐵𝐵(𝐵𝐵, 𝐵𝐵) ⨂𝐵𝐵 Loss Function 𝒚𝒚 𝒙𝒙 • Identity loss
- 17. Image registration 17 Experiment Application to Liver CT Registration (3D) Dataset • Multiphase abdominal CT images (from Asan Medical Center) • Does not have ground-truth registration fields
- 18. Image registration 18 Original image Deformed image Unenhanced Arterial Portal Delay Experiment Application to Liver CT Registration (3D)
- 19. Image registration 19 Original image Deformed image Unenhanced Arterial Portal Delay Experiment Application to Liver CT Registration (3D)
- 20. Image registration 20 Original image Deformed image Unenhanced Arterial Portal Delay Experiment Application to Liver CT Registration (3D)
- 21. Image registration 21 Original image Deformed image Unenhanced Arterial Portal Delay Experiment Application to Liver CT Registration (3D)
- 22. Image registration 22 Artery phase → Portal phase Original artery phase Moved artery phase Fixed portal phase Experiment Application to Liver CT Registration (3D)
- 23. Image registration 23 Tumor size measurement & Target registration error Experiment Application to Liver CT Registration (3D) Effect of cycle consistency : less folding problem
- 24. Image registration 24 Conclusion Advantages of Proposed Method • Does not require the ground-truth of deformation fields • Faster time for image registration • Topology preservation for forward and backward mapping • 3D image registration for any pair of images from a single network • Applicable to challenging tasks Unsupervised learning Cycle consistency