[BMVC 2022] DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection
Download as PPTX, PDF0 likes126 views
This document proposes a new domain adaptive class-incremental learning (DA-CIL) paradigm for 3D object detection. The method uses dual-domain copy-paste data augmentation to address data scarcity and domain shifts. It also employs dual-teacher knowledge distillation with multi-level consistency regularization between domains. Experimental results on the ScanNet and SUN RGB-D datasets show the method outperforms other class-incremental learning and domain adaptation baselines, and ablation studies validate the contributions of the dual-domain augmentation and consistency losses.
[BMVC 2022] DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection
- 1. DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection Ziyuan Zhao1,2, Mingxi Xu1,3, Peisheng Qian1, Ramanpreet Singh Pahwa1,2, and Richard Chang1 Presenter: Zhao Ziyuan 1 Institute for Infocomm Research (I2R), A*STAR, Singapore 2 Artificial Intelligence Analytics & Informatics (AI3), A*STAR, Singapore 3 Nanyang Technological University, Singapore Paper ID: 916
- 2. 2 Introduction – Catastrophic Forgetting & Domain Shift - Deep learning-based 3D object detection has received considerable attention in 3D point clouds. - Catastrophic forgetting: the performance of DL models on old classes tends to decrease substantially when trained on novel classes. - Domain Shift: DL models trained on one domain (i.e., source domain) always suffer tremendous performance degradation when evaluated on another domain (i.e., target domain).
- 3. 3 Motivation – Domain Adaptive Class-incremental Learning - Existing CIL and UDA methods are designed for addressing different challenges separately, ignoring that the catastrophic forgetting and domain shift problems concurrently exist in 3D object detection on point clouds of real-world environments. - We aim to leverage labeled old classes on the source domain and labeled new classes on the target domain to close the domain shift and adapt to new classes without forgetting old classes on the target domain in DA-CIL.
- 4. 4 Method – Transformation-consistent Meta-hallucination - We proposed a new CIL paradigm to enable Domain Adaptive Class-Incremental Learning for 3D object detection. - Dual-domain copy-paste (CP) address both in-domain data scarcity and cross-domain distribution shift. - Dual-teacher knowledge distillation (KD) enforces multi-level consistency regularization (MLC).
- 5. 5 Method (1) – Dual-Domain Copy-Paste - To relieve data scarcity and reduce the domain gap at the data level, we extensively leverage copy- paste (CP) augmentation techniques for creating cross-domain and in-domain point clouds.
- 6. 6 Method (2) – Multi-Level Consistency - To facilitate the dual-teacher knowledge transfer, we propose multi-level consistency regularization from two aspects. - Statistics-Level (SC) Consistency ( Batch Normalization parameters alignment) - Bounding Box-Level Consistency ( center-, class and size-level consistency loss) Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Sess: Self-ensembling semi-supervised 3d object detection. CVPR, pp.11079–11087, 2020.
- 7. 7 Method (3) – Dual-Teacher Training - Cross-domain Teacher: The student model learns the underlying knowledge in base classes from a cross-domain teacher via distillation loss (square of Euclidean distance between the classification logits of different bounding boxes). - In-domain Teacher: Meanwhile, the EMA in-domain teacher helps student model capture structure and semantic invariant information in objects with consistency loss - The mixed labels (pseudo base + real novel) are transformed by the same augmentation step that is applied on the augmented source domain to compute a supervised loss with the backbone VoteNet Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. CVPR, pp. 9277–9286, 2019.
- 8. 8 Experimental Results - Dataset - We evaluate our method on 3D object detection datasets, ScanNet (source) and SUN RGB-D (target). - 5 categories (bathtub, bed, bookshelf, chair, desk) were selected as base classes. - 5 additional categories (dresser, nightstand, sofa, table, toilet) in SUN RGB-D were selected as novel classes in the target domain. ScanNet dataset SUN RGB-D dataset
- 9. 9 Experimental Results (1) – Comparison with different methods - For class-incremental learning, we evaluated 2 naive transfer learning baselines Freeze-and-add, and Fine-tune. We also used SDCoT as a strong baseline for CIL. - For unsupervised domain adaptation, We implemented one recent popular self-ensembling method, mean teacher (MT). - Our method outperforms many methods by a clear margin in CIL under domain shift. Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Sess: Self-ensembling semi-supervised 3d object detection. CVPR, pp.11079–11087, 2020.
- 10. 10 Experimental Results (2) – Comparison with different augmentation techniques - We implemented two popular augmentation techniques for comparison, Mix3D and CutMix. - Mix3D: combines 2 point clouds into a mixed point cloud via concatenation, which results in excessive overlaps in the mixed point clouds - CutMix: randomly replaces patches of point clouds with patches from other point clouds, which unnecessarily introduces extra context from the source domain. - Our method outperforms the two augmentation techniques.
- 11. 11 Experimental Results (3) – Ablation Analysis - Without cross-domain copy-paste, the model achieves a lower result in base classes, which proves the effectiveness of cross-domain copy-paste augmentation in base class recognition. - Similarly, it can be inferred from results without in-domain augmentation that in-domain copy-paste can enhance the model performance in novel classes. - Moreover, statistics-level consistency provides around 0.8% performance increment in all classes.
- 12. 12 Experimental Results (4) – Qualitative Comparison (good cases) - Our method can accurately detect both old and novel classes in the target domain, overcoming the domain shift in old classes.
- 13. 13 Experimental Results (4) – Qualitative Comparison (failure cases) - Fail to detect the bookshelf in the point clouds. The failure is due to the loss of the geometric structure of the bookshelf in the point cloud data. - The desk object is misclassified as a table, which is likely due to geometric similarities between the 2 classes. - The bounding box of the desk object deviates from the ground-truth, which can be attributed to the large size and irregular shape of the object.
- 14. 14 Conclusions - We identify and explore a novel domain adaptive class-incremental learning paradigm for 3D object detection. - We propose a novel 3D object detection framework, in which we design a novel dual-domain copy- paste augmentation method to address both in-domain data scarcity and cross-domain distribution shift. - We further enhance the dual-teacher knowledge distillation with multi-level consistency between different domains. - We achieve superior performance in extensive experiments.
- 15. 15 Conclusions - We identify and explore a novel domain adaptive class-incremental learning paradigm for 3D object detection. - We propose a novel 3D object detection framework, in which we design a novel dual-domain copy- paste augmentation method to address both in-domain data scarcity and cross-domain distribution shift. - We further enhance the dual-teacher knowledge distillation with multi-level consistency between different domains. - We achieve superior performance in extensive experiments.
Editor's Notes
- #2: Hello everyone. Welcome to my presentation. My name is Zhao Ziyuan. Here I am presenting our paper titled DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection.
- #3: Deep learning-based 3D object detection has received considerable attention in 3D point clouds. But there are still many challenges in the real scenario. Firstly, the catastrophic forgetting problem seriously limits model performance in class incremental learning (CIL). Secondly, as shown in the figure, there are distribution gaps between two different datasets, and the distribution gap is part of the domain gap. So, in the presence of the domain shift, deep learning models trained on one domain always suffer tremendous performance degradation when evaluated on another domain.
- #4: Existing CIL and UDA methods address different challenges separately, ignoring that the catastrophic forgetting and domain shift concurrently exist in 3D object detection on point clouds of real-world environments. And we aim to leverage labeled old classes on the source domain and labeled new classes on the target domain to close the domain shift and adapt to new classes without forgetting old classes on the target domain in DA-CIL.
- #5: As fig3 shows, our DA-CIL framework contains two steps: First, cross-domain intermediate point clouds and in-domain augmented point clouds are generated by dual-domain copy-paste augmentation. Then, a dual-teacher network is constructed to distill knowledge from these domains for domain adaptive class-incremental learning.
- #6: As shown in fig 4, our dual-domain copy paste first extracts objects from the source domain to populate the ground-truth database. Then objects are copy-pasted into point clouds from the source domain and the target domain to generate the augmented domain and the intermediate domain, respectively.
- #7: To facilitate the dual-teacher knowledge transfer, we propose multi-level consistency regularization from two aspects: Statistics-Level Consistency and Bounding Box-Level Consistency. Statistics-Level Consistency is implemented in the feedforward process and Bounding Box-Level Consistency is calculated as above equations.
- #8: Read slide
- #9: We evaluate our method on 3D object detection datasets, ScanNet and SUN RGB-D. We set 5 categories as base classes and addition 5 categories in SUN RGB-D as novel classes. The above two figures present the gap between the two datasets intuitively.
- #10: We evaluated 2 naive transfer learning baselines Freeze-and-add, and Fine-tune, one recent popular self-ensembling method, Mean Teacher and SDCoT, the co-teaching architecture, as a strong baseline for class-incremental learning. As table 1 shows, our method outperforms many methods by a clear margin in CIL under domain shift.
- #11: Read slide
- #12: To evaluate the effectiveness of different components in our method for CIL under Domain Shift, we perform ablation studies by removing components in our approach. +slide
- #13: As illustrated in left figure, our method is able to accurately detect both old and novel classes in the target domain, overcoming the domain shift in old classes. Moreover, our model consistently outperforms the baseline SDCoT in both scenarios.
- #14: Read
- #15: Read slide
- #16: Read slide