Multi sensor calibration by deep learning
0 likes480 views
The document discusses various deep learning approaches for multimodal sensor calibration, including RegNet, CalibNet, RGGNet, CalibRCNN, LCCNet, and CFNet. Each method utilizes different neural network architectures to achieve real-time extrinsic calibration between lidar and camera systems, aiming to improve accuracy and reduce reliance on calibration targets. The proposed systems show significant advancements in calibration error reduction compared to traditional methods, with detailed descriptions of their architectures and performance metrics provided.
Multi sensor calibration by deep learning
- 1. Multi-Sensor Calibration by Deep Learning Yu Huang Sunnyvale, California Yu.huang07@gmail.com
- 2. Outline • RegNet: Multimodal Sensor Registration Using Deep Neural Networks • CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks • RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model • CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints • LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network • CFNet: LiDAR-Camera Registration Using Calibration Flow Network
- 3. RegNet: Multimodal Sensor Registration Using Deep Neural Networks • RegNet, the deep CNN to infer a 6 DOF extrinsic calibration between multimodal sensors, exemplified using a scanning LiDAR and a monocular camera. • Compared to existing approaches, RegNet casts all 3 conventional calibration steps (feature extraction, feature matching and global regression) into a single real-time capable CNN. • It does not require any human interaction and bridges the gap between classical offline and target-less online calibration approaches as it provides both a stable initial estimation as well as a continuous online correction of the extrinsic parameters. • During training, randomly decalibrate our system in order to train RegNet to infer the correspondence between projected depth measurements and RGB image and finally regress the extrinsic calibration. • Additionally, with an iterative execution of multiple CNNs, that are trained on different magnitudes of decalibration, it compares favorably to state-of-the-art methods in terms of a mean calibration error of 0.28◦ for the rotational and 6 cm for the translation components even for large decalibrations up to 1.5 m and 20◦ .
- 4. RegNet: Multimodal Sensor Registration Using Deep Neural Networks It estimates the calibration btw a depth and an RGB sensor. The depth points are projected on the RGB image using an initial calibration Hinit. In the 1st and 2nd part of the network, use NiN blocks to extract rich features for matching. The final part regresses decalibration by gathering global info. using two FCLs. During training φdecalib is randomly permutated resulting in different projections of the depth points.
- 5. RegNet: Multimodal Sensor Registration Using Deep Neural Networks
- 6. CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks • CalibNet: a self-supervised deep network capable of automatically estimating the 6-DoF rigid body transformation between a 3D LiDAR and a 2D camera in real-time. • CalibNet alleviates the need for calibration targets, thereby resulting in significant savings in calibration efforts. • During training, the network only takes as input a LiDAR point cloud, the corresponding monocular image, and the camera calibration matrix K. • At train time, no impose direct supervision (i.e., no directly regress to the calibration parameters, for example). • Instead, train the network to predict calibration parameters that maximize the geometric and photometric consistency of the input images and point clouds. • CalibNet learns to iteratively solve the underlying geometric problem and accurately predicts extrinsic calibration parameters for a wide range of mis-calibrations, without requiring retraining or domain adaptation. • Code: https://github.com/epiception/CalibNet.
- 7. CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks Input RGB image (a), a raw LiDAR point cloud (b), and outputs a transformation T that best aligns the two inputs. (c) the colorized point cloud output for a mis-calibrated setup, and (d) the output after calibration using CalibNet.
- 8. CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks Network architecture
- 9. CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks
- 10. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model • With increasing popularity of deep learning (DL), a few recent efforts have demonstrated the advantages of DL for feature extraction on this task. • However, their reported performances are not sufficiently satisfying yet. • One improvement can be the formulation with proper consideration of the underneath geometry. • Besides, existing online calibration methods focus on optimizing the calibration error while overlooking the tolerance within the error bounds. • To address the research gap, a DL-based LiDAR-camera calibration method, named as the RGGNet, is proposed by considering the Riemannian geometry and utilizing a deep generative model to learn an implicit tolerance model. • The code available at https://github.com/KleinYuan/RGGNet.
- 11. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model The architecture of the proposed RGGNet.
- 12. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model The architecture of the Tolerance Regularizer. VAE encoders contain four Conv-ReLU-BN layers with depths as 64/128/64/32, following with a FC-ReLU-BN layer with 1024 units and one FC layer with 200 units; decoders contain two FC-ReLU-BN FC layers with units as 200 and (HW ) 8 , three Deconv-ReLU-BN layers with depths as 32/64/128 and output Deconv-Sigmoid layer with 5 units. Kernel size of 4 and stride size of 2 are set for all convolution and deconvolution layers.
- 13. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model The intensity depth map and the point map are projected by the point clouds with the calibration parameters (K, Y ). Each feature map corresponds to an RGB image. Examples of RGGNet predictions. Reference objects for calibration are shown in bounding boxes.
- 14. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model One bad prediction example with the same input fromT2b test. a: initial de-calibrated input; b: the ground-truth; c: the prediction from -RGGNet, yielding a 0.0550 se3 error; d: the prediction from RGGNet, yielding a 0.060 se3 error. Images are cropped for demonstration.
- 15. RGGNet: Tolerance Aware LiDAR-Camera Online Calibration with Geometric Deep Learning and Generative Model TABLE I PERFORMANCE COMPARISONS WITH DNN BASED METHODS
- 16. CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints • Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. • Different from the existing methods, 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. • The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. • Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset.
- 17. CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints Architecture of CalibRCNN. (a) the RGB images of consecutive frames; (b) depth maps of consecutive frames generated from the mis-calibrated point clouds. By projecting 3D point cloud to the corresponding 2D RGB image we can obtain images similar to (c) and (d), where the color of the projected point represents its depth value. (c) is a 3D-2D projection image converted by mis-calibration parameters, while (d) is the projection image after calibration using our network. Red rectangles show the difference before and after calibration.
- 18. CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints Architecture of the proposed Calibration Recurrent Convolutional Neural Network.
- 19. CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints Pose transformation relationship diagram of continuous frames of LiDAR and camera data
- 20. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network • It is online LiDAR-Camera Self-calibration Network (LCCNet), different from the previous CNN-based methods. • LCCNet can be trained end-to-end and predict the extrinsic parameters in real-time. • In the LCCNet, exploit the cost volume layer to express the correlation between the RGB image features and the depth image projected from point clouds. • Besides using the smooth L1-Loss of the predicted extrinsic calibration parameters as a supervised signal, an additional self-supervised signal, point cloud distance loss, is applied during training. • Instead of directly regressing the extrinsic parameters, predict the decalibrated deviation from initial calibration to the ground truth. • The calibration error decreases further with iterative refinement and the temporal filtering approach in the inference stage. • The execution time of the calibration process is 24ms for each iteration on a single GPU. • LCCNet achieves a mean absolute calibration error of 0.297cm in translation and 0.017◦ in rotation with miscalibration magnitudes of up to 1.5m and 20◦ on the KITTI-odometry dataset, which is better than the state-of-the-art CNN-based calibration methods. • The code available at https://github.com/LvXudong-HIT/LCCNet.
- 21. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network The proposed LCCNet takes the RGB and the projecte depth image as inputs to predict the extrinsic parameters between the LiDAR and the camera. The point clouds are reprojected b the predicted extrinsic parameters. The reprojected depth image and the RGB image will be the subsequent inputs of the LCCNet This process is called iterative refinement. After five time iterative refinements, we obtain the final extrinsic parameters estimation.
- 22. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network The network takes an RGB image from a calibrated camera and a projected sparse depth image from a mis-calibrated LiDAR as input. The output is a 6-DoF rigid-body transformation that represents the deviation between the initial extrinsic and the ground truth extrinsic.
- 23. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network (a) Initial Calibration, (b) Ground truth, (c) Calibration results.
- 24. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network
- 25. CFNet: LiDAR-Camera Registration Using Calibration Flow Network • An online LiDAR-camera extrinsic calibration algorithm that combines the DL and the geometry methods. • Define a two-channel image named calibration flow to illustrate the deviation from the initial projection to the ground truth. • EPnP algorithm within the RANSAC scheme is applied to estimate the extrinsic parameters with 2D-3D correspondences constructed by the calibration flow. • A semantic initialization with the introduction of instance centroids (ICs). • The code available at https://github.com/LvXudong-HIT/CFNet.
- 26. CFNet: LiDAR-Camera Registration Using Calibration Flow Network (First Row) The 2D instance centroid (2D- IC) extracted from images. (Second Row) The 3D instance centroid (3D-IC) extracted from LiDAR point clouds. (Third Row) The initial calibration parameters provided by semantic initialization (registration of 2DIC and 3D-IC). (Fourth Row) The final calibration parameters predicted by CFNet. (Bottom Row) Three- dimensional map generated by fusing the sensors using calibration values provided by our approach.
- 27. CFNet: LiDAR-Camera Registration Using Calibration Flow Network CFNet, an automatic online extrinsic calibration method that estimates the transformation parameters between 3D LiDAR and 2D camera.
- 28. CFNet: LiDAR-Camera Registration Using Calibration Flow Network The Architecture of calibration network CFNet
- 29. CFNet: LiDAR-Camera Registration Using Calibration Flow Network
- 30. CFNet: LiDAR-Camera Registration Using Calibration Flow Network
- 31. CFNet: LiDAR-Camera Registration Using Calibration Flow Network