(Research Note) Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping
2 likes202 views
This document discusses advancements in model-aided monocular visual-inertial state estimation and dense mapping, focusing on addressing issues in global localization, drift, and sensing range. It presents contributions from the multi-state constraint Kalman filter (MSCKF) and introduces solutions for achieving real-time global localization and dense mapping. Future directions include enhancing mapping quality and refining 3D models using onboard visual information.
![Related works (1/5)
Global localization solutions based on place recognition [1][2] can only
obtain topological localization which is not accurate enough for closed-l
oop control
[1] G. Schindler, M. Brown, and R. Szeliski, “City-scale location recog- nition,” in Computer Vi
sion and Pattern Recognition, 2007. CVPR’07. IEEE Conference on. IEEE, 2007, pp. 1–7.
[2] M. Cummins and P. Newman, “Fab-map: Probabilistic localization and mapping in the spa
ce of appearance,” The International Journal of Robotics Research, vol. 27, no. 6, pp. 647–665,
2008.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 3
Problem 1: Global localization](https://image.slidesharecdn.com/01modelaided-180321060826/85/Research-Note-Model-Aided-Monocular-Visual-Inertial-State-Estimation-and-Dense-Mapping-3-320.jpg)
![Related works (4/5)
Computation Map
[10] high Dense
LSD-SLAM low Semi-dense
DTAM high Dense
REMODE high Dense(mono)
3D model based[15] low Dense
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 6](https://image.slidesharecdn.com/01modelaided-180321060826/85/Research-Note-Model-Aided-Monocular-Visual-Inertial-State-Estimation-and-Dense-Mapping-6-320.jpg)
![Related works (5/5)
Multistate constraint Kalman filter (MSCKF) [6] is a light-weight filter-ba
sed solution of fusing visual odometry and IMU data.
[6] A. I. Mourikis and S. I. Roumeliotis, “A multi-state constraint kalman filter for vision-aided
inertial navigation,” in Robotics and automation, 2007 IEEE international conference on. IEEE,
2007, pp. 3565–3572.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 7
MSCKF](https://image.slidesharecdn.com/01modelaided-180321060826/85/Research-Note-Model-Aided-Monocular-Visual-Inertial-State-Estimation-and-Dense-Mapping-7-320.jpg)
(Research Note) Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping
- 1. National Chung Cheng University, Taiwan Robot Vision Laboratory 2018/03/22 Jacky Liu (Research Note) Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping
- 2. About this work Model-Aided Monocular Visual-Inertial State Estimati on and Dense Mapping Kejie Qiu1 , Shaojie Shen1 IROS2017 - IEEE/RSJ International Conference on Intelligent Robots an d Systems 1. Department of Electronic and Computer Engineering, Hong Kong University of Science an d Technology, Hong Kong, China 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 2
- 3. Related works (1/5) Global localization solutions based on place recognition [1][2] can only obtain topological localization which is not accurate enough for closed-l oop control [1] G. Schindler, M. Brown, and R. Szeliski, “City-scale location recog- nition,” in Computer Vi sion and Pattern Recognition, 2007. CVPR’07. IEEE Conference on. IEEE, 2007, pp. 1–7. [2] M. Cummins and P. Newman, “Fab-map: Probabilistic localization and mapping in the spa ce of appearance,” The International Journal of Robotics Research, vol. 27, no. 6, pp. 647–665, 2008. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 3 Problem 1: Global localization
- 4. Related works (2/5) The odometry-based methods suffer from long-term drifting while SLA M-based approaches can not guarantee global consistency before a m ajor loop closures detection. Fusion of odometry, SLAM and GNSS may resolve the localization pro blem in most cases, but it still does not guarantee drift-free localization at all times. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 4 Problem 2: Drift
- 5. Related works (3/5) Depth camera • It has the intrinsic detection limitation impedes outdoor applications. Stereo camera • It has limited baseline constrains detection range. • The extrinsic calibration is another issue for easy use. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 5 Problem 3: Sensing range
- 6. Related works (4/5) Computation Map [10] high Dense LSD-SLAM low Semi-dense DTAM high Dense REMODE high Dense(mono) 3D model based[15] low Dense 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 6
- 7. Related works (5/5) Multistate constraint Kalman filter (MSCKF) [6] is a light-weight filter-ba sed solution of fusing visual odometry and IMU data. [6] A. I. Mourikis and S. I. Roumeliotis, “A multi-state constraint kalman filter for vision-aided inertial navigation,” in Robotics and automation, 2007 IEEE international conference on. IEEE, 2007, pp. 3565–3572. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 7 MSCKF
- 8. Contributions (1/2) 1. GPS not available => 3D model based localization 2. Drift problem => 3D model based localization 3. Sensing range => Monocular temporal stereo 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 8 Solutions
- 9. Contributions (2/2) 1. Handling simultaneously global localization and real- time dense mapping problem with minimum sensing and a rough prior 3D model. 2. Integrating the model-based global localization method with a tightly-coupled visual-inertial fusion method to get all-the-time global localization with high local accuracy. 3. Implementing motion stereo with depth prior rendered form a prior 3D model to realize accurate environment awareness. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 9
- 10. Overview 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 10
- 11. Overview 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 11
- 12. Method 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 12 1. Global pose fusion (MSCKF) 2. Fused state estimation 3. Semi-global matching
- 13. Global pose update for visual-inertial odometry The original visual-inertial odometry (VIO) can already handle local are a autonomous navigation robustly. They use MSCKF(multi-state constraint kalman filter) as the VIO imple mentation that is based on Kalman filter and treat the global pose upda te as an additional EKF update. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 13 Multi-view cost aggregation VIO 3D model global pose All-the-time global- consistent property Global Pose update Cost aggregation Semi-global matching
- 14. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 14 MSCKF parameters Global pose update for visual-inertial odometry Global Pose update Cost aggregation Semi-global matching rotation positionvelocity Bias of gryo and accelemeter frame 1 frame N Kalman gain residual
- 15. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 15 MSCKF parameters Global pose update for visual-inertial odometry Global Pose update Cost aggregation Semi-global matching
- 16. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 16 Global pose update for visual-inertial odometry Observation (sensing) Global Pose update Cost aggregation Semi-global matching MSCKF camera state global observation symbol ⊗ denotes quaternion multiplication
- 17. EKF 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 17 http://ais.informatik.uni-freiburg.de/teaching/ws12/mapping/pdf/slam04-ekf-slam.pdf
- 18. Quaternion Multiplication 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 18 https://www.mathworks.com/help/aeroblks/quaternionmultiplication.html
- 19. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 19 Global pose update for visual-inertial odometry Update (prediction) Global Pose update Cost aggregation Semi-global matching
- 20. Monocular dense mapping with depth prior constrains Different from spatial stereo where only two calibrated views are used for depth e stimation, multiple temporal camera views are used for depth estimation with prec ise pose estimation for every camera frame. The advantage of using multiple temporal camera views: 1. No baseline limitation (camera mounting distance limitation) Therefore, the same depth estimation scheme can be used both small indoor environments and large outdoor cases. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 20 A) Multi-view cost aggregation Cost aggregation Global Pose update Cost aggregation Semi-global matching
- 21. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 21 Global Pose update Cost aggregation Semi-global matching
- 22. Every re-projection pixel is found by back-projecting a pixel in the reference frame to a 3D point and re-projecting this 3D point into the measurement frame. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 22 Reference image measurement image Global Pose update Cost aggregation Semi-global matching
- 23. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 23 The multiplication with the inverse of the came ra matrix gives you a ray along which the 3D point is located. Depth pixel intensityworld frame rotation matrix world frame trans. matrix Global Pose update Cost aggregation Semi-global matching
- 24. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 24 Reference image measurement image Global Pose update Cost aggregation Semi-global matching
- 25. Global Pose update Cost aggregation Semi-global matching 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 25 Δ-Δ Huber loss
- 26. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 26 Huber loss Δ-Δ Global Pose update Cost aggregation Semi-global matching
- 27. Monocular dense mapping with depth prior constrains 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 27 B) Semi-global matching Pixel-wise cost Smoothness constraints (depth diff = 1) Smoothness constraints (depth diff > 1) Global minimization is an NP-complete problem which cannot be solved in polynomial time. Global Pose update Cost aggregation Semi-global matching
- 28. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 28 Global Pose update Cost aggregation Semi-global matching
- 29. Monocular dense mapping with depth prior constrains TSDF (truncated signed distance function) Each 3D voxel contains 1. TSDF Depth 2. Photometric intensity 3. Confidence weight of the measurements. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 29 C) 3D reconstruction
- 30. Experimental results • Implementation detail 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 30 752x480 12Hz 100Hz 160x108 12Hz 3D model constructed by Altizure.com MAV Nvidia Tegra X1 4 CPU 256 GPU core
- 31. Online SfM service Altizure.com 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 31
- 32. State estimation results 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 32 • The comparison of position, orientation of MSCKF and MSCKF+Model. (Motion capture Ground truth)
- 33. State estimation results 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 33 • The comparison of position, orientation of MSCKF and MSCKF+Model. (Motion capture Ground truth)
- 34. Mapping result • Depth prior-based (2nd row) has better mapping performance at the texture-less areas such as the green circular areas. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 34 Motion stereo Depth prior-based
- 35. Object removal (dynamic env.) 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 35
- 36. Conclusion 1. Global localization and dense mapping utilizing a known 3D texture d model 2. Combine several techniques to achieve real-time onboard dense S LAM. 1. Fast model view rendering 2. Image stabilization 3. Edge-based image alignment 4. Global pose fusion 5. Monocular dense mapping 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 36
- 37. Future work 1. Utilizing larger image to enhance mapping quality 2. Refining 3D model online using onboard visual information to acco unt for the differences between the model and the environment. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 37
- 38. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 38 J.Delmerico andD.Scaramuzza, “A Benchmark Comparison of M onocular Visual-Inertial Odometry Algorithms for Flying Robot s,” 2018.
- 39. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 39