"Using Inertial Sensors and Sensor Fusion to Enhance the Capabilities of Embedded Vision Systems," a Presentation from Sensor Platforms
3 likes889 views
The document discusses advancements and challenges in embedded vision technologies, emphasizing a shift from mobile to wearable devices and the need for improved accuracy in various applications like augmented reality and indoor navigation. It highlights the importance of sensor fusion by combining inertial and vision systems to overcome limitations such as scale issues and environmental constraints. The content also outlines the use of Kalman filters for state estimation and the significance of context and motion constraints in enhancing performance.
![• Noise Random walk
– Integrating noise causes a linear walk.
6/2/2014 Sensor Platforms Proprietary and Confidential Information 16
Measured Acc error [m/s/s] for an accelerometer when sitting still.](https://image.slidesharecdn.com/10embeddedvisionalliancemembermeetingmay2014sensorsshaw-150707185134-lva1-app6891/85/Using-Inertial-Sensors-and-Sensor-Fusion-to-Enhance-the-Capabilities-of-Embedded-Vision-Systems-a-Presentation-from-Sensor-Platforms-16-320.jpg)
![• State estimation from visual & inertial sources
– Combined measurements & physical models
6/2/2014 26
],,,,ˆ,[ gak bbvqpx ](https://image.slidesharecdn.com/10embeddedvisionalliancemembermeetingmay2014sensorsshaw-150707185134-lva1-app6891/85/Using-Inertial-Sensors-and-Sensor-Fusion-to-Enhance-the-Capabilities-of-Embedded-Vision-Systems-a-Presentation-from-Sensor-Platforms-26-320.jpg)
![• State estimation from visual & inertial sources
• Kalman filter
– Recursive linear quadratic estimator
– Combined measurements & physical models
6/2/2014 27
],,,,ˆ,[ gak bbvqpx ](https://image.slidesharecdn.com/10embeddedvisionalliancemembermeetingmay2014sensorsshaw-150707185134-lva1-app6891/85/Using-Inertial-Sensors-and-Sensor-Fusion-to-Enhance-the-Capabilities-of-Embedded-Vision-Systems-a-Presentation-from-Sensor-Platforms-27-320.jpg)
![6/2/2014 29
[Weiss & Siegwart, 2007]](https://image.slidesharecdn.com/10embeddedvisionalliancemembermeetingmay2014sensorsshaw-150707185134-lva1-app6891/85/Using-Inertial-Sensors-and-Sensor-Fusion-to-Enhance-the-Capabilities-of-Embedded-Vision-Systems-a-Presentation-from-Sensor-Platforms-29-320.jpg)
"Using Inertial Sensors and Sensor Fusion to Enhance the Capabilities of Embedded Vision Systems," a Presentation from Sensor Platforms
- 1. Kevin A. Shaw, Ph.D. Chief Technology Officer March 30th, 2014 Santa Clara, California, USA 2014 Embedded Vision Member Meeting kshaw@sensorplatforms.comR2
- 2. • Consumer games – More stable attitude • Augmented Reality (AR) – Needs improvement, better accuracy • Indoor Navigation – Need better accuracy; lower power • Hyper photography – Super resolution; intraframe deblur • Robotics – Visual odometry to detect egomotion – Always need better accuracy 6/2/2014 2
- 3. • Construction equipment – Perimeter safety • Context awareness – Understanding users better • Change from mobile to wearables – Shift from mostly-pocket to always-visual – Digital eyewear makes a big difference • Natural interfaces – Using the same wealth of information as humans do to understand the world 6/2/2014 3
- 4. • Change to always-on consumer vision products 6/2/2014 4
- 5. • How to solve some of limitations of vision systems using some of these sensors • Some limitations: – Lack of metric scale • The Dollhouse problem – Pose stability • Feature point robustness – Power consumption • Suitable for mobile products? 6/2/2014 5
- 6. 6/2/2014 6 Accelerometer Gyroscope Magnetometer Barometer Proximity Amb. Light sensor GPS WiFi Bluetooth GSM/CDMA Cell NFC Camera (front) Touch screen Camera (back) 20 sensors! Humidity Colorimeter CO2/VOC gas Microphones x 3 Fingerprint Thermal ambient
- 8. • What are they? – MEMS are tiny silicon structures – MicroElectroMechanical Systems – Leveraging semiconductor toolsets Bosch 6/2/2014 8
- 9. • Measures dynamic acceleration – Very versatile. – Result: Vibration, tilt, & position – Low power 1-10uA 6/2/2014 9 Hydrogen atom = 1Å MEMS displacement Resolution ~ 0.1Å
- 10. • Used to measure rotation • Absolute orientation reference for gyroscope • Power is moderate: 300-1500uA 6/2/2014 10
- 11. • Gyros don’t measure angle! – They measure the rate of change – Body rates: rotation about each axis • Rates are relative to starting point – Depend on Accel/Mag for start • Integrate to get angle • Power is high: 1-5mA or more Gyro SEM 𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0 6/2/2014 11
- 12. • Measures air pressure • Air pressure indicates altitude • Not good for absolute • Resolution of 1-2 feet • Low power: 1-5uA Melexis.com 6/2/2014 12
- 13. Sensor Fusion 13
- 14. • We want to know Position and Attitude (pose). – Inertial and Vision systems can each help find this 6/2/2014 14 𝑃𝑜𝑠𝑒 = 𝑝, 𝑞 = 𝑥, 𝑦, 𝑧, 𝑞0, 𝑞1, 𝑞2, 𝑞3,
- 15. • Position seems easy: double integrate • Angle is only a single integration. • No problem! 6/2/2014 15 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = 𝑣 𝑡 𝑑𝑡 + 𝑝0 𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 = 𝑎 𝑡 𝑑𝑡 + 𝑣0 𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0
- 16. • Noise Random walk – Integrating noise causes a linear walk. 6/2/2014 Sensor Platforms Proprietary and Confidential Information 16 Measured Acc error [m/s/s] for an accelerometer when sitting still.
- 17. • Noise Random walk – Integrating noise causes a linear walk. 6/2/2014 Sensor Platforms Proprietary and Confidential Information 17 dttatv )()( Measured)( ta
- 18. 6/2/2014 Sensor Platforms Proprietary and Confidential Information 18 dttvtp )()( dttatv )()( Measured)( ta
- 19. • Gravity gets in the way • And its big. • Need to subtract it off, but…. 6/2/2014 19
- 20. • Dead Reckoning – Over the past few years significant progress has been made – Stable solutions with consumer grade sensors – Graph (right) uses stock sensors on Galaxy S3 – Pedestrian walking constraints aid solution 6/2/2014 20 WaypointsMeasured Path
- 21. • Visual odometry – Visually tracking position (camera pose) through a space • Tracking feature points is a powerful way to understand the world 6/2/2014 21
- 22. • Limitations – Can't tell size of objects: i.e. scale • Doll house problem – Hard to map points between frames over time – Need cohesion over long time scales – Need robustness in dark spaces & low-texture surfaces – Need maintain vision lock (can get lost due to motion-blur) – Enormous computational load (ready for mobile?!?) • Can we aid the solution with more sensors? 6/2/2014 22
- 23. • Attitude estimates allow anticipated search space – Reduce computation for FP correspondence • Power reduction with reduced/opportunistic frame rates – Can trust INS when not moving – Or when spatial diversity is low • Vision System can be turned on only when high resolution navigation/alignment is needed – PDR to Statue, VS for precise AR overlay 6/2/2014 23
- 24. • Need to find metric scale – Without it the world makes no sense • Monocular / Binocular issues 6/2/2014 24 If you were a vision system, which one is real?
- 25. • Need some way to get extra scale information – Binocular cameras (like humans do; monocular is cheaper) – Reference object (hard to keep in sight; i.e. Ikea catalog) – Location estimates (GPS is not available indoors) – Mapped landmark (best but hard; humans do this) – Inertial estimates (tend to drift, but commonly available) – Depth cameras 6/2/2014 25
- 26. • State estimation from visual & inertial sources – Combined measurements & physical models 6/2/2014 26 ],,,,ˆ,[ gak bbvqpx
- 27. • State estimation from visual & inertial sources • Kalman filter – Recursive linear quadratic estimator – Combined measurements & physical models 6/2/2014 27 ],,,,ˆ,[ gak bbvqpx
- 28. – Closely coupled KF – Solve it all at once • Computationally expensive (Order n2 or n3) – Loosely coupled KF • Estimate visual delta-pose • Estimate inertial delta-pose • Combine with KF • But loose cross-correlations 6/2/2014 28
- 29. 6/2/2014 29 [Weiss & Siegwart, 2007]
- 30. – Recursive vs Batch solution • Kalman Filters are “recursive”; only one frame deep – Batch solution: • Compute solution across multiple frames • Bundle Adjustment with well selected keyframes • Much more stable, but computationally expensive • Asynchronous to frame updates; non-uniform keyframes 6/2/2014 30
- 31. – Consumer sensors are cheap • Free: they are already in place • Contextual & Motional – Need additional constraints • Contextual constraints – Are you moving? » Easy for robots » Harder for humans » Not only for robots anymore • Motion constraints – Wheel constraints help (only on flat ground with no slippage) – Pedestrian constraints (track steps, distance & direction) 6/2/2014 31
- 32. Demonstration 1: Ideal Vision & PDR Scenario ● Vision-only: 5%, PDR-only: 3%, Fused: 1.5% ● Use case: head-mounted AR, vision mapping Initial scale estimate Sensor Platforms, Inc. Confidential and Proprietary 32 End Start
- 33. Demonstration 2: Non-Ideal Vision/Ideal PDR ● Vision-only: 15%, PDR-only: 2%, Fused: 1% ● Use case: AR over large spaces Vision Outage Initial scale estimate Vision Outage Sensor Platforms, Inc. Confidential and Proprietary 33 Start End
- 34. Demonstration 3: Non-Ideal Vision/Non-Ideal PDR ● Vision-only: 8%, PDR-only: 15%, Fused: 5% ● Use case: intensive gaming Initial scale estimate PDR Outage Sensor Platforms, Inc. Confidential and Proprietary 34 Start End
- 35. • Simultaneous Location & Mapping 6/2/2014 35
- 36. • Optical Image Stabilization (OIS) – Optical (in-lens) with inertial attitude tracking; gyro based • Super resolution – Across multiple frames stabilized with pose tracking • Deblurring – Within frame pose-detection and deconvolution 6/2/2014 36
- 37. – Across multiple frames stabilized with pose tracking – Inertial data stabilizes the solution 6/2/2014 37
- 38. – Within frame pose-detection and deconvolution 6/2/2014 38