“When 2D Is Not Enough: An Overview of Optical Depth Sensing Technologies,” a Presentation from Ambarella
0 likes133 views
The document provides an overview of optical depth sensing technologies, focusing on stereo cameras, time of flight (ToF), and structured light modalities. It discusses their operating principles, design considerations, performance metrics, and various applications in fields such as autonomous driving and occupancy sensing. It emphasizes the importance of depth sensing in understanding spatial relationships and highlights the strengths and weaknesses of each technology.
“When 2D Is Not Enough: An Overview of Optical Depth Sensing Technologies,” a Presentation from Ambarella
- 1. © 2021 Ambarella Inc. When 2D Is Not Enough: An Overview of Optical Depth Sensing Technologies Dinesh Balasubramaniam Ambarella
- 2. © 2021 Ambarella Inc. 2 Ambarella Founded on the premise that video is a unique type of data requiring an optimized SoC architecture • DNN AI processors integrated with state-of-the-art video processors yield highly optimized computer vision SoCs Viewing and sensing SoCs with leading performance and efficiency • Low-power and high-performance SoCs for IoT and Automotive camera markets 2004 Ambarella founded 2012 IPO NASDAQ: AMBA 2021 280+ million HD/4K cameras enabled 2015 Acquisition of Santa Clara, CA Company HQ Tokyo Hsinchu Total Headcount Detroit Parma VisLab Shanghai Seoul Munich 785 Shenzhen World-class engineering team • >600 software, algorithm, VLSI and image processing engineers globally AI Vision Processors For Edge Applications
- 3. © 2021 Ambarella Inc. Outline • Why Depth? • Depth Sensing Modalities • Stereo Camera, Time of Flight, and Structured Light • Operating Principles • Design considerations: Resolution, Range, Precision, Bright light and Low-light performance • Review and comparison • Applications 3
- 4. © 2021 Ambarella Inc. Why Depth? 4
- 5. © 2021 Ambarella Inc. Why Depth? 5 Which face is real? What is in the way? Are all objects flat?
- 6. Modalities
- 7. © 2021 Ambarella Inc. Depth Sensing Modalities 7 Sonar Time of Flight module Stereo camera Radar Lidar Structured Light module
- 9. © 2021 Ambarella Inc. Operating Principle – Stereo Camera System = disparity Right Image Left Image
- 10. © 2021 Ambarella Inc. Operating Principle – Stereo Camera System B f dl dr 𝐵 𝐷 = 𝐵 − 𝑑𝑙 − 𝑑𝑟 𝐷 − 𝑓 𝐷 = 𝑓 𝐵 𝑑𝑙 − 𝑑𝑟 B – baseline (distance between optical center) D – distance to be measured f – focal length of lens d – distance of projected point from optical axis • Distance inversely proportional to disparity • Epipolar geometry helps to reduce search space for matching point • Finding matching point is called correspondence problem D
- 11. © 2021 Ambarella Inc. Design Considerations – Stereo Camera Systems • Systems are flexible and scalable – can accommodate many configurations with appropriate change in sensor resolution, field of view (FoV), and baseline. • Depth precision decreases with distance – non-linear error model. • Good performance in bright light conditions. Low-light performance depends on output of image pipeline. • Challenged on scenes with low texture – improved with active illumination. • Manufacturing process can be complicated – alignment of optical units is critical during and after manufacture. Improved with periodic calibration or autocalibration software. 11
- 12. © 2021 Ambarella Inc. Design Consideration – Stereo Camera Systems 12 Stereo Left Stereo Right
- 13. © 2021 Ambarella Inc. Performance – Stereo Camera Systems 13 Stereo Left Stereo Right
- 14. © 2021 Ambarella Inc. Design Considerations – Stereo Camera Systems 14
- 15. Time of Flight
- 16. © 2021 Ambarella Inc. Operating Principle – Time of Flight • Two types of ToF sensors: direct ToF and indirect ToF. • Direct Tof sensors emit a pulse of light and measure the time it takes for the reflection to arrive at the sensor. Pulse strength and duration can be varied to allow long range measurements (up to kms). • Indirect ToFs also called continuous-wave ToFs emit a pulsed, modulated light signal and measure the phase difference of the reflected signal to derive depth.
- 17. © 2021 Ambarella Inc. Operating Principle – Time of Flight Phase Depth Integration time Read out time ∅ = 90 ∅ = 0 ∅ = 180 ∅ = 270
- 18. © 2021 Ambarella Inc. Operating Principle – Time of Flight Depth Amplitude
- 19. © 2021 Ambarella Inc. • Depth range depends on modulating frequency of the IR source - maximum measurable distance for given frequency called unambiguous range. • LEDs can be used up to 30MHz. VCSELs (vertical-cavity surface-emitting laser) can be used if higher modulating frequencies are required (up to 100MHz). Design Consideration– Time of Flight Modulating Frequency (MHz) Unambiguous Range (m) 20 7.5 50 3 75 2 100 1.5
- 20. © 2021 Ambarella Inc. Design Considerations – Time of Flight • Comparatively low resolution (VGA current max) but unaffected by scene texture • Dense depth information where field of view and field of illumination overlap • Range limited by modulating frequency of illuminator. Depth precision decreases with distance for a given modulating frequency. • Good performance in low-light conditions. Sunlight affects bright light performance – improved when using 940 nm IR illuminator. • Scattered reflections and cross talk can impact depth measurement. • Integration time impacts number of fps but can handle motion. • Combining multiple systems can be challenging – can be separated by frequency. 20
- 21. © 2021 Ambarella Inc. Design Considerations – Time of Flight 21 * Data from vendor module
- 22. Structured Light
- 23. © 2021 Ambarella Inc. Operating Principle – Structured Light • Light source plus a pattern generator project a random pattern. • When image plane moves pattern shifts depending on object distance. • Reflected pattern correlated against a reference stored on the system. • Depth depends on disparity between the reference and reflected pattern.
- 24. © 2021 Ambarella Inc. Operating Principle – Structured Light B D f R R1 R2 p1 p2 𝑓 𝑝2 − 𝑝1 = 𝑅 𝑅2 − 𝑅1 𝐷 𝐵 = 𝑅 − 𝐷 𝑅2 − 𝑅1 𝐷 = 𝑅𝐵𝑓 𝑅 𝑝2 − 𝑝1 + 𝐵𝑓 B – baseline (distance between optical center) f – focal length of lens R – distance to reference plane D – distance to be measured 𝑅𝑛 – Point on reference plane 𝑝𝑛 – projected points on image plane
- 25. © 2021 Ambarella Inc. System Considerations – Structured Light • Moderately flexible – sensor resolution, baseline, and pattern density can be varied for different applications. • Depth precision decreases with distance. • Good performance in low-light conditions. Lower performance when sunlight is present – improved when using 940 nm IR illuminator. • Spatial resolution higher than time of flight sensors but depth density depends on projector pattern. • Frame rate depends on sensor and depth processing speed. 25
- 26. © 2021 Ambarella Inc. System Considerations – Structured Light 26
- 27. Comparison
- 28. © 2021 Ambarella Inc. Comparison Stereo Time of Flight Structured Light Depth Range Short to long range Low to mid range Low to mid range Depth Precision mm to cm mm to cm mm to cm Depth Resolution High – can detect finer features or objects further away Up to VGA Pattern dependent Active emitter Optional Required Required Bright light performance Good Fair – improved with 940 nm IR Fair – improved with 940 nm IR Low-light performance Depends on ISP Good Good Cost High Low Mid
- 29. Applications
- 30. © 2021 Ambarella Inc. Applications 30 Long-range stereo for autonomous driving ToF for occupancy sensing Structured light for access control
- 31. Thank You