MIPI DevCon Seoul 2018: Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors
1 like266 views
The document discusses the integration of various sensor types, including imaging cameras, lidar, and radar, into automotive, drone, and AR/VR environments through the use of MIPI CSI-2SM standards. It highlights challenges in virtual channel mapping and clocking modes when combining identical and dissimilar sensors, as well as the advantages of using FPGAs for a faster development cycle compared to ASICs. The emphasis is on enabling innovative vision systems by efficiently managing sensor data processing and integration.
MIPI DevCon Seoul 2018: Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors
- 1. Tom Watzka & Satwant Singh Lattice Semiconductor Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors
- 2. © 2018 MIPI Alliance, Inc. 2 Sensors are Proliferating Lattice Semiconductor SemicoResearchCorp. 0 100 200 300 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units 0 500 1000 1500 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units 0 10 20 30 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units Automotive 27% CAGR Drone 27% CAGR VR/AR/MR 166% CAGR
- 3. © 2018 MIPI Alliance, Inc. 3 Sensors in the Automotive Environment Lattice Semiconductor • Imaging cameras: surround view, parking assistance and traffic sign recognition. • Lidar: Emergency braking and collision avoidance. • Short and medium range radar: Traffic alert and blind spot detection. • Longer range radar: Cruise control and early forward vehicle detection. Lots of Dissimilar Sensors
- 4. © 2018 MIPI Alliance, Inc. 4 Sensors in the Drone Environment Lattice Semiconductor • Stereo Vision • Ultrasonic (Sonar) • Time-of-Flight • Lidar • Infrared • Monocular Vision Lots of Dissimilar Sensors
- 5. © 2018 MIPI Alliance, Inc. 5 Sensors in the AR/VR Environment Environmental Understanding Cameras Depth Cameras • Structured Infrared light projector/scanner • RGB Depth cameras • Time-of-flight camera Lattice Semiconductor Lots of Dissimilar Sensors
- 6. © 2018 MIPI Alliance, Inc. 6 • Region of Interest • Down sampling • Virtual Channels VC1 VC2 Lattice Semiconductor Sensors in the Industrial Environment Cameras with Virtual Channel Capability
- 7. © 2018 MIPI Alliance, Inc. 7 Combining Identical Sensors • Sync Sensors • Buffer Lines • Concatenate both lines at 2x frequency Lattice Semiconductor Same “nominal” clock rate
- 8. © 2018 MIPI Alliance, Inc. 8 • Sensors not synchronized • Different Clocks • Output Frequency >= Input 1 + Input 2 • Lines sent out FIFO via Virtual Channels Virtual Channel Output Lattice Semiconductor Combining Dissimilar Sensors Different Data Rates
- 9. © 2018 MIPI Alliance, Inc. 9 MIPI CSI-2SM Virtual Channels (VC) Lattice Semiconductor
- 10. © 2018 MIPI Alliance, Inc. 10 MIPI CSI-2SM v1.2/v1.3 MIPI CSI-2SM v2.0 KEY issue: difference in supported MIPI CSI-2SM version • MIPI CSI-2SM v1.2/v1.3 - only supports up to 4 slaves (2-bit VC) • MIPI CSI-2SM v2.0 - can support up to 16 or 32 slaves (4-bit VC for MIPI CSI-2SM over D- PHY , 5-bit VC for MIPI CSI-2SM over C-PHY) Lattice Semiconductor Challenges – Virtual Channel (VC) Fields Different VC Mappings between Different MIPI CSI-2SM Versions
- 11. © 2018 MIPI Alliance, Inc. 11 Challenges – VC Fields If there are no more than 4 input channels: • Bridge can perform simple mapping to MIPI CSI-2SM V1.2. If there are more than 4 input channels: • Bridge can perform simple mapping to MIPI CSI-2SM V2.0. • If processor cannot support MIPI CSI-2SM V2.0: • Must merge similar sensors into one stream MIPI CSI-2SM V1.2 VC 1&2 Bridge AP/ISP Bridge AP/ISP Lattice Semiconductor MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1,2,3,4 MIPI CSI-2SM V1.2 VC 1,2,3,4,5,6
- 12. © 2018 MIPI Alliance, Inc. 12 Continuous Clock Mode: • Simple Clocking Design • High power consumption Non-Continuous Mode: • Reference Clock Needed • Lower Power Consumption MIPI D-PHYSM Rx MIPI CSI-2SM MIPI CSI-2SM RefClk Continuous Mode Lattice Semiconductor Challenges – Clocking Continuous vs. Non Continuous Modes for MIPI CSI-2SM MIPI D-PHYSM Tx MIPI D-PHYSM Rx MIPI CSI-2SM MIPI CSI-2SM MIPI D-PHYSM Tx RefClk MIPI CSI-2SM Non-Continuous Mode MIPI CSI-2SM
- 13. © 2018 MIPI Alliance, Inc. 13 Word Clock Domain N Internal Clock Domain Internal Clock Domain D-PHY Clock Domain 0 Challenges – Multiple Clock Domains Lattice Semiconductor Input Buffer 0 FIFO : D-PHY Clock Domain N Input Buffer N FIFO Identical Sensors • Can use common reference clock • Separate Clock Domains • FIFO Resolves Meta-Stability • Internal Circuitry employs simple buffering adjusting for clock phasing Word Clock Domain 0D-PHY Clock Domain 0 Input Buffer 0 FIFO Sensor 0 Logic : D-PHY Clock Domain N Input Buffer N FIFO Internal Buffer 0 FIFO Internal Buffer N FIFO Sensor N Logic : Sensor 0 Logic Sensor N Logic Dissimilar Sensors (or cameras with different reference clocks) • First clock domain referenced to source clock • Second clock domain referenced to word clock. • Third clock domain referenced to internal clock
- 14. © 2018 MIPI Alliance, Inc. 14 Lattice Device Down Sample MIPI CCISM Manager VC Expand VC Combine + Frame ID MIPI CCISM ROI MIPI CSI-2SM Auto Image Adjust Optical VC1&2 IR Radar Lattice Semiconductor Virtual Channel Pre-Processing in Hardware Balancing Power and Performance MIPI CSI-2SM MIPI CSI-2SM MIPI CSI-2SM
- 15. © 2018 MIPI Alliance, Inc. 15 Why FPGAs? Lattice Semiconductor Design RTL Simulate Synthesize Debug FPGA Days to Weeks Dev Cycle Masks Fabrication Testing Qual Production ASIC FPGA ASIC Months to Years Dev Cycle • Product Development Cycle • Faster time to Market • Lower NRE • Enable Innovation • Size • Pin constrained – same as ASIC • Power • Largely a function of Interfaces Fewer and fewer designs are using ASICs
- 16. © 2018 MIPI Alliance, Inc. 16 • MIPI Camera WG: https://members.mipi.org/workgroup/join/154 • MIPI CSI-2SM V2.1: https://members.mipi.org/wg/All- Members/document/folder/11134 ADDITIONAL RESOURCES Lattice Semiconductor