MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems
5 likes3,139 views
The document discusses the increasing demand for multi-camera and sensor fusion systems in various applications such as mobile, IoT, and automotive. It highlights advancements in MIPI CSI-2 interfaces and their role in enhancing imaging performance, including the requirements for processing data from multiple sensors. Key topics include the evolution of CSI-2 technology, its application in advanced driver-assistance systems, and considerations for functional safety and system architecture.
MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems
- 1. MIPI CSI-2℠ CSI-2℠ Application for Vision and Sensor Fusion Systems Richard Sproul – Cadence Design Systems, IP Architect
- 2. Overview • The expanding demand for imaging and vision based systems in mobile, IoT and automotive products is creating the need for multi-camera and sensor fusion systems to look for novel ways to gather and process multiple camera/sensor data streams whilst still fitting into the mobile interface. • The presentation will highlight some of the key details and requirements for a system with image processing of a multi-camera/sensor system. 2
- 3. CSI-2 Application for Multi-Sensor Systems Multi-Camera Applications • Imaging applications are everywhere • Mobile Phone – Selfie Picture in Picture – Gesture Recognition • Video Games – Gesture Recognition • Autonomous Driving – Pedestrian Detection – Signage Recognition – Night Vision – Parallel Parking! • In-Car Control – Gesture Recognition
- 4. 4 More than the naked eye… CSI-2 Application for Multi-Sensor Systems Machine Vision
- 5. 5 CSI-2 Application for Multi-Sensor Systems Camera Applications Op0mal pathway for mul0ple forward-looking advancements in imaging – Key Drivers: Health, Convenience, Security, Lifestyle, Efficiency – High-perf pixel conduit needs met with C/D-PHY advancements – Broad defini0ons and fuzzy range: (i.e. Wearable: Near Body, On Body, In Body) • Explore possibili0es of overlap between Imaging and low-speed sensor requirements and solu0ons Camera Controller Interface (CCI/CCS) advancement considera0ons: - Point-to-Point and Mul0-Drop configura0ons - Energy consumed / Gb transfer - Limit latency for VB & HB - Precision Timing & Sync - Independent Transport: Pixel Data & Control - Channel Integrity (Error Detec0on) - FW Upload (ISP, Neural)
- 6. • Options for physical interface • Pins, legacy, bandwidth 6 CSI-2 Application for Multi-Sensor Systems MIPI CSI-2 Interfaces
- 7. • Evolution of the CSI-2 7 CSI-2 Application for Multi-Sensor Systems CSI-2 Generations
- 8. 8 CSI-2 Application for Multi-Sensor Systems CSI-2 Performance
- 9. • CSI-2 packets V1.x to V2.0 9 LP LP LP LP Transi0on between packets to LP state for PHY data lane (100ns) Transi0on between packets by using filler pa_erns CSI-2 Application for Multi-Sensor Systems CSI-2 Packet Structure
- 10. • Improve the effective bandwidth 10 0 20 40 60 80 100 120 1000 1250 1500 1800 2000 2250 2500 Frame Rate fps Bit Rate (Mbps) CSI-2 Frame Rate Improvement V1.x to V2.x 1920x1080 RAW12 FPS (V1.x) FPF (V2.x) CSI-2 Application for Multi-Sensor Systems CSI-2 Packet Transmission
- 11. • Automotive application for driver assistance - External systems and for in-car control • Objects • High resolu0on • Night image and IR • In-car gesture • People detec0on • Medium resolu0on • Road signage • Medium resolu0on Parking assistance CSI2 Application for Multi-Sensor Systems Advanced Driver-Assistance System
- 12. CSI-2 Application for Multi-Sensor Systems • Application in a Multi-Camera Platform 12 Automotive AV Reference Subsystem MIPI DPHY Audio DSP $I $D System Interconnect Image/Vision DSP DMA I-RAM D-RAM AXI2 AHB UART I2C 32b APB Timer I2S GPIO AHB2 APB 32b AHB QSP I Soun dWir e Audio USB 2/3 devic e Ethe rnet MAC On-Chip System SRAM 1300MT/s DDR3 Controller DDR-PHY 64b DDR3 SODIMM SD SDIO eMM C Displa y Inm. BR PHY USB PHY Pixel 2AXI Color Conver t Video Scalar HDMI PHY Image/Vision DSP DMA I-RAM D-RAM MIPI CSI-2 Rx MIPI CSI-2 RX MIPI CSI-2 Rx MIPI DPHY MIPI DPHY Sensor DSP $I
- 13. 13 CSI-2 Application for Multi-Sensor Systems Sensors Everywhere
- 14. • The data does not have to be images… • LiDAR • The resolution is low (IR RAW data, typically 64 pixels high, though much more horizontally) • Range is limited. Typical LiDARs see well to about 70 metres. • Refresh rates tend to be slower, at around 10fps. • RADAR • Long range – cruise control, brake assist • Ultrasonic • Short-range parking assist • Self parking ☺ • Protocol support with user-defined data to transfer the bytes 14 CSI-2 Application for Multi-Sensor Systems CSI-2 for Sensors
- 15. CSI-2 Application for Multi-Sensor Systems CSI-2 Example Video Frame • Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps • So with our 30fps, we have 200M bit to use • 3 HD camera RGB888 1920x1200x24=165.888M • Also adding 100ns gaps (150 bit clocks) • 3 x(1920x24) +3x150 = 138240 • Embedded data line with image processed data (clusters, edges)
- 16. 16 CSI-2 TX Controller Video Buffer Centralized ECU for Infotainment or ADAS Pixel Pixel Data Data Sensor DSP Sensor DSP PCIe Sensor DSP I2S I2S I2S SoundWire eMMC Apps USB DSI DDR DPHY PCIe PHY USB PHY Sensor Sensor Video EP Video EP PPI PPI DPHY DPHY Pixel Pixel PHY Data Data CSI-2 domain Video Buffer Data Buffer Data Buffer I2S SoundWire Sensor DSP CSI-2 RX CTRL Vision DSP CSI-2 Vision DSP CSI-2 CSI-2 Application for Multi-Sensor Systems CSI-2 Sensor Fusion Example • Sensor Fusion ADAS System Topology • Merge the data from image and other sensors • Pre-processing the inline data for the application
- 17. • Image Processing and the Application • Application processing will need to perform the ADAS system and sensor analysis 17 CSI2 Application for Multi-Sensor Systems CSI-2 Sensor Fusion Example
- 18. 18 CSI-2 Application for Multi-Sensor Systems Filling the Channels D PHY (MCNN) D-PHY (MFEN) Pixel Processor/ Application DP DP DN DN D PHY (SCNN) D-PHY (SFEN) Pixel Stream 0 PPI PPI D-PHY (MFEN) DP DN D-PHY (SFEN) PPI Pixel Stream 1 Pixel Stream 2 Pixel Stream 3 Sensor Processor/ Application Pixel Stream 4 Pixel Stream 5 Pixel Stream 6 Pixel Stream 7 CSI-2 Host Controller PPI PPI PPI D-PHY (MFEN) D-PHY (SFEN) DP DN CSI-2 Slave Controller PPI PPI Sensor Control Sensor Control Sensor Sensor Sensor Sensor Sensor DATA DATA DATA DATA DATA DATA DATA CSI-2 Slave Controller CSI-2 Host Controller
- 19. 19 CSI-2 Application for Multi-Sensor Systems Physical Interface
- 20. CSI-2 Application for Multi-Sensor Systems CSI-2 Example Video Frame • Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps – Using LS/LE to keep synchronisation and sequence – Use the virtual channel to identify the sensor – Use the data types (RAW, RGB, YUV and user defined) – Use the short packet sync events
- 21. • Functional safety considerations • DPHY BER, RX error detection • Packet header ECC • Payload CRC • SP sync sequences, counting values 21 CSI2 Application for Multi-Sensor Systems Functional Safety In CSI2 ADAS
- 22. • Pitfalls of interleaved streams 22 CSI-2 Application for Multi-Sensor Systems CSI-2 Interleaving Data
- 23. 23 CSI-2 Application for Multi-Sensor Systems CSI-2 Interleaving
- 25. CSI-2 Beyond Mobile • System architecture considerations for CNN applications: • Assist • Co-pilot • Automated • Optimal platform arch for the CNN engines • Central processing (+SW dev, lacks scalability / modularity, cost may not align w/ entry-level cars) • Distributed processing (plug-and-play, scalable, each camera unit enhances capability, complex system) • AlgoEngine: CPU / GPU / DSP / FPGA • Overall risks and uncertainty: • Market, product, execution, timing, regulators, infrastructure 25
- 26. CSI-2 Beyond Mobile • What can technology do for us? • Imaging: digital photography vs. vision • Scene capture, object capture & track, modeling & measurement • Perception and decision-making using real-time streaming image data: • Camera, RADAR, LiDAR, sonar (varying detection capabilities vs. cost) • Performance vs robustness – consequence of error • Camera position, lighting, environmental factors, required accuracy for object detection 26