"Dynamically Reconfigurable Processor Technology for Vision Processing," a Presentation from Renesas
1 like534 views
The document outlines Renesas's dynamically reconfigurable processor (DRP) technology aimed at enhancing vision processing capabilities by addressing limitations in existing architectures such as power consumption, scalability, and flexibility. It highlights the DRP's performance efficiency in processing image algorithms, showing significant speed advantages over traditional CPUs, and promotes partnership opportunities for advancing embedded AI and vision applications. Additionally, it provides resources, product specifications, and development support for the RZ/A2M processor incorporating DRP technology.
"Dynamically Reconfigurable Processor Technology for Vision Processing," a Presentation from Renesas
- 1. © 2019 Renesas Dynamically Reconfigurable Processor Technology for Vision Processing Yoshio Sato Renesas May 2019
- 2. © 2019 Renesas Renesas Broad Product Portfolio RZ 32/64-bit Arm-Based High-End Embedded MPU Family 2 8/16-bit Ultra-low energy Sensing and Motor Control Microcontrollers and Microprocessors 32/64-bit Arm-Based High-End Embedded MPU 32-bit High power efficiency Motor Control Renesas 40nm process Automotive …and more Analog & Mixed Signal, Power, Discrete ▪ Analog Devices ▪ Interface ▪ Memory ▪ Optoelectronics ▪ Power Management ▪ Sensors ▪ Space & Harsh Environment ▪ Timing & Digital Logic For automotive: ▪ Power Management ▪ Battery Management ▪ Video & Display …and more SoC, Integrated Platforms Factory automation Automotive AutomotiveRenesas autonomy™ IoTRenesas Synergy™ HMIRenesas RZ/G Linux
- 3. © 2019 Renesas Image Processing Market Dynamics Vision Evolution Drives Need for More Computing Power and Connectivity Bandwidth 3 1x 26x 1x 5x 1x 130x Resolution FPS VGA 30 10060 Consumer Healthcare Picking, Sorting 8MP+ Network Video Marketing Camera 2MP 1MP 5MP Motion Capture, Sports Analysis Intruder Alarm 150+ Security, Marketing Machine Vision 3MP Medical Camera (Endoscope) Digital Microscope Access Control Barcode, Character Visual Inspection
- 4. © 2019 Renesas Problem Statement Limitations of EXISTING ARCHITECTURES Multi processor • Limited scalability • Power hungry Dedicated IP • Adding functionality increases cost and power consumption • Not flexible FPGA • Power hungry • Area efficiency challenge 4
- 5. © 2019 Renesas Multi Processor Problem: Amdahl's Law Speedup is Limited by the Serial Part of the Program 5 https://en.wikipedia.org/wiki/Amdahl%27s_law#
- 6. © 2019 Renesas Multi Processor Problem: Von Neumann Architecture Data Transfer Drives Power Consumption 6 Conventional Software Implementation (CPU/DSP/GPU) Wired Logic Architecture (Dedicated IP/FPGA) Saving memory access by combining multiple functions → Reducing power by less memory access DRAM DRAM DRAM DRAM DRAM DRAM DRAM Sequential functions with many memory accesses → Consuming power only for data transfer Func 1 Func 2 Func 3 Func 4 Func. 1 Func. 2 Func. 3 Func. 4 Compute Local SRAM Main SRAM Main DRAM 1x 2x 128x CHIP Reference: presentation by Bert Moons Relative energy consumption per equivalent MAC operation I/O
- 7. © 2019 Renesas Dedicated IP Tradeoff Performance Increase at Cost of Larger Die Size and Power Consumption 7 0 10 20 30 40 50 Product Generation Gen.1 Gen.2 Gen.3 Functionality,Cost,Power
- 8. © 2019 Renesas Dedicated IP Limitations Limited Flexibility to Fix Bugs for Pre Production and Mass Production 8 Product GenerationGen 1 Gen 2 Functionality,Cost,Power Gen 2.1 Gen 2.2 Bug Fix Upgrade Optimize after market Time-to-market Bug Found! Hide it! Re-spin! What if we find the bug in the field?
- 9. © 2019 Renesas FPGA Problem Efficient Mapping is Challenging; Adding Features Increases Die Size and Power Programmability requires tradeoff of increased die size, power, and cost Mapping design efficiently to FPGA resources is difficult Adding new features may force move to larger, more expensive, and power-hungry FPGA 9 IO Pads Interconnect Switch Block Logic Block Free Area FreeFree
- 10. © 2019 Renesas Dynamically Reconfigurable Processor (DRP) Benefits of Software Flexibility with Pipelined Hardware Performance 10 Performance HighLow High Low DRP Dedicated IP or FPGA CPU Flexible, but limited performance High performance, but limited runtime reconfigurability Allocate system functions in complementary manner Runtime-reconfigurable hardwareProgrammability
- 11. © 2019 Renesas DRP (Dynamically Reconfigurable Processor) Spatial and Time-Multiplexed Computing 11 CPU GPU (SIMD) FPGA DRP ALU ALU ALU ALU Process 1 to a few operations in a cycle Process 10-100 same operations in a cycle Spatially expand various operations into LUT architecture Efficient if all parallel data are prepared High degree of freedom but limited performance High performance but use huge area due to expansion Pipelined processing Reg Reg ld ld add sub bne st ld ld add sub ld ld ld ld add add sub sub Time + × - Process spatially expanded operations Combine performance with expansion and area efficiency via dynamic reconfiguration +× × - Time Spatially pipelined Time-multiplexed
- 12. © 2019 Renesas DRP in the System Reconfigurable Array + Fast DMA 12 Fast and Intelligent Data Mover (DMA) ◼ Fast external memory access via dedicated DMA ◼ C programming with access API ◼ Dynamic configuration loading in less 1ms from external memory Reconfigurable Processing Elements Array ◼ Reconfigure as pipelined circuit to fit with algorithm ◼ Run complex algorithms by switching configurations within a nanosecond (up to 64 contexts) ◼ Programming with C language by high-level Synthesis tool Reduce memory bandwidth and achieve low latency Memory Controller DMA controller FIFO FIFO CPU External DRAM Output ImageInput Image DRP
- 13. © 2019 Renesas DMAC DMAC STC Custom Module-upper STC Custom Module-upper Processing Element Array Custom Module-lower Custom Module-lower DRP Hardware Architecture PE (Processor Elements), SRAMs, ALUs, STC (State Machine Controller) and DMAC 13 13 8b ALU Reg (7Byte) 8b ALU 512B 2p SRAM 16b Mul 16b Mul AXI(128-bit) Flexible coarse-grained reconfigurable architecture (Binary/8-bit) Tile-based scalability (48 PEs x 6 Tiles) Embedded Intelligent DMA controller 4KB 2p SRAM 16b Mul Reg (7Byte) One Tile (Executable Independently)
- 14. © 2019 Renesas DRP Reconfiguration Architecture State Machine Controller and Dynamically Reconfigurable Data-path 14 DMAC DMAC STC Custom Module-upper STC Custom Module-upper Processing Element Array Custom Module-lower Custom Module-lower ALUALU 512B 2p SRAM 16b Mul 16b Mul 4KB 2p SRAM 16b Mul Reg Reg The state defines the data-path structure, which can transition to a new configuration within one clock cycle
- 15. © 2019 Renesas DRP Programming Method C-to-Hardware Design Flow at a Glance 15 for( i = 0; i < N; i++ ){ for( j = 0; j < N; j++){ fn(i, j) = 5*f(i, j) – f(i, j-1) – f(i-1, j) – f(i+1, j) – f(i, j+1); } } Dedicated compiler Automatic synthesis from C to hardwired logic Automatic mapping from logic to DRP configuration Algorithm written in C Dynamically reconfigurable data-path
- 16. © 2019 Renesas STC PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem Mem MPY MPY MPY MPY MPY MPY MPY MPY １ 1 2 3 4 5 6 7 ３ ４ ５ State-by-State Reconfiguration Dynamic Reconfiguration Mechanism Cycle-base Configuration for STC (State Machine Controller) and Data-path Switch between multiple Data-paths with each DRP clock cycle to execute complex algorithms 16 + x - DP#1 - x DP#2 +x DP#3 - x DP#4 + x DP#5 x + x + DP#6 +x DP#7 + x - DP#1 x + DP#3 x - DP#4 + x DP#5 + x - DP#1 + x - DP#1 x + DP#3 x - DP#4 + x DP#5 + x - DP#1 Finite State Machine + Datapath Configurations DRP Hardware
- 17. © 2019 Renesas Dynamic Loading to Support Multiple Algorithms Switch to New Algorithm In As Little As One Millisecond 17 Dynamic reconfiguration in one DRP clock cycle • Up to 64 contexts (HW configurations) stored in DRP • Context switching managed by State Machine Controller Dynamic loading of new configuration in as little as 1 ms • Loads from external memory without interrupting execution • Change to HW with completely different function set • Time division execution of huge applications
- 18. © 2019 Renesas DRP vs CPU Performance Boosting Image Processing Algorithms DRP is 12 to 20 times faster than CPU in these image processing examples 18 Image size : 640x480 VGA Image Color : Grayscale 8BPP CPU : RZ/A2M Cortex-A9 @528MHz DRP : max.66MHz Video Frame Rate Example Algorithm: Harris Corner Detection Process Execution Time (ms) DRP CPU Canny Edge Detection 9.0 110.6 * Harris Corner Detection 11.6 235 * *CPU: Using Open CV) Processing Time Example Algorithm: Canny Edge Detection
- 19. © 2019 Renesas Dynamic Loading in Milliseconds Accelerate Sequential Algorithms / Parallel Processing Across Multiple DRP Slices 19 Image size : 640x480 VGA Image Color : RGB 24BPP, Grayscale 8BPP CPU : RZ/A2M Cortex-A9@528MHz DRP : max.66MHz
- 20. © 2019 Renesas DRP Benchmarking Over 10x Higher Performance and More Stable Execution Time than CPU DRP methodology converts source program to hard-wired circuit for high-speed, jitter-free execution 20 DRP: 10.4 ms; No JitterCPU: from 141.9 ms to 142.3 ms Canny Edge Detection Benchmark on Streaming Video
- 21. © 2019 Renesas RZ/A2M First RZ Product with DRP DRP Multipurpose Accelerator Large 4MB SRAM - High-speed access & Low BOM Cost High performance HMI functions - Cortex-A9 528MHz with NEON DSP - 2D Graphics accelerator & Sprite Engine - MIPI-CSI camera interface - JPEG Hardware codec Rich Connectivity - Dual Ethernet - USB/SDHI/MMC/NAND - 8bit DDR Memory Interface High Security - Arm TrustZone - Trusted Secure IP (TSIP) - Hardware crypto acceleration - Key management and storage - Access management Packages - 324pin BGA: 19mm/0.8mm pitch - 272pin BGA : 17mm/0.8mm pitch - 256pin BGA: 11mm/0.5mm pitch - 176pin BGA: 13mm/0.8mm pitch 21 Winner!
- 22. © 2019 Renesas RZ/A2M Development Support Software Packages and Tools Ready to Download Free RZ/A2M Software Package • Download from website • FreeRTOS v10 integrated Free Development Environment • Download e2studio IDE from website Free Compiler • Download GNU Arm Embedded Toolchain from GNU Arm Embedded Toolchain 6-2017-q2-update Fast and easy development 22 https://www.renesas.com/us/en/products/software-tools/software-os-middleware- driver/software-package/rza2-software-development-kit-free-rtos.html RZ/A2M Software Package Device Drivers Middle- ware Sample Projects DRP Library DRP Drivers e2 studio IDE and Tool Plug-ins Tool
- 23. © 2019 Renesas Sample Projects Using DRP Libraries Shrink Time-to-Market with Free Application Code 23 Simple Image Signal Processing (ISP) 2D Barcode Detection & Recognition Iris Detection & Recognition Image enhancement for fast, robust recognition 720p resolution, 10x faster than CPU decode, longer battery life 30 cm range, fast, encrypted identity storage and transmission https://www.renesas.com/us/en/products/software-tools/boards-and-kits/eval-demo/rz-a2m-evaluation-board-kit.html#downloads More to Come
- 24. © 2019 Renesas RZ/A2M Evaluation Kit First Prototyping-Ready Kit Complete RZ/A2M Evaluation Platform Evaluation of DRP enabled MIPI Camera Module (MIPI CSI) HyperMCP with HyperFlash™ and HyperRAM™ RGB conversion board for HDMI display 2ch Ethernet communication Other peripheral functions: SDHI, USB, etc. Segger J-Link Lite debugger Available now for RZ/A2M customer evaluation 24 Part number: RTK7921053S00000BE
- 25. © 2019 Renesas Summary and Next Steps Embedded Vision Evolution demands more computing power and connectivity bandwidth Available architectures have limitations of scalability, power consumption, flexibility, and efficiency DRP is solving these problems with Spatial and Time-Multiplexed Computing Roadmap for DRP technology to enhance embedded AI inference capabilities Renesas is looking for partners who would like to work with our DRP technology 25
- 26. © 2019 Renesas Dynamically Reconfigurable Processor (DRP) Technology Resources 26 RZA2 Support Material Visit RZ/A2 MPU Product Page Watch RZ/A2 MPU Overview Video Watch DRP Overview Video https://www.renesas.com/rza2m Buy RZ/A2M Evaluation Kit https://www.renesas.com/us/en/products/software- tools/boards-and-kits/eval-demo/rz-a2m-evaluation- board-kit.html Get RZA2M Demos, Downloads, and Application Guides https://www.renesas.com/us/en/products/software- tools/boards-and-kits/eval-demo/rz-a2m-evaluation- board-kit.html#downloads DRP Vision Processing Demo Please visit us in the Vision Technology Showcase to see a demonstration of the video processing capabilities of the RZ/A2M with DRP Embedded Vision Summit Dynamically Reconfigurable Processor Technology for Vision Processing
- 27. © 2019 Renesas Thank You 27
- 28. © 2019 Renesas Backup Material
- 29. © 2019 Renesas Acquisition of IDT Integrated Device Technology: the leading supplier of analog mixed-signal products including sensors, connectivity and wireless power Who We Are The World’s Leading Embedded Solution Provider 29 Renesas Technology Spin-off from Hitachi and Mitsubishi merger 2003 NEC Electronics Spin-off from NEC 2002 2019 Acquisition of Intersil Strengthen leadership in the analog market 2017 Renesas Electronics started operation NEC Electronics and Renesas Technology merged 2010 Originating from Hitachi, Mitsubishi, NEC, and Intersil Net sales 757.4 billion yen in 2018 19,000+ employees worldwide Headquartered in Tokyo, Japan
- 30. © 2019 Renesas RZ Family of Microprocessors Maximum Performance for Cognitive, HMI, Industrial Networks 30 RZ/G SeriesRZ/A Series www.renesas.com/rz
- 31. © 2019 Renesas RZ Series Target Applications 31
- 32. © 2019 Renesas RZ/A2M MPUs add DRP Technology to RZ/A1 Features 32
- 33. © 2019 Renesas RZ/G MPU Series Lineup Now with 2nd Generation RZ/G MPUs — Scalable from Entry-class RZ/G2E to High-end RZ/G2H 33
- 34. © 2019 Renesas Additional RZ MPU Resources RZ MPU Family Overview RZ Family of 64-Bit & 32-Bit Arm-Based High-End MPUs https://www.renesas.com/us/en/products/microcontrollers-microprocessors/rz.html RZ/A1 Support Material RZA1 Development Software and Kits https://www.renesas.com/us/en/products/microcontrollers-microprocessors/rz/rza/rza-startup.html RZ/G Support Material Get Started with the RZ/G Linux Platform https://www.renesas.com/us/en/products/rzg-linux-platform/get-started.html 34