“Open Standards Unleash Hardware Acceleration for Embedded Vision,” a Presentation from the Khronos Group
0 likes35 views
The document discusses the need for open standards in embedded vision to facilitate parallel processing and improve hardware acceleration. It introduces the Kamaros Embedded Camera API as a cross-vendor solution designed to streamline camera integration and enhance system performance. The Khronos Group aims to develop royalty-free API standards to support safety-critical applications and promote innovation in the embedded vision market.
“Open Standards Unleash Hardware Acceleration for Embedded Vision,” a Presentation from the Khronos Group
- 1. Open Standards Unleash Hardware Acceleration for Embedded Vision Neil Trevett President Khronos Group
- 2. • The need for parallel offload of vision and inferencing processing • The need for open standard APIs for embedded vision • Introduction to Khronos and its family of open API standards • Khronos and safety-critical APIs • Introducing the Kamaros embedded camera API in development • How to get involved! Topics © 2023 The Khronos Group Inc 2
- 3. 3 © 2023 The Khronos Group Inc Khronos Connects Software to Silicon Open, royalty-free interoperability standards to harness the power of GPU, XR and multiprocessor hardware 3D graphics, augmented and virtual reality, parallel programming, inferencing and vision acceleration Non-profit, member-driven standards organization, open to any company Proven multi-company governance and Intellectual Property Framework Founded in 2000 ~ 200 Members |~ 40% US, 30% Europe, 30% Asia
- 4. 4 © 2023 The Khronos Group Inc Increasing Embedded Vision Processing Demands Increasing number and diversity of sensors per system Including camera arrays and depth sensors such as Lidar Increasing sophistication of sensor and vision processing Including inferencing Rapidly increasing demand for higher performance vision processing
- 5. 5 © 2023 The Khronos Group Inc The Need for Parallel Processing Single Processor Simple to program but May not provide enough performance especially As Moore’s Law frequency/power scaling is slowing Multi-Processor Additional processors can process expanded workloads but Adds complexity to system design and programming, as there is the need to: (i) Distribute workload across processors (ii) Synchronize use of compute and memory resources (iii) Communicate intermediate data and results Can open standards help solve this complexity?
- 6. 6 © 2023 The Khronos Group Inc Open Standards Make Technology Pervasive Open Standard = Shared Specification Implementation Implementation Implementation Implementation Implementation Contributor Contributor Contributor Contributor Contributor Open Source = Shared Implementation Open standards with rigorous conformance testing enable consistency across multiple implementations that can meet the needs of diverse markets, price points, and use cases Open standards often use open source to spread the implementation effort for sample implementations, tools, samples, conformance tests, validators, etc. INTEROPERABILITY standards define precise COMMUNICATION E.g., software to hardware, client to server
- 7. 7 © 2023 The Khronos Group Inc Open Standard Why’s, When’s and How’s Why Grow Markets By reducing consumer confusion and increasing capabilities and usability Reduce Costs By sharing ecosystem development between many companies and driving volume Speed Time to Market With well-proven functionality, testing and interoperability Enable Innovation Free companies to compete on value differentiators: quality, performance, power etc. When Proven Technologies Don’t do R&D by standards committee Consensus Need The downside of not having a standard is clear to all How Multi-company governance Avoid single-company control or dependency Well-defined IPR Policy Royalty-free standards have clearer path to wide adoption With Extensibility Enable a responsive innovation pipeline to meet customer and market needs Thoughtful Abstraction Focus on interoperability and avoid over specifying implementation which stifles innovation
- 8. 8 © 2023 The Khronos Group Inc Khronos Active Standards 3D Graphics Desktop, Mobile and Web 3D Assets Authoring and Delivery Portable XR Augmented and Virtual Reality Parallel Computation Vision, Camera, Inferencing, Machine Learning Safety Critical APIs • Enable cross-platform software reusability • Decouple software and hardware for easier development and integration of new components • • Provide cross-generation reusability • Facilitate field upgradability Open Standard APIs in Embedded Markets
- 9. 9 © 2023 The Khronos Group Inc Khronos Compute Acceleration Standards Higher-level Languages and APIs Streamlined development and performance portability Lower-level Languages and APIs Direct hardware control and optimization GPU GPU rendering + compute acceleration Heterogeneous compute acceleration Single source C++ programming with compute acceleration Graph-based vision and inferencing acceleration Intermediate Representation (IR) supporting parallel execution and graphics GPU FPGA DSP Custom Hardware GPU CPU CPU CPU AI/Tensor HW Neural Network Exchange Format Trained Networks Multiple programming abstractions to meet the needs of diverse software stack architectures
- 10. 10 © 2023 The Khronos Group Inc OpenCL – Low-level Parallel Programing Complements GPU-only APIs Simpler programming model Relatively lightweight run-time More language flexibility, e.g., pointers Rigorously defined numeric precision OpenCL Kernel Code OpenCL Kernel Code OpenCL Kernel Code OpenCL C Kernel Code GPU DSP CPU CPU FPGA OpenCL Devices Host CPU NN HW Runtime OpenCL API to compile, load and execute kernels across devices Programming and Runtime Framework for Application Acceleration Offload compute-intensive kernels onto parallel heterogeneous processors CPUs, GPUs, DSPs, FPGAs, Tensor Processors OpenCL C or C++ kernel languages Platform Layer API Query, select and initialize compute devices Runtime API Build and execute kernels programs on multiple devices Explicit Application Control Which programs execute on what device Where data is stored in memories in the system When programs are run, and what operations are dependent on earlier operations
- 11. 11 © 2023 The Khronos Group Inc Executing OpenCL Programs Context Programs Kernels Memory Objects Command Queues Send for execution Programs Kernel0 Kernel1 Kernel2 Images Buffers In order & out of order Create data & arguments Compile A kernel program is the basic unit of executable code (similar to a C function) An OpenCL program is a collection of kernels and functions An OpenCL command queue is used by the host application to send kernels and data transfer functions to a device for execution By enqueueing commands into a command queue, kernels and data transfer functions may execute asynchronously and in parallel with application host code As an open standard, OpenCL is a well proven design, available from many silicon vendors with an extensive ecosystem of available tools, compilers, libraries and educational materials
- 12. 12 © 2023 The Khronos Group Inc Growing Need for APIs for Functional Safety 1990s Avionics 2010s Automotive 2020s… Everywhere Growing demand for accelerated graphics and compute growing where functional safety is paramount Autonomous vehicles and devices, avionics, medical, industrial, and energy Industry safety-critical standards include RTCA DO-178C (avionics) | ISO 26262 (automotive) IEC 61508 (industrial) | IEC 62304 (medical) Safety-critical APIs are designed to reduce system-level certification effort and costs 1) Streamlined to reduce documentation and testing surface area 2) Deterministic behavior to simplify system design and testing 3) Unambiguous and comprehensive fault handling
- 13. 13 © 2023 The Khronos Group Inc Khronos Safety Critical Standards Evolution OpenGL ES 1.0 - 2003 Fixed function graphics OpenGL ES 2.0 - 2007 Programmable Shaders OpenGL SC 1.0 - 2005 Safety-critical subset OpenGL SC 2.0 - 2016 Safety-critical subset Vulkan 1.2 - 2020 Explicit Graphics, Compute and Display Vulkan SC 1.0 - 2022 Safety-critical subset SYCL 2020 C++-based heterogeneous parallel programming March 2022 SYCL SC Working Group announced Khronos has 20 years experience in standards for safety-critical markets Leveraging proven mainstream standards with shipping implementations and developer tooling and familiarity A choice of abstraction levels to suit different markets and developer needs OpenVX SC Extension – 2017 Vision / inferencing OpenVX 1.3 – 2019 SC Extension integrated into core OpenVX
- 14. 14 © 2023 The Khronos Group Inc Need for Embedded Camera API Standard Embedded System Camera Sensor and ISP Vision and Inferencing Acceleration Sensor Stream Application controlling sensor stream generation and processing in real time Choice of multiple open standard, cross- vendor APIs No widely adopted, open standard, cross- vendor APIs Benefits of a Cross-vendor Embedded Camera API Portability of camera/sensor code for easier system integration of new sensors Preservation of application code across multiple generations of cameras and sensors Sophisticated control over sensor stream generation increases effectiveness of downstream accelerated processing An effective open, cross-vendor open standard for camera, sensor and ISP control is an ecosystem gap compared to processing APIs The cost and time to integrate and utilize sensors in embedded systems using proprietary APIs has become a major constraint on innovation and efficiency in the embedded vision market
- 15. 15 © 2023 The Khronos Group Inc Kamaros Embedded Camera API Application-facing API for cross-vendor, cross-platform portability Multiple Language Bindings IDL based API design with automatic generation of multiple language bindings to C, C++, Python, and other programming languages Loadable Layers Command dispatch system to enable developers to use installable layers for validation, profiling, and debugging, etc. Resource Management and Synchronization Vulkan-style command queues well-suited to camera request processing Camera Programming Model based on Pipeline Templates Defines available streams through sensor, ISP and other processing elements Exposes static and per-frame controls - ranging from simple to more complex Balances ‘simplicity and portability’ and ‘detailed HW control’ Tested pipeline templates can be vendor-supplied
- 16. 16 © 2023 The Khronos Group Inc Kamaros Working Group Organization Any company is welcome to join Khronos for access to any Khronos Working Group Advisory Panel Design suggestions and draft feedback Drafts Feedback Working Group Design decisions and execution responsibility Other Khronos Working Groups and Advisory Panels GenICam compatibility, layered libraries, and other machine vision initiatives leveraging Kamaros once public Khronos / EMVA Liaison Agreement for ongoing close coordination Joint membership privileges for designated liaisons Other invited industry experts
- 17. 17 © 2023 The Khronos Group Inc Get Involved! Khronos is developing a growing family of open, royalty-free API standards for embedded and safety-critical markets Any company is welcome to join Khronos to influence standards development https://www.khronos.org/members/ or email memberservices@khronosgroup.org More information on any Khronos APIs https://www.khronos.org/ Khronos members can participate in the Kamaros Camera Working Group https://www.khronos.org/kamaros