“Advancing Embedded Vision Systems: Harnessing Hardware Acceleration and Open Standards,” a Presentation from the Khronos Group
0 likes28 views
The document discusses the importance of open standards in embedded vision systems, emphasizing the need for parallel processing and interoperability to enhance hardware acceleration. It introduces the Kamaros Embedded Camera API, aimed at creating an open, royalty-free standard for camera system runtimes, addressing the complexities of diverse sensor integration. Additionally, the document outlines the role of the Khronos Group in developing these standards and invites participation from companies in the standards development process.
“Advancing Embedded Vision Systems: Harnessing Hardware Acceleration and Open Standards,” a Presentation from the Khronos Group
- 1. 1 © 2024 The Khronos Group Inc. Advancing Embedded Vision Systems: Harnessing Hardware Acceleration and Open Standards Neil Trevett President Khronos Group
- 2. 2 © 2024 The Khronos Group Inc. • The need for parallel offload of vision and inferencing processing • How open standard APIs add value for embedded vision • Introduction to Khronos and its family of open API standards • Khronos and Safety Critical APIs • Introduction to the new Kamaros Embedded Camera API in development • How to get involved! Topics
- 3. 3 © 2024 The Khronos Group Inc. Khronos Connects Software to Silicon Open, royalty-free interoperability standards to harness the power of GPU, XR and multiprocessor hardware 3D graphics, augmented and virtual reality, parallel programming, inferencing and vision acceleration Non-profit, member-driven standards organization, open to any company Proven multi-company governance and Intellectual Property Framework Founded in 2000 ~ 200 Members |~ 40% US, 30% Europe, 30% Asia
- 4. 4 © 2024 The Khronos Group Inc. The Need for Parallel Processing Single Processor Simple to program but may not provide enough performance especially As Moore’s Law frequency/power scaling is slowing Multi-Processor Additional processors can process expanded workloads but adds complexity to system design and programming: (i) Distribute workload across processors (ii) Synchronize use of compute and memory resources (iii) Communicate intermediate data and results Can open standards help solve this complexity?
- 5. 5 © 2024 The Khronos Group Inc. What is an Open Interoperability Standard? Open Standard = Shared Specification Implementation Implementation Implementation Implementation Implementation Contributor Contributor Contributor Contributor Contributor Open Source = Shared Implementation Open standards enable MULTIPLE IMPLEMENTATIONS that can meet the needs of diverse markets, price points, and use cases - with rigorous conformance testing to ensure cross-vendor consistency Open standards often use open source to share the implementation effort for sample implementations, tools, samples, conformance tests, validators, etc. INTEROPERABILITY standards define precise COMMUNICATION E.g., software to hardware, client to server OPEN standards are created through multi-company cooperation
- 6. 6 © 2024 The Khronos Group Inc. Open Standard Why’s, When's and How’s Why? Grow Markets Reduced consumer confusions and increased capabilities and usability Reduce Costs Shared development between many companies and driving volume Speed Time to Market Well-proven functionality, testing and interoperability Enable Innovation Free resources to compete on quality, performance, power etc. When? When Technologies are Proven Avoid research and development by standards committee Consensus Need The downsides of not having a standard are obvious How? Multi-company governance Avoid single-company control or dependency Well-defined IPR Policy Royalty-free standards have clearer path to wide adoption With Extensibility Enable a responsive innovation pipeline to meet customer and market needs Thoughtful Abstraction Encourage innovation through freedom to innovate implementation details
- 7. 7 © 2024 The Khronos Group Inc. The Value of Open Standards in Embedded Markets • Cross-platform software reusability • Decoupled software and hardware for easier development and integration • Cross-generation reusability and field upgradability • Developer training and familiarity • Shared development of an industry-wide ecosystem of tools and libraries Open standards can provide proven and accessible languages and APIs to enable effective parallel processing and hardware acceleration
- 8. 8 © 2024 The Khronos Group Inc. Khronos Active Standards 3D Graphics Desktop, Mobile, Web 3D Asset Authoring/Delivery Portable XR Augmented and Virtual Reality Parallel Computation Vision, Camera, Inferencing, Machine Learning Safety Critical APIs Khronos standards most relevant to the embedded vision industry
- 9. 9 © 2024 The Khronos Group Inc. Khronos Compute Acceleration Standards Heterogeneous compute acceleration Accelerated Single source C++ programming Graph-based vision and inferencing acceleration Intermediate Representation (IR) compiler target supporting parallel execution and graphics Higher-level Languages and APIs Streamlined development and performance portability GPU FPGA DSP Custom HW GPU CPU CPU CPU AI/Tensor HW GPU GPU rendering + compute acceleration Lower-level Languages and APIs Explicit hardware control Neural Network Exchange Format Trained Networks Multiple programming abstractions to meet the needs of diverse software stack architectures Third party vision, streaming and inferencing libraries Applications, libraries, and higher-level languages and APIs may use lower-level Khronos standards to access hardware acceleration Shaders Kernels
- 10. 10 © 2024 The Khronos Group Inc. OpenVX Partitioning OpenVX 2.0 implementations need to include just OpenVX Core OpenVX Functionality OpenVX 1.3.X Feature Sets Base Features Graph Framework + Data Objects Vision Nodes NN Nodes Enhanced Vision Nodes NNEF Import Nodes Extensions Vision Feature Set NN Feature Set NNEF Import Feature Set Enhanced Vision Feature set Binary Image Support Feature set OpenVX 1.3.X implementations need to include Base Features plus at least one of Vision/NN/NNEF feature sets OpenVX Core Essential Extensions Custom/Target Nodes Base Features Graph Framework + Data Objects Vision Nodes NN Nodes Enhanced Vision Nodes NNEF Import Nodes Extensions OpenVX 2.0 Core + Optional Extensions Focus on the OpenVX Graph Framework Flexible pipelined data flow through target hardware Seamless custom kernel support Extensions e.g., for vision/radar/lidar processing
- 11. 11 © 2024 The Khronos Group Inc. Growing Need for APIs for Functional Safety Safety-critical APIs reduce system-level certification effort where functional safety is paramount 1) Streamlined to reduce documentation and testing surface area 2) Deterministic behavior to simplify system design and testing 3) Unambiguous and comprehensive fault handling 1990s - Avionics 2010s - Automotive 2020s - Autonomous vehicles and devices, avionics, medical, industrial, and energy 2003 2005 2017 Safety Critical Extension 2020 (V1.3) 2022 2023 Working Group Formed Khronos has 20 years experience in adapting safety-critical API from proven mainstream APIs
- 12. 12 © 2024 The Khronos Group Inc. The Need for a Camera System API Standard Increasing Sensor Diversity Including camera arrays and depth sensors such as Lidar Increasing Sensor Processing Demands Including inferencing. Sensor outputs need to be flexibly and efficiently generated and streamed into acceleration processors Multiple Sensors Per System Synchronization and coordination become essential Proprietary APIs Hinder Innovation Vendor-specific APIs to control cameras, sensors and close-to-sensor ISPs prevent access of full camera capabilities The cost and time to integrate and utilize sensors in embedded systems has become a major constraint on innovation and efficiency in the embedded vision market
- 13. 13 © 2024 The Khronos Group Inc. What is Kamaros? EMBEDDED CAMERA SYSTEM API Jointly promoted by Khronos and the European Machine Vision Association (EMVA), the Kamaros API Working Group is developing an open, royalty-free standard for controlling camera system runtimes in embedded, mobile, industrial, XR, automotive, and scientific markets
- 14. 14 © 2024 The Khronos Group Inc. Need for Embedded Camera API Standard Embedded System Camera Sensor and ISP Vision and Inferencing Acceleration Sensor Stream Application controlling sensor stream generation and processing in real time Choice of multiple open standard, cross-vendor APIs No widely adopted, open standard, cross-vendor APIs Benefits of a Cross-vendor Embedded Camera API Portability of camera/sensor code for easier system integration of new sensors Preservation of application code across multiple generations of cameras and sensors Sophisticated control over sensor stream generation increases effectiveness of downstream accelerated processes An effective open, cross-vendor open standard for camera, sensor and ISP control is a major ecosystem gap compared to processing APIs An effective camera API abstraction will enable sensor vendors to expose fuller hardware capabilities without exposing proprietary implementation details while while gaining access to a larger ecosystem of libraries and applications
- 15. 15 © 2024 The Khronos Group Inc. Typical Kamaros Software Stack Frameworks & Middleware OS GStreamer Sensors Physical Devices Lenses Lights Processors Kamaros System Runtime Windows Linux Android Chrome RTOS Transport CSI-2 USB Ethernet Application OpenVX Libraries API in scope Some libraries may be in scope Kamaros API Names of transport layers, frameworks and operating systems are illustrative examples
- 16. 16 © 2024 The Khronos Group Inc. Structure of a Kamaros Application Request Pipeline Configuration Controls set in priority order Best Attempt Pipeline Configuration Achievable Control Values, Defaults and Constraints 2. Discover Pipeline Sets Controls per Frame Synchronize for Frame availability Process Returned Frame Per frame metadata provides applied settings and errors Stop Pipeline Free resources 4. Capture Frames Create Pipeline with initial Controls Allocate any needed resources e.g., buffers 3. Create Pipeline 1. Discover Camera Select Camera via Kamaros Loader Data provided on physical or geographic location, sensor and lens etc. Query Pipeline Templates for selected Camera Select Pipeline that has Controls and capability limits that satisfy use case Pipeline Templates Define controls and capabilities of available streams through camera sensor, ISP and processors Enable flexible and scalable discovery of camera capabilities Simpler Templates provide can application portability Customized Templates can provide detailed hardware control and may be vendor-supplied Application
- 17. 17 © 2024 The Khronos Group Inc. Kamaros, Khronos and EMVA Cooperation Drafts Feedback EMVA members are invited to join Kamaros Advisory Panel at no charge GenICam, layered libraries, and other EMVA machine vision initiatives may leverage Kamaros Khronos / EMVA have a Liaison Agreement for ongoing coordination and joint membership privileges for designated liaisons Working Group Design decisions and execution Other Khronos Working Groups and Advisory Panels Advisory Panel Design input and feedback
- 18. 18 © 2024 The Khronos Group Inc. Get Involved! Any company is welcome to join Khronos to influence standards development https://www.khronos.org/members/ or email memberservices@khronosgroup.org More information on any Khronos API https://www.khronos.org/ Khronos members can participate in the Kamaros Camera Working Group EMVA Members can join the Kamaros Advisory panel https://www.khronos.org/kamaros Khronos is developing a growing family of open, royalty-free API standards relevant to embedded and safety-critical markets