"APIs for Accelerating Vision and Inferencing: Options and Trade-offs," a Presentation from Khronos
0 likes224 views
The document discusses the evolving ecosystem of vision performance and portability, emphasizing the role of layered acceleration via diverse hardware and explicit kernel APIs. It highlights various technologies, software, and frameworks facilitating machine learning and neural network inferencing across platforms, as well as the need for standardized APIs for vision processing applications. The document also touches upon safety-critical requirements and the importance of industry collaboration for the development of next-generation standards in this field.
"APIs for Accelerating Vision and Inferencing: Options and Trade-offs," a Presentation from Khronos
- 1. Copyright © 2018 The Khronos Group 1
- 2. Copyright © 2018 The Khronos Group 2 The Search for Vision Performance and Portability Leads to a Layered Acceleration Ecosystem Explicit Kernel APIs Diverse Hardware (GPUs, DSPs, FPGAs) Single Source C++ Languages Libraries, Frameworks and run-times Code Intermediate Representations PTX HSAIL GPU FPGA DSP Dedicated Hardware Vision Processing Libraries Inferencing Run-times WinML/DirectMLAR Libraries Machine Learning Training Frameworks GPU-only APIs Heterogenous Computing
- 3. Copyright © 2018 The Khronos Group 3 Mobile Augmented Reality Libraries ARKit V1.0 is VR focused. Subsequent versions will extend to cross- platform AR Encapsulated Vision-based Functionality Also leveraging motion sensors AR Libraries typically use ARKit/ARCore if available or implement own tracking if not
- 4. Copyright © 2018 The Khronos Group 4 Vision and Neural Net Inferencing Runtimes Neural Network Workflow Applications Using Embedded Vision and Inferencing Desktop and Cloud Hardware Trained Networks cuDNN MIOpen MKL-DNN FPGA DSP GPU Custom Hardware Diverse Inferencing Acceleration Hardware GPU CPU CPU CPU Framework Specific Formats Training = Desktop / Cloud Neural Net Training Frameworks Neural Net Training Frameworks Neural Net Training Frameworks Neural Net Training Frameworks Deployment on Embedded Devices Compilation and Optimization TPU Authoring interchange Embedded Deployment WinML
- 5. Copyright © 2018 The Khronos Group 5 NNVM - Open Compiler for AI Inferencing http://www.tvmlang.org/2017/08/17/tvm-release-announcement.html SPIR-V IR for parallel accelerators Backend in development LLVM IR for CPUs 1.Import Trained Network Description 2. Graph-level Optimizations 3. Decompose into primitive instructions 4. Emit programs executable by run-times
- 6. Copyright © 2018 The Khronos Group 6 SPIR-V Ecosystem LLVM Third party kernel and shader languages • Khronos defined cross-API IR • Native graphics and parallel compute • Stable specification to complement LLVM Open Source Project OpenCL C++ Front-end OpenCL C Front-end glslang Khronos-hosted Open Source Projects IHV Driver Runtimes SPIR-V Validator SPIR-V (Dis)Assembler LLVM to SPIR-V Bi-directional Translators https://github.com/KhronosGroup/SPIRV-Tools GLSL HLSL Khronos cooperating with Clang/LLVM Community SPIRV-Cross GLSL HLSL MSL SPIRV-opt | SPIRV-remap Optimization Tools DXC SYCL Front-end
- 7. Copyright © 2018 The Khronos Group 7 Platform Neural Network Stacks Microsoft Windows Windows Machine Learning (WinML) Google Android Neural Network API (NNAPI) Apple MacOS and iOS CoreML Common Fundamental Steps 1. Import trained NN model file 2. Build optimized version of graph 3. Accelerate on GPU or other processor using available low-level API https://docs.microsoft.com/en-us/windows/uwp/machine-learning/ https://developer.android.com/ndk/guides/neuralnetworks/ https://developer.apple.com/documentation/coreml Core ML Model
- 8. Copyright © 2018 The Khronos Group 8 NNEF Ecosystem Files Syntax Parser/ Validator TensorFlow and Caffe Exporters Comparing Neural Network Exchange Industry Initiatives NNEF open source projects hosted on Khronos NNEF GitHub repository Apache 2.0 license https://github.com/KhronosGroup/NNEF-Tools TensorFlow and Caffe2 Importer / Exporters Google NNAPI Convertor NNEF 1.0 Provisional Released for industry feedback before finalization OpenVX Ingestion & Execution Live Imminent
- 9. Copyright © 2018 The Khronos Group 9 Network Data File Binary format contains parameter tensors Supports float and quantized (integer) data Flexible bit widths and quantization algorithms Quantization algorithms expressed as extensible compound operations Quantization info provided as hints for execution Network Data File Binary format contains parameter tensors Supports float and quantized (integer) data Flexible bit widths and quantization algorithms Quantization algorithms expressed as extensible compound operations Quantization info provided as hints for execution NNEF File Structure Network Structure File Distilled, platform independent network description Human readable, syntactical elements from Python Standardized Operations Rigorously defined semantics Linear, convolution, pooling, normalization, activation, unary/binary Supports fully connected, convolutional, recurrent architectures Two Levels of Expressiveness FLAT: Basic transfer of computation graphs with standardized operations Simple to parse and translate to vendor specific formats COMPOSITIONAL: Define custom compound operations Higher-level graph descriptions More complex to parse but offers more optimization hints Network Data File Binary format contains parameter tensors Supports float and quantized (integer) data Flexible bit widths and quantization algorithms Quantization algorithms expressed as extensible compound operations Quantization info provided as hints for execution Split Structure and Data files Easy independent access to network structure or individual parameter data Set of files can use a container such as tar or zip with optional compression and encryption Can associate multiple Data Files with one Network Structure File e.g. the same data in multiple formats
- 10. Copyright © 2018 The Khronos Group 10 • Convolution Neural Network topologies can be represented as OpenVX graphs • Can also combine traditional vision and neural network operations • OpenVX Neural Network Extension • Defines OpenVX nodes to represent many common NN layer types • Layer types include convolution, activation, pooling, fully-connected, soft-max • Defines multi-dimensional tensors objects to connect layers • Kernel Import Extension • Enables loading of external program representations into OpenVX graphs NNEF Execution within OpenVX Vision Node Vision Node Vision Node Downstream Application Processing Native Camera Control CNN Nodes An OpenVX graph mixing CNN nodes with traditional vision nodes Importer converts NNEF representation into OpenVX Graphs using Kernel Import Extension
- 11. Copyright © 2018 The Khronos Group 11 OpenCL Command Queue OpenVX / OpenCL Interop Extension Application Fully asynchronous host- device operations during data exchange RuntimeRuntime Map or copy OpenVX data objects into cl_mem buffers Copy or export cl_mem buffers into OpenVX data objects Enables custom OpenCL acceleration to be used within OpenVX User Kernels OpenVX user-kernels can access command queue and cl_mem objects to asynchronously schedule OpenCL kernel execution
- 12. Copyright © 2018 The Khronos Group 12 OpenCL Ecosystem Roadmap 2011 OpenCL 1.2 OpenCL C Kernel Language OpenCL 2.1 SPIR-V in Core 2015 SYCL 1.2 C++11 Single source programming OpenCL 2.2 C++ Kernel Language 2017 SYCL 1.2.1 C++11 Single source programming Work with industry to bring Heterogeneous compute to standard ISO C++ OpenCL ‘Next’ Flexible and efficient deployment of parallel computation across diverse processor architectures Single source C++ programming. Great for supporting C++ apps, libraries and frameworks More deployment options: Enabling dispatch of OpenCL C kernels from Vulkan runtimes
- 13. Copyright © 2018 The Khronos Group 13 • Single-source heterogeneous programming using STANDARD C++ • Use C++ templates and lambda functions for host & device code • Layered over OpenCL • Fast and powerful path for bring C++ apps and libraries to OpenCL • C++ Kernel Fusion - better performance on complex software than hand-coding • SYCLBLAS, SYCL Eigen, SYCL TensorFlow, SYCL DNN, SYCL GTX, VisionCpp • Close cooperation with ISO C++ • C++17 Parallel STL hosted by Khronos • C++20 Parallel STL with Ranges • Implementations • triSYCL, ComputeCpp, ComputeCpp SDK … • More information at http://sycl.tech SYCL Ecosystem
- 14. Copyright © 2018 The Khronos Group 14 Pervasive Vulkan 1.0 Major GPU Companies supporting Vulkan for Desktop and Mobile Platforms http://vulkan.gpuinfo.org/ Platforms Embedded Virtual Reality Cloud Services Mobile (Android 7.0+) Desktop Media Players Consoles
- 15. Copyright © 2018 The Khronos Group 15 • Experimental collaboration between Google, Codeplay, and Adobe - Successfully tested on over 200K lines of Adobe OpenCL C production code • Compiles OpenCL C to Vulkan’s SPIR-V execution environment - Proof-of-concept that OpenCL kernels can be brought seamlessly to Vulkan - Significant parts OpenCL C 1.2 so far – shaped by submitted workloads Clspv OpenCL C to Vulkan Compiler Clspv Compiler OpenCL C Source Runtime OpenCL Host Code Run-time API Translator Prototype open source project https://github.com/google/clspv Possible future project – if interest? Increasing deployment options for OpenCL kernel developers e.g. Vulkan is a supported API on Android
- 16. Copyright © 2018 The Khronos Group 16 • Vendors can support ANY combination of features to suit their hardware/market • If all exposed features are conformant – the implementation is conformant • Khronos will define Feature Sets equivalent to current profiles • Existing profiles and device types not going away! No changes to existing applications • Opportunity to coalesce industry support around market-focused feature sets • Khronos aiming to provide Feature Set infrastructure for the industry to leverage OpenCL Next – Deployment Flexibility OpenCL 2.2 Functionality = queryable, optional feature Khronos-defined OpenCL 2.2 Full Profile Feature Set Khronos-defined OpenCL 1.2 Full Profile Feature Set Industry-defined E.g. ‘Inferencing’ Feature Set
- 17. Copyright © 2018 The Khronos Group 17 Safety Critical APIs – Khronos Experience Need for new-generation APIs for safety certifiable vision, graphics and compute e.g. ISO 26262 and DO-178B/C OpenGL ES 1.0 - 2003 Fixed function graphics OpenGL ES 2.0 - 2007 Shader programmable pipeline OpenGL SC 1.0 - 2005 Fixed function graphics subset OpenGL SC 2.0 - April 2016 Shader programmable pipeline subset General lack of industry consensus on how APIs should be designed to streamline safety certification OpenVX SC 1.1 - May 2017 Restricted “deployment” implementation only executes pre- compiled binary format
- 18. Copyright © 2018 The Khronos Group 18 Khronos Safety Critical Advisory Forum OpenCL SC TSG Working on OpenCL SC Gathering requirements SYCL SC Guidelines to augment Industry First Safe and Secure Parallel and Heterogeneous C++ Safe AI for Automotive AESIN Automotive ADAS & AV + security https://aesin.org.uk MISRA C++ C++ WG23 Programming Vulnerabilities ISO C Safe and Secure SG ISO C++ Vulnerabilities Safety Critical SG Generating guidelines for designing safety critical APIs to ease system certification. Open to Khronos member AND industry experts https://www.khronos.org/advisors/kscaf Industry outreach and cooperation Khronos SC Activities OpenVX SC 1.1 - May 2017 Restricted “deployment” implementation only executes pre- compiled binary format We are inviting safety critical experts to join KSCAF! No cost or work commitment
- 19. Copyright © 2018 The Khronos Group 19 • Advanced control of sensor and camera subsystem – with cross-platform portability • Generate sophisticated image stream for advanced imaging & vision apps • No platform API currently fulfills all developer requirements • Portable access to growing sensor diversity: e.g. depth sensors and sensor arrays • Cross sensor synch: e.g. synch of camera and MEMS sensors • Advanced, high-frequency per-frame burst control of camera/sensor: e.g. ROI • Multiple input, output re-circulating streams with RAW, Bayer or YUV Processing Need for Camera Control API? Image Signal Processor (ISP) Defines control of Sensor, Color Filter Array Lens, Flash, Focus, Aperture Auto Exposure (AE) Auto White Balance (AWB) Auto Focus (AF) Stream of Images for Vision Processing OpenKCAM standard is currently on ice – do we need to restart?
- 20. Copyright © 2018 The Khronos Group 20 • Vision tools and API ecosystem becoming increasingly sophisticated • Layering libraries, languages and run-times • Machine learning stacks continue to grow in flexibility • Exchange formats and compiler technologies essential to flexible deployment • Open standards evolving alongside proprietary solutions • For developers that value cross-platform deployment • Safety-critical APIs becoming increasingly essential • Many vision applications need system certification • Still no cross-vendor camera APIs? • Is the time yet right for this to be a target for standardization? • Please join if your company interested in Khronos open standards • Neil Trevett | ntrevett@nvidia.com | @neilt3d Key Takeaways and What’s Next?