OpenVINO introduction
Download as PPTX, PDF3 likes2,167 views
The document discusses OpenVINOTM, an Intel toolkit that provides high performance computer vision and deep learning inference capabilities. It allows building applications that leverage OpenCV, deep learning models, and heterogeneous execution across CPU, GPU, FPGA and VPU hardware. Key benefits include portable deployment across platforms with a minimal footprint, optimized performance on Intel hardware, and pre-trained models for common tasks like object detection. The toolkit includes libraries, tools for model optimization, and samples to help developers build and deploy high performance computer vision and deep learning applications.
OpenVINO introduction
- 1. High performance applications using OpenVINO™ 1 Yury Gorbachev
- 2. Internet of Things Group Why? • DL Training and inference are essentially very different domains • Frameworks contain too much for training • Hardware requirements are very different (100W is pretty normal) • Performance goals are different (batch size, latency vs. throughput) • Environment is different (development vs. deployment) • Hard to find solution that does DL inference properly
- 3. Internet of Things Group 3 What is proper Deep Learning inference? • Highest performance characteristics • Inference/watt/$ is major concern, best possible performance is a must • Minimal footprint • Memory, binary size, execution overhead • Absolute minimum of dependencies • Cross platform portability • Backward compatibility and predictable maintenance
- 4. Internet of Things Group 4 Also • CV application is not just Deep Learning • “Classical” components widely used • OpenCV is undisputable champion of the CV world • Need to satisfy deployment requirements as well • Performance, footprint, legal cleanness, etc.
- 5. Internet of Things Group 5 OpenVINO™ Toolkit for best CV/DL applications • Development toolkit for high performance CV and DL inference • Solution for application designers • No training/research overhead or specifics, minimal footprint, highly portable code • Set of libraries to solve CV/DL deployment problems • Fastest OpenCV build • Deep Learning Inference Engine • Provides access to all Intel accelerators and heterogeneous execution model • Intel CPU, integrated GPU • Vision Processing Unit (VPU) and FPGA
- 6. Internet of Things Group OpenVINO™ vs. Computer Vision Input Object Person Face Emotion Gesture Text … Custom Components CV/non-DL Components Direct Coding Solution API Solution API Solution Custom Code VPU GPU CPUFPGA VPU GPU CPU DL Components Computer Vision Pipeline DL Inference Engine VPU GPU CPUFPGA OpenVINO ™
- 7. Internet of Things Group 7 OpenCV: OpenVINO vs. Open Source • Most performant and fine tuned build • SSE, AVX2 vectorization & TBB/OMP parallelism • GPU offload via Transparent API • DL Inference Engine by default for OpenCV DNN • Legally clean • Each build is checked with IP Protection tools, safe for production • Additional algorithms from Intel in binary form • Face Detection and analysis libraries
- 8. Internet of Things Group 8 Deep Learning Inference Engine • Pure inference oriented solution (no training included) • Superior performance on Intel platform, highly optimized • Minimal memory use • No framework required in runtime • Support for CPU, GPU, FPGA, Movidius • Heterogeneous execution support • Cross-platform portability
- 9. Internet of Things Group 9 Deep Learning performance using OpenVINO/CPU 3.6 3.5 2.7 3.0 Core™-i5 6500@2.9 GHZ
- 10. Internet of Things Group 10 Deep Learning performance using OpenVINO/GPU 3.6 4.0 3.0 4.9 5.7 Core™-i5 6500@2.9 GHZ
- 11. Internet of Things Group 11 Inference memory footprint on CPU 7.8 3.0 3.1 3.4 4.5 1.6 Core™-i5 6500@2.9 GHZ
- 12. Internet of Things Group 12 Pre-trained Models Video/Image Pre-trained Models Video/Image Once in Design Time IR User Application + Inference Engine Model Optimizer User Application + Framework Model Model With Deep Learning Frameworks With OpenVINO™ DL Inference Engine
- 13. Internet of Things Group Deep Learning Inference Engine (IE) DL Inference Engine API Deep Learning application IR Model Optimizer Design time CPU Plugin MKL- DNN Heterogeneous Execution Engine C++ layers GPU Plugin clDNN Movidius Plugin Custom OCL layers FPGA Plugin MVNC DLA Custom layers + 3 4 3 Framework independent lightweight internal representation 4 Customizations in C++ and OpenCL languages 2 Heterogeneous network execution across accelerators 1 2 1 Single API solution across accelerators
- 14. Internet of Things Group 14 Easier deployment • Accurate against original framework • Direct replacement of original framework calls • No retraining/fine-tuning required • Unified support for multiple OSes • Linux and Windows are equally supported and performant • Encapsulates basic preprocessing • Mean subtraction / normalization integrated into model
- 15. Internet of Things Group 15 Portability across platforms • Single API across platforms • No need to change SDKs and application codes depending on targets • Consistent set of layers and accurate results across targets • Verified against reference model/framework • Heterogeneous execution for missing pieces • CPU fallback whenever needed
- 16. Internet of Things Group 16 Additional portability benefits • Intel has rapidly developing hardware set • New SoCs and architectures are evolving • No need to wait for HW itself or emulator • Design app for existing targets first and move to new ones easy • (Bonus) Check algos on fastest, deploy on most suitable! Create application using Inference Engine API Design and validate on CPU/GPU clusters Deploy on Movidius/FPGA Targets
- 17. Internet of Things Group 17 Customization possibilities • OpenVINO™ is partially binary product right now • Possible to implement and add own layers • New topologies are easy to support • Most of the layers are delivered as a source • Check how it is done for known topologies • Base your implementation on those for faster TTM • MKLDNN for CPU and clDNN for GPU are already available in source form • Possible to check implementation there
- 18. Internet of Things Group 18 Open Model Zoo • Free reference models for Deep Learning Inference Engine • Object Detection (Face, People, Vehicles, etc.) • Object Analysis (Facial analysis, Head Pose, Vehicle attributes) • Superior performance on Intel • Core™ i5 CPU: SSD 300 (6 fps) vs. People Detection Model (68 fps) Significant reduction in development efforts, no dataset & training needed
- 19. Internet of Things Group 19 Samples • Basic samples to facilitate API understanding • Classification, object detection, segmentation • Target selection via command line • Extended samples using Model Zoo • Face analysis, Security camera sample • Interworking between Media SDK, OpenCV, DL IE • Automated public models downloader script