Achieving AI @scale on Mobile Devices
6 likes3,019 views
Qualcomm Technologies is leading advancements in mobile AI, emphasizing the Snapdragon platform's capabilities for efficient on-device machine learning and high performance. The integration of advanced algorithms and heterogeneous computing enhances user experiences while prioritizing privacy and low latency. This initiative supports various applications, from augmented reality to automotive technology, showcasing the transformative potential of AI on mobile devices.
Achieving AI @scale on Mobile Devices
- 1. Achieving AI @Scale on Mobile Devices Qualcomm Technologies, Inc. @qualcomm_techMarch 2018
- 2. 2 Mobile is the largest computing platform in the world Source: Gartner Mar. ‘17 >8.5 Billion Cumulative smartphone unit shipments forecast between 2017–2021
- 3. 3 Source: Qualcomm Incorporated data. Currently, Qualcomm semiconductors are products of Qualcomm Technologies, Inc. or its subsidiaries. MSM is a product of Qualcomm Technologies, Inc. IHS, Mar. ’17; Strategy Analytics, Mar. ’17 30 Years of driving the evolution of wireless # 1Fabless semiconductor company # 1 in 3G/4G LTE modem 842MMSM™ chipsets shipped FY’16
- 4. 4 Modem 3G (CDMA, GSM) 4G/LTE 5G Connectivity Bluetooth Smart Bluetooth Mesh 802.11ac/ax 802.11ad 802.11n 802.15.4 DSRC Powerline GNSS/Location Peripherals (PCIe, USB.) Processors CPU GPU DSP Multimedia Video Speech Display processing Media processing Camera processing Audio processing Computer Vision AR / VR Qualcomm Technologies’ success is based on RF Power amps Acoustic filters RF switches Other Machine learning NoC / MemCtrl Power management Security Touch Fingerprint Technology leadership Connectivity Computing
- 5. 5 1 chipset per year Chipset complexity is growing dramatically Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc. Source: Qualcomm internal data ~10 chipsets per year Transistors on a chip More frequent and more complex design cycles Today Transistors on a chip <10 Million Comparative scale Circa 2000 >3Billion
- 6. 6 A mobile processor today—Snapdragon 835 Highly integrated and complex SoC using 10nm process technology Qualcomm Spectra 180 Camera Qualcomm Mobile Security Kryo 280 CPU Wi-Fi Snapdragon X16 LTE modem Adreno 540 Graphics Processing Unit (GPU) Hexagon 682 DSP HVX All-Ways Aware Qualcomm® Location Suite Display Processing Unit (DPU) Video Processing Unit (VPU) Qualcomm® Aqstic Audio * Compared to Snapdragon 820 Qualcomm Adreno, Qualcomm Kryo, Qualcomm Hexagon, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Location Suite, Qualcomm All-Ways Aware, and Qualcomm Mobile Security are products of Qualcomm Technologies, Inc. Snapdragon X16 LTE World’s first announced gigabit-class LTE modem Qualcomm® Hexagon™ DSP Snapdragon Neural Processing Engine support Qualcomm® Mobile Security First to support full biometric suite Qualcomm® Adreno™ Visual Processing 25% faster graphics rendering 60x more display colors* Qualcomm® Kryo™ 280 CPU Our most power efficient architecture to date Qualcomm Spectra™ Camera ISP Smooth zoom | Fast-autofocus True to life colors
- 7. 7 Mobile scale changes everything 7 Superior scale Rapid replacement cycles Integrated and optimized technologies Industrial IoT Smart cities Automotive Smart homes Extended reality Wearables Healthcare Mobile computing Networking
- 8. 8 Intelligence is moving to the device Server/Cloud Training Execution/Inference Devices Execution/Inference Training (emerging)
- 9. 9 Privacy Reliability Low latency Efficient use of network bandwidth Process data closest to the source, complement the cloud On-device intelligence is paramount
- 10. 10 Mobile is becoming the pervasive AI platform
- 11. 11 Qualcomm Technologies is accelerating on-device AI Making efficient on-device machine learning possible for highly responsive, private, and intuitive user experiences
- 12. 1212 Qualcomm ® Artificial Intelligence Platform A high-performance platform designed to support myriad intelligent- on-device-capabilities that utilize: • Qualcomm® Snapdragon™ mobile platform’s heterogeneous compute capabilities within a highly integrated SoC • Innovations in machine learning algorithms and enabling software • Development frameworks to minimize the time and effort for integrating customer networks with our platform The platform for efficient on-device machine learning Qualcomm Artificial Intelligence Platform and Qualcomm Snapdragon are products of Qualcomm Technologies, Inc. Audio intelligence Intuitive security Visual intelligence
- 13. 13 Making on-device intelligence pervasive Focusing on high performance HW/SW and optimized network design Algorithmic advancements Algorithmic research that benefits from state-of-the-art deep neural networks Optimization for space and runtime efficiency Efficient hardware Developing heterogeneous compute to run demanding neural networks at low power and within thermal limits Selecting the right compute block for the right task Software tools Software accelerated run-time for deep learning SDK/development frameworks
- 14. 14 Algorithmic enhancements for space and runtime efficiency Improve performance by addressing model complexity Source: Qualcomm Research. Results from running a semantic segmentation network for automotive. Neural network optimizations for embedded • Improved network architecture • Focus on memory and storage ◦ Reduce bit widths ◦ Model compression ◦ Leverage sparsity • Architecture learning Required operations per image 0 10000 20000 30000 40000 Original QCOM V1 QCOM V2 QCOM V3 GOPS Network versions QTI V1 QTI V2 QTI V3 27x Reduction in complexity
- 15. 15 Snapdragon Neural Processing Engine SDK Software accelerated runtime for the execution of deep neural networks on device Qualcomm Kryo, Qualcomm Adreno and Qualcomm Hexagon are products of Qualcomm Technologies, Inc. Available at: developer.qualcomm.com Efficient execution on Snapdragon • Takes advantage of Snapdragon heterogeneous computing capabilities • Runtime and libraries accelerate deep neural net processing on all engines: CPU, GPU, and DSP with HVX Model framework/Network support • Convolutional neural networks and LSTMs • Support for Caffe/Caffe2, TensorFlow, and user/developer defined layers Optimization/Debugging tools • Offline network conversion tools • Debug and analyze network performance • API and SDK documentation with sample code • Ease of integration into customer applications Qualcomm® Kryo™ CPU Qualcomm® Adreno™ GPU Qualcomm® Hexagon™ DSP Sample code Offline conversion tools Analyze performance Ease of integration
- 16. 16 Robust software and tools simplify development Mobile Auto Home Camera XR NPE GoogleNet/ Inception SSD Alexnet ResNet SqueezeNet Object classification Face detection Scene segmentation Natural language understanding Speaker recognition Security/Authentication Resource management
- 17. 17 Optimized NPE Model (.dlc file) Model to runtime workflow: Training and inference Training: Machine learning experts build and train their network to solve their particular problem Inference: NPE enables the network to run on Snapdragon devices Model testing Good enough match? No Backward propagation Training data Test data Yes Deep learning frameworks (e.g., Caffe2, TensorFlow) Model conversion tools Application Model optimization tools Optional (quantization, compression, etc.) Data 1 Data 1 Model building and training NPE Enabled App NPE runtime .dlc file Model file Static weights and biases NPE Model (.dlc file)
- 18. 18 Application NPElibrary Computeruntimes High-level software architecture for the Snapdragon NPE HW Kernel space User space OS support: • ARM Android • ARM Linux • x86 Linux Kryo CPU Adreno GPU Hexagon DSP OS DSP kernel mode driver GPU kernel mode driver DL containerModel debugProfiling logging Model loader NPE API CPU QSMLSymphony User defined layers Float Application code Future User developed GPU Float OpenCL driver DSP Fixed 8 Fixed 16 FastRPC driver Half float
- 19. 19 Model Conversion Tool UDL .dlc file Model conversion tool User Defined Layer (UDL) workflow Supports prototyping of layers not yet supported by the Snapdragon NPE Note: there is runtime performance cost to inserting a UDL into a network, related to “context” switching. Execution context is transferred to user in CPU control plane. NPE model (.dlc file) UDL Model conversion tool UDL handler Model file (static weights and biases) UDL NPE workflow with UDL NPE model (.dlc file) Model conversion tool Model file (static weights and biases) NPE workflow User-provided implementation User-defined layer weights and parameters NPE runtime UDL implementation NPE runtime .dlc file
- 20. 20 Converting and quantizing a model Native model (e.g., Caffe) Model conversion tool NPE model (.dlc file) Model optimization tool NPE model (.dlc file) NPE enabled appOnly needed for certain features (e.g., activation quantization) Images Images Images Optional (quantization, compression, etc.) snpe-<framework>-to-dlc snpe-dlc-optimize snpe-<framework>-to-dlc (<framework> = Caffe | Caffe2 | TensorFlow) • Input is the model in native framework format • Output is a converted but not optimized NPE DLC file snpe-dlc-optimize • Converts non-quantized DLC models into 8-bit quantized DLC models ◦ Additionally implements further optimizations such as SVD compression • Quantized model is necessary for fixed-point NPE runtimes (e.g. DSP)
- 21. 21 Repeat API example usage NPE provides a simple C++ API with the following functionality • Load a DLC model and select the runtime • Execute the model • Debug support ◦ Dump the output of all layers in a model • Collect performance metrics ◦ Per-layer timing Green API calls are only required for UDLs Application InitializationRuntime Get input data and prepare InputTensor Process OutputTensorMap NPE new SNPEBuilder ( DLContainer ) SNPE->execute( InputTensor, OutputTensorMap ) Snapdragon NPE Object (Snapdragon NPE) SNPEBuilder Object(SBO) SBO->setRunTimeProcessor( RuntimeMode ) SBO->build( ) InputTensor = SNPEFactory: :getTensorFactory().createTensor() See SNPEBuilder.hpp for all builder APIs Process network See NativeCpp/BatchRun/ example in NPE For each UDL read from DLContainer IUDL object (IUDL) UdlFactory( UdlContext ) IUDL::Setup( ) SBO->setUdlBundle( UdlFactoryContextBundle ) IUDL->execute( InputTensor, OutputTensor )
- 22. 2222 Learn more at: developer.qualcomm.com AI + Qualcomm Technologies Massive scale of Millions of units 100s Based on Snapdragon Platforms capable of supporting Snapdragon NPE today.
- 23. 23 Qualcomm Technologies and Facebook collaboration for massive AI scale
- 24. 2424 Facebook + Qualcomm Technologies On-device AI with Snapdragon Facebook and Qualcomm’s Caffe2 collaboration • Demonstrated Caffe2 acceleration with NPE at F8 2017 • 5x performance upside on GPU (compared to CPU) • Announced commercial support of Caffe2 in July through Qualcomm Developer Network • Facebook AML has integrated the NPE with Caffe2 Future Caffe2/NPE research and development • Continue to work closely with Facebook to optimize key networks for maximum on- device performance • Enhancements to Caffe2 allowing Snapdragon specific SoC optimizations • More advanced AI-powered XR applications F8 2017 “On-device machine learning is made possible by the Qualcomm Snapdragon NPE which does the heavy lifting needed to run neural networks more efficiently on Snapdragon devices.” Source: XDA
- 25. 25 Enhancing the Facebook experience through on-device AI Augmented reality features potentially powered by AI • Style transfer and filters • Frames and masks • Photo and live videos, including 360° • Contextual awareness (e.g. location/sensor metadata) On-device acceleration benefits • Smooth UI with increased frame rate • Increased battery life More engaging social media with AI and AR *Requires network connection and will support up to 20 hours of battery life
- 26. 26 What’s next
- 27. 27 AI hardware What does the future look like? Neural processing architecture research Distributed computing architectures
- 28. 28 AI offers enhanced experiences and new capabilities for smartphones Superior photographyTrue personal assistance Extended battery life Enhanced security Natural user interfaces Enhanced connectivity A new development paradigm where things repeatedly improve
- 29. 29 AI will bring XR closer to the ultimate level of immersion Creating physical presence in real or imagined worlds Interactions Natural UI Depth estimation Sounds Audio filtering and cleanup Visuals Rendering techniques Battery efficiency Managing workload concurrency
- 30. 3030 AI is revolutionizing the car of the future Redefining the in-car experience • Natural user interfaces • Personalization • Driver awareness monitoring Paving the road to autonomy • Surround view perception • Sensor fusion • Path planning • Decision making
- 32. Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Thank you! Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm is a trademark of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.