SlideShare a Scribd company logo

FPGAs and Machine Learning

inside-BigData.com, profile picture
inside-BigData.com

The document outlines Intel's advancements in AI, machine learning, and deep learning across various sectors, emphasizing the need for diverse hardware architectures to meet increasing computing demands. It discusses tools and frameworks like OpenVINO, Intel’s deep learning acceleration suite, and highlights the performance variances in systems contingent on configurations. Additionally, it notes that optimizations are primarily intended for Intel processors and may not guarantee results on other non-Intel microprocessors.

Technology
1 of 179
Downloaded 239 times
1
2
Most read
3
4
5
Most read
6
7
8
9
10
11
12
13
14
15
Most read
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
Intel Network & Custom Logic Group Greg Nash, System Architect – Government, Aerospace & Military Jim Moawad, Technical Solution Specialist – FPGA Acceleration July 29, 2019 - Argonne Training Program on Extreme Scale Computing Public
2 © 2019 Intel Corporation Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No product or component can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. For more complete information about performance and benchmark results, visit http://www.intel.com/benchmarks . Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/benchmarks . Intel® Advanced Vector Extensions (Intel® AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at http://www.intel.com/go/turbo. Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Intel Atom, Intel Core, Intel Nervana, Intel Optane, Intel Xeon, Iris, Movidius, OpenVINO, Pentium, Celeron, Stratix, Arria and the Stratix logo and are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as property of others. notices&disclaimers
3 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
4 © 2019 Intel Corporation Wide variety of market segments incorporating AI and Deep Learning which is increasing compute demands across the board. MachineLearningAppliestoNearlyEveryMarket Achieve higher yields & increase efficiency Maximize production and uptime Transform the learning experience Enhance safety, research, and more Turn data into valuable intelligence Revolutionize patient outcomes Empower truly intelligent Industry 4.0 Create thrilling experiences Transform stores and inventory Enable homes that see, hear, and respond Drive network and operational efficiency Agriculture Energy Education Government Finance Health Industrial Media Retail SmartHome Telecom Transport Automated driving
5 © 2019 Intel Corporation No one size fits all approach to AI AI&MachineLearningisaVastFieldofStudy Deep learning Machine learning AI Supervised Learning Reinforcement Learning unSupervised Learning Regression Classification Clustering Decision Trees Data Generation Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning
6 © 2019 Intel Corporation AIIsthedrivingforce Foresight Predictive Analytics Forecast Prescriptive Analytics Act/adapt Cognitive Analytics Hindsight Descriptive Analytics insight Diagnostic Analytics DataIsDrivingtheDesireforAI AnalyticsCurve 25GBper month Internet User 1 Datadeluge(2019) Insights Business Operational Security 50GBper day Smart Car 2 3TB per day Smart Hospital 2 40TBper day Airplane Data 2 1pBper day Smart Factory 2 50PBper day City Safety 2 1. Source: http://www.cisco.com/c/en/us/solutions/service-provider/vni-network-traffic-forecast/infographic.html 2. Source: https://www.cisco.com/c/dam/m/en_us/service-provider/ciscoknowledgenetwork/files/547_11_10-15-DocumentsCisco_GCI_Deck_2014-2019_for_CKN__10NOV2015_.pdf
7 © 2019 Intel Corporation The future is a diverse mix of scalar, vector, matrix, and spatial architectures deployed in CPU, GPU, AI, FPGA and other accelerators DiverseWorkloadsrequire DIVERSEarchitectures Spatial FPGA Matrix AI Vector GPU Scalar CPU SVMS
8 © 2019 Intel Corporation Training Highly efficient multi-model inferencing for cloud, data center and appliances at scale Inference Delivering Real time workflows with AI for optimized system performance GPU discrete & Integrated, optimized for highly parallel workload acceleration Fastest time-to-train at scale with high bandwidth AI server connections for the most persistent, intense usage Foundation for all workloads including AI Intel® DL Boost for inference & training acceleration Energy efficient Edge AI for Video/Vision CPU Accelerators FPGA,VPU,GPU Multi models – high performance across broad set of existing and emerging DL models for applications like Natural Language Processing, Speech & Text, Recommendation Engines, and Video & Image search & filtering A+ - Combined workflows where use cases span AI plus additional workloads (Eg - Immersive Media - Media + Graphics in GPU or L3 forwarding in FPGA) AIComputeGuide NNP-I NNP-T
9 © 2019 Intel Corporation Applications in HPC, Big Data, HPDA, AI & more that have access to a common compute and data pool ingest Store Process AnalyzeSource(s)? Structured? Volume? DURABILITY? Streaming? LOCALITY? GOVERNANCE? Other? Tool for live streaming data ingestion from Internet of Things (IOT) sensors in endpoint devices File, block or object-based storage solution given cost, access, volume and perf requirements Integration, cleaning, normalization and more transformations on batch and/or streaming data Visit: intel.com/content/www/us/en/analytics/tame-the-data-deluge.html *Other names and brands may be claimed as the property of others Tameyourdata with a robust data layer
10 © 2019 Intel Corporation Chooseanyapproach Reinforce- ment Learning Un- supervised Learning Statistical Methods Machine Learning Clustering kmeans Neural Nets Hidden Markov Models GMM Hier- archical Supervised Learning Regres- sion Class- ification Linear SVR GPR Decision Trees Ensemble Methods Logistic Neural Nets SVM Naïve Bayes Nearest Neighbor Discri- minant Analysis from machine to deep learning Visit: software.intel.com/ai/courses
1111 © 2019 Intel Corporation 1 An open source version is available at: 01.org/openvinotoolkit *Other names and brands may be claimed as the property of others. Developer personas show above represent the primary user base for each row, but are not mutually-exclusive All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Speedupdevelopment Optimization Notice TOOLKITS App developers libraries Data scientists Kernels Library developers Open source platform for building E2E Analytics & AI applications on Apache Spark* with distributed TensorFlow*, Keras*, BigDL Deep learning inference deployment on CPU/GPU/FPGA/VPU for Caffe*, TensorFlow*, MXNet*, ONNX*, Kaldi* Open source, scalable, and extensible distributed deep learning platform built on Kubernetes (BETA) Intel-optimized Frameworks And more framework optimizations underway including PaddlePaddle*, Chainer*, CNTK* & others Python R Distributed • Scikit- learn • Pandas • NumPy • Cart • Random Forest • e1071 • MlLib (on Spark) • Mahout Intel® Distribution for Python* Intel® Data Analytics Acceleration Library (DAAL) Intel distribution optimized for machine learning High performance machine learning & data analytics library Open source compiler for deep learning model computations optimized for multiple devices (CPU, GPU, NNP) from multiple frameworks (TF, MXNet, ONNX) Intel® Math Kernel Library for Deep Neural Networks (MKL-DNN) Open source DNN functions for CPU / integrated graphics Machinelearning Deeplearning * * * * * using open AI software Visit: www.intel.ai/technology
12 © 2019 Intel Corporation DeployAIanywhere Visit: www.intel.ai/technology All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. with unprecedented hardware choice C l o u d D e v I c e e d g e Intel GPU Automated Driving Dedicated Media/Vision Flexible Acceleration Dedicated DLInference Dedicated DLTraining Graphics,Media &Analytics NNP-TNNP-I Ifneeded
13 © 2019 Intel Corporation DeeplearningDeployed Tools Containers Intel-optimized Frameworks Libraries Training Trained Model Weights Inference Cloud Intel® Distribution of OpenVINO™ Toolkit Multi-Use Cluster (Spark/HPC/other) Cloud Multi-Use Cluster (Spark/HPC/other) Dedicated Training Cluster (with Nauta) End-to-end deep learning on Intel Data Model Optimization Edge Server Device -OR- -OR- -OR- -OR- -OR- -OR- -OR- All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Optimization Notice
FPGAs and Machine Learning
15 © 2019 Intel Corporation 2nd generationINTEL®XEON®SCALABLEPROCESSOR Cascade Lake PerformanceTCO/Flexibility SecUrity Built-in Acceleration with Intel® Deep Learning Boost…  IMT – Intel® Infrastructure Management Technologies  ADQ – Application Device Queues  SST – Intel® Speed Select Technology  Intel® Security Essentials  Intel® SecL: Intel® Security Libraries for Data Center  TDT – Intel® Threat Detection TechnologyThroughput (img/s) Drop-in compatible CPU on Intel® Xeon® Scalable platform formerly known as Begin your AI journey efficiently, now with even more agility… Hardware-Enhanced Security… deep learning throughput!1 Up to 30X 1 Based on Intel internal testing: 1X,5.7x,14x and 30x performance improvement based on Intel® Optimization for Café ResNet-50 inference throughput performance on Intel® Xeon® Scalable Processor. See Configuration Details 3 Performance results are based on testing as of 7/11/2017(1x) ,11/8/2018 (5.7x), 2/20/2019 (14x) and 2/26/2019 (30x) and may not reflect all publically available security updates. No product can be absolutely secure. See configuration disclosure for details. , Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance
16 © 2019 Intel Corporation Intel®DeepLearningBoost(DLBoost) INT8 07 06 05 04 03 02 01 00 Sign Mantissa featuring Vector Neural Network Instructions (VNNI) NEW vpdpbusd OUTPUT INT32 CONSTANT INT32 INPUT INT8 INPUT INT8 AVX-512 (VNNI) instruction to accelerate INT8 convolutions: vpdpbusd INPUT INT8 INPUT INT8 vpmaddubsw vpmaddwd vpaddd OUTPUT INT16 OUTPUT INT32 CONSTANT INT16 CONSTANT INT32 OUTPUT INT32 Current AVX-512 instructions to perform INT8 convolutions: vpmaddubsw, vpmaddwd, vpaddd
17 1 5.7x inference throughput improvement with Intel® Optimizations for Caffe ResNet-50 on Intel® Xeon® Platinum 8180 Processor in Feb 2019 compared to performance at launch in July 2017. See configuration details on Config 1 Performance results are based on testing as of dates shown in configuration and may not reflect all publicly available security updates.28/24/2018) Results have been estimated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. No product can be absolutely secure. See configuration disclosure for details. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance Baseline 50x 285x 2S Intel® Xeon® Scalable Processor (Skylake) July 2017 July 2017 Skylake launch February 2019 vs. Baseline vs. Baseline DeeplearningperformanceonCPU I N T 8 V N N I Resnet-50 1.9x2 Inception v3 1.8x2 SSD-VGG16 2.0x2 Caffe Resnet-50 1.9x2 Inception v3 1.8x2 TensorFlow SKYLAKE CASCADE LAKE Hardware + Software Improvements for Intel® Xeon® Processors
18 © 2019 Intel Corporation Intel®Nervana™neuralnetworkprocessors(NNP) ¥ ‡The Intel® Nervana™ Neural Network Processor is a future product that is not broadly available today All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. NNP-T NNP-I Highly-efficient multi-model inferencing for cloud, data center and intense appliances Fastest time-to-train with high bandwidth AI server connections for the most persistent, intense usage DEDICATED DL TRAINING DEDICATED DL INFERENCE
19 © 2019 Intel Corporation FIRSTTOMARKETTOACCELERATE EvolvingAIWORKLOADS Precision Latency Sparsity AdversarialNetworks ReinforcementLearning NeuromorphicComputing … Real-time workloads Rnn Lstm SpeechWL DeliveringAI+forFlexible systemlevelfunctionality AI+I/OIngest AI+Networking Ai+security Ai+pre/postprocessing … Falcon Mesa Intel®FPGAforAI Enabling real-time AI in a wide range of embedded, edge and cloud apps All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
20 © 2019 Intel Corporation DRONES • Sense and avoid • GPS denied hovering • Pixel labeling • Video, image capture Surveillance • Detection/classification • Identification • Multi-nodal systems • Multimodal sensing • Video, image capture SMARTHOME • Detection, tracking • Perimeter, presence monitoring • Recognition, classification • Multi-nodal systems • Multimodal sensing • Video, image capture SERVICEROBOTS • Navigation • 3D Vol. mapping • Multimodal sensing WEARABLES • Detection, tracking • Recognition • Video, image, session capture AR-VRHMD • 6DOF pose, position, mapping • Gaze, eye tracking • Gesture tracking, recognition • See-through camera Intel®Movidius™Visionprocessingunit(vPU) Power-efficient image processing, computer vision & deep learning for devices All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
21 © 2019 Intel Corporation https://software.intel.com/en- us/iot/speech-enabling-dev-kit TryitTODAY! Intel® Speech Enabling Developer Kit Learn more: https://sigport.org/sites/default/files/docs/PosterFinal.pdf Amplethroughput For speech, language, and other sensing inference Lowpower <100 mW power consumption for always-on applications Flexibility Gaussian mixture model (GMM) and neural network inference support Intel® GNA (IP) DSP Intel®Gaussianneuralaccelerator(GNA) Streaming Co-Processor for Low-Power Audio Inference & More All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
22 © 2019 Intel Corporation Hardwareintegration Softwaresupport MacOS (CoreML and MPS1) Windows O/S (WinML) OpenVINO™ Toolkit (Win, Linux) clDNN Ubiquity/Scalability • Shipped in 1 billion+ Intel® SoCs • Broad choice of performance/power across Intel Atom®, Intel® Core™, and Intel® Xeon® processors Powerful,FlexibleArchitecture • Rich data type support for 32bitFP, 16bitFP, 32bitInteger, 16bitInteger with SIMD multiply-accumulate instructions MediaLeadership • Intel® Quick Sync Video – fixed-function media blocks to improve power and performance • Intel® Media SDK – API that provides access to hardware-accelerated codecs MemoryArchitecture • Shared memory architecture on die between CPU and GPU to enable lower latency and power Intelintegratedprocessorgraphics Built-in deep learning inference acceleration All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
23 © 2019 Intel Corporation Visit www.mobileye.com Mobileye–anintelcompany Automated Driving Platform All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
24 Framework Topology Library DeeplearningSoftware Provides a standard way to customize and deploy Deep Neural Networks Supplies generic functionality which can be selectively extended with user defined additions to enhance specific tasks A set of algorithms modeled loosely after the human brain that forms a “network” designed to recognize complex patterns Also referred to as a network, NN, or model. Highly optimized functions intended to be used in Neural Network implementations to maximize performance Libraries are framework and model agnostic, hardware specific MACHINE LEARNING LIBRARIES Python R Distributed • Scikit-learn • Pandas • NumPy • Cart • Random Forest • e1071 • MlLib (on Spark) • Mahout DAAL MKL-DNN clDNN Intel® Data Analytics Acceleration Library (includes machine learning) Open-source deep neural network functions for CPU / integrated graphics
25 © 2019 Intel Corporation Frameworks AmazonGoogle FacebookMicrosoft https://github.com/onnx/onnx Open Neural Network Exchange (ONNX) • Enabling interoperability between different frameworks • Streamlining the path from research to production Caffe2, PyTorch, Microsoft Cognitive Toolkit, Apache MXNet ONNX promises to ease interoperability between frameworks.
26 ConvolutionalNeuralNetwork(CNN) → Layers of neurons connected by learned weights → Three layer classes: • Convolution Layer • Pooling Layer • Fully connected Layer → Connections between the layers point in one direction (Feed Forward Network) → Used to extract features from images for recognition or detection RecurrentNeuralNetwork(RNN) → Allows long-term dependencies to affect the output by relying on internal memory to process inputs → Types of RNN: • LSTM (Long Short Term Memory) • GRU (Gated Recurrent Unit) → Designed to analyze sequential data (i.e., NLP) = x1 x2 x3 x4 y1 y2 y3
27 © 2019 Intel Corporation IMAGE RECOGNITION http://ieeexplore.ieee.org/document/7124883/ Workload: ResNet50, InceptionV3, SSD-VGG16 SPEECH RECOGNITION https://www.news- medical.net/whitepaper/20170821/Speech-Recognition-in- Healthcare-a-Significant-Improvement-or-Severe- Headache.aspx Workload: DeepSpeech2 TEXT PROCESSING https://www.microsoft.com/en-us/translator/mt.aspx#nnt Workload: Neural Machine Translation CNN RNN
28 © 2019 Intel Corporation Purpose Neural Network Type Neural Network Example Image Recognition Classify image(s) CNN - Convolutional Neural Network ResNet50, Inception V3, Inception_ResV2, MobileNet, SqueezeNet, DenseNet Object Detection Locate AND classify object(s) in image CNN SSD– Single Shot Detector, R-FCN, Yolo_V2, VGG_16, DDRN/D-DBPN Natural Language Processing (NLP) Extract context & meaning from text LTSM (RNN)- Long Short Term Memory Google NMT Speech Recognition Convert speech to text RNN - Recursive Neural Network Baidu DeepSpeech, Transformer Text-to-Speech Convert text to speech GAN - Generative Adversarial Network with CNN/RNN WaveNet Recommendation Systems Recommend ads, search, apps, etc. MLP - Multilayer Perceptron Wide & Deep MLP, NCF Data Generation Create images like training data GAN with CNN DCGAN, DRAW Reinforcement Learning Learning from feedback on action CNN or CNN+RNN A3C All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. TopologiesforDeeplearning Golden Retrieve r
30 © 2019 Intel Corporation Comparisonofdifferentnetworks An Analysis of Deep Neural Network Models for Practical Applications Alfredo Canziani & Eugenio Culurciello & Adam Paszke Single-Crop top-1 validation accuracies AlexNet depth: 8 layers, GoogleNet depth: 22 layers, VGG depth: 16 layers or 19 layers, ResNet depth: 50 layers or 101 layers Single-Crop top-1 validation accuracies versus amount of operation for a single forward pass Legend in gray circles in the image shows number of network parameters The number of network parameters captured and depth of the networks is not necessarily linear with Top-1 accuracy that can be achieved CNN
31 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
FPGAs and Machine Learning
33 © 2019 Intel Corporation Deep Learning Development Cycle Data acquisition and organization Integrate trained models with application code Create models Adjust models to meet performance and accuracy objectives Intel® Distribution OpenVINO™ Toolkit Provides Deployment from Intel® Edge to Cloud
34 © 2019 Intel Corporation Lots of Labeled Data! Training Inference Forward Backward Model Weights Forward “Bicycle”? “Strawberry” “Bicycle”? Error Human Bicycle Strawberry ?????? Data Set Size Accuracy Did You Know? Training requires a very large data set and deep neural network (many layers) to achieve the highest accuracy in most cases Deep Learning: Training vs. Inference
35 © 2019 Intel Corporation Latest version is 2018 R5 What it is A toolkit to accelerate development of high performance computer vision & deep learning into vision applications from device to cloud. It enables deep learning on hardware accelerators and easy deployment across multiple types of Intel® platforms. Who needs this product?  Computer vision/deep learning software developers  Data scientists  OEMs, ISVs, System Integrators Usages Security surveillance, robotics, retail, healthcare, AI, office automation, transportation, non-vision use cases (speech, text) & more. Free Download  software.intel.com/openvino-toolkit Open Source version  01.org/openvinotoolkit Highperformance,performaiattheedge Streamlined&Optimizeddeeplearninginference Heterogeneous,cross-platformflexibility
36 © 2019 Intel Corporation Intel®distributionofOpenvino™toolkit software.intel.com/openvino-toolkit Supports 100+ public models, incl. 30+ pretrained models D E E P L E A R N I N G C O M P U T E R V I S I O N Model Optimizer Inference Engine Computer vision library (kernel & graphic APIs) Multiple products launched based on this toolkit Breadth of product portfolioRapid adoption by developers SupportsmajorAIframeworks Cross-platformflexibility HighPerformance,highEfficiency Optimized media encode/decode functions OpenCL™ Strong Adoption + Rapidly Expanding Capability Optimization Notice
37 © 2019 Intel Corporation Caffe* TensorFlow* MxNet* .dataIR IR IR = Intermediate Representation format Load, infer CPU Plugin GPU Plugin FPGA Plugin NCS Plugin Model Optimizer Convert & Optimize Model Optimizer  What it is: A Python*-based tool to import trained models and convert them to Intermediate representation.  Why important: Optimizes for performance/space with conservative topology transformations; biggest boost is from conversion to data types matching hardware. Inference Engine  What it is: High-level inference API  Why important: Interface is implemented as dynamically loaded plugins for each hardware type. Delivers best performance for each type without requiring users to implement and maintain multiple code pathways. Trained Models Inference Engine Common API (C++ / Python) Optimized cross- platform inference OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos GPU = Intel CPU with integrated graphics processing unit/Intel® Processor Graphics Kaldi* ONNX* (Pytorch, Caffe2 & more) GNA Plugin Extendibility C++ Extendibility OpenCL™ Extendibility OpenCL™ VAD Plugin VAD = Vision Accelerator Design Products; includes FPGA and 8 MyriadX versions Optimization Notice Intel®DeeplearningdeploymenttoolkitFor Deep Learning Inference – Part of Intel® Distribution of OpenVINO toolkit
38 © 2019 Intel Corporation Deploy/Visualize App Developers Model Data Scientists Prepare Data Engineers LibraryDevelopers AISoftwareinThebigpicture Ingest/Store Architects API’s Enterprise Applications Other names and brands may be claimed as the property of others. Note: displayed logos are not a complete list of solutions/providers in each category, personas are not mutually-exclusive by workflow step, and categorization is generalized Manage Metadata And more… Collect, Integrate, ETL & ELT Open (Managed)Open And more… Proprietary Pre-Process Data And more… Visualize Proprietary And more… Machine Learning & Analytics Proprietary And more… Store & Manage Big Data Open (Managed)Open And more…Proprietary: IT Systems Management And more… Unified Workflows And more… Intel® DL Studio Deploy Inference And more… Deep Learning And more… Optimization Notice
39 © 2019 Intel Corporation Quick Guide: What’s Inside the Intel Distribution vs Open Source version of OpenVINO™ toolkit 1Intel MKL is not open source but does provide the best performance 2Refer to readme file for validated versions Tool/Component Intel® Distribution of OpenVINO™ toolkit OpenVINO™ toolkit (open source) Open Source Directory https://github.com Installer (including necessary drivers) P Intel® Deep Learning Deployment toolkit Model Optimizer P P /opencv/dldt/tree/2018/model-optimizer Inference Engine P P /opencv/dldt/tree/2018/inference-engine Intel CPU plug-in P Intel® Math Kernel Library (Intel® MKL) only1 P BLAS, Intel® MKL1 , jit (Intel MKL) /opencv/dldt/tree/2018/inference-engine Intel GPU (Intel® Processor Graphics) plug-in P P /opencv/dldt/tree/2018/inference-engine Heterogeneous plug-in P P /opencv/dldt/tree/2018/inference-engine Intel GNA plug-in P Intel® FPGA plug-in P Intel® Neural Compute Stick (1 & 2) VPU plug-in P Intel® Vision Accelerator based on Movidius plug-in P 30+ Pretrained Models - incl. Model Zoo (IR models that run in IE + open sources models) P P /opencv/open_model_zoo Samples (APIs) P P /opencv/dldt/tree/2018/inference-engine Demos P P /opencv/open_model_zoo Traditional Computer Vision OpenCV* P P /opencv/opencv OpenVX (with samples) P Intel® Media SDK P P2 /Intel-Media-SDK/MediaSDK OpenCL™ Drivers & Runtimes P P2 /intel/compute-runtime FPGA RunTime Environment, Deep Learning Acceleration & Bitstreams (Linux* only) P
40 © 2019 Intel Corporation Expedite development, accelerate deep learning inference performance, and speed production deployment. Speed Deployment with Pre-trained Models & Samples  Age & Gender  Face Detection–standard & enhanced  Head Position  Human Detection–eye-level & high-angle detection  Detect People, Vehicles & Bikes  License Plate Detection: small & front facing  Vehicle Metadata  Human Pose Estimation  Action recognition – encoder & decoder  Text Detection & Recognition  Vehicle Detection  Retail Environment  Pedestrian Detection  Pedestrian & Vehicle Detection  Person Attributes Recognition Crossroad  Emotion Recognition  Identify Someone from Different Videos– standard & enhanced  Facial Landmarks  Gaze estimation Pretrained Models in Intel® Distribution of OpenVINO™ toolkit  Identify Roadside objects  Advanced Roadside Identification  Person Detection & Action Recognition  Person Re-identification–ultra small/ultra fast  Face Re-identification  Landmarks Regression  Smart Classroom Use Cases  Single image Super Resolution (3 models)  Instance segmentation  and more… Binary Models  Face Detection Binary  Pedestrian Detection Binary  Vehicle Detection Binary  ResNet50 Binary
41 © 2019 Intel Corporation  Object Detection  Hello Infer Classification  Standard & Pipelined Image Classification  Interactive Face Detection  Security Barrier  Image Segmentation  Object Detection SSD  Validation Application  Neural Style Transfer  Multi-channel Face Detection  Object Detection for Single Shot Multibox Detector using Asynch API+ Save Time with Deep Learning Samples Use Model Optimizer & Inference Engine for public models & Intel pretrained models
42 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
43
44 VisionInferencerequirementsVary Design Variable Considerations Performance Frames/sec, latency (real-time constraints) Power Power envelope, power per inference Cost Hardware cost, development cost, cost per inference Batch size 1, 8, 32, etc. Precision FP32, FP16, FP11 Image size 720p, 1080p, 4K Camera input Raw video, encoded Network support CNN, RNN, LSTM, custom Operating environment Temperature controlled environment, outdoor use, 24/7 operation Product life Operational lifespan, product availability Regulation Security, privacy There is no “one-size-fits-all” solution . . . . . . and sometimes requirements change
45 • Field Programmable Gate Array (FPGA) – Millions of logic elements – Thousands of embedded memory blocks – Thousands of DSP blocks – Programmable routing – High speed transceivers – Various built-in hardened IP • Used to create Custom Hardware • Well-suited for Matrix Multiplation DSP Block Memory Block Programmable Routing Switch Logic Modules FPGAOverview
46 Intel® FPGA PAC N3000 for networking Intel® FPGA PAC D5005 for Datacentre Mustang-F-100-A10* Intel®FPGA–applicationAccelerationfromEdgetoCloud Combining Intel FPGA hardware and software to efficiently accelerate workloads for processing-intense tasks Programmable Acceleration Card (PAC) offering Devices/edge Network/NFV Cloud/enterprise * Other names and brands may be claimed as the property of others. Intel® FPGA PAC with Arria®10 GX
47 WhyFPGAForDeepLearningInference Keydifferentiators• Low latency • Low power • High throughput (FPS) • Flexibility to adapt to new, evolving, and custom networks • Supports large image sizes (e.g., 4K) • Large networks (up to 4 billion parameters) • Security / Encryption • Wide ambient temperature range (-40° C to 105° C)* • 24/7/365 operation • Long lifespan (8–10 years) Vision Acceleration Design with Intel® Arria® 10 FPGA *-40°C to 105°C temp range is for chip down design. Board temp range is 0° to 65°C.
48 IncreaseDeepLearningWorkloadPerformanceonPublicModels usingIntel®DistributionofOpenVINO™toolkit&Intel®Architecture Optimize with OpenVINO on CPU, Get an even Bigger Performance Boost with Intel® FPGA 1Depending on workload, quality/resolution for FP16 may be marginally impacted. A performance/quality tradeoff from FP32 to FP16 can affect accuracy; customers are encouraged to experiment to find what works best for their situation. Performance results are based on testing as of June 13, 2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Configuration: Testing by Intel as of June 13, 2018. Intel® Core™ i7-6700K CPU @ 2.90GHz fixed, GPU GT2 @ 1.00GHz fixed Internal ONLY testing, Test v3.15.21 – Ubuntu* 16.04, OpenVINO 2018 RC4, Intel® Arria® 10 FPGA 1150GX. Tests were based on various parameters such as model used (these are public), batch size, and other factors. Different models can be accelerated with different Intel hardware solutions, yet use the same Intel software tools. Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Public Models (Batch Size) RelativePerformance Improvement Standard Caffe* Baseline 19.9x1 ComparisonofFramesperSecond(FPS)
49 Solving Machine Learning Challenges with FPGA Real-Time deterministic lowlatency Ease-of-use softwareabstraction, platforms&libraries Flexibility customizablehardware fornextgenDNNarchitectures Intel FPGAs can be customized to enable advances in machine learning algorithms. Intel FPGA hardware implements a deterministic low latency data path unlike any other competing compute device. Intel FPGA solutions enable software-defined programming of customized machine learning accelerator libraries.
50 Public Intel FPGA Machine Learning Success Microsoft: “Microsoft has revealed that Altera FPGAs have been installed across every Azure cloud server, creating what the company is calling “the world’s first AI supercomputer.” -Sept 2016 NEC: “To create the NeoFace Accelerator, the engine software IP was integrated into an Intel Arria 10 FPGA, which operate in Xeon processor–based servers. ” -June 2017 JD.COM: “Arria®10 FPGA can achieve 5x improvement in the performance of LSTM accelerator card compared to GPU.” –Apr 2017 Inspur/iFlytech: “Leading server vendor Inspur Group and Altera today launched a speech recognition acceleration solution based on Altera's Arria® 10 FPGAs and DNN algorithm from iFLYTEK.” -Dec 2016 ZTE: “Using Intel’s Arria 10 FPGA, ZTE engineers were able to achieve more than 1000 images per second in facial recognition.” –Jan 2017
51 Why FPGAs for Deep Learning? PE PE PE PE PE PE PE PE feeder feeder feeder feeder feeder feeder feeder feeder PE PE PE PE PE PE PE PE
52 Customized Performance Convolutional Neural Networks are Compute Intensive Feature Benefit Highly parallel architecture Facilitates efficient low-batch video stream processing and reduces latency Configurable Distributed Floating Point DSP Blocks FP32 9Tflops, FP16, FP11 Accelerates computation by tuning compute performance Tightly coupled high-bandwidth memory >50TB/s on chip SRAM bandwidth, random access, reduces latency, minimizes external memory access Programmable Data Path Reduces unnecessary data movement, improving latency and efficiency Configurability Support for variable precision (trade-off throughput and accuracy). Future proof designs, and system connectivity Fine-grained & low latency between compute and memory Convolutional Neural Networks are Compute Intensive Function 2Function 1 Function 3 IO IO Optional Memory Optional Memory Pipeline Parallelism
53 © 2019 Intel Corporation FPGAs System Flexibility to Control the Data path Storage Acceleration  Machine learning  Cryptography  Compression  Indexing Inline Data Flow Processing  Machine learning  Object detection and recognition  Advanced driver assistance system (ADAS)  Gesture recognition  Face detection Compute Acceleration/Offload  Workload agnostic compute  FPGAaaS  Virtualization Intel® Xeon® Processor
54 © 2019 Intel Corporation FPGA Flexibility Supports Arbitrary Architectures Many efforts to improve efficiency – Batching – Reduce bit width – Sparse weights – Sparse activations – Weight sharing – Compact network I W O I X W = ··· ··· O 2 3 3 2 1 3 13 1 3 Shared Weights SparseCNN [CVPR’15] Spatially SparseCNN [CIFAR-10 winner ‘14] Pruning [NIPS’15] TernaryConnect [ICLR’16] BinaryConnect [NIPS’15] DeepComp [ICLR’16] HashedNets [ICML’15] XNORNet SqueezeNet LeNet [IEEE} AlexNet [ILSVRC’12} VGG [ILSVRC’14} GoogleNet [ILSVRC’14} ResNet [ILSVRC’15}
55 © 2019 Intel Corporation FPGAs Provide Deterministic System Latency FPGAs can leveraging parallelism across the entire chip reducing the compute time to a fraction F P G A X e o n I/O Latency Compute Latency System Latency = I/O Latency + Compute Latency 0.25ms 1ms 0.3ms
56 © 2019 Intel Corporation Deep Learning Topology Inference Processing “head” 1 “head” 2 “head” 10 Neural net “the body” image … Most of the compute is here Vision: CNNs (ResNet) Similar for Speech and Translation features Feature vector for index Tags Object detect Post-processing Intel® FPGA Deep Learning Acceleration Suite Re-size / crop image Pre-processing
57 © 2019 Intel Corporation Intel® FPGA Deep Learning Acceleration Suite Features  CNN acceleration engine for common topologies executed in a graph loop architecture – AlexNet, GoogleNet, LeNet, SqueezeNet, VGG16, ResNet, Yolo, SSD…  Software Deployment – No FPGA compile required – Run-time reconfigurable  Customized Hardware Development – Custom architecture creation w/ parameters – Custom primitives using OpenCL™ flow Convolution PE Array Crossbar prim prim prim custom DDR Memory Reader /Writer Feature Map Cache DDR Config Engine
58 © 2019 Intel Corporation FPGA Usage with Intel® Distribution of OpenVINO™ toolkit  Supports common software frameworks (Caffe*, TensorFlow*)  Model Optimizer enhances model for improved execution, storage, and transmission  Inference Engine optimizes inference execution across Intel® hardware solutions using unified deployment API  Intel® FPGA DLA Suite provides turn-key or customized CNN acceleration for common topologies GoogLeNet Optimized Template ResNet Optimized Template Additional, Generic CNN Templates SqueezeNet Optimized Template VGG Optimized Template Deep Learning Deployment Toolkit Intel® Xeon® Processor Conv PE Array Crossbar DDR Memor y Reader /Writer Feature Map Cache DDR DDR DDR Config Engine Trained Model Caffe*, TensorFlow*, etc… Heterogenous CPU/FPGA Deployment Optimized Acceleration Engine Pre-compiled Graph Architectures Hardware Customization Supported Model Optimizer FP Quantize Model Compress Model Analysis Inference Engine DLA Runtime Engine MKL-DNN Intel® FPGA Intermediate Representation *Other names and brands may be claimed as the property of others
59 © 2019 Intel Corporation OpenVINO™ with DLA User Flows IP Architect Neural Net Design Intel® FPGA SDK for OpenCL™ Bitstream Library Application Developer Compile User Customization of DLA Suite Source Code OpenVINO™ toolkit Turnkey Software Deployment Flow FPGA Architecture Development Flow Design ProgramRun Choose from many precompiled FPGA Images Or Custom create FPGA bitstream
60 © 2019 Intel Corporation Machine Learning on Intel® FPGA Platform Acceleration Stack Platform Solution DLA Runtime Engine DLA Workload OpenCL™ Runtime FPGA Interface Manager Hardware Platform & IP Software Stack DL Deployment Toolkit Acceleration Stack Application Intel® Programmable Acceleration Card Intel® Xeon CPU ML Framework (Caffe*, TensorFlow*) Intel® FPGA Acceleration Hub Edge Computing Solution DLA Runtime Engine DLA Workload OpenCL™ Runtime OpenCL™ BSP Hardware Platform & IP Software Stack DL Deployment Toolkit OpenCL™ BSP Driver Application Intel® Vision Accelerator Design with FPGA Intel® CPU ML Framework (Caffe*, TensorFlow*) Intel® Vision Accelerator Design Products
61 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
62 Intel® FPGADeepLearningAcceleration(DLAforOpenVINO) Pre-compiledGraphArchitecture ExampleTopologies DDR DDR DDR DDR Configuration Engine AlexNet GoogLeNet Tiny Yolo SqueezeNetVGG16 ResNet 18 …* ResNet 50ResNet 101 Crossbar CUSTOM* PRIM Conv PE Array Feature Map Cache PRIM PRIM *More topologies added with every release MobileNet ResNet SSD SqueezeNet SDD Memory Reader /Writer Convolution Processing Element Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine *Deeper customization options COMING SOON! * Other names and brands may be claimed as the property of others.
63 © 2019 Intel Corporation DLA Architecture: Built for Performance  Maximize Parallelism on the FPGA – Filter Parallelism (Processing Elements) – Input-Depth Parallelism – Winograd Transformation – Batching – Feature Stream Buffer – Filter Cache  Choosing FPGA Bitstream – Data Type / Design Exploration – Primitive Support ReLU Convolution / Fully Connected Norm MaxPool Stream Buffer Conv PE Array Crossbar ReL U Max Pool DDR Memory Reader /Writer Feature Map Cache DDR DDR DDR Config EngineNor m Execute
64 © 2019 Intel Corporation CNN Computation in One Slide Inew 𝑥 𝑦 = 𝑥′=−1 1 𝑦′=−1 1 Iold 𝑥 + 𝑥′ 𝑦 + 𝑦′ × F 𝑥′ 𝑦′ Input Feature Map (Set of 2D Images) Filter (3D Space) Output Feature Map Repeat for Multiple Filters to Create Multiple “Layers” of Output Feature Map
65 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer
66 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
67 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
68 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Stream Buffer output input Blocks are run-time reconfigurable and bypassable
69 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
70 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input MaxPool
71 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
72 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
73 © 2019 Intel Corporation Mapping Graphs in DLA Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
74 © 2019 Intel Corporation Efficient Parallel Execution of Convolutions  Parallel Convolutions – Different filters of the same convolution layer processed in parallel in different processing elements (PEs)  Vectored Operations – Across the depth of feature map  PE Array geometry can be customized to hyperparameters of given topology FPGA Double-Buffer On-Chip RAM Filters (on-chip RAM) FilterParallelism (OutputDepth) External DDR
75 © 2019 Intel Corporation Winograd Transformation  Perform convolutions with fewer multiplication – Allows more convolutions to be done on FPGA  Take 6 input features elements and 3 filter elements – Standard convolution requires 12 multiplies – Transformed convolution requires just 6 multiplies ReLU Convolution / Fully Connected Norm MaxPool Winograd Transform Stream Buffer Winograd Transform
76 © 2019 Intel Corporation Fully Connected Computation and Batching  Fully Connected Layer computation does not allow for data reuse of weights – Different from convolutions – Very memory bandwidth intensive  Solution: Batch up images – Weights reused across multiple images 𝑜 = 𝑰 𝒗𝒆𝒄 ∗ 𝑾 𝒗𝒆𝒄 … 𝑶 𝒗𝒆𝒄 = 𝑰 𝒎𝒂𝒕 ∗ 𝑾 𝒗𝒆𝒄 Batching
77 © 2019 Intel Corporation Feature Cache Feature data cached on-chip  Streamed to a daisy chain of parallel processing elements  Double buffered – Overlap convolution with cache updates – Output of one subgraph becomes input of another – Eliminates unnecessary external memory accesses Double-Buffer On-Chip RAM Stream buffer size
78 © 2019 Intel Corporation Filter Cache Filter weights cached in each processing element  Double buffered in order to support prefetching – While one set is used to calculate output feature maps, another set is prefetched DDR Conv Conv DDR
79 © 2019 Intel Corporation Design Exploration with Reduced Precision Tradeoff between performance and accuracy  Reduced precision allows more processing to be done in parallel  Using smaller Floating Point format does not require retraining of network  FP11 benefit over using INT8/9 – No need to retrain, better performance, less accuracy loss FP11 FP10 FP9 Sign, 5-bit exponent, 10-bit mantissaFP16 Sign, 5-bit exponent, 5-bit mantissa Sign, 5-bit exponent, 4-bit mantissa Sign, 5-bit exponent, 3-bit mantissa
80 © 2019 Intel Corporation Support for Different Topologies Tradeoff between features and performance Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine LRN Permute Concat FlattenSoftMax Reshape vs
81 © 2019 Intel Corporation Optimize for Best Performance Tradeoff between size of Feature Map cache and convolutional PE array Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine vs Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine
82 FeatureCacheandPEArrayTradeoffs Performance Feature Cache Size Small Feature cache Bigger PE array, but more slicing Performance bottleneck = DDR Big Feature cache Less slicing, but smaller PE array Performance bottleneck = PE compute Sweet spot varies based on graph dimensions PE Array Size
83 FeatureCacheandPEArrayTradeoffs
84 Featurecacheandslicingforbiggraphs Convolution 73x73x128 73x73x32 Un-Sliced Graph 720 M20Ks Convolution Slice#1 73x73x128 73x37x32 Sliced Graph Feature Cache Feature Cache 180 M20Ks Convolution Slice#2 DDR Feature Cache Feature Cache DDR Feature Cache Required: 900 M20Ks 73x36x32 73x37x128 73x36x128 73x73x16 Feature Cache Required: 384 M20Ks
85 © 2019 Intel Corporation Precompiled Architectures (aocx) included with OpenVINO™ toolkit  Precompiled and included architectures are optimized for standard topologies  Better performance maybe achieved with custom architectures for custom deep learning networks – Size requirement maybe different from standard topology – Primitive requirement may be different from standard networks
86 © 2019 Intel Corporation Supported Primitives and Topologies TopologiesPrimitives  Conv  ReLu, Leaky ReLU  Concat  Eltwise  Pooling  Power  ScaleShift  Batch Norm  Fully Connected  LRM Normalization  Custom  AlexNet  GoogLeNet  SSD  ResNet-18/50/101/152  SSD  SqueezeNet  SSD  VGG-16/19  SSD  Tiny Yolo  LeNet  MobileNet v1/v2
87 © 2019 Intel Corporation Customizable DLA Architecture  Many aspects of a DLA architecture can be customized – Convolution Processing Element parallelization – Datatype – Primitive Support – Stream Buffer Size – Custom Primitives – etc. Convolution PE Array Crossbar prim prim prim DDR Memory Reader /Writer Feature Map Cache DDR Config Engine Customizable prim
88 © 2019 Intel Corporation End-to-End Machine Learning 1 Train 2 Prepare model 3 Inference Trained Model Caffe* Tensorflow* MXNet* ONNX* Model Optimizer Optimized Model GPUCPU MKLDNN clDNN Inference Engine Real-time Data DLA Runtime FPGA
89 © 2019 Intel Corporation DLA SW API Intel FPGA Deep Learning Acceleration Suite GoogleNet Optimized Template ResNet Optimized Template Additional, Generic CNN Templates SqueezeNet Optimized Template VGG Optimized Template • Supports common software frameworks (Caffe, Tensorflow) • Intel DL software stack provides graph optimizations • Intel FPGA Deep Learning Acceleration Suite provides turn- key or customized CNN acceleration for common topologies Caffe TensorFlow Intel® Xeon® Processor Intel Arria® FPGA Inference Engine Model Optimizer Conv PE Array Crossbar DD R Memor y Reader /Writer Feature Map Cache DD R DD R DD R Config Engine Optimized Acceleration Engine Standard ML Frameworks Intel Deep Learning Deployment Toolkit Heterogenous CPU/FPGA Deployment Pre-compiled Graph Architectures Hardware Customization Supported
90 © 2019 Intel Corporation Intel® FPGA Deep Learning Acceleration Suite  Contents – Pre-compiled FPGA Bitstreams – To support various layers required by different networks with different precisions – Primitives supported: – Convolution, Fully Connected, Inner Product, ReLU, pReLU, Pooling, Norm, Concat, Elsewise Add, etc… – Software runtime library – Board Support Packages to support various boards – Tools – Diagnose – Bitstream customization and generation
91 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
92 © 2019 Intel Corporation
93 © 2019 Intel Corporation Model Optimizer  Convert models from various frameworks (Caffe, TensorFlow, MXNet, ONNX)  Converts to a unified Model (IR, later n-graph)  Optimizes topologies (Node merging, batch normalization elimination, performing horizontal fusion)  Folds constants paths in graph
94 © 2019 Intel Corporation Improve Performance with Model Optimizer  Easy to use, Python*-based workflow does not require rebuilding frameworks.  Import Models from various supported frameworks - Caffe*, TensorFlow*, MXNet*, ONNX*, Kaldi*.  100+ models for Caffe, MXNet and TensorFlow validated. All public models on ONNX* model zoo supported.  With Kaldi support, the model optimizer extends inferencing for non-vision networks.  IR files for models using standard layers or user-provided custom layers do not require Caffe.  Fallback to original framework is possible in cases of unsupported layers, but requires original framework. Trained Model Model Optimizer Analyze Quantize Optimizetopology Convert Intermediate Representation (IR) file
95 © 2019 Intel Corporation Model Optimizer Model optimizer performs generic optimization:  Node merging  Horizontal fusion  Batch normalization to scale shift  Fold scale shift with convolution  Drop unused layers (dropout)  FP16/Int8 quantization  Model optimizer can add normalization and mean operations, so some preprocessing is ‘added’ to the IR --mean_values (104.006, 116.66, 122.67) --scale_values (0.07, 0.075, 0.084) FP11 no no no yes INT8 yes no no no
96 © 2019 Intel Corporation Model Optimization Techniques Linear Operation Fusing & Grouped Convolutions Fusing Linear Operation Fusing: 3 stages 1. BatchNorm and ScaleShift decomposition: BN layers decomposes to Mul->Add->Mul->Add sequence; ScaleShift layers decomposes to Mul- >Add sequence. 2. Linear operations merge: Merges sequences of Mul and Add operations to the single Mul->Add instance. 3. Linear operations fusion: Fuses Mul and Add operations to Convolution or FullybConnected layers. Grouped Convolutions Fusing Specific optimization that applies for TensorFlow* topologies. (Xception*) Caffe Resnet269 block (from Netscope) Merged Caffe* Resnet269 block (from Netscope*) Split->Convolutions->Concat block from TensorBoard*
97 © 2019 Intel Corporation Model Optimizer: Linear Operation Fusing Example 1. Remove Batch normalization stage. 2. Recalculate the weights to ‘include’ the operation. 3. Merge Convolution and ReLU into one optimized kernel.
98 © 2019 Intel Corporation Model Optimizer: Cutting Off Parts of a Model Model optimizer can cut out a portion of the network:  Model has pre/post-processing parts that cannot be mapped to existing layers.  Model has a training part that is not used during inference.  Model is too complex and cannot be converted in one shot. Command line options  Model Optimizer provides command line options --input and --output to specify new entry and exit nodes ignoring the rest of the model:  --input option accepts a comma-separated list of layer names of the input model that should be treated as new entry points to the model;  --output option accepts a comma-separated list of layer names of the input model that should be treated as new exit points from the model.
99 © 2019 Intel Corporation Intermediate Representation (IR) Squeezenet1.1.caffemodel Squeezenet1.1.bin Squeezenet1.1.prototxt Squeezenet1.1.xml
100 © 2019 Intel Corporation Model Optimizer Options Option for Deployment Description --input_model Network binary weights file TensorFlow* .pb Caffe* .caffemodel MXNet* .params --input_proto Caffe .prototxt file --data_type IP Precision (i.e. FP16) --scale Network normalization factor (Optional) --output_dir Output directory path (Optional) Python script: $MO_DIR/mo.py Full Model Optimizer options covered in OpenVINO™ documentations
101 © 2019 Intel Corporation Run Model Optimizer  To generate IR .xml and .bin files for Inference Engine $ source $MO_DIR/venv/bin/activate $ cd $MO_DIR/ $ python mo.py --input_model <model dir>/<weights>.caffemodel --scale 1 --data_type FP16 --output_dir <output dir> Model Optimizer arguments Batch: 1 Precision of IR: FP16 Enable fusing: True Enable gfusing: True … [ SUCCESS ] Generated IR model. [ SUCCESS ] XML file: … [ SUCCESS ] BIN file: …
102 © 2019 Intel Corporation
103 © 2019 Intel Corporation Inference Engine Common API Plug-InArchitecture Inference Engine Runtime Movidius API Intel® Movidius™ Myriad™ X DLA Intel® Processor Graphics (GPU) CPU: Intel® Xeon®/Core™/Atom® clDNN Plugin Intel® Math Kernel Library (MKLDNN)Plugin OpenCL™Intrinsics FPGA Plugin Applications/Service Intel® Arria 10 FPGA  Simple & Unified API for Inference across all Intel® architecture  Optimized inference on large IA hardware targets (CPU/GEN/FPGA)  Heterogeneity support allows execution of layers across hardware types  Asynchronous execution improves performance  Futureproof/scale your development for future Intel® processors Transform Models & Data into Results & Intelligence Movidius Plugin OpenVX and the OpenVX logo are trademarks of the Khronos Group Inc. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos GPU = Intel CPU with integrated graphics/Intel® Processor Graphics/GEN GNA = Gaussian mixture model and Neural Network Accelerator GNA API Intel® GNA GNA* Plugin Optimal Model Performance Using the Inference Engine
104 Create IE Instance Load IE Plugin (MKL-DNN/clDNN/DLA) Read an IR Model Set Target Device (CPU/GPU/Movidius/FPGA) Load network to plugin Allocate input, output buffers Fill input buffer with data Run inference Interpret output results Initialization Main loop Initialize • Fill Input • Infer • Interpret Output Inference Engine Workflow
105 © 2019 Intel Corporation Inference Engine Details A runtime with a unified API for integrating inference with application logic – Delivers optimized inference solution with reduced footprint on inference execution for target hardware  libinference_engine.so library implements core functions – Loading and parsing of model IR – Preparation of input and output blobs – Triggers inference using specified plug-in  Include file: inference_engine.hpp
106 © 2019 Intel Corporation Inference Engine Plugins  CPU MKLDNN Plugin (Intel® Math Kernel Library for Deep Neural Networks) – Supports Intel® Xeon®/Core®/Atom® platform – Widest set of network classes supported, easiest way to enable topology  GPU clDNN Plugin (Compute Library for Deep Neural Networks) – Supports 9th generation and above Intel® HD and Iris graphics processors – Extensibility mechanism to develop custom layers through OpenCL™  FPGA DLA Plugin – Supports Intel® FPGAs – Basic set of layers are supported on FPGA, non-supported layers inferred through other plugins
107 © 2019 Intel Corporation Heterogeneous Plugin  Enables inference of one network on several devices. – Calculate heaviest parts of network on accelerator, unsupported fallback to CPU – To utilize all available hardware more efficiently during one inference  Heterogeneous execution can be triggered in two independent steps: – Loading a network to the Heterogeneous plugin, splitting the network to parts, and executing them through the plugin – Setting of affinity to layers (binding them to devices in InferenceEngine::ICNNNetwork) – Can be done automatically or in manual mode
108 © 2019 Intel Corporation Pre-processing  Most image formats are interleaved (RGB, BGR, BGRA, etc.)  Models usually expect RGB planar format: – R-plane, G-plane, B-plane Interleaved Planar
109 © 2019 Intel Corporation Post-processing Developers are responsible for parsing inference output. Many output formats are available. Some examples include: • Simple Classification (alexnet*): an array of float confidence scores, # of elements=# of classes in the model • SSD: Many “boxes” with a confidence score, label #, xmin,ymin, xmax,ymax Unless a model is well documented, the output pattern may not be immediately obvious.
110 © 2019 Intel Corporation Inference Engine Classes Class Details InferencePlugin Main plugin interface PluginDispatcher Finds suitable plug-in for specified device CNNNetReader Build and parse a network from given IR CNNNetwork Neural Network and binary information Blob, BlobMaps Container object representing a tensor (e.g. input/output) InputsDataMap Information about input of the network ExecutableNetwork Loaded Network InferRequest Interact with inputs/outputs and performs inference using namespace InferenceEngine;
111 © 2019 Intel Corporation Inference Engine API Usage (1) 1. Load Plugin – FPGA Plugin: libdliaPlugin.so – Others: libclDNNPlugin.so (GPU), libMKLDNNPlugin.so (CPU) – Plugin Dir: <OpenVINO install dir>/inference_engine/lib/<OS>/intel64 2. Load Network InferenceEngine::PluginDispatcher dispatcher(<pluginDir>); InferencePlugin plugin = dispatcher.getPluginByDevice(“HETERO:FPGA, CPU”); InferenceEngine::CNNNetReader netBuilder netBuilder.ReadNetwork(“<Model>.xml”); netBuilder.ReadWeights(“<Model>.bin”);
112 © 2019 Intel Corporation Inference Engine API Usage (2) 3. Prepare Input and Output Blobs – For Inputs – Allocate based on the size of the input, number of channels, batch size, etc. – Set input precision – Set layout – Fill in data (i.e. from rgb value of image) – For Outptus – Set output precision – Allocate based on output format
113 © 2019 Intel Corporation Inference Engine API Usage (3) 4. Load the model to the plugin 5. Perform inference 6. Process output blobs ExecutableNetwork exeNet = plugin.LoadNetwork(netBuilder.getNetwork(), {}); InferRequest infer_request = exeNet.CreateInferRequest(); infer_request.Infer(); const Blob::Ptr output_blob = infer_request.GetBlob(firstOutputName);
114 © 2019 Intel Corporation Using the Inference Engine API IR Parse (using CNNNetReader) Load Network Create Engine Instance Infer CNNNetReader netBuilder; netBuilder->ReadNetwork(“Model.xml”); netBuilder->ReadWeights(“Model.bin") InferencePlugin plugin PluginDispatcher({}).getPluginbyDevice(“HETERO:FPGA, CPU”); ExecutableNetwork exeNet = plugin.LoadNetwork(netBuilder->getNetwork(), {}); InferRequest infer_request = exeNet.CreateInferRequest(); //Create Input Blob, iterate through and fill in data infer_request.Infer(); //Process Output CNNNetwork instance CNNNetwork instance
115 © 2019 Intel Corporation Batch Execution  For better performance, using a larger batch size may help  Allocate input and output Blob according to batch size  Set the Batch size on the network netBuilder.getNetwork().setBatchSize(<size>);
116 © 2019 Intel Corporation Inference Engine Example Applications  Execute samples and demos with FPGA support – Shipped with the OpenVINO™ toolkit – classification_sample – object_detection_sample_ssd – many others…
117 © 2019 Intel Corporation Example Application using FPGAs $ ./object_detection_sample_ssd -m <path_to_model>/Model.xml -i <path_to_pictures>/picture.jpg -d HETERO:FPGA,CPU The “priorities” defines a greedy behavior:  Keeps all layers that can be executed on the device (FPGA)  Carefully respects topological and other limitations  Follows priorities when searching (for example, CPU)
118 © 2019 Intel Corporation DLA FPGA IP CNN primitives (Conv, FC, ReLU, MaxPool, Norm..) IE Software Architecture with FPGAs Board Support Package DLA Software Runtime Inference Engine OpenCL™ Runtime Driver to the FPGA board OpenCL API to access FPGA DLA API Unified API for Intel® platforms Inference Engine Plugin HETERO/FPGA Plugin Deep Learning Deployment Toolkit Intel® FPGA DLA Suite
119 © 2019 Intel Corporation Software Requirements  Intel® Distribution of OpenVINO™ toolkit (Linux for FPGA) – Includes Intel® FPGA Runtime (RTE) for OpenCL™  For Vision Accelerator Design with FPGA board – OpenCL™ Board Support Package for Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA – Intel® Quartus® Prime Pro Edition Programmer  For Programmable Acceleration Card – Intel® Acceleration Stack for Runtime – Includes Programmer, RTE for OpenCL, and OpenCL BSP DLA Runtime Engine OpenCL™ Runtime Software Stack DL Deployment Toolkit Acceleration Stack User DL Inference Application OpenCL™ BSP Driver OpenVINO
120 © 2019 Intel Corporation FPGA Software Components  Intel® FPGA OpenCL™ Runtime – Included as part of OpenVINO™ – Inference acceleration done through OpenCL calls  Board Support Package – Translates OpenCL calls to low-level device driver calls – Allows programming of FPGA with different bitstreams – Included as part of Acceleration Stack for PAC boards  Intel® Quartus® Prime Programmer – Possibly needed to flash/program the board with FPGA image to communicate with host DLA Runtime Engine OpenCL™ Runtime Software Stack DL Deployment Toolkit Acceleration Stack User DL Inference Application OpenCL™ BSP Driver OpenVINO
121 © 2019 Intel Corporation Prepare FPGA Environment for Inference  Once software stack is installed and environment set  Ensure PCIe* connection is operating – lspci | grep Altera  Ensure FPGA OpenCL is working correctly – aocl diagnose – Ensure DIAGNOSTIC_PASSED
122 © 2019 Intel Corporation Load FPGA Image and Execute IE Application  FPGAs needs to be preconfigured with primitives prior to application execution  Choose FPGA bitstream from the DLA suite – Based on topology needs and data type requirements – Option to create custom FPGA bitstream based on requirements  Execute User or Example Application aocl program acl0 /opt/intel/openvino/bitstreams/<board>/2019R1_PL1_FP11_MobileNet_SqueeeNet_VGG.aocx
123 © 2019 Intel Corporation FPGA Image Selection  Precompiled FPGA images included  Choose image based on – Primitives needed (architecture) – Data type support for accuracy/performance trade-off – K (filter) and C (channel depth) vectorization for performance – Data path width – On-chip stream buffer depth  May also generate customized FPGA image
124 © 2019 Intel Corporation Use OpenVINO™ Running SqueezeNet example manually or create your own application 1. Create a new test directory and cd into it, ensure environment script is ran 2. Copy original Caffe* files from the demo directory into the new directory 3. Copy the imagenet labels from the demo directory 4. Execute model optimizer to convert to intermediate representation 5. Execute classification sample cp ~/openvino_models/models/FP32/classification/squeeenet/1.1/caffe/squeeenet1.1.* . cp ~/openvino_models/ir/FP32/classification/squeeenet/1.1/caffe/squeeenet1.1.labels . mo –input_model squeezenet1.1.caffemodel classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png –d HETERO:FPGA,CPU classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png –d HETERO:FPGA,CPU –ni 100
125 © 2019 Intel Corporation Summary  The Intel® Deep Learning Acceleration Suite part of Intel® Distribution of OpenVINO™ tookit provides out-of-box deployment of deep learning inference on FPGAs  Deep Learning Deployment Toolkit can be used as a frontend for the DLA – Consists of the Model Optimizer and Inference Engine – Use Model Optimizer to optimize and convert framework model to unified internal representation – Use Inference Engine unified API to deploy network on various devices including the FPGA
126 © 2019 Intel Corporation References  Follow-on Training – Deploying FPGAs for Deep Learning Inference with Intel® Distribution of OpenVINO™ toolkit  Intel® AI – www.intel.com/ai  Intel® Distribution of OpenVINO™ toolkit – https://software.intel.com/en-us/openvino-toolkit  Intel® FPGAs for AI  Intel® FPGA Acceleration Stack for Intel® Xeon® CPU with FPGAs
127 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
Intel® Agilex® FPGAs
Introducing Intel® Agilex™ FPGAs Intel innovation for ultimate agility and flexibility AnyDeveloper Intel® Quartus® Prime Design Tool for Hardware Developers One API for Software Developers Any-to-anyintegration Memory, Analog, Logic Any Node, Supplier, IP Rapid Intel® eASIC™ Devices Optimization 10nm Process Memory-Coherent Intel® Xeon® Processor Acceleration Massive Bandwidth High-Performancecompute 129
The FPGA for the Data-Centric World 130 Upto 40% Higher performance1,3 Upto 40% Lower Power1,3 Process Data Store Data Move Data DDR5& HBM Intel®Optane™DC PersistentMemorysupport Transceiver datarates Intel®Xeon®Processorcoherent connectivity&PCIe Gen5 112G 1 Compared to Intel® Stratix® 10 FPGAs 2 With FP16 configuration 3 Based on current estimates, see slide 180 for details Upto 40TFLOPS DSPPerformance 2 Generation Intel® Hyperflex™ Architecture nd2 2,3
Any-to-Any Heterogenous 3D Integration 131 Chiplet-BasedArchitecture Library of chiplets: transceivers, custom I/O & custom compute tiles Intel®eASIC™TileCustomization Ultimate customization with integrated Intel® eASIC™ tile Embedded Multi-dieinterconnectbridge No-compromise die-to-die 3D packaging interconnect
Custom Logic Continuum 132 Flexible and Agile Optimization Across Product Lifecycle FPGA Fastest Time to Market Highest Flexibility Highest Performance Lowest Power & Cost* Optimized TTM with Performance / Cost eASIC ASIC Reusable IP 1Refers to Intel® eASIC™ Devices * Compared to other options shown 1
First Memory-Coherent Accelerator for Intel® Xeon® Scalable Processors 133 FPGA Memory Processor Memory Compute Express Link* (CXL) Accelerates Diverse Workloads • LowLatency • MemoryCoherence • IncreasedMemorySpace (Including data analytics, database acceleration and function-as-a-service)
134 112G Pcie Gen5 * High Bandwidth for Applications Including 400G Networks, Edge Analytics, Data Center Workloads Transceiver Leadership
Well Suited for Artificial Intelligence Applications 3,4 based on current estimates, see slide 180 for details 135 Complementary to: Flexibility for evolving workloads and multi- function integration AI+ usage model Only FPGA supporting hardened BFLOAT16 & FP16 Configurable DSP • FP32 • BFLOAT16 • FP16 • INT8 through INT2 One APIOpenVINO™ toolkit 40TFLOPS 3 Upto FP16Performance Upto 92TOPS INT8Performance 4
Intel® Agilex™ FPGA Tools for Developers 136  Single source, heterogenous programming environment  Support for common performance library APIs  FPGA support with Intel software development tools including Intel® VTune™ Amplifier & Intel® Advisor 5See slide 180 for details  Higher productivity:  30% improvement in compile times5  New productivity flows and usability features for faster design convergence  Higher efficiency: 15% improvement in memory utilization5 Hardware Developers Software Developers ONEAPI
Intel® Agilex™ FPGA Family 137 F-Series For wide range of applications Up to 58G transceivers PCIe* Gen4 DDR4 I-SeriesFor high-performance processor interface and bandwidth-intensive applications Up to 112G transceivers PCIe* Gen5 DDR4 Quad-Core ARM* Cortex*-A53 M-Series For compute-intensive applications Up to 112G transceivers PCIe* Gen5 DDR4, DDR5 and Intel® Optane™ DC Persistent Memory support Quad-Core ARM* Cortex*-A53 Quad-Core ARM* Cortex*-A53 SoC Option HBM Option Coherent attach to Intel® Xeon® Scalable Processor option (CXL) Coherent attach to Intel® Xeon® Scalable Processor option (CXL) Intel® Quartus® Prime Design Software Support April 2019; First Device Availability Q3 2019
Efficient Network Transformation First FPGA Providing Flexibility and Agility From 100Gbps to 1Tbps InfrastructureOffload OptimizedArchitecture vSwitch | vRouter | Security SmallFormFactor&LowPower Wide Range of Servers DatapathAcceleration VNFPerformanceOptimization Load Balancing | Data Integrity Network Translation SignificantImprovements Throughput | Jitter | Latency 28G-112G per channel PCIe Gen 4/5 138 1 1 Compared to Intel® Stratix® 10 FPGAs
Converged Workload Acceleration for the Datacenter First FPGA with Comprehensive Memory Support and Coherent Attach to Intel® Xeon® Scalable Processors Memory Coherent (CXL) infrastructureacceleration Network | Security | Remote Memory Access Applicationacceleration AI | Search | Video Transcode | Database 40 TFLOPs of DSP Performance1 StorageAcceleration Compression | Decompression | Encryption Memory Hierarchy ManagementDDR5 SSD 139 1 with FP16 configuration, based on current estimates, see slide 180 for details
Radio Baseband Agility & Flexibility for All Stages of 5G Implementation + FPGAFlexibility High Flexibility | Short Time-to-Market RapidIntel®eASIC™Deviceoptimization Power & Cost Optimization FullCustomASICOPtimization Best Power1 | Best Performance1 | Best Cost1 Application-SpecificTileOptions Data Converter | Vector Engine | Custom Compute Data Converter DFE Beamforming Fronthaul Only Intel® Provides All of These Customization Options CustomLogicContinuum FEC (Turbo/LDPC) uRLLC Fronthaul 140 1 Compared to the other customization options shown
Intel® Agilex™ FPGAs for the Data-Centric World Any-to-Any integration enables Intel® FPGAs with application-specific optimization and customization, delivering new levels of flexibility & agility Intel® Agilex™ FPGAs for transformative applications in edge computing, embedded, networking (5G/NFV) and data centers First FPGAs leveraging Intel’s unmatched innovation: 10nm process, 3D integration, Intel® Xeon® Scalable Processor memory coherency (CXL), 112G XCVRs, PCIe* Gen 5, Intel® eASIC™ devices, One API, Intel® Optane™ DC Persistent Memory support 141
142 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
143 © 2019 Intel Corporation oneAPISingle Programming Model to Deliver Cross-Architecture Performance
144 © 2019 Intel Corporation The future is a diverse mix of scalar, vector, matrix, and spatial architectures deployed in CPU, GPU, AI, FPGA and other accelerators DiverseWorkloadsrequire DIVERSEarchitectures Spatial FPGA Matrix AI Vector GPU Scalar CPU SVMS
145 © 2019 Intel Corporation Project oneAPI delivers a unified programming model to simplify development across diverse architectures Common developer experience across Scalar, Vector, Matrix and Spatial (SVMS) architecture Unified and simplified language and libraries for expressing parallelism Uncompromised native high-level language performance Support for CPU, GPU, AI and FPGA Based on industry standards and open specifications Optimized Applications Optimized Middleware / Frameworks oneAPI Language & Libraries Intel’sONEAPI CoreConcept FPGAAIGPUCPU Scalar Vector Matrix Spatial oneAPI Tools
146 © 2019 Intel Corporation OneAPIforcross-architectureperformance Some capabilities may differ per architecture. Optimized Applications Optimized Middleware & Frameworks Scalar Vector Matrix Spatial FPGAAIGPUCPU Direct Programming Data Parallel C++ API-Based Programming Threading Video Processing Math Analytics/ML DNN ML Comm DPC++ Library oneAPI Product Analysis & Debug Tools VTune™ Advisor Debugger Compatibility Tool
© 2019 Intel Corporation Thankyou
FPGAs and Machine Learning
149 © 2019 Intel Corporation software.intel.com/bigdl AnalyticsZoo Build E2E Analytics & AI Applications for Big Data at Scale Optimization Notice All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. *Other names and brands may be claimed as the property of others UnifiedBigDataAnalytics+AIOpenSourcePlatform Reference Use Cases  Anomaly detection, sentiment analysis, fraud detection, image generation, chatbot, sequence prediction, etc. Built-In Deep Learning Models  Image classification, object detection, text classification, recommendations, GANs, Sequence to Sequence, etc. Feature Engineering Feature transformations for  Image, text, 3D imaging, time series, speech, etc. High-Level Pipeline APIs  Distributed Tensorflow* & Keras* on Apache Spark/BigDL  Support for Autograd*, transfer learning, Spark DataFrame and ML Pipeline  Model serving API for model serving/inference pipelines Backends Apache Spark, TensorFlow*, BigDL, Python, etc. https://analytics-zoo.github.io/ Play the Video
150 © 2019 Intel Corporation software.intel.com/intel-distribution-for-python Easy, Out-of-the-box Access to High Performance Python*  Prebuilt, delivers faster, close-to- native code application performance for machine learning and scientific computing from device to cloud  Drop in replacement for your existing Python (no code changes required) Drive Performance with Multiple Optimization Techniques  Accelerated NumPy/SciPy/Scikit- Learn with Intel® Math Kernel Library  Data analytics with pyDAAL, enhanced thread scheduling with Threading Building Blocks, Jupyter* Notebook interface, Numba, Cython  Scale easily with optimized MPI4Py and Jupyter notebooks Faster Access to Latest Optimizations for Intel® Architecture  Distribution and individual optimized packages available through conda* and Anaconda Cloud*  Optimizations upstreamed back to main Python trunk For developers using the most popular and fastest growing programming language for AI All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Inteldistributionforpython* Advancing Python Performance Closer to Native Speeds Optimization Notice
151 © 2019 Intel Corporation Building blocks for all data analytics stages, including data preparation, data mining & machine learning Open Source, Apache* 2.0 License Common Python*, Java*, C++ APIs across all Intel hardware Optimizes data ingestion & algorithmic compute together for highest performance Supports offline, streaming & distributed usage models for a range of application needs Flexible interfaces to leading big data platforms including Spark* Split analytics workloads between edge devices & cloud to optimize overall application throughput Pre-processing Transformation Analysis Modeling Decision MakingValidation All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Intel®DataAnalyticsAccelerationLibrary(Intel®DAAL) High Performance Machine Learning & Data Analytics Library Optimization Notice
152 © 2019 Intel Corporation Distribution Details  Open Source  Apache* 2.0 License  Common DNN APIs across all Intel hardware.  Rapid release cycles, iterated with the DL community, to best support industry framework integration.  Highly vectorized & threaded for maximal performance, based on the popular Intel® Math Kernel Library. For developers of deep learning frameworks featuring optimized performance on Intel hardware github.com/01org/mkl-dnn Direct 2D Convolution Rectified linear unit neuron activation (ReLU) Maximum pooling Inner product Local response normalization (LRN) Examples: Intel®MathKerneLfordeepneuralNetworks(intel®mkl-DNN) Accelerate Performance of Deep Learning Models All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Optimization Notice
153 © 2019 Intel Corporation Work in progress github.com/NervanaSystems/ngraph *Other names and brands may be claimed as the property of others. All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. nGraph More… Single API for multiple frameworks & backends Lotus NVIDIA* GPU CPU NNP VPU Processor Graphics GPU FPGA Deep learning training & inference More… Intel®nGRAPH™Compiler Open-source C++ library, compiler & runtime for deep learning enabling flexibility to run models across a variety of frameworks & hardware Optimization Notice
154 © 2019 Intel Corporation Nauta Open-Source Distributed Deep Learning Platform for Kubernetes* All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. * Other names and brands may be claimed as the property of others. Batch inference Inference deployment Export to edge devices Multi-user collaboration Interactive sessions Template functionality Fast training Batch training Experiment tracking Multi-node distribution Analytics & visualization using TensorBoard* build train evaluate github.com/intelAI/NautaOptimization Notice
FPGAs and Machine Learning
156 © 2019 Intel Corporation UnsupervisedlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning Machinelearning(Clustering) An ‘Unsupervised Learning’ Example K-Means Revenue Purchasing Power Revenue Purchasing Power Choose the right AI approach for your challenge
157 © 2019 Intel Corporation SupervisedlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning TrainingInference Model Weights Forward “Bicycle”? ?????? Lots of tagged data Forward Backward “Strawberry” “Bicycle”? Error Human Bicycle Strawberry Deeplearning(imagerecognition) A ‘Supervised Learning’ Example Choose the right AI approach for your challenge
158 © 2019 Intel Corporation ReinforcementlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning Deeplearning(Reinforcementlearning) A ‘Deep Reinforcement Learning’ Example Feedback Action EnvironmentAgent Choose the right AI approach for your challenge
159 Open-source development environments Specific neural network implementations Library Optimized primitive functions for AI Intel® Distribution for Python Pandas NumPy Scikit-Learn Spark MlLib Mahout MKL-DNN DAAL DeeplearningGlossary Learn more about “what is AI” at software.intel.com/ai/course Framework Topology container Pre-configured AI environments ready to depoy
160 © 2019 Intel Corporation Machine Learning How do you engineer the best features? 𝑁 × 𝑁 Arjun NEURAL NETWORK 𝒇 𝟏, 𝒇 𝟐, … , 𝒇 𝑲 Roundness of face Dist between eyes Nose width Eye socket depth Cheek bone structure Jaw line length …etc. CLASSIFIER ALGORITHM SVM Random Forest Naïve Bayes Decision Trees Logistic Regression Ensemble methods 𝑁 × 𝑁 Arjun Deep Learning How do you guide the model to find the best features? Machinevs.Deeplearning
161 © 2019 Intel Corporation Whichapproachisright? Challenge Bestapproach Approach Answer How many widgets should we manufacture? Analyze historical supply/demand Analytics/ Business Intelligence 10,000 What will our yield be? Algorithm that correlates many variables to yield Statistical/ Machine Learning At current conditions, yield will be at 90% with 10% loss expected Which widgets have visual defects? Algorithm that learns to identify defects in images Deep Learning Widget 1003, Widget 1094 . . . A large manufacturer uses data to improve their operations, with each challenge using a different approach to deliver maximum business value at the lowest possible cost Learn Morein theNext slides
162 © 2019 Intel Corporation Aicl serlook Machinelearning DeepLearning reasoning Algorithms designed to deliver better insight with more data Neural networks used to infer meaning from large dense datasets Hybrid of analytics & AI techniques designed to find meaning in diverse datasets Regression (Linear/Logistic) Classification (Support Vector Machines/SVM, Naïve Bayes) Clustering (Hierarchical, Bayesian, K- Means, DBSCAN) Decision Trees (RandomForest) Extrapolation (Hidden Markov Models/HMM) More… Image Recognition (Convolutional Neural Networks/CNN, Single-Shot Detector/SSD) Speech Recognition (Recurrent Neural Network/RNN) Natural Language Processing (Long-Short Term Memory/LSTM) Data Generation (Generative Adversarial Networks/GAN) Recommender System (Multi-Layer Perceptron/MLP) Time-Series Analysis (LSTM, RNN) Reinforcement Learning (CNN, RNN) More… Associative Memory (Intel® Saffron AI memory base)  See also: machine & deep learning techniques More…
163 © 2019 Intel Corporation Lots of labeled data! Training Inference Forward Backward Model weights Forward “Bicycle”? “Strawberry” “Bicycle”? Error Human Bicycle Strawberry ?????? DeeplearningBasics Data set size Accuracy Didyouknow? Training with a large data set AND deep (many layered) neural network often leads to the highest accuracy inference
164 © 2019 Intel Corporation Framework Open-source software environments that facilitate deep learning model development & deployment through built-in components and the ability to customize code Topology Wide variety of algorithms modeled loosely after the human brain that use neural networks to recognize complex patterns in data that are otherwise difficult to reverse engineer Library Hardware-optimized mathematical and other primitive functions that are commonly used in machine & deep learning algorithms, topologies & frameworks Intel® Distribution for Python Pandas NumPy Scikit-Learn Spark MlLib Mahout MKL-DNN DAAL Deeplearningglossary Translating common deep learning terminology
165 © 2019 Intel Corporation Image Recognition Object Detection Image Segmentation Language Translation Text to Speech Recommendation System Deeplearningusages&Keytopologies Resnet-50 Inception V3 MobileNet SqueezeNet R-FCN Faster-RCNN Yolo V2 SSD-VGG16, SSD-MobileNet Mask R-CNN GNMT Wavenet Wide & Deep, NCF There are many deep learning usages and topologies for each
166 © 2019 Intel Corporation 0% 8% 15% 23% 30% Human 2010 Present Imagerecognition 0% 8% 15% 23% 30% 2000 Present SpeechrecognitionError Error Human 97% person 99% “play song” Source: ILSVRC ImageNet winning entry classification error rate each year 2010-2016 (Left), https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/ (Right) Deeplearningbreakthroughs Machines able to meet or exceed human image & speech recognition Tumor Detection Document Sorting Oil&Gas search Voice Assistant Defect detection Genome sequencing e.g.
167 © 2019 Intel Corporation All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Deep Learning Training Inference AI HARDWARE Mainstream intensive Multi-purpose to purpose-built AI compute from cloud to device Most other Onesizedoesnotfitall
168 © 2019 Intel Corporation Large scale data centers such as public cloud or comms service providers, gov’t and academia, large enterprise IT User-touch end point devices with lower power requirements such as laptops, tablets, smart home devices, drones Small scale data centers, small business IT infrastructure, to few on-premise server racks and workstations All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Varies to <1ms <5ms <10-40ms ~100ms DatacenterEdgeEndpoint HARDWARE Multi-purpose to purpose-built AI compute from device to cloud Aiisexpanding
169 © 2019 Intel Corporation 1GNA=Gaussian Neural Accelerator All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Images are examples of intended applications but not an exhaustive list. IOT SENSORS (Security, home, retail, industrial…) Display, Video, AR/VR, Gestures, Speech DESKTOP & MOBILITY Vision & Inference Speech SELF-DRIVING VEHICLES Autonomous Driving SERVERS, APPLIANCES & GATEWAYS Latency- Bound Inference Basic Inference, Media & Vision Most Use Cases SERVERS & APPLIANCES DatacenterEdgeEndpoint Flexible & Memory Bandwidth-Bound Use Cases Onesizedoesnotfitall HARDWARE Multi-purpose to purpose-built AI compute from device to cloud Varies to <1ms <5ms <10-40ms ~100ms Dedicated Media & Vision Inference Most Use Cases Most Intensive Use Cases NNP-T M.2 CardSOC Special Purpose Special Purpose
170 © 2019 Intel Corporation hardware Multi-purpose to purpose-built AI compute from cloud to device solutions Partner ecosystem to facilitate AI in finance, health, retail, industrial & more Intel analytics ecosystem to get your data ready Data Driving AI forward through R&D, investments and policy Future tools Software to accelerate development and deployment of real solutions Bring Your AI Vision to Life Using Our Complete Portfolio
FPGAs and Machine Learning
172 Configurations for Continued Innovation Driving DL Gains on Xeon® (March 2019) 1x inference throughput improvement on Intel® Xeon® Platinum 8180 processor (July 2017) baseline : Tested by Intel as of July 11th 2017: Platform: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),and https://github.com/soumith/convnet-benchmarks/tree/master/caffe/imagenet_winners (ConvNet benchmarks; files were updated to use newer Caffe prototxt format but are functionally equivalent). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 5.7x inference throughput improvement on Intel® Xeon® Platinum 8180 processor (December 2018) with continued optimizations : Tested by Intel as of November 11th 2018 :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux-7.3.1611-Core, kernel: 3.10.0-862.3.3.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimization for caffe version: 551a53d63a6183c233abaa1a19458a25b672ad41 Topology::ResNet_50_v1 BIOS:SE5C620.86B.00.01.0014.070920180847 MKLDNN: 4e333787e0d66a1dca1218e99a891d493dbc8ef1 instances: 2 instances socket:2 (Results on Intel® Xeon® Scalable Processor were measured running multiple instances of the framework. Methodology described here: https://software.intel.com/en-us/articles/boosting-deep-learning-training-inference-performance-on-xeon-and-xeon-phi) Synthetic data. Datatype: INT8 Batchsize=64 vs Tested by Intel as of July 11th 2017:2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency- set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 14x inference throughput improvement on Intel® Xeon® Platinum 8280 processor with Intel® DL Boost: Tested by Intel as of 2/20/2019. 2 socket Intel® Xeon® Platinum 8280 Processor, 28 cores HT On Turbo ON Total Memory 384 GB (12 slots/ 32GB/ 2933 MHz), BIOS: SE5C620.86B.0D.01.0271.120720180605 (ucode: 0x200004d), Ubuntu 18.04.1 LTS, kernel 4.15.0-45-generic, SSD 1x sda INTEL SSDSC2BA80 SSD 745.2GB, nvme1n1 INTEL SSDPE2KX040T7 SSD 3.7TB, Deep Learning Framework: Intel® Optimization for Caffe version: 1.1.3 (commit hash: 7010334f159da247db3fe3a9d96a3116ca06b09a) , ICC version 18.0.1, MKL DNN version: v0.17 (commit hash: 830a10059a018cd2634d94195140cf2d8790a75a, model: https://github.com/intel/caffe/blob/master/models/intel_optimized_models/int8/resnet50_int8_full_conv.prototxt, BS=64, syntheticData, 4 instance/2 socket, Datatype: INT8 vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time -- forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 30x inference throughput improvement on Intel® Xeon® Platinum 9282 processor with Intel® DL Boost : Tested by Intel as of 2/26/2019. Platform: Dragon rock 2 socket Intel® Xeon® Platinum 9282(56 cores per socket), HT ON, turbo ON, Total Memory 768 GB (24 slots/ 32 GB/ 2933 MHz), BIOS:SE5C620.86B.0D.01.0241.112020180249, Centos 7 Kernel 3.10.0-957.5.1.el7.x86_64, Deep Learning Framework: Intel® Optimization for Caffe version: https://github.com/intel/caffe d554cbf1, ICC 2019.2.187, MKL DNN version: v0.17 (commit hash: 830a10059a018cd2634d94195140cf2d8790a75a), model: https://github.com/intel/caffe/blob/master/models/intel_optimized_models/int8/resnet50_int8_full_conv.prototxt, BS=64, No datalayer syntheticData:3x224x224, 56 instance/2 socket, Datatype: INT8 vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0- 514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“.
173 © 2019 Intel Corporation This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Intel inside, the Intel inside logo, Xeon, the Xeon logo, Xeon Phi, the Xeon Phi logo, Core, the Core logo, Atom, the Atom logo, Movidius, the Movidius logo, Stratix, the Stratix logo, Arria, the Arria logo, Myriad, Nervana and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit © 2019 Intel Corporation. Configurationdetails
174 © 2019 Intel Corporation Configuration: AI Performance – Software + Hardware INFERENCE using FP32 Batch Size Caffe GoogleNet v1 128 AlexNet 256. The benchmark results may need to be revised as additional testing is conducted. The results depend on the specific platform configurations and workloads utilized in the testing, and may not be applicable to any particular user's components, computer system or workloads. The results are not necessarily representative of other benchmarks and other benchmark results may show greater or lesser impact from mitigations. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance Source: Intel measured as of June 2017 Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Configurations for Inference throughput Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611-Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology:GoogleNet v1 BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1449.9 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320191901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configuration for training throughput: Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611-Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc765b1162ab9940d Topology:alexnet BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1257 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320191901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configurationdetails(Cont’d)
175 © 2019 Intel Corporation Configurationdetails(Cont’d) INFERENCE using FP32 Batch Size Caffe GoogleNet v1 128 AlexNet 256. The benchmark results may need to be revised as additional testing is conducted. The results depend on the specific platform configurations and workloads utilized in the testing, and may not be applicable to any particular user's components, computer system or workloads. The results are not necessarily representative of other benchmarks and other benchmark results may show greater or lesser impact from mitigations. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance Source: Intel measured as of June 2017 Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Configurations for Inference throughput Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611- Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology:GoogleNet v1 BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1449.9 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320181901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configuration for training throughput: Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611- Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc765b1162ab9940d Topology:alexnet BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1257 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320181901, CentOS Linux- 7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594
176 © 2019 Intel Corporation Configurationdetails(Cont’d) 1.4x training throughput improvement in August 2018: Tested by Intel as of measured August 2nd 2018. Processor: 2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux- 7.3.1611-Core kernel 3.10.0-693.11.6.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimizations for caffe version:a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology::resnet_50 BIOS:SE5C620.86B.00.01.0013.030920180427 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 123 imgs/sec vs Intel tested July 11th 2017 Platform: Platform: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (GoogLeNet, AlexNet, and ResNet-50), https://github.com/intel/caffe/tree/master/models/default_vgg_19 (VGG-19), and https://github.com/soumith/convnet-benchmarks/tree/master/caffe/imagenet_winners (ConvNet benchmarks; files were updated to use newer Caffe prototxt format but are functionally equivalent). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 5.4x inference throughput improvement in August 2018: Tested by Intel as of measured July 26th 2018 :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux-7.3.1611-Core, kernel: 3.10.0-862.3.3.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimized caffe version:a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology::resnet_50_v1 BIOS:SE5C620.86B.00.01.0013.030920180427 MKLDNN: version:464c268e544bae26f9b85a2acb9122c766a4c396 instances: 2 instances socket:2 (Results on Intel® Xeon® Scalable Processor were measured running multiple instances of the framework. Methodology described here: https://software.intel.com/en-us/articles/boosting-deep-learning- training-inference-performance-on-xeon-and-xeon-phi) NoDataLayer. Datatype: INT8 Batchsize=64 Measured: 1233.39 imgs/sec vs Tested by Intel as of July 11th 2017:2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 11X inference thoughput improvement with CascadeLake: Future Intel Xeon Scalable processor (codename Cascade Lake) results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“.
177 © 2019 Intel Corporation Configurationdetails(cont’d) Intel® Arria® 10 – 1150 FPGA energy efficiency on Caffe/AlexNet up to 25 img/s/w with FP16 at 297MHz Vanilla AlexNet Classification Implementation as specified by http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf, Training Parameters taken from Caffe open-source Framework are 224x224x3 Input, 1000x1 Output, FP16 with Shared Block-Exponents, All compute layers (incl. Fully Connected) done on the FPGA except for Softmax, Arria 10-1150 FPGA, -1 Speed Grade on Altera PCIe DevKit with x72 DDR4 @ 1333 MHz, Power measured through on-board power monitor (FPGA POWER ONLY), ACDS 16.1 Internal Builds + OpenCL SDK 16.1 Internal Build, Compute machine is an HP Z620 Workstation, Xeon E5-1660 at 3.3 GHz with 32GB RAM. The Xeon is not used for compute.
DETAILS ON INTEL® AGILEX™ FPGA PERFORMANCE, POWER AND SOFTWARE SUPPORT NUMBERS (1) Up to 40% Higher Performance Compared to Intel Stratix 10 FPGAs Derived from benchmarking an example design suite comparing maximum clock speed (Fmax) achieved in Intel Stratix 10 devices with the Fmax achieved in Intel Agilex devices, using Intel Quartus Prime Software. On average, designs running in the fastest speed grade of Intel Agilex FPGAs achieve a 40% improvement in Fmax compared to the same designs running in the most popular speed grade of Stratix 10 devices (-2 speed grade), tested February 2019. (2) Up to 40% Lower Total Power Compared to Intel Stratix 10 FPGAs Derived from benchmarking an example design suite comparing total power estimates of each design running in Intel Stratix 10 FPGAs compared to the total power consumed by the same design running in Intel Agilex FPGAs. Power estimates of Intel Stratix 10 FPGA designs are obtained from Intel Stratix 10 Early Power Estimator; power estimates for Intel Agilex FPGA designs are obtained using internal Intel analysis and architecture simulation and modeling, tested February 2019. (3) Up to 40 TFLOPs of DSP Performance (FP16 Configuration) Each Intel Agilex DSP block can perform two FP16 floating-point operations (FLOPs) per clock cycle. Total FLOPs for FP16 configuration is derived by multiplying 2x the maximum number of DSP blocks to be offered in a single Intel Agilex FPGA by the maximum clock frequency that will be specified for that block. (4) Up to 92 TOPs of DSP Performance (INT8 Configuration) Each Intel Agilex DSP block can perform eight INT8 operations per clock cycle. Total TOPs for INT8 configuration is derived by multiplying 8x the maximum number of DSP blocks to be offered in a single Intel Agilex FPGA by the maximum clock frequency that will be specified for that block. (5) 30% Improvement in Compile Times / 15% Improvement in Memory Utilization Comparison is made between Intel Quartus Prime Software 18.1 and Intel Quartus Prime 19.1. Derived from benchmarking an example design suite comparing compile times and memory utilization for designs in Intel Quartus Prime Software 18.1 with compile times and memory utilization for same designs in Intel Quartus Prime Software 19.1, tested February 2019. 178 Results have been estimated or simulated using internal Intel analysis, architecture simulation, and modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.
LEGAL NOTICES & DISCLAIMERS This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Atom, Xeon and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2019 Intel Corporation. 179
LEGAL NOTICES & DISCLAIMERS FTC Optimization Notice If you make any claim about the performance benefits that can be achieved by using Intel software developer tools (compilers or libraries), you must use the entire text on the same viewing plane (slide) as the claim. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804 180

More Related Content

PPTX
AI Hardware Landscape 2021
Grigory Sapunov
 
PPTX
GPU Architecture NVIDIA (GTX GeForce 480)
Fatima Qayyum
 
PDF
Hardware Acceleration for Machine Learning
CastLabKAIST
 
PDF
FPGA Hardware Accelerator for Machine Learning
Dr. Swaminathan Kathirvel
 
PDF
Deep learning: Hardware Landscape
Grigory Sapunov
 
PDF
AI Chip Trends and Forecast
CastLabKAIST
 
PDF
Architecture of TPU, GPU and CPU
GlobalLogic Ukraine
 
PDF
Basic of AI Accelerator Design using Verilog HDL
Joohan KIM
 
AI Hardware Landscape 2021
Grigory Sapunov
 
GPU Architecture NVIDIA (GTX GeForce 480)
Fatima Qayyum
 
Hardware Acceleration for Machine Learning
CastLabKAIST
 
FPGA Hardware Accelerator for Machine Learning
Dr. Swaminathan Kathirvel
 
Deep learning: Hardware Landscape
Grigory Sapunov
 
AI Chip Trends and Forecast
CastLabKAIST
 
Architecture of TPU, GPU and CPU
GlobalLogic Ukraine
 
Basic of AI Accelerator Design using Verilog HDL
Joohan KIM
 

What's hot (20)

PPTX
Software Defined Network - SDN
Venkata Naga Ravi
 
PPTX
Federated learning in brief
Shashi Perera
 
PDF
AI On the Edge: Model Compression
Apache MXNet
 
PDF
5G Multi-Access Edge Compute
Michelle Holley
 
PPTX
Snapdragon Processor
Krishna Gehlot
 
PPTX
IoT
Ananth Kumar
 
PDF
On-Device AI
LGCNSairesearch
 
PDF
Inside Microsoft's FPGA-Based Configurable Cloud
inside-BigData.com
 
PDF
Federated learning
Mindos Cheng
 
PPTX
MemVerge: The Software Stack for CXL Environments
Memory Fabric Forum
 
PDF
Introduction to Software Defined Networking (SDN)
Bangladesh Network Operators Group
 
PDF
"Accelerating Deep Learning Using Altera FPGAs," a Presentation from Intel
Edge AI and Vision Alliance
 
PPTX
Broadcom PCIe & CXL Switches OCP Final.pptx
Memory Fabric Forum
 
PDF
Ericsson 5 g platform
Ericsson
 
DOCX
5G NETWORK AND INTERNET OF THINGS doc
Bhadra Gowdra
 
PPTX
Internet of things using Raspberry Pi
Yash Gajera
 
PDF
Neuromorphic computing for neural networks
Claudio Gallicchio
 
PDF
Latest Seminar Topics for Engineering,MCA,MSc Students
Arun Kumar
 
PPT
Multi core processors
Ummiya Mohammedi
 
PDF
Enabling the metaverse with 5G- web.pdf
Qualcomm Research
 
Software Defined Network - SDN
Venkata Naga Ravi
 
Federated learning in brief
Shashi Perera
 
AI On the Edge: Model Compression
Apache MXNet
 
5G Multi-Access Edge Compute
Michelle Holley
 
Snapdragon Processor
Krishna Gehlot
 
IoT
Ananth Kumar
 
On-Device AI
LGCNSairesearch
 
Inside Microsoft's FPGA-Based Configurable Cloud
inside-BigData.com
 
Federated learning
Mindos Cheng
 
MemVerge: The Software Stack for CXL Environments
Memory Fabric Forum
 
Introduction to Software Defined Networking (SDN)
Bangladesh Network Operators Group
 
"Accelerating Deep Learning Using Altera FPGAs," a Presentation from Intel
Edge AI and Vision Alliance
 
Broadcom PCIe & CXL Switches OCP Final.pptx
Memory Fabric Forum
 
Ericsson 5 g platform
Ericsson
 
5G NETWORK AND INTERNET OF THINGS doc
Bhadra Gowdra
 
Internet of things using Raspberry Pi
Yash Gajera
 
Neuromorphic computing for neural networks
Claudio Gallicchio
 
Latest Seminar Topics for Engineering,MCA,MSc Students
Arun Kumar
 
Multi core processors
Ummiya Mohammedi
 
Enabling the metaverse with 5G- web.pdf
Qualcomm Research
 
Ad

Similar to FPGAs and Machine Learning (20)

PDF
AIDC Summit LA- Hands-on Training
Intel® Software
 
PDF
Microsoft Build 2019- Intel AI Workshop
Intel® Software
 
PDF
AIDC India - AI on IA
Intel® Software
 
PDF
TDC2018SP | Trilha IA - Inteligencia Artificial na Arquitetura Intel
tdc-globalcode
 
PDF
Accelerate Machine Learning Software on Intel Architecture
Intel® Software
 
PDF
Accelerating AI Adoption with Partners
Sri Ambati
 
PDF
Intel Powered AI Applications for Telco
Michelle Holley
 
PDF
Driving Industrial InnovationOn the Path to Exascale
Intel IT Center
 
PDF
“Acceleration of Deep Learning Using OpenVINO: 3D Seismic Case Study,” a Pres...
Edge AI and Vision Alliance
 
PDF
TDC2017 | São Paulo - Trilha Machine Learning How we figured out we had a SRE...
tdc-globalcode
 
PDF
Tendências da junção entre Big Data Analytics, Machine Learning e Supercomput...
Igor José F. Freitas
 
PPTX
Gary Brown (Movidius, Intel): Deep Learning in AR: the 3 Year Horizon
AugmentedWorldExpo
 
PDF
Como criar um mundo autônomo e conectado - Jomar Silva
iMasters
 
PDF
QATCodec: past, present and future
boxu42
 
PDF
AI & Computer Vision (OpenVINO) - CPBR12
Jomar Silva
 
PDF
AI Crash Course- Supercomputing
Intel IT Center
 
PDF
TDC2019 Intel Software Day - Inferencia de IA em edge devices
tdc-globalcode
 
PDF
High Performance Computing: The Essential tool for a Knowledge Economy
Intel IT Center
 
PPTX
Meg Mude, Intel - Data Engineering Lifecycle Optimized on Intel - H2O World S...
Sri Ambati
 
PDF
“Intel Video AI Box—Converging AI, Media and Computing in a Compact and Open ...
Edge AI and Vision Alliance
 
AIDC Summit LA- Hands-on Training
Intel® Software
 
Microsoft Build 2019- Intel AI Workshop
Intel® Software
 
AIDC India - AI on IA
Intel® Software
 
TDC2018SP | Trilha IA - Inteligencia Artificial na Arquitetura Intel
tdc-globalcode
 
Accelerate Machine Learning Software on Intel Architecture
Intel® Software
 
Accelerating AI Adoption with Partners
Sri Ambati
 
Intel Powered AI Applications for Telco
Michelle Holley
 
Driving Industrial InnovationOn the Path to Exascale
Intel IT Center
 
“Acceleration of Deep Learning Using OpenVINO: 3D Seismic Case Study,” a Pres...
Edge AI and Vision Alliance
 
TDC2017 | São Paulo - Trilha Machine Learning How we figured out we had a SRE...
tdc-globalcode
 
Tendências da junção entre Big Data Analytics, Machine Learning e Supercomput...
Igor José F. Freitas
 
Gary Brown (Movidius, Intel): Deep Learning in AR: the 3 Year Horizon
AugmentedWorldExpo
 
Como criar um mundo autônomo e conectado - Jomar Silva
iMasters
 
QATCodec: past, present and future
boxu42
 
AI & Computer Vision (OpenVINO) - CPBR12
Jomar Silva
 
AI Crash Course- Supercomputing
Intel IT Center
 
TDC2019 Intel Software Day - Inferencia de IA em edge devices
tdc-globalcode
 
High Performance Computing: The Essential tool for a Knowledge Economy
Intel IT Center
 
Meg Mude, Intel - Data Engineering Lifecycle Optimized on Intel - H2O World S...
Sri Ambati
 
“Intel Video AI Box—Converging AI, Media and Computing in a Compact and Open ...
Edge AI and Vision Alliance
 
Ad

More from inside-BigData.com (20)

PDF
Major Market Shifts in IT
inside-BigData.com
 
PDF
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
PPTX
Transforming Private 5G Networks
inside-BigData.com
 
PDF
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
inside-BigData.com
 
PDF
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
inside-BigData.com
 
PDF
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
inside-BigData.com
 
PDF
HPC Impact: EDA Telemetry Neural Networks
inside-BigData.com
 
PDF
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
inside-BigData.com
 
PDF
Machine Learning for Weather Forecasts
inside-BigData.com
 
PPTX
HPC AI Advisory Council Update
inside-BigData.com
 
PDF
Fugaku Supercomputer joins fight against COVID-19
inside-BigData.com
 
PDF
Energy Efficient Computing using Dynamic Tuning
inside-BigData.com
 
PDF
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
inside-BigData.com
 
PDF
State of ARM-based HPC
inside-BigData.com
 
PDF
Versal Premium ACAP for Network and Cloud Acceleration
inside-BigData.com
 
PDF
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
inside-BigData.com
 
PDF
Scaling TCO in a Post Moore's Era
inside-BigData.com
 
PDF
CUDA-Python and RAPIDS for blazing fast scientific computing
inside-BigData.com
 
PDF
Introducing HPC with a Raspberry Pi Cluster
inside-BigData.com
 
PDF
Overview of HPC Interconnects
inside-BigData.com
 
Major Market Shifts in IT
inside-BigData.com
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
Transforming Private 5G Networks
inside-BigData.com
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
inside-BigData.com
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
inside-BigData.com
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
inside-BigData.com
 
HPC Impact: EDA Telemetry Neural Networks
inside-BigData.com
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
inside-BigData.com
 
Machine Learning for Weather Forecasts
inside-BigData.com
 
HPC AI Advisory Council Update
inside-BigData.com
 
Fugaku Supercomputer joins fight against COVID-19
inside-BigData.com
 
Energy Efficient Computing using Dynamic Tuning
inside-BigData.com
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
inside-BigData.com
 
State of ARM-based HPC
inside-BigData.com
 
Versal Premium ACAP for Network and Cloud Acceleration
inside-BigData.com
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
inside-BigData.com
 
Scaling TCO in a Post Moore's Era
inside-BigData.com
 
CUDA-Python and RAPIDS for blazing fast scientific computing
inside-BigData.com
 
Introducing HPC with a Raspberry Pi Cluster
inside-BigData.com
 
Overview of HPC Interconnects
inside-BigData.com
 

Recently uploaded (20)

PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Teaching material agriculture food technology
LiaRayya
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Advanced IT Governance
University of Hertfordshire
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 

FPGAs and Machine Learning

  • 1. Intel Network & Custom Logic Group Greg Nash, System Architect – Government, Aerospace & Military Jim Moawad, Technical Solution Specialist – FPGA Acceleration July 29, 2019 - Argonne Training Program on Extreme Scale Computing Public
  • 2. 2 © 2019 Intel Corporation Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No product or component can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. For more complete information about performance and benchmark results, visit http://www.intel.com/benchmarks . Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/benchmarks . Intel® Advanced Vector Extensions (Intel® AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at http://www.intel.com/go/turbo. Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Intel Atom, Intel Core, Intel Nervana, Intel Optane, Intel Xeon, Iris, Movidius, OpenVINO, Pentium, Celeron, Stratix, Arria and the Stratix logo and are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as property of others. notices&disclaimers
  • 3. 3 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 4. 4 © 2019 Intel Corporation Wide variety of market segments incorporating AI and Deep Learning which is increasing compute demands across the board. MachineLearningAppliestoNearlyEveryMarket Achieve higher yields & increase efficiency Maximize production and uptime Transform the learning experience Enhance safety, research, and more Turn data into valuable intelligence Revolutionize patient outcomes Empower truly intelligent Industry 4.0 Create thrilling experiences Transform stores and inventory Enable homes that see, hear, and respond Drive network and operational efficiency Agriculture Energy Education Government Finance Health Industrial Media Retail SmartHome Telecom Transport Automated driving
  • 5. 5 © 2019 Intel Corporation No one size fits all approach to AI AI&MachineLearningisaVastFieldofStudy Deep learning Machine learning AI Supervised Learning Reinforcement Learning unSupervised Learning Regression Classification Clustering Decision Trees Data Generation Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning
  • 6. 6 © 2019 Intel Corporation AIIsthedrivingforce Foresight Predictive Analytics Forecast Prescriptive Analytics Act/adapt Cognitive Analytics Hindsight Descriptive Analytics insight Diagnostic Analytics DataIsDrivingtheDesireforAI AnalyticsCurve 25GBper month Internet User 1 Datadeluge(2019) Insights Business Operational Security 50GBper day Smart Car 2 3TB per day Smart Hospital 2 40TBper day Airplane Data 2 1pBper day Smart Factory 2 50PBper day City Safety 2 1. Source: http://www.cisco.com/c/en/us/solutions/service-provider/vni-network-traffic-forecast/infographic.html 2. Source: https://www.cisco.com/c/dam/m/en_us/service-provider/ciscoknowledgenetwork/files/547_11_10-15-DocumentsCisco_GCI_Deck_2014-2019_for_CKN__10NOV2015_.pdf
  • 7. 7 © 2019 Intel Corporation The future is a diverse mix of scalar, vector, matrix, and spatial architectures deployed in CPU, GPU, AI, FPGA and other accelerators DiverseWorkloadsrequire DIVERSEarchitectures Spatial FPGA Matrix AI Vector GPU Scalar CPU SVMS
  • 8. 8 © 2019 Intel Corporation Training Highly efficient multi-model inferencing for cloud, data center and appliances at scale Inference Delivering Real time workflows with AI for optimized system performance GPU discrete & Integrated, optimized for highly parallel workload acceleration Fastest time-to-train at scale with high bandwidth AI server connections for the most persistent, intense usage Foundation for all workloads including AI Intel® DL Boost for inference & training acceleration Energy efficient Edge AI for Video/Vision CPU Accelerators FPGA,VPU,GPU Multi models – high performance across broad set of existing and emerging DL models for applications like Natural Language Processing, Speech & Text, Recommendation Engines, and Video & Image search & filtering A+ - Combined workflows where use cases span AI plus additional workloads (Eg - Immersive Media - Media + Graphics in GPU or L3 forwarding in FPGA) AIComputeGuide NNP-I NNP-T
  • 9. 9 © 2019 Intel Corporation Applications in HPC, Big Data, HPDA, AI & more that have access to a common compute and data pool ingest Store Process AnalyzeSource(s)? Structured? Volume? DURABILITY? Streaming? LOCALITY? GOVERNANCE? Other? Tool for live streaming data ingestion from Internet of Things (IOT) sensors in endpoint devices File, block or object-based storage solution given cost, access, volume and perf requirements Integration, cleaning, normalization and more transformations on batch and/or streaming data Visit: intel.com/content/www/us/en/analytics/tame-the-data-deluge.html *Other names and brands may be claimed as the property of others Tameyourdata with a robust data layer
  • 10. 10 © 2019 Intel Corporation Chooseanyapproach Reinforce- ment Learning Un- supervised Learning Statistical Methods Machine Learning Clustering kmeans Neural Nets Hidden Markov Models GMM Hier- archical Supervised Learning Regres- sion Class- ification Linear SVR GPR Decision Trees Ensemble Methods Logistic Neural Nets SVM Naïve Bayes Nearest Neighbor Discri- minant Analysis from machine to deep learning Visit: software.intel.com/ai/courses
  • 11. 1111 © 2019 Intel Corporation 1 An open source version is available at: 01.org/openvinotoolkit *Other names and brands may be claimed as the property of others. Developer personas show above represent the primary user base for each row, but are not mutually-exclusive All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Speedupdevelopment Optimization Notice TOOLKITS App developers libraries Data scientists Kernels Library developers Open source platform for building E2E Analytics & AI applications on Apache Spark* with distributed TensorFlow*, Keras*, BigDL Deep learning inference deployment on CPU/GPU/FPGA/VPU for Caffe*, TensorFlow*, MXNet*, ONNX*, Kaldi* Open source, scalable, and extensible distributed deep learning platform built on Kubernetes (BETA) Intel-optimized Frameworks And more framework optimizations underway including PaddlePaddle*, Chainer*, CNTK* & others Python R Distributed • Scikit- learn • Pandas • NumPy • Cart • Random Forest • e1071 • MlLib (on Spark) • Mahout Intel® Distribution for Python* Intel® Data Analytics Acceleration Library (DAAL) Intel distribution optimized for machine learning High performance machine learning & data analytics library Open source compiler for deep learning model computations optimized for multiple devices (CPU, GPU, NNP) from multiple frameworks (TF, MXNet, ONNX) Intel® Math Kernel Library for Deep Neural Networks (MKL-DNN) Open source DNN functions for CPU / integrated graphics Machinelearning Deeplearning * * * * * using open AI software Visit: www.intel.ai/technology
  • 12. 12 © 2019 Intel Corporation DeployAIanywhere Visit: www.intel.ai/technology All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. with unprecedented hardware choice C l o u d D e v I c e e d g e Intel GPU Automated Driving Dedicated Media/Vision Flexible Acceleration Dedicated DLInference Dedicated DLTraining Graphics,Media &Analytics NNP-TNNP-I Ifneeded
  • 13. 13 © 2019 Intel Corporation DeeplearningDeployed Tools Containers Intel-optimized Frameworks Libraries Training Trained Model Weights Inference Cloud Intel® Distribution of OpenVINO™ Toolkit Multi-Use Cluster (Spark/HPC/other) Cloud Multi-Use Cluster (Spark/HPC/other) Dedicated Training Cluster (with Nauta) End-to-end deep learning on Intel Data Model Optimization Edge Server Device -OR- -OR- -OR- -OR- -OR- -OR- -OR- All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Optimization Notice
  • 15. 15 © 2019 Intel Corporation 2nd generationINTEL®XEON®SCALABLEPROCESSOR Cascade Lake PerformanceTCO/Flexibility SecUrity Built-in Acceleration with Intel® Deep Learning Boost…  IMT – Intel® Infrastructure Management Technologies  ADQ – Application Device Queues  SST – Intel® Speed Select Technology  Intel® Security Essentials  Intel® SecL: Intel® Security Libraries for Data Center  TDT – Intel® Threat Detection TechnologyThroughput (img/s) Drop-in compatible CPU on Intel® Xeon® Scalable platform formerly known as Begin your AI journey efficiently, now with even more agility… Hardware-Enhanced Security… deep learning throughput!1 Up to 30X 1 Based on Intel internal testing: 1X,5.7x,14x and 30x performance improvement based on Intel® Optimization for Café ResNet-50 inference throughput performance on Intel® Xeon® Scalable Processor. See Configuration Details 3 Performance results are based on testing as of 7/11/2017(1x) ,11/8/2018 (5.7x), 2/20/2019 (14x) and 2/26/2019 (30x) and may not reflect all publically available security updates. No product can be absolutely secure. See configuration disclosure for details. , Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance
  • 16. 16 © 2019 Intel Corporation Intel®DeepLearningBoost(DLBoost) INT8 07 06 05 04 03 02 01 00 Sign Mantissa featuring Vector Neural Network Instructions (VNNI) NEW vpdpbusd OUTPUT INT32 CONSTANT INT32 INPUT INT8 INPUT INT8 AVX-512 (VNNI) instruction to accelerate INT8 convolutions: vpdpbusd INPUT INT8 INPUT INT8 vpmaddubsw vpmaddwd vpaddd OUTPUT INT16 OUTPUT INT32 CONSTANT INT16 CONSTANT INT32 OUTPUT INT32 Current AVX-512 instructions to perform INT8 convolutions: vpmaddubsw, vpmaddwd, vpaddd
  • 17. 17 1 5.7x inference throughput improvement with Intel® Optimizations for Caffe ResNet-50 on Intel® Xeon® Platinum 8180 Processor in Feb 2019 compared to performance at launch in July 2017. See configuration details on Config 1 Performance results are based on testing as of dates shown in configuration and may not reflect all publicly available security updates.28/24/2018) Results have been estimated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. No product can be absolutely secure. See configuration disclosure for details. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance Baseline 50x 285x 2S Intel® Xeon® Scalable Processor (Skylake) July 2017 July 2017 Skylake launch February 2019 vs. Baseline vs. Baseline DeeplearningperformanceonCPU I N T 8 V N N I Resnet-50 1.9x2 Inception v3 1.8x2 SSD-VGG16 2.0x2 Caffe Resnet-50 1.9x2 Inception v3 1.8x2 TensorFlow SKYLAKE CASCADE LAKE Hardware + Software Improvements for Intel® Xeon® Processors
  • 18. 18 © 2019 Intel Corporation Intel®Nervana™neuralnetworkprocessors(NNP) ¥ ‡The Intel® Nervana™ Neural Network Processor is a future product that is not broadly available today All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. NNP-T NNP-I Highly-efficient multi-model inferencing for cloud, data center and intense appliances Fastest time-to-train with high bandwidth AI server connections for the most persistent, intense usage DEDICATED DL TRAINING DEDICATED DL INFERENCE
  • 19. 19 © 2019 Intel Corporation FIRSTTOMARKETTOACCELERATE EvolvingAIWORKLOADS Precision Latency Sparsity AdversarialNetworks ReinforcementLearning NeuromorphicComputing … Real-time workloads Rnn Lstm SpeechWL DeliveringAI+forFlexible systemlevelfunctionality AI+I/OIngest AI+Networking Ai+security Ai+pre/postprocessing … Falcon Mesa Intel®FPGAforAI Enabling real-time AI in a wide range of embedded, edge and cloud apps All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
  • 20. 20 © 2019 Intel Corporation DRONES • Sense and avoid • GPS denied hovering • Pixel labeling • Video, image capture Surveillance • Detection/classification • Identification • Multi-nodal systems • Multimodal sensing • Video, image capture SMARTHOME • Detection, tracking • Perimeter, presence monitoring • Recognition, classification • Multi-nodal systems • Multimodal sensing • Video, image capture SERVICEROBOTS • Navigation • 3D Vol. mapping • Multimodal sensing WEARABLES • Detection, tracking • Recognition • Video, image, session capture AR-VRHMD • 6DOF pose, position, mapping • Gaze, eye tracking • Gesture tracking, recognition • See-through camera Intel®Movidius™Visionprocessingunit(vPU) Power-efficient image processing, computer vision & deep learning for devices All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
  • 21. 21 © 2019 Intel Corporation https://software.intel.com/en- us/iot/speech-enabling-dev-kit TryitTODAY! Intel® Speech Enabling Developer Kit Learn more: https://sigport.org/sites/default/files/docs/PosterFinal.pdf Amplethroughput For speech, language, and other sensing inference Lowpower <100 mW power consumption for always-on applications Flexibility Gaussian mixture model (GMM) and neural network inference support Intel® GNA (IP) DSP Intel®Gaussianneuralaccelerator(GNA) Streaming Co-Processor for Low-Power Audio Inference & More All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
  • 22. 22 © 2019 Intel Corporation Hardwareintegration Softwaresupport MacOS (CoreML and MPS1) Windows O/S (WinML) OpenVINO™ Toolkit (Win, Linux) clDNN Ubiquity/Scalability • Shipped in 1 billion+ Intel® SoCs • Broad choice of performance/power across Intel Atom®, Intel® Core™, and Intel® Xeon® processors Powerful,FlexibleArchitecture • Rich data type support for 32bitFP, 16bitFP, 32bitInteger, 16bitInteger with SIMD multiply-accumulate instructions MediaLeadership • Intel® Quick Sync Video – fixed-function media blocks to improve power and performance • Intel® Media SDK – API that provides access to hardware-accelerated codecs MemoryArchitecture • Shared memory architecture on die between CPU and GPU to enable lower latency and power Intelintegratedprocessorgraphics Built-in deep learning inference acceleration All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
  • 23. 23 © 2019 Intel Corporation Visit www.mobileye.com Mobileye–anintelcompany Automated Driving Platform All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.
  • 24. 24 Framework Topology Library DeeplearningSoftware Provides a standard way to customize and deploy Deep Neural Networks Supplies generic functionality which can be selectively extended with user defined additions to enhance specific tasks A set of algorithms modeled loosely after the human brain that forms a “network” designed to recognize complex patterns Also referred to as a network, NN, or model. Highly optimized functions intended to be used in Neural Network implementations to maximize performance Libraries are framework and model agnostic, hardware specific MACHINE LEARNING LIBRARIES Python R Distributed • Scikit-learn • Pandas • NumPy • Cart • Random Forest • e1071 • MlLib (on Spark) • Mahout DAAL MKL-DNN clDNN Intel® Data Analytics Acceleration Library (includes machine learning) Open-source deep neural network functions for CPU / integrated graphics
  • 25. 25 © 2019 Intel Corporation Frameworks AmazonGoogle FacebookMicrosoft https://github.com/onnx/onnx Open Neural Network Exchange (ONNX) • Enabling interoperability between different frameworks • Streamlining the path from research to production Caffe2, PyTorch, Microsoft Cognitive Toolkit, Apache MXNet ONNX promises to ease interoperability between frameworks.
  • 26. 26 ConvolutionalNeuralNetwork(CNN) → Layers of neurons connected by learned weights → Three layer classes: • Convolution Layer • Pooling Layer • Fully connected Layer → Connections between the layers point in one direction (Feed Forward Network) → Used to extract features from images for recognition or detection RecurrentNeuralNetwork(RNN) → Allows long-term dependencies to affect the output by relying on internal memory to process inputs → Types of RNN: • LSTM (Long Short Term Memory) • GRU (Gated Recurrent Unit) → Designed to analyze sequential data (i.e., NLP) = x1 x2 x3 x4 y1 y2 y3
  • 27. 27 © 2019 Intel Corporation IMAGE RECOGNITION http://ieeexplore.ieee.org/document/7124883/ Workload: ResNet50, InceptionV3, SSD-VGG16 SPEECH RECOGNITION https://www.news- medical.net/whitepaper/20170821/Speech-Recognition-in- Healthcare-a-Significant-Improvement-or-Severe- Headache.aspx Workload: DeepSpeech2 TEXT PROCESSING https://www.microsoft.com/en-us/translator/mt.aspx#nnt Workload: Neural Machine Translation CNN RNN
  • 28. 28 © 2019 Intel Corporation Purpose Neural Network Type Neural Network Example Image Recognition Classify image(s) CNN - Convolutional Neural Network ResNet50, Inception V3, Inception_ResV2, MobileNet, SqueezeNet, DenseNet Object Detection Locate AND classify object(s) in image CNN SSD– Single Shot Detector, R-FCN, Yolo_V2, VGG_16, DDRN/D-DBPN Natural Language Processing (NLP) Extract context & meaning from text LTSM (RNN)- Long Short Term Memory Google NMT Speech Recognition Convert speech to text RNN - Recursive Neural Network Baidu DeepSpeech, Transformer Text-to-Speech Convert text to speech GAN - Generative Adversarial Network with CNN/RNN WaveNet Recommendation Systems Recommend ads, search, apps, etc. MLP - Multilayer Perceptron Wide & Deep MLP, NCF Data Generation Create images like training data GAN with CNN DCGAN, DRAW Reinforcement Learning Learning from feedback on action CNN or CNN+RNN A3C All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. TopologiesforDeeplearning Golden Retrieve r
  • 29. 30 © 2019 Intel Corporation Comparisonofdifferentnetworks An Analysis of Deep Neural Network Models for Practical Applications Alfredo Canziani & Eugenio Culurciello & Adam Paszke Single-Crop top-1 validation accuracies AlexNet depth: 8 layers, GoogleNet depth: 22 layers, VGG depth: 16 layers or 19 layers, ResNet depth: 50 layers or 101 layers Single-Crop top-1 validation accuracies versus amount of operation for a single forward pass Legend in gray circles in the image shows number of network parameters The number of network parameters captured and depth of the networks is not necessarily linear with Top-1 accuracy that can be achieved CNN
  • 30. 31 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 32. 33 © 2019 Intel Corporation Deep Learning Development Cycle Data acquisition and organization Integrate trained models with application code Create models Adjust models to meet performance and accuracy objectives Intel® Distribution OpenVINO™ Toolkit Provides Deployment from Intel® Edge to Cloud
  • 33. 34 © 2019 Intel Corporation Lots of Labeled Data! Training Inference Forward Backward Model Weights Forward “Bicycle”? “Strawberry” “Bicycle”? Error Human Bicycle Strawberry ?????? Data Set Size Accuracy Did You Know? Training requires a very large data set and deep neural network (many layers) to achieve the highest accuracy in most cases Deep Learning: Training vs. Inference
  • 34. 35 © 2019 Intel Corporation Latest version is 2018 R5 What it is A toolkit to accelerate development of high performance computer vision & deep learning into vision applications from device to cloud. It enables deep learning on hardware accelerators and easy deployment across multiple types of Intel® platforms. Who needs this product?  Computer vision/deep learning software developers  Data scientists  OEMs, ISVs, System Integrators Usages Security surveillance, robotics, retail, healthcare, AI, office automation, transportation, non-vision use cases (speech, text) & more. Free Download  software.intel.com/openvino-toolkit Open Source version  01.org/openvinotoolkit Highperformance,performaiattheedge Streamlined&Optimizeddeeplearninginference Heterogeneous,cross-platformflexibility
  • 35. 36 © 2019 Intel Corporation Intel®distributionofOpenvino™toolkit software.intel.com/openvino-toolkit Supports 100+ public models, incl. 30+ pretrained models D E E P L E A R N I N G C O M P U T E R V I S I O N Model Optimizer Inference Engine Computer vision library (kernel & graphic APIs) Multiple products launched based on this toolkit Breadth of product portfolioRapid adoption by developers SupportsmajorAIframeworks Cross-platformflexibility HighPerformance,highEfficiency Optimized media encode/decode functions OpenCL™ Strong Adoption + Rapidly Expanding Capability Optimization Notice
  • 36. 37 © 2019 Intel Corporation Caffe* TensorFlow* MxNet* .dataIR IR IR = Intermediate Representation format Load, infer CPU Plugin GPU Plugin FPGA Plugin NCS Plugin Model Optimizer Convert & Optimize Model Optimizer  What it is: A Python*-based tool to import trained models and convert them to Intermediate representation.  Why important: Optimizes for performance/space with conservative topology transformations; biggest boost is from conversion to data types matching hardware. Inference Engine  What it is: High-level inference API  Why important: Interface is implemented as dynamically loaded plugins for each hardware type. Delivers best performance for each type without requiring users to implement and maintain multiple code pathways. Trained Models Inference Engine Common API (C++ / Python) Optimized cross- platform inference OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos GPU = Intel CPU with integrated graphics processing unit/Intel® Processor Graphics Kaldi* ONNX* (Pytorch, Caffe2 & more) GNA Plugin Extendibility C++ Extendibility OpenCL™ Extendibility OpenCL™ VAD Plugin VAD = Vision Accelerator Design Products; includes FPGA and 8 MyriadX versions Optimization Notice Intel®DeeplearningdeploymenttoolkitFor Deep Learning Inference – Part of Intel® Distribution of OpenVINO toolkit
  • 37. 38 © 2019 Intel Corporation Deploy/Visualize App Developers Model Data Scientists Prepare Data Engineers LibraryDevelopers AISoftwareinThebigpicture Ingest/Store Architects API’s Enterprise Applications Other names and brands may be claimed as the property of others. Note: displayed logos are not a complete list of solutions/providers in each category, personas are not mutually-exclusive by workflow step, and categorization is generalized Manage Metadata And more… Collect, Integrate, ETL & ELT Open (Managed)Open And more… Proprietary Pre-Process Data And more… Visualize Proprietary And more… Machine Learning & Analytics Proprietary And more… Store & Manage Big Data Open (Managed)Open And more…Proprietary: IT Systems Management And more… Unified Workflows And more… Intel® DL Studio Deploy Inference And more… Deep Learning And more… Optimization Notice
  • 38. 39 © 2019 Intel Corporation Quick Guide: What’s Inside the Intel Distribution vs Open Source version of OpenVINO™ toolkit 1Intel MKL is not open source but does provide the best performance 2Refer to readme file for validated versions Tool/Component Intel® Distribution of OpenVINO™ toolkit OpenVINO™ toolkit (open source) Open Source Directory https://github.com Installer (including necessary drivers) P Intel® Deep Learning Deployment toolkit Model Optimizer P P /opencv/dldt/tree/2018/model-optimizer Inference Engine P P /opencv/dldt/tree/2018/inference-engine Intel CPU plug-in P Intel® Math Kernel Library (Intel® MKL) only1 P BLAS, Intel® MKL1 , jit (Intel MKL) /opencv/dldt/tree/2018/inference-engine Intel GPU (Intel® Processor Graphics) plug-in P P /opencv/dldt/tree/2018/inference-engine Heterogeneous plug-in P P /opencv/dldt/tree/2018/inference-engine Intel GNA plug-in P Intel® FPGA plug-in P Intel® Neural Compute Stick (1 & 2) VPU plug-in P Intel® Vision Accelerator based on Movidius plug-in P 30+ Pretrained Models - incl. Model Zoo (IR models that run in IE + open sources models) P P /opencv/open_model_zoo Samples (APIs) P P /opencv/dldt/tree/2018/inference-engine Demos P P /opencv/open_model_zoo Traditional Computer Vision OpenCV* P P /opencv/opencv OpenVX (with samples) P Intel® Media SDK P P2 /Intel-Media-SDK/MediaSDK OpenCL™ Drivers & Runtimes P P2 /intel/compute-runtime FPGA RunTime Environment, Deep Learning Acceleration & Bitstreams (Linux* only) P
  • 39. 40 © 2019 Intel Corporation Expedite development, accelerate deep learning inference performance, and speed production deployment. Speed Deployment with Pre-trained Models & Samples  Age & Gender  Face Detection–standard & enhanced  Head Position  Human Detection–eye-level & high-angle detection  Detect People, Vehicles & Bikes  License Plate Detection: small & front facing  Vehicle Metadata  Human Pose Estimation  Action recognition – encoder & decoder  Text Detection & Recognition  Vehicle Detection  Retail Environment  Pedestrian Detection  Pedestrian & Vehicle Detection  Person Attributes Recognition Crossroad  Emotion Recognition  Identify Someone from Different Videos– standard & enhanced  Facial Landmarks  Gaze estimation Pretrained Models in Intel® Distribution of OpenVINO™ toolkit  Identify Roadside objects  Advanced Roadside Identification  Person Detection & Action Recognition  Person Re-identification–ultra small/ultra fast  Face Re-identification  Landmarks Regression  Smart Classroom Use Cases  Single image Super Resolution (3 models)  Instance segmentation  and more… Binary Models  Face Detection Binary  Pedestrian Detection Binary  Vehicle Detection Binary  ResNet50 Binary
  • 40. 41 © 2019 Intel Corporation  Object Detection  Hello Infer Classification  Standard & Pipelined Image Classification  Interactive Face Detection  Security Barrier  Image Segmentation  Object Detection SSD  Validation Application  Neural Style Transfer  Multi-channel Face Detection  Object Detection for Single Shot Multibox Detector using Asynch API+ Save Time with Deep Learning Samples Use Model Optimizer & Inference Engine for public models & Intel pretrained models
  • 41. 42 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 42. 43
  • 43. 44 VisionInferencerequirementsVary Design Variable Considerations Performance Frames/sec, latency (real-time constraints) Power Power envelope, power per inference Cost Hardware cost, development cost, cost per inference Batch size 1, 8, 32, etc. Precision FP32, FP16, FP11 Image size 720p, 1080p, 4K Camera input Raw video, encoded Network support CNN, RNN, LSTM, custom Operating environment Temperature controlled environment, outdoor use, 24/7 operation Product life Operational lifespan, product availability Regulation Security, privacy There is no “one-size-fits-all” solution . . . . . . and sometimes requirements change
  • 44. 45 • Field Programmable Gate Array (FPGA) – Millions of logic elements – Thousands of embedded memory blocks – Thousands of DSP blocks – Programmable routing – High speed transceivers – Various built-in hardened IP • Used to create Custom Hardware • Well-suited for Matrix Multiplation DSP Block Memory Block Programmable Routing Switch Logic Modules FPGAOverview
  • 45. 46 Intel® FPGA PAC N3000 for networking Intel® FPGA PAC D5005 for Datacentre Mustang-F-100-A10* Intel®FPGA–applicationAccelerationfromEdgetoCloud Combining Intel FPGA hardware and software to efficiently accelerate workloads for processing-intense tasks Programmable Acceleration Card (PAC) offering Devices/edge Network/NFV Cloud/enterprise * Other names and brands may be claimed as the property of others. Intel® FPGA PAC with Arria®10 GX
  • 46. 47 WhyFPGAForDeepLearningInference Keydifferentiators• Low latency • Low power • High throughput (FPS) • Flexibility to adapt to new, evolving, and custom networks • Supports large image sizes (e.g., 4K) • Large networks (up to 4 billion parameters) • Security / Encryption • Wide ambient temperature range (-40° C to 105° C)* • 24/7/365 operation • Long lifespan (8–10 years) Vision Acceleration Design with Intel® Arria® 10 FPGA *-40°C to 105°C temp range is for chip down design. Board temp range is 0° to 65°C.
  • 47. 48 IncreaseDeepLearningWorkloadPerformanceonPublicModels usingIntel®DistributionofOpenVINO™toolkit&Intel®Architecture Optimize with OpenVINO on CPU, Get an even Bigger Performance Boost with Intel® FPGA 1Depending on workload, quality/resolution for FP16 may be marginally impacted. A performance/quality tradeoff from FP32 to FP16 can affect accuracy; customers are encouraged to experiment to find what works best for their situation. Performance results are based on testing as of June 13, 2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Configuration: Testing by Intel as of June 13, 2018. Intel® Core™ i7-6700K CPU @ 2.90GHz fixed, GPU GT2 @ 1.00GHz fixed Internal ONLY testing, Test v3.15.21 – Ubuntu* 16.04, OpenVINO 2018 RC4, Intel® Arria® 10 FPGA 1150GX. Tests were based on various parameters such as model used (these are public), batch size, and other factors. Different models can be accelerated with different Intel hardware solutions, yet use the same Intel software tools. Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Public Models (Batch Size) RelativePerformance Improvement Standard Caffe* Baseline 19.9x1 ComparisonofFramesperSecond(FPS)
  • 48. 49 Solving Machine Learning Challenges with FPGA Real-Time deterministic lowlatency Ease-of-use softwareabstraction, platforms&libraries Flexibility customizablehardware fornextgenDNNarchitectures Intel FPGAs can be customized to enable advances in machine learning algorithms. Intel FPGA hardware implements a deterministic low latency data path unlike any other competing compute device. Intel FPGA solutions enable software-defined programming of customized machine learning accelerator libraries.
  • 49. 50 Public Intel FPGA Machine Learning Success Microsoft: “Microsoft has revealed that Altera FPGAs have been installed across every Azure cloud server, creating what the company is calling “the world’s first AI supercomputer.” -Sept 2016 NEC: “To create the NeoFace Accelerator, the engine software IP was integrated into an Intel Arria 10 FPGA, which operate in Xeon processor–based servers. ” -June 2017 JD.COM: “Arria®10 FPGA can achieve 5x improvement in the performance of LSTM accelerator card compared to GPU.” –Apr 2017 Inspur/iFlytech: “Leading server vendor Inspur Group and Altera today launched a speech recognition acceleration solution based on Altera's Arria® 10 FPGAs and DNN algorithm from iFLYTEK.” -Dec 2016 ZTE: “Using Intel’s Arria 10 FPGA, ZTE engineers were able to achieve more than 1000 images per second in facial recognition.” –Jan 2017
  • 50. 51 Why FPGAs for Deep Learning? PE PE PE PE PE PE PE PE feeder feeder feeder feeder feeder feeder feeder feeder PE PE PE PE PE PE PE PE
  • 51. 52 Customized Performance Convolutional Neural Networks are Compute Intensive Feature Benefit Highly parallel architecture Facilitates efficient low-batch video stream processing and reduces latency Configurable Distributed Floating Point DSP Blocks FP32 9Tflops, FP16, FP11 Accelerates computation by tuning compute performance Tightly coupled high-bandwidth memory >50TB/s on chip SRAM bandwidth, random access, reduces latency, minimizes external memory access Programmable Data Path Reduces unnecessary data movement, improving latency and efficiency Configurability Support for variable precision (trade-off throughput and accuracy). Future proof designs, and system connectivity Fine-grained & low latency between compute and memory Convolutional Neural Networks are Compute Intensive Function 2Function 1 Function 3 IO IO Optional Memory Optional Memory Pipeline Parallelism
  • 52. 53 © 2019 Intel Corporation FPGAs System Flexibility to Control the Data path Storage Acceleration  Machine learning  Cryptography  Compression  Indexing Inline Data Flow Processing  Machine learning  Object detection and recognition  Advanced driver assistance system (ADAS)  Gesture recognition  Face detection Compute Acceleration/Offload  Workload agnostic compute  FPGAaaS  Virtualization Intel® Xeon® Processor
  • 53. 54 © 2019 Intel Corporation FPGA Flexibility Supports Arbitrary Architectures Many efforts to improve efficiency – Batching – Reduce bit width – Sparse weights – Sparse activations – Weight sharing – Compact network I W O I X W = ··· ··· O 2 3 3 2 1 3 13 1 3 Shared Weights SparseCNN [CVPR’15] Spatially SparseCNN [CIFAR-10 winner ‘14] Pruning [NIPS’15] TernaryConnect [ICLR’16] BinaryConnect [NIPS’15] DeepComp [ICLR’16] HashedNets [ICML’15] XNORNet SqueezeNet LeNet [IEEE} AlexNet [ILSVRC’12} VGG [ILSVRC’14} GoogleNet [ILSVRC’14} ResNet [ILSVRC’15}
  • 54. 55 © 2019 Intel Corporation FPGAs Provide Deterministic System Latency FPGAs can leveraging parallelism across the entire chip reducing the compute time to a fraction F P G A X e o n I/O Latency Compute Latency System Latency = I/O Latency + Compute Latency 0.25ms 1ms 0.3ms
  • 55. 56 © 2019 Intel Corporation Deep Learning Topology Inference Processing “head” 1 “head” 2 “head” 10 Neural net “the body” image … Most of the compute is here Vision: CNNs (ResNet) Similar for Speech and Translation features Feature vector for index Tags Object detect Post-processing Intel® FPGA Deep Learning Acceleration Suite Re-size / crop image Pre-processing
  • 56. 57 © 2019 Intel Corporation Intel® FPGA Deep Learning Acceleration Suite Features  CNN acceleration engine for common topologies executed in a graph loop architecture – AlexNet, GoogleNet, LeNet, SqueezeNet, VGG16, ResNet, Yolo, SSD…  Software Deployment – No FPGA compile required – Run-time reconfigurable  Customized Hardware Development – Custom architecture creation w/ parameters – Custom primitives using OpenCL™ flow Convolution PE Array Crossbar prim prim prim custom DDR Memory Reader /Writer Feature Map Cache DDR Config Engine
  • 57. 58 © 2019 Intel Corporation FPGA Usage with Intel® Distribution of OpenVINO™ toolkit  Supports common software frameworks (Caffe*, TensorFlow*)  Model Optimizer enhances model for improved execution, storage, and transmission  Inference Engine optimizes inference execution across Intel® hardware solutions using unified deployment API  Intel® FPGA DLA Suite provides turn-key or customized CNN acceleration for common topologies GoogLeNet Optimized Template ResNet Optimized Template Additional, Generic CNN Templates SqueezeNet Optimized Template VGG Optimized Template Deep Learning Deployment Toolkit Intel® Xeon® Processor Conv PE Array Crossbar DDR Memor y Reader /Writer Feature Map Cache DDR DDR DDR Config Engine Trained Model Caffe*, TensorFlow*, etc… Heterogenous CPU/FPGA Deployment Optimized Acceleration Engine Pre-compiled Graph Architectures Hardware Customization Supported Model Optimizer FP Quantize Model Compress Model Analysis Inference Engine DLA Runtime Engine MKL-DNN Intel® FPGA Intermediate Representation *Other names and brands may be claimed as the property of others
  • 58. 59 © 2019 Intel Corporation OpenVINO™ with DLA User Flows IP Architect Neural Net Design Intel® FPGA SDK for OpenCL™ Bitstream Library Application Developer Compile User Customization of DLA Suite Source Code OpenVINO™ toolkit Turnkey Software Deployment Flow FPGA Architecture Development Flow Design ProgramRun Choose from many precompiled FPGA Images Or Custom create FPGA bitstream
  • 59. 60 © 2019 Intel Corporation Machine Learning on Intel® FPGA Platform Acceleration Stack Platform Solution DLA Runtime Engine DLA Workload OpenCL™ Runtime FPGA Interface Manager Hardware Platform & IP Software Stack DL Deployment Toolkit Acceleration Stack Application Intel® Programmable Acceleration Card Intel® Xeon CPU ML Framework (Caffe*, TensorFlow*) Intel® FPGA Acceleration Hub Edge Computing Solution DLA Runtime Engine DLA Workload OpenCL™ Runtime OpenCL™ BSP Hardware Platform & IP Software Stack DL Deployment Toolkit OpenCL™ BSP Driver Application Intel® Vision Accelerator Design with FPGA Intel® CPU ML Framework (Caffe*, TensorFlow*) Intel® Vision Accelerator Design Products
  • 60. 61 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 61. 62 Intel® FPGADeepLearningAcceleration(DLAforOpenVINO) Pre-compiledGraphArchitecture ExampleTopologies DDR DDR DDR DDR Configuration Engine AlexNet GoogLeNet Tiny Yolo SqueezeNetVGG16 ResNet 18 …* ResNet 50ResNet 101 Crossbar CUSTOM* PRIM Conv PE Array Feature Map Cache PRIM PRIM *More topologies added with every release MobileNet ResNet SSD SqueezeNet SDD Memory Reader /Writer Convolution Processing Element Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine *Deeper customization options COMING SOON! * Other names and brands may be claimed as the property of others.
  • 62. 63 © 2019 Intel Corporation DLA Architecture: Built for Performance  Maximize Parallelism on the FPGA – Filter Parallelism (Processing Elements) – Input-Depth Parallelism – Winograd Transformation – Batching – Feature Stream Buffer – Filter Cache  Choosing FPGA Bitstream – Data Type / Design Exploration – Primitive Support ReLU Convolution / Fully Connected Norm MaxPool Stream Buffer Conv PE Array Crossbar ReL U Max Pool DDR Memory Reader /Writer Feature Map Cache DDR DDR DDR Config EngineNor m Execute
  • 63. 64 © 2019 Intel Corporation CNN Computation in One Slide Inew 𝑥 𝑦 = 𝑥′=−1 1 𝑦′=−1 1 Iold 𝑥 + 𝑥′ 𝑦 + 𝑦′ × F 𝑥′ 𝑦′ Input Feature Map (Set of 2D Images) Filter (3D Space) Output Feature Map Repeat for Multiple Filters to Create Multiple “Layers” of Output Feature Map
  • 64. 65 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer
  • 65. 66 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 66. 67 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected Norm MaxPool AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 67. 68 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Stream Buffer output input Blocks are run-time reconfigurable and bypassable
  • 68. 69 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 69. 70 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input MaxPool
  • 70. 71 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 71. 72 © 2019 Intel Corporation Mapping Graphs in DLA ReLU Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 72. 73 © 2019 Intel Corporation Mapping Graphs in DLA Convolution / Fully Connected AlexNet Graph Conv ReLu Norm MaxPool Fully Conn. Blocks are run-time reconfigurable and bypassable Stream Buffer output input
  • 73. 74 © 2019 Intel Corporation Efficient Parallel Execution of Convolutions  Parallel Convolutions – Different filters of the same convolution layer processed in parallel in different processing elements (PEs)  Vectored Operations – Across the depth of feature map  PE Array geometry can be customized to hyperparameters of given topology FPGA Double-Buffer On-Chip RAM Filters (on-chip RAM) FilterParallelism (OutputDepth) External DDR
  • 74. 75 © 2019 Intel Corporation Winograd Transformation  Perform convolutions with fewer multiplication – Allows more convolutions to be done on FPGA  Take 6 input features elements and 3 filter elements – Standard convolution requires 12 multiplies – Transformed convolution requires just 6 multiplies ReLU Convolution / Fully Connected Norm MaxPool Winograd Transform Stream Buffer Winograd Transform
  • 75. 76 © 2019 Intel Corporation Fully Connected Computation and Batching  Fully Connected Layer computation does not allow for data reuse of weights – Different from convolutions – Very memory bandwidth intensive  Solution: Batch up images – Weights reused across multiple images 𝑜 = 𝑰 𝒗𝒆𝒄 ∗ 𝑾 𝒗𝒆𝒄 … 𝑶 𝒗𝒆𝒄 = 𝑰 𝒎𝒂𝒕 ∗ 𝑾 𝒗𝒆𝒄 Batching
  • 76. 77 © 2019 Intel Corporation Feature Cache Feature data cached on-chip  Streamed to a daisy chain of parallel processing elements  Double buffered – Overlap convolution with cache updates – Output of one subgraph becomes input of another – Eliminates unnecessary external memory accesses Double-Buffer On-Chip RAM Stream buffer size
  • 77. 78 © 2019 Intel Corporation Filter Cache Filter weights cached in each processing element  Double buffered in order to support prefetching – While one set is used to calculate output feature maps, another set is prefetched DDR Conv Conv DDR
  • 78. 79 © 2019 Intel Corporation Design Exploration with Reduced Precision Tradeoff between performance and accuracy  Reduced precision allows more processing to be done in parallel  Using smaller Floating Point format does not require retraining of network  FP11 benefit over using INT8/9 – No need to retrain, better performance, less accuracy loss FP11 FP10 FP9 Sign, 5-bit exponent, 10-bit mantissaFP16 Sign, 5-bit exponent, 5-bit mantissa Sign, 5-bit exponent, 4-bit mantissa Sign, 5-bit exponent, 3-bit mantissa
  • 79. 80 © 2019 Intel Corporation Support for Different Topologies Tradeoff between features and performance Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine LRN Permute Concat FlattenSoftMax Reshape vs
  • 80. 81 © 2019 Intel Corporation Optimize for Best Performance Tradeoff between size of Feature Map cache and convolutional PE array Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine vs Convolution PE Array Crossbar ReLU Norm MaxPool Memory Reader /Writer Feature Map Cache Config Engine
  • 81. 82 FeatureCacheandPEArrayTradeoffs Performance Feature Cache Size Small Feature cache Bigger PE array, but more slicing Performance bottleneck = DDR Big Feature cache Less slicing, but smaller PE array Performance bottleneck = PE compute Sweet spot varies based on graph dimensions PE Array Size
  • 83. 84 Featurecacheandslicingforbiggraphs Convolution 73x73x128 73x73x32 Un-Sliced Graph 720 M20Ks Convolution Slice#1 73x73x128 73x37x32 Sliced Graph Feature Cache Feature Cache 180 M20Ks Convolution Slice#2 DDR Feature Cache Feature Cache DDR Feature Cache Required: 900 M20Ks 73x36x32 73x37x128 73x36x128 73x73x16 Feature Cache Required: 384 M20Ks
  • 84. 85 © 2019 Intel Corporation Precompiled Architectures (aocx) included with OpenVINO™ toolkit  Precompiled and included architectures are optimized for standard topologies  Better performance maybe achieved with custom architectures for custom deep learning networks – Size requirement maybe different from standard topology – Primitive requirement may be different from standard networks
  • 85. 86 © 2019 Intel Corporation Supported Primitives and Topologies TopologiesPrimitives  Conv  ReLu, Leaky ReLU  Concat  Eltwise  Pooling  Power  ScaleShift  Batch Norm  Fully Connected  LRM Normalization  Custom  AlexNet  GoogLeNet  SSD  ResNet-18/50/101/152  SSD  SqueezeNet  SSD  VGG-16/19  SSD  Tiny Yolo  LeNet  MobileNet v1/v2
  • 86. 87 © 2019 Intel Corporation Customizable DLA Architecture  Many aspects of a DLA architecture can be customized – Convolution Processing Element parallelization – Datatype – Primitive Support – Stream Buffer Size – Custom Primitives – etc. Convolution PE Array Crossbar prim prim prim DDR Memory Reader /Writer Feature Map Cache DDR Config Engine Customizable prim
  • 87. 88 © 2019 Intel Corporation End-to-End Machine Learning 1 Train 2 Prepare model 3 Inference Trained Model Caffe* Tensorflow* MXNet* ONNX* Model Optimizer Optimized Model GPUCPU MKLDNN clDNN Inference Engine Real-time Data DLA Runtime FPGA
  • 88. 89 © 2019 Intel Corporation DLA SW API Intel FPGA Deep Learning Acceleration Suite GoogleNet Optimized Template ResNet Optimized Template Additional, Generic CNN Templates SqueezeNet Optimized Template VGG Optimized Template • Supports common software frameworks (Caffe, Tensorflow) • Intel DL software stack provides graph optimizations • Intel FPGA Deep Learning Acceleration Suite provides turn- key or customized CNN acceleration for common topologies Caffe TensorFlow Intel® Xeon® Processor Intel Arria® FPGA Inference Engine Model Optimizer Conv PE Array Crossbar DD R Memor y Reader /Writer Feature Map Cache DD R DD R DD R Config Engine Optimized Acceleration Engine Standard ML Frameworks Intel Deep Learning Deployment Toolkit Heterogenous CPU/FPGA Deployment Pre-compiled Graph Architectures Hardware Customization Supported
  • 89. 90 © 2019 Intel Corporation Intel® FPGA Deep Learning Acceleration Suite  Contents – Pre-compiled FPGA Bitstreams – To support various layers required by different networks with different precisions – Primitives supported: – Convolution, Fully Connected, Inner Product, ReLU, pReLU, Pooling, Norm, Concat, Elsewise Add, etc… – Software runtime library – Board Support Packages to support various boards – Tools – Diagnose – Bitstream customization and generation
  • 90. 91 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 91. 92 © 2019 Intel Corporation
  • 92. 93 © 2019 Intel Corporation Model Optimizer  Convert models from various frameworks (Caffe, TensorFlow, MXNet, ONNX)  Converts to a unified Model (IR, later n-graph)  Optimizes topologies (Node merging, batch normalization elimination, performing horizontal fusion)  Folds constants paths in graph
  • 93. 94 © 2019 Intel Corporation Improve Performance with Model Optimizer  Easy to use, Python*-based workflow does not require rebuilding frameworks.  Import Models from various supported frameworks - Caffe*, TensorFlow*, MXNet*, ONNX*, Kaldi*.  100+ models for Caffe, MXNet and TensorFlow validated. All public models on ONNX* model zoo supported.  With Kaldi support, the model optimizer extends inferencing for non-vision networks.  IR files for models using standard layers or user-provided custom layers do not require Caffe.  Fallback to original framework is possible in cases of unsupported layers, but requires original framework. Trained Model Model Optimizer Analyze Quantize Optimizetopology Convert Intermediate Representation (IR) file
  • 94. 95 © 2019 Intel Corporation Model Optimizer Model optimizer performs generic optimization:  Node merging  Horizontal fusion  Batch normalization to scale shift  Fold scale shift with convolution  Drop unused layers (dropout)  FP16/Int8 quantization  Model optimizer can add normalization and mean operations, so some preprocessing is ‘added’ to the IR --mean_values (104.006, 116.66, 122.67) --scale_values (0.07, 0.075, 0.084) FP11 no no no yes INT8 yes no no no
  • 95. 96 © 2019 Intel Corporation Model Optimization Techniques Linear Operation Fusing & Grouped Convolutions Fusing Linear Operation Fusing: 3 stages 1. BatchNorm and ScaleShift decomposition: BN layers decomposes to Mul->Add->Mul->Add sequence; ScaleShift layers decomposes to Mul- >Add sequence. 2. Linear operations merge: Merges sequences of Mul and Add operations to the single Mul->Add instance. 3. Linear operations fusion: Fuses Mul and Add operations to Convolution or FullybConnected layers. Grouped Convolutions Fusing Specific optimization that applies for TensorFlow* topologies. (Xception*) Caffe Resnet269 block (from Netscope) Merged Caffe* Resnet269 block (from Netscope*) Split->Convolutions->Concat block from TensorBoard*
  • 96. 97 © 2019 Intel Corporation Model Optimizer: Linear Operation Fusing Example 1. Remove Batch normalization stage. 2. Recalculate the weights to ‘include’ the operation. 3. Merge Convolution and ReLU into one optimized kernel.
  • 97. 98 © 2019 Intel Corporation Model Optimizer: Cutting Off Parts of a Model Model optimizer can cut out a portion of the network:  Model has pre/post-processing parts that cannot be mapped to existing layers.  Model has a training part that is not used during inference.  Model is too complex and cannot be converted in one shot. Command line options  Model Optimizer provides command line options --input and --output to specify new entry and exit nodes ignoring the rest of the model:  --input option accepts a comma-separated list of layer names of the input model that should be treated as new entry points to the model;  --output option accepts a comma-separated list of layer names of the input model that should be treated as new exit points from the model.
  • 98. 99 © 2019 Intel Corporation Intermediate Representation (IR) Squeezenet1.1.caffemodel Squeezenet1.1.bin Squeezenet1.1.prototxt Squeezenet1.1.xml
  • 99. 100 © 2019 Intel Corporation Model Optimizer Options Option for Deployment Description --input_model Network binary weights file TensorFlow* .pb Caffe* .caffemodel MXNet* .params --input_proto Caffe .prototxt file --data_type IP Precision (i.e. FP16) --scale Network normalization factor (Optional) --output_dir Output directory path (Optional) Python script: $MO_DIR/mo.py Full Model Optimizer options covered in OpenVINO™ documentations
  • 100. 101 © 2019 Intel Corporation Run Model Optimizer  To generate IR .xml and .bin files for Inference Engine $ source $MO_DIR/venv/bin/activate $ cd $MO_DIR/ $ python mo.py --input_model <model dir>/<weights>.caffemodel --scale 1 --data_type FP16 --output_dir <output dir> Model Optimizer arguments Batch: 1 Precision of IR: FP16 Enable fusing: True Enable gfusing: True … [ SUCCESS ] Generated IR model. [ SUCCESS ] XML file: … [ SUCCESS ] BIN file: …
  • 101. 102 © 2019 Intel Corporation
  • 102. 103 © 2019 Intel Corporation Inference Engine Common API Plug-InArchitecture Inference Engine Runtime Movidius API Intel® Movidius™ Myriad™ X DLA Intel® Processor Graphics (GPU) CPU: Intel® Xeon®/Core™/Atom® clDNN Plugin Intel® Math Kernel Library (MKLDNN)Plugin OpenCL™Intrinsics FPGA Plugin Applications/Service Intel® Arria 10 FPGA  Simple & Unified API for Inference across all Intel® architecture  Optimized inference on large IA hardware targets (CPU/GEN/FPGA)  Heterogeneity support allows execution of layers across hardware types  Asynchronous execution improves performance  Futureproof/scale your development for future Intel® processors Transform Models & Data into Results & Intelligence Movidius Plugin OpenVX and the OpenVX logo are trademarks of the Khronos Group Inc. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos GPU = Intel CPU with integrated graphics/Intel® Processor Graphics/GEN GNA = Gaussian mixture model and Neural Network Accelerator GNA API Intel® GNA GNA* Plugin Optimal Model Performance Using the Inference Engine
  • 103. 104 Create IE Instance Load IE Plugin (MKL-DNN/clDNN/DLA) Read an IR Model Set Target Device (CPU/GPU/Movidius/FPGA) Load network to plugin Allocate input, output buffers Fill input buffer with data Run inference Interpret output results Initialization Main loop Initialize • Fill Input • Infer • Interpret Output Inference Engine Workflow
  • 104. 105 © 2019 Intel Corporation Inference Engine Details A runtime with a unified API for integrating inference with application logic – Delivers optimized inference solution with reduced footprint on inference execution for target hardware  libinference_engine.so library implements core functions – Loading and parsing of model IR – Preparation of input and output blobs – Triggers inference using specified plug-in  Include file: inference_engine.hpp
  • 105. 106 © 2019 Intel Corporation Inference Engine Plugins  CPU MKLDNN Plugin (Intel® Math Kernel Library for Deep Neural Networks) – Supports Intel® Xeon®/Core®/Atom® platform – Widest set of network classes supported, easiest way to enable topology  GPU clDNN Plugin (Compute Library for Deep Neural Networks) – Supports 9th generation and above Intel® HD and Iris graphics processors – Extensibility mechanism to develop custom layers through OpenCL™  FPGA DLA Plugin – Supports Intel® FPGAs – Basic set of layers are supported on FPGA, non-supported layers inferred through other plugins
  • 106. 107 © 2019 Intel Corporation Heterogeneous Plugin  Enables inference of one network on several devices. – Calculate heaviest parts of network on accelerator, unsupported fallback to CPU – To utilize all available hardware more efficiently during one inference  Heterogeneous execution can be triggered in two independent steps: – Loading a network to the Heterogeneous plugin, splitting the network to parts, and executing them through the plugin – Setting of affinity to layers (binding them to devices in InferenceEngine::ICNNNetwork) – Can be done automatically or in manual mode
  • 107. 108 © 2019 Intel Corporation Pre-processing  Most image formats are interleaved (RGB, BGR, BGRA, etc.)  Models usually expect RGB planar format: – R-plane, G-plane, B-plane Interleaved Planar
  • 108. 109 © 2019 Intel Corporation Post-processing Developers are responsible for parsing inference output. Many output formats are available. Some examples include: • Simple Classification (alexnet*): an array of float confidence scores, # of elements=# of classes in the model • SSD: Many “boxes” with a confidence score, label #, xmin,ymin, xmax,ymax Unless a model is well documented, the output pattern may not be immediately obvious.
  • 109. 110 © 2019 Intel Corporation Inference Engine Classes Class Details InferencePlugin Main plugin interface PluginDispatcher Finds suitable plug-in for specified device CNNNetReader Build and parse a network from given IR CNNNetwork Neural Network and binary information Blob, BlobMaps Container object representing a tensor (e.g. input/output) InputsDataMap Information about input of the network ExecutableNetwork Loaded Network InferRequest Interact with inputs/outputs and performs inference using namespace InferenceEngine;
  • 110. 111 © 2019 Intel Corporation Inference Engine API Usage (1) 1. Load Plugin – FPGA Plugin: libdliaPlugin.so – Others: libclDNNPlugin.so (GPU), libMKLDNNPlugin.so (CPU) – Plugin Dir: <OpenVINO install dir>/inference_engine/lib/<OS>/intel64 2. Load Network InferenceEngine::PluginDispatcher dispatcher(<pluginDir>); InferencePlugin plugin = dispatcher.getPluginByDevice(“HETERO:FPGA, CPU”); InferenceEngine::CNNNetReader netBuilder netBuilder.ReadNetwork(“<Model>.xml”); netBuilder.ReadWeights(“<Model>.bin”);
  • 111. 112 © 2019 Intel Corporation Inference Engine API Usage (2) 3. Prepare Input and Output Blobs – For Inputs – Allocate based on the size of the input, number of channels, batch size, etc. – Set input precision – Set layout – Fill in data (i.e. from rgb value of image) – For Outptus – Set output precision – Allocate based on output format
  • 112. 113 © 2019 Intel Corporation Inference Engine API Usage (3) 4. Load the model to the plugin 5. Perform inference 6. Process output blobs ExecutableNetwork exeNet = plugin.LoadNetwork(netBuilder.getNetwork(), {}); InferRequest infer_request = exeNet.CreateInferRequest(); infer_request.Infer(); const Blob::Ptr output_blob = infer_request.GetBlob(firstOutputName);
  • 113. 114 © 2019 Intel Corporation Using the Inference Engine API IR Parse (using CNNNetReader) Load Network Create Engine Instance Infer CNNNetReader netBuilder; netBuilder->ReadNetwork(“Model.xml”); netBuilder->ReadWeights(“Model.bin") InferencePlugin plugin PluginDispatcher({}).getPluginbyDevice(“HETERO:FPGA, CPU”); ExecutableNetwork exeNet = plugin.LoadNetwork(netBuilder->getNetwork(), {}); InferRequest infer_request = exeNet.CreateInferRequest(); //Create Input Blob, iterate through and fill in data infer_request.Infer(); //Process Output CNNNetwork instance CNNNetwork instance
  • 114. 115 © 2019 Intel Corporation Batch Execution  For better performance, using a larger batch size may help  Allocate input and output Blob according to batch size  Set the Batch size on the network netBuilder.getNetwork().setBatchSize(<size>);
  • 115. 116 © 2019 Intel Corporation Inference Engine Example Applications  Execute samples and demos with FPGA support – Shipped with the OpenVINO™ toolkit – classification_sample – object_detection_sample_ssd – many others…
  • 116. 117 © 2019 Intel Corporation Example Application using FPGAs $ ./object_detection_sample_ssd -m <path_to_model>/Model.xml -i <path_to_pictures>/picture.jpg -d HETERO:FPGA,CPU The “priorities” defines a greedy behavior:  Keeps all layers that can be executed on the device (FPGA)  Carefully respects topological and other limitations  Follows priorities when searching (for example, CPU)
  • 117. 118 © 2019 Intel Corporation DLA FPGA IP CNN primitives (Conv, FC, ReLU, MaxPool, Norm..) IE Software Architecture with FPGAs Board Support Package DLA Software Runtime Inference Engine OpenCL™ Runtime Driver to the FPGA board OpenCL API to access FPGA DLA API Unified API for Intel® platforms Inference Engine Plugin HETERO/FPGA Plugin Deep Learning Deployment Toolkit Intel® FPGA DLA Suite
  • 118. 119 © 2019 Intel Corporation Software Requirements  Intel® Distribution of OpenVINO™ toolkit (Linux for FPGA) – Includes Intel® FPGA Runtime (RTE) for OpenCL™  For Vision Accelerator Design with FPGA board – OpenCL™ Board Support Package for Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA – Intel® Quartus® Prime Pro Edition Programmer  For Programmable Acceleration Card – Intel® Acceleration Stack for Runtime – Includes Programmer, RTE for OpenCL, and OpenCL BSP DLA Runtime Engine OpenCL™ Runtime Software Stack DL Deployment Toolkit Acceleration Stack User DL Inference Application OpenCL™ BSP Driver OpenVINO
  • 119. 120 © 2019 Intel Corporation FPGA Software Components  Intel® FPGA OpenCL™ Runtime – Included as part of OpenVINO™ – Inference acceleration done through OpenCL calls  Board Support Package – Translates OpenCL calls to low-level device driver calls – Allows programming of FPGA with different bitstreams – Included as part of Acceleration Stack for PAC boards  Intel® Quartus® Prime Programmer – Possibly needed to flash/program the board with FPGA image to communicate with host DLA Runtime Engine OpenCL™ Runtime Software Stack DL Deployment Toolkit Acceleration Stack User DL Inference Application OpenCL™ BSP Driver OpenVINO
  • 120. 121 © 2019 Intel Corporation Prepare FPGA Environment for Inference  Once software stack is installed and environment set  Ensure PCIe* connection is operating – lspci | grep Altera  Ensure FPGA OpenCL is working correctly – aocl diagnose – Ensure DIAGNOSTIC_PASSED
  • 121. 122 © 2019 Intel Corporation Load FPGA Image and Execute IE Application  FPGAs needs to be preconfigured with primitives prior to application execution  Choose FPGA bitstream from the DLA suite – Based on topology needs and data type requirements – Option to create custom FPGA bitstream based on requirements  Execute User or Example Application aocl program acl0 /opt/intel/openvino/bitstreams/<board>/2019R1_PL1_FP11_MobileNet_SqueeeNet_VGG.aocx
  • 122. 123 © 2019 Intel Corporation FPGA Image Selection  Precompiled FPGA images included  Choose image based on – Primitives needed (architecture) – Data type support for accuracy/performance trade-off – K (filter) and C (channel depth) vectorization for performance – Data path width – On-chip stream buffer depth  May also generate customized FPGA image
  • 123. 124 © 2019 Intel Corporation Use OpenVINO™ Running SqueezeNet example manually or create your own application 1. Create a new test directory and cd into it, ensure environment script is ran 2. Copy original Caffe* files from the demo directory into the new directory 3. Copy the imagenet labels from the demo directory 4. Execute model optimizer to convert to intermediate representation 5. Execute classification sample cp ~/openvino_models/models/FP32/classification/squeeenet/1.1/caffe/squeeenet1.1.* . cp ~/openvino_models/ir/FP32/classification/squeeenet/1.1/caffe/squeeenet1.1.labels . mo –input_model squeezenet1.1.caffemodel classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png –d HETERO:FPGA,CPU classification_sample –m squeezenet1.1.xml –i $IE_INSTALL/demo/car.png –d HETERO:FPGA,CPU –ni 100
  • 124. 125 © 2019 Intel Corporation Summary  The Intel® Deep Learning Acceleration Suite part of Intel® Distribution of OpenVINO™ tookit provides out-of-box deployment of deep learning inference on FPGAs  Deep Learning Deployment Toolkit can be used as a frontend for the DLA – Consists of the Model Optimizer and Inference Engine – Use Model Optimizer to optimize and convert framework model to unified internal representation – Use Inference Engine unified API to deploy network on various devices including the FPGA
  • 125. 126 © 2019 Intel Corporation References  Follow-on Training – Deploying FPGAs for Deep Learning Inference with Intel® Distribution of OpenVINO™ toolkit  Intel® AI – www.intel.com/ai  Intel® Distribution of OpenVINO™ toolkit – https://software.intel.com/en-us/openvino-toolkit  Intel® FPGAs for AI  Intel® FPGA Acceleration Stack for Intel® Xeon® CPU with FPGAs
  • 126. 127 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 128. Introducing Intel® Agilex™ FPGAs Intel innovation for ultimate agility and flexibility AnyDeveloper Intel® Quartus® Prime Design Tool for Hardware Developers One API for Software Developers Any-to-anyintegration Memory, Analog, Logic Any Node, Supplier, IP Rapid Intel® eASIC™ Devices Optimization 10nm Process Memory-Coherent Intel® Xeon® Processor Acceleration Massive Bandwidth High-Performancecompute 129
  • 129. The FPGA for the Data-Centric World 130 Upto 40% Higher performance1,3 Upto 40% Lower Power1,3 Process Data Store Data Move Data DDR5& HBM Intel®Optane™DC PersistentMemorysupport Transceiver datarates Intel®Xeon®Processorcoherent connectivity&PCIe Gen5 112G 1 Compared to Intel® Stratix® 10 FPGAs 2 With FP16 configuration 3 Based on current estimates, see slide 180 for details Upto 40TFLOPS DSPPerformance 2 Generation Intel® Hyperflex™ Architecture nd2 2,3
  • 130. Any-to-Any Heterogenous 3D Integration 131 Chiplet-BasedArchitecture Library of chiplets: transceivers, custom I/O & custom compute tiles Intel®eASIC™TileCustomization Ultimate customization with integrated Intel® eASIC™ tile Embedded Multi-dieinterconnectbridge No-compromise die-to-die 3D packaging interconnect
  • 131. Custom Logic Continuum 132 Flexible and Agile Optimization Across Product Lifecycle FPGA Fastest Time to Market Highest Flexibility Highest Performance Lowest Power & Cost* Optimized TTM with Performance / Cost eASIC ASIC Reusable IP 1Refers to Intel® eASIC™ Devices * Compared to other options shown 1
  • 132. First Memory-Coherent Accelerator for Intel® Xeon® Scalable Processors 133 FPGA Memory Processor Memory Compute Express Link* (CXL) Accelerates Diverse Workloads • LowLatency • MemoryCoherence • IncreasedMemorySpace (Including data analytics, database acceleration and function-as-a-service)
  • 133. 134 112G Pcie Gen5 * High Bandwidth for Applications Including 400G Networks, Edge Analytics, Data Center Workloads Transceiver Leadership
  • 134. Well Suited for Artificial Intelligence Applications 3,4 based on current estimates, see slide 180 for details 135 Complementary to: Flexibility for evolving workloads and multi- function integration AI+ usage model Only FPGA supporting hardened BFLOAT16 & FP16 Configurable DSP • FP32 • BFLOAT16 • FP16 • INT8 through INT2 One APIOpenVINO™ toolkit 40TFLOPS 3 Upto FP16Performance Upto 92TOPS INT8Performance 4
  • 135. Intel® Agilex™ FPGA Tools for Developers 136  Single source, heterogenous programming environment  Support for common performance library APIs  FPGA support with Intel software development tools including Intel® VTune™ Amplifier & Intel® Advisor 5See slide 180 for details  Higher productivity:  30% improvement in compile times5  New productivity flows and usability features for faster design convergence  Higher efficiency: 15% improvement in memory utilization5 Hardware Developers Software Developers ONEAPI
  • 136. Intel® Agilex™ FPGA Family 137 F-Series For wide range of applications Up to 58G transceivers PCIe* Gen4 DDR4 I-SeriesFor high-performance processor interface and bandwidth-intensive applications Up to 112G transceivers PCIe* Gen5 DDR4 Quad-Core ARM* Cortex*-A53 M-Series For compute-intensive applications Up to 112G transceivers PCIe* Gen5 DDR4, DDR5 and Intel® Optane™ DC Persistent Memory support Quad-Core ARM* Cortex*-A53 Quad-Core ARM* Cortex*-A53 SoC Option HBM Option Coherent attach to Intel® Xeon® Scalable Processor option (CXL) Coherent attach to Intel® Xeon® Scalable Processor option (CXL) Intel® Quartus® Prime Design Software Support April 2019; First Device Availability Q3 2019
  • 137. Efficient Network Transformation First FPGA Providing Flexibility and Agility From 100Gbps to 1Tbps InfrastructureOffload OptimizedArchitecture vSwitch | vRouter | Security SmallFormFactor&LowPower Wide Range of Servers DatapathAcceleration VNFPerformanceOptimization Load Balancing | Data Integrity Network Translation SignificantImprovements Throughput | Jitter | Latency 28G-112G per channel PCIe Gen 4/5 138 1 1 Compared to Intel® Stratix® 10 FPGAs
  • 138. Converged Workload Acceleration for the Datacenter First FPGA with Comprehensive Memory Support and Coherent Attach to Intel® Xeon® Scalable Processors Memory Coherent (CXL) infrastructureacceleration Network | Security | Remote Memory Access Applicationacceleration AI | Search | Video Transcode | Database 40 TFLOPs of DSP Performance1 StorageAcceleration Compression | Decompression | Encryption Memory Hierarchy ManagementDDR5 SSD 139 1 with FP16 configuration, based on current estimates, see slide 180 for details
  • 139. Radio Baseband Agility & Flexibility for All Stages of 5G Implementation + FPGAFlexibility High Flexibility | Short Time-to-Market RapidIntel®eASIC™Deviceoptimization Power & Cost Optimization FullCustomASICOPtimization Best Power1 | Best Performance1 | Best Cost1 Application-SpecificTileOptions Data Converter | Vector Engine | Custom Compute Data Converter DFE Beamforming Fronthaul Only Intel® Provides All of These Customization Options CustomLogicContinuum FEC (Turbo/LDPC) uRLLC Fronthaul 140 1 Compared to the other customization options shown
  • 140. Intel® Agilex™ FPGAs for the Data-Centric World Any-to-Any integration enables Intel® FPGAs with application-specific optimization and customization, delivering new levels of flexibility & agility Intel® Agilex™ FPGAs for transformative applications in edge computing, embedded, networking (5G/NFV) and data centers First FPGAs leveraging Intel’s unmatched innovation: 10nm process, 3D integration, Intel® Xeon® Scalable Processor memory coherency (CXL), 112G XCVRs, PCIe* Gen 5, Intel® eASIC™ devices, One API, Intel® Optane™ DC Persistent Memory support 141
  • 141. 142 Agenda • Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI
  • 142. 143 © 2019 Intel Corporation oneAPISingle Programming Model to Deliver Cross-Architecture Performance
  • 143. 144 © 2019 Intel Corporation The future is a diverse mix of scalar, vector, matrix, and spatial architectures deployed in CPU, GPU, AI, FPGA and other accelerators DiverseWorkloadsrequire DIVERSEarchitectures Spatial FPGA Matrix AI Vector GPU Scalar CPU SVMS
  • 144. 145 © 2019 Intel Corporation Project oneAPI delivers a unified programming model to simplify development across diverse architectures Common developer experience across Scalar, Vector, Matrix and Spatial (SVMS) architecture Unified and simplified language and libraries for expressing parallelism Uncompromised native high-level language performance Support for CPU, GPU, AI and FPGA Based on industry standards and open specifications Optimized Applications Optimized Middleware / Frameworks oneAPI Language & Libraries Intel’sONEAPI CoreConcept FPGAAIGPUCPU Scalar Vector Matrix Spatial oneAPI Tools
  • 145. 146 © 2019 Intel Corporation OneAPIforcross-architectureperformance Some capabilities may differ per architecture. Optimized Applications Optimized Middleware & Frameworks Scalar Vector Matrix Spatial FPGAAIGPUCPU Direct Programming Data Parallel C++ API-Based Programming Threading Video Processing Math Analytics/ML DNN ML Comm DPC++ Library oneAPI Product Analysis & Debug Tools VTune™ Advisor Debugger Compatibility Tool
  • 146. © 2019 Intel Corporation Thankyou
  • 148. 149 © 2019 Intel Corporation software.intel.com/bigdl AnalyticsZoo Build E2E Analytics & AI Applications for Big Data at Scale Optimization Notice All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. *Other names and brands may be claimed as the property of others UnifiedBigDataAnalytics+AIOpenSourcePlatform Reference Use Cases  Anomaly detection, sentiment analysis, fraud detection, image generation, chatbot, sequence prediction, etc. Built-In Deep Learning Models  Image classification, object detection, text classification, recommendations, GANs, Sequence to Sequence, etc. Feature Engineering Feature transformations for  Image, text, 3D imaging, time series, speech, etc. High-Level Pipeline APIs  Distributed Tensorflow* & Keras* on Apache Spark/BigDL  Support for Autograd*, transfer learning, Spark DataFrame and ML Pipeline  Model serving API for model serving/inference pipelines Backends Apache Spark, TensorFlow*, BigDL, Python, etc. https://analytics-zoo.github.io/ Play the Video
  • 149. 150 © 2019 Intel Corporation software.intel.com/intel-distribution-for-python Easy, Out-of-the-box Access to High Performance Python*  Prebuilt, delivers faster, close-to- native code application performance for machine learning and scientific computing from device to cloud  Drop in replacement for your existing Python (no code changes required) Drive Performance with Multiple Optimization Techniques  Accelerated NumPy/SciPy/Scikit- Learn with Intel® Math Kernel Library  Data analytics with pyDAAL, enhanced thread scheduling with Threading Building Blocks, Jupyter* Notebook interface, Numba, Cython  Scale easily with optimized MPI4Py and Jupyter notebooks Faster Access to Latest Optimizations for Intel® Architecture  Distribution and individual optimized packages available through conda* and Anaconda Cloud*  Optimizations upstreamed back to main Python trunk For developers using the most popular and fastest growing programming language for AI All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Inteldistributionforpython* Advancing Python Performance Closer to Native Speeds Optimization Notice
  • 150. 151 © 2019 Intel Corporation Building blocks for all data analytics stages, including data preparation, data mining & machine learning Open Source, Apache* 2.0 License Common Python*, Java*, C++ APIs across all Intel hardware Optimizes data ingestion & algorithmic compute together for highest performance Supports offline, streaming & distributed usage models for a range of application needs Flexible interfaces to leading big data platforms including Spark* Split analytics workloads between edge devices & cloud to optimize overall application throughput Pre-processing Transformation Analysis Modeling Decision MakingValidation All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Intel®DataAnalyticsAccelerationLibrary(Intel®DAAL) High Performance Machine Learning & Data Analytics Library Optimization Notice
  • 151. 152 © 2019 Intel Corporation Distribution Details  Open Source  Apache* 2.0 License  Common DNN APIs across all Intel hardware.  Rapid release cycles, iterated with the DL community, to best support industry framework integration.  Highly vectorized & threaded for maximal performance, based on the popular Intel® Math Kernel Library. For developers of deep learning frameworks featuring optimized performance on Intel hardware github.com/01org/mkl-dnn Direct 2D Convolution Rectified linear unit neuron activation (ReLU) Maximum pooling Inner product Local response normalization (LRN) Examples: Intel®MathKerneLfordeepneuralNetworks(intel®mkl-DNN) Accelerate Performance of Deep Learning Models All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Optimization Notice
  • 152. 153 © 2019 Intel Corporation Work in progress github.com/NervanaSystems/ngraph *Other names and brands may be claimed as the property of others. All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. nGraph More… Single API for multiple frameworks & backends Lotus NVIDIA* GPU CPU NNP VPU Processor Graphics GPU FPGA Deep learning training & inference More… Intel®nGRAPH™Compiler Open-source C++ library, compiler & runtime for deep learning enabling flexibility to run models across a variety of frameworks & hardware Optimization Notice
  • 153. 154 © 2019 Intel Corporation Nauta Open-Source Distributed Deep Learning Platform for Kubernetes* All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. * Other names and brands may be claimed as the property of others. Batch inference Inference deployment Export to edge devices Multi-user collaboration Interactive sessions Template functionality Fast training Batch training Experiment tracking Multi-node distribution Analytics & visualization using TensorBoard* build train evaluate github.com/intelAI/NautaOptimization Notice
  • 155. 156 © 2019 Intel Corporation UnsupervisedlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning Machinelearning(Clustering) An ‘Unsupervised Learning’ Example K-Means Revenue Purchasing Power Revenue Purchasing Power Choose the right AI approach for your challenge
  • 156. 157 © 2019 Intel Corporation SupervisedlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning TrainingInference Model Weights Forward “Bicycle”? ?????? Lots of tagged data Forward Backward “Strawberry” “Bicycle”? Error Human Bicycle Strawberry Deeplearning(imagerecognition) A ‘Supervised Learning’ Example Choose the right AI approach for your challenge
  • 157. 158 © 2019 Intel Corporation ReinforcementlearningexampleMachinelearning Regression Classification Clustering Decision Trees Data Generation DeepLearning Image Processing Speech Processing Natural Language Processing Recommender Systems Adversarial Networks Reinforcement Learning Deeplearning(Reinforcementlearning) A ‘Deep Reinforcement Learning’ Example Feedback Action EnvironmentAgent Choose the right AI approach for your challenge
  • 158. 159 Open-source development environments Specific neural network implementations Library Optimized primitive functions for AI Intel® Distribution for Python Pandas NumPy Scikit-Learn Spark MlLib Mahout MKL-DNN DAAL DeeplearningGlossary Learn more about “what is AI” at software.intel.com/ai/course Framework Topology container Pre-configured AI environments ready to depoy
  • 159. 160 © 2019 Intel Corporation Machine Learning How do you engineer the best features? 𝑁 × 𝑁 Arjun NEURAL NETWORK 𝒇 𝟏, 𝒇 𝟐, … , 𝒇 𝑲 Roundness of face Dist between eyes Nose width Eye socket depth Cheek bone structure Jaw line length …etc. CLASSIFIER ALGORITHM SVM Random Forest Naïve Bayes Decision Trees Logistic Regression Ensemble methods 𝑁 × 𝑁 Arjun Deep Learning How do you guide the model to find the best features? Machinevs.Deeplearning
  • 160. 161 © 2019 Intel Corporation Whichapproachisright? Challenge Bestapproach Approach Answer How many widgets should we manufacture? Analyze historical supply/demand Analytics/ Business Intelligence 10,000 What will our yield be? Algorithm that correlates many variables to yield Statistical/ Machine Learning At current conditions, yield will be at 90% with 10% loss expected Which widgets have visual defects? Algorithm that learns to identify defects in images Deep Learning Widget 1003, Widget 1094 . . . A large manufacturer uses data to improve their operations, with each challenge using a different approach to deliver maximum business value at the lowest possible cost Learn Morein theNext slides
  • 161. 162 © 2019 Intel Corporation Aicl serlook Machinelearning DeepLearning reasoning Algorithms designed to deliver better insight with more data Neural networks used to infer meaning from large dense datasets Hybrid of analytics & AI techniques designed to find meaning in diverse datasets Regression (Linear/Logistic) Classification (Support Vector Machines/SVM, Naïve Bayes) Clustering (Hierarchical, Bayesian, K- Means, DBSCAN) Decision Trees (RandomForest) Extrapolation (Hidden Markov Models/HMM) More… Image Recognition (Convolutional Neural Networks/CNN, Single-Shot Detector/SSD) Speech Recognition (Recurrent Neural Network/RNN) Natural Language Processing (Long-Short Term Memory/LSTM) Data Generation (Generative Adversarial Networks/GAN) Recommender System (Multi-Layer Perceptron/MLP) Time-Series Analysis (LSTM, RNN) Reinforcement Learning (CNN, RNN) More… Associative Memory (Intel® Saffron AI memory base)  See also: machine & deep learning techniques More…
  • 162. 163 © 2019 Intel Corporation Lots of labeled data! Training Inference Forward Backward Model weights Forward “Bicycle”? “Strawberry” “Bicycle”? Error Human Bicycle Strawberry ?????? DeeplearningBasics Data set size Accuracy Didyouknow? Training with a large data set AND deep (many layered) neural network often leads to the highest accuracy inference
  • 163. 164 © 2019 Intel Corporation Framework Open-source software environments that facilitate deep learning model development & deployment through built-in components and the ability to customize code Topology Wide variety of algorithms modeled loosely after the human brain that use neural networks to recognize complex patterns in data that are otherwise difficult to reverse engineer Library Hardware-optimized mathematical and other primitive functions that are commonly used in machine & deep learning algorithms, topologies & frameworks Intel® Distribution for Python Pandas NumPy Scikit-Learn Spark MlLib Mahout MKL-DNN DAAL Deeplearningglossary Translating common deep learning terminology
  • 164. 165 © 2019 Intel Corporation Image Recognition Object Detection Image Segmentation Language Translation Text to Speech Recommendation System Deeplearningusages&Keytopologies Resnet-50 Inception V3 MobileNet SqueezeNet R-FCN Faster-RCNN Yolo V2 SSD-VGG16, SSD-MobileNet Mask R-CNN GNMT Wavenet Wide & Deep, NCF There are many deep learning usages and topologies for each
  • 165. 166 © 2019 Intel Corporation 0% 8% 15% 23% 30% Human 2010 Present Imagerecognition 0% 8% 15% 23% 30% 2000 Present SpeechrecognitionError Error Human 97% person 99% “play song” Source: ILSVRC ImageNet winning entry classification error rate each year 2010-2016 (Left), https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/ (Right) Deeplearningbreakthroughs Machines able to meet or exceed human image & speech recognition Tumor Detection Document Sorting Oil&Gas search Voice Assistant Defect detection Genome sequencing e.g.
  • 166. 167 © 2019 Intel Corporation All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Deep Learning Training Inference AI HARDWARE Mainstream intensive Multi-purpose to purpose-built AI compute from cloud to device Most other Onesizedoesnotfitall
  • 167. 168 © 2019 Intel Corporation Large scale data centers such as public cloud or comms service providers, gov’t and academia, large enterprise IT User-touch end point devices with lower power requirements such as laptops, tablets, smart home devices, drones Small scale data centers, small business IT infrastructure, to few on-premise server racks and workstations All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Varies to <1ms <5ms <10-40ms ~100ms DatacenterEdgeEndpoint HARDWARE Multi-purpose to purpose-built AI compute from device to cloud Aiisexpanding
  • 168. 169 © 2019 Intel Corporation 1GNA=Gaussian Neural Accelerator All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Images are examples of intended applications but not an exhaustive list. IOT SENSORS (Security, home, retail, industrial…) Display, Video, AR/VR, Gestures, Speech DESKTOP & MOBILITY Vision & Inference Speech SELF-DRIVING VEHICLES Autonomous Driving SERVERS, APPLIANCES & GATEWAYS Latency- Bound Inference Basic Inference, Media & Vision Most Use Cases SERVERS & APPLIANCES DatacenterEdgeEndpoint Flexible & Memory Bandwidth-Bound Use Cases Onesizedoesnotfitall HARDWARE Multi-purpose to purpose-built AI compute from device to cloud Varies to <1ms <5ms <10-40ms ~100ms Dedicated Media & Vision Inference Most Use Cases Most Intensive Use Cases NNP-T M.2 CardSOC Special Purpose Special Purpose
  • 169. 170 © 2019 Intel Corporation hardware Multi-purpose to purpose-built AI compute from cloud to device solutions Partner ecosystem to facilitate AI in finance, health, retail, industrial & more Intel analytics ecosystem to get your data ready Data Driving AI forward through R&D, investments and policy Future tools Software to accelerate development and deployment of real solutions Bring Your AI Vision to Life Using Our Complete Portfolio
  • 171. 172 Configurations for Continued Innovation Driving DL Gains on Xeon® (March 2019) 1x inference throughput improvement on Intel® Xeon® Platinum 8180 processor (July 2017) baseline : Tested by Intel as of July 11th 2017: Platform: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),and https://github.com/soumith/convnet-benchmarks/tree/master/caffe/imagenet_winners (ConvNet benchmarks; files were updated to use newer Caffe prototxt format but are functionally equivalent). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 5.7x inference throughput improvement on Intel® Xeon® Platinum 8180 processor (December 2018) with continued optimizations : Tested by Intel as of November 11th 2018 :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux-7.3.1611-Core, kernel: 3.10.0-862.3.3.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimization for caffe version: 551a53d63a6183c233abaa1a19458a25b672ad41 Topology::ResNet_50_v1 BIOS:SE5C620.86B.00.01.0014.070920180847 MKLDNN: 4e333787e0d66a1dca1218e99a891d493dbc8ef1 instances: 2 instances socket:2 (Results on Intel® Xeon® Scalable Processor were measured running multiple instances of the framework. Methodology described here: https://software.intel.com/en-us/articles/boosting-deep-learning-training-inference-performance-on-xeon-and-xeon-phi) Synthetic data. Datatype: INT8 Batchsize=64 vs Tested by Intel as of July 11th 2017:2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency- set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 14x inference throughput improvement on Intel® Xeon® Platinum 8280 processor with Intel® DL Boost: Tested by Intel as of 2/20/2019. 2 socket Intel® Xeon® Platinum 8280 Processor, 28 cores HT On Turbo ON Total Memory 384 GB (12 slots/ 32GB/ 2933 MHz), BIOS: SE5C620.86B.0D.01.0271.120720180605 (ucode: 0x200004d), Ubuntu 18.04.1 LTS, kernel 4.15.0-45-generic, SSD 1x sda INTEL SSDSC2BA80 SSD 745.2GB, nvme1n1 INTEL SSDPE2KX040T7 SSD 3.7TB, Deep Learning Framework: Intel® Optimization for Caffe version: 1.1.3 (commit hash: 7010334f159da247db3fe3a9d96a3116ca06b09a) , ICC version 18.0.1, MKL DNN version: v0.17 (commit hash: 830a10059a018cd2634d94195140cf2d8790a75a, model: https://github.com/intel/caffe/blob/master/models/intel_optimized_models/int8/resnet50_int8_full_conv.prototxt, BS=64, syntheticData, 4 instance/2 socket, Datatype: INT8 vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time -- forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 30x inference throughput improvement on Intel® Xeon® Platinum 9282 processor with Intel® DL Boost : Tested by Intel as of 2/26/2019. Platform: Dragon rock 2 socket Intel® Xeon® Platinum 9282(56 cores per socket), HT ON, turbo ON, Total Memory 768 GB (24 slots/ 32 GB/ 2933 MHz), BIOS:SE5C620.86B.0D.01.0241.112020180249, Centos 7 Kernel 3.10.0-957.5.1.el7.x86_64, Deep Learning Framework: Intel® Optimization for Caffe version: https://github.com/intel/caffe d554cbf1, ICC 2019.2.187, MKL DNN version: v0.17 (commit hash: 830a10059a018cd2634d94195140cf2d8790a75a), model: https://github.com/intel/caffe/blob/master/models/intel_optimized_models/int8/resnet50_int8_full_conv.prototxt, BS=64, No datalayer syntheticData:3x224x224, 56 instance/2 socket, Datatype: INT8 vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0- 514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, synthetic dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“.
  • 172. 173 © 2019 Intel Corporation This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Intel inside, the Intel inside logo, Xeon, the Xeon logo, Xeon Phi, the Xeon Phi logo, Core, the Core logo, Atom, the Atom logo, Movidius, the Movidius logo, Stratix, the Stratix logo, Arria, the Arria logo, Myriad, Nervana and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit © 2019 Intel Corporation. Configurationdetails
  • 173. 174 © 2019 Intel Corporation Configuration: AI Performance – Software + Hardware INFERENCE using FP32 Batch Size Caffe GoogleNet v1 128 AlexNet 256. The benchmark results may need to be revised as additional testing is conducted. The results depend on the specific platform configurations and workloads utilized in the testing, and may not be applicable to any particular user's components, computer system or workloads. The results are not necessarily representative of other benchmarks and other benchmark results may show greater or lesser impact from mitigations. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance Source: Intel measured as of June 2017 Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Configurations for Inference throughput Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611-Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology:GoogleNet v1 BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1449.9 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320191901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configuration for training throughput: Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611-Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc765b1162ab9940d Topology:alexnet BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1257 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320191901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configurationdetails(Cont’d)
  • 174. 175 © 2019 Intel Corporation Configurationdetails(Cont’d) INFERENCE using FP32 Batch Size Caffe GoogleNet v1 128 AlexNet 256. The benchmark results may need to be revised as additional testing is conducted. The results depend on the specific platform configurations and workloads utilized in the testing, and may not be applicable to any particular user's components, computer system or workloads. The results are not necessarily representative of other benchmarks and other benchmark results may show greater or lesser impact from mitigations. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance Source: Intel measured as of June 2017 Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Configurations for Inference throughput Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611- Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology:GoogleNet v1 BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1449.9 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320181901, CentOS Linux-7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594 Configuration for training throughput: Platform :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.28GB (12slots / 32 GB / 2666 MHz),4 instances of the framework, CentOS Linux-7.3.1611- Core , SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework caffe version: a3d5b022fe026e9092fc7abc765b1162ab9940d Topology:alexnet BIOS:SE5C620.86B.00.01.0004.071220170215 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 1257 imgs/sec vs Platform: 2S Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 64GB DDR4-2133 ECC RAM. BIOS: SE5C610.86B.01.01.0024.021320181901, CentOS Linux- 7.5.1804(Core) kernel 3.10.0-862.3.2.el7.x86_64, SSD sdb INTEL SSDSC2BW24 SSD 223.6GB. Framework BVLC-Caffe: https://github.com/BVLC/caffe, Inference & Training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. BVLC Caffe (http://github.com/BVLC/caffe), revision 2a1c552b66f026c7508d390b526f2495ed3be594
  • 175. 176 © 2019 Intel Corporation Configurationdetails(Cont’d) 1.4x training throughput improvement in August 2018: Tested by Intel as of measured August 2nd 2018. Processor: 2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux- 7.3.1611-Core kernel 3.10.0-693.11.6.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimizations for caffe version:a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology::resnet_50 BIOS:SE5C620.86B.00.01.0013.030920180427 MKLDNN: version: 464c268e544bae26f9b85a2acb9122c766a4c396 NoDataLayer. Measured: 123 imgs/sec vs Intel tested July 11th 2017 Platform: Platform: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (GoogLeNet, AlexNet, and ResNet-50), https://github.com/intel/caffe/tree/master/models/default_vgg_19 (VGG-19), and https://github.com/soumith/convnet-benchmarks/tree/master/caffe/imagenet_winners (ConvNet benchmarks; files were updated to use newer Caffe prototxt format but are functionally equivalent). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 5.4x inference throughput improvement in August 2018: Tested by Intel as of measured July 26th 2018 :2 socket Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz / 28 cores HT ON , Turbo ON Total Memory 376.46GB (12slots / 32 GB / 2666 MHz). CentOS Linux-7.3.1611-Core, kernel: 3.10.0-862.3.3.el7.x86_64, SSD sda RS3WC080 HDD 744.1GB,sdb RS3WC080 HDD 1.5TB,sdc RS3WC080 HDD 5.5TB , Deep Learning Framework Intel® Optimized caffe version:a3d5b022fe026e9092fc7abc7654b1162ab9940d Topology::resnet_50_v1 BIOS:SE5C620.86B.00.01.0013.030920180427 MKLDNN: version:464c268e544bae26f9b85a2acb9122c766a4c396 instances: 2 instances socket:2 (Results on Intel® Xeon® Scalable Processor were measured running multiple instances of the framework. Methodology described here: https://software.intel.com/en-us/articles/boosting-deep-learning- training-inference-performance-on-xeon-and-xeon-phi) NoDataLayer. Datatype: INT8 Batchsize=64 Measured: 1233.39 imgs/sec vs Tested by Intel as of July 11th 2017:2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50). Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“. 11X inference thoughput improvement with CascadeLake: Future Intel Xeon Scalable processor (codename Cascade Lake) results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance vs Tested by Intel as of July 11th 2017: 2S Intel® Xeon® Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance. Caffe: (http://github.com/intel/caffe/), revision f96b759f71b2281835f690af267158b82b150b5c. Inference measured with “caffe time --forward_only” command, training measured with “caffe time” command. For “ConvNet” topologies, dummy dataset was used. For other topologies, data was stored on local storage and cached in memory before training. Topology specs from https://github.com/intel/caffe/tree/master/models/intel_optimized_models (ResNet-50),. Intel C++ compiler ver. 17.0.2 20170213, Intel MKL small libraries version 2018.0.20170425. Caffe run with “numactl -l“.
  • 176. 177 © 2019 Intel Corporation Configurationdetails(cont’d) Intel® Arria® 10 – 1150 FPGA energy efficiency on Caffe/AlexNet up to 25 img/s/w with FP16 at 297MHz Vanilla AlexNet Classification Implementation as specified by http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf, Training Parameters taken from Caffe open-source Framework are 224x224x3 Input, 1000x1 Output, FP16 with Shared Block-Exponents, All compute layers (incl. Fully Connected) done on the FPGA except for Softmax, Arria 10-1150 FPGA, -1 Speed Grade on Altera PCIe DevKit with x72 DDR4 @ 1333 MHz, Power measured through on-board power monitor (FPGA POWER ONLY), ACDS 16.1 Internal Builds + OpenCL SDK 16.1 Internal Build, Compute machine is an HP Z620 Workstation, Xeon E5-1660 at 3.3 GHz with 32GB RAM. The Xeon is not used for compute.
  • 177. DETAILS ON INTEL® AGILEX™ FPGA PERFORMANCE, POWER AND SOFTWARE SUPPORT NUMBERS (1) Up to 40% Higher Performance Compared to Intel Stratix 10 FPGAs Derived from benchmarking an example design suite comparing maximum clock speed (Fmax) achieved in Intel Stratix 10 devices with the Fmax achieved in Intel Agilex devices, using Intel Quartus Prime Software. On average, designs running in the fastest speed grade of Intel Agilex FPGAs achieve a 40% improvement in Fmax compared to the same designs running in the most popular speed grade of Stratix 10 devices (-2 speed grade), tested February 2019. (2) Up to 40% Lower Total Power Compared to Intel Stratix 10 FPGAs Derived from benchmarking an example design suite comparing total power estimates of each design running in Intel Stratix 10 FPGAs compared to the total power consumed by the same design running in Intel Agilex FPGAs. Power estimates of Intel Stratix 10 FPGA designs are obtained from Intel Stratix 10 Early Power Estimator; power estimates for Intel Agilex FPGA designs are obtained using internal Intel analysis and architecture simulation and modeling, tested February 2019. (3) Up to 40 TFLOPs of DSP Performance (FP16 Configuration) Each Intel Agilex DSP block can perform two FP16 floating-point operations (FLOPs) per clock cycle. Total FLOPs for FP16 configuration is derived by multiplying 2x the maximum number of DSP blocks to be offered in a single Intel Agilex FPGA by the maximum clock frequency that will be specified for that block. (4) Up to 92 TOPs of DSP Performance (INT8 Configuration) Each Intel Agilex DSP block can perform eight INT8 operations per clock cycle. Total TOPs for INT8 configuration is derived by multiplying 8x the maximum number of DSP blocks to be offered in a single Intel Agilex FPGA by the maximum clock frequency that will be specified for that block. (5) 30% Improvement in Compile Times / 15% Improvement in Memory Utilization Comparison is made between Intel Quartus Prime Software 18.1 and Intel Quartus Prime 19.1. Derived from benchmarking an example design suite comparing compile times and memory utilization for designs in Intel Quartus Prime Software 18.1 with compile times and memory utilization for same designs in Intel Quartus Prime Software 19.1, tested February 2019. 178 Results have been estimated or simulated using internal Intel analysis, architecture simulation, and modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.
  • 178. LEGAL NOTICES & DISCLAIMERS This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Atom, Xeon and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2019 Intel Corporation. 179
  • 179. LEGAL NOTICES & DISCLAIMERS FTC Optimization Notice If you make any claim about the performance benefits that can be achieved by using Intel software developer tools (compilers or libraries), you must use the entire text on the same viewing plane (slide) as the claim. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804 180