Vertex Perspectives | AI Optimized Chipsets | Part II
8 likes2,419 views
The document discusses the evolution of AI-optimized chipsets driven by the demands of deep learning and neural networks, highlighting the shift from general computing to performance optimization for specialized tasks. It covers various technologies like GPUs, FPGAs, and ASICs while emphasizing the need for innovations to support high-performance, memory-efficient processing for AI applications, particularly in edge computing. The future is expected to involve a significant increase in custom chip designs tailored for specific applications, alongside a possible shift in training and inference capabilities closer to end devices.
![And graph processing….
Scalar Processing
• Processes an operation per instruction
• CPUs run at clock speeds in the GHz range
• Might take a long time to execute large matrix operations
via a sequence of scalar operations
A1
A2
An
+ B1
B2
Bn
…
+
+
=
=
=
C1
C2
Cn
a[i] + b[i] = c[i]
for i = 1 to n
A1
A2
An
B1
B2
Bn
…
+ =
C1
C2
Cn
a[1:n] + b[1:n] = c[1:n]
.
.
.
.
.
.
.
.
.
.
.
Source: Spark 2.x - 2nd generation Tungsten Engine
Vector Processing
• Same operation performed concurrently across a large
number of data elements at the same time
• GPUs are effectively vector processors
Graph Processing
• Runs many computational processes (vertices)
• Calculates the effects these vertices on other points with
which they interact via lines (i.e. edges)
• Overall processing works on many vertices and points
simultaneously
• Low precision needed](https://image.slidesharecdn.com/vertexperspectives-aioptimizedchipsetspartii-180514061034/85/Vertex-Perspectives-AI-Optimized-Chipsets-Part-II-9-320.jpg)
Vertex Perspectives | AI Optimized Chipsets | Part II
- 1. AI-Optimized Chipsets May 2018 Part II: Computing Hardware
- 2. Computing Hardware Previously in Part I, we reviewed the ADAC loop and key factors driving innovation for AI-optimized chipsets. In this instalment, we explore how AI-led computing demands are powering these trends: • Deep learning is expected to drive training for neural networks requiring massive datasets for AI algorithm development • This in turn leads to a shift in the performance focus of computing from general application to neural nets, increasing demand for high performance computing • Deep learning is both computationally and memory intensive, necessitating enhancements in processor performance • Hence, the rise of startups adopting alternative, innovative approaches and how this is expected to pave the way for different types of AI-optimized chipsets
- 3. Source: Nvidia | Graphcore Deep learning is expected to drive training for neural networks Training Inference “Dog” “Cat” Untrained Neural Network Model “Cat” Trained Model Optimized for Performance • Training refers to neural network learning with significant data • AI algorithms are developed via training • Consumes significant computing power • Training loads can be divided into many concurrent tasks. This is ideal for the GPU’s double floating point precision and huge core counts • Training can also be conducted using FPGAs • Requires calculations with relatively high precision, often using 32-bit floating-point operations • Inference refers to the neutral network interpreting new data to generate accurate results • Typically conducted at the application or client end-point (i.e. edge), rather than on the server or cloud • Requires fewer hardware resources and depending on the application, can be performed using CPUs • This could be FPGAs, ASICs, Digital Signal Processors (DSPs) etc • Inference is expected to shift locally to mobile devices • Precision can be sacrificed in favor of greater speed or less power consumption
- 4. “The workloads are changing dramatically” for computing, as a result of machine learning…and whenever workloads have changed in computing, it has always created an opportunity for new kinds of computing.” Andrew Feldman CEO | Cerebras Source: Intel, NVIDIA, ImageNet, Ark Invest Management LLC Deep Learning Growth Drivers With massive datasets required for AI algorithm development and inference
- 5. Source: Deep Learning: An Artificial Intelligence Revolution by ARK Investment | Learning both Weights and Connections for Efficient Neural Networks by Song Han et al. | Icon made by Those Icons from www.flaticon.com Shifting the performance focus of computing from general application to neural nets
- 6. Source: Deep Learning: An Artificial Intelligence Revolution by ARK Investment | Learning both Weights and Connections for Efficient Neural Networks by Song Han et al. | Convolutional Neural Network by Mathworks Deep learning chipsets are designed to optimize performance, power and memory. • Algorithms tend to be highly parallel • Requires data splitting between different processing units • Connecting the pipeline in the most efficient manner is key • Significant transfer of data back and forth between memory • For instance, convolutional neural networks require convolution operations to be repeated throughout the pipeline and the number of operations can be extremely significant Example of a neural network with many convolutional layers Deep learning is both computationally and memory intensive
- 7. • A neural network takes input data, multiplies them with a weight matrix and applies an activation function • Multiplying matrices is often the most computationally intensive part of running a trained model Driving enhancements in processor performance via matrix multiplication…. The outputs of this matrix multiplication are then processed further by an activation function This sequence of multiplications and additions can be written as a matrix multiplication X1 X2 X3 Y1 Y2 Input Neurons = f (W11X1 + W12X2 + W13X3) = f (W21X1 + W22X2 + W23X3) Source: An in-depth look at Google’s first Tensor Processing Unit (TPU) by Kat Sato
- 8. Quantization in neural networks Quantization is a process of converting a range of input values into a smaller set of output values that closely approximates the original data • Reduces the cost of neural network predictions and memory usage ▪ Especially for mobile and embedded deployments • Neural network predictions may not require the precision of 16-bit or 32-bit floating point calculations ▪ For example, if it is raining - knowing whether it is light or heavy will suffice, there is no need to know how many droplets of water are falling per second • 8-bit integers can still be used to calculate a neural network prediction while maintaining the appropriate level of accuracy Source: An in-depth look at Google’s first Tensor Processing Unit (TPU) by Kat Sato Quantization in TensorFlow
- 9. And graph processing…. Scalar Processing • Processes an operation per instruction • CPUs run at clock speeds in the GHz range • Might take a long time to execute large matrix operations via a sequence of scalar operations A1 A2 An + B1 B2 Bn … + + = = = C1 C2 Cn a[i] + b[i] = c[i] for i = 1 to n A1 A2 An B1 B2 Bn … + = C1 C2 Cn a[1:n] + b[1:n] = c[1:n] . . . . . . . . . . . Source: Spark 2.x - 2nd generation Tungsten Engine Vector Processing • Same operation performed concurrently across a large number of data elements at the same time • GPUs are effectively vector processors Graph Processing • Runs many computational processes (vertices) • Calculates the effects these vertices on other points with which they interact via lines (i.e. edges) • Overall processing works on many vertices and points simultaneously • Low precision needed
- 10. Source: Cerebras Founder Feldman Contemplates the A.I. Chip Age by Barron’s | : Suffering Ceepie-Geepies! Do We Need a New Processor Architecture? By The Register | Startup Unveils Graph Processor at Hot Chips by EETimes • The key to a “graph” machine is software that captures the “intent” of the graph problems it needs to solve ▪ Processing in parallel instead of sequential • ThinCI’s Graph Streaming Processor (GSP) is designed to understand the complex data dependencies and flow • GSPs manage this entirely on the chip with: ▪ Minimal software intervention ▪ Extremely low memory bandwidth needs • Reduces or eliminates inter- processor communications and synchronizations • A microprocessor wastes a lot of effort with a sparse matrix multiplying by zero ▪ A sparse matrix is a matrix that has many elements that are zero • A new chip is needed to: ▪ Handle sparse matrix math ▪ Emphasize communications between inputs and outputs of calculations • Machine learning methods (e.g. convolutional neural networks) involve: ▪ Recursion ▪ Feedback ▪ Computations in one instance feed into computations elsewhere in the process • Cerebras’ solution: Simple on compute, on arithmetic and very intense on communications Creating new approaches that focus on graph processing and sparse matrix math, emphasizing communications between inputs and outputs of calculations • Graphcore’s Intelligence Processing Unit (IPU) has a structure which provides: ▪ Efficient massive compute parallelism ▪ Huge memory bandwidth • Both factors essential for delivering a significant step-up in graph processing power needed for machine intelligence • The graph is a highly-parallel execution plan for the IPU • Expected to increase the speed of machine learning workloads significantly: ▪ General: by 5x ▪ Specific: by 50 - 100x (e.g. autonomous vehicle workloads)
- 11. Source: Horizon Robotics | Hailo | Gyrfalcon Technology As well as AI processing in memory architectures and massively parallel compute capabilities • Deep learning processor for edge devices offering datacenter class performance in an embedded device • Dataflow approach, based on the structure of Neural Networks (NNs) • Distributed memory fabric, combined with purpose-made pipeline elements, allowing very low power memory access (without using batch processing) • Novel control scheme based on combination of hardware and software, reaching very low Joules/operation metrics with a high degree of flexibility • Extremely efficient computational elements, which can be variably applied according to need • Low overhead interconnect, allowing Near Memory Processing (NMP) and balancing changing requirements of memory, compute and control along the NN • Gyrfalcon’s Intelligent Matrix Processor: Lightspeeur® 2801S delivers a APiM (AI Processing in memory) architecture which features massively parallel compute capabilities • Its APiM architecture, uses memory as the AI processing unit. This eliminates the huge data movement that results in high power consumption • The architecture features true, on-chip parallelism, in situ computing, and eliminates memory bottlenecks. It has roughly 28,000 parallel computing cores and does not require external memory for AI inference • It runs in various open frameworks like TensorFlow, Caffe and others to complete deep learning training and inference tasks • The Brain Processing Unit (BPU) by Horizon Robotics is a heterogeneous Multiple Instruction, Multiple Data (MIMD) computation system • By heterogeneity, the BPU uses multiple kinds of Processing Units (PU) that were designed specifically for neural network inference. It gains performance or energy efficiency by adding dissimilar PUs, incorporating specialized processing capabilities to handle particular tasks • MIMD is a technique employed to achieve parallelism, with a number of PUs that function asynchronously and independently • At any one time, different PUs may be executing different instructions on different pieces of data • The first generation BPU employs a Gaussian architecture - allowing each vision task to be divided into 2 stages (i.e. attention and cognition) for optimal allocation of computations. This offers a parallel and fast filter of task-irrelevant information, on-demand cognition and edge learning to adjust models after deployment • This design enables the BPU to achieve a performance of up to 1TOPS at a low-power of 1.5W. It can process the 1080P video input at 30 frames per second, as well as detect and recognize up to 200 objects per frame
- 12. The choice of chipset depends on use - for training, inference, in the cloud, at the edge or a hybrid of both Cloud • Some cloud providers have been creating their own chips • Using alternative architectures to GPUs (e.g. FPGAs and ASICs) • Cloud-based systems can handle neural network training and inference Edge • Edge devices, from phones to drones, to focus mainly on inference, due to energy efficiency and low-latency computation considerations • Inference will be moved to edge devices for most applications (AR expected to be a key driver) • New entrants will have the best chance of success in the end-device market given its nascence • Chips for end-devices have power requirements as low as 1 watt • Devices market is too large and diverse for a single chip design to address, and customers will ultimately want custom designs
- 13. With industry players adopting different approaches Cloud Edge • Google TPUs are ASICs • The high non-recurring costs associated with designing the ASIC can be adsorbed due to Google’s large scale • Using TPUs across multiple operations help save costs, ranging from Street View to search queries • TPUs save more power than GPUs • Rolling out FPGAs in its own datacenter revamp ▪ Similar to ASICs ▪ But reprogrammable so that their algorithms can be updated • Smartphone System-on-Chips (SoCs) are likely to incorporate ASIC logic blocks • Creates opportunities for new IP licensing companies. (e.g. Cambricon has licensed its ASIC design to Huawei for its Kirin 970 SoC) • Specialized chips for mobile devices - an increasing trend with: ▪ Dedicated AI chips appearing in Apple’s iPhone X, Huawei’s Mate 10, and Google’s Pixel 2 ▪ ARM has reconfigured its chip design to optimize AI ▪ Qualcomm launched its own mobile AI chips Huawei Mate 10’s Kirin 970 Source: Google | Microsoft | Huawei
- 14. Source: Artificial Intelligence: 10 Trends to Watch in 2017 and Beyond by Tractica | Expect Deeper and Cheaper Machine Learning by IEEE Spectrum | MIT Technology in Review | Google Rattles the Tech World with a New AI Chip for All by Wired | Back to the Edge: AI Will Force Distributed Intelligence Everywhere by Azeem | When Moore’s Law Met AI – Artificial Intelligence and the Future of Computing by Azeem Latency and contextualization of locales are key drivers of edge computing Key Drivers of Edge Computing • Learning typically happens in the cloud • Devices do not do any learning from their environment or experience • Besides inference, it will also be essential to push training to the edge Latency • For many applications that delay will be unacceptable (e.g. the high latency risk of sending signal data to the cloud for self-driving prediction, even with 5G networks) Context • Devices will soon need to be powerful enough to learn at the edge of the network • Devices will be used in situ and those locales will be increasingly contextualized • The environment where the device is placed will be a key input to its operation. Allowing the network to learn from the experience of edge devices and the environment
- 15. Source: Artificial Intelligence: 10 Trends to Watch in 2017 and Beyond by Tractica | Expect Deeper and Cheaper Machine Learning by IEEE Spectrum | MIT Technology in Review | Google Rattles the Tech World with a New AI Chip for All by Wired | Back to the Edge: AI Will Force Distributed Intelligence Everywhere by Azeem | When Moore’s Law Met AI – Artificial Intelligence and the Going forward, we are likely to see Federated Learning - a multi-faceted infrastructure where learning happens on the edge of the network and in the cloud Federated Learning • Allows for smarter models, lower latency and power consumption, while availing differential privacy and personalized experiences • Allows the network to learn from the experience of many edge devices and their experiences of the environment • In a federated environment, edge devices could do some learning and efficiency send back deltas (or weights) to the cloud where a central model could be more efficiently updated, instead of sending their raw experiential data back to the cloud for analysis • Differential privacy also ensures that the aggregate data in a database capture significant patterns, while protecting individual privacy • Google designed its original TPU for execution. Its new cloud TPU offers a chip that handles training as well • Amazon and Microsoft offering GPU processing via cloud services, but they do not offer bespoke AI chips for both training and executing neural networks • Bitmain claims to have built 70% of all the computers on the Bitcoin network. It makes specialized chips to perform the critical hash functions involved in mining and trading bitcoins, and packages those chips into the top mining rig - the Antminer S9 • In 2017, Bitmain unveiled details its new AI chip, the Sophon BM1680 - specialized for both training and executing deep learning algorithms Google’s cloud TPU Bitmain’s Sophon BM1680
- 16. • AI applications increasingly demand higher performance and lower power but deep learning technology has primarily been a software play • Need for hardware acceleration was only recognized recently. Top global semiconductor companies and a number of startups ventured to develop specialized chipsets to address these demands • Current chipset market is dominated by GPUs and CPUs • An expanded role for other chipset types, including ASICs and FPGAs, is expected to exist in future • According to Tractica, deep learning chipset shipments are expected to increase from 863K units in 2016 to 41.2M units by 2025, with revenue growing from USD 513M to USD 12.2B at a CAGR of 42.2% Source: Tractica | Graphcore This is expected to drive increasing production of ASICs, FPGAs, and other emerging chipsets Deep Learning Chipset Unit Shipment by Type (Global, 2016-2025) “We believe that intelligence is the future of computing, and graph processing is the future of computers.” Nigel Toon CEO | Graphcore 863K units 41.2M units Source: Tractica
- 17. In Part I, we note that deep learning technology has primarily been a software play to date. The rise of new applications (e.g. autonomous driving) is expected to create substantial demand for computing. Existing processors were not originally designed for these new applications, hence the need to develop AI-optimized chipsets. Currently, most of the computing happens in the cloud. As AI applications become more ubiquitous, we expect a shift in inference and training closer to where it is needed, resulting in a relative increase in intelligence at the network edge. This is the end of Part II of a 4-part series of Vertex Perspectives that seeks to understand key factors driving innovation for AI-optimized chipsets, their industry landscape and development trajectory. In Part III, we assess the dominance of tech giants in the cloud, coupled with disruptive startups adopting cloud-first or edge-first approaches to AI-optimized chips. Most industry players are expected to focus on the cloud, with ASIC startups featuring prominently in the cloud and at the edge. Importantly, we look at what these opportunities mean for investors and entrepreneurs. Finally in Part IV, we look at other emerging technologies including neuromorphic chips and quantum computing systems, to explore their promise as alternative AI-optimized chipsets. Do let us know if you would like to subscribe to future Vertex Perspectives. Key Takeaways
- 18. Disclaimer This presentation has been compiled for informational purposes only. It does not constitute a recommendation to any party. The presentation relies on data and insights from a wide range of sources including public and private companies, market research firms, government agencies and industry professionals. We cite specific sources where information is public. The presentation is also informed by non-public information and insights. Information provided by third parties may not have been independently verified. Vertex Holdings believes such information to be reliable and adequately comprehensive but does not represent that such information is in all respects accurate or complete. Vertex Holdings shall not be held liable for any information provided. Any information or opinions provided in this report are as of the date of the report and Vertex Holdings is under no obligation to update the information or communicate that any updates have been made. About Vertex Ventures Vertex Ventures is a global network of operator-investors who manage portfolios in the U.S., China, Israel, India and Southeast Asia. Vertex teams combine firsthand experience in transformational technologies; on-the-ground knowledge in the world’s major innovation centers; and global context, connections and customers. About the Authors Emanuel TIMOR General Partner Vertex Ventures Israel emanuel@vertexventures.com XIA Zhi Jin Partner Vertex Ventures China xiazj@vertexventures.com Brian TOH Director Vertex Holdings btoh@vertexholdings.com Tracy JIN Director Vertex Holdings tjin@vertexholdings.com
- 19. 19