Generative AI at the edge.pdf
2 likes3,731 views
The document discusses Qualcomm's advancements in on-device generative AI, emphasizing the importance of full-stack AI optimizations across various models including diffusion and large language models. It highlights the efficiency and capabilities of their Snapdragon platform, with real-life applications in image generation, coding, and content creation. The presentation also showcases the benefits of hybrid AI, underlining Qualcomm's leadership in enabling generative AI to scale effectively on mobile devices.
![22
1: Perplexity is average over several test sets, including wikitext and c4 (subset)
Quantization-aware training with knowledge distillation
Reduces memory footprint while solving quantization challenges of
maintaining model accuracy and the lack of original training pipeline
<1
Point increase
in perplexity1
<1%
Decrease in
accuracy
Construct a
training loop
that can run
two models
on the same
input data
Teacher
Llama-2-Chat 7B
[FP16]
Student
Llama-2-Chat 7B
[INT4]
Dataset
true labels
Teacher logits
Student logits
Loss1: KL loss
(Teacher soft logits,
student soft logits)
Loss2: Cross entropy
loss (True labels,
student hard logits)
KD loss function combines the KL divergence
loss and hard-label based CE loss
Hard logits
(no temperature)
Soft logits
(temperature = 4)
Classes
Probability](https://image.slidesharecdn.com/generativeaiattheedge-231109012741-94fccb4d/85/Generative-AI-at-the-edge-pdf-22-320.jpg)
Generative AI at the edge.pdf
- 1. Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. November 8, 2023 @QCOMResearch Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Generative AI at the edge Joseph Soriaga Senior Director, Technology Qualcomm Technologies, Inc.
- 2. 2 Today’s agenda Why on-device generative AI is key Full-stack AI optimizations for diffusion models — Stable Diffusion Full-stack AI optimizations for large language models — Llama 2 Hybrid AI technologies and architectures Q&A 2
- 3. 3 AIMET is a product of Qualcomm Innovation Center, Inc. Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. LLM: Large language mode; LVM: Language vision model Leading machine learning research for on-device AI across the entire spectrum of topics Platform research Applied research Fundamental research AI research Generative AI G-CNN Self-supervised learning Reinforcement learning Causality and system-2 Model quantization, compression, and NAS HW-SW co-design Distillation of generative models Power management AI Model Efficiency Toolkit (AIMET) Deep learning for 3D/geometry Audio and video compression AI for wireless and RF sensing Energy-efficient perception AI for chip design On-device learning Bayesian distributed learning Graph and kernel optimization Federated learning Deep learning for graphics Video recognition and prediction Virtual AI assistant (LLM) Diffusion-based image generation (LVM) Voice UI
- 4. 4 Full-stack AI research & optimization Model, hardware, and software innovation across each layer to accelerate AI applications Early R&D and technology inventions essential to leading the ecosystem forward Transfer tech to commercial teams and influence future research with learnings from deployment Vision Identify a problem or need; establish requirements Ecosystem collaboration Collaborate and drive the ecosystem toward rapid commercialization at scale ~2-3 years Model quantization & optimization Develop tech & tools to quantize weights and modify architecture to run efficiently on hardware Software compilation Develop tech & tools to improve graph-level and kernel-level software compilation performance Proof of concept Target teams integrate models into final application for stable and intuitive demonstration Invention Invent new methods that set state-of-the-art
- 5. 5 World’s first on-device demo of Stable Diffusion running on an Android phone 1B+ parameter generative AI model runs efficiently and interactively Full-stack AI optimization to achieve sub-15 second latency for 20 inference steps Enhanced privacy, security, reliability, and cost with on-device processing Fast development enabled by Qualcomm AI Research and Qualcomm® AI Stack At MWC 2023
- 6. 6 Text generation (ChatGPT, Bard, Llama, etc.) Image generation (Stable Diffusion, MidJourney, etc.) Code generation (Codex, etc.) 6 Input prompts “Write a lullaby about cats and dogs to help a child fall asleep, include a golden shepherd” A great lullaby is created in seconds Real-life application of this platform • Communications, • Journalism, • Publishing, • Creative writing • Writing assistance Input prompts “Super cute fluffy cat warrior in armor” Real-life application of this platform • Advertisements • Published illustrations • Corporate visuals • Novel image generation Input prompts “Create code for a pool cleaning website with tab for cleaning, repairs, and testimonials” Real-life application of this platform • Web design • Software development • Coding • Technology A beautiful website is created in seconds What is generative AI? AI models that create new and original content like text, images, video, audio, or other data Generative AI, foundational models, and large language models are sometimes used interchangeably
- 7. 7 7 Infrastructure Cloud Hyperscaler datacenters, enterprise servers Assistant app (using foundation models) Vertical applications for consumers and knowledge workers to assist with various tasks such as writing content, coding, designing etc. Tooling/orchestration Developer tools and platforms for generative AI Foundation model Generic models General purpose LLM and others; exposed functionality the APIs Domain specific models Purpose-specific model development and/or training (enterprise, pro photo/video, simulated data) Assistant app (using own model) Vertical application implementation from model (e.g., LLM) development and training to user app Machine learning apps Labeling, training, model hub, optimization, etc. The generative AI ecosystem stack is allowing many apps to proliferate
- 8. 8 8 Generative AI will impact use cases across device categories Gen AI can help improve customer and employee experience in retail, such as providing recommendations for inventory and store layout “Suggest inventory and store layout changes to increase user satisfaction in the sports section” IoT Gen AI is transforming productivity by composing emails, creating presentations, and writing code PC Phone “Make me reservations for a weekend getaway at the place Bob recommended” Gen AI can become a true digital assistant XR Gen AI can help create immersive 3D virtual worlds based on simple prompts Automotive Gen AI can be used for ADAS/AD to help improve drive policy by predicting the trajectory and behavior of various agents “Make me a status presentation for my boss based on inputs from my team”
- 9. 9 9 Generative AI with diffusion models for robotics path planning 9 Stable Diffusion Denoising an image with a diffusion model Generating robot trajectories Instead of diffusing an image we diffuse a robot trajectory
- 10. 10 2024 2023 Assuming INT4 parameters On-device AI can support a variety of Gen AI models A broad number of Gen AI capabilities can run on device using models that range from 1 to 10 billion parameters We can run models with over 1 billion parameters on device today and anticipate this growing to over 10 billion parameters in the coming months 0.1 1 10 100 1000 Collaborative robotics Video understanding Image understanding Combinatorial optimization Mathematical reasoning Programming Dialog and NLP Text-to-image Model size (billions of parameters)
- 11. 11 Knowledge distillation Create a smaller model with fewer parameters Run faster inference on target deployment Maintain prediction quality close to the teacher Less training time Training a smaller “student” model to mimic a larger “teacher” model Teacher model Training data Student model Logits Knowledge distillation Logits Soft labels Match logits of the models to transfer teacher model representation and minimize distillation loss (KL divergence) Output Output Cross entropy loss Ground truth
- 12. 12 On-device intelligence is paramount Process data closest to the source, complement the cloud Privacy Reliability Low latency Cost Energy Personalization
- 13. 13 Output image VAE: Variational Auto Encoder; CLIP: Contrastive Language-Image Pre-Training What is diffusion? Reverse diffusion (subtract noise or denoise) Forward diffusion (add noise) Image generation Stable Diffusion architecture UNet is the biggest component model of Stable Diffusion Many steps, often 20 or more, are used for generating high-quality images Significant compute is required Input prompt Stable Diffusion (1B+ parameters) CLIP text encoder (123M parameters) Scheduler UNet (860M parameters) VAE decoder (50M parameters) Step Vase in Greek style with intricate patterns and design
- 14. 14 14 Original Stable Diffusion UNet Pruning & knowledge distillation More efficient architecture design through pruning and knowledge distillation Reducing UNet compute (FLOPs), model size, and peak memory usage Efficient UNet Convolutional block Attention block
- 15. 15 15 DDIM: Denoising Diffusion Implicit Models; MSE: Mean-squared error Step distillation for the DDIM scheduler Teach the student model to achieve in one step what the teacher achieves in multiple steps Teacher: 2 UNets Student: 1 UNet MSE loss
- 16. 16 16 FID↓ CLIP ↑ Inference latency Baseline (SD-1.5) Fast SD 17.14* 0.3037 5.05 seconds 20.08 0.3004 0.56 seconds 16 Fast Stable Diffusion Reduces UNet forward passes to less than 20 Step distillation Combines conditional and unconditional generation Guidance conditioning Reduces compute (FLOPs), model size, peak memory usage Efficient UNet Reparameterization from epsilon to velocity space for robust distillation *: These results are not directly comparable since baseline Stable Diffusion was trained with over 20x larger dataset than fast Stable Diffusion. SD: Stable Diffusion Our full-stack AI optimization of Stable Diffusion significantly improves latency while maintaining accuracy e-to-v Baseline Stable Diffusion speedup vs baseline Stable Diffusion 9x
- 17. 17 17 Fast Stable Diffusion Stable Diffusion V1.5 Similar image quality between our fast implementation and baseline model Panoramic view of mountains of Vestrahorn and perfect reflection in shallow water, soon after sunrise, Stokksnes, South Iceland, Polar Regions, natural lighting A hyper realistic photo of a beautiful cabin inside of a forest and full of trees and plants, with large aurora borealis in the sky Underwater world, plants, flowers, shells, creatures, high detail, sharp focus, 4k High quality colored pencil sketch portrait of an anthro furry fursona blue fox, handsome eyes, sketch doodles surrounding it, photo of notebook sketch Japanese garden at wild life river and mountain range, highly detailed, digital illustration
- 18. 18 World’s fastest AI text-to-image generative AI on a phone Takes less than 0.6 seconds for generating 512x512 images from text prompts Efficient UNet architecture, guidance conditioning, and step distillation Full-stack AI optimization to achieve this improvement
- 19. 19 LVM: Language vision model AI acceleration on the Qualcomm® Hexagon™ NPU of the Snapdragon® 8 Gen 3 Mobile Processor Full-stack AI optimization for LVM Runs completely on the device Significantly reduces runtime latency and power consumption Continuously improves the Qualcomm® AI Stack Qualcomm® AI Engine direct for improved performance and minimized memory spillage Knowledge distillation for pruning and removing of attention blocks, resulting in accurate model with improved performance and power efficiency Designing an efficient diffusion model through knowledge distillation for high accuracy
- 20. 20 20 LLMs are highly bandwidth limited rather than compute limited Illustration of autoregressive language modeling Single-token generation architecture of large languages models results in high memory bandwidth Recite the first law of robotics Recite the first law of robotics A robot may not injure a human being A robot may not injure a human robot may not injure a human A Huge bandwidth Each parameter of the model must be read to generate each token (e.g., read 7B parameters for Llama 7B to generate a single token) DRAM NPU DDR TCM Transformer layer 1 Transformer layer N Embeddings LM head LLM
- 21. 21 LLM quantization motivations LLM quantization challenges A 4x smaller model (i.e., FP16 -> INT4) Reduce memory bandwidth and storage Reduce latency Reduce power consumption Maintain accuracy of FP published models Post-training quantization (PTQ) may not be accurate enough for 4-bit The training pipeline (e.g., data or rewards) is not available for quantization aware training (QAT) Shrinking an LLM for increased performance while maintaining accuracy is challenging
- 22. 22 1: Perplexity is average over several test sets, including wikitext and c4 (subset) Quantization-aware training with knowledge distillation Reduces memory footprint while solving quantization challenges of maintaining model accuracy and the lack of original training pipeline <1 Point increase in perplexity1 <1% Decrease in accuracy Construct a training loop that can run two models on the same input data Teacher Llama-2-Chat 7B [FP16] Student Llama-2-Chat 7B [INT4] Dataset true labels Teacher logits Student logits Loss1: KL loss (Teacher soft logits, student soft logits) Loss2: Cross entropy loss (True labels, student hard logits) KD loss function combines the KL divergence loss and hard-label based CE loss Hard logits (no temperature) Soft logits (temperature = 4) Classes Probability
- 23. 23 Recite the first law of robotics Recite the first law of robotics A robot may not injure those human being A robot may not injure a human robot may not injure those human A Recite the first law of robotics Recite the first law of robotics A robot may not injure a human being A robot may not injure a human robot may not injure a human A A robot may A robot may not injure a not injure injure a a Llama 2 Llama 2 draft not not not Speculative decoding speeds up token rate by trading off compute for bandwidth A good draft model predicts with a high acceptance rate Draft model generates a few speculative tokens at a time Target model decides which to accept in one pass
- 24. Train a significantly smaller draft LLM for speculative decoding while maintaining enough accuracy is challenging Small draft model motivations Small draft model challenges 10x smaller draft model than target model Fast results Reduce memory bandwidth, storage, latency, and power consumption The training pipeline (e.g., data or rewards) is not available Cover multiple families, e.g., 7B and 13B models Match the distribution of the target model for higher acceptance rate
- 25. 25 Speculative decoding provides speedup with no accuracy loss Using our research techniques on Llama 2-7B Chat, we achieved Upto 20 tokens per second
- 26. 26 AI assistant enables basic chat and chat-assisted apps on device Orchestration across different tasks based on user query Powered by Llama 2 Chat (7B) Voice UI with Snapdragon Voice Activation and Whisper-Small (244M) Orchestrator Task classification Travel planner API Llama 2 Chat (7B param) User interface Voice/text/browser miniLM (∼33M param) Snapdragon Voice Activation & Whisper-Small (∼244M param) LM: Language model
- 27. 27
- 28. 28 World’s fastest Llama 2-7B on a phone Up to 20 tokens per second Demonstrating both chat and application interaction on device World’s first demonstration of speculative decoding running on a phone At Snapdragon Summit 2023
- 29. 29 QAT: Quantization-aware training; LLM: Large language model AI acceleration on the Qualcomm® Hexagon™ NPU of the Snapdragon® 8 Gen 3 Mobile Processor Full-stack AI optimization for LLM Runs completely on the device Significantly reduces runtime latency and power consumption Continuously improves the Qualcomm® AI Stack Qualcomm AI Engine direct for improved performance and minimized memory spillage QAT with knowledge distillation for accurate INT4 target LLM for improved performance and power efficiency Designing a good draft model for given target model through knowledge distillation for high acceptance and no accuracy loss
- 30. 30 30 1: Reuters 2023 Cloud economics will not allow generative AI to scale Cost per query1 Gen AI applications Coding assistant Copy creation Web search Personal assistant Image & video creation Text summarization Conversational chatbots … Billions of users (e.g. web search) Traditional Generative AI ~10x
- 31. 31 We are a leader in the realization of the hybrid AI Convergence of: Wireless connectivity Efficient computing Distributed AI Unlocking the data that will fuel our digital future and generative AI To scale, the center of gravity of AI processing is moving to the edge Central cloud Edge cloud On device Hybrid AI 31 Cost Energy Reliability, latency, & performance Privacy & security Personalization
- 32. 32 Device-centric hybrid AI On-device neural network or rules-based arbiter will decide where to run the model More complex models will use the cloud as needed It will be seamless to the user On-device neural network or rules-based arbiter Yes Is cloud needed? No The device acts as the anchor point
- 33. 33 33 33 ASR: Automatic speech recognition; CV: Computer vision; TTS: Text to speech Device-sensing hybrid AI The device acts as the eyes and ears Simple model ASR, CV, TTS Speech Image/video LLM Text Text answer TTS Speech Image/video LLM Improved prompt Text answer TTS Advanced model ASR, CV, TTS • Sensor and human-machine interface processing run on device • ASR, CV, TTS • LLM runs in the cloud • For advanced version, an on-device orchestrator uses on-device learning and personal data to provided improved prompts to the LLM
- 34. 34 34 34 Joint-processing hybrid AI Multi-token speculative decoding as an example • LLMs are memory-bound and produce a single token per inference, reading in all the weights • The smaller draft model runs on device, sequentially • The larger target model runs on the cloud, in parallel and speculatively • The good tokens are accepted • Results in net speedup in tokens per unit time and energy savings Predict draft model Four tokens sequentially computed on device Accept Average 2 to 3 are correct and accepted Verify target model Four tokens speculatively computed in parallel in cloud 1 2 3 4 1 2 X X 1 2 3 4
- 35. 35 On-device generative AI offers many benefits Generative AI is happening now on the device Our on-device AI leadership is enabling generative AI to scale Hybrid AI is the future 35
- 36. 36 www.qualcomm.com/news/onq Connect with us Questions www.youtube.com/c/QualcommResearch www.slideshare.net/qualcommwirelessevolution www.qualcomm.com/research/artificial-intelligence @QCOMResearch https://assets.qualcomm.com/mobile- computing-newsletter-sign-up.html
- 37. Follow us on: For more information, visit us at: qualcomm.com & qualcomm.com/blog Thank you Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2018-2023 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, Adreno, Hexagon, Kryo, FastConnect, and Qualcomm Spectra are trademarks or registered trademarks of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes our licensing business, QTL, and the vast majority of our patent portfolio. Qualcomm Technologies, Inc., a subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of our engineering, research and development functions, and substantially all of our products and services businesses, including our QCT semiconductor business. Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Qualcomm patented technologies are licensed by Qualcomm Incorporated.