How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
0 likes929 views
The document discusses how Roblox scaled the BERT model to handle over 1 billion daily requests on CPU, focusing on optimizing latency and throughput for real-time text classification. Key strategies included using smaller models like DistilBERT, optimizing input sizes, applying quantization, and caching results to reduce requests. The success of these optimizations led to a significant improvement in performance while maintaining a minimal impact on accuracy.
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
- 1. How We Scaled BERT To Serve 1+ Billion Daily Requests on CPU Quoc N. Le, Data Scientist, Roblox Kip Kaehler, Engineering Manager, Roblox
- 3. Deep Learning for Text Classification 4 ● Text Classification is a key capability on Roblox platform ● BERT is a deep learning model that has transformed the Natural Language Processing (NLP) landscape ● Performance (Precision/Recall Area Under Curve) of our text classifiers improved by 10 percentage points fine- tuning BERT versus classical machine learning
- 4. Beyond Accuracy: Latency and Throughput 5 Latency: Speed of Request Analogy -> How long it takes for a single person to cross a bridge We required latency under 20ms Throughput: Completed Requests Per Second Analogy -> How many people can cross a bridge in a period of time We required over 50k requests per sec We want a short, wide bridge to maximize both Latency and Throughput in a realtime environment
- 5. Our Deep Learning Tech Stack
- 6. GPU vs CPU (for our application) Higher Throughput for Real-Time Inference Higher Throughput for Model Training ~GPU (TESLA V100) 10x faster at processing training examples than CPU, due to efficiency in doing large batch matrix operations OR ~CPU (Intel Xeon Scalable Processor) 5x more throughput than GPU due to CPU-specific optimizations and spreading real- time inference requests across cores (with latency < 20ms) (on cost equivalent hardware in 2020)
- 7. Which Comes First? 8 Build an Accurate Model Make It Fast Choose a Known Fast Model Make It Accurate
- 8. Know Your Quoc Le’s Quoc N. Le Quoc V. Le Has over 85k citations as an AI researcher according to Google Scholar Once got kicked out of the Boomtown Casino in Reno for counting cards in blackjack
- 9. Our Scaling Playbook on CPU: Less Is More! 10 ❏ Smaller Model (Distillation) ❏ Smaller Inputs (Dynamic Inputs) ❏ Smaller Weights (Quantization) ❏ Smaller Number of Requests (Caching) ❏ Smaller Number of Threads per Core (Thread Tuning)
- 10. Where We Started Smaller Model Smaller Inputs Smaller Weights Benchmarks Run on Intel Xeon Scalable Processors Baseline BERT
- 11. Smaller Model (DistilBERT) Smaller Model Smaller Inputs Smaller Weights Benchmarks Run on Intel Xeon Scalable Processors Baseline BERT
- 12. Smaller Model (DistilBERT) Bert Base has 110m parameters DistilBert has 66m parameters Tradeoff: < 1% negative impact on “Accuracy” (PR AUC) Text Input .. .. .. .. .. Predict Layer Transformer Layer .. .. .. .. Transformer Layer Transformer Layer .. .. .. .. Transformer Layer Predict Layer Text Input Student Model (DistilBERT) Teacher Model (BERT) DistilBERT: https://arxiv.org/pdf/1910.01108.pdf .. Knowledge Distillation
- 13. Smaller Inputs (Dynamic Shapes) Smaller Model Smaller Inputs Smaller Weights Benchmarks Run on Intel Xeon Scalable Processors Baseline BERT
- 14. Smaller Inputs (Dynamic Shapes) Fixed Shape Inputs (Zero pad until all inputs have same shape) Dynamic Shape Inputs (Do not zero pad)
- 15. Smaller Weights (Quantization) Smaller Model Smaller Inputs Smaller Weights Benchmarks Run on Intel Xeon Scalable Processors Baseline BERT
- 16. Smaller Weights (Quantization) Image Credit: https://towardsdatascience.com/how-to- accelerate-and-compress-neural-networks-with- quantization-edfbbabb6af7 Tradeoff: < 1% negative impact on “Accuracy” (PR AUC) “Dynamic Quantization” One-liner
- 18. Smaller Number of Requests to Model (Caching) Image Credit: https://peltarion.com/blog/data- science/illustration-3d-bert Text Classification Service Cache DistilBERT Model 1. Retrieve text classification result from cache (we’re done if it’s there) 2. Else call deep learning model for result 3. Add result to cache, then return result to service
- 19. Smaller Number Threads Per Core (Thread Tuning)
- 20. Our Scaling Playbook on CPU: Less Is More! 21 ✓ Smaller Model (Distillation) ✓ Smaller Inputs (Dynamic Inputs) ✓ Smaller Weights (Quantization) ✓ Smaller Number of Requests (Caching) ✓ Smaller Number of Threads per Core (Thread Tuning)
- 21. 30x Improvement in Latency and Throughput on CPU Smaller Model Smaller Inputs Smaller Weights Benchmarks Run on Intel Xeon Scalable Processors Baseline BERT
- 22. Takeaways 23 ● For certain real-time deep learning applications, it is feasible/natural to super- scale inferences on CPU ● The key to scaling is making things smaller, as shown in this presentation ● Many optimizations that enabled scale are easy to implement (one-liners) ● Check out our blog for more details: https://robloxtechblog.com/how-we-scaled- bert-to-serve-1-billion-daily-requests-on-cpus- d99be090db26
- 23. Questions? Suggestions? 24 We are always looking to get more performance from our models. Please reach out to kkaehler@roblox.com PS We are always hiring 🤓
- 24. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.