Building an Efficient AI Training Platform at bilibili with Alluxio
1 like274 views
The document discusses the integration of Alluxio with Bilibili's Coeus cloud-native AI platform to enhance machine learning operations. It highlights various performance improvements, challenges faced in data management, and benefits of using Alluxio for large-scale training tasks, including specific use cases in audio language recognition and video portrait matting. Additionally, it outlines the advantages of serverless deployments and the need for efficient data handling in cloud environments.
Building an Efficient AI Training Platform at bilibili with Alluxio
- 1. bilibili Coeus ML Platform with Alluxio Lei Li, AI Platform Lead, bilibili Zifan Ni, Sr. SWE, bilibili
- 2. $whoami Lei Li • Worked in • Working in • Wide interest in cloud-native, kubenetes, MLOps, serverless, etc. • ✈ ♠ 🏃
- 3. $whoami Zifan Ni • Worked in • Working in • Lovely Puppy, Da Vinci. • 👨🍳 🍹 🎮
- 4. 2 3 4 1CONTENTS Best Practice Alluxio Meets Coeus bilibili Coeus Alluxio Perf
- 5. bilibili • Leading video community • MAU reached 230 million in last financial report • Aim to enrich everyday life of young generation in China
- 6. bilibili Coeus • cloud-native AI platform • supporting ü ADs ü CV ü NLP ü Speech ü e-commerce ü etc.
- 7. model dev model training model storage model serving VPA Hawkeye Cloud-Native Observability System of bilibili Alluxio + Fluid OSS HDFS Volcano bilibili Coeus
- 8. 2 3 4 1 CONTENTS Best Practice Alluxio Meets Coeus bilibili Coeus Alluxio Perf 2 1
- 9. Coeus without Alluxio OSS HDFS Scenario 1 Container Crash Training Container
- 10. Coeus without Alluxio Scenario 1 Container Crash Training Container OSS HDFS Download Data
- 11. Coeus without Alluxio Scenario 1 Container Crash Training Container OSS HDFS Download Data Has to be Downloaded Again…
- 12. Coeus without Alluxio Data is too huge to fit into one single machine, Users have to • (Re)write difficult APIs for different storage to consume data in pipe. • Handle retry, reconnection logic, etc. Scenario 2 Data is too huge
- 13. Alluxio meets Coeus Cache Wanted! • Distributed • Hides difficult details of reading data!
- 14. • Holds huge data in distributed workers • Fuse simplifies data reading (os.open()) Alluxio meets Coeus Cache Wanted! • Distributed • Hides difficult details of reading data! Alluxio + Fluid
- 15. 2 2 4 CONTENTS Best Practice Alluxio Meets Coeus bilibili Coeus Alluxio Perf 1 3
- 16. Serverless Deployment Pod0 1. Why Serverless GPU0 GPU1
- 17. Serverless Fuse Pod0 Alluxio Master0 GPU1 Alluxio Worker0 Alluxio Fuse Pod1 1. Why Serverless GPU0
- 18. Serverless Fuse Pod0 Alluxio Master0 Pending… GPU1 Alluxio Worker0 Alluxio Fuse Pod1 1. Why Serverless GPU0
- 19. Serverless Fuse 2. Serverless Solution Pod0 Alluxio Master0 Alluxio Worker0 Pod1 Alluxio Fuse GPU0 GPU1
- 20. Serverless Fuse Worker Worker Data Panel Control Panel Sidecar Control Panel Runtime Controller Pod Application Master Runtime Fuse Sidecar Fluid watch mount 3. Serverless Details
- 21. Alluxio Tuning Problem Solution 1.Master stop world gc 30s / 30m • Java: MaxGCPauseMillis 👇 , ParallelGCThreads👆 • Alluxio Master: Memory request👆 2. Fuse frequently throw I/O exception • alluxio.user.rpc.retry.max.duration👆 • alluxio.user.rpc.retry.base.sleep👆 3. Poor performance in reading bunches of small files • Upgrade Alluxio >= 2.6.2
- 22. 2 4 CONTENTS Best Practice Alluxio Meets Coeus bilibili Coeus Alluxio Perf 1 3
- 23. Framework Pytorch Network TDNN neural network model Total Epoches 20 Number of Files ~2.55Millions File Size ～300Ki/file; ~800Gi total GPU 4 * Nivida V100/16G GPU Alluxio 2 * 500 Gi worker Case 1 : Audio Language Recognition Model Alluxio Performance
- 24. Speed Comparison OSS S3Fuse Local SSD OSS Alluxio Cache Completion Time 242.56h 63.48h 64.17h Speed-up Ratio 1 3.82 3.78 Alluxio Performance User Case 1 : Audio Language Recognition Model
- 25. Case 2 : Video Portrait Matting Framework Pytorch Network TDNN neural network model Total Epoches 50 Number of Files ~20Millions File Size ~100K/file; ~2Ti in total GPU 4 * Nividia V100/32G Alluxio 4 * 600 Gi worker Alluxio Performance
- 26. • not enough disks to support large- scale training in local storage • S3Fuse: bad performance and availability with large metadata • each epoch completed stably in ~18 hours • quality of trained model significantly improved • intersection of union(IoU) of portrait matting improved by about 2% Alluxio Performance W/O Alluxio With Alluxio Case 2 : Video Portrait Matting
- 27. Q&A Lei Li, lilei06@bilibili.com Zifan Ni, nizifan@bilibili.com Thanksto LuQiu@alluxio BaoyouChen,XinguiZeng@bilibili YangChe@Alibaba