Virtual Flink Forward 2020: Machine learning with Flink in Weibo - Yu Qian
0 likes241 views
The document provides an overview of Weibo's machine learning platform (WML) and its integration with Flink for online and offline data processing and model training. It outlines the architecture, capabilities, and various components of WML, including high-performance computing clusters and real-time feature engineering. The document also discusses next steps in leveraging Flink for multimedia processing and real-time data warehousing.
Virtual Flink Forward 2020: Machine learning with Flink in Weibo - Yu Qian
- 1. Qian Yu yuqian8@staff.weibo.com Weibo Machine Learning Team Machine Learning with Flink in Weibo
- 2. Agenda 1 About Weibo Weibo ML Platform (WML) overview Flink in WML 2 3 Next steps with Flink4
- 3. About Weibo • Most popular and largest social media platform in China • Based on social relationship, conduct, post, and share information to the world • Weibo built a social network that connecting people, interests and content 222M DAU 516M MAU
- 4. Machine learning Platform in Weibo (WML) —— Overview W M L Offline computing cluster Cluster High-performance computing cluster Online computing cluster Online Predicting WeiServing (self-developed) WeiPS (self-developed) Compute Platform WeiLearn (self-developed) Flink/StormHadoop/Spark TensorFlow Model Training LR DNN/RNNFM/FFMGBDT CF/MF … Model Zoo WeiClient WAIC UI Sample Pool Server Lake Schedule WeiBox (self-developed) K8sYarn Weiflow (self-developed) Application Feature Generating Online Model Training Sample Service Online Inference
- 5. Machine learning Platform in Weibo (WML) —— Development Two layer of DAGs design: WeiLearn Input Process Output Source Process Sink WeiFlow Task Task Task Task
- 6. 1.0 2.0 3.0 4.0 5.0 WeiLearn 6.0 Parameter scale Model count Model service QPS Iteration cycle monthly 50k 10+ 100M 10min 950k 190+ 100B WeiLearn weekly 100k 20+ 1B weekly 230k 30+ 2B daily 550k 80+ 10B + LR + GBDT + DNN Offline machine learning: LR Offline machine learning: LR GBDT GBDT+LR Offline machine learning: LR GBDT GBDT+LR Deep learning: Wide&Deep + Online FM + Online DNN Online deep learning: Enforce the expressive power of online machine learning model + FM Online machine learning： Onilne FM/FFM Offline machine learning： LR/GBDT/GBDT+LR FM/MF Deep learning： Wide&Deep DeepFM DSSM Offline machine learning: LR GBDT GBDT+LR FM MF Deep learning: Wide&Deep After successive iterations, Weibo machine learning platform (WML), can support over 100B parameters, 1m QPS， and iteration cycle around 10 minutes now. Machine learning Platform in Weibo (WML) —— CTR model iteration
- 7. Flink in WML —— Overview Business Interest Feed Relation Feed Video Recommendation Content monitoring Applications Multimedia feature generating platform Content Deduplication Data Synchronization Real-time feature engineering Sample service Platform Online Model Training Image Recommendation Computation WeiPig + WeiStream WeiLearn + WeiFlink Infrastructure Storm Cluster Flink Cluster (K8s, Yarn) MCQ Redis Kafka HDFS
- 8. Apply unified Flink APIs to both online and offline ML pipelines Online pipeline Offline pipeline Stream MCQ Stream processing Stream MCQ Online training Model Serving Recommendation System Stream MCQ HDFS Batch processing HDFS Offline training Model Serving Recommendation System Unified pipeline Stream MCQ HDFS Model Serving Recommendation System Batch/Stream processing Stream MCQ HDFS Model training Online/Offline mode Flink in WML —— Overview
- 9. Interaction log Click log Read log Expo log Sample Service Data filter & map Multi-stream joining Flink/timer/state Sample Pool Feature Processor Forward Predictor Gradient Computer Weips agent Model Training WeiLearn PSserver PSserver PSserver PSschudler ZooKeeper File/Queue checkpoint PSserver PSserver PSserver PSschudler checkpoint WeiPS Model Zoo Weips proxy Feature Processor Predictor Model Loader Model/Version Manager fbthrift rpc Model Predicting WeiServing Metric/AUC/ACC/MSE Model stablization Consistency check CICD / CTCD Blue green Deployment Model Collection Model Deployment Flow shift Model Evaluation Service discovery Sorting • Real-time logs：100+ • Click behavior delay：min -> second • User behaviors count：200+ Flink in WML —— Online model training • Sample alignment • Performance improvement • High availability & Fault Tolerant
- 10. Flink in WML —— Sample Service Sample Service Offline Data Real-time Logs Offline Data processing Online Data processing Data Processing Sample Pool Model Training Feature Generating Post, User, Relationship, Content, Multimedia (Picture, Video, Audio) General computing, Multi-stream joining, Deep learning …
- 11. input filter & map input filter & map … input filter & map Data Source 1 Data Source 2 … Data Source N Distributed by key Joining Window（State -> RocksDB/Gemini -> Checkpoint） After joining filter & map & append features WML Sample Pool Multiple stream joining: UDF: Flink in WML —— Sample Service
- 12. Joining time window & sample alignment Flink in WML —— Sample Service Interaction log Expo log Click log Read log Joining time window (10min) Sample stream State (key, value) RocksDB/Gemini timestamp HDFS timestamp + 10min Improvements: 1. Flush out sample immediately when joining finished 2. Sample compensation 3. RocksDB vs Gemini 4. Balance between success rate and time window size
- 13. Job status Management Metrics monitoring & Alert Job History Twinkle/VVP -> Flink Cluster Filter & Map UDFsSample ID Data source conf … Input Filter & Map UDFs Joining window … Process Sample ID Data sink conf … Output Interaction Layer Weiclient WAIC UI • Further encapsulation -> Weiflink + Weiplugin • Use Jenkins to do UDF cicd • Unified Steaming Data using sample IDs • Develop inner DAG logic inside WeiLearn • Grafana: Monitoring and alerting • Twinkle/VVP & WAIC UI to manage jobs • Review job history in HDFS • WAIC UI : Select your data and UDFs • Weiclient : CLT to submit job to different cluster Flink in WML —— Sample Service
- 14. Deep Learning & online inference with Flink: Multimedia online computing and inference platform Distributed model training service Multimedia online inference Multimedia features Model Zoo processing ü One click CICD ü Distributed training ü Support different types of model Offline training Online inference ü Success rate: 99.99%+ ü Delay: seconds ü Develop mode: ConfigurableOnline GPU(k8s) Offline GPU(k8s) Business & Applications Text stream Image stream Video stream Reconciliation system End-to-end monitoring Case tracing Service assurance Data center Sample Pool Data source Flink in WML —— Multimedia feature generating
- 15. Monitoring & Case tracing Flink in WML —— Multimedia feature generating High availability & fault tolerant 1. Automatically restart 2. Restore from checkpoints and snapshots
- 16. Next steps with Flink —— Real time data warehousing Real time data warehousing: unifying batch and stream schema & APIs Development Abstract APIs Offline Engine Offline Storage One single code base: SQL + UDF Everything is table: Unified table register APIs and meta data Online Engine Online Storage Mapred Flink HDFS Table schema Schema + format + connector
- 17. Deep learning with Flink: dealing with complex computation online Distributed model training service Multimedia online inference Multimedia features Model Zoo processing ü One click CICD ü Distributed training ü Support different types of model training inference ü Success rate: 99.99%+ ü Delay: seconds ü Develop mode: ConfigurableOnline GPU(k8s) TF on Flink Business & Applications Text stream Image stream Video stream Reconciliation system End-to-end monitoring Case tracing Service assurance Data center Sample Pool Data source Next steps with Flink —— DL in Flink
- 18. Qian Yu yuqian8@staff.weibo.com Weibo Machine Learning Team Thanks!