Apache Flink @ Alibaba - Seattle Apache Flink Meetup
- 2. About me • 2017.3 – present Alibaba Compute Platform • 2012 – 2017.3 Microsoft China • Spark as a Service, with a focus on streaming • Skype for Business data • 2000 – 2012 Microsoft US • SQL Server engine • Speech API & Speech Server 2
- 4. About Alibaba Data EBs Total PBs Everyday 100Ms Events/secTs Events/Day
- 5. Web Tier DB Tier MQ DataHub Data Pipeline HBase Dashboard Exactly-Once Lots of Events Sub second latency Why Streaming? Highly Available
- 6. Challenges of Data Infra at Alibaba Lots of JobsTons of Data Exactly-OnceComplex Logic Thousands of Machines Strict SLA Low LatencyHigh Throughput
- 7. Introducing Alibaba Blink + = Blink Apache Flink Alibaba’s Improvements Alibaba Blink
- 8. Blink numbers + = Blink Apache Flink Alibaba’s Improvements Alibaba Blink Unprecedented scale on 2017-11-11 472M events per second 10s of milliseconds latency Accurate results
- 9. Improvements in Blink Runtime Async IO Increment CP Process & Deployment Metric
- 10. Declarative Optimizable Understandable Why SQL? Stable Unify
- 12. SQL Improvements UDF/UDTF/UDAF Stream JOIN, etc. Retraction Window AGG DML: INSERT etc. DDL
- 15. Use Case — Real-time A/B Test (Analytics) Parser Parser Parser Filter Filter Filter Join UDF Agg Impression Click Transaction Druid Online Logs
- 16. Use Case — Search Index Build & Update HBase ClusterMysql Cluster Mysql IC2 Mysql IC3 Mysql IC1 Mysql UIC2 Mysql UIC3 Mysql UIC1 HBase IC HBase UIC HBase Result Engine search Sync Join Export
- 17. Use Case — Online Machine Learning Online Logs HDFS Feature Feature Compute Online Learning Model Export Engine online
- 18. Flink Advantages • Event time / processing time • Watermark, window, trigger • Temporal join, retraction, … Streaming semantics • Managed state • Exactly once • Distributed checkpoints • Queryable stable Fault tolerant stateful processing at scale • Process Functions, DataStream / DataSet, Table API / SQL • Streaming, batch, event driven apps, CEP, ML, graph Rich programming models and APIs
- 19. Blink Focuses • Single job 10K+ parallelism, 10s TB state size Scale • Efficient resource utilization (FLIP-6) • Runtime improvements • Metrics & monitoring Performance, costs and SLA • SQL • Platform – develop, debug, deploy, monitor, migration • Connectors Productivity
- 20. Blink Ecosystem in Alibaba Cluster Resource Management Search Storage StreamCompute Platform Blink Alibaba Apps Recommendation BI Lots more DataStream API Runtime Engine Ads DataSet API SQL & Table API Machine Learning Platform
- 21. Blink & Flink YARN (Resource Management) HDFS (Persistent Storage) Blink Runtime YARN App Master Resource Manager Job Manager Task Manager Tasks Rocksdb State Backend Web Monitor Flink Client Alibaba Data Lake Submit Job Launch AM Request TM Launch TM Metrics Apache Flink Alibaba Blink Alibaba Monitor SystemMetric Reporter Connectors Read/Write Checkpoint Incrementally Debug Task Scheduling Checkpoint Coordination
- 22. Examples of recent Blink runtime optimizations • Credit-based network stack • Dynamic load balancing • Improved check-pointing for large scale jobs • Some of these work has been contributed to Flink and will be released in flink-1.5
- 23. Some New Flink Features • Incremental Checkpoints • Fine grained recovery Flink 1.3 • Queryable states improvements • Table API & SQL enhancements Flink 1.4 • FLIP-6 • Network stack improvements • State replication • Eager State Declaration • State evolution Flink 1.5+
- 24. Evolution of Large State Handling & Recovery 24
- 25. G H C D Full Checkpoints 25 Checkpoint 1 Checkpoint 2 Checkpoint 3 I E A B C D A B C D A F C D E @t1 @t2 @t3 A F C D E G H C D I E
- 26. G H C D Incremental Checkpoints 26 Checkpoint 1 Checkpoint 2 Checkpoint 3 I E A B C D A B C D A F C D E E F G H I @t1 @t2 @t3
- 27. Incremental Checkpoints 27 Checkpoint 1 Checkpoint 2 Checkpoint 3 Checkpoint 4 C1 C3C1 C1 Chunk 1 Chunk 2 Chunk 3 Chunk 4 Storage C2 C4C3
- 28. Network Stack Improvements • Removal of redundant copy operations • Event driven network transfer • Removal of artificial latency source • Introduction of flow control • Better control of back pressure 28
- 29. State Replication • Decouple state from Tasks • Faster recovery in case of Task failure • Replicate state between Task Managers • Faster failure recovery in case of machine failure • High throughput queryable state 29