Flink Forward San Francisco 2019: Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream - Stephan Ewen & Aljoscha Krettek
Download as PPTX, PDF2 likes1,234 views
The document discusses the rethinking of the Apache Flink stack and APIs to unify batch and streaming processing. It highlights the challenges of maintaining separate APIs and processing styles, as well as the benefits of creating a unified system to improve efficiency and ease of use. Future improvements include a new unified source interface and enhancements to existing components to better support both data processing paradigms.
Flink Forward San Francisco 2019: Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream - Stephan Ewen & Aljoscha Krettek
- 1. © 2019 Ververica Aljoscha Krettek – Software Engineer, Flink PMC Stephan Ewen – CTO, Flink PMC/Chair Towards Flink 2.0: Rethinking the Stack and APIs to unify Batch & Streaming
- 2. © 2019 Ververica2 This is joint work with many members of the Apache Flink community Timo, Dawid, Shaoxuan, Kurt, Guowei, Becket, Jincheng, Fabian, Till, Andrey, Gary, Piotr, Stefan, etc. And many others …
- 3. © 2019 Ververica3 Some of this presents work that is in progress in the Flink community. Other things are planned and/or have design documents. Some were discussed at one point or another on the mailing lists or in person. This represents our understaning of the current state, this is not a fixed roadmap, Flink is an open-source Apache project.
- 4. © 2019 Ververica Batch and Streaming
- 5. © 2019 Ververica5 Everything Is a Stream
- 6. © 2019 Ververica6 What changes faster? Data or Query? ad-hoc queries, data exploration, ML training and (hyper) parameter tuning continuous applications, data pipelines, standing queries, anomaly detection, ML evaluation, … Data changes slowly compared to fast changing queries Data changes fast application logic is long-lived “batch” “streaming”
- 7. © 2019 Ververica7 Latency vs. Completeness (for geeks) 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time 2016 Rogue One III.5 2017 Episode VIII
- 8. © 2019 Ververica8 Latency vs. Completeness (more formally) *from the excellent Streaming 101 by Tyler Akidau: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-
- 9. © 2019 Ververica9 Latency vs. Completeness Bounded/ Batch Unbounded/ Streaming Data is as complete as it gets within that Batch Job No fine latency control Trade of latency versus completeness Processing-time timers for latency control
- 10. © 2019 Ververica10 What does this mean for processing? older more recent files, message bus, or …
- 11. © 2019 Ververica11 Stream-style processing older more recent watermark unprocesse d
- 12. © 2019 Ververica12 Batch-style processing older more recent watermark unprocesse d
- 13. © 2019 Ververica13 Batch and Streaming Processing Styles S S S M M M R R R S S S S S S M M M R R R S S S more batch-y more stream-y running not running can do things one-by-one everything is always-on running
- 14. © 2019 Ververica14 So this is the reason why we have different APIs, different batch and stream processing systems? • Different requirements • Optimization potential for batch and streaming • Also: historic developments and slow- changing organizations 💡
- 15. © 2019 Ververica Current Stack
- 16. © 2019 Ververica16 Runtime Tasks / JobGraph / Network Interconnect / Fault Tolerance DataSet “Operators“ and Drivers / Operator Graph / Visitors DataStream StreamOperators / StreamTransformation Graph* / Monolithic Translators Table API / SQL Logical Nodes* / Different translation paths
- 17. © 2019 Ververica17 What could be improved? • Each API has its own internal graph representation code duplication • Multiple translation components between the different graphs code duplication – DataStream API has an intermediate graph structure: StreamTransformation StreamGraph JobGraph • Separate (incompatible) operator implementations – DataStream API has StreamOperator, DataSet API has Drivers two map operators, two flatMap operators – These are run by different lower-level Tasks – DataSet operators are optimized for different requirements than DataSet operators • Table API is translated to two distinct lower-level APIs two different translation stacks – ”project operator” for DataStream and for DataSet • Connectors for each API are separate a whole bunch of connectors all over the From a system design/code quality/architecture/development perspective
- 18. © 2019 Ververica18 What does this mean for users? • You have to decide between DataSet and DataStream when writing a job – Two (slightly) different APIs, with different capabilities – Different set of supported connectors: no Kafka DataSet connector, no HBase DataStream connector – Different performance characteristics – Different fault-tolerance behavior – Different scheduling logic • With Table API, you only have to learn one API – Still, the set of supported connectors depends on the underlying execution API – Feature set depends on whether there is an implementation for your underlying API • You cannot combine more batch-y with more stream-y sources/sinks • A “soft problem”: with two stacks of everything, less developer power will go into each one individual stack less features, worse performance, more bugs that are fixed slower
- 19. © 2019 Ververica Future Stack
- 20. © 2019 Ververica20 DataSet “Operators“ and Drivers / Operator Graph / Visitors DataStream StreamOperator / StreamTransformation Graph* / Monolithic Translators Batch is a subset of streaming! Can’t we just? ✅❌ Done! 🥳
- 21. © 2019 Ververica21 Unifying the Batch and Streaming APIs • DataStream API functionality is already a superset of DataSet API functionality • We need to introduce BoundedStream to harness optimization potential, semantics are clear from earlier: –No processing-time timers –Watermark “jumps” from –Infinity to +Infinity at end of processing • DataStream translation and runtime (operators) need to be enhanced to use the added optimization potential • Streaming execution is the generic case that always works, “batch” enables additional “optimization rules”: bounded operators, different scheduling we get feature parity automatically ✅ • Sources need to be unified as well see later
- 22. © 2019 Ververica22 A typical “unified” use case: Bootstrapping state * See for example Bootstrapping State in Flink by Gregory Fee https://sf-2018.flink- “stream ” source “batch” source Stateful operatio n batch-y partstream-y part • We have a streaming use case • We want to bootstrap the state of some operations from a historical source • First execute bounded parts of the graph, then start the rest
- 23. © 2019 Ververica23 Under-the-hood Changes • StreamTransformation/StreamGraph need to be beefed up to carry the additional information about boundedness • Translation, scheduling, deployment, memory management and network stack needs to take this into account Graph Representation / DAG Operator / Task • StreamOperator needs to support batch-style execution see next slide • Network stack must eventually support blocking inputs
- 24. © 2019 Ververica24 Selective Push Model Operator FLINK-11875 batch: pull-based operator (or Driver) streaming: push-based StreamOperator • StreamOperator needs additional API to tell the runtime which input to consume • Network stack/graph needs to be enhanced to deal with blocking inputs, 😱
- 25. © 2019 Ververica25 Current Source Interfaces InputFormat createInputSplits(): splits openSplit(split) assignInputSplit() nextRecord(): T closeCurrentSplit() SourceFunction run(OutputContext) close() batch streaming
- 26. © 2019 Ververica26 Batch InputFormat Processing TaskManager TaskManager TaskManager JobManager (1) request split (2) send split (3) process split • Splits are assigned to TaskManagers by the JobManager, which runs a copy of the InputFormat Flink knows about splits and can be clever about scheduling, be reactive • Splits can be processed in arbitrary order • Split processing pulls records from the InputFormat • InputFormat knows nothing about watermarks, timestamps, checkpointing bad for streaming
- 27. © 2019 Ververica27 Stream SourceFunction Processing • Source have a run-loop that they manage completely on their own • Sources have flexibility and can efficiently work with the source system: batch accesses, dealing with multiple topics from one consumer, threading model, etc… • Flink does not know what’s going on inside and can’t be clever about it • Sources have to implement their own per-partition watermarking, idleness tracking, what have you TaskManagerTaskManager TaskManager (1) do your thing
- 28. © 2019 Ververica28 A New (unified) Source Interface • This must support both batch and streaming use cases, allow Flink to be clever, be able to deal with event-time, watermarks, source idiosyncrasies, and enable snapshotting • This should enable new features: generic idleness detection, event-time alignment FLIP-27 Source createSplitEnumerator() createSplitReader SplitEnumerator discoverNewSplits() nextSplit() snapshotState() isDone() SplitReader addSplit() hasAvailable(): Future snapshotState() emitNext(Context): Status * FLINK-10886: Event-time alignment for sources; Jamie Grier (Lyft) contributed the first parts of this
- 29. © 2019 Ververica29 A New (unified) SourceInterface: Execution Style I TaskManager TaskManager TaskManager JobManager (1) request split (2) send split (3) process split • Splits are assigned to TaskManagers by the JobManager, which runs a copy of the SplitEnumerator Flink knows about splits and can be clever about scheduling, be reactive • Splits can be processed in arbitrary order • Split processing is driven by the TaskManager working with SplitReader • SplitReader emits watermarks but Flink deals with idleness, per-split watermarking
- 30. © 2019 Ververica30 A New (unified) SourceInterface: Execution Style II
- 31. © 2019 Ververica31 Table API / SQL • API: easy, it’s already unified ✅ • Translation and runtime (operators) need to be enhanced to use the added optimization potential but use the StreamOperator for both batch and streaming style execution • Streaming execution is the generic case that always works, “batch” enables additional “optimization rules”: bounded operators, different scheduling we get feature parity automatically ✅ • Sources will be unified from the unified source interface ✅ • This is already available in the Blink fork (by Alibaba), FLINK-11439 is the effort of getting that into Flink
- 32. © 2019 Ververica32 StreamTransformation DAG / StreamOperator DataStream “Physical“ Application API Table API / SQL Declarative API Runtime The Future Stack
- 34. © 2019 Ververica34 Summary • Semantics of unified processing are quite clear already • For some things, work is ongoing and there are design documents (FLIPs) • Some things are farther in the future • This project requires changes in all components/layers of Flink: –API, deployment, network stack, scheduling, fault tolerance • You can follow all of this on the public mailing lists and FLIPs! • The Flink tech stack is going to be quite nice! 😎
- 35. © 2019 Ververica www.ververica.com @VervericaDataaljoscha@ververica.com stephan@ververica.com
- 36. © 2019 Ververica Thank you! Questions?
Editor's Notes
- #7: Streaming Keep up with real time, some extra capacity for catch-up Receive data roughly in order as produced Latency is important Batch Fast forward through months/years of history Massively parallel unordered reads Throughput most important
- #12: Time in data stream must be quasi monotonous to produce time progress (watermarks) Always have close-to-latest incremental results Resource requirements change over time Recovery must catch up very fast
- #13: Order of time in data does not matter (parallel unordered reads) Bulk operations (2 phase hash/sort) Longer time for recovery (no low latency SLA) Resource requirements change fast throughout the execution of a single job
- #14: Understanding this difference will help later, when we discuss scheduling changes.
- #18: Possibly put these on separate slides, with fewer words. Or even some graphics.
- #19: Possibly put these on separate slides, with fewer words. Or even some graphics.
- #22: There are some quirks when you use DataStream for batch a groupReduce would be window with a GlobalWindow MapPartition would have to finalizing things in close() Joins would have to specify global window Of course, state requirements are bad for the naïve approach, i.e. large state, inefficient access patterns Joins and grouping can be a lot faster with specific algorithms Hash Join, Merge join, etc…
- #23: Recall the earlier processing-styles slide: batch wants step by step streaming is all at once This has been mentioned a lot. Lyft has given a talk about this at last FF
- #24: For example different window operator Different join implementations The scheduling stuff and networking would be a whole talk on their own. Memory management is another issue.
- #25: Pull-based operator is how most databases were/are implemented. Note how the pull model enables hash join, merge join, … Side inputs benefit from a pull-based model Bring the dog-drinking-from-hose example, also for Join operator This will allow porting batch operators/algorithms to StreamOperator
- #27: Note that this nicely jibes with the pull-based model. Enables the things we need for batch.
- #28: Mention the dog with the hose. Sources just keep spitting out records as fast as they can.