Apache Flink Deep Dive
6 likes1,816 views
Flink provides concise summaries of key points: 1) After submitting a Flink job, the client creates and submits the job graph to the JobManager, which then creates an execution graph and deploys tasks across TaskManagers for parallel execution. 2) The batch optimizer chooses optimal execution plans by evaluating physical execution strategies like join algorithms and data shipping approaches to minimize data shuffling and network usage. 3) Flink iterations are optimized by having the runtime directly handle caching, state maintenance, and pushing work out of loops to avoid scheduling overhead between iterations. Delta iterations further improve efficiency by only updating changed elements in each iteration.
![DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
broadcast forward
Combine
GroupRed
sort
DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
hash-part [0] hash-part [0]
hash-part [0,1]
GroupRed
sort
forwardBest plan
depends on
relative sizes
of input files
Alternative Execution Plans
16](https://image.slidesharecdn.com/flink-meetupinternals-150512152148-lva1-app6892/85/Apache-Flink-Deep-Dive-16-320.jpg)
![case class PageVisit(url: String, ip: String, userId: Long)
case class User(id: Long, name: String, email: String, country: String)
// get your data from somewhere
val visits: DataSet[PageVisit] = ...
val users: DataSet[User] = ...
// filter the users data set
val germanUsers = users.filter((u) => u.country.equals("de"))
// join data sets
val germanVisits: DataSet[(PageVisit, User)] =
// equi-join condition (PageVisit.userId = User.id)
visits.join(germanUsers).where("userId").equalTo("id")
Example: Distributed Joins
The join operator needs to
create all the pairs of
elements from the two
inputs, for which the join
condition evaluates to true
19](https://image.slidesharecdn.com/flink-meetupinternals-150512152148-lva1-app6892/85/Apache-Flink-Deep-Dive-19-320.jpg)
Apache Flink Deep Dive
- 1. Apache Flink Deep Dive Vasia Kalavri Flink Committer & KTH PhD student vasia@apache.org 1st Apache Flink Meetup Stockholm May 11, 2015
- 2. Flink Internals ● Job Life-Cycle ○ what happens after you submit a Flink job? ● The Batch Optimizer ○ how are execution plans chosen? ● Delta Iterations ○ how are Flink iterations special for Graph and ML apps? 2
- 3. what happens after you submit a Flink job?
- 4. The Flink Stack Python Gelly Table FlinkML SAMOA Batch Optimizer DataSet (Java/Scala) DataStream (Java/Scala)Hadoop M/R Flink Runtime Local Remote Yarn Tez Embedded Dataflow *current Flink master + few PRs Streaming Optimizer 4
- 5. DataSet<String> text = env.readTextFile(input); DataSet<Tuple2<String, Integer>> result = text .flatMap((str, out) -> { for (String token : value.split("W")) { out.collect(new Tuple2(token, 1)); }) .groupBy(0).aggregate(SUM, 1); 1 3 2 Program Life-Cycle 4 5
- 6. Task Manager Job Manager Task Manager Flink Client & Optimizer DataSet<String> text = env.readTextFile(input); DataSet<Tuple2<String, Integer>> result = text .flatMap((str, out) -> { for (String token : value.split("W")) { out.collect(new Tuple2(token, 1)); }) .groupBy(0).aggregate(SUM, 1); O Romeo, Romeo, wherefore art thou Romeo? O, 1 Romeo, 3 wherefore, 1 art, 1 thou, 1 6 Nor arm, nor face, nor any other part nor, 3 arm, 1 face, 1, any, 1, other, 1 part, 1 creates and submits the job graph creates the execution graph and deploys tasks execute tasks and send status updates
- 7. Input First SecondX Y Operator X Operator Y ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<String> input = env.readTextFile(input); DataSet<String> first = input.filter (str -> str.contains(“Apache Flink“)); DataSet<String> second = first.filter (str -> str.length() > 40); second.print() env.execute(); Series of Transformations 7
- 8. DataSet Abstraction Think of it as a collection of data elements that can be produced/recovered in several ways: … like a Java collection … like an RDD … perhaps it is never fully materialized (because the program does not need it to) … implicitly updated in an iteration → this is transparent to the user 8
- 9. Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 Example: grep 9
- 10. Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 Stage 1: Create/cache Log Subsequent stages: Grep log for matches Caching in-memory and disk if needed Staged (batch) execution 10
- 11. Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 001100110011001100110011 Stage 1: Deploy and start operators Data transfer in-memory and disk if needed Note: Log DataSet is never “created”! Pipelined execution 11
- 12. 12
- 13. how are execution plans chosen?
- 14. Flink Batch Optimizer Inspired by database optimizers, it creates and selects the execution plan for a user program 14
- 15. DataSet<Tuple5<Integer, String, String, String, Integer>> orders = … DataSet<Tuple2<Integer, Double>> lineitems = … DataSet<Tuple2<Integer, Integer>> filteredOrders = orders .filter(. . .) .project(0,4).types(Integer.class, Integer.class); DataSet<Tuple3<Integer, Integer, Double>> lineitemsOfOrders = filteredOrders .join(lineitems) .where(0).equalTo(0) .projectFirst(0,1).projectSecond(1) .types(Integer.class, Integer.class, Double.class); DataSet<Tuple3<Integer, Integer, Double>> priceSums = lineitemsOfOrders .groupBy(0,1).aggregate(Aggregations.SUM, 2); priceSums.writeAsCsv(outputPath); A Simple Program 15
- 16. DataSource orders.tbl Filter Map DataSource lineitem.tbl Join Hybrid Hash buildHT probe broadcast forward Combine GroupRed sort DataSource orders.tbl Filter Map DataSource lineitem.tbl Join Hybrid Hash buildHT probe hash-part [0] hash-part [0] hash-part [0,1] GroupRed sort forwardBest plan depends on relative sizes of input files Alternative Execution Plans 16
- 17. 17
- 18. ● Evaluates physical execution strategies ○ e.g. hash-join vs. sort-merge join ● Chooses data shipping strategies ○ e.g. broadcast vs. partition ● Reuses partitioning and sort orders ● Decides to cache loop-invariant data in iterations Optimization Examples 18
- 19. case class PageVisit(url: String, ip: String, userId: Long) case class User(id: Long, name: String, email: String, country: String) // get your data from somewhere val visits: DataSet[PageVisit] = ... val users: DataSet[User] = ... // filter the users data set val germanUsers = users.filter((u) => u.country.equals("de")) // join data sets val germanVisits: DataSet[(PageVisit, User)] = // equi-join condition (PageVisit.userId = User.id) visits.join(germanUsers).where("userId").equalTo("id") Example: Distributed Joins The join operator needs to create all the pairs of elements from the two inputs, for which the join condition evaluates to true 19
- 20. Example: Distributed Joins ● Ship Strategy: The input data is distributed across all parallel instances that participate in the join ● Local Strategy: Each parallel instance performs a join algorithm on its local partition For both steps, there are multiple valid strategies which are favorable in different situations. 20
- 21. Repartition-Repartition Strategy Partitions both inputs using the same partitioning function. All elements that share the same join key are shipped to the same parallel instance and can be locally joined. 21
- 22. Broadcast-Forward Strategy Sends one complete data set to each parallel instance that holds a partition of the other data. The other Dataset remains local and is not shipped at all. 22
- 23. The optimizer will compute cost estimates for execution plans and will pick the “cheapest” plan: ● amount of data shipped over the the network ● if the data of one input is already partitioned R-R Cost: Full shuffle of both data sets over the network B-F Cost: Depends on the size of the dataset that is broadcasted and the number of parallel instances Read more: http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html How does the Optimizer choose? 23
- 24. how are Flink iterations special?
- 25. ● for/while loop in client submits one job per iteration step ● Data reuse by caching in memory and/or disk Step Step Step Step Step Client Iterate by unrolling 25
- 26. Native Iterations ● the runtime is aware of the iterative execution ● no scheduling overhead between iterations ● caching and state maintenance are handled automatically Caching Loop-invariant DataPushing work “out of the loop” Maintain state as index 26
- 27. Flink Iteration Operators Iterate IterateDelta Input Iterative Update Function Result Replace Workset Iterative Update Function Result Solution Set State 27
- 28. Delta Iteration ● Not all the elements of the state are updated in each iteration. ● The elements that require an update, are stored in the workset. ● The step function is applied only to the workset elements. 28
- 29. Partition a graph into components by iteratively propagating the min vertex ID among neighbors Example: Connected Components 29
- 31. 31
- 32. Performance 32
- 33. Read the documentation and our blog posts! ● Memory Management ● Serialization and Type Extraction ● Streaming Optimizations ● Fault-Tolerance Want to learn more? 33
- 34. Apache Flink Deep Dive Vasia Kalavri Flink Committer & KTH PhD student vasia@apache.org 1st Apache Flink Meetup Stockholm May 11, 2015