Online performance analysis of distributed dataflow systems (O'Reilly Velocity 2017)
1 like377 views
The document discusses the online performance analysis of distributed dataflow systems, particularly focusing on a framework called Strymon for real-time datacenter analytics and management. It highlights the challenges in analyzing performance metrics and the importance of critical path analysis to optimize execution in large-scale streaming applications. Additionally, it emphasizes the capabilities of Strymon in providing low-latency insights and enhancing performance through its online profiling tools.
![36
CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
set of N transient paths of snapshot G[ts,te]. The tran-
path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
e following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Sp
di↵er
ysis:
graph
whos
ers (t
graph
all wo
tions
1 We p
4](https://image.slidesharecdn.com/snailtrail-velocity-171023123622/85/Online-performance-analysis-of-distributed-dataflow-systems-O-Reilly-Velocity-2017-61-320.jpg)
![36
CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
Can be computed
without path
enumeration!
tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
set of N transient paths of snapshot G[ts,te]. The tran-
path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
e following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Sp
di↵er
ysis:
graph
whos
ers (t
graph
all wo
tions
1 We p
4](https://image.slidesharecdn.com/snailtrail-velocity-171023123622/85/Online-performance-analysis-of-distributed-dataflow-systems-O-Reilly-Velocity-2017-62-320.jpg)
Online performance analysis of distributed dataflow systems (O'Reilly Velocity 2017)
- 1. ONLINE PERFORMANCE ANALYSIS OF DISTRIBUTED DATAFLOW SYSTEMS Vasia Kalavri kalavriv@inf.ethz.ch Support: O’Reilly Velocity London 20 October 2017
- 2. ABOUT ME ▸ Postdoc at ETH Zürich ▸ Systems Group: https://www.systems.ethz.ch/ ▸ PMC member of Apache Flink ▸ Research interests ▸ Large-scale graph processing ▸ Streaming dataflow engines ▸ Current project ▸ Predictive datacenter analytics and management 2 @vkalavri
- 3. 33 traces, configuration, topology updates, … Datacenter Strymon queries, complex analytics, simulations, … policy enforcement, what-if scenarios, … STRYMON: ONLINE DATACENTER ANALYTICS AND MANAGEMENT Datacenter model event streams strymon.systems.ethz.ch
- 4. STRYMON IS BUILT ON TIMELY 4 ▸ A steaming framework for data-parallel computations ▸ Arbitrary cyclic dataflows ▸ Logical timestamps (epochs) ▸ Asynchronous execution ▸ Low latency D. Murray, F. McSherry, M. Isard, R. Isaacs, P. Barham, M. Abadi. Naiad: A Timely Dataflow System. In SOSP, 2013. 4 https://github.com/frankmcsherry/timely-dataflow
- 5. What-if scenarios Real-time datacenter analytics Incremental network routing Online critical path analysis Explaining simulations Strymon 5
- 6. Apache Flink SnailTrail: Strymon’s component for online performance analysis of distributed dataflows 6 What-if scenarios Real-time datacenter analytics Incremental network routing Online critical path analysis Explaining simulations Strymon
- 8. UNDERSTANDING THE PERFORMANCE OF DISTRIBUTED DATAFLOWS Hard to troubleshoot ▸ long-running, dynamic workloads ▸ many tasks, activities, operators, dependencies ▸ bottleneck causes are usually not isolated but span multiple processes 8 client W1 W1 scheduler
- 12. PERFORMANCE METRICS 9 Duration Aggregate data exchange Dataflow graph
- 13. PERFORMANCE METRICS 9 Duration Aggregate data exchange Custom aggregate metrics Dataflow graph
- 14. W1 W2 W3 serialization processing waiting deserialization 10 processing message A PARALLEL EXECUTION
- 15. W1 W2 W3 serialization processing waiting deserialization 10 processing message A PARALLEL EXECUTION
- 16. W1 W2 W3 serialization processing waiting deserialization 10 processing messageProcessing is the most time-consuming activity A PARALLEL EXECUTION
- 17. W1 W2 W3 serialization processing waiting deserialization 11 processing What if we optimize it? A PARALLEL EXECUTION
- 21. 13 W1 W2 W3 serialization waiting deserialization No performance benefit for the parallel execution! A PARALLEL EXECUTION
- 24. THE PROGRAM ACTIVITY GRAPH (PAG) 16 W1 W2 W3 a b c d t=k t=k+1 x u v z
- 25. THE PROGRAM ACTIVITY GRAPH (PAG) 16 W1 W2 W3 a b c d t=k t=k+1 x u v z Nodes are timestamped events: start or end of a worker activity
- 26. THE PROGRAM ACTIVITY GRAPH (PAG) 16 W1 W2 W3 a b c d t=k t=k+1 x u v z Nodes are timestamped events: start or end of a worker activity u = { timestamp: k+1, worker: 2 }
- 27. THE PROGRAM ACTIVITY GRAPH (PAG) 17 W1 W2 W3 a b c d t=k t=k+1 x u v z Edges represent activities annotated with a type and duration
- 28. THE PROGRAM ACTIVITY GRAPH (PAG) 17 W1 W2 W3 a b c d t=k t=k+1 x u v z Edges represent activities annotated with a type and duration (u, v) = { type: serialization duration: 1 }
- 29. W1 W2 W3 The Program Activity Graph captures computational dependencies among parallel workers ▸ Which activities delay the overall execution? ▸ i.e. which activities lie on the critical path of execution? 18
- 30. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 31. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 32. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 33. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 34. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 35. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 36. CRITICAL PATH 19 W1 W2 W3 a b c d The longest path in the execution history (not considering waiting activities)
- 37. 20 W1 W2 W3 a b c d CRITICAL PATH
- 39. CRITICAL PATH 21 W1 W2 W3 a b c d Reduced execution time
- 40. POST-MORTEM CRITICAL PATH ANALYSIS IS EASY 1. Collect traces during execution job start job end profiler 22
- 41. POST-MORTEM CRITICAL PATH ANALYSIS IS EASY 1. Collect traces during execution job start job end profiler 22 2. Analyze traces offline analyzer performance summaries
- 42. How to compute the critical path for continuously running, distributed streaming applications, with potentially unbounded input? 23
- 43. How to compute the critical path for continuously running, distributed streaming applications, with potentially unbounded input? ▸ There might be no “job end” ▸ The PAG and critical path are continuously evolving ▸ Stale profiling information is not useful 23
- 44. ONLINE CRITICAL PATH ANALYSIS
- 45. ONLINE ANALYSIS OF TRACE SNAPSHOTS 25 input stream output stream
- 46. ONLINE ANALYSIS OF TRACE SNAPSHOTS 25 input stream output stream periodic snapshot
- 47. ONLINE ANALYSIS OF TRACE SNAPSHOTS 25 input stream output stream periodic snapshot trace snapshot stream analyzer performance summaries stream
- 48. PROGRAM ACTIVITY GRAPH SNAPSHOT 26 W1 W2 W3 a b c d t=k t=k+1 x u v z ts te
- 49. W1 W2 W3 a b c d x u v z ts te 27
- 50. ▸ All paths have the same length: te - ts W1 W2 W3 a b c d x u v z ts te 27
- 51. W1 W2 W3 a b c d x u v z ts te ▸ All paths have the same length: te - ts 28
- 52. W1 W2 W3 a b c d x u v z ts te ▸ All paths have the same length: te - ts 29
- 53. W1 W2 W3 a b c d x u v z ts te ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities 30
- 54. W1 W2 W3 a b c d x u v z ts te ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities 31
- 55. W1 W2 W3 a b c d x u v z ts te ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities 31
- 56. ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities ▸ Enumerating all paths is impractical W1 W2 W3 a b c d x u v z ts te 32
- 57. W1 W2 W3 a b c d x u v z ts te How to rank activities with regard to criticality? All paths are potentially part of the evolving critical path 33
- 58. W1 W2 W3 a b c d x u v z ts te How to rank activities with regard to criticality? All paths are potentially part of the evolving critical path Intuition: the more paths an activity appears on the more probable it is that this activity is critical 33
- 59. 1 2 3 4 5 6 7 8 9 W1 W2 W3 a b c d x u v z ts te 34
- 60. 1 2 3 4 5 6 7 8 9 W1 W2 W3 a b c d x u v z ts te 9 0 0 6 6 35
- 61. 36 CRITICAL PARTICIPATION (CP METRIC) An estimation of the activity’s participation in the critical path total number of paths in the snapshot activity duration: edge weight centrality: the number of paths this activity appears on tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN} set of N transient paths of snapshot G[ts,te]. The tran- path centrality of an edge e 2 G[ts,te] is defined as c(e) = NX i=1 ci(e), where ci(e) = 8 >>< >>: 0 : e < ~pi 1 : e 2 ~pi e following holds: CPa = TPC(a) · aw N(te ts) (3) Sp di↵er ysis: graph whos ers (t graph all wo tions 1 We p 4
- 62. 36 CRITICAL PARTICIPATION (CP METRIC) An estimation of the activity’s participation in the critical path total number of paths in the snapshot activity duration: edge weight centrality: the number of paths this activity appears on Can be computed without path enumeration! tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN} set of N transient paths of snapshot G[ts,te]. The tran- path centrality of an edge e 2 G[ts,te] is defined as c(e) = NX i=1 ci(e), where ci(e) = 8 >>< >>: 0 : e < ~pi 1 : e 2 ~pi e following holds: CPa = TPC(a) · aw N(te ts) (3) Sp di↵er ysis: graph whos ers (t graph all wo tions 1 We p 4
- 63. ONLINE PERFORMANCE ANALYSIS WITH SNAILTRAIL
- 64. 38 reference application SnailTrail Timely Trace ingestion CP-based performance summaries PAG construction CP computation and activity ranking trace streams Profiling Trace generation Apache Flink, Apache Spark, TensorFlow, Heron, Timely Dataflow, ... SNAILTRAIL IN ACTION
- 65. EXAMPLE: TASK SCHEDULING IN APACHE SPARK 39 DRIVER W1 W2 W3 Venkataraman, Shivaram, et al. "Drizzle: Fast and adaptable stream processing at scale." Spark Summit (2016).
- 66. SCHEDULING BOTTLENECK IN APACHE SPARK 40 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 5 10 15 Snapshot %weight Processing Scheduling Conventional ProfilingSnailTrail Profiling Apache Spark: Yahoo! Streaming Benchmark, 16 workers, 8s snapshots
- 67. SNAILTRAIL CP-BASED SUMMARIES ▸ Activity Summary ▸ which activity type is a bottleneck? 41
- 68. 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP DataMessage Unknown Buffer Deserialization Serialization Processing Activity Summary Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots 42
- 69. 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP DataMessage Unknown Buffer Deserialization Serialization Processing Activity Summary Optimize serialization! Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots 42
- 70. SNAILTRAIL CP-BASED SUMMARIES ▸ Activity Summary ▸ which activity type is a bottleneck? ▸ Straggler Summary ▸ which worker is a bottleneck? 43
- 71. 0 5 10 15 Snapshot 0.00 0.05 0.10 0.15 CP W1 W2 W3 W4 Straggler Summary Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots 44
- 72. 0 5 10 15 Snapshot 0.00 0.05 0.10 0.15 CP W1 W2 W3 W4 Straggler Summary Steal work from W1 Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots 44
- 73. SNAILTRAIL CP-BASED SUMMARIES ▸ Activity Summary ▸ which activity type is a bottleneck? ▸ Straggler Summary ▸ which worker is a bottleneck? ▸ Operator Summary ▸ which operator is a bottleneck? 45
- 74. Operator Summary Apache Flink: Dhalion WordCount Benchmark, 10 workers, 1s snapshots 46
- 75. Operator Summary Increase flatmap’s parallelism! Apache Flink: Dhalion WordCount Benchmark, 10 workers, 1s snapshots 46
- 76. SNAILTRAIL CP-BASED SUMMARIES ▸ Activity Summary ▸ which activity type is a bottleneck? ▸ Straggler Summary ▸ which worker is a bottleneck? ▸ Operator Summary ▸ which operator is a bottleneck? ▸ Communication Summary ▸ which communication channels are bottlenecks? 47
- 77. Communication Criticality Communication Summary 48 0 1 2 3 4 5 6 7 8 9 10 11 12 Worker 0 1 2 3 4 5 6 7 8 9 10 11 12 Worker 1 32 54 86 7 109 1211 Worker Worker 1 2 3 4 5 6 7 8 9 10 11 12
- 78. Communication Criticality Communication Summary 48 0 1 2 3 4 5 6 7 8 9 10 11 12 Worker 0 1 2 3 4 5 6 7 8 9 10 11 12 Worker 1 32 54 86 7 109 1211 Worker Worker 1 2 3 4 5 6 7 8 9 10 11 12 Collocate worker 5 with 2, 9, 10, 11
- 79. SNAILTRAIL PERFORMANCE ▸ Low instrumentation overhead ▸ < 10% for all reference systems ▸ High throughput ▸ 1.2 million events per second ▸ Always online ▸ 1s of traces in 6ms, 256s of traces in < 25s 49 Traces from Apache Flink Sessionization, 48 workers, 1s-256s snapshots SnailTrail on Intel Xeon E5-4640, 2.40GHz, 32 cores, 512GB RAM
- 80. RECAP 50
- 81. RECAP 50 Strymon: online datacenter analytics and management
- 82. RECAP 50 Strymon: online datacenter analytics and management 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 5 10 15 Snapshot %weight Processing Scheduling Conventional profiling is misleading
- 83. RECAP 50 Strymon: online datacenter analytics and management 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 5 10 15 Snapshot %weight Processing Scheduling Conventional profiling is misleading CP-metric: online critical path analysis
- 84. RECAP 50 Strymon: online datacenter analytics and management 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 5 10 15 Snapshot %weight Processing Scheduling Conventional profiling is misleading CP-metric: online critical path analysis SnailTrail: online CP-based summaries
- 85. THE STRYMON TEAM & FRIENDS 51 Vasiliki Kalavri Zaheer Chothia Andrea Lattuada Prof. Timothy Roscoe Sebastian Wicki Moritz Hoffmann Desislava Dimitrova John Liagouris Frank McSherry
- 86. THE STRYMON TEAM & FRIENDS 51 Vasiliki Kalavri Zaheer Chothia Andrea Lattuada Prof. Timothy Roscoe Sebastian Wicki Moritz Hoffmann Desislava Dimitrova John Liagouris ? ? Frank McSherry
- 87. THE STRYMON TEAM & FRIENDS 51 Vasiliki Kalavri Zaheer Chothia Andrea Lattuada Prof. Timothy Roscoe Sebastian Wicki Moritz Hoffmann Desislava Dimitrova John Liagouris ? ? Frank McSherry IT COULD BE YOU! strymon.systems.ethz.ch www.systems.ethz.ch/positions
- 88. ONLINE PERFORMANCE ANALYSIS OF DISTRIBUTED DATAFLOW SYSTEMS Vasia Kalavri kalavriv@inf.ethz.ch Support: O’Reilly Velocity London 20 October 2017