Self Managed and Automatically Reconfigurable Stream Processing - Vasiliki Kalavri, ETH Zurich
0 likes380 views
The document discusses advanced techniques in stream processing, particularly focusing on automatic scaling and performance optimization in systems like Apache Flink and Spark. It introduces concepts such as adaptive scheduling, critical path analysis, and metrics for assessing activity criticality within streaming applications. The findings emphasize the need for accurate scaling decisions to enhance the efficiency of distributed streaming dataflows.
![CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
23
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
Definition 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
be the set of N transient paths of snapshot G[ts,te]. The tran-
sient path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
The following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Spark, Flink
di↵erent, but act
ysis: all execute
graphs whose v
whose edges den
ers (threads, pr
graph can be tran
all workers appl
tions of the data
1 We provide proofs
4](https://image.slidesharecdn.com/self-managedandautomaticallyreconfigurablestreamprocessing-vasilikikalavriethzurich-191014111757/85/Self-Managed-and-Automatically-Reconfigurable-Stream-Processing-Vasiliki-Kalavri-ETH-Zurich-41-320.jpg)
![CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
23
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
Can be computed
without path
enumeration!
Definition 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
be the set of N transient paths of snapshot G[ts,te]. The tran-
sient path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
The following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Spark, Flink
di↵erent, but act
ysis: all execute
graphs whose v
whose edges den
ers (threads, pr
graph can be tran
all workers appl
tions of the data
1 We provide proofs
4](https://image.slidesharecdn.com/self-managedandautomaticallyreconfigurablestreamprocessing-vasilikikalavriethzurich-191014111757/85/Self-Managed-and-Automatically-Reconfigurable-Stream-Processing-Vasiliki-Kalavri-ETH-Zurich-42-320.jpg)
Self Managed and Automatically Reconfigurable Stream Processing - Vasiliki Kalavri, ETH Zurich
- 1. Flink Forward Europe 8 October 2019 VASILIKI KALAVRI vasia@apache.org SELF-MANAGED AND AUTOMATICALLY RECONFIGURABLE STREAM PROCESSING @vkalavri
- 2. 2 20001992 2013 MapReduce 2004 Tapestry NiagaraCQ Aurora TelegraphCQ STREAM Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Next-gen streaming Now
- 3. Single-node execution Synopses and sketches Stream Database Systems 2 20001992 2013 MapReduce 2004 Tapestry NiagaraCQ Aurora TelegraphCQ STREAM Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Next-gen streaming Now Tapestry NiagaraCQ Aurora TelegraphCQ STREAM
- 4. Dataflow Systems Distributed execution Partitioned state Single-node execution Synopses and sketches Stream Database Systems 2 20001992 2013 MapReduce 2004 Tapestry NiagaraCQ Aurora TelegraphCQ STREAM Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Next-gen streaming Now Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Next-gen streaming Tapestry NiagaraCQ Aurora TelegraphCQ STREAM
- 5. 3 2013 MapReduce 2004 Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Now Next-gen streaming
- 6. 3 2013 MapReduce 2004 Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Now Re-configurable Systems Automatic scaling Analyzer invoke re-configure job performance metrics decision Profiler Adaptive scheduling Straggler mitigation Query optimization Instrumented stream processor Next-gen streaming
- 7. SNAILTRAIL: GENERALIZING CRITICAL PATHS FOR ONLINE ANALYSIS OF DISTRIBUTED DATAFLOWS NSDI’18
- 8. CONVENTIONAL PROFILING TELLS ONLY PART OF THE STORY 5 Duration Aggregate data exchange Dataflow graph
- 9. CONVENTIONAL PROFILING TELLS ONLY PART OF THE STORY 5 Duration Aggregate data exchange Dataflow graph Custom aggregate metrics
- 11. 6 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 5 10 15 Snapshot %weight Processing Scheduling DRIVER W1 W2 W3 PROFILING SPARK SCHEDULING processing scheduling
- 12. 7 worker 1 worker 2 worker 3 receive message deserialization processing serialization send message waiting waiting
- 13. 8 worker 1 worker 2 worker 3 processing
- 14. OPTIMIZING PROCESSING… 9 worker 1 worker 2 worker 3
- 15. OPTIMIZING PROCESSING INCREASED WAITING 10 worker 1 worker 2 worker 3
- 17. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 18. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 19. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 20. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 21. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 22. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 23. CRITICAL PATH: LONGEST EXECUTION PATH (not considering waiting activities) 12 W1 W2 W3 a b c d
- 24. OPTIMIZING CRITICAL ACTIVITIES CAN REDUCE LATENCY 13 W1 W2 W3 a b c d
- 25. 14 W1 W2 W3 a b c d Reduced execution time OPTIMIZING CRITICAL ACTIVITIES CAN REDUCE LATENCY
- 26. ONLINE CRITICAL PATH ANALYSIS
- 27. ONLINE ANALYSIS OF TRACE SNAPSHOTS 16 input stream output stream
- 28. ONLINE ANALYSIS OF TRACE SNAPSHOTS 16 input stream output stream periodic snapshot trace snapshot stream analyzer performance summaries stream
- 29. 17 W1 W2 W3 a b c d x u v z ts te
- 30. 17 All paths are potentially part of an evolving critical path W1 W2 W3 a b c d x u v z ts te
- 31. W1 W2 W3 a b c d x u v z ts te 18 ▸ All paths have the same length: te - ts
- 32. W1 W2 W3 a b c d x u v z ts te 19 ▸ All paths have the same length: te - ts
- 33. W1 W2 W3 a b c d x u v z ts te 20 ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities
- 34. W1 W2 W3 a b c d x u v z ts te 20 ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities
- 35. W1 W2 W3 a b c d x u v z ts te 20 ▸ All paths have the same length: te - ts ▸ Choosing a random path might miss critical activities
- 36. 21 How to rank activities with regard to criticality?
- 37. 21 How to rank activities with regard to criticality? Intuition: the more paths an activity appears on the more probable this activity is critical
- 38. 1 2 3 4 5 6 7 8 9 22 W1 W2 W3 a b c d x u v z ts te
- 39. 1 2 3 4 5 6 7 8 9 22 W1 W2 W3 a b c d x u v z ts te
- 40. 1 2 3 4 5 6 7 8 9 22 W1 W2 W3 a b c d x u v z ts te 9 0 0 6 6
- 41. CRITICAL PARTICIPATION (CP METRIC) An estimation of the activity’s participation in the critical path 23 total number of paths in the snapshot activity duration: edge weight centrality: the number of paths this activity appears on Definition 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN} be the set of N transient paths of snapshot G[ts,te]. The tran- sient path centrality of an edge e 2 G[ts,te] is defined as c(e) = NX i=1 ci(e), where ci(e) = 8 >>< >>: 0 : e < ~pi 1 : e 2 ~pi The following holds: CPa = TPC(a) · aw N(te ts) (3) Spark, Flink di↵erent, but act ysis: all execute graphs whose v whose edges den ers (threads, pr graph can be tran all workers appl tions of the data 1 We provide proofs 4
- 42. CRITICAL PARTICIPATION (CP METRIC) An estimation of the activity’s participation in the critical path 23 total number of paths in the snapshot activity duration: edge weight centrality: the number of paths this activity appears on Can be computed without path enumeration! Definition 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN} be the set of N transient paths of snapshot G[ts,te]. The tran- sient path centrality of an edge e 2 G[ts,te] is defined as c(e) = NX i=1 ci(e), where ci(e) = 8 >>< >>: 0 : e < ~pi 1 : e 2 ~pi The following holds: CPa = TPC(a) · aw N(te ts) (3) Spark, Flink di↵erent, but act ysis: all execute graphs whose v whose edges den ers (threads, pr graph can be tran all workers appl tions of the data 1 We provide proofs 4
- 44. 25 reference application SnailTrail Timely Trace ingestion CP-based performance summaries PAG construction CP computation and activity ranking trace streams Profiling Trace generation Apache Flink, Apache Spark, TensorFlow, Heron, Timely Dataflow, ...
- 45. 26 DRIVER W1 W2 W3 DRIVER SCHEDULING IS CRITICAL processing scheduling
- 46. 26 DRIVER W1 W2 W3 0 5 10 15 Snapshot 0.0 0.2 0.4 0.6 0.8 CP 0 %weight Processing DRIVER SCHEDULING IS CRITICAL processing scheduling
- 48. 28 2013 MapReduce 2004 Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Now Re-configurable Systems Automatic scaling Analyzer invoke re-configure job performance metrics decision Profiler Adaptive scheduling Straggler mitigation Query optimization Instrumented stream processor Next-gen streaming
- 49. FAST AND ACCURATE AUTOMATIC SCALING DECISIONS FOR DISTRIBUTED STREAMING DATAFLOWS OSDI’18
- 50. 30 Streaming systems must be capable of adapting the level of parallelism when conditions change at runtime events/s time : input rate : throughput Data loss SLO violationsIdle resources events/s time events/s time
- 51. AUTOMATIC SCALING OVERVIEW 31 scaling controller detect symptoms decide whether to scale decide how much to scale metrics policy scaling action
- 52. HEURISTIC SCALING APPROACHES 32 CPU utilization backlog, tuples/s backpressure signal threshold and rule-based if CPU > 80% => scale small changes, one operator at a time Borealis StreamCloud Seep IBM Streams Spark Streaming Google Dataflow Dhalion scaling actionmetrics policy
- 53. HEURISTIC SCALING APPROACHES 32 CPU utilization backlog, tuples/s backpressure signal threshold and rule-based if CPU > 80% => scale small changes, one operator at a time Problematic under interference, multi-tenancy Sensitive to noise, manual, hard to tune Non-predictive, speculative steps Borealis StreamCloud Seep IBM Streams Spark Streaming Google Dataflow Dhalion scaling actionmetrics policy
- 54. Effect of Dhalion’s scaling actions in an initially under-provisioned wordcount dataflow 33
- 55. Effect of Dhalion’s scaling actions in an initially under-provisioned wordcount dataflow 33 o1src o2 back-pressure! target: 40 rec/s
- 56. Effect of Dhalion’s scaling actions in an initially under-provisioned wordcount dataflow 33 o1src o2 back-pressure! target: 40 rec/s 10 rec/s 100 rec/s
- 57. Effect of Dhalion’s scaling actions in an initially under-provisioned wordcount dataflow 33 o1src o2 back-pressure! target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if we scale ο1 x 4? How much to scale ο2?
- 58. 34 o1src o2 back-pressure! target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if we scale ο1 x 4? How much to scale ο2?
- 59. 34 o1src o2 back-pressure! target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if we scale ο1 x 4? How much to scale ο2? o1 cannot keep up waiting for output waiting for input src o1 o2
- 60. 34 o1src o2 back-pressure! target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if we scale ο1 x 4? How much to scale ο2? o1 cannot keep up waiting for output waiting for input src o1 o2 o2 cannot keep up src o1 o2
- 61. THE DS2 MODEL
- 62. 36 src o1 o2 10 recs 10 recs 1 2 3 4 100 rec 100 recs Intuition: use the dataflow graph to extract operator dependencies and system instrumentation to collect accurate, representative metrics. target: 40 rec/s 0.5s
- 63. 36 src o1 o2 10 recs 10 recs 1 2 3 4 100 rec 100 recs Intuition: use the dataflow graph to extract operator dependencies and system instrumentation to collect accurate, representative metrics. x4 instances to keep up with src rate target: 40 rec/s 0.5s
- 64. 36 src o1 o2 10 recs 10 recs 1 2 3 4 100 rec 100 recs Intuition: use the dataflow graph to extract operator dependencies and system instrumentation to collect accurate, representative metrics. True rate = 200 recs/s x4 instances to keep up with src rate target: 40 rec/s 0.5s
- 65. 36 src o1 o2 10 recs 10 recs 1 2 3 4 100 rec 100 recs Intuition: use the dataflow graph to extract operator dependencies and system instrumentation to collect accurate, representative metrics. True rate = 200 recs/s x4 instances to keep up with src rate x2 instances to keep up with x4 o1 instances target: 40 rec/s 0.5s
- 66. If operator scaling is linear, then: ▸ no overshoot when scaling up ▸ no undershoot when scaling down 37 parallelism initial rate target prediction p0 p1 parallelism initial rate target p0p1 prediction DS2 MAKES LINEAR PREDICTIONS
- 67. If operator scaling is linear, then: ▸ no overshoot when scaling up ▸ no undershoot when scaling down 37 parallelism initial rate target prediction p0 p1 parallelism initial rate target p0p1 prediction DS2 MAKES LINEAR PREDICTIONS x x p’ p’
- 68. If operator scaling is linear, then: ▸ no overshoot when scaling up ▸ no undershoot when scaling down 37 parallelism initial rate target prediction p0 p1 parallelism initial rate target p0p1 Ideal rates act as un upper bound when scaling up and as a lower bound when scaling down: ▸ DS2 will converge monotonically to the target rate prediction DS2 MAKES LINEAR PREDICTIONS p’ p’
- 69. If operator scaling is linear, then: ▸ no overshoot when scaling up ▸ no undershoot when scaling down 37 parallelism initial rate target prediction p0 p1 parallelism initial rate target p0p1 Ideal rates act as un upper bound when scaling up and as a lower bound when scaling down: ▸ DS2 will converge monotonically to the target rate prediction DS2 MAKES LINEAR PREDICTIONS actual actual
- 70. DS2 MINIMIZES THE ERROR UNTIL CONVERGENCE 38 parallelism initial rate target actual error p0 p1 prediction x x x
- 71. DS2 MINIMIZES THE ERROR UNTIL CONVERGENCE 38 parallelism initial rate target actual p0 p1 x new prediction
- 72. DS2 MINIMIZES THE ERROR UNTIL CONVERGENCE 38 parallelism initial rate target actual p0 p1 x error p1’ new prediction Gradually minimizes error
- 73. EVALUATION
- 74. 40 Scaling Manager Scaling Policy Metrics Repository invoke re-scale job report metrics monitor pull metrics decision Timely dataflow Apache Flink Instrumented stream processor
- 75. DS2 VS. STATE-OF-THE-ART ON HERON 41 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s
- 76. DS2 VS. STATE-OF-THE-ART ON HERON 41 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s
- 77. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s
- 78. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s DS2 converges in a single step for both operators
- 79. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s DS2 converges in a single step for both operators and converges in 60s, as soon as it receives the Heron metrics
- 80. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s DS2 converges in a single step for both operators Dhalion scales one operator at a time, and needs six steps in total 1 6 5 43 2and converges in 60s, as soon as it receives the Heron metrics
- 81. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow Target rate: 16.700 rec/s DS2 converges in a single step for both operators and converges in 2000s Dhalion scales one operator at a time, and needs six steps in total 1 6 5 43 2and converges in 60s, as soon as it receives the Heron metrics
- 82. DS2 VS. STATE-OF-THE-ART ON HERON 42 Initially under-provisioned wordcount dataflow +10 counts +12 mappers Target rate: 16.700 rec/s DS2 converges in a single step for both operators and converges in 2000s Dhalion scales one operator at a time, and needs six steps in total 1 6 5 43 2and converges in 60s, as soon as it receives the Heron metrics
- 83. DS2 ON APACHE FLINK 43 Initially under-provisioned wordcount Target rate: 2.000.000 rec/s, drops to half at 800s
- 84. DS2 ON APACHE FLINK 43 Initially under-provisioned wordcount Target rate: 2.000.000 rec/s, drops to half at 800s DS2 converges in 2 steps for both operators 1 2
- 85. DS2 ON APACHE FLINK 43 Initially under-provisioned wordcount Target rate: 2.000.000 rec/s, drops to half at 800s DS2 reacts within 3s when the target rate drops DS2 converges in 2 steps for both operators 1 2
- 86. DS2 ON APACHE FLINK 43 Initially under-provisioned wordcount Target rate: 2.000.000 rec/s, drops to half at 800s DS2 reacts within 3s when the target rate drops DS2 converges in 2 steps for both operators 1 2 Transient underpovisioning by 1 instance
- 87. 44 github.com/strymon-system Kalavri V, Liagouris J, Hoffmann M, Dimitrova D, Forshaw M, Roscoe T. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. OSDI ’18. Hoffmann M, Lattuada A, Liagouris J, Kalavri V, Dimitrova D, Wicki S, Chothia Z, Roscoe T. Snailtrail: Generalizing critical paths for online analysis of distributed dataflows. NSDI’18. github.com/li1/snailtrail
- 88. 45 Zaheer Chothia Andrea Lattuada Timothy Roscoe Moritz Hoffmann Desislava Dimitrova John Liagouris Malte Sandstede Matthew ForshawSebastian Wicki strymon.systems.ethz.ch
- 89. 46 www.bu.edu/cs/phd-program/phd/ Let’s work on streaming research together
- 90. Flink Forward Europe 8 October 2019 VASILIKI KALAVRI vasia@apache.org SELF-MANAGED AND AUTOMATICALLY RECONFIGURABLE STREAM PROCESSING @vkalavri