Introducing Arc: A Common Intermediate Language for Unified Batch and Stream Analytics - Max Meldrum & Klas Segeljakt, KTH
1 like128 views
The document presents the ARC intermediate representation (IR), designed for unified batch and stream analytics using Apache Flink, addressing language barriers and optimization challenges in data processing. It introduces ARC's features, including values, builders, and a dataflow model, while demonstrating its integration with various frameworks to enhance performance for data scientists. Furthermore, it discusses experimental results, example code, and future goals for ARC and its implementation as a Rust-based dataflow engine.
![7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-22-320.jpg)
![7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-23-320.jpg)
![7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])
• Values are written to builders, and builders are lazily materialised back into values](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-24-320.jpg)
![7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])
• Values are written to builders, and builders are lazily materialised back into values
➡ Dependencies between values and builders form a dataflow graph](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-25-320.jpg)
![8
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Arc
source
evenSink
oddSink
map(v+5)
filter(v%2==0)
filter(v%2!=0)
Arc Example](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-27-320.jpg)
![10
source
evenSink
oddSink
map(v+5) then if(x%2==0)
Arc Example (Fused)
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Unfused](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-29-320.jpg)
![10
source
evenSink
oddSink
map(v+5) then if(x%2==0)
Arc Example (Fused)
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Unfused
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
for(source,
{evenSink,oddSink},
|out, v|
let x = v + 5;
if (x % 2 == 0,
{merge(out.$1, x), out.$2},
{out.$1, merge(out.$2, x)}))
Fused](https://image.slidesharecdn.com/introducingarcacommonintermediatelanguageforunifiedbatchandstreamanalytics-maxmeldrumriseklassegelja-191014105834/85/Introducing-Arc-A-Common-Intermediate-Language-for-Unified-Batch-and-Stream-Analytics-Max-Meldrum-Klas-Segeljakt-KTH-30-320.jpg)
Introducing Arc: A Common Intermediate Language for Unified Batch and Stream Analytics - Max Meldrum & Klas Segeljakt, KTH
- 1. Introducing Arc Klas Segeljakt Max Meldrum @FlinkForward A Common Intermediate Language for Unified Batch and Stream Analytics
- 2. Outline • Project Introduction • The Arc Intermediate Representation (IR) • Arc Examples • Arc + Flink Integration? • Conclusions 2
- 3. The Big Picture 3 • Flink plays an important role in the data science landscape ? ? ? ? ? ?
- 4. The Big Picture 3 • Flink plays an important role in the data science landscape • Combining Flink with other frameworks can lead to interesting applications ? ? ? ? ? ?
- 5. The Big Picture 3 • Flink plays an important role in the data science landscape • Combining Flink with other frameworks can lead to interesting applications • However, there is a language barrier ? ? ? ? ? ?
- 7. Intuition 4 f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing
- 8. Intuition 4 f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing • Data movement costs ( )
- 9. Intuition 4 #Frameworks Performance f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing • Data movement costs ( )
- 10. Intuition 4 #Frameworks Performance f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing • Data movement costs ( )
- 11. Intuition 4 #Frameworks Performance f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing • Data movement costs ( ) f1 f2 f3IR IR IR
- 12. Intuition 4 #Frameworks Performance f1 f2 f3 • No cross-optimisation optimisation is possible, e.g. resource sharing • Data movement costs ( ) f1 f2 f3IR IR IR f1 + f2 + f3IR
- 13. The Arc IR 5 • Streams • Tables • Linear algebra High-level • Runners • Hardware Low-level
- 14. Arc The Arc IR 5 • Streams • Tables • Linear algebra High-level • Runners • Hardware Low-level
- 15. Arc The Arc IR 5 • Streams • Tables • Linear algebra High-level • Runners • Hardware Low-level Abstractions • Pipelines (Operators/Sources/Sinks) • User-defined Windows • Out-of-Order Processing, ...
- 16. Arc The Arc IR 5 • Streams • Tables • Linear algebra High-level • Runners • Hardware Low-level Optimisations • Compiler: Partial evaluation, ... • Dataflow: Operator fusion, fission, reordering, ... Abstractions • Pipelines (Operators/Sources/Sinks) • User-defined Windows • Out-of-Order Processing, ...
- 17. Compiler Pipeline 6 Arc (High Level IR) Frontends Logical Dataflow IR Binaries Physical Dataflow IR
- 18. Compiler Pipeline 6 Arc (High Level IR) Frontends Logical Dataflow IR Binaries Physical Dataflow IR Flink Backend Flink Frontend
- 19. 7 What is Arc?
- 20. 7 What is Arc? A restrictive language for describing batch and stream transformations
- 21. 7 What is Arc? A restrictive language for describing batch and stream transformations Transformations are modelled through:
- 22. 7 What is Arc? A restrictive language for describing batch and stream transformations Transformations are modelled through: • Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
- 23. 7 What is Arc? A restrictive language for describing batch and stream transformations Transformations are modelled through: • Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64) • Builders: Write-only data types (e.g. Appender[T])
- 24. 7 What is Arc? A restrictive language for describing batch and stream transformations Transformations are modelled through: • Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64) • Builders: Write-only data types (e.g. Appender[T]) • Values are written to builders, and builders are lazily materialised back into values
- 25. 7 What is Arc? A restrictive language for describing batch and stream transformations Transformations are modelled through: • Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64) • Builders: Write-only data types (e.g. Appender[T]) • Values are written to builders, and builders are lazily materialised back into values ➡ Dependencies between values and builders form a dataflow graph
- 27. 8 |source:Stream[i32], evenSink:StreamAppender[i32], oddSink:StreamAppender[i32]| let mapped = result(for(source, StreamAppender[i32], |out, v| merge(out, v + 5))); for(mapped, evenSink, |out, v| if (v % 2 == 0, merge(out, v), out)); for(mapped, oddSink, |out, v| if (v % 2 != 0, merge(out, v), out)) Arc source evenSink oddSink map(v+5) filter(v%2==0) filter(v%2!=0) Arc Example
- 28. 9 source evenSink oddSink map(v+5) filter(v%2==0) filter(v%2!=0) Arc Example source evenSink oddSink map(v+5) then if(x%2==0)
- 29. 10 source evenSink oddSink map(v+5) then if(x%2==0) Arc Example (Fused) |source:Stream[i32], evenSink:StreamAppender[i32], oddSink:StreamAppender[i32]| let mapped = result(for(source, StreamAppender[i32], |out, v| merge(out, v + 5))); for(mapped, evenSink, |out, v| if (v % 2 == 0, merge(out, v), out)); for(mapped, oddSink, |out, v| if (v % 2 != 0, merge(out, v), out)) Unfused
- 30. 10 source evenSink oddSink map(v+5) then if(x%2==0) Arc Example (Fused) |source:Stream[i32], evenSink:StreamAppender[i32], oddSink:StreamAppender[i32]| let mapped = result(for(source, StreamAppender[i32], |out, v| merge(out, v + 5))); for(mapped, evenSink, |out, v| if (v % 2 == 0, merge(out, v), out)); for(mapped, oddSink, |out, v| if (v % 2 != 0, merge(out, v), out)) Unfused |source:Stream[i32], evenSink:StreamAppender[i32], oddSink:StreamAppender[i32]| for(source, {evenSink,oddSink}, |out, v| let x = v + 5; if (x % 2 == 0, {merge(out.$1, x), out.$2}, {out.$1, merge(out.$2, x)})) Fused
- 31. 11 Arc + Flink?
- 32. • Benefits: 11 Arc + Flink?
- 33. • Benefits: • Enable stronger optimisations 11 Arc + Flink?
- 34. • Benefits: • Enable stronger optimisations • Use your other favourite libraries together with Flink 11 Arc + Flink?
- 35. • Benefits: • Enable stronger optimisations • Use your other favourite libraries together with Flink • Make life easier for data scientists 11 Arc + Flink?
- 36. The black box problem UDFs are black boxes 12 stream.map( ) .filter( ) .reduce( )
- 37. The black box problem UDFs are black boxes ➡ Flink is unaware of what is being executed inside of each black box 12 stream.map( ) .filter( ) .reduce( )
- 38. Fusion Levels 13 = Flink Task ~ Thread
- 39. Fusion Levels 13 = Flink Task ~ Thread x + 1 x + 1 x + 1 x + 11. No Fusion
- 40. Fusion Levels 13 = Flink Task ~ Thread x + 1 x + 1 x + 1 x + 11. No Fusion x + 1 x + 1 x + 1 x + 12.Task Fusion
- 41. Fusion Levels 13 = Flink Task ~ Thread x + 1 x + 1 x + 1 x + 11. No Fusion x + 1 x + 1 x + 1 x + 12.Task Fusion 3. Invocation-level Fusion x + 1 for-loop 4X
- 42. Fusion Levels 13 = Flink Task ~ Thread x + 1 x + 1 x + 1 x + 11. No Fusion x + 1 x + 1 x + 1 x + 12.Task Fusion 4. Instruction-level Fusion x + 4 3. Invocation-level Fusion x + 1 for-loop 4X
- 43. 14 Experiment Results 100 101 102 103 ExecutionTime(seconds) None Task(Flink) Invocation Instruction 50 maps on 10M elements N one Task(Flink) Invocation Instruction Optimisation level (Lower is better)
- 44. Example Frontend Code (Pandas + Beam) 15
- 45. Example Frontend Code (Pandas + Beam) 15
- 46. Example Frontend Code (Pandas + Beam) 15 import arc.beam as beam import arc.beam.transforms.window as window import arc.beam.transforms.combiners as combiners import arc.pandas as pandas
- 47. Example Frontend Code (Pandas + Beam) 15 import arc.beam as beam import arc.beam.transforms.window as window import arc.beam.transforms.combiners as combiners import arc.pandas as pandas def normalise(elements): series = pandas.Series(elements) avg = series.sum() / series.count() return series / avg
- 48. Example Frontend Code (Pandas + Beam) 15 import arc.beam as beam import arc.beam.transforms.window as window import arc.beam.transforms.combiners as combiners import arc.pandas as pandas def normalise(elements): series = pandas.Series(elements) avg = series.sum() / series.count() return series / avg p = beam.Pipeline() (p | beam.io.ReadFromText(path='input.txt').with_output_types(int) | beam.WindowInto(window.FixedWindows(size=5)) | beam.CombineGlobally(normalise) | combiners.ToList() | beam.io.WriteToText(path='output.txt')) p.run()
- 49. Example Frontend Code (Pandas + Beam) 15 import arc.beam as beam import arc.beam.transforms.window as window import arc.beam.transforms.combiners as combiners import arc.pandas as pandas def normalise(elements): series = pandas.Series(elements) avg = series.sum() / series.count() return series / avg p = beam.Pipeline() (p | beam.io.ReadFromText(path='input.txt').with_output_types(int) | beam.WindowInto(window.FixedWindows(size=5)) | beam.CombineGlobally(normalise) | combiners.ToList() | beam.io.WriteToText(path='output.txt')) p.run() ?
- 51. • Flink inspired dataflow engine built in Rust • Goals: • Common runtime for Arc applications • Support dynamic task execution • First-class support for hardware acceleration 17 Arcon: Native Arc Runner
- 52. • Arc is an IR for batch and stream programming. • By raising the level of abstraction, Arc is able to both optimise the dataflow and the code within it. 18 Conclusions Arc and experiments can be found at https://github.com/cda-group Contact info: klasseg@kth.se & mmeldrum@kth.se
- 53. References Publications: • Kroll, L., Segeljakt, K., Carbone, P., Schulte, C. and Haridi, S., 2019, June. Arc: an IR for batch and stream programming. In Proceedings of the 17th ACM SIGPLAN International Symposium on Database Programming Languages (pp. 53-58). ACM. • Meldrum, M., Segeljakt, K., Kroll, L., Carbone, P., Schulte, C. and Haridi, S., 2019, August. Arcon: Continuous and Deep Data Stream Analytics. In Proceedings of Real-Time Business Intelligence and Analytics (p. 3). ACM. 19