SlideShare a Scribd company logo

Reactive Streams / Akka Streams - GeeCON Prague 2014

Konrad Malawski, profile picture
Konrad Malawski

This document contains the slides from a presentation on Akka Streams and Reactive Streams. It discusses key concepts like back pressure in Reactive Streams which allows asynchronous stream processing between different libraries. It also summarizes how Akka Streams works with linear flows, materialization, and integrating with actors to consume streams in an advanced way by treating grouped streams as individual streams processed by child actors.

Technology
1 of 124
Downloaded 96 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Akka Streams Konrad `@ktosopl` Malawski OST: Tomas Dvorak ./‘ ./‘
hAkker @ Konrad `@ktosopl` Malawski Akka Team, Reactive Streams TCK
hAkker @ Konrad `@ktosopl` Malawski typesafe.com geecon.org Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow
You? ?
Czech User Group? You?
You? ? z ?
You? ? z ? ?
You? ? z ? ? ?
Streams
Streams
Streams “You cannot enter the same river twice” ~ Heraclitus http://en.wikiquote.org/wiki/Heraclitus
Streams Real Time Stream Processing When you attach “late” to a Publisher, you may miss initial elements – it’s a river of data. http://en.wikiquote.org/wiki/Heraclitus
Reactive Streams
Reactive Streams Stream processing
Reactive Streams Back-pressured Stream processing
Reactive Streams Back-pressured Asynchronous Stream processing
Reactive Streams Back-pressured Asynchronous Stream processing Standardised (!)
Reactive Streams: Goals 1. Back-pressured Asynchronous Stream processing 2. Standard implemented by many libraries
Reactive Streams - Specification & TCK http://reactive-streams.org
Reactive Streams - Who? Kaazing Corp. rxJava @ Netflix, reactor @ Pivotal (SpringSource), vert.x @ Red Hat, Twitter, akka-streams @ Typesafe, spray @ Spray.io, Oracle, java (?) – Doug Lea - SUNY Oswego … http://reactive-streams.org
Reactive Streams - Inter-op We want to make different implementations co-operate with each other. http://reactive-streams.org
Reactive Streams - Inter-op The different implementations “talk to each other” using the Reactive Streams protocol. http://reactive-streams.org
Reactive Streams - Inter-op The Reactive Streams SPI is NOT meant to be user-api. You should use one of the implementing libraries. http://reactive-streams.org
What is back-pressure？
Back-pressure? Example Without Publisher[T] Subscriber[T]
Back-pressure? Example Without Fast Publisher Slow Subscriber
Back-pressure? “Why would I need that!?”
Back-pressure? Push + NACK model
Back-pressure? Push + NACK model Subscriber usually has some kind of buffer.
Back-pressure? Push + NACK model
Back-pressure? Push + NACK model
Back-pressure? Push + NACK model What if the buffer overflows?
Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending
Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending Kernel does this! Routers do this! (TCP)
Back-pressure? Push + NACK model (b) Increase buffer size… Well, while you have memory available!
Back-pressure? Push + NACK model (b)
Back-pressure? NACKing is NOT enough.
Negative ACKnowledgement
Back-pressure? Example NACKing Buffer overflow is imminent!
Back-pressure? Example NACKing Telling the Publisher to slow down / stop sending…
Back-pressure? Example NACKing NACK did not make it in time, because M was in-flight!
Back-pressure? speed(publisher) < speed(subscriber)
Back-pressure? Fast Subscriber, No Problem No problem!
Back-pressure? Reactive-Streams = “Dynamic Push/Pull”
Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber
Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber Solution: Dynamic adjustment
Back-pressure? RS: Dynamic Push/Pull Slow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.
Back-pressure? RS: Dynamic Push/Pull Fast Publisher will send at-most 3 elements. This is pull-based-backpressure.
Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives!
Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives. Publisher can accumulate demand.
Back-pressure? RS: Accumulate demand Publisher accumulates total demand per subscriber.
Back-pressure? RS: Accumulate demand Total demand of elements is safe to publish. Subscriber’s buffer will not overflow.
Back-pressure? RS: Requesting “a lot” Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)
Back-pressure? RS: Dynamic Push/Pull MAX speed
Back-pressure? RS: Dynamic Push/Pull Easy MAX speed
How does fit all this?
Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model: An Actor can only: • Send and receive messages • Create Actors • Change it’s behaviour
Akka class Player extends Actor { def receive = { case NextTurn => sender() ! decideOnMove() } def decideOnMove(): Move = ??? }
Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
Akka Streams 0.9 early preview
Akka Streams – Linear Flow
Akka Streams – Linear Flow
Akka Streams – Linear Flow
Akka Streams – Linear Flow
Akka Streams – Linear Flow Flow[Double].map(_.toInt). [...] No Source attached yet. “Pipe ready to work with Doubles”.
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") An ActorSystem is the world in which Actors live in. AkkaStreams uses Actors, so it needs ActorSystem.
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() Contains logic on HOW to materialise the stream.
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() A materialiser chooses HOW to materialise a Stream. The Flow’s AST is fully “lifted”. The Materialiser can choose to materialise the Flow in any way it sees fit. Our implementation uses Actors. But you could easily plug in an SparkMaterializer!
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() You can configure it’s buffer sizes etc.
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = Source(1 to 3).runWith(foreachSink)
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = FlowFrom(1 to 3).runWith(foreachSink)(mat) Uses the implicit FlowMaterializer
Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() // sugar for runWith Source(1 to 3).foreach(println)
Akka Streams – Linear Flow val mf = Flow[Int]. map(_ * 2). runWith(Sink.foreach(println)) // is missing a Source, // can NOT run == won’t compile!
Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run()
Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run() With a Source attached… it can run()
Akka Streams – Linear Flow Flow[Int]. map(_.toString). runWith(Source(1 to 10), Sink.ignore) Connects Source and Sink, then runs
Akka Streams – Flows are reusable f.withSource(IterableSource(1 to 10)).run() f.withSource(IterableSource(1 to 100)).run() f.withSource(IterableSource(1 to 1000)).run()
Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber) Each “group” is a stream too! It’s a “Stream of Streams”.
Akka Streams <-> Actors – Advanced groupBy(_.last). GroupBy groups “11” to group “1”, “12” to group “2” etc.
Akka Streams <-> Actors – Advanced groupBy(_.last). Source It offers (groupKey, subStreamSource) to Subscriber
Akka Streams <-> Actors – Advanced groupBy(_.last). Source It can then start children, to handle the sub-flows!
Akka Streams <-> Actors – Advanced groupBy(_.last). Source For example, one child for each group.
Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber) The Actor, will consume SubStream offers.
Consuming streams with Actors (advanced)
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber { override def requestStrategy = WatermarkRequestStrategy(highWatermark = 10) override def receive = { case OnNext(n: String) => println(s”n = $n”) } }
Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber { override def requestStrategy = WatermarkRequestStrategy(highWatermark = 10) override def receive = { case OnNext(n: String) => println(s”n = $n”) } }
Akka Streams – FlowGraph FlowGraph
Akka Streams – FlowGraph Linear Flows or non-akka pipelines Could be another RS implementation!
Akka Streams – GraphFlow Fan-out elements and Fan-in elements
Akka Streams – GraphFlow // first define some pipeline pieces val f1 = Flow[Input].map(_.toIntermediate) val f2 = Flow[Intermediate].map(_.enrich) val f3 = Flow[Enriched].filter(_.isImportant) val f4 = Flow[Intermediate].mapFuture(_.enrichAsync) // then add input and output placeholders val in = SubscriberSource[Input] val out = PublisherSink[Enriched]
Akka Streams – GraphFlow
Akka Streams – GraphFlow val b3 = Broadcast[Int]("b3") val b7 = Broadcast[Int]("b7") val b11 = Broadcast[Int]("b11") val m8 = Merge[Int]("m8") val m9 = Merge[Int]("m9") val m10 = Merge[Int]("m10") val m11 = Merge[Int]("m11") val in3 = Source(List(3)) val in5 = Source(List(5)) val in7 = Source(List(7))
Akka Streams – GraphFlow
Akka Streams – GraphFlow // First layer in7 ~> b7 b7 ~> m11 b7 ~> m8 in5 ~> m11 in3 ~> b3 b3 ~> m8 b3 ~> m10
Akka Streams – GraphFlow // Second layer m11 ~> b11 b11 ~> Flow[Int].grouped(1000) ~> resultFuture2 b11 ~> m9 b11 ~> m10 m8 ~> m9
Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
Akka Streams – GraphFlow Sinks and Sources are “keys” which can be addressed within the graph val resultFuture2 = Sink.future[Seq[Int]] val resultFuture9 = Sink.future[Seq[Int]] val resultFuture10 = Sink.future[Seq[Int]] val g = FlowGraph { implicit b => // ... m10 ~> Flow[Int].grouped(1000) ~> resultFuture10 // ... }.run() Await.result(g.get(resultFuture2), 3.seconds).sorted should be(List(5, 7))
Akka Streams – GraphFlow Sinks and Sources are “keys” which can be addressed within the graph val resultFuture2 = Sink.future[Seq[Int]] val resultFuture9 = Sink.future[Seq[Int]] val resultFuture10 = Sink.future[Seq[Int]] val g = FlowGraph { implicit b => // ... m10 ~> Flow[Int].grouped(1000) ~> resultFuture10 // ... }.run() Await.result(g.get(resultFuture2), 3.seconds).sorted should be(List(5, 7))
Akka Streams – GraphFlow val g = FlowGraph {} FlowGraph is immutable and safe to share and re-use!
Available Elements 0.9 early preview
Available Sources • FutureSource • IterableSource • IteratorSource • PublisherSource • SubscriberSource • ThunkSource • TickSource (timer based) • … easy to add your own! 0.9 early preview
Available operations • drop / dropWithin • take / takeWithin • filter • groupBy • grouped • map • prefixAndTail • … easy to add your own! “Rate – detaching” operations: • buffer • collect • concat • conflate 0.9 early preview
Available Sinks • BlackHoleSink • FoldSink • ForeachSink • FutureSink • OnCompleteSink • PublisherSink / FanoutPublisherSink • SubscriberSink • … easy to add your own! 0.9 early preview
Available Junctions • Broadcast • Merge • FlexiMerge • Zip • Unzip • Concat • … easy to add your own! 0.9 early preview
Akka-Streams Coming soon!
Rough plans • 0.9 released • 0.10 in 2~3 weeks • 1.0 “soon” after… • Means “stabilised APIs” • Will not yet be performance tuned, though it’s already pretty good… we know where and how we can tune it for 1.x. 0.7 early preview
Java DSL • Partial Java DSL in 0.9 (released) • Full Java DSL in 0.10 (in 2~3 weeks) • as 1st class citizen (!) 0.7 early preview
Spray => Akka-Http && ReactiveStreams Spray is now merged into Akka, as Akka-Http Works on Reactive Streams Streaming end-to-end!
Links • http://akka.io • http://reactive-streams.org • https://groups.google.com/group/akka-user • 0.7 release http://akka.io/news/2014/09/12/akka-streams-0.7-released.html • javadsl https://github.com/akka/akka/pulls?q=is%3Apr+javadsl
Děkuji vám! Dzięki! ありがとう！ Ask questions, get Stickers! (for real!) http://akka.io ktoso @ typesafe.com twitter: ktosopl github: ktoso team blog: letitcrash.com
©Typesafe 2014 – All Rights Reserved

More Related Content

PPTX
Kafka 101
Clement Demonchy
 
PDF
Building robust CDC pipeline with Apache Hudi and Debezium
Tathastu.ai
 
PDF
Automate Your Kafka Cluster with Kubernetes Custom Resources
confluent
 
PPTX
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
 
PPTX
Oracle REST Data Services: Options for your Web Services
Jeff Smith
 
ODP
Introduction To RabbitMQ
Knoldus Inc.
 
PDF
Ingesting and Processing IoT Data Using MQTT, Kafka Connect and Kafka Streams...
confluent
 
PPT
Wireshark Inroduction Li In
mhaviv
 
Kafka 101
Clement Demonchy
 
Building robust CDC pipeline with Apache Hudi and Debezium
Tathastu.ai
 
Automate Your Kafka Cluster with Kubernetes Custom Resources
confluent
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
 
Oracle REST Data Services: Options for your Web Services
Jeff Smith
 
Introduction To RabbitMQ
Knoldus Inc.
 
Ingesting and Processing IoT Data Using MQTT, Kafka Connect and Kafka Streams...
confluent
 
Wireshark Inroduction Li In
mhaviv
 

What's hot (20)

PDF
Sqoop on Spark for Data Ingestion
DataWorks Summit
 
PDF
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
PDF
Event Driven-Architecture from a Scalability perspective
Jonas Bonér
 
PPTX
Best Practices in Security with PostgreSQL
EDB
 
PDF
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Databricks
 
PPTX
Web services SOAP
princeirfancivil
 
PDF
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
PDF
An Introduction to BGP Flow Spec
ShortestPathFirst
 
PPT
Data Loss and Duplication in Kafka
Jayesh Thakrar
 
PPT
Introduction to the Web API
Brad Genereaux
 
PPTX
Logstash
Rajgourav Jain
 
PPTX
Apache Hive Tutorial
Sandeep Patil
 
PDF
Cassandra Introduction & Features
DataStax Academy
 
PPTX
Spark architecture
GauravBiswas9
 
PDF
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Databricks
 
PDF
Developing Real-Time Data Pipelines with Apache Kafka
Joe Stein
 
PDF
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
mumrah
 
PDF
Kafka Deep Dive
Knoldus Inc.
 
PPTX
Kafka Connect - debezium
Kasun Don
 
PDF
Securing Kafka
confluent
 
Sqoop on Spark for Data Ingestion
DataWorks Summit
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
Event Driven-Architecture from a Scalability perspective
Jonas Bonér
 
Best Practices in Security with PostgreSQL
EDB
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Databricks
 
Web services SOAP
princeirfancivil
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
An Introduction to BGP Flow Spec
ShortestPathFirst
 
Data Loss and Duplication in Kafka
Jayesh Thakrar
 
Introduction to the Web API
Brad Genereaux
 
Logstash
Rajgourav Jain
 
Apache Hive Tutorial
Sandeep Patil
 
Cassandra Introduction & Features
DataStax Academy
 
Spark architecture
GauravBiswas9
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Databricks
 
Developing Real-Time Data Pipelines with Apache Kafka
Joe Stein
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
mumrah
 
Kafka Deep Dive
Knoldus Inc.
 
Kafka Connect - debezium
Kasun Don
 
Securing Kafka
confluent
 
Ad

Viewers also liked (20)

PDF
[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)
Konrad Malawski
 
PDF
Fresh from the Oven (04.2015): Experimental Akka Typed and Akka Streams
Konrad Malawski
 
PDF
DDDing Tools = Akka Persistence
Konrad Malawski
 
PDF
The Cloud-natives are RESTless @ JavaOne
Konrad Malawski
 
PDF
Akka Streams and HTTP
Roland Kuhn
 
PDF
Akka persistence == event sourcing in 30 minutes
Konrad Malawski
 
PDF
Akka-chan's Survival Guide for the Streaming World
Konrad Malawski
 
PDF
Android my Scala @ JFokus 2013
Konrad Malawski
 
PDF
100th SCKRK Meeting - best software engineering papers of 5 years of SCKRK
Konrad Malawski
 
PDF
The things we don't see – stories of Software, Scala and Akka
Konrad Malawski
 
PDF
[Japanese] How Reactive Streams and Akka Streams change the JVM Ecosystem @ R...
Konrad Malawski
 
PDF
The Need for Async @ ScalaWorld
Konrad Malawski
 
PDF
Disrupt 2 Grow - Devoxx 2013
Konrad Malawski
 
PDF
Akka Streams in Action @ ScalaDays Berlin 2016
Konrad Malawski
 
PDF
2014 akka-streams-tokyo-japanese
Konrad Malawski
 
PDF
How Reactive Streams & Akka Streams change the JVM Ecosystem
Konrad Malawski
 
PDF
End to End Akka Streams / Reactive Streams - from Business to Socket
Konrad Malawski
 
PDF
Reactive Streams, j.u.concurrent & Beyond!
Konrad Malawski
 
PDF
Distributed Consensus A.K.A. "What do we eat for lunch?"
Konrad Malawski
 
PDF
Reactive Stream Processing with Akka Streams
Konrad Malawski
 
[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)
Konrad Malawski
 
Fresh from the Oven (04.2015): Experimental Akka Typed and Akka Streams
Konrad Malawski
 
DDDing Tools = Akka Persistence
Konrad Malawski
 
The Cloud-natives are RESTless @ JavaOne
Konrad Malawski
 
Akka Streams and HTTP
Roland Kuhn
 
Akka persistence == event sourcing in 30 minutes
Konrad Malawski
 
Akka-chan's Survival Guide for the Streaming World
Konrad Malawski
 
Android my Scala @ JFokus 2013
Konrad Malawski
 
100th SCKRK Meeting - best software engineering papers of 5 years of SCKRK
Konrad Malawski
 
The things we don't see – stories of Software, Scala and Akka
Konrad Malawski
 
[Japanese] How Reactive Streams and Akka Streams change the JVM Ecosystem @ R...
Konrad Malawski
 
The Need for Async @ ScalaWorld
Konrad Malawski
 
Disrupt 2 Grow - Devoxx 2013
Konrad Malawski
 
Akka Streams in Action @ ScalaDays Berlin 2016
Konrad Malawski
 
2014 akka-streams-tokyo-japanese
Konrad Malawski
 
How Reactive Streams & Akka Streams change the JVM Ecosystem
Konrad Malawski
 
End to End Akka Streams / Reactive Streams - from Business to Socket
Konrad Malawski
 
Reactive Streams, j.u.concurrent & Beyond!
Konrad Malawski
 
Distributed Consensus A.K.A. "What do we eat for lunch?"
Konrad Malawski
 
Reactive Stream Processing with Akka Streams
Konrad Malawski
 
Ad

Similar to Reactive Streams / Akka Streams - GeeCON Prague 2014 (20)

PDF
Reactive Streams 1.0 and Akka Streams
Dean Wampler
 
PDF
Reactive integrations with Akka Streams
Konrad Malawski
 
PDF
Reactive stream processing using Akka streams
Johan Andrén
 
PDF
Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Lightbend
 
PPTX
Intro to Akka Streams
Michael Kendra
 
PDF
Streaming all the things with akka streams
Johan Andrén
 
PDF
Understanding Akka Streams, Back Pressure, and Asynchronous Architectures
Lightbend
 
PDF
Building a Reactive System with Akka - Workshop @ O'Reilly SAConf NYC
Konrad Malawski
 
PDF
Akka streams
mircodotta
 
PDF
Asynchronous stream processing with Akka Streams
Johan Andrén
 
PDF
VJUG24 - Reactive Integrations with Akka Streams
Johan Andrén
 
PDF
Build Real-Time Streaming ETL Pipelines With Akka Streams, Alpakka And Apache...
Lightbend
 
PPTX
Intro to Reactive Thinking and RxJava 2
JollyRogers5
 
PDF
Building Stateful Microservices With Akka
Yaroslav Tkachenko
 
ODP
Introduction to Akka Streams [Part-I]
Knoldus Inc.
 
PPTX
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Kafka
Akara Sucharitakul
 
PPTX
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...
Reactivesummit
 
PDF
Akka streams - Umeå java usergroup
Johan Andrén
 
PPTX
Reactive programming for java developers
Constantin Popa
 
PDF
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
Dan Halperin
 
Reactive Streams 1.0 and Akka Streams
Dean Wampler
 
Reactive integrations with Akka Streams
Konrad Malawski
 
Reactive stream processing using Akka streams
Johan Andrén
 
Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Lightbend
 
Intro to Akka Streams
Michael Kendra
 
Streaming all the things with akka streams
Johan Andrén
 
Understanding Akka Streams, Back Pressure, and Asynchronous Architectures
Lightbend
 
Building a Reactive System with Akka - Workshop @ O'Reilly SAConf NYC
Konrad Malawski
 
Akka streams
mircodotta
 
Asynchronous stream processing with Akka Streams
Johan Andrén
 
VJUG24 - Reactive Integrations with Akka Streams
Johan Andrén
 
Build Real-Time Streaming ETL Pipelines With Akka Streams, Alpakka And Apache...
Lightbend
 
Intro to Reactive Thinking and RxJava 2
JollyRogers5
 
Building Stateful Microservices With Akka
Yaroslav Tkachenko
 
Introduction to Akka Streams [Part-I]
Knoldus Inc.
 
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Kafka
Akara Sucharitakul
 
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...
Reactivesummit
 
Akka streams - Umeå java usergroup
Johan Andrén
 
Reactive programming for java developers
Constantin Popa
 
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
Dan Halperin
 

More from Konrad Malawski (11)

PDF
Networks and Types - the Future of Akka @ ScalaDays NYC 2018
Konrad Malawski
 
PDF
Akka Typed (quick talk) - JFokus 2018
Konrad Malawski
 
PDF
ScalaSwarm 2017 Keynote: Tough this be madness yet theres method in't
Konrad Malawski
 
PDF
State of Akka 2017 - The best is yet to come
Konrad Malawski
 
PDF
Not Only Streams for Akademia JLabs
Konrad Malawski
 
PDF
Krakow communities @ 2016
Konrad Malawski
 
PDF
Zen of Akka
Konrad Malawski
 
PDF
Need for Async: Hot pursuit for scalable applications
Konrad Malawski
 
PDF
Open soucerers - jak zacząć swoją przygodę z open source
Konrad Malawski
 
PDF
HBase RowKey design for Akka Persistence
Konrad Malawski
 
PDF
Scalding - the not-so-basics @ ScalaDays 2014
Konrad Malawski
 
Networks and Types - the Future of Akka @ ScalaDays NYC 2018
Konrad Malawski
 
Akka Typed (quick talk) - JFokus 2018
Konrad Malawski
 
ScalaSwarm 2017 Keynote: Tough this be madness yet theres method in't
Konrad Malawski
 
State of Akka 2017 - The best is yet to come
Konrad Malawski
 
Not Only Streams for Akademia JLabs
Konrad Malawski
 
Krakow communities @ 2016
Konrad Malawski
 
Zen of Akka
Konrad Malawski
 
Need for Async: Hot pursuit for scalable applications
Konrad Malawski
 
Open soucerers - jak zacząć swoją przygodę z open source
Konrad Malawski
 
HBase RowKey design for Akka Persistence
Konrad Malawski
 
Scalding - the not-so-basics @ ScalaDays 2014
Konrad Malawski
 

Recently uploaded (20)

PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Advanced IT Governance
University of Hertfordshire
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
Approach and Philosophy of On baking technology
30anthuong17
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Teaching material agriculture food technology
LiaRayya
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
KodekX | Application Modernization Development
KodekX
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 

Reactive Streams / Akka Streams - GeeCON Prague 2014

  • 1. Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Akka Streams Konrad `@ktosopl` Malawski OST: Tomas Dvorak ./‘ ./‘
  • 2. hAkker @ Konrad `@ktosopl` Malawski Akka Team, Reactive Streams TCK
  • 3. hAkker @ Konrad `@ktosopl` Malawski typesafe.com geecon.org Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow
  • 7. You? ? z ? ?
  • 8. You? ? z ? ? ?
  • 11. Streams “You cannot enter the same river twice” ~ Heraclitus http://en.wikiquote.org/wiki/Heraclitus
  • 12. Streams Real Time Stream Processing When you attach “late” to a Publisher, you may miss initial elements – it’s a river of data. http://en.wikiquote.org/wiki/Heraclitus
  • 15. Reactive Streams Back-pressured Stream processing
  • 16. Reactive Streams Back-pressured Asynchronous Stream processing
  • 17. Reactive Streams Back-pressured Asynchronous Stream processing Standardised (!)
  • 18. Reactive Streams: Goals 1. Back-pressured Asynchronous Stream processing 2. Standard implemented by many libraries
  • 19. Reactive Streams - Specification & TCK http://reactive-streams.org
  • 20. Reactive Streams - Who? Kaazing Corp. rxJava @ Netflix, reactor @ Pivotal (SpringSource), vert.x @ Red Hat, Twitter, akka-streams @ Typesafe, spray @ Spray.io, Oracle, java (?) – Doug Lea - SUNY Oswego … http://reactive-streams.org
  • 21. Reactive Streams - Inter-op We want to make different implementations co-operate with each other. http://reactive-streams.org
  • 22. Reactive Streams - Inter-op The different implementations “talk to each other” using the Reactive Streams protocol. http://reactive-streams.org
  • 23. Reactive Streams - Inter-op The Reactive Streams SPI is NOT meant to be user-api. You should use one of the implementing libraries. http://reactive-streams.org
  • 25. Back-pressure? Example Without Publisher[T] Subscriber[T]
  • 26. Back-pressure? Example Without Fast Publisher Slow Subscriber
  • 27. Back-pressure? “Why would I need that!?”
  • 28. Back-pressure? Push + NACK model
  • 29. Back-pressure? Push + NACK model Subscriber usually has some kind of buffer.
  • 30. Back-pressure? Push + NACK model
  • 31. Back-pressure? Push + NACK model
  • 32. Back-pressure? Push + NACK model What if the buffer overflows?
  • 33. Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending
  • 34. Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending Kernel does this! Routers do this! (TCP)
  • 35. Back-pressure? Push + NACK model (b) Increase buffer size… Well, while you have memory available!
  • 36. Back-pressure? Push + NACK model (b)
  • 39. Back-pressure? Example NACKing Buffer overflow is imminent!
  • 40. Back-pressure? Example NACKing Telling the Publisher to slow down / stop sending…
  • 41. Back-pressure? Example NACKing NACK did not make it in time, because M was in-flight!
  • 43. Back-pressure? Fast Subscriber, No Problem No problem!
  • 44. Back-pressure? Reactive-Streams = “Dynamic Push/Pull”
  • 45. Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber
  • 46. Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber Solution: Dynamic adjustment
  • 47. Back-pressure? RS: Dynamic Push/Pull Slow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.
  • 48. Back-pressure? RS: Dynamic Push/Pull Fast Publisher will send at-most 3 elements. This is pull-based-backpressure.
  • 49. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives!
  • 50. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives. Publisher can accumulate demand.
  • 51. Back-pressure? RS: Accumulate demand Publisher accumulates total demand per subscriber.
  • 52. Back-pressure? RS: Accumulate demand Total demand of elements is safe to publish. Subscriber’s buffer will not overflow.
  • 53. Back-pressure? RS: Requesting “a lot” Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)
  • 54. Back-pressure? RS: Dynamic Push/Pull MAX speed
  • 55. Back-pressure? RS: Dynamic Push/Pull Easy MAX speed
  • 56. How does fit all this?
  • 57. Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
  • 58. Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
  • 59. Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model: An Actor can only: • Send and receive messages • Create Actors • Change it’s behaviour
  • 60. Akka class Player extends Actor { def receive = { case NextTurn => sender() ! decideOnMove() } def decideOnMove(): Move = ??? }
  • 61. Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
  • 62. Akka Streams 0.9 early preview
  • 63. Akka Streams – Linear Flow
  • 64. Akka Streams – Linear Flow
  • 65. Akka Streams – Linear Flow
  • 66. Akka Streams – Linear Flow
  • 67. Akka Streams – Linear Flow Flow[Double].map(_.toInt). [...] No Source attached yet. “Pipe ready to work with Doubles”.
  • 68. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") An ActorSystem is the world in which Actors live in. AkkaStreams uses Actors, so it needs ActorSystem.
  • 69. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() Contains logic on HOW to materialise the stream.
  • 70. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() A materialiser chooses HOW to materialise a Stream. The Flow’s AST is fully “lifted”. The Materialiser can choose to materialise the Flow in any way it sees fit. Our implementation uses Actors. But you could easily plug in an SparkMaterializer!
  • 71. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() You can configure it’s buffer sizes etc.
  • 72. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = Source(1 to 3).runWith(foreachSink)
  • 73. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = FlowFrom(1 to 3).runWith(foreachSink)(mat) Uses the implicit FlowMaterializer
  • 74. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() // sugar for runWith Source(1 to 3).foreach(println)
  • 75. Akka Streams – Linear Flow val mf = Flow[Int]. map(_ * 2). runWith(Sink.foreach(println)) // is missing a Source, // can NOT run == won’t compile!
  • 76. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  • 77. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  • 78. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  • 79. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run()
  • 80. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run() With a Source attached… it can run()
  • 81. Akka Streams – Linear Flow Flow[Int]. map(_.toString). runWith(Source(1 to 10), Sink.ignore) Connects Source and Sink, then runs
  • 82. Akka Streams – Flows are reusable f.withSource(IterableSource(1 to 10)).run() f.withSource(IterableSource(1 to 100)).run() f.withSource(IterableSource(1 to 1000)).run()
  • 83. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
  • 84. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber) Each “group” is a stream too! It’s a “Stream of Streams”.
  • 85. Akka Streams <-> Actors – Advanced groupBy(_.last). GroupBy groups “11” to group “1”, “12” to group “2” etc.
  • 86. Akka Streams <-> Actors – Advanced groupBy(_.last). Source It offers (groupKey, subStreamSource) to Subscriber
  • 87. Akka Streams <-> Actors – Advanced groupBy(_.last). Source It can then start children, to handle the sub-flows!
  • 88. Akka Streams <-> Actors – Advanced groupBy(_.last). Source For example, one child for each group.
  • 89. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber) The Actor, will consume SubStream offers.
  • 90. Consuming streams with Actors (advanced)
  • 91. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
  • 92. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
  • 93. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
  • 94. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
  • 95. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber with ImplicitFlowMaterializer with ActorLogging { override def requestStrategy = OneByOneRequestStrategy override def receive = { case OnNext((groupId: String, subStream: Source[String])) => val subSub = context.actorOf(Props[SubStreamSubscriber], s"sub-$groupId") subStream.runWith(Sink.subscriber(ActorSubscriber(subSub))) } }
  • 96. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber { override def requestStrategy = WatermarkRequestStrategy(highWatermark = 10) override def receive = { case OnNext(n: String) => println(s”n = $n”) } }
  • 97. Akka Streams <-> Actors – Advanced class SubStreamParent extends ActorSubscriber { override def requestStrategy = WatermarkRequestStrategy(highWatermark = 10) override def receive = { case OnNext(n: String) => println(s”n = $n”) } }
  • 98. Akka Streams – FlowGraph FlowGraph
  • 99. Akka Streams – FlowGraph Linear Flows or non-akka pipelines Could be another RS implementation!
  • 100. Akka Streams – GraphFlow Fan-out elements and Fan-in elements
  • 101. Akka Streams – GraphFlow // first define some pipeline pieces val f1 = Flow[Input].map(_.toIntermediate) val f2 = Flow[Intermediate].map(_.enrich) val f3 = Flow[Enriched].filter(_.isImportant) val f4 = Flow[Intermediate].mapFuture(_.enrichAsync) // then add input and output placeholders val in = SubscriberSource[Input] val out = PublisherSink[Enriched]
  • 102. Akka Streams – GraphFlow
  • 103. Akka Streams – GraphFlow val b3 = Broadcast[Int]("b3") val b7 = Broadcast[Int]("b7") val b11 = Broadcast[Int]("b11") val m8 = Merge[Int]("m8") val m9 = Merge[Int]("m9") val m10 = Merge[Int]("m10") val m11 = Merge[Int]("m11") val in3 = Source(List(3)) val in5 = Source(List(5)) val in7 = Source(List(7))
  • 104. Akka Streams – GraphFlow
  • 105. Akka Streams – GraphFlow // First layer in7 ~> b7 b7 ~> m11 b7 ~> m8 in5 ~> m11 in3 ~> b3 b3 ~> m8 b3 ~> m10
  • 106. Akka Streams – GraphFlow // Second layer m11 ~> b11 b11 ~> Flow[Int].grouped(1000) ~> resultFuture2 b11 ~> m9 b11 ~> m10 m8 ~> m9
  • 107. Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
  • 108. Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
  • 109. Akka Streams – GraphFlow // Third layer m9 ~> Flow[Int].grouped(1000) ~> resultFuture9 m10 ~> Flow[Int].grouped(1000) ~> resultFuture10
  • 110. Akka Streams – GraphFlow Sinks and Sources are “keys” which can be addressed within the graph val resultFuture2 = Sink.future[Seq[Int]] val resultFuture9 = Sink.future[Seq[Int]] val resultFuture10 = Sink.future[Seq[Int]] val g = FlowGraph { implicit b => // ... m10 ~> Flow[Int].grouped(1000) ~> resultFuture10 // ... }.run() Await.result(g.get(resultFuture2), 3.seconds).sorted should be(List(5, 7))
  • 111. Akka Streams – GraphFlow Sinks and Sources are “keys” which can be addressed within the graph val resultFuture2 = Sink.future[Seq[Int]] val resultFuture9 = Sink.future[Seq[Int]] val resultFuture10 = Sink.future[Seq[Int]] val g = FlowGraph { implicit b => // ... m10 ~> Flow[Int].grouped(1000) ~> resultFuture10 // ... }.run() Await.result(g.get(resultFuture2), 3.seconds).sorted should be(List(5, 7))
  • 112. Akka Streams – GraphFlow val g = FlowGraph {} FlowGraph is immutable and safe to share and re-use!
  • 113. Available Elements 0.9 early preview
  • 114. Available Sources • FutureSource • IterableSource • IteratorSource • PublisherSource • SubscriberSource • ThunkSource • TickSource (timer based) • … easy to add your own! 0.9 early preview
  • 115. Available operations • drop / dropWithin • take / takeWithin • filter • groupBy • grouped • map • prefixAndTail • … easy to add your own! “Rate – detaching” operations: • buffer • collect • concat • conflate 0.9 early preview
  • 116. Available Sinks • BlackHoleSink • FoldSink • ForeachSink • FutureSink • OnCompleteSink • PublisherSink / FanoutPublisherSink • SubscriberSink • … easy to add your own! 0.9 early preview
  • 117. Available Junctions • Broadcast • Merge • FlexiMerge • Zip • Unzip • Concat • … easy to add your own! 0.9 early preview
  • 119. Rough plans • 0.9 released • 0.10 in 2~3 weeks • 1.0 “soon” after… • Means “stabilised APIs” • Will not yet be performance tuned, though it’s already pretty good… we know where and how we can tune it for 1.x. 0.7 early preview
  • 120. Java DSL • Partial Java DSL in 0.9 (released) • Full Java DSL in 0.10 (in 2~3 weeks) • as 1st class citizen (!) 0.7 early preview
  • 121. Spray => Akka-Http && ReactiveStreams Spray is now merged into Akka, as Akka-Http Works on Reactive Streams Streaming end-to-end!
  • 122. Links • http://akka.io • http://reactive-streams.org • https://groups.google.com/group/akka-user • 0.7 release http://akka.io/news/2014/09/12/akka-streams-0.7-released.html • javadsl https://github.com/akka/akka/pulls?q=is%3Apr+javadsl
  • 123. Děkuji vám! Dzięki! ありがとう！ Ask questions, get Stickers! (for real!) http://akka.io ktoso @ typesafe.com twitter: ktosopl github: ktoso team blog: letitcrash.com
  • 124. ©Typesafe 2014 – All Rights Reserved