Introduction to Akka Streams [Part-II]
0 likes1,816 views
This document discusses Akka Streams including flows, operator fusion, and the graph DSL. It provides examples of using flows to perform transformations on streaming data between a source and sink. Operator fusion is described as optimizing performance by mitigating the cost of passing elements between stages. The graph DSL allows modeling data flows that can't be represented linearly, such as those involving fan-in and fan-out patterns. Partial graphs are discussed as reusable components that can be used to construct graphs according to different requirements.
![Akka Stream: Example
object StreamingCopyApp extends App {
val spath = Paths.get("/home/harmeet/akkastream/knolx.log")
val source: Source[ByteString, Future[IOResult]] =
FileIO.fromPath(spath)
val dpath = Paths.get("/home/harmeet/akkastreamcopy")
val sink: Sink[ByteString, Future[IOResult]] = FileIO.toPath(dpath)
val runnableGraph: RunnableGraph[Future[IOResult]] = source.to(sink)
implicit val system = ActorSystem("akkastream")
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
runnableGraph.run().foreach { result =>
println(s"${result.status}, ${result.count} bytes read")
system.terminate()
}
}](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-5-320.jpg)
![Akka Stream: Flow Example
val frame: Flow[ByteString, String, NotUsed] =
Framing.delimiter(ByteString("n"), 10240).map(_.decodeString("UTF8"))
val parse: Flow[String, Event, NotUsed] = Flow[String].map(Event.parsing)
val filter: Flow[Event, Event, NotUsed] = Flow[Event].filter(_.log ==
"DEBUG")
val serialize: Flow[Event, ByteString, NotUsed] = Flow[Event].map{ event =>
ByteString(event.toJson.compactPrint)
}
val composedFlow: Flow[ByteString, ByteString, NotUsed] = frame.via(parse)
.via(filter)
.via(serialize)
val runnableGraph: RunnableGraph[Future[IOResult]] =
source.via(composedFlow).toMat(sink) (Keep.right)](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-6-320.jpg)
![Akka Stream: Flow Example
val composedFlow: Flow[ByteString, ByteString, NotUsed] =
Framing.delimiter(ByteString("n"), 10240)
.map(_.decodeString("UTF8"))
.map(Event.parsing)
.filter(_.log == "DEBUG")
.map { event =>
ByteString(event.toJson.compactPrint)
}
val runnableGraph: RunnableGraph[Future[IOResult]] =
source.via(composedFlow)
.toMat(sink) (Keep.right)](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-7-320.jpg)
![Akka Stream: Graph DSL Example
val in = Source(1 to 5)
val f1 = Flow[Int].map(_ * 2)
val f2 = Flow[Int].map(_ * 1)](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-13-320.jpg)
![Akka Stream: Graph DSL Example
val f3 = Flow[Int].map(_ * 2)
val f4 = Flow[Int].map(_ + 1)
val out = Sink.foreach[Int](println)](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-14-320.jpg)
)
val merge = builder.add(Merge[Int](2))](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-15-320.jpg)
![Akka Stream: Graph DSL Example
val graph = RunnableGraph.fromGraph(GraphDSL.create() {
implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source(1 to 5)
val out = Sink.foreach[Int](println)
val f1 = Flow[Int].map(_ * 2)
val f2 = Flow[Int].map(_ * 1)
val f3 = Flow[Int].map(_ * 2)
val f4 = Flow[Int].map(_ + 1)
val bcast = builder.add(Broadcast[Int](2))
val merge = builder.add(Merge[Int](2))
in ~> f1 ~> bcast ~> f2 ~> merge ~> f4 ~> out
bcast ~> f3 ~> merge
ClosedShape
})](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-18-320.jpg)
![Introduction to Akka Streams [Part-II]](https://image.slidesharecdn.com/babystepsinakkastreampart-ii1-171219071747/85/Introduction-to-Akka-Streams-Part-II-23-320.jpg)
Introduction to Akka Streams [Part-II]
- 1. By: Harmeet Singh (Taara) Sr. Software Consultant
- 2. Agenda 1. Flows 2. Operator Fusion. 3. Graph DSL. 4. Leftover.
- 3. Akka Stream: Flows The way to perform transformations to the data as it flows between Source and Sink.
- 6. Akka Stream: Flow Example val frame: Flow[ByteString, String, NotUsed] = Framing.delimiter(ByteString("n"), 10240).map(_.decodeString("UTF8")) val parse: Flow[String, Event, NotUsed] = Flow[String].map(Event.parsing) val filter: Flow[Event, Event, NotUsed] = Flow[Event].filter(_.log == "DEBUG") val serialize: Flow[Event, ByteString, NotUsed] = Flow[Event].map{ event => ByteString(event.toJson.compactPrint) } val composedFlow: Flow[ByteString, ByteString, NotUsed] = frame.via(parse) .via(filter) .via(serialize) val runnableGraph: RunnableGraph[Future[IOResult]] = source.via(composedFlow).toMat(sink) (Keep.right)
- 7. Akka Stream: Flow Example val composedFlow: Flow[ByteString, ByteString, NotUsed] = Framing.delimiter(ByteString("n"), 10240) .map(_.decodeString("UTF8")) .map(Event.parsing) .filter(_.log == "DEBUG") .map { event => ByteString(event.toJson.compactPrint) } val runnableGraph: RunnableGraph[Future[IOResult]] = source.via(composedFlow) .toMat(sink) (Keep.right)
- 8. Akka Stream: Operator Fusion Operator Fusion is used to performance optimization to mitigate the cost of passing elements between the different stages. Two main consequences are: 1. Passing elements fused stages faster. 2. Fused stages no longer run in parallel.
- 9. Akka Stream: Operator Fusion Q. How can we execute fused stages in parallel??? Ans: 1. By using async method 2. Turn off auto-fusing (akka.stream.materializer.auto fusing=off) Example: Source(List(1, 2, 3)).map(_ + 1) .async .map(_ * 2).to(Sink.ignore)
- 10. Akka Stream: Operator Fusion
- 11. Akka Stream: Graph DSL What if your data flow cannot be modeled as a simple linear process? What if process is better modeled as computation graph instead?
- 12. Akka Stream: Graph DSL The Graph DSL is kind of diagram ASCII art – in many cases you could translate a white board diagram of a graph into the DSL. A graph problem is one that typically involves the concept of fanning out and fanning in. ● Fanning out: Single input to multiple output. ● Fanning in: Multiple inputs single output.
- 13. Akka Stream: Graph DSL Example val in = Source(1 to 5) val f1 = Flow[Int].map(_ * 2) val f2 = Flow[Int].map(_ * 1)
- 14. Akka Stream: Graph DSL Example val f3 = Flow[Int].map(_ * 2) val f4 = Flow[Int].map(_ + 1) val out = Sink.foreach[Int](println)
- 15. Akka Stream: Graph DSL Example val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2))
- 16. Akka Stream: Graph DSL Example in ~> f1 ~> bcast ~> f2 ~> merge ~> f4 ~> out
- 17. Akka Stream: Graph DSL Example bcast ~> f3 ~> merge
- 18. Akka Stream: Graph DSL Example val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] => import GraphDSL.Implicits._ val in = Source(1 to 5) val out = Sink.foreach[Int](println) val f1 = Flow[Int].map(_ * 2) val f2 = Flow[Int].map(_ * 1) val f3 = Flow[Int].map(_ * 2) val f4 = Flow[Int].map(_ + 1) val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) in ~> f1 ~> bcast ~> f2 ~> merge ~> f4 ~> out bcast ~> f3 ~> merge ClosedShape })
- 19. Akka Stream: Graph DSL ClosedShape Thegraph itself wasfully self-contained and complete. OR It doesnot haveany open input port and output port like aclosed circuit.
- 20. Akka Stream: Partial Graphs Whenever weneed to createareusablecomponent for graph, which wecan useaccording to requirementswith existing or new graph. https://doc.akka.io/docs/akka/2.5/scala/stream/stream-graphs.html#constructing-and-combinin
- 21. Leftovers ➔ API’s ➔ Error Handling. ➔ Streaming HTTP ➔ Handling IO Stream ➔ Unit Testing
- 22. References: ✔ AkkainAction by Raymond Roestenburg, Rob Bakker, Rob Williams. ✔ MasteringAkkaby Christian Baxter ✔ https://doc.akka.io/api/akka/current/akka/stream/index.html?_ga=2.25006541