A dive into akka streams: from the basics to a real-world scenario
26 likes17,416 views
The document discusses the use of Akka Streams in a data processing pipeline, detailing the transition from traditional methods to a reactive architecture. It outlines key concepts, such as the functioning of source, flow, and sink, as well as backpressure mechanisms, and how to create custom processing stages. Additionally, it summarizes important lessons learned during the implementation process, including the need for effective recovery mechanisms and the richness of the Akka Streams API.
![val source = Source(1 to 42)
val flow = Flow[Int].map(_ + 1)
val sink = Sink.foreach(println)
val graph = source.via(flow).to(sink)
graph.run()](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-14-320.jpg)
![Flow[ByteString]
.via(chunkBuffer)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-19-320.jpg)
![Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-21-320.jpg)
![Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-22-320.jpg)
![class PersistentBuffer[A] (...)
extends GraphStage[FlowShape[A, A]] {
val in =
Inlet[A]("PersistentBuffer.in")
val out =
Outlet[A]("PersistentBuffer.out")
override val shape = FlowShape.of(in,out)
When the available stages are not enough, write your own](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-23-320.jpg)
![Custom stages: State
override def createLogic(
attr: Attributes
): GraphStageLogic =
new GraphStageLogic(shape) {
var state: StageState[A] = initialState[A]](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-24-320.jpg)
![val cb = getAsyncCallback[Try[A]] {
case Success(elements:A) =>
state = state.copy(...)
...
}
queue.getCurrentQueue
.onComplete(cb.invoke)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-25-320.jpg)
![Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)
.via(kinesisPublisher)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-28-320.jpg)
![Delegating work to actors
Flow[KinesisEvent]
.mapAsync(1) { event =>
(publisher ? Publish(event))
.mapTo[PublishCompleted]
}](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-29-320.jpg)
![Flow[KinesisEvent]
.mapAsync(1) { event =>
(publisher ? Publish(event))
.mapTo[Future[PublishCompleted]]
.flatten
}](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-30-320.jpg)
![Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)
.via(kinesisPublisher)
.alsoTo(countEvents)](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-31-320.jpg)
![Extra side-effects
val countEvents = Sink.foreach[Event] { _ =>
metricEventsPerTimeUnitOfSensor.mark()
metricEventsPerTimeUnit.mark()
}](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-32-320.jpg)
![Materialized Values
val sink: Sink[Any, Future[Done]] = Sink.ignore
.runWith(sink) is a shortcut for .toMat(sink)(Keep.Right).run
STREAM RUN
Obtain a materialized
value per each stage](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-34-320.jpg)
![Materialized Values Example: TestKit
val myFlow = Flow[String].map { v => v.take(1) }
val (pub, sub) = TestSource.probe[String]
.via(myFlow)
.toMat(TestSink.probe[Any])(Keep.both)
.run()
sub.request(1)
pub.sendNext("Gathering")
sub.expectNext("G")](https://image.slidesharecdn.com/adiveintoakkastreams-fromthebasicstoarealworldscenario-160515091425/85/A-dive-into-akka-streams-from-the-basics-to-a-real-world-scenario-35-320.jpg)
A dive into akka streams: from the basics to a real-world scenario
- 1. A dive into Akka Streams From the basics to a real-world scenario Gioia Ballin, Simone Fabiano - SW developers
- 2. Who are these guys? Gioia Ballin Twitter / @GioiaBallin Simone Fabiano Twitter / @mone_tpatpc R&D Data Analysis Microservices Infrastructure WebBlog
- 3. Agenda The Problem Akka Streams Basics The Collector Service Recap and Learnings
- 4. Data processing pipeline COLLECTOR SVC AMAZON KINESIS WI-FI/3GSENSORS
- 5. The Collector Service: two months ago... AMAZON KINESIS INBOX ACTOR TCP Kinesis Client INBOX ACTOR Kinesis Client INBOX ACTOR Kinesis Client TCP TCP chunks chunks chunks events events events put put put
- 7. Data may be lost COLLECTOR SVC AMAZON KINESIS WI-FI/3GSENSORS CONNECTION ISSUES MESSAGE OVERLOAD
- 8. Not REALLY reactive: why? Lack of... RECOVERY MECHANISMS BACKPRESSURE
- 9. Recovery mechanisms Upload failure? Retry Kinesis unavailable? Buffer
- 10. Ingestion speed under control Take advantage of sensors buffer Sensors switch from busier to less busy services
- 11. Welcome Akka Streams! Backpressure on the network + Easy fit for stream paradigm Typed flow & Increased Readability
- 12. Akka Streams Basics - 3 2 1 Go!
- 13. SOURCE FLOW SINK
- 14. val source = Source(1 to 42) val flow = Flow[Int].map(_ + 1) val sink = Sink.foreach(println) val graph = source.via(flow).to(sink) graph.run()
- 15. Flows are collections (almost) OK map, filter, fold, … No flatMap! (there’s mapConcat though) Extras mapAsync, mapAsyncUnordered, buffer..
- 16. How backpressure works Source Flow Sink pull pull push push
- 17. Collector with Akka Streams
- 18. path("data") { post { extractRequest { request => val source = request.entity.dataBytes val flow = processing() val sink = Sink.ignore source.via(flow).runWith(sink) … } }
- 20. Transformation tools: Framing.delimiter val chunkBuffer = Framing.delimiter( ByteString("n"), maxPresenceEventBytes, false ) .map(_.dropRight(1)) .map(_.utf8String)
- 23. class PersistentBuffer[A] (...) extends GraphStage[FlowShape[A, A]] { val in = Inlet[A]("PersistentBuffer.in") val out = Outlet[A]("PersistentBuffer.out") override val shape = FlowShape.of(in,out) When the available stages are not enough, write your own
- 24. Custom stages: State override def createLogic( attr: Attributes ): GraphStageLogic = new GraphStageLogic(shape) { var state: StageState[A] = initialState[A]
- 25. val cb = getAsyncCallback[Try[A]] { case Success(elements:A) => state = state.copy(...) ... } queue.getCurrentQueue .onComplete(cb.invoke)
- 26. Custom Stages: ports setHandler(in, new InHandler { override def onPush() = { val element = grab(in) pull(in) ... } ... }) setHandler(out, new OutHandler { override def onPull(): Unit = { push(out, something) } ... })
- 27. Custom Stages: Start and stop override def postStop(): Unit = { ... } override def preStart(): Unit = { ... }
- 29. Delegating work to actors Flow[KinesisEvent] .mapAsync(1) { event => (publisher ? Publish(event)) .mapTo[PublishCompleted] }
- 30. Flow[KinesisEvent] .mapAsync(1) { event => (publisher ? Publish(event)) .mapTo[Future[PublishCompleted]] .flatten }
- 32. Extra side-effects val countEvents = Sink.foreach[Event] { _ => metricEventsPerTimeUnitOfSensor.mark() metricEventsPerTimeUnit.mark() }
- 33. Automatic Fusion and Async Stages A stream is handled by 1 thread NO CROSS-THREAD COMMUNICATIONS NO PARALLELISM FASTER CODE! SLOWER CODE! A boundary will split your stream on different threads source .via(doSomething).async .via(doSomething).async .runWith(Sink.foreach(println))
- 34. Materialized Values val sink: Sink[Any, Future[Done]] = Sink.ignore .runWith(sink) is a shortcut for .toMat(sink)(Keep.Right).run STREAM RUN Obtain a materialized value per each stage
- 35. Materialized Values Example: TestKit val myFlow = Flow[String].map { v => v.take(1) } val (pub, sub) = TestSource.probe[String] .via(myFlow) .toMat(TestSink.probe[Any])(Keep.both) .run() sub.request(1) pub.sendNext("Gathering") sub.expectNext("G")
- 36. Stream Ending: Supervisioning val flow = otherFlow .withAttributes ActorAttributes.supervisionStrategy { case _ => Supervision.resume } )
- 37. Stream Ending: Supervisioning RESUME RESET STOP > drop failing elem > keep going > drop failing elem > reset stage state > keep going fail all the things!
- 38. Other shapes FAN OUT FAN IN WHATEVER
- 40. In two months to production... Memory Leaks Data Losses Backpressure on the network Recovery mechanisms
- 41. Key learnings Actors aren’t sufficient to be reactive Backpressure as a building block Akka Streams API is rich
- 42. THANK YOU! Sleep more worry less