Risking Everything with Akka Streams

H O P E S, R E G RETS A N D
“ B E ST P R ACTICES”
RISKING
EVERYT HING
W IT H
AKKA ST REAMS
08-12-20 1 6
JOACHIM HOFER
(@JOHOF ER )

2
0. What We Do
1. First Impressions
2. Lessons Learnt
3. Awesomeness
4. Ops
5. Verdict
T ABLE OF
CONT ENTS

4
net sales 2015: 3 billion euros
several 1000 updates / second
latency: ideally seconds
WHAT WE DO: SOME NUMBERS

6
Akka Streams vs. Futures, RxScala, Actors
Tech Blog “Which shoe fits you?”
https://github.com/zalando/
scala-concurrency-playground
Akka Streams fits us best!
Image: Nevit Dilmen (CC BY-SA 3.0)
FIRST IMPRESSIONS — PREPARATION

7
H U H?
FIRST IMPRESSIONS — GRAPH DSL
import GraphDSL.Implicits._
val bcSqsIn = b add Broadcast[StreamCreationEvent](2)
val rules = b add ruleStore.stage.async
val eval = b add evaluator.stage.async
val publish = b add nakadi.stage.async
val ack = b add ackStage.stage.async
bcSqsIn ~> rules ~> eval.in0
bcSqsIn ~> eval.in1; eval.out ~> publish ~> ack
FlowShape(bcSqsIn.in, ack.out)

8
A H !
FIRST IMPRESSIONS — PLAIN SOURCES
products
.mapAsync(parallelism = 5)(ruleEvaluatorEvent(flowId))
.groupedWithin(3, 100 millis)
.filter(_.nonEmpty)
.mapAsync(parallelism = 50)(sqsGateway.send)
.runForeach(result => log.info(result.getFailed.size))

9 Images: Jerry Daykin (CC BY 2.0, left), The Present Group (CC BY 3.0 US, right)
FIRST IMPRESSIONS — OTHER BIG SCARIES
MAT ERIALISATION B A C K P R E S SU R E

1 0
T AST Y DOCUMENTATION
Image: Wikivisual (CC BY-NC-SA 3.0)

1 1
MISTAKES W ERE MADE
Image: Hapesoft (public domain)

1 2
Caused by onError
Completes the stream
Recover using recover / recoverWithRetries
FAILURES (VS ERRORS)

1 3
Caused by exceptions
Get escalated to failures by default
Recover using a Supervision Strategy
Can be ignored easily (“Resume” strategy)
ERRORS (VS FAILURES)

1 4
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E1

1 5
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
1

1 6
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1

1 7
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2

1 8
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2
2

1 9
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2
2

2 0
G O IN G O U T - OF-SYNC
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
12
2

2 1
Unconsumed response entities
Solution: discardEntityBytes
LESSONS LEARNT — AKKA-HT TP CLIENTS (1)

2 2
No response from the server
“Currently Akka HTTP doesn’t implement client-side request timeout
checking itself as this functionality can be regarded as a more general
purpose streaming infrastructure feature.” (akka http docs)
Solution: Use an explicit xxxTimeout stage
LESSONS LEARNT — AKKA-HT TP CLIENTS (2)

2 3
Default: only up to 16?!
16 (buffer size) x 50 (parallel flows) x 10 (events per batch) = 8000 events
Solution: keep low, use explicit buffer stages
LESSONS LEARNT — INTERNAL BUFFERS

2 4
C O N FIGURATION A S A S O U RCE?
Image: Alpha (CC BY-SA 2.0)
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
CONFI GU R A TI ON

2 6
Backpressure
Automatically asynchronous
AWESOMENESS — REACTIVE

2 7
AWESOMENESS — TESTABILITY
EVALUAT ETEST EVENTS OUTPUT S TO CHECK

2 8
groupedWithin
throttle
mapConcat
recoverWithRetries
…
AWESOMENESS — BUILT -I N STAGES
Flow[RuleEvaluationEvent]
.groupedWithin(batchSize, batchTimeout)
.mapAsync(parallelism = 1)(provider.deleteFromStreamCreation)
.mapConcat(identity)
.via(logAndMonitor(Checkpoint.Acknowledged, "acknowledged msg"))

2 9
no out-of-the-box solution
built-in stage monitor not very helpful
no access to internal buffers
OPS — MONITORI NG: THE BAD

3 0
Trace events along streams
Create your own monitoring stage
OPS — MONITORI NG: THE GOOD

3 1
E X A MPLE P A S S -THROUGH S T AG E L O G IC
OPS — MONITORI NG
…
new GraphStageLogic(shape) {
setHandlers(in, out, new InHandler with OutHandler {
override def onPush(): Unit = {
push(out, grab(in))
}
override def onPull(): Unit = {
pull(in)
}
})
}
…

3 2
E X A MPLE P A S S -THROUGH S T AG E L O G IC
OPS — MONITORI NG
…
new GraphStageLogic(shape) {
setHandlers(in, out, new InHandler with OutHandler {
override def onPush(): Unit = {
stats.countPush()
push(out, grab(in))
}
override def onPull(): Unit = {
stats.countPull()
pull(in)
}
})
}
…

3 3
easy to tune
very efficient
easy to scale
see also: Gearpump Materializer
Image: Duncan Rawlinson (CC BY 2.0)
OPS — TUNING AND SCALING

3 4
everything under control
just keeps on running
OPS — RELIABILITY

3 6
Takes time to understand
Very potent
Image: Twice25 (CC BY-SA 2.5)
VERDICT: IT’S MAGIC!

joachim.hofer@zalando.de
@johofer
J O A CHIM H O F ER
Availability Engineering
Backend Engineer
08-12-20 1 6

Risking Everything with Akka Streams

More Related Content

Similar to Risking Everything with Akka Streams (20)

Recently uploaded (20)

Risking Everything with Akka Streams

Editor's Notes