SlideShare a Scribd company logo

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Andrzej Ludwikowski, profile picture
Andrzej Ludwikowski

The document provides an in-depth exploration of event sourcing, detailing its history, benefits, implementation levels, and various considerations including performance, scalability, and eventual consistency. It discusses the challenges and complexities involved in using event sourcing with specific frameworks like Akka and Lagom, while also addressing serialization, schema evolution, and event handling. Additionally, it highlights alternatives to event sourcing and the implications of using techniques such as change data capture and command sourcing.

Software
Related topics:
Software Architecture
1 of 128
Downloaded 11 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
Event Sourcing what could possibly go wrong? Andrzej Ludwikowski
About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski ➔ academy.softwaremill.com
SoftwareMill
SoftwareMill
SoftwareMill
SoftwareMill JVM BLOGGERS
SoftwareMill ● Hibernate Envers ● Alpakka Kafka ● Sttp ● Scala Clippy ● Quicklens ● MacWire
What is Event Sourcing? DB Order { items=[itemA, itemB] }
What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability ● microservices integration pattern
ES and CQRS Client Query Service Data access Queries Read model Read model Read models Command Service Domain Events Commands
ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Transaction Command Service Domain Events Commands
ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● Implementing event sourcing using a relational database ● slick-eventsourcing
ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction Command Service Domain Events Commands
ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load ? - lags
ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction event bus Command Service Domain Events Commands
ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction At-least-once delivery Command Service Domain Events Commands
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction ?
ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation - locking vs Single Writer
ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of DB?
Not covered but worth to check ● Command Sourcing ● Event Collaboration
ES implementation? ● custom ● library ● framework
ES from domain perspective ● commands, events, state ● 2 methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
ES packaging ● github.com/aludwiko/event-sourcing-akka-persistence
import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} case class User (userId: UserId, name: String, email: Email, funds: Funds) { def process(command: UserCommand): Either[List[UserEvent]] = command match { case c: UpdateEmail => updateEmail(c) case c: DepositFunds => deposit(c) case c: WithdrawFunds => withdraw(c) ... } def apply(event: UserEvent): User = ??? //pattern matching }
ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
ES packaging ● domain logic ● domain validation ● 0 ES framework imports
library vs. framework ● Akka Persistence vs. Lagom ● Akka Persistence Typed
Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
Plain text Binary human-readable deserialization required
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
Speed https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
Size https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
● backward - V2 can read V1 Full compatibility Application Events V1 V2
● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
● forward - V2 can read V3 Full compatibility Events Read model Read model Read models Projector V2 V3
Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
Protobuf schema management + clean domain - a lot of boilerplate code
Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
Avro deserialization Bytes Deserialization Object Reader Schema Writer Schema
Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
Avro vs. Protocol Buffers ● The best serialization strategy for Event Sourcing
Event payload ● delta event ● rich event (event enrichment) ● + metadata ○ seq_num ○ created_at ○ event_id ○ command_id ○ correlation_id
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
State replay time ● snapshotting ● write-through cache
Memory consumption
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
Fixing state ● healing command
Updating all aggregates User(id) Command(user_id) Event(user_id) Event(user_id) Event(user_id)
Handling duplicates Events Read model Read model Read models Projector event bus At-least-once delivery
https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/
Handling duplicates Events Read model Read model Read models Projector event bus At-least-once delivery Events
Handling duplicates Events Read model Read model Read models Projector event bus idempotent updates Events
Event + seq_no Event + seq_no Handling duplicates Events Read model Read model Read models Projector event bus Event + seq_no read model update + seq_no Events
Broken read model Events ad model ead model Read models Projector event bus Events
Broken read model Events ad model ead model Read models Projector event bus read model update + offset (manual offset management) Events
Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
Pessimistic compensation action User account Cinema show
Pessimistic compensation action User account Cinema show charged
Pessimistic compensation action User account Cinema show charged booked
Pessimistic compensation action User account Cinema show charged booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out refund
Optimistic compensation action User account Cinema show charged booked sold out
Optimistic compensation action User account Cinema show booked booked sold out overbooked
Optimistic compensation action User account Cinema show charged booked booked sold out overbooked ?
Saga Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Updater event bus Transaction
Saga - choreography Command Service Domain Events Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
Saga - orchestration Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
Saga ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
Saga ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
Cluster = split brain 1 5 4 3 Load balancer 2
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
Cluster = split brain 1 5 4 3 Load balancer 2
Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
Summary ● carefully choose ES lib/framework ● there is no perfect database for event sourcing ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
Want to learn more? ● Reactive Event Sourcing with Akka (Java/Scala) ● Akka Typed Fundamentals (Java/Scala) ● Akka Cluster ● Performance tests with Gatling
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
Rate me please :)
Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
WE WANT YOU
Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski

More Related Content

PDF
Introduction To Appium With Robotframework
Syam Sasi
 
PPTX
ATLAS Automation POC
aakashmc
 
PPT
Content Management With Apache Jackrabbit
Jukka Zitting
 
PDF
Getting Started With Cypress
Knoldus Inc.
 
PPT
An Introduction to Solr
tomhill
 
PPTX
Schema registry
Whiteklay
 
PPTX
Evaluation of TPC-H on Spark and Spark SQL in ALOJA
DataWorks Summit
 
PDF
SplunkSummit 2015 - A Quick Guide to Search Optimization
Splunk
 
Introduction To Appium With Robotframework
Syam Sasi
 
ATLAS Automation POC
aakashmc
 
Content Management With Apache Jackrabbit
Jukka Zitting
 
Getting Started With Cypress
Knoldus Inc.
 
An Introduction to Solr
tomhill
 
Schema registry
Whiteklay
 
Evaluation of TPC-H on Spark and Spark SQL in ALOJA
DataWorks Summit
 
SplunkSummit 2015 - A Quick Guide to Search Optimization
Splunk
 

What's hot (20)

PDF
Tame the small files problem and optimize data layout for streaming ingestion...
Flink Forward
 
PPTX
Apache Pulsar: Why Unified Messaging and Streaming Is the Future - Pulsar Sum...
StreamNative
 
PDF
What is WebElement in Selenium | Web Elements & Element Locators | Edureka
Edureka!
 
PPTX
Apache Flink and what it is used for
Aljoscha Krettek
 
ODP
Introduction to BDD
Knoldus Inc.
 
PDF
Selenium with Cucumber
Knoldus Inc.
 
PDF
API_Testing_with_Postman
Mithilesh Singh
 
PDF
Postman
Igor Shubovych
 
PDF
5 Levels of Agile Planning Explained Simply
Russell Pannone
 
PPT
Introduction to Hibernate
Krishnakanth Goud
 
PDF
Appium & Robot Framework
Furkan Ertürk
 
PDF
Cypress Automation Testing Tutorial (Part 1).pdf
bacancytechnology
 
PPTX
Agile vs waterfall
Lohet Ramesh
 
PPTX
BDD presentation
temebele
 
PPTX
REST API testing with SpecFlow
Aiste Stikliute
 
PPTX
Test Automation and Selenium
Karapet Sarkisyan
 
PDF
Successfully Implementing BDD in an Agile World
SmartBear
 
PPTX
Data driven Automation Framework with Selenium
Edureka!
 
PPT
How Retail Banks Use MongoDB
MongoDB
 
PPT
Agile Scrum Methodology
Rajeev Misra
 
Tame the small files problem and optimize data layout for streaming ingestion...
Flink Forward
 
Apache Pulsar: Why Unified Messaging and Streaming Is the Future - Pulsar Sum...
StreamNative
 
What is WebElement in Selenium | Web Elements & Element Locators | Edureka
Edureka!
 
Apache Flink and what it is used for
Aljoscha Krettek
 
Introduction to BDD
Knoldus Inc.
 
Selenium with Cucumber
Knoldus Inc.
 
API_Testing_with_Postman
Mithilesh Singh
 
Postman
Igor Shubovych
 
5 Levels of Agile Planning Explained Simply
Russell Pannone
 
Introduction to Hibernate
Krishnakanth Goud
 
Appium & Robot Framework
Furkan Ertürk
 
Cypress Automation Testing Tutorial (Part 1).pdf
bacancytechnology
 
Agile vs waterfall
Lohet Ramesh
 
BDD presentation
temebele
 
REST API testing with SpecFlow
Aiste Stikliute
 
Test Automation and Selenium
Karapet Sarkisyan
 
Successfully Implementing BDD in an Agile World
SmartBear
 
Data driven Automation Framework with Selenium
Edureka!
 
How Retail Banks Use MongoDB
MongoDB
 
Agile Scrum Methodology
Rajeev Misra
 
Ad

Similar to Event sourcing - what could possibly go wrong ? Devoxx PL 2021 (20)

PDF
Event Sourcing - what could go wrong - Jfokus 2022
Andrzej Ludwikowski
 
PDF
Event Sourcing - what could go wrong - Devoxx BE
Andrzej Ludwikowski
 
PDF
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?
SegFaultConf
 
PDF
Event Sourcing - what could possibly go wrong?
Andrzej Ludwikowski
 
PDF
Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemo...
Codemotion
 
PDF
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Anton Kirillov
 
PPTX
Docker & ECS: Secure Nearline Execution
Brennan Saeta
 
PDF
Building a serverless company on AWS lambda and Serverless framework
Luciano Mammino
 
PDF
Spring data requery
Sunghyouk Bae
 
PDF
Architecting Alive Apps
Jorge Ortiz
 
PDF
Immutable Deployments with AWS CloudFormation and AWS Lambda
AOE
 
PDF
Big Data Tools in AWS
Shu-Jeng Hsieh
 
PPTX
FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time...
Zhenzhong Xu
 
PDF
Online Meetup: Why should container system / platform builders care about con...
Docker, Inc.
 
PDF
GTS Episode 1: Reactive programming in the wild
Omer Iqbal
 
PDF
Serverless Java on Kubernetes
Krzysztof Sobkowiak
 
PDF
Lightbend Lagom: Microservices Just Right
mircodotta
 
PPTX
Practical AngularJS
Wei Ru
 
PPTX
Fabric - Realtime stream processing framework
Shashank Gautam
 
PDF
Anatomy of an action
Gordon Chung
 
Event Sourcing - what could go wrong - Jfokus 2022
Andrzej Ludwikowski
 
Event Sourcing - what could go wrong - Devoxx BE
Andrzej Ludwikowski
 
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?
SegFaultConf
 
Event Sourcing - what could possibly go wrong?
Andrzej Ludwikowski
 
Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemo...
Codemotion
 
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Anton Kirillov
 
Docker & ECS: Secure Nearline Execution
Brennan Saeta
 
Building a serverless company on AWS lambda and Serverless framework
Luciano Mammino
 
Spring data requery
Sunghyouk Bae
 
Architecting Alive Apps
Jorge Ortiz
 
Immutable Deployments with AWS CloudFormation and AWS Lambda
AOE
 
Big Data Tools in AWS
Shu-Jeng Hsieh
 
FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time...
Zhenzhong Xu
 
Online Meetup: Why should container system / platform builders care about con...
Docker, Inc.
 
GTS Episode 1: Reactive programming in the wild
Omer Iqbal
 
Serverless Java on Kubernetes
Krzysztof Sobkowiak
 
Lightbend Lagom: Microservices Just Right
mircodotta
 
Practical AngularJS
Wei Ru
 
Fabric - Realtime stream processing framework
Shashank Gautam
 
Anatomy of an action
Gordon Chung
 
Ad

More from Andrzej Ludwikowski (8)

PDF
Event-driven systems without pulling your hair out
Andrzej Ludwikowski
 
PDF
Performance tests - it's a trap
Andrzej Ludwikowski
 
PDF
Cassandra lesson learned - extended
Andrzej Ludwikowski
 
PDF
Performance tests with Gatling (extended)
Andrzej Ludwikowski
 
PPTX
Stress test your backend with Gatling
Andrzej Ludwikowski
 
PPTX
Performance tests with Gatling
Andrzej Ludwikowski
 
PDF
Cassandra - lesson learned
Andrzej Ludwikowski
 
PPTX
Annotation processing tool
Andrzej Ludwikowski
 
Event-driven systems without pulling your hair out
Andrzej Ludwikowski
 
Performance tests - it's a trap
Andrzej Ludwikowski
 
Cassandra lesson learned - extended
Andrzej Ludwikowski
 
Performance tests with Gatling (extended)
Andrzej Ludwikowski
 
Stress test your backend with Gatling
Andrzej Ludwikowski
 
Performance tests with Gatling
Andrzej Ludwikowski
 
Cassandra - lesson learned
Andrzej Ludwikowski
 
Annotation processing tool
Andrzej Ludwikowski
 

Recently uploaded (20)

PPTX
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
PDF
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PPTX
Introduction to Artificial Intelligence
lohedi9363
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
Transform Your Business with a Software ERP System
Canada Software Company
 
System and Network Administraation Chapter 3
jaramharo
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Introduction to Artificial Intelligence
lohedi9363
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Introduction Database Management System for Course Database
ReinertYosua
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

  • 1. Event Sourcing what could possibly go wrong? Andrzej Ludwikowski
  • 2. About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski ➔ academy.softwaremill.com
  • 7. SoftwareMill ● Hibernate Envers ● Alpakka Kafka ● Sttp ● Scala Clippy ● Quicklens ● MacWire
  • 8. What is Event Sourcing? DB Order { items=[itemA, itemB] }
  • 9. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  • 10. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  • 11. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
  • 12. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
  • 13. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability ● microservices integration pattern
  • 14. ES and CQRS Client Query Service Data access Queries Read model Read model Read models Command Service Domain Events Commands
  • 15. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Transaction Command Service Domain Events Commands
  • 16. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● Implementing event sourcing using a relational database ● slick-eventsourcing
  • 17. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
  • 18. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction Command Service Domain Events Commands
  • 19. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load ? - lags
  • 20. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction event bus Command Service Domain Events Commands
  • 21. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
  • 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
  • 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction At-least-once delivery Command Service Domain Events Commands
  • 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 25. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 26. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 27. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 28. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 29. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction ?
  • 30. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
  • 31. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation - locking vs Single Writer
  • 32. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of DB?
  • 33. Not covered but worth to check ● Command Sourcing ● Event Collaboration
  • 34. ES implementation? ● custom ● library ● framework
  • 35. ES from domain perspective ● commands, events, state ● 2 methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
  • 36. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  • 37. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  • 38. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  • 39. import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
  • 40. import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
  • 42. import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} case class User (userId: UserId, name: String, email: Email, funds: Funds) { def process(command: UserCommand): Either[List[UserEvent]] = command match { case c: UpdateEmail => updateEmail(c) case c: DepositFunds => deposit(c) case c: WithdrawFunds => withdraw(c) ... } def apply(event: UserEvent): User = ??? //pattern matching }
  • 43. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  • 44. ES packaging ● domain logic ● domain validation ● 0 ES framework imports
  • 45. library vs. framework ● Akka Persistence vs. Lagom ● Akka Persistence Typed
  • 46. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  • 47. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  • 48. akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
  • 49. akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
  • 50. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
  • 51. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
  • 52. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
  • 53. Plain text Binary human-readable deserialization required
  • 54. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
  • 55. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
  • 56. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
  • 57. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
  • 58. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 59. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 60. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 61. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 62. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
  • 65. ● backward - V2 can read V1 Full compatibility Application Events V1 V2
  • 66. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
  • 67. ● forward - V2 can read V3 Full compatibility Events Read model Read model Read models Projector V2 V3
  • 68. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
  • 69. Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  • 70. Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  • 71. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
  • 72. Protobuf schema management + clean domain - a lot of boilerplate code
  • 73. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  • 74. Avro deserialization Bytes Deserialization Object Reader Schema Writer Schema
  • 75. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
  • 76. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  • 77. Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
  • 78. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
  • 79. Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
  • 80. Avro vs. Protocol Buffers ● The best serialization strategy for Event Sourcing
  • 81. Event payload ● delta event ● rich event (event enrichment) ● + metadata ○ seq_num ○ created_at ○ event_id ○ command_id ○ correlation_id
  • 84. State replay time ● snapshotting ● write-through cache
  • 87. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
  • 89. Updating all aggregates User(id) Command(user_id) Event(user_id) Event(user_id) Event(user_id)
  • 94. Event + seq_no Event + seq_no Handling duplicates Events Read model Read model Read models Projector event bus Event + seq_no read model update + seq_no Events
  • 95. Broken read model Events ad model ead model Read models Projector event bus Events
  • 96. Broken read model Events ad model ead model Read models Projector event bus read model update + offset (manual offset management) Events
  • 97. Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
  • 99. Pessimistic compensation action User account Cinema show charged
  • 100. Pessimistic compensation action User account Cinema show charged booked
  • 101. Pessimistic compensation action User account Cinema show charged booked sold out
  • 102. Pessimistic compensation action User account Cinema show charged booked booked sold out
  • 103. Pessimistic compensation action User account Cinema show charged booked booked sold out
  • 104. Pessimistic compensation action User account Cinema show charged booked booked sold out refund
  • 105. Optimistic compensation action User account Cinema show charged booked sold out
  • 106. Optimistic compensation action User account Cinema show booked booked sold out overbooked
  • 107. Optimistic compensation action User account Cinema show charged booked booked sold out overbooked ?
  • 108. Saga Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Updater event bus Transaction
  • 109. Saga - choreography Command Service Domain Events Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 110. Saga - orchestration Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Events Commands
  • 111. Saga ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
  • 112. Saga ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
  • 113. ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
  • 114. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
  • 115. Cluster = split brain 1 5 4 3 Load balancer 2
  • 116. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
  • 117. Cluster = split brain 1 5 4 3 Load balancer 2
  • 118. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
  • 119. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
  • 120. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
  • 121. Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
  • 122. Summary ● carefully choose ES lib/framework ● there is no perfect database for event sourcing ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
  • 123. Want to learn more? ● Reactive Event Sourcing with Akka (Java/Scala) ● Akka Typed Fundamentals (Java/Scala) ● Akka Cluster ● Performance tests with Gatling
  • 126. Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
  • 128. Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski