Event Sourcing & CQRS, Kafka, Rabbit MQ

@arafkarsh arafkarsh
ARAF KARSH HAMID
Co-Founder / CTO
MetaMagic Global Inc., NJ, USA
@arafkarsh
arafkarsh
Microservice
Architecture Series
Building Cloud Native Apps
Kafka, Replication
Event Storming
Distributed Tx
Case Studies
Part 2 of 11

@arafkarsh arafkarsh 2
Slides are color coded based on the topic colors.
Event Streaming
Kafka
Replication
1
Event Storming
Event Sourcing & CQRS
2
Distributed
Transactions 3
Case Studies
4

Agile
Scrum (4-6 Weeks)
Developer Journey
Monolithic
Domain Driven Design
Event Sourcing and CQRS
Waterfall
Optional
Design
Patterns
Continuous Integration (CI)
6/12 Months
Enterprise Service Bus
Relational Database [SQL] / NoSQL
Development QA / QC Ops
3
Microservices
Scrum / Kanban (1-5 Days)
Mandatory
Design
Patterns
Infrastructure Design Patterns
CI
DevOps
Event Streaming / Replicated Logs
SQL NoSQL
CD
Container Orchestrator Service Mesh

Messaging / Event Streaming
• Problem Statement
• Rabbit MQ
• Kafka
4
1

Problem Statement – Synchronous Calls
Check Out
Order Inventory
Notification Service
eMail SMS
Cart
1. Complex and Error prone.
2. Tightly Coupled Systems
3. Performance Issues
4. Scalability
Issues
5

Problem Statement – Async Calls : Queue Based
Check Out
Order Inventory
eMail SMS
Cart
• Scalability Issues
• Multiple Sub Scribers are not
allowed (to the same topic)
Issues
6

Rabbit MQ Messaging System
• Fanout Exchange
• Direct Exchange
• Topic Exchange
• Header Exchange
7

Async Calls : Fanout Exchange
Check Out
Order Inventory
eMail SMS
Cart
1. Loosely Coupled Systems
2. Scalable
Exchange
Duplicates the message &
sends it to respective
Queues
Binding key
8

Async Calls : Direct Exchange
Check Out
Order Inventory
eMail SMS
Cart
2. Scalable
Exchange
Queues
Binding key
Message Contains Routing
Key which needs to match
with Binding Key
9

Async Calls : Topic Exchange
Check Out
Order Inventory
eMail SMS
Cart
2. Scalable
Exchange
Queues
order.any
Message Contains Routing Key
which says order.phone then it can
do a partial match with order.any
10

Async Calls : Header Exchange
Check Out
Order Inventory
eMail SMS
Cart
2. Scalable
Exchange
Queues
order.any
Message Routing happens
based on the Header
11

Async Calls : Default Exchange
Check Out
Order Inventory
eMail SMS
Cart
2. Scalable
Exchange
Queues
Binding key
Message is moved forward
if the Routing Key matches
the Queue Name que-inv
que-inv
que-ord
que-notification
12

Discovering Microservices Principles….
Components
via
Services
Organized around
Business
Capabilities
Products
NOT
Projects
Smart
Endpoints
& Dumb Pipes
Decentralized
Governance &
Data Management
Infrastructure
Automation
Design for
Failure
Evolutionary
Design
How does the routing rules defy Microservices Principles?
13

Discovering Microservices Principles….
Components
via
Services
Organized around
Business
Capabilities
Products
NOT
Projects
Smart
Endpoints
& Dumb Pipes
Decentralized
Governance &
Data Management
Infrastructure
Automation
Design for
Failure
Evolutionary
Design
How does the routing rules defy Microservices Principles?
14

Kafka – Event Streaming
• Kafka Solution
• Replication, HA & Load Balancing
• Ordering Guarantee
• Concepts : Queues / Pub – Sub / Event Streaming
• Why is it different from traditional message queues?
• Kafka Solution - Revisited
15

Async Calls : Kafka Solution
Check Out
Cart
1. Highly Scalable
2. Multi Subscriber
4. Data Durability (Replication)
5. Ordering Guarantee (Per Partition)
Use Partition Key
Kafka
Producer API
Kafka
Consumer API
eMail SMS
1 2 3 4
1 2 3 4
1 2 Service
Instances
Order Topic (Total Partitions 6)
Kafka Storage
Replicated Logs
Kafka Cluster
5 6 7 8
7 8
What will happen to
Inventory Instance 7 and 8?
Order Consumer Group Inv Consumer Group
Notification Consumer Multiple Subscriber
As there are only 6 Partitions
Kafka can serve ONLY 6
consumers within a partition
16

Check Out
Cart
4. Data Durability (Replication)
5. Ordering Guarantee (Per
Partition)
Use Partition Key
Kafka
Producer API
Kafka
Consumer API
1 2 3 4
1 2 3 4
3 4
Service
Instances
Kafka Storage
Replicated Logs
Kafka Cluster
5 6 7 8
7 8
What will happen to
Inventory Instance 7 and 8?
Order Consumer Group Inv Consumer Group Multiple Subscriber
As there are only 6 Partitions
Kafka can serve ONLY 6
consumers within a partition
2 5
1
Broadcast Orders to following
Consumers
All the above Consumers will get same
orders available in the Order Topic
1. Highly Scalable
2. Multi Subscriber
3. Loosely Coupled
Systems
17

Kafka Replication, HA & Load Balancing
Order Consumer Group
1 2 3
3 Instances of
Order Service
Server 2
P1 P3 P6
Server 1
P2 P4 P5
Order Topic Partitions 6 – Split into 2 Servers
P1 P3 P6 P2 P4 P5
Replication
1. Partitions are replicated in both Server 1 and
Server 2
2. P1, P3, P6 are Leaders in Server 1 and followers
in Server 2
3. P2, P4, P5 are Leaders in Server 2 and followers
in Server 1
High Availability
1. If Server 1 goes down then followers in Server 2
will become the leader and vice versa.
Load Balancing, Performance and Scalability
1. Horizontal Scalability is achieved by adding more
servers.
2. Partitions from both servers are assigned to
various consumers of Order Service.
Order Consumer
• C1 = P1, P2
• C2 = P3, P4
• C3 = P6, P5
2 Partitions
each
Pn
Leader in
Server 1
Follower in
Server 2
Pn
Leader in
Server 2
Follower in
Server 1
18

Kafka – Ordering Guarantees
1. Messages that require relative ordering needs to be sent to
the same Partition. Kafka takes care of this part.
2. Supply the same key for all the messages that require Relative
Ordering.
3. For example if the Customer Order requires relative ordering
then Order ID will be the Key. All the messages for the same
Order ID will be sent to the same partition.
4. To maintain a Global Ordering without a key then use a Single
Partition Topic. This will limit the scalability,
19

Traditional Queue / Pub-Sub Vs. Kafka
20
0 1 2 3 4 5 6 7 8 9
8
7
9 Consumer 1
Consumer 2
Consumer 3
Queues
Data
Data can be partitioned for scalability for parallel
processing by same type of consumers
Pros:
Cons:
1. Queues are NOT multi subscribers compare to
Pub Sub.
2. Once a Consumer reads the data, it’s gone from
the queue.
3. Ordering of records will be lost in asynchronous
parallel processing.
0 1 2 3 4 5 6 7 8 9
9
9
9 Consumer 1
Consumer 2
Consumer 3
Pub – Sub
Data
Multiple subscribers can get the same data.
Pros:
Scaling is difficult as every message goes to every
subscriber.
Cons:

Traditional Queue, Pub Sub Vs. Kafka
1 2 3 4
1 2 3
Service
Instances
Order Topic Total Partitions 6 – Split into 2 Servers
Server 1 Server 2
P1 P3 P6 P2 P4 P5
Queue Implementation
1. Partition replaces the Queues & Consumer
(within a Group) will retrieve the message from
1 or more Partition.
2. Each Consumer from within a Group will be
assigned different partitions.
3. Load Balancing (Assigning Partitions to
Consumer) happens based on the number of
Consumers in the Group.
4. If a Consumer drops out the partitions will re-
assigned to another consumer within the group.
5. No. of Partitions must be greater than the
Consumers within a Group.
Pub Sub Implementation
1. With Multiple Consumer Group (Ex., Order &
Inventory) the Same Message (Event) is available
to all the groups subscribed to the same Topic.
• Order Consumer
• C1 = P1, P3
• C2 = P6, P2
• C3 = P4, P5
2
Partitions
each
• Inventory Consumer
• I1 = P1, P4
• I2 = P3, P5
• I3= P6
• I4 = P2
2
Partitions
1Partition each
21

Check Out
1 2 3 4
1 2 3
Kafka
Producer API
Kafka
Consumer API
Kafka Storage
Replicated Logs
Service
Instances
Kafka Cluster
Server 1 Server 2
P1 P3 P6 P2 P4 P5
• Each Order Consumer
has 2 Partitions each
• C1 = P1, P3
• C2 = P6, P2
• C3 = P4, P5
• Inventory Consumer has
• I1 = P1, P4
• I2 = P3, P5
• I3= P3
• I4 = P6
1. Highly Scalable
2. Multi Subscriber
3. Loosely Coupled
Systems
4. Data Durability
(Replication)
5. Ordering
Guarantee (Per
Partition)
Use Partition Key
22

Kafka Core Concepts
23
Publish & Subscribe
Read and write streams of data
like a messaging system
Process
Write scalable stream processing
apps that react to events in real-
time.
Store
Store streams of data safely in a
distributed, replicated, fault
tolerant cluster.

Kafka APIs
24
Source : https://kafka.apache.org/documentation/#gettingStarted
• The Producer API allows an application to publish a
stream of records to one or more Kafka topics.
• The Consumer API allows an application to subscribe
to one or more topics and process the stream of
records produced to them.
• The Streams API allows an application to act as
a stream processor, consuming an input stream from
one or more topics and producing an output stream
to one or more output topics, effectively transforming
the input streams to output streams.
• The Connector API allows building and running
reusable producers or consumers that connect Kafka
topics to existing applications or data systems. For
example, a connector to a relational database might
capture every change to a table.

Kafka – Fault Tolerance
Data Durability in Kafka
25

Kafka Cluster
m1
m2
m3
Leader (A)
m1
m2
Follower (B)
m1
Follower (C)
A,B,C are 3
servers in
Kafka Cluster
m1
m2
m3
Leader (A)
m1
m2
Follower (B)
m1
Follower (C)
m1
m2
m3
Leader (A)
m1
m2
Leader (B)
m1
Follower (C) Server B
becomes the
new Leader
Server A Fails
m2
26

Kafka Cluster – Topics & Partitions
• The partitions of the log are distributed over the servers in the Kafka cluster with each server handling
data and requests for a share of the partitions.
Source : https://kafka.apache.org/intro
m1, m2
Broker 1
Leader (A)
Broker 2
Follower (B)
m1,m2
Broker 3
Follower C
p1
Broker 4
Follower (B,C)
m1
p1,p2
Broker 5
Leader A
p1,p2
Partition 1
Partition 0
Topic ABC
• Each server acts as a leader for some of its partitions and a follower for others so load is well balanced
within the cluster.
• Each partition has one server which acts as the "leader" and zero or more servers which act as "followers".
27

Record Commit Process
Broker 1
Leader
Topic 1
Broker 2
Follower
Producer
Consumer
3
3
Commit
2
ack
• Each partition is replicated across a configurable
number of servers for fault tolerance.
• The leader handles all read and write requests for
the partition while the followers passively replicate
the leader.
• If the leader fails, one of the followers will
automatically become the new leader.
1
Message with Offset
4
777743
Broker 3
Follower
Data Durability From Kafka v0.8.0 onwards
acks Acknowledgement Description
0
If set to zero then the producer will NOT wait for any
acknowledgment from the server at all. The record will be
immediately added to the socket buffer and considered sent.
No guarantee can be made that the server has received the
record in this case, and the retries configuration will not take
effect (as the client won't generally know of any failures). The
offset given back for each record will always be set to -1.
1
This will mean the leader will write the record to its local log
but will respond without awaiting full acknowledgement
from all followers. In this case should the leader fail
immediately after acknowledging the record but before the
followers have replicated it then the record will be lost.
All /
-1
This means the leader will wait for the full set of in-sync
replicas to acknowledge the record. This guarantees that the
record will not be lost as long as at least one in-sync replica
remains alive. This is the strongest available guarantee. This is
equivalent to the acks=-1 setting.
Source: https://kafka.apache.org/documentation/#topicconfigs
acks Steps
0 1
1 1,2
-1 1,2,3
Producer Configuration
28

Message Acknowledgements
m1
Follower (B)
m2 m3 m4
m1
Follower (C)
m2 m3 m4
m1
Leader (A)
m2 m3 m4
Producer
acks=0 m5
Ack
m1
Follower (B)
m2 m3 m4 m5
m1
Follower (C)
m2 m3 m4 m5
m1
Leader (A)
m2 m3 m4 m5
Producer
acks=all m5
Ack
m1
Follower (B)
m2 m3 m4
m1
Follower (C)
m2 m3 m4
m1
Leader (A)
m2 m3 m4 m5
Producer
acks=1 m5
Ack
Producer get Ack
before even the
message reached
the Leader.
Producer get Ack
after the Leader
commits the
message.
Producer get Ack
after all the ISR (In
Sync Replicas)
confirms the
commit.
29

Message Acknowledgements
m1
Follower (B)
m2 m3 m4 m5
m1
Follower (C)
m2 m3 m4
m1
Leader (A)
m2 m3 m4 m5
Producer
acks=all m5
m1
Follower (B)
m2 m3 m4 m5
m1
Follower (C)
m2 m3 m4
m1
Leader (A)
m2 m3 m4 m5
Producer
acks=all
min.insync.replicas=2
m5
Ack
Producer get Ack
after the available
ISR = min in sync
replicas = X
Producer won’t get
Ack as all the ISR(In
Sync Replica) are
not available.
Because all the 3 ISR (In Sync Replicas) are Alive. Kafka Broker will send the Ack back ONLY after
receiving the ack from all the three ISRs.
Why is the Ack
Not Coming –
even after the
min in sync
replicas = 2?
m1
Follower (B)
m2 m3 m4 m5
m1
Follower (C)
m2 m3 m4
m1
Leader (A)
m2 m3 m4 m5
Producer
acks=all m5
min.insync.replicas=2
30

Replication
m1
m2
m3
L(A)
m1
m2
F(B)
m1
F(C)
ISR = (A, B, C)
Leader A commits Message
m1. Message m2 & m3 not
yet committed.
1
m1
m2
F(C)
m1
m2
L(B)
m1
m2
m3
L(A)
ISR = (B,C)
A fails and B is the new
Leader. B commits m2
2
m1
m2
m3
L(A)
m1
m2
L(B)
m4
m5
m1
m2
F(C)
m4
m5
ISR = (B,C)
B commits new messages
m4 and m5
3
m1
m2
L(B)
m4
m5
m1
m2
F(C)
m4
m5
m1
F(A)
ISR = (A, B,C)
A comes back, restores to
last commit and catches
up to latest messages.
4
m1
m2
L(B)
m4
m5
m1
m2
F(C)
m4
m5
m1
m2
F(A)
m4
m5
ISR – In-sync Replica
• Instead of majority vote, Kafka
dynamically maintains a set of in-sync
replicas (ISR) that are caught-up to the
leader.
• Only members of this set are eligible for
election as leader.
• A write to a Kafka partition is not
considered committed until all in-sync
replicas have received the write.
• This ISR set is persisted to ZooKeeper
whenever it changes. Because of this, any
replica in the ISR is eligible to be elected
leader.
31

Kafka Topic and Durability
1. Anatomy of Topic
2. Partition Log Segment
3. Cluster – Topic and Partitions
4. Record Commit Process
5. Consumer Access & Retention Policy
32

Anatomy of a Topic
33
• A Topic is a category or feed name to which
records are published.
• Topics in Kafka are always multi subscriber.
• Each Partition is an ordered, immutable
sequence of records that is continually
appended to—a structured commit log.
• A Partition is nothing but a directory of Log
Files
• The records in the partitions are each assigned a sequential id number called
the offset that uniquely identifies each record within the partition.

Partition Log Segment
• Partition (Kafka’s Storage unit) is Directory of
Log Files.
• A partition cannot be split across multiple
brokers or even multiple disks
• Partitions are split into Segments
• Segments are two files: 000.log & 000.index
• Segments are named by their base offset.
The base offset of a segment is an offset
greater than offsets in previous segments and
less than or equal to offsets in that segment.
0 1 2 3 4 5 6 7 8 9
Partition
Data
6
3
0 Segment 0
Segment 3
Segment 6
9 Segment 9 - Active
$ tree kafka-logs | head -n 6
kafka-logs
|──── SigmaHawk-2
| |──── 00000000006109871597.index
| |──── 00000000006109871597.log
| |──── 00000000007306321253.index
| |──── 00000000007306321253.log
Topic /
Partition
Segment 1
Segment 2
4 Bytes 4 Bytes

Partition Log Segment
• Indexes store offsets relative to its segments base offset
• Indexes map each offset to their message position in the log and
they are used to look up messages.
• Purging of data is based on oldest segment and one segment at
a time.
Rel.Offset, Position Offset, Position, Size, Payload
0000.index 0000.log
0 0 0 0 7 ABCDE67
1 7 1 7 4 ABC4
2 11 2 11 9 ABCDEF89
4 Bytes 4 Bytes
$ tree kafka-logs | head -n 6
kafka-logs
|──── SigmaHawk-2
| |──── 00000000006109871597.index
| |──── 00000000006109871597.log
| |──── 00000000007306321253.index
| |──── 00000000007306321253.log
Topic /
Partition
Segment 1
Segment 2
3 20 3 20 3 AB3

Consumer Access & Data Retention
36
• For example, if the retention policy is set to 2 days, then for the two days
after a record is published, it is available for consumption, after which it
will be discarded to free up space.
• The Kafka cluster retains all published records—whether or not they
have been consumed—using a configurable retention period
• Kafka's performance is effectively constant with respect to data
size so storing data for a long time is not a problem.
• Only metadata retained on a per-consumer basis is the offset or position of that consumer
in the log. This offset is controlled by the consumer: normally a consumer will advance its
offset linearly as it reads records, but, in fact, since the position is controlled by the
consumer it can consume records in any order it likes.
777743
777742
777741
777740
777739
777738
777737
777736
Producer
Consumer
Consumer
Consumer
• Producers Push Data
• Consumers Poll Data
Writes
Reads
Offset=37
Offset=38
Offset=41

Kafka Data Structures
Mentoring Session
• Kafka Record v1
• Kafka Record v2
• Kafka Record Batch
37

Kafka Record / Message Structure
38
Magic Attr
CRC
int64
int32 int8
Timestamp
Header
Key (Variable Length)
Value (Variable Length)
Payload
v1 (Supported since 0.10.0)
Field Description
CRC
The CRC is the CRC32 of the remainder of the message bytes. This
is used to check the integrity of the message on the broker and
consumer.
Magic Byte
This is a version id used to allow backwards compatible evolution
of the message binary format. The current value is 2.
Attributes
Bit 0-2 Compression Codec
0 No Compression
1 Gzip Compression
2 Snappy Compression
Bit 3 Timestamp Type: 0 for Create Time Stamp,
1 for Log Append Time Stamp
Bit. 4 is Transactional (0 means Transactional)
Bit 5 is Control Batch (0 means Control Batch)
Bit >5. Un used
Timestamp
This is the timestamp of the message. The timestamp type is
indicated in the attributes. Unit is milliseconds since beginning of
the epoch (midnight Jan 1, 1970 (UTC)).
Key
The key is an optional message key that was used for partition
assignment. The key can be null.
Value
The value is the actual message contents as an opaque byte array.
Kafka supports recursive messages in which case this may itself
contain a message set. The message can be null.
int8
Source: https://kafka.apache.org/documentation/#messages

Kafka Record Structure
39
Length (varint) Attr
int8
Timestamp Delta (varint)
Offset Delta (varint) Key Length (varint)
Key (varint) Value Length (varint)
Value (varint) Headers (Header Array)
Header Key Length (varint) Header Key (varint)
Header Value Length (varint) Header Value (varint)
Header
Record
• In Kafka 0.11, the structure of the 'Message
Set' and 'Message' were significantly
changed.
• A 'Message Set' is now called a 'Record
Batch', which contains one or more 'Records'
(and not 'Messages').
• The recursive nature of the previous versions
of the message format was eliminated in
favor of a flat structure.
• When compression is enabled, the Record
Batch header remains uncompressed, but the
Records are compressed together.
• Multiple fields in the 'Record' are varint
encoded, which leads to significant space
savings for larger batches.

Kafka Record Batch Structure
40
Field Description
First Offset
Denotes the first offset in the Record Batch. The 'offset Delta' of each
Record in the batch would be be computed relative to this First Offset.
Partition
Leader Epoch
this is set by the broker upon receipt of a produce request and is
used to ensure no loss of data when there are leader changes
with log truncation.
Attributes
The fifth lowest bit indicates whether the Record Batch is part of a
transaction or not. 0 indicates that the Record Batch is not
transactional, while 1 indicates that it is. (since 0.11.0.0)
Last Offset
Delta
The offset of the last message in the Record Batch. This is used by
the broker to ensure correct behavior even when Records within a
batch are compacted out.
First
Timestamp
The timestamp of the first Record in the batch. The timestamp of
each Record in the Record Batch is its 'Timestamp Delta' + 'First
Timestamp'.
Max
Timestamp
The timestamp of the last Record in the batch. This is used by the
broker to ensure the correct behavior even when Records within
the batch are compacted out.
Producer ID
This is the broker assigned producer Id received by the 'Init
Producer Id' request.
Producer
Epoch
This is the broker assigned producer Epoch received by the 'Init
Producer Id' request.
First Sequence
This is the producer assigned sequence number which is used by
the broker to de-duplicate messages. The sequence number for
each Record in the Record Batch is its Offset Delta + First
Sequence.
First Offset
int64
Length
int32
Partition Leader Epoch
int32
Magic
int8
CRC
int32
Attr
int16
Last offset Delta
int32
First Timestamp
int64
Max Timestamp
int64
Producer
Epoch
int16
Producer ID
int64
First Sequence
int32
Records (Record Array)

Kafka Operations
Kafka Setup
Kafka Producer
Kafka Consumer
41

Kafka Quick Setup & Demo
42
1. install the most recent version from Kafka download page
2. Extract the binaries into a /…./Softwares/kafka folder. For the current version it's kafka_2.11-1.0.0.0.
3. Change your current directory to point to the new folder.
4. Start the Zookeeper server by executing the command:
bin/zookeeper-server-start.sh config/zookeeper.properties.
5. Start the Kafka server by executing the command:
bin/kafka-server-start.sh config/server.properties.
6. Create a Test topic that you can use for testing:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic KafkaSigmaTest
7. Start a simple console Consumer that can consume messages published to a given topic, such as KafkaSigmaTest:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic KafkaSigmaTest --from-beginning.
8. Start up a simple Producer console that can publish messages to the test topic:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic KafkaSigmaTest
9. Try typing one or two messages into the producer console. Your messages should show in the consumer console.

Kafka Producer (Java)
43
Kafka Producer Configuration Kafka Producer Synchronous

Kafka Consumer (Java)
44
Kafka Consumer Configuration Kafka Consumer

ProtoBuf v3.0
45

ProtoBuf - Java
46

ProtoBuf – Object to JSON & JSON to Object
47
Object
to JSON
JSON to
Object

Kafka Performance
• Kafka / Pulsar / RabbitMQ
• LinkedIn Kafka Cluster
• Uber Kafka Cluster
48

Kafka Use Cases – High Volume Events
1. Social Media
2. E-Commerce – especially on a Single Day Sale
3. Location Sharing – Ride Sharing Apps
4. Data Gathering
1. Music Streaming Service
2. Web Site Analytics
49

Kafka / RabbitMQ / Pulsar
Tests Kafka Pulsar
Rabbit MQ
(Mirrored)
Peak Throughput (MB/s)
605
MB/s
305
MB/s
38
MB/s
p99 Latency (ms)
5 ms
(200 MB/s load)
25 ms
(200 MB/s load)
1 ms*
(reduced 30 MB/s
load)
Source: https://www.confluent.io/blog/kafka-fastest-messaging-system/
50

LinkedIn Kafka Cluster
51
Brokers
60
Partitions
50K
Messages / Second
800K
MB / Second inbound
300
MB / Second Outbound
1024
The tuning looks fairly
aggressive, but all of the
brokers in that cluster
have a 90% GC pause
time of about 21ms, and
they’re doing less than 1
young GC per second.

Uber Kafka Cluster
52
Topics
10K+
Events / Second
11M
Petabytes of Data
1PB+

Kafka Summary
53
1. Combined Best of Queues and Pub / Sub Model.
2. Data Durability
3. Fastest Messaging Infrastructure
4. Streaming capabilities
5. Replication

Event Storming
• Event sourcing / cqrs
• Case study: Shopping portal
• Case study: Restaurant app
• Case study: movie booking
• Case study: movie streaming
• Case Study: Patient Health Care
54
2

Mind Shift : From Object Modeling to Process Modeling
55
Developers with Strong Object Modeling experience
will have trouble making Events a first-class citizen.
• How do I start Event Sourcing?
• Where do I Start on Event Sourcing / CQRS?
The Key is:
1. App User’s Journey
2. Business Process
3. Ubiquitous Language – DDD
4. Capability Centric Design
5. Outcome Oriented The Best tool to define your process and its tasks.
How do you define your End User’s Journey & Business Process?
• Think It
• Build It
• Run IT

Process
• Define your Business Processes. Eg. Various aspects of Order
Processing in an E-Commerce Site, Movie Ticket Booking,
Patient visit in Hospital.
1
Commands • Define the Commands (End-User interaction with your App) to
execute the Process. Eg. Add Item to Cart is a Command.
2
Event Sourced
Aggregate
• Current state of the Aggregate is always derived from the Event
Store. Eg. Shopping Cart, Order etc. This will be part of the Rich
Domain Model (Bounded Context) of the Micro Service.
4
Projections
• Projections focuses on the View perspective of the Application.
As the Read & Write are different Models, you can have
different Projections based on your View perspective.
5
Write
Data
Read
Data
Events • Commands generates the Events to be stored in Event Store.
Eg. Item Added Event (in the Shopping Cart).
3
Event Storming – Concept

Event Sourcing Intro
Standard CRUD Operations – Customer Profile – Aggregate Root
Profile
Address
Title
Profile Created
Profile
Address
New Title
Title Updated
Profile
New
Address
New Title
New Address added
Derived
Profile
Address
Notes
Notes Removed
Time T1 T2 T4
T3
Event Sourcing and Derived Aggregate Root
Commands
1. Create Profile
2. Update Title
3. Add Address
4. Delete Notes
2
Events
1. Profile Created Event
2. Title Updated Event
3. Address Added Event
4. Notes Deleted Event
3
Profile
Address
New Title
Current State of the
Customer Profile
4
Event store
Single Source of Truth
Greg
Young
57

Event Sourcing & CQRS (Command and Query Responsibility Segregation)
• In traditional data management systems, both
commands (updates to the data) and queries
(requests for data) are executed against the
same set of entities in a single data repository.
• CQRS is a pattern that segregates the
operations that read data (Queries) from the
operations that update data (Commands) by
using separate interfaces.
• CQRS should only be used on specific portions
of a system in Bounded Context (in DDD).
• CQRS should be used along with Event Sourcing.
58
MSDN – Microsoft https://msdn.microsoft.com/en-us/library/dn568103.aspx |
Martin Fowler : CQRS – http://martinfowler.com/bliki/CQRS.html
CQS :
Bertrand Meyer
Axon
Framework
For Java
Java Axon Framework Resource : http://www.axonframework.org
Greg
Young

Differences between Commands, Events & Queries
Behavior /
Stage Change
Includes a
Response
Command Requested to Happen Maybe
Event Just Happened Never
Query None Always
1. Events are Facts and Notification
2. Event wear 2 hats: Data Hats (Fact) and Notification Hats
59

Case Study: Shopping Site – Event Sourcing / CQRS
Catalogue Shopping Cart Order Payment
• Search Products
• Add Products
• Update Products
Commands
• Add to Cart
• Remove Item
• Update Quantity
Customer
• Select Address
• Select Delivery Mode
• Process Order
Events
• Product Added
• Product Updated
• Product Discontinued
• Item Added
• Item Removed /
Discontinued
• Item Updated
• Order Initiated
• Address Selected
• Delivery Mode Selected
• Order Created
• Confirm Order for
Payment
• Proceed for Payment
• Cancel Order
• Payment Initiated
• Order Cancelled
• Order Confirmed
• OTP Send
• Payment Approved
• Payment Declined
Microservices
• Customer
• Shopping Cart
• Order
Customer Journey thru Shopping Process
2
Processes 1 Customers will browse through the Product catalogue to find the products, its ratings and reviews. Once the product is narrowed
down the customer will add the product to shopping cart. Once the customer is ready for the purchase, he/she will start the
order processing by selecting the Delivery address, delivery method, payment option. Once the payment is done customer will
get the order tracking details.
ES Aggregate 4
Core Domain
Supporting Domain Supporting Domain Supporting Domain Generic Domain
3
60

Distributed
Transactions
• Saga Design Pattern
• Features
• Handling Invariants
• Forward recovery
• Local Saga Feature
• Distributed Saga
• Use Case: Distributed Saga
61
3

Distributed Tx: SAGA Design Pattern instead of 2PC
62
Long Lived Transactions (LLTs) hold on to DB resources for relatively long periods of time, significantly delaying
the termination of shorter and more common transactions.
Source: SAGAS (1987) Hector Garcia Molina / Kenneth Salem,
Dept. of Computer Science, Princeton University, NJ, USA
T1 T2 Tn
Local Transactions
C1 C2 Cn-1
Compensating Transaction
Divide long–lived, distributed transactions into quick local ones with compensating actions for
recovery.
Travel : Flight Ticket & Hotel Booking Example
BASE (Basic Availability, Soft
State, Eventual Consistency)
Room Reserved
T1
Room Payment
T2
Seat Reserved
T3
Ticket Payment
T4
Cancelled Room Reservation
C1
Cancelled Room Payment
C2
Cancelled Ticket Reservation
C3

SAGA Design Pattern Features
63
1. Backward Recovery (Rollback)
T1 T2 T3 T4 C3 C2 C1
Order Processing, Banking
Transactions, Ticket Booking
Examples
Updating individual scores in
a Team Game.
2. Forward Recovery with Save Points
T1 (sp) T2 (sp) T3 (sp)
• To recover from Hardware Failures, SAGA needs to be persistent.
• Save Points are available for both Forward and Backward Recovery.
Type
Source: SAGAS (1987) Hector Garcia Molina / Kenneth Salem, Dept. of Computer Science, Princeton University, NJ, USA

Handling Invariants – Monolithic to Micro Services
64
In a typical Monolithic App
Customer Credit Limit info and
the order processing is part of
the same App. Following is a
typical pseudo code.
Order Created
T1
Order
Microservice
Credit Reserved
T2
Customer
Microservice
In Micro Services world with Event Sourcing, it’s a
distributed environment. The order is cancelled if
the Credit is NOT available. If the Payment
Processing is failed then the Credit Reserved is
cancelled.
Payment
Microservice
Payment Processed
T3
Order Cancelled
C1
Credit Cancelled due to
payment failure
C2
Begin Transaction
If Order Value <= Available
Credit
Process Order
Process Payments
End Transaction
Monolithic 2 Phase Commit
https://en.wikipedia.org/wiki/Invariant_(computer_science)

Use Case : Restaurant – Forward Recovery
Domain
The example focus on a
concept of a Restaurant
which tracks the visit of
an individual or group
to the Restaurant. When
people arrive at the
Restaurant and take a
table, a table is opened.
They may then order
drinks and food. Drinks
are served immediately
by the table staff,
however food must be
cooked by a chef. Once
the chef prepared the
food it can then be
served.
Payment
Billing
Dining
Source: http://cqrs.nu/tutorial/cs/01-design
Soda Cancelled
Table Opened
Juice Ordered
Soda Ordered
Appetizer Ordered
Soup Ordered
Food Ordered
Juice Served
Food Prepared
Food Served
Appetizer Served
Table Closed
Aggregate Root : Dinning Order
Billed Order
T1
Payment CC
T2
Payment Cash
T3
T1 (sp) T2 (sp) T3 (sp)
Event Stream
Aggregate Root : Food Bill
Transaction doesn't rollback if one payment
method is failed. It moves forward to the
NEXT one.
sp
Network
Error
C1 sp

Local SAGA Features
1. Part of the Micro Services
2. Local Transactions and Compensation
Transactions
3. SAGA State is persisted
4. All the Local transactions are based on
Single Phase Commit (1 PC)
5. Developers need to ensure that
appropriate compensating
transactions are Raised in the event of
a failure.
API Examples
@StartSaga(name=“HotelBooking”)
public void reserveRoom(…) {
}
@EndSaga(name=“HotelBooking”)
public void payForTickets(…) {
}
@AbortSaga(name=“HotelBooking”)
public void cancelBooking(…) {
}
@CompensationTx()
public void cancelReservation(…) {
}
66

SAGA Execution Container
1. SEC is a separate Process
2. Stateless in nature and Saga state is stored in a
messaging system (Kafka is a Good choice).
3. SEC process failure MUST not affect Saga Execution as
the restart of the SEC must start from where the Saga
left.
4. SEC – No Single Point of Failure (Master Slave Model).
5. Distributed SAGA Rules are defined using a DSL.
67

Use Case : Travel Booking – Distributed Saga (SEC)
Hotel Booking
Car Booking
Flight Booking
Saga
Execution
Container
Start Saga
{Booking Request}
Payment
End
Saga
Start
Saga
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
Saga Log
End Saga
{Booking Confirmed}
SEC knows the structure of the
distributed Saga and for each
of the Request Which Service
needs to be called and what
kind of Recovery mechanism it
needs to be followed.
SEC can parallelize the calls
to multiple services to
improve the performance.
The Rollback or Roll forward
will be dependent on the
business case.
Source: Distributed Sagas By Catitie McCaffrey, June 6, 2017
68

Use Case : Travel Booking – Rollback
Hotel Booking
Car Booking
Flight Booking
Saga
Execution
Container
Start Saga
{Booking Request}
Payment
Start
Comp
Saga
End
Comp
Saga
Start Hotel
End Hotel
Start Car
Abort Car
Cancel Hotel
Cancel Flight
Saga Log
End Saga
{Booking Cancelled}
Kafka is a good choice to
implement the SEC log.
SEC is completely STATELESS in
nature. Master Slave model
can be implemented to avoid
the Single Point of Failure.
Source: Distributed Sagas By Catitie McCaffrey, June 6, 2017
69

Scalability Requirement in Cloud
1. Availability and Partition Tolerance is more important
than immediate Consistency.
2. Eventual Consistency is more suitable in a highly
scalable Cloud Environment
3. Two Phase Commit has its limitations from Scalability
perspective and it’s a Single Point of Failure.
4. Scalability examples from eBay, Amazon, Netflix, Uber,
Airbnb etc.
70

Summary:
71
1. 2 Phase Commit
Doesn’t scale well in cloud environment
2. SAGA Design Pattern
Raise compensating events when the local transaction fails.
3. SAGA Supports Rollbacks & Roll
Forwards
Critical pattern to address distributed transactions.

Case Studies
• Case Study: Shopping Portal
• Case Study: Movie Streaming
• Case Study: Patient Care
• Case Study: Restaurant Dinning
• Case Study: Movie Ticket Booking
4
72

Process
• Define your Business Processes. Eg. Various aspects of Order
Processing in an E-Commerce Site, Movie Ticket Booking,
Patient visit in Hospital.
1
Commands • Define the Commands (End-User interaction with your App) to
execute the Process. Eg. Add Item to Cart is a Command.
2
Event Sourced
Aggregate
• Current state of the Aggregate is always derived from the Event
Store. Eg. Shopping Cart, Order etc. This will be part of the Rich
Domain Model (Bounded Context) of the Micro Service.
4
Projections
• Projections focuses on the View perspective of the Application.
As the Read & Write are different Models, you can have
different Projections based on your View perspective.
5
Write
Data
Read
Data
Events • Commands generates the Events to be stored in Event Store.
Eg. Item Added Event (in the Shopping Cart).
3
Event Storming – Concept
73

Shopping Portal Services – Code Packaging
Auth Products Cart Order
Customer
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Packaging Structure
Bounded Context
Implementation
(Repositories, Business Services, Web Services)
Domain Models
(Entities, Value Objects, DTOs)
Entity Factories
Interfaces (Ports)
74

Case Study: Shopping Site – Event Sourcing / CQRS
Catalogue Shopping Cart Order Payment
• Search Products
• Add Products
• Update Products
Commands
• Add to Cart
• Remove Item
• Update Quantity
Customer
• Select Address
• Select Delivery Mode
• Process Order
Events
• Product Added
• Product Updated
• Product Discontinued
• Item Added
• Item Removed /
Discontinued
• Item Updated
• Order Initiated
• Address Selected
• Delivery Mode Selected
• Order Created
Payment
• Cancel Order
• Order Cancelled
• Order Confirmed
• OTP Send
Microservices
• Customer
• Shopping Cart
• Order
Customer Journey thru Shopping Process
2
Processes 1 Customers will browse through the Product catalogue to find the products, its ratings and reviews. Once the product is narrowed
down the customer will add the product to shopping cart. Once the customer is ready for the purchase, he/she will start the
order processing by selecting the Delivery address, delivery method, payment option. Once the payment is done customer will
get the order tracking details.
ES Aggregate 4
Core Domain
Sub Domain Sub Domain Sub Domain Generic Domain
3
75

DDD: Use Case Order Service
Models
Value Object
• Currency
• Item Value
• Order Status
• Payment Type
• Record State
• Audit Log
Entity
• Order (Aggregate Root)
• Order Item
• Shipping Address
• Payment
DTO
• Order
• Order Item
• Shipping Address
• Payment
Domain Layer Adapters
• Order Repository
• Order Service
• Order Web Service
• Order Query Web Service
• Shipping Address Web Service
• Payment Web Service
Adapters Consists of Actual
Implementation of the Ports like
Database Access, Web Services
API etc.
Converters are used to convert
an Enum value to a proper
Integer value in the Database.
For Example Order Status
Complete is mapped to integer
value 100 in the database.
Services / Ports
• Order Service
Utils
• Order Factory
• Order Status Converter
• Record State Converter
• Shipping Address Web Service
Shopping Portal
76

Shopping Portal Design based on Hexagonal Architecture
Monolithic App Design using DDD
Domain Driven Design helps you to migrate your monolithic App to Microservices based Apps
77

Movie Streaming Services – Code Packaging
Auth Streaming License Subscription
Discovery
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Domain Layer
• Models
• Repo
• Services
• Factories
Adapters
• Repo
• Services
• Web Services
Packaging Structure
Bounded Context
Implementation
Domain Models
(Entities, Value Objects, DTOs)
Entity Factories
Interfaces (Ports)
78

Case Study: Movie Streaming – Event Sourcing / CQRS
Subscription Payment
• Search Movies
• Add Movies
• Update Movies
Commands
• Request Streaming
• Start Movie Streaming
• Pause Movie Streaming
• Validate Streaming
License
• Validate Download
License
Events
• Movie Added
• Movie Updated
• Movie Discontinued
• Streaming Requested
• Streaming Started
• Streaming Paused
• Streaming Done
• Streaming Request
Accepted
• Streaming Request
Denied
• Subscribe Monthly
• Subscribe Annually
• Monthly
Subscription Added
• Yearly Subscription
Added
Discovery
Microservices
Customer will search for specific movie or pick up a new episode from a TV Series from the watch list. Once the streaming
request is authorized by the license service, video streaming will start. Customer can pause, fast forward and restart the
movie streaming. Movie streaming will be based on Customer subscription to the service.
• Stream List
• Favorite List
Customer Journey thru Streaming Movie / TV Show
The purpose of this example is to demonstrate the concept of ES / CQRS thru Event Storming principles.
License
Streaming
Processes 1
2 ES Aggregate 4
Core Domain
Sub Domain Sub Domain
Sub Domain Generic Domain
3
79

DDD: Use Case Subscription Service
Models
Value Object
• Currency
• Subscription Value
• Subscription Type
• Subscription Status
• Payment Type
• Record State
• Audit Log
Entity
• Subscription (Aggregate
Root)
• Customer
• Payment
DTO
• Subscription
• Payment
Domain Layer Adapters
• Order Service
Adapters Consists of Actual
Implementation of the Ports like
Database Access, Web Services
API etc.
Converters are used to convert
an Enum value to a proper
Integer value in the Database.
For Example Order Status
Complete is mapped to integer
value 100 in the database.
Services / Ports
• Order Service
Utils
• Order Factory
• Order Status Converter
• Record State Converter
• Streaming Web Service
Movie Streaming
80

Case Study: Patient Diagnosis and Treatment
Payment
• Register
Patient
• Search Doctor
Commands
• Add Patient
Info
• Add Details
• Add BP
• Add Diagnosis
• Add
Prescription
Events
• Doctor
Scheduled
• Patient Added
• Patient Info
Added
• Details Added
• BP Added
• Diagnosis
Added
• Prescription
Added
• Add
Medicine
• Add Bill
• Medicine
Added
• Bill Prepared
• Payment
Approved
• Cash Paid
Patient registers and takes an appointment with the doctor. Patient details and history is recorded. Doctor
does the diagnosis and creates the prescription. Patient buys the medicine from the Pharmacy. If patient
needs to be admitted, then ward appointment is scheduled and admitted to the ward. Once the treatment is
over patient is discharged from the Hospital.
Microservices
• Diagnosis
• Prescription
• Hospital Bill
• Discharge Summary
Patient Journey thru Treatment Process
Registration
• Add Doctor
• Add
Appointment
• Add Patient File
• Doctor Added
• Appointment
Added
• Patient File Added
ES Aggregate
2 4
Processes 1
Doctors Diagnosis Pharmacy
Ward
Patient
• Add Checkup
• Add Treatment
• Add Food
• Add Discharge
• Checkup Added
• Treatment
Added
• Food Added
• Discharge Added
Core Domain Sub Domain Sub Domain
Sub Domain
Sub Domain
3
81

Case Study: Movie Booking – Event Sourcing / CQRS
Order Payment
• Search Movies
• Add Movies
• Update Movies
Commands
• Select Movie
• Select Theatre / Show
• Select Seats
• Process Order
• Select Food
• Food Removed
• Skip Food
• Process Order
Events
• Movie Added
• Movie Updated
• Movie Discontinued
• Movie Added
• Theatre / Show Added
• Seats Added
• Order Initiated
• Popcorn Added
• Drinks Added
• Popcorn Removed
• Order Finalized
Payment
• Cancel Order
• Order Cancelled
• Order Confirmed
• OTP Send
Movies Theatres Food
Microservices
Customer's will Search for the Movies after selecting the City. Once the movie is selected then they will identify a theatre and
check for the show Times and then select the seats. Once the seats are selected then a choice is given to add Snacks after
that the Customer will proceed to payments. Once the payment is done then the tickets are confirmed.
• Theatre
• Show
• Order
Customer Journey thru booking Movie Ticket
The purpose of this example is to demonstrate the concept of ES / CQRS thru Event Storming principles.
Processes 1
2 ES Aggregate 4
Core Domain
3
82

Case Study: Restaurant Dining – Event Sourcing and CQRS
Order Payment
• Add Drinks
• Add Food
• Update Food
Commands • Open Table
• Add Juice
• Add Soda
• Add Appetizer 1
• Add Appetizer 2
• Serve Drinks
• Prepare Food
• Serve Food
Events
• Drinks Added
• Food Added
• Food Updated
• Food Discontinued
• Table Opened
• Juice Added
• Soda Added
• Appetizer 1 Added
• Appetizer 2 Added
• Juice Served
• Soda Served
• Appetizer Served
• Food Prepared
• Food Served
• Prepare Bill
• Process
Payment
• Bill Prepared
• Payment Processed
• Cash Paid
When people arrive at the Restaurant and take a table, a Table is opened. They may then order drinks and
food. Drinks are served immediately by the table staff; however, food must be cooked by a chef. Once the
chef prepared the food it can then be served. The Bill is prepared when the Table is closed.
Microservices
• Dinning Order
• Billable Order
Customer Journey thru Dinning Processes
Food Menu Kitchen
Dining
• Remove Soda
• Add Food 1
• Add Food 2
• Place Order
• Close Table
• Remove Soda
• Food 1 Added
• Food 2 Added
• Order Placed
• Table Closed
ES Aggregate
2 4
Processes 1
Core Domain
3
83

Summary: User Journey / CCD / DDD / Event Sourcing & CQRS
User Journey
Bounded
Context
1
Bounded
Context
2
Bounded
Context
3
1. Bounded Contexts
2. Entity
3. Value Objects
4. Aggregate Roots
5. Domain Events
6. Repository
7. Service
8. Factory
Process
1
Commands
2
Projections
5
ES Aggregate
4
Events
3
Event Sourcing & CQRS
Domain Expert Analyst Architect QA
Design Docs Test Cases Code
Developers
Ubiquitous Language
Core
Domain
Sub
Domain
Generic
Domain
Vertically sliced Product Team
FE
BE
DB
Business
Capability 1
QA
Team
PO
FE
BE
DB
Business
Capability 2
QA
Team
PO
FE
BE
DB
Business
Capability n
QA
Team
PO
84

100s Microservices
1,000s Releases / Day
10,000s Virtual Machines
100K+ User actions / Second
81 M Customers Globally
1 B Time series Metrics
10 B Hours of video streaming
every quarter
Source: NetFlix: : https://www.youtube.com/watch?v=UTKIT6STSVM
10s OPs Engineers
0 NOC
0 Data Centers
So what do NetFlix think about DevOps?
No DevOps
Don’t do lot of Process / Procedures
Freedom for Developers & be Accountable
Trust people you Hire
No Controls / Silos / Walls / Fences
Ownership – You Build it, You Run it.

Design Patterns are
solutions to general
problems that
software developers
faced during software
development.
Design Patterns

DREAM | AUTOMATE | EMPOWER
Araf Karsh Hamid :
India: +91.999.545.8627
http://www.slideshare.net/arafkarsh
https://www.linkedin.com/in/arafkarsh/
https://www.youtube.com/user/arafkarsh/playlists
http://www.arafkarsh.com/
@arafkarsh
arafkarsh

Source Code: https://github.com/MetaArivu Web Site: https://metarivu.com/ https://pyxida.cloud/

http://www.slideshare.net/arafkarsh

References
1. July 15, 2015 – Agile is Dead : GoTo 2015 By Dave Thomas
2. Apr 7, 2016 - Agile Project Management with Kanban | Eric Brechner | Talks at Google
3. Sep 27, 2017 - Scrum vs Kanban - Two Agile Teams Go Head-to-Head
4. Feb 17, 2019 - Lean vs Agile vs Design Thinking
5. Dec 17, 2020 - Scrum vs Kanban | Differences & Similarities Between Scrum & Kanban
6. Feb 24, 2021 - Agile Methodology Tutorial for Beginners | Jira Tutorial | Agile Methodology Explained.
Agile Methodologies
90

References
1. Vmware: What is Cloud Architecture?
2. Redhat: What is Cloud Architecture?
3. Cloud Computing Architecture
4. Cloud Adoption Essentials:
5. Google: Hybrid and Multi Cloud
6. IBM: Hybrid Cloud Architecture Intro
7. IBM: Hybrid Cloud Architecture: Part 1
8. IBM: Hybrid Cloud Architecture: Part 2
9. Cloud Computing Basics: IaaS, PaaS, SaaS
91
1. IBM: IaaS Explained
2. IBM: PaaS Explained
3. IBM: SaaS Explained
4. IBM: FaaS Explained
5. IBM: What is Hypervisor?
Cloud Architecture

References
Microservices
1. Microservices Definition by Martin Fowler
2. When to use Microservices By Martin Fowler
3. GoTo: Sep 3, 2020: When to use Microservices By Martin Fowler
4. GoTo: Feb 26, 2020: Monolith Decomposition Pattern
5. Thought Works: Microservices in a Nutshell
6. Microservices Prerequisites
7. What do you mean by Event Driven?
8. Understanding Event Driven Design Patterns for Microservices
92

References – Microservices – Videos
93
1. Martin Fowler – Micro Services : https://www.youtube.com/watch?v=2yko4TbC8cI&feature=youtu.be&t=15m53s
2. GOTO 2016 – Microservices at NetFlix Scale: Principles, Tradeoffs & Lessons Learned. By R Meshenberg
3. Mastering Chaos – A NetFlix Guide to Microservices. By Josh Evans
4. GOTO 2015 – Challenges Implementing Micro Services By Fred George
5. GOTO 2016 – From Monolith to Microservices at Zalando. By Rodrigue Scaefer
6. GOTO 2015 – Microservices @ Spotify. By Kevin Goldsmith
7. Modelling Microservices @ Spotify : https://www.youtube.com/watch?v=7XDA044tl8k
8. GOTO 2015 – DDD & Microservices: At last, Some Boundaries By Eric Evans
9. GOTO 2016 – What I wish I had known before Scaling Uber to 1000 Services. By Matt Ranney
10. DDD Europe – Tackling Complexity in the Heart of Software By Eric Evans, April 11, 2016
11. AWS re:Invent 2016 – From Monolithic to Microservices: Evolving Architecture Patterns. By Emerson L, Gilt D. Chiles
12. AWS 2017 – An overview of designing Microservices based Applications on AWS. By Peter Dalbhanjan
13. GOTO Jun, 2017 – Effective Microservices in a Data Centric World. By Randy Shoup.
14. GOTO July, 2017 – The Seven (more) Deadly Sins of Microservices. By Daniel Bryant
15. Sept, 2017 – Airbnb, From Monolith to Microservices: How to scale your Architecture. By Melanie Cubula
16. GOTO Sept, 2017 – Rethinking Microservices with Stateful Streams. By Ben Stopford.
17. GOTO 2017 – Microservices without Servers. By Glynn Bird.

References
94
1. Oct 27, 2012 What I have learned about DDD Since the book. By Eric Evans
2. Mar 19, 2013 Domain Driven Design By Eric Evans
3. Jun 02, 2015 Applied DDD in Java EE 7 and Open Source World
4. Aug 23, 2016 Domain Driven Design the Good Parts By Jimmy Bogard
5. Sep 22, 2016 GOTO 2015 – DDD & REST Domain Driven API’s for the Web. By Oliver Gierke
6. Jan 24, 2017 Spring Developer – Developing Micro Services with Aggregates. By Chris Richardson
7. May 17. 2017 DEVOXX – The Art of Discovering Bounded Contexts. By Nick Tune
8. Dec 21, 2019 What is DDD - Eric Evans - DDD Europe 2019. By Eric Evans
9. Oct 2, 2020 - Bounded Contexts - Eric Evans - DDD Europe 2020. By. Eric Evans
10. Oct 2, 2020 - DDD By Example - Paul Rayner - DDD Europe 2020. By Paul Rayner

References
1. IBM: Event Driven Architecture – Mar 21, 2021
2. Martin Fowler: Event Driven Architecture – GOTO 2017
3. Greg Young: A Decade of DDD, Event Sourcing & CQRS – April 11, 2016
4. Nov 13, 2014 GOTO 2014 – Event Sourcing. By Greg Young
5. Mar 22, 2016 Building Micro Services with Event Sourcing and CQRS
6. Apr 15, 2016 YOW! Nights – Event Sourcing. By Martin Fowler
7. May 08, 2017 When Micro Services Meet Event Sourcing. By Vinicius Gomes
95

References
96
Kafka
1. Understanding Kafka
2. Understanding RabbitMQ
3. IBM: Apache Kafka – Sept 18, 2020
4. Confluent: Apache Kafka Fundamentals – April 25, 2020
5. Confluent: How Kafka Works – Aug 25, 2020
6. Confluent: How to integrate Kafka into your environment – Aug 25, 2020
7. Kafka Streams – Sept 4, 2021
8. Kafka: Processing Streaming Data with KSQL – Jul 16, 2018
9. Kafka: Processing Streaming Data with KSQL – Nov 28, 2019

References
Databases: Big Data / Cloud Databases
1. Google: How to Choose the right database?
2. AWS: Choosing the right Database
3. IBM: NoSQL Vs. SQL
4. A Guide to NoSQL Databases
5. How does NoSQL Databases Work?
6. What is Better? SQL or NoSQL?
7. What is DBaaS?
8. NoSQL Concepts
9. Key Value Databases
10. Document Databases
11. Jun 29, 2012 – Google I/O 2012 - SQL vs NoSQL: Battle of the Backends
12. Feb 19, 2013 - Introduction to NoSQL • Martin Fowler • GOTO 2012
13. Jul 25, 2018 - SQL vs NoSQL or MySQL vs MongoDB
14. Oct 30, 2020 - Column vs Row Oriented Databases Explained
15. Dec 9, 2020 - How do NoSQL databases work? Simply Explained!
1. Graph Databases
2. Column Databases
3. Row Vs. Column Oriented Databases
4. Database Indexing Explained
5. MongoDB Indexing
6. AWS: DynamoDB Global Indexing
7. AWS: DynamoDB Local Indexing
8. Google Cloud Spanner
9. AWS: DynamoDB Design Patterns
10. Cloud Provider Database Comparisons
11. CockroachDB: When to use a Cloud DB?
97

References
Docker / Kubernetes / Istio
1. IBM: Virtual Machines and Containers
2. IBM: What is a Hypervisor?
3. IBM: Docker Vs. Kubernetes
4. IBM: Containerization Explained
5. IBM: Kubernetes Explained
6. IBM: Kubernetes Ingress in 5 Minutes
7. Microsoft: How Service Mesh works in Kubernetes
8. IBM: Istio Service Mesh Explained
9. IBM: Kubernetes and OpenShift
10. IBM: Kubernetes Operators
11. 10 Consideration for Kubernetes Deployments
Istio – Metrics
1. Istio – Metrics
2. Monitoring Istio Mesh with Grafana
3. Visualize your Istio Service Mesh
4. Security and Monitoring with Istio
5. Observing Services using Prometheus, Grafana, Kiali
6. Istio Cookbook: Kiali Recipe
7. Kubernetes: Open Telemetry
8. Open Telemetry
9. How Prometheus works
10. IBM: Observability vs. Monitoring
98

References
99
1. Feb 6, 2020 – An introduction to TDD
2. Aug 14, 2019 – Component Software Testing
3. May 30, 2020 – What is Component Testing?
4. Apr 23, 2013 – Component Test By Martin Fowler
5. Jan 12, 2011 – Contract Testing By Martin Fowler
6. Jan 16, 2018 – Integration Testing By Martin Fowler
7. Testing Strategies in Microservices Architecture
8. Practical Test Pyramid By Ham Vocke
Testing – TDD / BDD

1. Simoorg : LinkedIn’s own failure inducer framework. It was designed to be easy to extend and
most of the important components are plug‐ gable.
2. Pumba : A chaos testing and network emulation tool for Docker.
3. Chaos Lemur : Self-hostable application to randomly destroy virtual machines in a BOSH-
managed environment, as an aid to resilience testing of high-availability systems.
4. Chaos Lambda : Randomly terminate AWS ASG instances during business hours.
5. Blockade : Docker-based utility for testing network failures and partitions in distributed
applications.
6. Chaos-http-proxy : Introduces failures into HTTP requests via a proxy server.
7. Monkey-ops : Monkey-Ops is a simple service implemented in Go, which is deployed into an
OpenShift V3.X and generates some chaos within it. Monkey-Ops seeks some OpenShift
components like Pods or Deployment Configs and randomly terminates them.
8. Chaos Dingo : Chaos Dingo currently supports performing operations on Azure VMs and VMSS
deployed to an Azure Resource Manager-based resource group.
9. Tugbot : Testing in Production (TiP) framework for Docker.
Testing tools

References
CI / CD
1. What is Continuous Integration?
2. What is Continuous Delivery?
3. CI / CD Pipeline
4. What is CI / CD Pipeline?
5. CI / CD Explained
6. CI / CD Pipeline using Java Example Part 1
7. CI / CD Pipeline using Ansible Part 2
8. Declarative Pipeline vs Scripted Pipeline
9. Complete Jenkins Pipeline Tutorial
10. Common Pipeline Mistakes
11. CI / CD for a Docker Application
101

References
102
DevOps
1. IBM: What is DevOps?
2. IBM: Cloud Native DevOps Explained
3. IBM: Application Transformation
4. IBM: Virtualization Explained
5. What is DevOps? Easy Way
6. DevOps?! How to become a DevOps Engineer???
7. Amazon: https://www.youtube.com/watch?v=mBU3AJ3j1rg
8. NetFlix: https://www.youtube.com/watch?v=UTKIT6STSVM
9. DevOps and SRE: https://www.youtube.com/watch?v=uTEL8Ff1Zvk
10. SLI, SLO, SLA : https://www.youtube.com/watch?v=tEylFyxbDLE
11. DevOps and SRE : Risks and Budgets : https://www.youtube.com/watch?v=y2ILKr8kCJU
12. SRE @ Google: https://www.youtube.com/watch?v=d2wn_E1jxn4

References
103
1. Lewis, James, and Martin Fowler. “Microservices: A Definition of This New Architectural Term”, March 25, 2014.
2. Miller, Matt. “Innovate or Die: The Rise of Microservices”. e Wall Street Journal, October 5, 2015.
3. Newman, Sam. Building Microservices. O’Reilly Media, 2015.
4. Alagarasan, Vijay. “Seven Microservices Anti-patterns”, August 24, 2015.
5. Cockcroft, Adrian. “State of the Art in Microservices”, December 4, 2014.
6. Fowler, Martin. “Microservice Prerequisites”, August 28, 2014.
7. Fowler, Martin. “Microservice Tradeoffs”, July 1, 2015.
8. Humble, Jez. “Four Principles of Low-Risk Software Release”, February 16, 2012.
9. Zuul Edge Server, Ketan Gote, May 22, 2017
10. Ribbon, Hysterix using Spring Feign, Ketan Gote, May 22, 2017
11. Eureka Server with Spring Cloud, Ketan Gote, May 22, 2017
12. Apache Kafka, A Distributed Streaming Platform, Ketan Gote, May 20, 2017
13. Functional Reactive Programming, Araf Karsh Hamid, August 7, 2016
14. Enterprise Software Architectures, Araf Karsh Hamid, July 30, 2016
15. Docker and Linux Containers, Araf Karsh Hamid, April 28, 2015

References
104
16. MSDN – Microsoft https://msdn.microsoft.com/en-us/library/dn568103.aspx
17. Martin Fowler : CQRS – http://martinfowler.com/bliki/CQRS.html
18. Udi Dahan : CQRS – http://www.udidahan.com/2009/12/09/clarified-cqrs/
19. Greg Young : CQRS - https://www.youtube.com/watch?v=JHGkaShoyNs
20. Bertrand Meyer – CQS - http://en.wikipedia.org/wiki/Bertrand_Meyer
21. CQS : http://en.wikipedia.org/wiki/Command–query_separation
22. CAP Theorem : http://en.wikipedia.org/wiki/CAP_theorem
23. CAP Theorem : http://www.julianbrowne.com/article/viewer/brewers-cap-theorem
24. CAP 12 years how the rules have changed
25. EBay Scalability Best Practices : http://www.infoq.com/articles/ebay-scalability-best-practices
26. Pat Helland (Amazon) : Life beyond distributed transactions
27. Stanford University: Rx https://www.youtube.com/watch?v=y9xudo3C1Cw
28. Princeton University: SAGAS (1987) Hector Garcia Molina / Kenneth Salem
29. Rx Observable : https://dzone.com/articles/using-rx-java-observable

Event Sourcing & CQRS, Kafka, Rabbit MQ

More Related Content

What's hot (20)

Similar to Event Sourcing & CQRS, Kafka, Rabbit MQ (20)

More from Araf Karsh Hamid (15)

Recently uploaded (20)

Event Sourcing & CQRS, Kafka, Rabbit MQ

Editor's Notes