Microservices interaction at scale using Apache Kafka

Microservices interaction at scale
using Apache Kafka

• Software engineer at Upwork
• 3500 logged hours
• ivanursul.com
2

Agenda
1.Monolith to Microservices
2.Communication patterns
3.Apache Kafka
4.Kafka in details
3

Anti - Agenda
1.Pros/cons of microservices
architecture
2.How to build microservices right
4

Monolith to Microservices
Module 1 Module 2 Module 3 Module 4 Module 5
Database layer
Network layer DTO
Business
Model
ER
Model
5

Scaling in terms of people
4 engineers 2 engineers 1 engineer
9

Microservices
communication
12

Database per service pattern
15

Services equality
Microservice 1 Microservice 2
2000 rps
KO
2000 rps
200
rps
17

External request
Critical path
18

Two types of data
Data-on-the-
outside
for your
storage
Data-on-the-
inside
for other services
21

“Pluggable” architecture
22

What is Apache Kafka ?
1.Distributed publish-subscribe
messaging system
2.Fast, Scalable, Durable
3.Created at LinkedIn, is now an Apache
project
4.confluent.io 24

time
Commit Log
msg
1
msg
4
msg
7
msg
10
msg
13P0
P1
P2
2x
2x
2x
Distributed Commit Log
Distributed, Replicated
Commit Log
msg
2
msg
5
msg
8
msg
11
msg
14
msg
3
msg
6
msg
9
msg
12
msg
15
25

Apache
Zookeper
Broker
Broke
r
Broke
r
Produc
er
Consum
er
Kafka Streams
Connec
t
Conne
ct
Send messagesRead messages
Zookeper for configuration
Adapter for
third-party
sources
At least one Broker/Node
Consume, process
and produce messages
Apache Kafka architecture
Kafka Cluster
26

Kafka usage patterns
1.Event Sourcing
2.Change Data Capture
3.Kafka Connect
27

Microservice 1 Microservice 2
Apache
Kafka
29
Write Request Read Request
Kafka Producer
Microservice 1

Command Query Responsibility
Segregation
UI
Write Model Read Model
Write DB Read DB
Event
s

Service structure
Microservice group
Write
Service
Event
Handler
Read
Service
2 instances 2 instances 3 instances

Benefits
32
1.Separate read and writes
2.Data can be replayed
3.Debugging
32

Drawbac
ks
1.Higher learning curve
2.Backward compatibility support
3333

3535
Database
Cache
Search engine
Request
update/invalidat
e

Database
Request
insert
updat
e
delete insert
updat
e
Cache Search Index
WAL
3636

Apache
Kafka
Microservice infrastructureLegacy monolith
extract changes
38

Apache
Kafka
extract changes
39

Benefits
40
1.Useful for legacy databases
2.Eliminate dual writes

Drawbac
ks
1.Not every DB has support for CDC
2.Database upgrade
41

Benefits
44
1.Useful for legacy Databases
2.Eliminate dual writes
3.Supported by all storages
4.Custom connectors

Drawbac
ks
1.Additional queries
2.Indicator fields
45

Demo
Key
Roles:
Order
Service
Customer
Service
Frameworks
46

Kaf
ka
Customer
Service
Order
Service
New Order
5 order.payment.processed
4 order.processed
event handler
1 order.created
2 order.created
event handler
47

Anatomy of topic
Kafka
Topic
P0
P1
P2
msg 0
Topic is
divided
into
multiple
partitions
(P0, P1,
P2)
Each message
has an identifier
called offset
Producer
s write
message
s to the
end of
partitionPartition is a
unit of
parallelism
Consumers
can read from
any offset
Message
can have
key:value
msg 3
msg 1 msg 4 msg 7 msg 8
msg 2 msg 5
msg 6
49

Producers
• Publish to topics
• Round-robin
• Murmur 2
• Retry, Batching
• Async
• Broker list
50

Kafka Broker
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5 0 1 2 3 4 5
Leader
partition
Replica
Each partition has
exactly one
leader
0 1 2 3 4 5 0 1 2 3 4 5
And >= 0 replicas.
They’re called In-
Sync Replicas
All read and
writes are made
using leader
partitionBroker 1 Broker 2 Broker 3
0 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
P
0
P
1
P
2
51

Consumers
• Multiple consumers
• List of partitions
• Offset management
• Offset backup to consumer_offsets
52

Consumer Groups
P0 P1 P2
Instance 1 Instance 1 Instance 2 Instance 1Instance 2Instance 3Instance 4
consumer group 1 consumer group 2 consumer group 3
Kafka
Cluster
53

Historical rewind
P0
Current Offset
Instance 1
Instance 1
Service 1
ms
g 1
msg
2
msg
3
msg
4
msg
5
54

Retention period
P0
P1
P2
msg
12
msg
15
msg
11
msg
14
msg
10
msg
13
msg
1
msg
4
msg
7
msg
2
msg
5
msg
8
msg
3
msg
6
msg
9
55

Log compaction
key 1 ->
value
key 2 ->
value
key 1 -> new value
56

Guarantees
• Messages are appended in the order
they send
• Ordering is guaranteed within partition
• Consumer reads a messages in the
order they are stored
• N - 1 Broker failures57

Delivery semantics
• at least once - by default
• at most once - configurable
• exactly once - Kafka Transactions, June
2017
58

Worst scenario
Unclean Leader Election
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5 6 7
0 1 2 3
P
1
P
2
0 1 2 3 4 5
P
0
0 1 2 3 4 5
Leader
partition
Replica
Broker
1
Broker
2
Broker
3
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5 6 7
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 30 1 2 3
Sync
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
Choose one of
the
unsynchronized
followers
Wait for Broker 1
to recover
59

Service Deployment
• No significant problems in producer side
• Consumers may have issues with blue-
green deployment
• Remember to reserve additional
partitions for blue and green stacks
60

Monitoring
• Broker, Producer and Consumer has
JMX support out of the box
• Additional option is to write custom
reporters
61

• Useful for large services
• Fault tolerant
• Consume, process and produce
messages
• Aggregations
• Stateful processing
• Distributed joins 64

Why Kafka ?
• Kafka doesn’t acknowledge messages
• Easy to use
• Configurable
• Super fast for reading tail messages

Summary
• Build an event driven backbone
• Add Service interfaces where needed
• Don’t forget about CDC and Kafka
Connect
66

www.slideshare.net/KennyBastani/in-the-eventual-consistency-of-succeeding-at-microservi
Do what makes sense
But don’t build microservices without events

Demo, Slides
ivanursul.com/microservices-interaction-apache-kafka
68

Microservices interaction at scale using Apache Kafka

More Related Content

What's hot (20)

Similar to Microservices interaction at scale using Apache Kafka (20)

Recently uploaded (20)

Microservices interaction at scale using Apache Kafka

Editor's Notes