An introduction to Apache Kafka and Kafka ecosystem at LinkedIn

©2017 LinkedIn Corporation. All Rights Reserved.
An Introduction to Apache Kafka and
Kafka Ecosystem at LinkedIn
Dong Lin
Data Infra Streaming @ LinkedIn
Open Data Science Conference

Agenda
▪ Kafka basics (50 min)
▪ Kafka ecosystem at LinkedIn (40 min)
▪ Hands-on (30 min)

©2017 LinkedIn Corporation. All Rights Reserved. 3
Kafka basics
▪ What is Kafka?
– Motivation and design philosophy
▪ Who uses Kafka?
– Adoption in the open source community and use-cases at LinkedIn
▪ What is the fundamental design of Kafka?
– Partition and replication model
▪ How to configure Kafka for your use-case?
– Tradeoff among performance, persistence, availability and message order
▪ What is the development roadmap of Kafka?
– Recent and upcoming features

Publish/Subscribe Messaging
• Multiple producers
• Multiple consumers
• Scalable and durable
• Created by LinkedIn
• Open sourced under Apache

PageViewEvent
Hadoop
Direct transmission
Web server

Many problems
Multiple
consumers
Destination
is slow
Destination
permanent
failure
Bug in
downstream
application
Destination
temporarily
unavailable
Multiple
producers
At least once
delivery
6
PageViewEvent
HadoopWeb server

Use a publish-subscribe messaging system
Multiple
consumers
Destination
permanent
failure
Bug in
downstream
application
Multiple
producers
Destination
temporarily
unavailable
Pub/sub
system
7
Hadoop
Destination
is slow
At least once
delivery
Web server

Use Kafka
Spark streaming
Multiple
consumers
Destination
permanent
failure
Bug in
downstream
application
FunctionalityPersistent
Delivery semanticsPerformance
Destination
temporarily
unavailable
Availability
8
Destination
is slow
At least once
delivery
Multiple
producers
Web server

Problem: closely-coupled pipelines
▪ O(N^2) pipelines – limited organizational scalability
▪ Messages are duplicated proportional to number of clients
9

Solution: publish-subscribe messaging system
▪ O(N) pipelines
▪ Space efficient
▪ Producers are decoupled from consumers
10

Kafka as Unix Pipes
$ cat *.txt | tr A-Z a-z | grep hello
$ tail –F *.txt | tr A-Z a-z | grep hello
producer kafka Hadoop kafka Hadoop
Samza kafka Samza
Reference: http://www.confluent.io/blog
11

Fan In
12

Fan Out
13

Add Branch
14

Switch Branch
15

Delete Branch
16

Parallel Consumption
17

Kafka basics
▪ What is Kafka?
▪ Who uses Kafka?

Companies that use Kafka
LinkedIn Yahoo Twitter Airbnb
Pinterest Square Coursera Uber
Goldman Sachs Box Paypal Cisco
Dropbox Spotify Wikipedia Microsoft
Netflix CloudFlare Hotels.com …
Reference: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
19

Apache projects integrated with Kafka
• Stream processing
• Apache Storm
• Apache Samza
• Apache Spark Streaming
• Search and Query
• Apache Hive
• Presto
• Apache Hadoop
…
20

Kafka volume at LinkedIn
21
• Produced
• Per day
2Trillion
messages
• Single cluster
• Unique data
5Gbps
Inbound
• Average 3X
consumption
• Before mirroring
18Gbps
Outbound
• Largest cluster has
250k partitions
• Up to 10k partitions
per broker
2.5M
Partitions

Kafka use-cases at LinkedIn
22
• Member-related
Activity
Tracking Metrics Queuing Logging
• Application
metrics, service
calls
• Internal
application data,
messaging
• Largest users
are Samza and
Search
• Dedicated
cluster for
application logs
going to ELK
• High volume, low
retention

Kafka basics
▪ What is Kafka?
▪ Who uses Kafka?

Design goal
▪ Performance
– High throughput
– Low latency
– Scalable
▪ Persistence and availability
– Data should be available in the event of (permanent) server failure
▪ Functionality
– Rewind back in time
▪ Strong delivery semantics
– At-least-once delivery / exactly-once delivery
– In-order message delivery within partition
24

Characteristics
• High throughput (～300 MBps per machine)
– Immutable append-only data structure for fast disk access
– Efficient data transfer via zero copy
– Mostly messages are read directly from page cache
– Partitioning model for scalability
– Batching and compression
• Low latency (～2 ms)
– Make data universally available in near real-time
• Strong guarantees about messages
– Messages strictly ordered within partition
– All data persistent on disk with replication
– Exactly once delivery
25

Is disk slow?
26

Traditional data copy
27
▪ 4 copies
▪ 2 context switches

Efficient zero copy
28
▪ 3 copies
▪ 0 context switch
▪ Only 2 copies if consumers
are mostly caught up

Kafka as log
29

Producer -> Topic -> Consumer
30

Topic divided into partitions
• Partitions are distributed and replicated across brokers
• Parallel produce/consume
• Messages with the same key go to the same partition
31

Old New
Partition consists of messages with offsets
• Append only
• Strict order
• Messages assigned with incremental offsets
32

▪ Disk/network/CPU load
distributed across brokers in
unit of partitions
Broker in Kafka

Producer in Kafka
▪ Messages with same key go
to the same partition
▪ Messages without a key go to
a random partition
34

Consumer in Kafka
▪ Consume can belong to a
consumer group (CG)
▪ Consumes in the same CG
– Parallel processing of messages
– Share the consumer offset
35

When a broker fails…
X
36

Partition replication in Kafka
▪Brokers can fail
– Controlled: e.g., upgrades/config changes
– Uncontrolled: disk failure, power outage, out-of-memory etc.
▪Need high availability
– Typical failover < 10 ms
▪Need data persistence
37

Partition replica assignment
▪ Replicas are laid out evenly across brokers
▪ First assigned replica is preferred as leader.
▪ Writes/reads go to leader, which sends message to followers
38

Replication (at a high-level)
39

40

41

42

Kafka basics
▪ What is Kafka?
▪ Who uses Kafka?

No one-size-fits-all configuration
44

Tradeoff between performance and persistence
• Should broker send ack to producer right after step 1?
• Higher persistence and lower throughput with acks = -1 in producer config
X
45

Tradeoff between performance and message order
46
• Should producer send new message before ack of the last message?
• In-order delivery and lower throughput with
max.in.flight.requests.per.connection = 1 in producer config
Kafka BrokerProducer
message 1
message 0 failed
retry message 0
message 0

Tradeoff between persistence and availability
• Should we allow message produce if all in-sync replicas are offline?
• Higher availability and weaker persistence with
unclean.leader.election.enable = true in broker config
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5 60 1 2 3 4 5
Follower 1 Follower 2
Leader
Read Read
47
7 8
X
X

Tradeoff between availability and cost
• Do we need more replicas for the topic?
• Higher availability and higher cost with RF=3 in comparison to RF=2)
48
producer
Broker
Broker Broker
producer
Broker
Broker
RF=3 RF=2

Kafka basics
▪ What is Kafka?
▪ Who uses Kafka?

Kafka provides great performance, availability and data persistence
Are there other features that will be valuable to users?
50

Improved support for multi-tenancy
▪ Sasl/Kerberos and SSL support (KIP-12)
▪ Quota (KIP-13)
▪ Namespace in Kafka topics (KIP-37)
▪ Zookeeper authentication (KIP-38)
▪ End-to-end encryption
51

Reduced hardware and operational cost
▪ Dynamic configuration (KIP-21)
▪ Rack aware replica assignment (KIP-36)
▪ Self healing (KIP-46)
▪ On demand data deletion (KIP-107)
▪ JBOD support (KIP-112 and KIP-113)
52

Additional functionality for broader use-cases
▪ Kafka connect for data import/export (KIP-26)
▪ Streaming processor (KIP-28)
▪ Timestamp in message (KIP-32)
▪ Exactly-once delivery and transactional messaging (KIP-98)
53

Learn more about Kafka
▪ Stream processing meetup
▪ Kafka summit
▪ Kafka improvement proposals
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
▪ LinkedIn engineering blog https://engineering.linkedin.com/blog
54

Agenda
– Projects to monitor and manage Kafka servers
– Projects to monitor and debug Kafka clients
– Projects to make Kafka easier to use
– Projects that are built on Kafka
▪ Hands on (30 min)

Projects to monitor and manage Kafka servers
▪ cruise-control for automatically balancing partitions across brokers
▪ kafka-monitor for monitoring kafka service availability etc.
▪ kafka-audit for monitoring data loss
▪ InGraph for monitoring all JMX metrics from Kafka as time-series graph
57

Problems before having Cruise Control
▪ SRE needs to wake up at night to move partitions in case of hardware failure
▪ SRE needs to manually move partitions to balance load across brokers
▪ Reduced availability due to need to wait for manual recovery
▪ The partition movement may impact production traffic
58
Open sourced on Github in Aug, 2017

Cruise Control Architecture
59
▪ Self-heal from broker failure
▪ Balance load across brokers
without manual intervention
▪ Controlled impact on PROD
traffic when moving partitions

Example Cruise Control goals
▪ Partitions should be distributed across brokers in a rack-aware manner
▪ Broker resource utilization should be below the user-specified threshold
▪ Try to evenly distribute resource utilization across brokers
60

61

Problems before having Kafka Monitor
▪ Some issues are discovered only after bug report from Kafka user
▪ Can not quantify the availability and the latency of Kafka cluster
▪ Can not quantify the availability and the latency of Kafka mirrored pipeline
62

Kafka Monitor Architecture
63
▪ Alert on service unavailability
▪ Quantify service availability
▪ Measure end-to-end latency
▪ Detect violation of Kafka semantics
Our availability SLA is 99.99%

Other Kafka Monitor features
64
▪ Automatically distribute partitions of the monitor topic evenly across brokers
▪ Extensible module to export JMX metrics to various stores (e.g. Graphite)
▪ Pluggable interface to test Kafka service with your own client implementation
Open sourced on Github in May, 2016

65

Problems before having Kafka Audit
▪ Hard to help user identify why their message is not received
▪ Hard to detect and debug message loss in Kafka pipelines
66

Kafka Audit Architecture
67
▪ Detect messages loss
▪ Debug message loss
▪ Audit Kafka resource usage

Example Kafka Audit UI
68
When, where and how many of messages are delivered to Kafka

69

InGraph Architecture
70
Metric topic
in
Kafka Cluster
Broker
Broker
Client
InGraph
with
UI
Metric
messages
metric
messages

Example InGraph UI
71

Projects to monitor and debug Kafka clients
▪ Burrow for monitoring offset lag of consumer groups
▪ kafka-audit for monitoring Kafka resource usage per client
72

Burrow Architecture
▪ Detect lagging consumers
▪ Detect stalled consumers
▪ Detect stopped consumers
▪ Detect offset rewind
▪ Open sourced on Github
73

Projects to monitor and debug Kafka clients
▪ Burrow for monitoring offset lag of consumer groups
▪ kafka-audit for monitoring Kafka resource usage per client
74
Attribute the hardware cost in $$ to users of Kafka
and reduce unnecessary usage of Kafka

Projects to make Kafka easier to use
▪ kafka-rest to allow non-Java client to produce and consume from Kafka cluster
▪ schema-registry for conversion between binary data and IndexedRecord
▪ li-apache-kafka-clients to support large message etc.
▪ Nuage for users to create and manage properties (e.g. retention time) of their
topic by themselves
75

Kafka Rest Architecture
76
▪ Support non-Java clients
▪ No need to maintain client
libraries in multiple languages

topic by themselves
77

Schema Registry Architecture
78
▪ Enable efficient binary
encoding of schema in the
Kafka message
▪ Track schema evolution
for forward and backward
compatibility
Kafka Cluster
LiProducer
with
Schema cache
LiConsumer
with
Schema cache
IndexedRecord
IndexedRecord
Binary
data
Binary
data
Schema Registry
Register schema Fetch schema
User application User application

topic by themselves
79

Large message support in li-apache-kafka-clients
80

topic by themselves
81

Put things together
82

Help yourself with these open source projects
▪ Cruise Control (https://github.com/linkedin/cruise-control)
▪ Kafka Monitor (https://github.com/linkedin/kafka-monitor)
▪ Burrow (https://github.com/linkedin/burrow)
▪ li-apache-kafka-clients (https://github.com/linkedin/li-apache-kafka-clients)
▪ Future projects open sourced by LinkedIn streaming team can be found at
https://github.com/linkedin/streaming
83
All projects are actively maintained and used in LinkedIn production environment
100% free of charge!

Projects at LinkedIn that are built on Kafka
▪ Stream processing – Apache Samza
▪ Change data capture – Brooklin
▪ Strongly consistent key-value store – Espresso
▪ Efficient key-value store for derived data – Venice
84

Agenda
▪ Hands-on (30 min)

Hands-on
▪ Visit goo.gl/D7GFfB
▪ Single cluster
– Download and compile Apache Kafka
– Setup a cluster of one broker
– Create and describe topic
– Produce and consume using Apache Kafka tools
– Monitor availability of your cluster using Kafka Monitor
▪ Mirrored pipeline
– Setup another cluster of one broker
– Setup MM to mirror traffic from the source cluster to the destination cluster
– Produce to the source cluster and consume from the destination cluster
– Monitor availability of your pipeline using Kafka Monitor

An introduction to Apache Kafka and Kafka ecosystem at LinkedIn

More Related Content

What's hot (20)

Similar to An introduction to Apache Kafka and Kafka ecosystem at LinkedIn (20)

More from Dong Lin (6)

Recently uploaded (20)

An introduction to Apache Kafka and Kafka ecosystem at LinkedIn