SlideShare a Scribd company logo

Using Chaos Engineering to Level up Apache Kafka Skills

confluent, profile picture
confluent

The document discusses chaos engineering in the context of Kafka systems, emphasizing the importance of experimenting to increase confidence in system robustness. It presents various scenarios demonstrating how to handle broker failures, recover offline partitions, and the implications of data loss, along with replication strategies and deployment methods. The conclusions highlight that Kafka is not self-healing, and manual intervention is often necessary to maintain cluster health and data integrity.

Technology
1 of 52
Downloaded 39 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
1 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Kafka Chaos Engineering Shlomi Hassan & Yaniv Ranen
2 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. It’s a lovely day, let’s upgrade the production cluster
3 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3 We didn’t even get a chance to say goodbye
4 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Where did we go wrong?
5 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. Chaos engineering
6 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Hello! Yaniv Ranen Shlomi Hassan
7 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 77 ZipRecruiter’s mission We actively connect people to their next great opportunity.
8 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Logging infrastructure scheme
9 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Logging Kafka cluster Cluster spec 1) Kafka a) 8 EC2 - m4.4xlarge b) EBS - Io1 2) Zookeeper a) 5 EC2 - m4.large b) EBS - Io1 3) Cluster volume - 3TB/day 4) Broker data spread on Multi AZ KafkaZookeeper
10 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Basic Kafka terminology
11 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Basic Kafka terminology P0 R1 P1 R1 broker1 P2 R1 P3 R1 P0 R2 P1 R2 broker2 P2 R2 P3 R2 broker3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x Leader Replica is in bold Active controller is in bold
12 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
13 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Chaos engineering method ＋ Ask a question ＋ Write it down as a scenario in a collaborative document ＋ Check the behaviour in a controlled environment ＋ Document the exact commands and output
14 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 1414 Scenario #1 ＋ What happens if we kill one broker? ＋ Will Kafka self-heal?
15 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0Under replicated
16 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 54 Under replicated
17 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes
18 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes
19 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0Under replicatedReassign partitions
20 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2020 Demonstration
21 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2121 Scenario #1: Stopping a broker ＋ Kafka is not self-healing ＋ Manual reassign partitions is needed
22 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2 ＋ What can we do to revive an offline partition?
23 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions
24 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
25 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions Reassign partitions
26 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions Reassign partitions
27 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
28 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
29 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition Broker 1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
30 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions New topic Data loss
31 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3131 Scenario #2: Reviving an offline partition ＋ Restarting broker revives the offline partition ＋ Partition reassignment doesn’t work when it is offline ＋ Recreation of a broker ○ Data loss ○ Inconsistent state
32 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3 ＋ How can we recover data from an offline partition?
33 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions P0 R2 P1 R2 P2 R2 P3 R2
34 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions
35 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions
36 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 "Broker 1” ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions P0 - P1 - P2 - P3 -
37 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 “Broker 1” ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4under replicated P0 R2 P1 R2 P2 R2 P3 R2
38 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3838 Scenario #3: How can we recover from a lost AZ? ＋ Spawning a replacement node using the same EBS volumes ＋ Maintain the same broker ID ＋ Consumer group offsets are kept
39 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3939 Our Conclusions ＋ Chaos engineering helps in gaining knowledge ＋ Kafka is not self-healing ＋ Offline partitions can be brought back using EBS volumes ＋ Problems with the health check ＋ Using different versions of Kafka might introduce lag ＋ consider freezing the old version protocol Inter.broker.protocol.version,log.message.format.version ＋ Upgrade active controller last
40 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Deployment strategies ＋ In-place deployment might prove risky
41 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green deployment P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers
42 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Disconnecting producers and consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2
43 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Draining the old cluster into the new one P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool
44 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Reconnecting the producers and consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
45 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. The new cluster becomes the production one P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers
46 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Replication = data + offsets metadata
47 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Replication tools Tools name Pros Cons Mirror Maker Open source Relatively easy to use Not a real mirror UReplicator Creates a real mirror Scalable Open source Maintained mainly/only by Uber Kafka connect - Replicator Supports smart replication Creates a real mirror Based on Kafka Connect ecosystem Paid solution
48 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green as a failover solution P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Passive cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
49 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green as a failover solution P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Passive cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
50 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Kafka Pro Tips ＋ Be part of the community ○ Join confluent Slack team ○ follow /suggest new KIP (Kafka Improvement) ○ Contribute fixes to Kafka and it’s ecosystem ＋ Use smart metrics (like health check) for better visibility ＋ Try chaos engineering at home use our Github repo ＋ Don’t stay behind ○ Use updated Kafka consumers/producers ○ Update your cluster regularly ＋ Follow confluent white papers ＋ Kafka Health check repo
51 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Q & A
52 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Thank you https://www.linkedin.com/in/yaniv-ranen-284b003/ https://www.linkedin.com/in/shlomihassan/

More Related Content

PPTX
SIP: Call Id, Cseq, Via-branch, From & To-tag role play
Sridhar Kumar N
 
PPTX
SQL for pattern matching (Oracle 12c)
Logan Palanisamy
 
PDF
Choosing Mikrotik Platform x86 vs chr
GLC Networks
 
PDF
Building the Ultimate Device Matrix
Carly Vanderwert
 
PDF
Routing Information Protocol (RIP)
Tharindu Sankalpa
 
PPTX
Enable DPDK and SR-IOV for containerized virtual network functions with zun
heut2008
 
PDF
BPF: Tracing and more
Brendan Gregg
 
PDF
VXLAN and FRRouting
Faisal Reza
 
SIP: Call Id, Cseq, Via-branch, From & To-tag role play
Sridhar Kumar N
 
SQL for pattern matching (Oracle 12c)
Logan Palanisamy
 
Choosing Mikrotik Platform x86 vs chr
GLC Networks
 
Building the Ultimate Device Matrix
Carly Vanderwert
 
Routing Information Protocol (RIP)
Tharindu Sankalpa
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
heut2008
 
BPF: Tracing and more
Brendan Gregg
 
VXLAN and FRRouting
Faisal Reza
 

What's hot (20)

PDF
LSFMM 2019 BPF Observability
Brendan Gregg
 
PDF
MUM Melbourne : Build Enterprise Wireless with CAPsMAN
GLC Networks
 
PDF
From Mediasoup WebRTC to Livekit Self-Hosted .pdf
atyenoria
 
PDF
Kamailio, FreeSWITCH, and You
Fred Posner
 
PPTX
NGINX High-performance Caching
NGINX, Inc.
 
DOCX
Packet Tracer: Nat protocol
Rafat Khandaker
 
PPTX
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...
InfluxData
 
PDF
A whirlwind tour of the LLVM optimizer
Nikita Popov
 
PDF
OAuth and STUN, TURN in WebRTC context RFC7635
Mihály Mészáros
 
PDF
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Bruno Teixeira
 
DOCX
Packet Tracer: Load Balancing with GLBP and FHRP
Rafat Khandaker
 
PDF
Introduction to gRPC: A general RPC framework that puts mobile and HTTP/2 fir...
Codemotion
 
PPTX
How to Introduce Telemetry Streaming (gNMI) in Your Network with SNMP with Te...
InfluxData
 
PDF
Routing fundamentals with mikrotik
Achmad Mardiansyah
 
PDF
BGP Techniques for Network Operators
APNIC
 
PDF
BGP on mikrotik
Achmad Mardiansyah
 
PDF
Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...
Vietnam Open Infrastructure User Group
 
PDF
Linux BPF Superpowers
Brendan Gregg
 
PPTX
The TCP/IP Stack in the Linux Kernel
Divye Kapoor
 
PDF
How to Handle Asynchronous Behaviors Using SVA
DVClub
 
LSFMM 2019 BPF Observability
Brendan Gregg
 
MUM Melbourne : Build Enterprise Wireless with CAPsMAN
GLC Networks
 
From Mediasoup WebRTC to Livekit Self-Hosted .pdf
atyenoria
 
Kamailio, FreeSWITCH, and You
Fred Posner
 
NGINX High-performance Caching
NGINX, Inc.
 
Packet Tracer: Nat protocol
Rafat Khandaker
 
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...
InfluxData
 
A whirlwind tour of the LLVM optimizer
Nikita Popov
 
OAuth and STUN, TURN in WebRTC context RFC7635
Mihály Mészáros
 
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Bruno Teixeira
 
Packet Tracer: Load Balancing with GLBP and FHRP
Rafat Khandaker
 
Introduction to gRPC: A general RPC framework that puts mobile and HTTP/2 fir...
Codemotion
 
How to Introduce Telemetry Streaming (gNMI) in Your Network with SNMP with Te...
InfluxData
 
Routing fundamentals with mikrotik
Achmad Mardiansyah
 
BGP Techniques for Network Operators
APNIC
 
BGP on mikrotik
Achmad Mardiansyah
 
Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...
Vietnam Open Infrastructure User Group
 
Linux BPF Superpowers
Brendan Gregg
 
The TCP/IP Stack in the Linux Kernel
Divye Kapoor
 
How to Handle Asynchronous Behaviors Using SVA
DVClub
 
Ad

Similar to Using Chaos Engineering to Level up Apache Kafka Skills (20)

PDF
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
PROIDEA
 
PDF
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Priyanka Aash
 
PDF
Integrating Vert.x
Kirk Pepperdine
 
PDF
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
CODE BLUE
 
PPTX
How to Get Going with Kubernetes
Ted Dunning
 
PDF
Big Data LDN 2018: PROGRESS FOR BIG DATA IN KUBERNETES
Matt Stubbs
 
PPTX
Dataviz in a collaborative mixed reality space
Stefan Wasserbauer
 
PDF
HITCON FreeTalk 20240726 - Dark side of the Force - 探索暗網威脅【 議題三：藍隊的暗網事件應變守則】
Hacks in Taiwan (HITCON)
 
PDF
Trouble with memory
Kirk Pepperdine
 
PDF
Webinar: Why evasive zero day attacks are killing traditional sandboxing
Cyren, Inc
 
PDF
C-SEC|2016 Session 2 The Security Game : You Failed at the Beginning By Incog...
acinfotec
 
PDF
Rooted2020 the day i_ruled_the_world_deceiving_software_developers_through_op...
RootedCON
 
PDF
The day I ruled the world (RootedCON 2020)
Javier Junquera
 
PDF
Visão Geral de Inteligência Artificial
Amazon Web Services LATAM
 
PPTX
Conférence - Arbor Edge Defense, Première et dernière ligne de défense intell...
African Cyber Security Summit
 
PDF
Black Clouds and Silver Linings in Node.js Security - Liran Tal Snyk OWASP Gl...
Liran Tal
 
PPTX
Advanced Threat Hunting - Botconf 2017
Kevin Finley
 
PPTX
Advanced Threat Hunting - BotConf 2017
ThreatConnect
 
PPTX
My Baby Done Bad Crypto - My Sweet Baby Done Me Wrong
Dan Houser
 
PDF
Apache Pulsar at Yahoo! Japan
StreamNative
 
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
PROIDEA
 
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Priyanka Aash
 
Integrating Vert.x
Kirk Pepperdine
 
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
CODE BLUE
 
How to Get Going with Kubernetes
Ted Dunning
 
Big Data LDN 2018: PROGRESS FOR BIG DATA IN KUBERNETES
Matt Stubbs
 
Dataviz in a collaborative mixed reality space
Stefan Wasserbauer
 
HITCON FreeTalk 20240726 - Dark side of the Force - 探索暗網威脅【 議題三：藍隊的暗網事件應變守則】
Hacks in Taiwan (HITCON)
 
Trouble with memory
Kirk Pepperdine
 
Webinar: Why evasive zero day attacks are killing traditional sandboxing
Cyren, Inc
 
C-SEC|2016 Session 2 The Security Game : You Failed at the Beginning By Incog...
acinfotec
 
Rooted2020 the day i_ruled_the_world_deceiving_software_developers_through_op...
RootedCON
 
The day I ruled the world (RootedCON 2020)
Javier Junquera
 
Visão Geral de Inteligência Artificial
Amazon Web Services LATAM
 
Conférence - Arbor Edge Defense, Première et dernière ligne de défense intell...
African Cyber Security Summit
 
Black Clouds and Silver Linings in Node.js Security - Liran Tal Snyk OWASP Gl...
Liran Tal
 
Advanced Threat Hunting - Botconf 2017
Kevin Finley
 
Advanced Threat Hunting - BotConf 2017
ThreatConnect
 
My Baby Done Bad Crypto - My Sweet Baby Done Me Wrong
Dan Houser
 
Apache Pulsar at Yahoo! Japan
StreamNative
 
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
PDF
Migration, backup and restore made easy using Kannika
confluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
PDF
Unlocking value with event-driven architecture by Confluent
confluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
PDF
Building API data products on top of your real-time data infrastructure
confluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
confluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
Migration, backup and restore made easy using Kannika
confluent
 
Five Things You Need to Know About Data Streaming in 2025
confluent
 
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
Unlocking value with event-driven architecture by Confluent
confluent
 
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
Building API data products on top of your real-time data infrastructure
confluent
 
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 

Recently uploaded (20)

DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
A Presentation on Artificial Intelligence
memembruh336
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Encapsulation theory and applications.pdf
gurumoop
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 
Modernizing your data center with Dell and AMD
Principled Technologies
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 

Using Chaos Engineering to Level up Apache Kafka Skills

  • 1. 1 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Kafka Chaos Engineering Shlomi Hassan & Yaniv Ranen
  • 2. 2 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. It’s a lovely day, let’s upgrade the production cluster
  • 3. 3 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3 We didn’t even get a chance to say goodbye
  • 4. 4 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Where did we go wrong?
  • 5. 5 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. Chaos engineering
  • 6. 6 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Hello! Yaniv Ranen Shlomi Hassan
  • 7. 7 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 77 ZipRecruiter’s mission We actively connect people to their next great opportunity.
  • 8. 8 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Logging infrastructure scheme
  • 9. 9 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Logging Kafka cluster Cluster spec 1) Kafka a) 8 EC2 - m4.4xlarge b) EBS - Io1 2) Zookeeper a) 5 EC2 - m4.large b) EBS - Io1 3) Cluster volume - 3TB/day 4) Broker data spread on Multi AZ KafkaZookeeper
  • 10. 10 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Basic Kafka terminology
  • 11. 11 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Basic Kafka terminology P0 R1 P1 R1 broker1 P2 R1 P3 R1 P0 R2 P1 R2 broker2 P2 R2 P3 R2 broker3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x Leader Replica is in bold Active controller is in bold
  • 12. 12 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
  • 13. 13 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Chaos engineering method ＋ Ask a question ＋ Write it down as a scenario in a collaborative document ＋ Check the behaviour in a controlled environment ＋ Document the exact commands and output
  • 14. 14 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 1414 Scenario #1 ＋ What happens if we kill one broker? ＋ Will Kafka self-heal?
  • 15. 15 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0Under replicated
  • 16. 16 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 54 Under replicated
  • 17. 17 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes
  • 18. 18 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes
  • 19. 19 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #1: Stopping a broker P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 P0 R2 P1 R2 Broker 2 P2 R2 P3 R2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0Under replicatedReassign partitions
  • 20. 20 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2020 Demonstration
  • 21. 21 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2121 Scenario #1: Stopping a broker ＋ Kafka is not self-healing ＋ Manual reassign partitions is needed
  • 22. 22 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2 ＋ What can we do to revive an offline partition?
  • 23. 23 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions
  • 24. 24 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
  • 25. 25 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions Reassign partitions
  • 26. 26 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions Reassign partitions
  • 27. 27 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
  • 28. 28 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
  • 29. 29 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition Broker 1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 4offline partitions
  • 30. 30 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #2: Reviving an offline partition P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions New topic Data loss
  • 31. 31 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3131 Scenario #2: Reviving an offline partition ＋ Restarting broker revives the offline partition ＋ Partition reassignment doesn’t work when it is offline ＋ Recreation of a broker ○ Data loss ○ Inconsistent state
  • 32. 32 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3 ＋ How can we recover data from an offline partition?
  • 33. 33 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes Cluster health 0offline partitions P0 R2 P1 R2 P2 R2 P3 R2
  • 34. 34 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions
  • 35. 35 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions
  • 36. 36 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 "Broker 1” ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4offline partitions P0 - P1 - P2 - P3 -
  • 37. 37 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Scenario #3: Recovering offline partitions without data loss P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 “Broker 1” ZooKeeper Kafka cluster Producer Consumer Px Ry Replica y of partition x EBS volumes P0 R2 P1 R2 P2 R2 P3 R2 Cluster health 4under replicated P0 R2 P1 R2 P2 R2 P3 R2
  • 38. 38 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3838 Scenario #3: How can we recover from a lost AZ? ＋ Spawning a replacement node using the same EBS volumes ＋ Maintain the same broker ID ＋ Consumer group offsets are kept
  • 39. 39 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3939 Our Conclusions ＋ Chaos engineering helps in gaining knowledge ＋ Kafka is not self-healing ＋ Offline partitions can be brought back using EBS volumes ＋ Problems with the health check ＋ Using different versions of Kafka might introduce lag ＋ consider freezing the old version protocol Inter.broker.protocol.version,log.message.format.version ＋ Upgrade active controller last
  • 40. 40 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Deployment strategies ＋ In-place deployment might prove risky
  • 41. 41 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green deployment P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers
  • 42. 42 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Disconnecting producers and consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2
  • 43. 43 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Draining the old cluster into the new one P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool
  • 44. 44 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Reconnecting the producers and consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
  • 45. 45 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. The new cluster becomes the production one P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper New cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers
  • 46. 46 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Replication = data + offsets metadata
  • 47. 47 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Replication tools Tools name Pros Cons Mirror Maker Open source Relatively easy to use Not a real mirror UReplicator Creates a real mirror Scalable Open source Maintained mainly/only by Uber Kafka connect - Replicator Supports smart replication Creates a real mirror Based on Kafka Connect ecosystem Paid solution
  • 48. 48 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green as a failover solution P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Passive cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
  • 49. 49 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Blue green as a failover solution P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Old cluster P0 R2 P1 R2 P2 R2 P3 R2 Producers Consumers P0 R1 P1 R1 Broker 1 P2 R1 P3 R1 Broker 2 Broker 3 ZooKeeper Passive cluster P0 R2 P1 R2 P2 R2 P3 R2 Replication tool Producers Consumers
  • 50. 50 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Kafka Pro Tips ＋ Be part of the community ○ Join confluent Slack team ○ follow /suggest new KIP (Kafka Improvement) ○ Contribute fixes to Kafka and it’s ecosystem ＋ Use smart metrics (like health check) for better visibility ＋ Try chaos engineering at home use our Github repo ＋ Don’t stay behind ○ Use updated Kafka consumers/producers ○ Update your cluster regularly ＋ Follow confluent white papers ＋ Kafka Health check repo
  • 51. 51 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Q & A
  • 52. 52 ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. ZipRecruiter, Inc. Proprietary and Confidential. Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. Thank you https://www.linkedin.com/in/yaniv-ranen-284b003/ https://www.linkedin.com/in/shlomihassan/