A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya and Rajini Sivaram | Kafka Summit London 2022
0 likes1,436 views
The document provides an in-depth overview of Apache Kafka replication, including its structure, necessity for replication, and its capabilities for durability and availability during node failures. It discusses multi-zone high availability, global replication for disaster protection, and various deployment strategies such as active-passive and active-active setups. Security measures, network considerations, and operational recovery strategies are also highlighted to ensure robust Kafka cluster management.
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya and Rajini Sivaram | Kafka Summit London 2022
- 1. A Hitchhiker's Guide to Apache Kafka® Geo-Replication Sanjana Kaundinya | Senior Software Engineer Rajini Sivaram | Principal Software Engineer
- 2. Overview of Kafka Replication
- 3. Kafka Overview ● Broker - Stores messages in partitions ● Topic - Virtual Group of one or more partitions ● Partitions - Log files on disk with only sequential writes. Kafka guarantees message ordering in a partition. Broker T1 P P2 P1 C C C P
- 4. CG1 Kafka Log Offsets P1 C1 P C2 4 5 6 7 8 9 0 1 2 3 P2 P3 P4 Partition 1 __consumer_offsets startOffset CG1 CG2 HW LEO Produce ● Append Only Log ● Log End Offset ● High Watermark ● Consumer Offsets
- 5. Why do we Need Replication ? How can a broker go down? ● Controlled shutdown ● Uncontrolled shutdown What happens when a broker goes down? ● Durability ● Availability
- 6. Kafka Replication ● Partition replicas are evenly distributed ● Byte for byte copy of each other ● One replica is a leader and all writes go to the leader ● Leader decides when to commit data P1 (L) P2 (L) P4 P1 P2 P3 P4 P3 (L) P1 P3 P4 (L) P2 Replication Factor = 3
- 7. How are messages committed ? ● Leader maintains in sync replicas (ISR) ● Failure cases are handled with the use of a leader epoch ● The leader epoch is part of the message(KIP-101) R1 (L) R2 (L) R4 (L) R2 R1 R3 R4 R3 R3 R3 (L) R4 R2 P1 (L)
- 8. Salient Points for Replication ● Intra cluster Replication helps improve durability and availability for node level failures. ● Offsets are core piece of Kafka producer and consumer ecosystem. ● Kafka Replication protocol ensures strong consistency through byte for byte replication and providing message ordering guarantees.
- 9. Multi Zone(MZ) HA Kafka Cluster B B ZK zk P C B zk AZ1 AZ2 AZ3 Inter Zone Latency <10 ms Typical ~3 ms ZK ZK
- 10. Why Do We Need To Globally Replicate ? ● Global Availability ● Protection against disasters ○ Natural disaster ○ Cloud provider outage ● Regulatory Compliance ● Aggregate Clusters ● IOT use cases ● Migration from one region to another
- 11. Differences Among Multi-DC Solutions Stretched Clusters Connected Clusters
- 13. Stretched Clusters ● Offset Preserving ● Fast Disaster Recovery ● Automated Client Failover with No Custom Code ● Sync or Async Replication per Topic with Confluent’s Multi-Region Clusters 13
- 14. 3 DC Stretched Cluster
- 15. 2.5 DC Stretched Cluster
- 16. Fetch from Followers ● With KIP-392, consumers can read from the closest replica ● This helps to save on networking costs and helps with overall latency 16
- 17. Confluent Multi-Region Clusters (MRC) Leader Follower Observer ● Sync vs Async replication ● Replica placement
- 18. MRC: Automatic Observer Promotion Leader Follower Observer observerPromotionPolicy ● under-min-isr ● under-replicated ● leader-is-observer
- 19. Network Considerations ● Single Kafka Cluster with bi-directional connectivity ● Cost of cross-DC traffic ● Network Latency: < 50ms between DCs ○ Sync: client impact ○ Async: durability impact ● Network partitions ● Replication tuning: buffer sizes, fetcher threads 19
- 20. Security Considerations ● Authentication using SSL or SASL_SSL for inter-broker connections ● Wire-encryption using TLS ● Single Kafka Cluster ○ Single account and access management for clients ○ ACLs apply across whole cluster 20
- 22. Clusters can replicate using Kafka Connect ● Have two separate Kafka clusters in use ● Different from a single stretched cluster ● Offset Translation ● MirrorMaker 2.0 and Confluent Replicator Connect based Replication 22 C
- 23. Fundamentals of Kafka Connect ● Offset management ● Elastic scalability ● Parallelization ● Task distribution ● Failure & Retries ● Configuration Management ● REST API
- 24. Multi-Geo Replication Through MirrorMaker 2 MirrorMaker 2
- 25. Offset Translation in MirrorMaker 2.0 offset_sync topic, partition, src offset, matching dest offset checkpoints topic, partition, group name, consumer group src offset, matching dest offset Consumer translateOffsets Destination Cluster
- 26. Offset Translation in Replicator 26
- 27. Network Considerations ● Where to run Connect based clusters? ○ local producer, remote consumer ● Connectivity from Connect to source and destination brokers ○ Firewalls ● High Latency networks ○ Kafka batch sizes ○ TCP buffers: OS level and application level ○ Automatic window scaling 27
- 28. Security Considerations ● Credentials ○ Source credentials ○ Destination credentials ○ Externalize passwords ● Wire encryption using TLS ● Access control ○ Access to read from source cluster ○ Access to write to destination cluster ○ Naming conventions: prefixed ACLs 28
- 29. Connecting Clusters Sans Kafka Connect ● Multi continent replication without the an external system ● Offset preserving, eliminating need for offset translation ● Has similar use cases as Kafka Connect based architectures Cluster Linking 29
- 30. Multi-Geographic Deployment Strategies with Apache Kafka
- 31. Active-Passive ● One cluster is the primary, other cluster is the standby ● The primary cluster is the only one written to ● Commonly used topology used for regulatory compliance 31 Producer Active DC Passive DC Consumer Consumer Replication
- 32. Active-Active ● Two clusters replicate to each other ● Records are produced to both clusters and seen by clients in both clusters ● Used for a globally distributed architecture, data needs to be regionally available 32 Producer Active DC Active DC Consumer Consumer Producer Replication Replication
- 33. Preventing Cyclic Replication in an Active-Active Setup How do connected clusters prevent cyclic replication? ● MirrorMaker 2.0 uses alias detection ● Confluent Replicator adds a provenance header to each record which contains: ○ ID of the origin cluster ○ Name of the topic ○ Timestamp
- 34. Fan-In AKA Aggregation ● Multiple clusters write to one centralized cluster ● Can aggregate into one centralized topic or do this on the central cluster ● Use cases: aggregation, analytics, IOT 34 DC Producer Producer DC Aggregate DC DC Producer R e p l i c a t i o n Replication Replication Consumer
- 35. Fan-Out ● One cluster writes out to multiple other clusters ● Only one cluster is actively produced to ● Use cases: expanded version of active-passive setups, IOT 35 DC Consumer DC Central DC DC R e p l i c a t i o n Replication Replication Producer Consumer Consumer
- 36. Disaster Recovery: Failing Over ● If primary cluster goes down, all producers have to be move to the secondary cluster ● Need to ensure that consumer applications can resume where they last left off 36 R R R A - Primary ZK R R R B - Secondary ZK Producer Replication Consumer
- 37. 37 R R R B - Secondary ZK Disaster Recovery: Failing Back ● Once the disaster is mitigated, switch back to the primary cluster ● Have to ensure client applications can write back to the original cluster R R R A - Primary ZK Producer Consumer Resume Replication Reconciliation
- 38. Operational Last point system was operational Disaster Disaster strikes and system goes down 2 1 Recovery Begin recovery after disaster strikes Normalcy System back to being operational 4 3 38 Disaster Recovery: Metrics Recovery Point Objective Recovery Time Objective
- 39. Which multi-geo deployment to choose? ● It really depends! ● Considerations: ○ Cost ○ Business Requirements ○ Use Case ○ Regulatory Compliance ● Two must haves: ○ Resilient to disasters ○ Security
- 40. Questions?