Fundamentals and Architecture of Apache Kafka
Download as PPTX, PDF0 likes460 views
The document provides an overview of Apache Kafka, a distributed streaming platform known for its scalability, fault tolerance, and high throughput, and describes its architecture consisting of producers, consumers, brokers, and Zookeeper. It highlights the advantages of Kafka over traditional messaging queues, including message durability, real-time processing capabilities, and data partitioning for performance. Additionally, the document explains key components, the replication process for data durability, and Kafka's metrics for monitoring performance.
Fundamentals and Architecture of Apache Kafka
- 1. Fundamentals and Architecture of Apache Kafka® Angelo Cesaro
- 2. Who am I? • I’m Angelo! • Consultant and Data Engineer at Cesaro.io • More than 10 years of experience • Worked at ServiceNow, Sky • Follow me on https://www.linkedin.com/in/angelocesaro https://twitter.com/angelocesaro https://github.com/cesaroangelo
- 3. Apache Kafka – Overview • A distributed streaming platform used for building real time data pipelines and mission-critical streaming applications with the following characteristics: 1. Horizontally scalable 2. Fault tolerant 3. Really fast 4. Used by thousands of companies in production
- 4. Kafka’s benefits over traditional messages queues There are few key differences between kafka and other traditional messages queues • Durability and availability 1. Cluster can handle broker failures 2. Messages are replicated for reliability • Very high throughput • Data retention • Excellent scalability 1. A small kafka cluster can process a large number of messages • Support real-time and batch consumption 1. Kafka was born for real time processing of data, but can also handle batch oriented jobs, for example feeding data to Hadoop or a data warehouse
- 5. High level of a Kafka cluster • Producers send data to the kafka cluster • Consumers read data from the kafka cluster • Brokers are the main storage and messaging components of the kafka cluster Note: the components above can be physical machines, VMs or docker containers, kafka works the same on of those platforms.
- 6. Messages • The basic unit of data in kafka is a message and the messages are the atomic unit of data sent by producers • A message is a key-value pair: • All the data is stored in Kafka as byte arrays (very important!) • Producer provides serializers to convert the key and value to byte arrays • Key and value can be any data type
- 7. Topic • Kafka keeps streams of messages called topic and they categorize messages into groups • Developers can decide which topics have to exist and by default Kafka auto-create topics when they are first used • Kafka has no limit to the number of topics that can be used • Topics are logical representation that spans across brokers Note: By analogy, we can think topics as tables in a dbms, just like we separate data in a db in different tables, we do the same with topics
- 8. Data partitioning • Producers shard data over a group of partitions and this is needed to allow for parallel access to the topic for increased throughput • Each partition contains a subset of messages and they are ordered and immutable • Usually the message key is used to control which partition a message is assigned to
- 9. Kafka components • 4 key components are in a kafka system • Brokers • Producers • Consumers • Zookeeper
- 10. Kafka broker • Brokers receive and store data sent by the producers • Brokers are server class systems that provide messages to the consumers when requested • Messages are spread across multiple partitions in different brokers • Kafka provides a configurable retention policy for messages and each message is identified by its offset number • The commit log is an append only data structure that lives in ram for fast access and it’s flushed to disk periodically • Producer sends requests to the brokers that append messages to the end of the log • Consumers consumes from a specific offset (usually the lowest available) and consumes all messages sequentially
- 11. Kafka producers • Each producer writes data as messages to the kafka cluster • Producers can be written in any language • Kafka provides a tool to send messages to the cluster • Confluent develops a rest (representational state transfer) server which can be used by clients written in any language • Confluent Enterprise includes a MQTT (message queuing telemetry transport) proxy that allows direct ingestion of IoT data
- 12. Kafka consumers • Each consumer pull events from topics as they are written • The latest message read are kept tracked in a special ‘consumer offset’ topic • If necessary the consumers can be reset to start reading from a specific offset (parameter to set in the configuration for the default behavior) Note: other similar solutions use to push events
- 13. Distributed consumption • The way kafka uses to scale the consumption is the combination of multiple consumers into consumer groups • Each consumer in that scenario will be assigned a subset of partitions for consumption It’s important to know that traditional systems tend to be point to point, that means that a message is gone once it has been consumed and can’t be read again. Kafka was designed to work differently, to allow to use the data multiple times
- 14. Zookeeper • Zookeeper is a centralized and distributed service that can be used to enable highly reliable distributed coordination • It maintains configuration information (in this context kafka cluster configurations) • It provides distributed synchronization • It runs in cluster and provides resiliency against failures
- 15. Kafka & Zookeeper Kafka uses Zookeeper for various important features • Cluster management • Storage of ACLs and passwords • Failure detection and recovery Note: 1. kafka can’t run without zookeeper 2. In the previous kafka releases (<0.11), the clients had to access to zookeeper, from 0.11 only the brokers need that access and then the cluster is isolated from the clients for better security and performance
- 16. Advantages of a pull architecture • Ability to add more consumers to the system without reconfiguring the cluster • Ability for a consumer to go offline and return back later, resuming from where it left off • Consumer won’t get overwhelmed by data, consumer decides what speed to get data and slow consumers won’t affect fast producers
- 17. Speeding up data transfer Kafka is fast, but why? • Kafka uses system page cache for producing and consuming messages. (linux kernel feature) • The use of page cache enables zero-copy, the feature that allows to transfer data directly from local file channel to a remote socket. that saves cpu cycles and memory bandwidth.
- 18. Kafka metrics • Kafka metrics can be exposed via jmx and showed through jmx clients • Type of metrics exposed are: 1. Gauge: instantaneous measurement of one value 2. Meter: measurement of ticks in a time range. E.g. one minute rate, 10 minutes rate, etc 3. Histogram: measurement of a value variants. E.g. 50th percentile, 98th percentile, etc 4. Timer: measurement of timings meter + histogram Kafka uses yammer metrics on the broker and in the older <0.9 clients. New clients uses new internal metric package. Confluent plans to consolidate the jmx metrics packages in the future.
- 19. Why Replication? • Each partition is stored in a broker • If we wouldn’t have any replication and if a broker goes offline, then the partitions stored in that broker won’t be available and a permanent data loss could occur • Without redundancy, partitions will be not available for reads and writes if the server goes offline and if the server has a fatal crash, the data is gone permanently Kafka uses replication for durability and availability
- 20. Replica • Each partition can have replicas • Each replica is placed on different brokers • Replicas are spread evenly across brokers for load balancing We specify the replication factor at topic creation time
- 21. Rack awareness of replicas • Rack awareness enables each replica to be placed on brokers in different racks. That helps to improve performance and fault tolerance. • Each broker can be configured with a broker.rack property, e.g. rack-1, us-east-1a • It’s useful if we need to deploy kafka on AWS across availability zones • Rack awareness was introduced in Confluent 3.0
- 22. Replica configurations • Increase the replication factor for better durability • For auto created topics, by default Kafka use replication factor 1, that needs to be configured accordingly in server.properties Kafka-topics –create –zookeeper zookeeper:2181 –partitions 1 – replication-factor 3 –topic mytopic
- 23. How brokers are involved in replication • Brokers ensure strongly consistent replicas • One replica is on the leader broker • All messages produced go to the leader • The leader propagates those messages to the followers brokers • All consumers read messages from the leader Note: very important to understand this above in case of troubleshooting ;)
- 24. Leaders and followers • Leader: 1. Accepts all reads and writes 2. Manages replicas 3. Leader election rate (meter metric): kafka.controller:type=controllerstatus,name=leaderelectionrateandtime ms • Follower: • Provide fault tolerance • keep up with the leader • There is a special thread running in the cluster that manage the current list of leaders and followers for every partition. It’s a complex and mission- critical task, for this reason there is a replica of this information in zookeeper and then cached on every broker for faster access
- 25. Partition leaders • Leaders have to be evenly distributed across all brokers for 2 main reasons: • Leaders can change in case of failure • Leaders do more work as discussed in the previous slides
- 26. Preferred replica • When we create a topic the preferred replica is set automatically. • It’s the first replica in the list of assigned replicas • Kafka-topics –zookeeper zookeeper:2181 –describe –topic my-topic Topic:my-topic PartitionCount:1 ReplicationFactor:3 Configs: Topic: my- topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
- 27. In Sync Replica (ISR) • In sync replica is a list of the replicas – leader + followers • A message is committed if it’s received by every replica in the list Note for troubleshooting: Where is it kept the isr list? It’s in the leader
- 28. What does committed mean? • Committed means in this context that the message is received and written to the disk by all replicas • The data is not available for consuming if it’s not committed • Who decides when to commit a message? The leader has this responsibility
- 29. Using Kafka command line tools #create topic with replication factor 1 and partition 1 • kafka-topics.sh --create --zookeeper localhost:2181 --replication- factor 1 --partitions 1 --topic test #delete topic with name test • kafka-topics.sh --delete --zookeeper localhost:2181 --topic test #list info regarding topic • kafka-topics.sh --describe --zookeeper localhost:2181 --topic test #list topics • kafka-topics.sh --list --zookeeper localhost:2181
- 30. Links! • https://kafka.apache.org • https://www.confluent.io • https://www.cesaro.io