Stream Processing Metamorphosis - A Kafka's tale
2 likes1,684 views
The document presents an overview of Kafka, a distributed streaming platform, detailing its architecture, components, and operational mechanisms. It highlights Kafka's advantages over other messaging systems, such as its capacity for real-time data handling, scalability, and durability through concepts like topic partitioning and replication. Additionally, it discusses Kafka's evolution and its role in modern streaming architectures, like Kappa and Lambda architectures.
Stream Processing Metamorphosis - A Kafka's tale
- 1. TUGA IT 2017 LISBON, PORTUGAL
- 3. Stream Processing Metamorphosis A Kafka’s tale JLVV
- 4. “ “Paths are made by walking” Frank Kafka JLVV
- 5. João Vazão Vasques Data Engineer at Talkdesk Scala, Clojure, Ruby, Python @JoaoVasques
- 6. Agenda Kafka Motivation Main concepts Architecture & Design Comparison of other message queue systems Main concepts Architecture Community Metamorphosis Surprise surprise JLVV
- 7. 1 JLVV
- 8. Motivation Why do we need Kafka? JLVV
- 9. Motivation - Why Kafka Client Backend 1 It all starts like this JLVV
- 10. Motivation - Why Kafka Client Backend 1 Client Backend 2 JLVV
- 11. Motivation - Why Kafka Client Backend 1 Client Client Client Backend 2 Backend 3 Backend 4 Then it starts to look like this JLVV
- 12. Motivation - Why Kafka LinkedIn before Kafka https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying JLVV
- 13. Motivation - Why Kafka There must be a better way to handle this… JLVV
- 14. Motivation - Why Kafka Source System Kafka decouples data pipelines Source System Source System Real-Time monitoring Billing …. JLVV
- 15. Motivation - Why Kafka Kafka decouples data pipelines https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying LinkedIn after Kafka JLVV
- 17. A log JLVV
- 18. JLVV
- 20. 1 2 3 4 5 6 7 8 First record 0 Next record 9 “Append-only sequence of records ordered by time” Jay Kreps JLVV
- 21. JLVV
- 24. Concepts Broker Topics Stream of messages of a particular type (e.g. calls, user clicks, likes, page views, etc..) JLVV
- 25. Concepts Broker Topics Producers Publish messages to a Kafka topic JLVV
- 26. Concepts Broker Topics Producers Consumers Subscribe to a topic and process the published messages JLVV
- 27. Architecture Broker Broker Broker Consumer Consumer Producer Producer Zookeeper JLVV
- 28. Broker JLVV
- 29. Broker Node in a cluster running a Kafka server JLVV
- 30. Broker Node in a cluster running a Kafka server Stores one of more partitions per topic JLVV
- 31. Topics JLVV
- 32. Topics - partitions 1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5 6 partition 1 partition 2 partition 3 Old New • A Kafka topic, T, is split in P partitions JLVV
- 33. Topics - partitions 1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5 6 partition 1 partition 2 partition 3 Old New • A Kafka topic, T, is split in P partitions • Messages in each partition are identified by sequential id (offset) JLVV
- 34. Topics - partitions 1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5 6 partition 1 partition 2 partition 3 Old New • A Kafka topic, T, is split in P partitions • Messages in each partition are identified by sequential id (offset) • Partition is the smallest unit of parallelism in Kafka (ordering) JLVV
- 35. Topics - partitions 1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5 6 partition 1 partition 2 partition 3 Old New • A Kafka topic, T, is split in P partitions • Messages in each partition are identified by sequential id (offset) • Partition is the smallest unit of parallelism in Kafka (ordering) • Each message in a topic has a time-based retention policy JLVV
- 36. Topics - Replication • Topics should be replicated JLVV
- 37. Topics - Replication • Topics should be replicated • Each partition has 1 leader and 0 or more replicas JLVV
- 38. Topics - Replication • Topics should be replicated • Each partition has 1 leader and 0 or more replicas • Replica is in-sync if: • can communicate with Zookeeper • Not far behind the leader JLVV
- 39. Topics - Replication • Topics should be replicated • Each partition has 1 leader and 0 or more replicas • Replica is in-sync if: • can communicate with Zookeeper • Not far behind the leader • Replication factor cannot be lowered JLVV
- 40. Topics - Log compaction • Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition JLVV
- 41. Topics - Log compaction • Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition JLVV
- 42. Producers JLVV
- 43. Producers ◦ Publish messages to a partition in a topic JLVV 1 2 3 4 5 6 7
- 44. Producers ◦ Publish messages to a partition in a topic JLVV 1 2 3 4 5 6 7 8
- 45. Producers ◦ Publish messages to a partition in a topic ◦ Load balancing (using partitions) ◦ round robin (if no key) JLVV 1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5 6 partition 1 partition 2 partition 3 Old New
- 46. Producers ◦ Publish messages to a partition in a topic ◦ Load balancing (using partitions) ◦ round robin (if no key) ◦ Hash function based on key & number of partitions (if key) JLVV
- 47. Producers ◦ Publish messages to a partition in a topic ◦ Load balancing (using partitions) ◦ round robin (if no key) ◦ Hash function based on key and number of partitions ◦ Replication Acks Durability Latency 0 some data loss no latency 1 (wait for leader) a few data loss 1 network roundtrip B (all brokers) no data loss B -1 network roundtrips JLVV
- 48. Consumers JLVV
- 49. Consumers ◦ Multiple consumers can read from the same topic (but from a single partition) 23 24 25 26 27 28 Consumer Consumer Consumer Fetch Fetch Fetch JLVV
- 50. Consumers ◦ Multiple consumers can read from the same topic (but from a single partition) ◦ Messages stay in Kafka after they’re consumed 23 24 25 26 27 28 Consumer Consumer Consumer Fetch Fetch Fetch JLVV
- 51. Consumers ◦ Multiple consumers can read from the same topic (but from a single partition) ◦ Messages stay in Kafka after they’re consumed ◦ Consumers can go away 23 24 25 26 27 28 Consumer Consumer Fetch Fetch JLVV
- 52. Consumers ◦ Multiple consumers can read from the same topic (but from a single partition) ◦ Messages stay in Kafka after they’re consumed ◦ Consumers can go away… and them come back 23 24 25 26 27 28 Consumer Consumer Fetch Fetch Consumer Fetch JLVV
- 53. Consumer groups ◦ Consumers can be organised into consumer groups. Each group has a coordinator JLVV
- 54. Consumer groups ◦ Consumers can be organised into consumer groups Pattern I All consumer instances in one group. Acts like a queue with load balancing JLVV
- 55. Consumer groups ◦ Consumers can be organised into consumer groups Pattern II All consumers instances in different groups. All messages are broadcast to all consumer instances Pattern I All consumer instances in one group. Acts like a queue with load balancing JLVV
- 56. Consumer groups ◦ Consumers can be organised into consumer groups Pattern II All consumers instances in different groups. All messages are broadcast to all consumer instances Pattern III Many consumer instances in a group. Added for scalability and fault tolerance. Each reads from one or more partitions of a topic. Pattern I All consumer instances in one group. Acts like a queue with load balancing JLVV
- 58. Comparison with other systems Let’s compare Kafka with another very popular messaging system JLVV
- 59. Comparing Kafka and RabbitMQ System Language Rate Routing Tooling Reprocessing Slow consumers? Kafka Scala + Java ~100K/sec No Decent Yes Ok Rabbit Erlang ~20K/sec Yes Very good No Hell JLVV
- 61. Why Kafka is like arriving to Mars
- 63. JLVV The rise of streaming architectures Kappa Architecture • Pure streaming system (no batch layer a la Lambda) • Stream + retention (Kafka) —> Real Time —> Serving • Streaming system: • Kafka Streams • Flink • Spark Streaming • Storm
- 65. 1012 events per day 20M events/sec Numbers of April 2017 3 Petabytes data per day At peak JLVV
- 67. Thanks! ANY QUESTIONS? You can find me at @JoaoVasques
- 68. THANK YOU TO OUR SPONSORS PLATINUM GOLD SILVER