Intoduction to Apache Kafka
- 1. Apache Kafka Veysel Gündüzalp - Senior Software Engineer @Trendyol twitter.com/v_gunduzalp linkedin.com/in/vgunduzalp
- 2. Why we need Apache Kafka - Is it a good technology? - Hype? - Over-Engineering?
- 3. Communication Types - request-response - callback - push - pull - poll
- 4. Disadvantage of Request-Response - Timeout - Client has to wait - Insufficient server resource - The application is closed and data loss - Network problems
- 5. Why Apache Kafka? - Low latency - Active-active working - High availability - Scalability and extensibility - Multiple consumers - Async processing - Creating language-independent applications
- 6. Broker - A Kafka broker acts as a server or a node in a Kafka cluster, and is responsible for managing the storage and replication of messages across multiple nodes - What is the difference from database?
- 7. Topic - Topic is a category or feed name to which messages are published by producers - The message order is only guarantee inside a partition (FIFO) - You can’t delete message - How should the topic naming be
- 8. Topic Partitions - Topic partition is a logical divisionof a topic's data into multiple, ordered segments that can be distributed across multiple brokers in a Kafka cluster. - Each partition is a sequence of immutable, ordered messages that can be processed independently from other partitions. - Each partition can be processed independently, allowing for parallel processing of messages across multiple consumers.
- 9. Topic Replication - Topic replication is a key feature of Kafka that provides high availability, fault tolerance, and scalability for message storage and processing in distributed systems. - Replication Factor should be at least 2, recommended 3, up to 4.
- 11. Message
- 12. Message Key - Message key is an optional attribute that is used to control how messages are partitioned and distributed across the topic's partitions - You can use for ordered messages
- 13. Message Header Message header is a set of key-value pairs that provide metadata and context information for a message - X-CorrelationId - X-AgentName - RetryCount
- 14. Producer Producer is a component that sends messages to a Kafka topic
- 16. Consumer - Consumer is an application or process that reads messages from a Kafka topic. - The consumer periodically polls Kafka for new messages and processes them one at a time - Batch? - Async?
- 17. Consumer Delivery Semantics At mostonce(0, 1)
- 18. Consumer Delivery Semantics At leastonce(1, n)
- 19. Consumer Delivery Semantics Exactlyonce - Consuming from a Kafka topic and producing to another topic (as in a Kafka Streams application) - Uses transactional producer ve consumer - Uses a two-phase commit protocol for both producers and consumers.
- 21. Cleanup Policy (Delete-Compaction) - Log compaction is a feature in Apache Kafka that allows for the selective removal of duplicate records from a topic's log
- 22. What is the lag? Kafka used to store and manage consumer group offsets in the __consumer_offsets topic
- 23. Why is Kafka fast? - Zero copy is a shortcut to save the multiple data copies between application context and kernel context. - This approach brings down the time by approximately 65%. https://twitter.com/alexxubyte/status/1506663791961919488 https://medium.com/free-code-camp/what-makes-apache-kafka-so-fast-a8d4f94ab145
- 24. Questions? Veysel Gündüzalp - Senior Software Engineer @Trendyol twitter.com/v_gunduzalp linkedin.com/in/vgunduzalp