Kafka syed academy_v1_introduction
0 likes162 views
This document provides an introduction to Apache Kafka. It discusses why Kafka is needed for real-time streaming data processing and real-time analytics. It also outlines some of Kafka's key features like scalability, reliability, replication, and fault tolerance. The document summarizes common use cases for Kafka and examples of large companies that use it. Finally, it describes Kafka's core architecture including topics, partitions, producers, consumers, and how it integrates with Zookeeper.
Kafka syed academy_v1_introduction
- 1. Kafka Syed Solutions Engineer - Big Data mail.syed786@gmail.com info.syedacademy@gmail.com +91-9030477368
- 3. Kafka grow ing Why Kafka? Kafka adoption is on the rise but why
- 4. Kafka growth exploding 1/3 of all Fortune 500 companies Top ten travel companies, 7 of top ten banks, 8 of top ten insurance companies, 9 of top ten telecom companies LinkedIn, Microsoft and Netflix process 4 comma message a day with Kafka (1,000,000,000,000) Real-time streams of data, used to collect big data or to do real time analysis (or both)
- 5. Why Kafka is Needed? Real time streaming data processed for real time analytics Service calls, track every call, IOT sensors Apache Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system Kafka is often used instead of JMS, RabbitMQ and AMQP higher throughput, reliability and replication Kafka can works in combination with • Flume, Spark Streaming, Storm, HBase and Spark for real-time analysis and processing of streaming data • Feed your data lakes with data streams Kafka brokers support massive message streams for follow-up analysis in Hadoop or Spark Kafka Streaming (subproject) can be used for real-time analytics
- 6. Kafka Use Cases Stream Processing Website Activity Tracking Metrics Collection and Monitoring Log Aggregation Real time analytics Capture and ingest data into Spark / Hadoop Guaranteed distributed commit log for in-memory computing Who uses Kafka? LinkedIn: Activity data and operational metrics Twitter: Uses it as part of Storm – stream processing infrastructure Square: Kafka as bus to move all system events to various Square data centers (logs, custom events, metrics, an so on). Outputs to Splunk, Graphite, Esper-like alerting systems Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box, Cisco, CloudFlare, DataDog, LucidWorks,
- 7. Why is Kafka Popular? Great performance Operational Simplicity, easy to setup and use, easy to reason Stable, Reliable Durability, Flexible Publish-subscribe/queue (scales with N- number of consumer groups), Robust Replication, Producer Tunable Consistency Guarantees, Ordering Preserved at shard level (Topic Partition) Works well with systems that have data streams to process, aggregate, transform & load into other stores Why is Kafka so fast? Zero Copy - calls the OS kernel direct rather to move data fast Batch Data in Chunks - Batches data into chunks end to end from Producer to file system to Consumer Provides More efficient data compression. Reduces I/O latency Sequential Disk Writes - Avoids Random Disk Access writes to immutable commit log. No slow disk seeking. No random I/O operations. Disk accessed in sequential manner Horizontal Scale - uses 100s to thousands of partitions for a single topic spread out to thousands of servers handle massive load
- 9. Distributed Streaming Platform Publish and Subscribe to streams of records Fault tolerant storage Replicates Topic Log Partitions to multiple servers Process records as they occur Fast, efficient IO, batching, compression, and more Used to decouple data streams Features of Kafka
- 10. Kafka Decoupling Data Streams
- 11. Kafka Polyglot clients /Wire protocol Kafka communication from clients and servers wire protocol over TCP protocol Protocol versioned Maintains backwards compatibility Many languages supported Kafka REST proxy allows easy integration Also provides Avro/Schema registry support via Kafka ecosystem Kafka Record Retention Kafka cluster retains all published records Time based – configurable retention period Size based - configurable based on size Compaction - keeps latest record Retention policy of three days or two weeks or a month It is available for consumption until discarded by time, size or compaction Consumption speed not impacted by size
- 12. Kafka scalable message storage Kafka acts as a good storage system for records/messages Records written to Kafka topics are persisted to disk and replicated to other servers for fault-tolerance Kafka Producers can wait on acknowledgment Write not complete until fully replicated Kafka disk structures scales well Writing in large streaming batches is fast Clients/Consumers can control read position (offset) Kafka acts like high-speed file system for commit log storage, replication
- 14. Kafka Fundamentals • Records have a key (optional), value and timestamp; Immutable • Topic a stream of records (“/orders”, “/user-signups”), feed name – Log topic storage on disk – Partition / Segments (parts of Topic Log) • Producer API to produce a streams or records • Consumer API to consume a stream of records • Broker : Kafka server that runs in a Kafka Cluster. Brokers form a cluster. Cluster consists on many Kafka Brokers on many servers. • ZooKeeper: Does coordination of brokers/cluster topology. Consistent file system for configuration information and leadership election for Broker Topic Partition Leaders
- 15. Kafka: Topics, Producers, and Consumers Kafka Cluster Topic Producer Producer Producer Consumer Consumer Consumer record record
- 16. Core Kafka
- 17. Kafka needs Zookeeper Zookeeper helps with leadership election of Kafka Broker and Topic Partition pairs Zookeeper manages service discovery for Kafka Brokers that form the cluster Zookeeper sends changes to Kafka New Broker join, Broker died, etc. Topic removed, Topic added, etc. Zookeeper provides in-sync view of Kafka Cluster configuration
- 18. Kafka Producer/Consumer Details Producers write to and Consumers read from Topic(s) Topic associated with a log which is data structure on disk Producer(s) append Records at end of Topic log Topic Log consist of Partitions - Spread to multiple files on multiple nodes Consumers read from Kafka at their own cadence Each Consumer (Consumer Group) tracks offset from where they left off reading Partitions can be distributed on different machines in a cluster high performance with horizontal scalability and failover with replication
- 19. Kafka Topic Partition, Consumers, Producers 0 1 42 3 5 6 7 8 9 10 11 Partition 0 Consumer Group A Producer Consumer Group B Consumer groups remember offset where they left off. Consumers groups each have their own offset. Producer writing to offset 12 of Partition 0 while… Consumer Group A is reading from offset 6. Consumer Group B is reading from offset 9.
- 20. Kafka Scale and Speed How can Kafka scale if multiple producers and consumers read/write to same Kafka Topic log? Writes fast: Sequential writes to filesystem are fast (700 MB or more a second) Scales writes and reads by sharding: Topic logs into Partitions (parts of a Topic log) Topics logs can be split into multiple Partitions different machines/different disks Multiple Producers can write to different Partitions of the same Topic Multiple Consumers Groups can read from different partitions efficiently
- 21. Kafka Brokers Kafka Cluster is made up of multiple Kafka Brokers Each Broker has an ID (number) Brokers contain topic log partitions Connecting to one broker bootstraps client to entire cluster Start with at least three brokers, cluster can have, 10, 100, 1000 brokers if needed Kafka Cluster, Failover, ISRs Topic Partitions can be replicated across multiple nodes for failover Topic should have a replication factor greater than 1 (2, or 3) Failover if one Kafka Broker goes down then Kafka Broker with ISR (in-sync replica) can serve data
- 22. ZooKeeper does coordination for Kafka Cluster Kafka BrokerProducer Producer Producer Consumer Consumer Consumer Kafka Broker Kafka Broker Topic ZooKeeper
- 23. Failover vs. Disaster Recovery Replication of Kafka Topic Log partitions allows for failure of a rack or AWS availability zone You need a replication factor of at least 3 Kafka Replication is for Failover Mirror Maker is used for Disaster Recovery Mirror Maker replicates a Kafka cluster to another data-center or AWS region Called mirroring since replication happens within a cluster
- 25. Topics, Logs, Partitions Kafka Topic is a stream of records Topics stored in log Log broken up into partitions and segments Topic is a category or stream name or feed Topics are pub/sub Can have zero or many subscribers - consumer groups Topics are broken up and spread by partitions for speed and size
- 26. Topic Partitions Topics are broken up into partitions Partitions decided usually by key of record Key of record determines which partition Partitions are used to scale Kafka across many servers Record sent to correct partition by key Partitions are used to facilitate parallel consumers Records are consumed in parallel up to the number of partitions Order guaranteed per partition Partitions can be replicated to multiple brokers
- 27. Topic Partition Log Order is maintained only in a single partition Partition is ordered, immutable sequence of records that is continually appended to—a structured commit log Records in partitions are assigned sequential id number called the offset Offset identifies each record within the partition Topic Partitions allow Kafka log to scale beyond a size that will fit on a single server Topic partition must fit on servers that host it topic can span many partitions hosted on many servers Topic Partitions are unit of parallelism - a partition can only be used by one consumer in group at a time Consumers can run in their own process or their own thread If a consumer stops, Kafka spreads partitions across remaining consumer in group
- 28. Topic Parallelism and Consumers Topic Partitions are unit of parallelism - a partition can only be used by one consumer in group at a time Consumers can run in their own process or their own thread If a consumer stops, Kafka spreads partitions across remaining consumer in group #of Consumers you can run per Consumer Group limited by #of Partitions
- 29. Kafka Topic Partitions Layout 0 1 42 3 5 6 7 8 9 10 11 0 1 42 3 5 6 7 8 0 1 42 3 5 6 7 8 9 10 Older Newer 0 1 42 3 5 6 7 Partition 0 Partition 1 Partition 2 Partition 3 Writes
- 30. Replication: Kafka Partition Distribution Each partition has leader server and zero or more follower servers Leader handles all read and write requests for partition Followers replicate leader, and take over if leader dies Used for parallel consumer handling within a group Partitions of log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of partitions Each partition can be replicated across a configurable number of Kafka servers - Used for fault tolerance Replication: Kafka Partition Leader One node/partition’s replicas is chosen as leader. Leader handles all reads and writes of Records for partition Writes to partition are replicated to followers (node/partition pair) An follower that is in-sync is called an ISR (in- sync replica) If a partition leader fails, one ISR is chosen as new
- 31. Kafka Replication to Partition 0 Kafka Broker 0 Partition 0 Partition 1 Partition 2 Partition 3 Partition 4 Kafka Broker 1 Partition 0 Partition 1 Partition 2 Partition 3 Partition 4 Kafka Broker 2 Partition 1 Partition 2 Partition 3 Partition 4 Client Producer 1) Write record Partition 0 2) Replicate record 2) Replicate record Leader Red Follower Blue Record is considered "committed" when all ISRs for partition wrote to their log. Only committed records are readable from consumer
- 32. Kafka Replication to Partitions 1 Kafka Broker 0 Partition 0 Partition 1 Partition 2 Partition 3 Partition 4 Kafka Broker 1 Partition 0 Partition 1 Partition 2 Partition 3 Partition 4 Kafka Broker 2 Partition 1 Partition 2 Partition 3 Partition 4 Client Producer 1) Write record Partition 02) Replicate record 2) Replicate record Another partition can be owned by another leader on another Kafka broker Leader Red Follower Blue
- 33. Kafka versus
- 34. Kafka vs JMS, SQS, RabbitMQ Messaging Is Kafka a Queue or a Pub/Sub/Topic? Yes Kafka is like a Queue per consumer group Kafka is a queue system per consumer in consumer group so load balancing like JMS, RabbitMQ queue Kafka is like Topics in JMS, RabbitMQ, MOM Topic/pub/sub by offering Consumer Groups which act like subscriptions Broadcast to multiple consumer groups MOM = JMS, ActiveMQ, RabbitMQ, IBM MQ Series, Tibco, etc.
- 35. Kafka vs MOM By design, Kafka is better suited for scale than traditional MOM systems due to partition topic log Load divided among Consumers for read by partition Handles parallel consumers better than traditional MOM Also by moving location (partition offset) in log to client/consumer side of equation instead of the broker, less tracking required by Broker and more flexible consumers Kafka written with mechanical sympathy, modern hardware, cloud in mind Disks are faster Servers have tons of system memory
- 36. Kinesis and Kafka are similar Kinesis Streams is like Kafka Core Kinesis Analytics is like Kafka Streams Kinesis Shard is like Kafka Partition Similar and get used in similar use cases In Kinesis, data is stored in shards. In Kafka, data is stored in partitions Kinesis Analytics allows you to perform SQL like queries on data streams Kafka Streaming allows you to perform functional aggregations and mutations Kafka integrates well with Spark and Flink which allows SQL like queries on streams
- 37. Kafka vs. Kinesis Data is stored in Kinesis for default 24 hours, and you can increase that up to 7 days. Kafka records default stored for 7 days can increase until you run out of disk space. Decide by the size of data or by date. Can use compaction with Kafka so it only stores the latest timestamp per key per record in the log With Kinesis data can be analyzed by lambda before it gets sent to S3 or RedShift With Kinesis you pay for use, by buying read and write units. Kafka is more flexible than Kinesis but you have to manage your own clusters, and requires some dedicated DevOps resources to keep it going Kinesis is sold as a service and does not require a DevOps team to keep it going (can be more expensive and less flexible, but much easier to setup and run)