Kafka Tutorial, Kafka ecosystem with clustering examples

™
Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka
Consulting
Cassandra and Kafka Support on AWS/EC2
Cloudurable
Introduction to Kafka
Support around Cassandra
and Kafka running in EC2

™
Consulting

™
Consulting
Kafka growing
Why Kafka?
Kafka adoption is on the rise
but why
What is Kafka?

Consulting
™
Kafka growth exploding
❖ Kafka growth exploding
❖ 1/3 of all Fortune 500 companies
❖ Top ten travel companies, 7 of top ten banks, 8 of top
ten insurance companies, 9 of top ten telecom
companies
❖ LinkedIn, Microsoft and Netflix process 4 comma
message a day with Kafka (1,000,000,000,000)
❖ Real-time streams of data, used to collect big data or to
do real time analysis (or both)

Consulting
™
Why Kafka is Needed?
❖ Real time streaming data processed for real time
analytics
❖ Service calls, track every call, IOT sensors
❖ Apache Kafka is a fast, scalable, durable, and fault-
tolerant publish-subscribe messaging system
❖ Kafka is often used instead of JMS, RabbitMQ and
AMQP
❖ higher throughput, reliability and replication

Consulting
™
Why is Kafka needed? 2
❖ Kafka can works in combination with
❖ Flume/Flafka, Spark Streaming, Storm, HBase and Spark
for real-time analysis and processing of streaming data
❖ Feed your data lakes with data streams
❖ Kafka brokers support massive message streams for follow-
up analysis in Hadoop or Spark
❖ Kafka Streaming (subproject) can be used for real-time
analytics

Consulting
™
Kafka Use Cases
❖ Stream Processing
❖ Website Activity Tracking
❖ Metrics Collection and Monitoring
❖ Log Aggregation
❖ Real time analytics
❖ Capture and ingest data into Spark / Hadoop
❖ CRQS, replay, error recovery
❖ Guaranteed distributed commit log for in-memory computing

Consulting
™
Who uses Kafka?
❖ LinkedIn: Activity data and operational metrics
❖ Twitter: Uses it as part of Storm – stream processing
infrastructure
❖ Square: Kafka as bus to move all system events to various
Square data centers (logs, custom events, metrics, an so
on). Outputs to Splunk, Graphite, Esper-like alerting
systems
❖ Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box,
Cisco, CloudFlare, DataDog, LucidWorks, MailChimp,
NetFlix, etc.

Consulting
™
Why is Kafka Popular?
❖ Great performance
❖ Operational Simplicity, easy to setup and use, easy to reason
❖ Stable, Reliable Durability,
❖ Flexible Publish-subscribe/queue (scales with N-number of consumer groups),
❖ Robust Replication,
❖ Producer Tunable Consistency Guarantees,
❖ Ordering Preserved at shard level (Topic Partition)
❖ Works well with systems that have data streams to process, aggregate,
transform & load into other stores
Most important reason: Kafka’s great performance: throughput, latency, obtained through great engineering

Consulting
™
Why is Kafka so fast?
❖ Zero Copy - calls the OS kernel direct rather to move data fast
❖ Batch Data in Chunks - Batches data into chunks
❖ end to end from Producer to file system to Consumer
❖ Provides More efficient data compression. Reduces I/O latency
❖ Sequential Disk Writes - Avoids Random Disk Access
❖ writes to immutable commit log. No slow disk seeking. No random I/O
operations. Disk accessed in sequential manner
❖ Horizontal Scale - uses 100s to thousands of partitions for a single topic
❖ spread out to thousands of servers
❖ handle massive load

Consulting
™
Streaming Architecture

Consulting
™
Why Kafka Review
❖ Why is Kafka so fast?
❖ How fast is Kafka usage growing?
❖ How is Kafka getting used?
❖ Where does Kafka fit in the Big Data Architecture?
❖ How does Kafka relate to real-time analytics?
❖ Who uses Kafka?

™
Consulting
Cassandra / Kafka Support in EC2/AWS
What is Kafka? Kafka messaging
Kafka Overview

Consulting
™
What is Kafka?
❖ Distributed Streaming Platform
❖ Publish and Subscribe to streams of records
❖ Fault tolerant storage
❖ Replicates Topic Log Partitions to multiple servers
❖ Process records as they occur
❖ Fast, efficient IO, batching, compression, and more
❖ Used to decouple data streams

Consulting
™
Kafka Decoupling Data
Streams

Consulting
™
Kafka Polyglot clients / Wire
protocol
❖ Kafka communication from clients and servers wire
protocol over TCP protocol
❖ Protocol versioned
❖ Maintains backwards compatibility
❖ Many languages supported
❖ Kafka REST proxy allows easy integration
❖ Also provides Avro/Schema registry support via Kafka
ecosystem

Consulting
™
Kafka Usage
❖ Build real-time streaming applications that react to streams
❖ Real-time data analytics
❖ Transform, react, aggregate, join real-time data flows
❖ Feed events to CEP for complex event processing
❖ Feed data lakes
❖ Build real-time streaming data pipe-lines
❖ Enable in-memory microservices (actors, Akka, Vert.x, Qbit,
RxJava)

Consulting
™
Kafka Use Cases
❖ Metrics / KPIs gathering
❖ Aggregate statistics from many sources
❖ Event Sourcing
❖ Used with microservices (in-memory) and actor systems
❖ Commit Log
❖ External commit log for distributed systems. Replicated
data between nodes, re-sync for nodes to restore state
❖ Real-time data analytics, Stream Processing, Log
Aggregation, Messaging, Click-stream tracking, Audit trail,
etc.

Consulting
™
Kafka Record Retention
❖ Kafka cluster retains all published records
❖ Time based – configurable retention period
❖ Size based - configurable based on size
❖ Compaction - keeps latest record
❖ Retention policy of three days or two weeks or a month
❖ It is available for consumption until discarded by time, size or
compaction
❖ Consumption speed not impacted by size

Consulting
™
Kafka scalable message
storage
❖ Kafka acts as a good storage system for records/messages
❖ Records written to Kafka topics are persisted to disk and replicated to
other servers for fault-tolerance
❖ Kafka Producers can wait on acknowledgment
❖ Write not complete until fully replicated
❖ Kafka disk structures scales well
❖ Writing in large streaming batches is fast
❖ Clients/Consumers can control read position (offset)
❖ Kafka acts like high-speed file system for commit log storage,
replication

Consulting
™
Kafka Review
❖ How does Kafka decouple streams of data?
❖ What are some use cases for Kafka where you work?
❖ What are some common use cases for Kafka?
❖ How is Kafka like a distributed message storage
system?
❖ How does Kafka know when to delete old messages?
❖ Which programming languages does Kafka support?

™
Consulting
Kafka Architecture

Consulting
™
Kafka Fundamentals
❖ Records have a key (optional), value and timestamp; Immutable
❖ Topic a stream of records (“/orders”, “/user-signups”), feed name
❖ Log topic storage on disk
❖ Partition / Segments (parts of Topic Log)
❖ Producer API to produce a streams or records
❖ Consumer API to consume a stream of records
❖ Broker: Kafka server that runs in a Kafka Cluster. Brokers form a cluster.
Cluster consists on many Kafka Brokers on many servers.
❖ ZooKeeper: Does coordination of brokers/cluster topology. Consistent file
system for configuration information and leadership election for Broker Topic
Partition Leaders

Consulting
™
Kafka: Topics, Producers, and
Consumers
Kafka
Cluster
Topic
Producer
Producer
Producer
Consumer
Consumer
Consumer
record
record

Consulting
™
Core Kafka

Consulting
™
Kafka needs Zookeeper
❖ Zookeeper helps with leadership election of Kafka Broker and
Topic Partition pairs
❖ Zookeeper manages service discovery for Kafka Brokers that
form the cluster
❖ Zookeeper sends changes to Kafka
❖ New Broker join, Broker died, etc.
❖ Topic removed, Topic added, etc.
❖ Zookeeper provides in-sync view of Kafka Cluster configuration

Consulting
™
Kafka Producer/Consumer
Details
❖ Producers write to and Consumers read from Topic(s)
❖ Topic associated with a log which is data structure on disk
❖ Producer(s) append Records at end of Topic log
❖ Topic Log consist of Partitions -
❖ Spread to multiple files on multiple nodes
❖ Consumers read from Kafka at their own cadence
❖ Each Consumer (Consumer Group) tracks offset from where they left off reading
❖ Partitions can be distributed on different machines in a cluster
❖ high performance with horizontal scalability and failover with replication

Consulting
™
Kafka Topic Partition, Consumers,
Producers
0 1 42 3 5 6 7 8 9 10 11
Partition
0
Consumer Group A
Producer
Consumer Group B
Consumer groups remember offset where they left off.
Consumers groups each have their own offset.
Producer writing to offset 12 of Partition 0 while…
Consumer Group A is reading from offset 6.
Consumer Group B is reading from offset 9.

Consulting
™
Kafka Scale and Speed
❖ How can Kafka scale if multiple producers and consumers read/write to
same Kafka Topic log?
❖ Writes fast: Sequential writes to filesystem are fast (700 MB or more a
second)
❖ Scales writes and reads by sharding:
❖ Topic logs into Partitions (parts of a Topic log)
❖ Topics logs can be split into multiple Partitions different
machines/different disks
❖ Multiple Producers can write to different Partitions of the same Topic
❖ Multiple Consumers Groups can read from different partitions efficiently

Consulting
™
Kafka Brokers
❖ Kafka Cluster is made up of multiple Kafka Brokers
❖ Each Broker has an ID (number)
❖ Brokers contain topic log partitions
❖ Connecting to one broker bootstraps client to entire
cluster
❖ Start with at least three brokers, cluster can have, 10,
100, 1000 brokers if needed

Consulting
™
Kafka Cluster, Failover, ISRs
❖ Topic Partitions can be replicated
❖ across multiple nodes for failover
❖ Topic should have a replication factor greater than 1
❖ (2, or 3)
❖ Failover
❖ if one Kafka Broker goes down then Kafka Broker with
ISR (in-sync replica) can serve data

Consulting
™
ZooKeeper does coordination for Kafka
Cluster
Kafka BrokerProducer
Producer
Producer
Consumer
Consumer
Consumer
Kafka Broker
Kafka Broker
Topic
ZooKeeper

Consulting
™
Failover vs. Disaster Recovery
❖ Replication of Kafka Topic Log partitions allows for failure of
a rack or AWS availability zone
❖ You need a replication factor of at least 3
❖ Kafka Replication is for Failover
❖ Mirror Maker is used for Disaster Recovery
❖ Mirror Maker replicates a Kafka cluster to another data-center
or AWS region
❖ Called mirroring since replication happens within a cluster

Consulting
™
Kafka Review
❖ How does Kafka decouple streams of data?
❖ What are some use cases for Kafka where you work?
❖ What are some common use cases for Kafka?
❖ What is a Topic?
❖ What is a Broker?
❖ What is a Partition? Offset?
❖ Can Kafka run without Zookeeper?
❖ How do implement failover in Kafka?
❖ How do you implement disaster recovery in Kafka?

™
Consulting
Kafka versus

Consulting
™
Kafka vs JMS, SQS, RabbitMQ
Messaging
❖ Is Kafka a Queue or a Pub/Sub/Topic?
❖ Yes
❖ Kafka is like a Queue per consumer group
❖ Kafka is a queue system per consumer in consumer group so load
balancing like JMS, RabbitMQ queue
❖ Kafka is like Topics in JMS, RabbitMQ, MOM
❖ Topic/pub/sub by offering Consumer Groups which act like
subscriptions
❖ Broadcast to multiple consumer groups
❖ MOM = JMS, ActiveMQ, RabbitMQ, IBM MQ Series, Tibco, etc.

Consulting
™
Kafka vs MOM
❖ By design, Kafka is better suited for scale than traditional MOM systems due
to partition topic log
❖ Load divided among Consumers for read by partition
❖ Handles parallel consumers better than traditional MOM
❖ Also by moving location (partition offset) in log to client/consumer side of
equation instead of the broker, less tracking required by Broker and more
flexible consumers
❖ Kafka written with mechanical sympathy, modern hardware, cloud in mind
❖ Disks are faster
❖ Servers have tons of system memory
❖ Easier to spin up servers for scale out

Consulting
™
Kinesis and Kafka are similar
❖ Kinesis Streams is like Kafka Core
❖ Kinesis Analytics is like Kafka Streams
❖ Kinesis Shard is like Kafka Partition
❖ Similar and get used in similar use cases
❖ In Kinesis, data is stored in shards. In Kafka, data is stored in partitions
❖ Kinesis Analytics allows you to perform SQL like queries on data streams
❖ Kafka Streaming allows you to perform functional aggregations and mutations
❖ Kafka integrates well with Spark and Flink which allows SQL like queries on
streams

Consulting
™
Kafka vs. Kinesis
❖ Data is stored in Kinesis for default 24 hours, and you can increase that up to 7 days.
❖ Kafka records default stored for 7 days
❖ can increase until you run out of disk space.
❖ Decide by the size of data or by date.
❖ Can use compaction with Kafka so it only stores the latest timestamp per key per record
in the log
❖ With Kinesis data can be analyzed by lambda before it gets sent to S3 or RedShift
❖ With Kinesis you pay for use, by buying read and write units.
❖ Kafka is more flexible than Kinesis but you have to manage your own clusters, and requires
some dedicated DevOps resources to keep it going
❖ Kinesis is sold as a service and does not require a DevOps team to keep it going (can be
more expensive and less flexible, but much easier to setup and run)

™
Consulting
Kafka Topics
Kafka Topics
Architecture
Replication
Failover
Parallel processing
Kafka Topic Architecture

Consulting
™
Topics, Logs, Partitions
❖ Kafka Topic is a stream of records
❖ Topics stored in log
❖ Log broken up into partitions and segments
❖ Topic is a category or stream name or feed
❖ Topics are pub/sub
❖ Can have zero or many subscribers - consumer groups
❖ Topics are broken up and spread by partitions for speed and
size

Consulting
™
Topic Partitions
❖ Topics are broken up into partitions
❖ Partitions decided usually by key of record
❖ Key of record determines which partition
❖ Partitions are used to scale Kafka across many servers
❖ Record sent to correct partition by key
❖ Partitions are used to facilitate parallel consumers
❖ Records are consumed in parallel up to the number of partitions
❖ Order guaranteed per partition
❖ Partitions can be replicated to multiple brokers

Consulting
™
Topic Partition Log
❖ Order is maintained only in a single partition
❖ Partition is ordered, immutable sequence of records that is continually appended
to—a structured commit log
❖ Records in partitions are assigned sequential id number called the offset
❖ Offset identifies each record within the partition
❖ Topic Partitions allow Kafka log to scale beyond a size that will fit on a single server
❖ Topic partition must fit on servers that host it
❖ topic can span many partitions hosted on many servers
❖ Topic Partitions are unit of parallelism - a partition can only be used by one
consumer in group at a time
❖ Consumers can run in their own process or their own thread
❖ If a consumer stops, Kafka spreads partitions across remaining consumer in group

Consulting
™
Kafka Topic Partitions Layout
0 1 42 3 5 6 7 8 9 10 11
0 1 42 3 5 6 7 8
0 1 42 3 5 6 7 8 9 10
Older Newer
0 1 42 3 5 6 7
Partition
0
Partition
1
Partition
2
Partition
3
Writes

Consulting
™
Replication: Kafka Partition
Distribution
❖ Each partition has leader server and zero or more follower
servers
❖ Leader handles all read and write requests for partition
❖ Followers replicate leader, and take over if leader dies
❖ Used for parallel consumer handling within a group
❖ Partitions of log are distributed over the servers in the Kafka cluster
with each server handling data and requests for a share of
partitions
❖ Each partition can be replicated across a configurable number of
Kafka servers - Used for fault tolerance

Consulting
™
Replication: Kafka Partition
Leader
❖ One node/partition’s replicas is chosen as leader
❖ Leader handles all reads and writes of Records for
partition
❖ Writes to partition are replicated to followers
(node/partition pair)
❖ An follower that is in-sync is called an ISR (in-sync
replica)
❖ If a partition leader fails, one ISR is chosen as new
leader

Consulting
™
Kafka Replication to Partition 0
Kafka Broker 0
Partition 0
Partition 1
Partition 2
Partition 3
Partition 4
Kafka Broker 1
Partition 0
Partition 1
Partition 2
Partition 3
Partition 4
Kafka Broker 2
Partition 1
Partition 2
Partition 3
Partition 4
Client Producer
1) Write record
Partition 0
2) Replicate
record
2) Replicate
record
Leader Red
Follower Blue
Record is considered "committed"
when all ISRs for partition
wrote to their log.
Only committed records are
readable from consumer

Consulting
™
Kafka Replication to Partitions
1
Kafka Broker 0
Partition 0
Partition 1
Partition 2
Partition 3
Partition 4
Kafka Broker 1
Partition 0
Partition 1
Partition 2
Partition 3
Partition 4
Kafka Broker 2
Partition 1
Partition 2
Partition 3
Partition 4
Client Producer
1) Write record
Partition 0
2) Replicate
record
2) Replicate
record
Another partition can
be owned
by another leader
on another Kafka broker
Leader Red
Follower Blue

Consulting
™
Topic Review
❖ What is an ISR?
❖ How does Kafka scale Consumers?
❖ What are leaders? followers?
❖ How does Kafka perform failover for Consumers?
❖ How does Kafka perform failover for Brokers?

™
Consulting
Kafka Producers
Kafka Producers
Partition selection
Durability
Kafka Producer Architecture

Consulting
™
Kafka Producers
❖ Producers send records to topics
❖ Producer picks which partition to send record to per topic
❖ Can be done in a round-robin
❖ Can be based on priority
❖ Typically based on key of record
❖ Kafka default partitioner for Java uses hash of keys to
choose partitions, or a round-robin strategy if no key
❖ Important: Producer picks partition

Consulting
™
Kafka Producers and
Consumers
0 1 42 3 5 6 7 8 9 10 11
Partition
0
Producers
Consumer Group A
Producers are writing at Offset 12
Consumer Group A is Reading from Offset 9.

Consulting
™
Kafka Producers
❖ Producers write at their own cadence so order of Records
cannot be guaranteed across partitions
❖ Producer configures consistency level (ack=0, ack=all,
ack=1)
❖ Producers pick the partition such that Record/messages
goes to a given same partition based on the data
❖ Example have all the events of a certain 'employeeId' go to
same partition
❖ If order within a partition is not needed, a 'Round Robin'
partition strategy can be used so Records are evenly
distributed across partitions.

Consulting
™
Producer Review
❖ Can Producers occasionally write faster than
consumers?
❖ What is the default partition strategy for Producers
without using a key?
❖ What is the default partition strategy for Producers using
a key?
❖ What picks which partition a record is sent to?

™
Consulting
Kafka Consumers
Load balancing consumers
Failover for consumers
Offset management per consumer
group
Kafka Consumer Architecture

Consulting
™
Kafka Consumer Groups
❖ Consumers are grouped into a Consumer Group
❖ Consumer group has a unique id
❖ Each consumer group is a subscriber
❖ Each consumer group maintains its own offset
❖ Multiple subscribers = multiple consumer groups
❖ Each has different function: one might delivering records to
microservices while another is streaming records to Hadoop
❖ A Record is delivered to one Consumer in a Consumer Group
❖ Each consumer in consumer groups takes records and only one consumer
in group gets same record
❖ Consumers in Consumer Group load balance record consumption

Consulting
™
Kafka Consumer Load Share
❖ Kafka Consumer consumption divides partitions over consumers in a
Consumer Group
❖ Each Consumer is exclusive consumer of a "fair share" of partitions
❖ This is Load Balancing
❖ Consumer membership in Consumer Group is handled by the Kafka
protocol dynamically
❖ If new Consumers join Consumer group, it gets a share of partitions
❖ If Consumer dies, its partitions are split among remaining live Consumers
in Consumer Group

Consulting
™
0 1 42 3 5 6 7 8 9 10 11
Partition
0
Consumer Group A
Producers
Consumer Group B
Consumers remember offset where they left off.
Consumers groups each have their own offset per partition.

Consulting
™
Processing
❖ How does Kafka divide up topic so multiple Consumers in a Consumer
Group can process a topic?
❖ You group consumers into consumers group with a group id
❖ Consumers with same id belong in same Consumer Group
❖ One Kafka broker becomes group coordinator for Consumer Group
❖ assigns partitions when new members arrive (older clients would talk
direct to ZooKeeper now broker does coordination)
❖ or reassign partitions when group members leave or topic changes
(config / meta-data change
❖ When Consumer group is created, offset set according to reset policy of
topic

Consulting
™
Kafka Consumer Failover
❖ Consumers notify broker when it successfully processed a
record
❖ advances offset
❖ If Consumer fails before sending commit offset to Kafka
broker,
❖ different Consumer can continue from the last committed
offset
❖ some Kafka records could be reprocessed
❖ at least once behavior
❖ messages should be idempotent

Consulting
™
Kafka Consumer Offsets and
Recovery
❖ Kafka stores offsets in topic called “__consumer_offset”
❖ Uses Topic Log Compaction
❖ When a consumer has processed data, it should
commit offsets
❖ If consumer process dies, it will be able to start up and
start reading where it left off based on offset stored in
“__consumer_offset”

Consulting
™
Kafka Consumer: What can be
consumed?
❖ "Log end offset" is offset of last record written
to log partition and where Producers write to
next
❖ "High watermark" is offset of last record
successfully replicated to all partitions followers
❖ Consumer only reads up to “high watermark”.
Consumer can’t read un-replicated data

Consulting
™
Consumer to Partition
Cardinality
❖ Only a single Consumer from the same Consumer
Group can access a single Partition
❖ If Consumer Group count exceeds Partition count:
❖ Extra Consumers remain idle; can be used for failover
❖ If more Partitions than Consumer Group instances,
❖ Some Consumers will read from more than one
partition

Consulting
™
2 server Kafka cluster hosting 4 partitions (P0-P5)
Kafka Cluster
Server 2
P0 P1 P5
Server 1
P2 P3 P4
Consumer Group A
C0 C1 C3
Consumer Group B
C0 C1 C3

Consulting
™
Multi-threaded Consumers
❖ You can run more than one Consumer in a JVM process
❖ If processing records takes a while, a single Consumer can run multiple threads to process
records
❖ Harder to manage offset for each Thread/Task
❖ One Consumer runs multiple threads
❖ 2 messages on same partitions being processed by two different threads
❖ Hard to guarantee order without threads coordination
❖ PREFER: Multiple Consumers can run each processing record batches in their own thread
❖ Easier to manage offset
❖ Each Consumer runs in its thread
❖ Easier to mange failover (each process runs X num of Consumer threads)

Consulting
™
Consumer Review
❖ What is a consumer group?
❖ Does each consumer have its own offset?
❖ When can a consumer see a record?
❖ What happens if there are more consumers than
partitions?
❖ What happens if you run multiple consumers in many
thread in the same JVM?

™
Consulting
Using Kafka Single Node
Using Kafka Single
Node
Run ZooKeeper, Kafka
Create a topic
Send messages from command
line
Read messages from command
line
Tutorial Using Kafka Single Node

Consulting
™
Run Kafka
❖ Run ZooKeeper start up script
❖ Run Kafka Server/Broker start up script
❖ Create Kafka Topic from command line
❖ Run producer from command line
❖ Run consumer from command line

Consulting
™
Run ZooKeeper

Consulting
™
Run Kafka Server

Consulting
™
Create Kafka Topic

Consulting
™
List Topics

Consulting
™
Run Kafka Producer

Consulting
™
Run Kafka Consumer

Consulting
™
Running Kafka Producer and
Consumer

Consulting
™
Kafka Single Node Review
❖ What server do you run first?
❖ What tool do you use to create a topic?
❖ What tool do you use to see topics?
❖ What tool did we use to send messages on the command line?
❖ What tool did we use to view messages in a topic?
❖ Why were the messages coming out of order?
❖ How could we get the messages to come in order from the
consumer?

™
Consulting
Use Kafka to send and receive messages
Lab Use Kafka
Use single server version of
Kafka.
Setup single node.
Single ZooKeeper.
Create a topic.
Produce and consume messages
from the command line.

™
Consulting
Using Kafka Cluster
and Failover
Demonstrate Kafka Cluster
Create topic with replication
Show consumer failover
Show broker failover
Kafka Tutorial Cluster and
Failover

Consulting
™
Objectives
❖ Run many Kafka Brokers
❖ Create a replicated topic
❖ Demonstrate Pub / Sub
❖ Demonstrate load balancing
consumers
❖ Demonstrate consumer failover
❖ Demonstrate broker failover

Consulting
™
Running many nodes
❖ If not already running, start up ZooKeeper
❖ Shutdown Kafka from first lab
❖ Copy server properties for three brokers
❖ Modify properties files, Change port, Change Kafka log
location
❖ Start up many Kafka server instances
❖ Create Replicated Topic
❖ Use the replicated topic

Consulting
™
Create three new server-
n.properties files
❖ Copy existing server.properties to server-
0.properties, server-1.properties, server-2.properties
❖ Change server-1.properties to use log.dirs
“./logs/kafka-logs-0”
❖ Change server-1.properties to use port 9093, broker
id 1, and log.dirs “./logs/kafka-logs-1”
❖ Change server-2.properties to use port 9094, broker
id 2, and log.dirs “./logs/kafka-logs-2”

Consulting
™
Modify server-x.properties
❖ Each have different
broker.id
❖ Each have different
log.dirs
❖ Each had different
port

Consulting
™
Create Startup scripts for three Kafka
servers
❖ Passing properties files
from last step

Consulting
™
Run Servers

Consulting
™
Create Kafka replicated topic my-
failsafe-topic
❖ Replication Factor is set to 3
❖ Topic name is my-failsafe-topic
❖ Partitions is 13

Consulting
™
Start Kafka Consumer
❖ Pass list of Kafka servers to bootstrap-
server
❖ We pass two of the three
❖ Only one needed, it learns about the rest

Consulting
™
Start Kafka Producer
❖ Start producer
❖ Pass list of Kafka Brokers

Consulting
™
Kafka 1 consumer and 1 producer
running

Consulting
™
Start a second and third
consumer
❖ Acts like pub/sub
❖ Each consumer
in its own group
❖ Message goes to
each
❖ How do we load
share?

Consulting
™
Running consumers in same
group
❖ Modify start consumer script
❖ Add the consumers to a group called
mygroup
❖ Now they will share load

Consulting
™
Start up three consumers
again
❖ Start up producer and three consumers
❖ Send 7 messages
❖ Notice how messages are spread among 3 consumers

Consulting
™
Consumer Failover
❖ Kill one consumer
❖ Send seven more
messages
❖ Load is spread to
remaining consumers
❖ Failover WORK!

Consulting
™
Create Kafka Describe Topic
❖ —describe will show list partitions, ISRs, and
partition leadership

Consulting
™
Use Describe Topics
❖ Lists which broker owns (leader of) which partition
❖ Lists Replicas and ISR (replicas that are up to
date)
❖ Notice there are 13 topics

Consulting
™
Test Broker Failover: Kill 1st
server
se Kafka topic describe to see that a new leader was elected!
Kill the first server

Consulting
™
Show Broker Failover Worked
❖ Send two more
messages from the
producer
❖ Notice that the
consumer gets the
messages
❖ Broker Failover WORKS!

Consulting
™
Kafka Cluster Review
❖ Why did the three consumers not load share the
messages at first?
❖ How did we demonstrate failover for consumers?
❖ How did we demonstrate failover for producers?
❖ What tool and option did we use to show ownership of
partitions and the ISRs?

™
Consulting
Use Kafka to send and receive messages
Lab 2 Use Kafka
multiple nodes
Use a Kafka Cluster to
replicate a Kafka topic log

™
Consulting
Kafka Ecosystem
Kafka Connect
Kafka Streaming
Kafka Schema Registry
Kafka REST

Consulting
™
Kafka Ecosystem
❖ Kafka Streams
❖ Streams API to transform, aggregate, process records from a stream and
produce derivative streams
❖ Kafka Connect
❖ Connector API reusable producers and consumers
❖ (e.g., stream of changes from DynamoDB)
❖ Kafka REST Proxy
❖ Producers and Consumers over REST (HTTP)
❖ Schema Registry - Manages schemas using Avro for Kafka Records
❖ Kafka MirrorMaker - Replicate cluster data to another cluster

Consulting
™
Kafka REST Proxy and
Kafka Schema Registry

Consulting
™
Kafka Ecosystem

Consulting
™
Kafka Stream Processing
❖ Kafka Streams for Stream Processing
❖ Kafka enable real-time processing of streams.
❖ Kafka Streams supports Stream Processor
❖ processing, transformation, aggregation, and produces 1 to * output streams
❖ Example: video player app sends events videos watched, videos paused
❖ output a new stream of user preferences
❖ can gear new video recommendations based on recent user activity
❖ can aggregate activity of many users to see what new videos are hot
❖ Solves hard problems: out of order records, aggregating/joining across streams, stateful
computations, and more

Consulting
™
Kafka Connectors and
Streams
Kafka
Cluster
App
App
App
App
App
App
DB DB
App App
Connectors
Producers
Consumers
Streams

Consulting
™
Kafka Ecosystem review
❖ What is Kafka Streams?
❖ What is Kafka Connect?
❖ What is the Schema Registry?
❖ What is Kafka Mirror Maker?
❖ When might you use Kafka REST Proxy?

Consulting
™
References
❖ Learning Apache Kafka, Second Edition 2nd Edition by Nishant Garg (Author),
2015, ISBN 978-1784393090, Packet Press
❖ Apache Kafka Cookbook, 1st Edition, Kindle Edition by Saurabh Minni (Author),
2015, ISBN 978-1785882449, Packet Press
❖ Kafka Streams for Stream processing: A few words about how Kafka works,
Serban Balamaci, 2017, Blog: Plain Ol' Java
❖ Kafka official documentation, 2017
❖ Why we need Kafka? Quora
❖ Why is Kafka Popular? Quora
❖ Why is Kafka so Fast? Stackoverflow
❖ Kafka growth exploding (Tech Republic)

Kafka Tutorial, Kafka ecosystem with clustering examples

More Related Content

What's hot (20)

Similar to Kafka Tutorial, Kafka ecosystem with clustering examples (20)

Recently uploaded (20)

Kafka Tutorial, Kafka ecosystem with clustering examples