Kafka 0.9, Things you should know

©2016 LinkedIn Corporation. All Rights Reserved.
Kafka 0.9, Things you should know

Ratish Ravindran
Site Reliability Engineer
LinkedIn, Data Infrastructure Streaming

©2016 LinkedIn Corporation. All Rights Reserved. 3
Agenda
 Security
 Kafka Connect
 User defined quota
 New consumer
 Notable improvements and fixes
 Upgrading from kafka 0.8
 Kafka 0.10 - highlights

Security
 Why ?
 Multitenant cluster
 Multiple clusters
 Network ACLs

Security
 Authentication
 Kerberos
 TLS
 Unix like permission
 "Principal P is [Allowed/Denied] Operation O From Host H On Resource R"
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --
allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --
allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
 Encryption

Kafka Connect
 Why ?
 Multiple tools for importing and exporting
 High engineering and operational overhead
 Some tools are poor fit for job

Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1T1
T2
T3 T6
T5
T4

Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1
KafkaConnect
KafkaConnect

Kafka Connect
Key Properties:
 Broad copying by default
 Streaming and batch
 Scales to application
 Focus on copying data only
 Parallel
 Connector API

Kafka Connect
Advantages :
 Fault tolerance
 Partitioning
 Offset management
 Delivery semantics
 Operations
 Monitoring

User defined quota
 Why ?
 High reads
 High writes
 SLAs

User defined quota
 Single large cluster
 Producer side (quota.producer.default)
 Consumer side (quota.consumer.default)
 Per client , Per broker
 Quota override
./bin/kafka-config.sh --alter
--add-config ‘producer_byte_rate=1048576,consumer_byte_rate=1048576’
--entity-type clients
--entity-name TestTopic
--zookeeper localhost:2181

New consumer
Motivation :
 Thin consumer client
 Central co-ordination
 Allow manual partition assignment
 Allow manual offset management
 Invocation of user specified callback on rebalance
 Non blocking consumer APIs

New consumer
Features:
 Group management protocol
 Consumer

New consumer
State Diagram - consumer

New consumer
Features:
 Consumer
 Co-ordinator

New consumer
State Diagram – Co-ordinator

New consumer
Features:
 Consumer
 Co-ordinator
 Failure detection protocol

New consumer
Interesting scenarios:
 Co-ordinator failover/connection loss
 Partition changes for subscribed topics
 Offset commit during rebalance
 Hearbeats during rebalance
 Slow consumers

Notable improvements and fixes
 Automated replica lag tuning (replica.lag.time.max.ms)
 New purgatory design – low memory overhead
 Auto-assign node ids
 No data loss in Mirror Maker – unclean shutdown
 Log compaction for compressed topics
 Handling of corrupt index files

Upgrading from kafka 0.8
 inter.broker.protocol.version=0.8.2.x
 Update code and restart
 inter.broker.protocol.version=0.9.0.0
 Restart brokers again

Potential Breaking Changes:
 Java 1.6 and Scala 2.9 are not supported
 Broker IDs > 1000 ( reserved.broker.max.id and
broker.id.generation.enable )
 replica.lag.max.messages removed
 replica.lag.time.max.ms
 No compaction for topics without key

Potential Breaking Changes contd….
 Changes in default JVM options

Why not kafka 0.10 ?

Kafka 0.10 - Highlights
 Kafka Streams
 Rack Awareness
 Timestamps in messages
 SASL improvements
 Kafka consumer max record
 Protocol version improvements

References
 http://kafka.apache.org/090/documentation.html
 http://www.confluent.io/blog/apache-kafka-0.9-is-released
 http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-
platform-3.0
 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+R
ewrite+Design

Kafka 0.9, Things you should know

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Kafka 0.9, Things you should know (20)

Recently uploaded (20)

Kafka 0.9, Things you should know