Kafka practical experience
1 like723 views
This document provides an overview of Kafka including its architecture, key concepts, and performance tuning. It describes how Kafka is a distributed streaming platform popular for use cases like logging, metrics, and messaging. The architecture explained includes Kafka brokers that make up clusters, Zookeeper for coordination, producers that publish messages, consumers that subscribe to messages, and topics for categorizing data. It also covers message delivery guarantees, monitoring tools, and ways to optimize producer, broker, consumer, and JVM performance such as configuration settings for throughput, latency, and durability.
Kafka practical experience
- 2. Agenda ◎Kafka Overview ◎AFT Kafka Architecture .Net Client -Pub/Sub Key Terms Introduction Message Delivery ◎Monitor Kafka Architecture ◎Kafka Performance Tuning Producer Broker Consumer JVM
- 3. Kafka Overview • High-performance distributed streaming platform • Popular project on githup: star=6389 , fork=3909 • Simple installation、Easy scale-out、More resource(Linux) • High availability、High failover、High reliability、Auto balance、Message persistence.. • Use Cases:Log Aggregation、Metrics、Event Sourcing、 Messaging… • Power by: LinkedIn、airbnb、Mozilla、Twitter、LINE、 skyscanner、trivago、Hotel.com、PayPal、Uber、 Yahoo…
- 4. AFT Kafka Architecture Kafka cluster(P2P) broker1 broker2 broker3 zookeeper cluster(Master-Salve) node1 node2 node3 UGS API UGS WEB UGS WinSvr Producers Consumers Serialization publish message(Batch、 Fire and forget) Middleware Consumer group Logger Services Logger Services Logger Services deserialization subscribe message (message set、 async) Topic’s configuration、broker status、 Cluster membership、controller process、 Coordinator process Message(binary) queue、 partition、offset manager、 Leader cache、topic、 ReplicaManager、 GroupCoordinator(rebalance) Replicat e Replicat e TCP TCP Heartbeat
- 6. Key Terms Introduction • Broker: MQ process(Minimum unit in kafka cluster) • Topic: Category of message(data is store in) • Producer: push message to broker(write data) • Consumer: pull message from broker(read data) • ConsumerGroup: provide tolerance、 scalability、parallel for Consumer • Partition: provide tolerance、 scalability、parallel • Offset: Message position on each partition
- 7. Message Delivery At most once: Messages may be lost but are never redelivered At least once: Messages are never lost but may be redelivered Exactly once: Each message is delivered once and only once(0.11.x) Messages sent by a producer to a particular topic partition will be appended in the order they are sent A consumer instance sees records in the order they are stored in the log. Tolerate up to N-1 server failures.(depends replication factors)
- 8. Monitor Kafka Architecture Telegraf http (every 10 sec) Influxdb 2.Result Grafana JMX 1.QL http (every 10 sec) Kafka- Manager Kafka Eagle Mysql 2.Result TCP 1.Collec t 2.Store TCP
- 9. High level architecture blueprint UGS Platform /Producer Logger Services /Consumer Channel SQL Server Kafka eagle /Consumer
- 10. Kafka Performance Tuning • Producer • Broker • Consumer • JVM
- 11. Producer • Load balancing(sends data directly to the broker that is the leader for the partition) • Acks=0 producer no wait any acknowledgment from the broker at all. Lowest latency at the cost of durability but high data lost. • Acks=1 producer gets an acknowledgment after the leader wrote the record to its local log, but will respond without awaiting full acknowledgement from all followers. Maybe follower will be lost data if leader commit after. • Acks=-1 producer gets all acknowledgment after all in-sync replicas has received the data. Strong guarantee data not be lost. • batch.size=100 ,net client • send.buffer.bytes=100*1024 • producer.type=async • compression.type=none • max.in.flight.requests.per.connection=3 Note: min.insync.replicas>=2 ACKs Throughput Latency Durability 0 High Low No Gurantee 1 Medium Medium Leader -1 Low High ISR
- 12. Broker • More partition = more concurrent process = more memory = more io access =increase throughput= increase latency (brokers have to distribution on each partition) P.S single topic less than 1024 partitions • Number of Factors = two brokers at least num.io.threads=8 num.network.threads=3 background.threads=10 queued.max.requests=500 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 num.recovery.threads.per.data.dir=2 Log.retention.hours=24 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.cleanup.policy=delete log.cleaner.enable=true log.cleaner.threads=1 log.cleaner.backoff.ms=30000 log.segment.bytes=1073741824 replica.fetch.min.bytes=1 replica.high.watermark.checkpoint.interval.ms=5000 replica.fetch.wait.max.ms=500 min.insync.replicas=2
- 13. Consumer • Need enough partitions to handle message from producer • Maximum number of consumer = a multiple of broker(balance is better) • max.poll.records=5000 • enable.auto.commit=true • auto.commit.interval.ms=5000 • fetch.max.wait.ms=500 • fetch.min.bytes=1 • keep small Batch size in our .net client(for real time consumer data)
- 14. JVM • Avoid out of memory • Avoid high frequency trigger GC -Xmx8g –Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 - XX:MaxMetaspaceFreeRatio=80 -XX:MinMetaspaceFreeRatio=50 - XX:G1HeapRegionSize=16M -XX:InitiatingHeapOccupancyPercent=35 -Xms: Set initial Java heap size -Xmx: Set maximum Java heap size +UseG1GC: Enable G1 GC MaxGCPauseMillis: Set maximum pause MaxMetaspaceFreeRatio: Set maximun metaspace free ratio MinMetaspaceFreeRatio: Set minimun metaspace free ratio G1HeapRegionSize: Adjust G1 region on each heap InitiatingHeapOccupancyPercent: initial Java heap occupancy threshold
- 15. Q & A
- 16. Reference • https://kafka.apache.org/ • https://github.com/apache/kafka • http://www.oracle.com/technetwork/articles/java/g1gc- 1984535.html • https://docs.oracle.com/cd/E40972_01/doc.70/e40973/cn f_jvmgc.htm • RiCo’s blog