Apache Kafka 下一代分布式消息系统
15 likes1,448 views
该文档概述了Apache Kafka的特性与架构，包括消息队列系统的解耦、冗余和可扩展性等核心概念。文中详细讨论了消息持久化、分布式负载均衡及与Zookeeper的集成，同时还比较了不同消息队列技术的优缺点。最后，文档探讨了Kafka的内部机制、领导者选举和复制设计等高级主题。
Apache Kafka 下一代分布式消息系统
- 3. 1
- 5. Top 10Uses For A Message Queue Decoupling 解耦 Redundancy 冗余 Scalability 扩展性 Elasticity & Spikability 灵活 ＆ 消除峰值 Resilliency 可恢复性 Understanding Data Flow 理解数据流 Asynchronous Communication 异步通信 Delivery Guarantees 送达保证 Ordering Guarantees 顺序保证 Buffering 缓冲
- 6. 消息语义 持久性 安全 淘汰 交付 路由 批量 消息过滤 排队标准 已收通知
- 7. 消息队列系统 / / TCP / UDP / HTTP / SOAP … 消息接收器 消息发送器 流入路由器 流出消息转换器 流出路由器 消息持久 消息状态 元数据 故障恢复 淘汰策略 容器组件 拦截器 流入消息转换器 拦截器服务组件调用
- 9. IO
- 10. 硬盘 VS. 内存 随机读 随机写 硬盘 VS. 顺序读 顺序写
- 11. 机械磁盘 慢 现代操作系统优化 1. 使用 read-ahead 和 write-behind 技术，预读取成块数据，将微小琐 碎的逻辑写入组织成一次较大的物理写入 2. 常用空闲内存用作磁盘缓存 3. 线性的访问磁盘，很多时候比随机的内存访问快得多
- 12. JVM 2个事实 1. Java对象占用空间非常大，差不多要存储的数据的两倍甚至更高 2. 随着堆中数据量的增加，GC（垃圾回收）变的越来越困难
- 13. JVM 1个假设 在64G内存的机器上，不得不使用到50G～56G的内存空间 当系统重启的时候，又必须要将数据刷到内存中（ 每分钟1GB 内存），即使冷刷 新（在使用数据的时候发现没有再刷到内存）也会导致最初的时候性能非常慢
- 15. 图解 零拷贝 Application Buffer Read buffer Socket buffer NIC buffer Application Context Kernel Context Figure1.Traditional data copying approach DMA copy CPU copy Figure2.Traditional Context Switches context switching before read before write Syscall read Syscall write U U U K K U User context Kernel context next cycle K
- 16. 图解 零拷贝 Read buffer Socket buffer NIC buffer Figure3.Data copy with transferTo() Appliacation Context Kernel Context TransferTo() before transfer to() Syscall read and send U K context switching Next cycle U Figure4.Context switching with transferTo()
- 17. 图解 零拷贝 Figure5.Data copies when transferTo(and) gather operations are used Application Context Kernel Context TransferTo() Read buffer Descriptor NIC buffer
- 18. 设计 常量复杂度 的磁盘操作 B树的复杂度是O(logN)，通常被认为就是常量复杂度 1. 线性的访问减小磁盘寻道 2. 压缩数据以减少IO压力 3. 使用零拷贝（zero copy）技术 但对于磁盘操作来说并非如此：磁盘进行一次搜索需要10ms，每个磁盘在同一时间只能进行一次搜索，并发处理困难 对树结构的性能的观察结果表明：其性能往往随着数据的增长而线性下降，数据增长一倍，速度就会降低一倍
- 19. Kafka2
- 24. 支持在线平滑水平扩展 水平扩展
- 25. Kafka名词解释 3. Replication & Replication Leader & Replication Follower 2. Topic & Partition & Segment & Offset 1. Broker & Controller & Producer & Consumer & Consumer Group 4. Assigned Replications & Preferred Replication 5. Message & Message Set
- 26. Producer1 Producer2 broker1 broker2 broker3 broker4 Consumer1 Consumer2 Consumer3 zookeeper Topic1 Topic2 Consumer Group 网络拓扑
- 27. Partition0 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition1 0 1 2 3 4 5 6 7 8 9 Partition2 0 1 2 3 4 5 6 7 8 9 10 11 12 Anatomy of a Topic Old New Write(Append) Topic消息流 ProducerA ConsumerA ConsumerB ConsumerC Topic ProducerB Partition1 0 1 Partition2 0 1 2 Partition3 0 1
- 28. Segment File Implementation Active Segment List 34477849968 - 35551592051 topic/34477849968.index 35551592052 - 36625333894 topic/35551592052.index 36625333895 - 37699075830 topic/36625333895.index 37699075831 - 38772817944 topic/37699075831.index … … … … … … 79575006771 - 80648748860 topic/79575006771.index 80648748861 - 81722490796 topic/80648748861.index 81722490797 - 82796232651 topic/81722490797.index 82796232652 - 83869974631 topic/82796232652.index topic/34477849968.kafka Message 34477849968 Message 34477850175 … … Message 35551591806 Message 35551592051 ConsistentViews topic/82796232652.kafka Message 82796232652 Message 82796232859 … … Message 83869974383 Message 83869974631 Deletes Appends Reads … …34477849968.index 34477849968 34477850630 34477852008 … 35551590905 35551592051 34477849968.kafka Message 34477849968 Message 34477850175 Message 34477850630 Message 34477851038 Message 34477851603 Message 34477852008 … Message 35551590905 Message 35551591309 Message 35551591806 Message 35551592051
- 29. broker1 broker2 broker3 broker4 Broker & Topic Partition topic1-part1 topic2-part1 topic2-part2 topic1-part2 topic1-part1 topic2-part1 topic2-part2 topic1-part2 topic1-part1 topic2-part1 topic2-part2 topic1-part2
- 30. Message Deliver payloadcrcmagicsize payloadcrcmagicsize … payloadcrcmagicsize Message Message Set Producer Consumer offset Consumer Buffer ByteMessageSet FileMessageSet
- 32. In-Sync Semantics 1. 每一个Broker节点必须维护和Zookeeper的连接Session，Zookeeper通过心跳机 制检查每个结点的连接 2. Follower Broker节点必须及时同步Leader Broker节点，不能落后Leader Broker 节点太多
- 33. Replication & Commit-Log 1. 当且仅当Message被所有的Replication写入到Log中，才算”Committed” 2. 只有Committed的Message，才会被Consumer读取
- 34. Persistence & Efficiency 1. 每一个Follower都只从Leader Pull数据 2. 每一个Follower收到数据后，立即向Leader发送ACK，而非等到数据写入Log后
- 35. Consumer & Partition 1. 同一Consumer Group中Consumer竞争Partition，即队列语义 2. 不同Consumer Group中Consumer共享Partition，即主题语义
- 36. Message Delivery Semantics 2. At least once - 消息绝不会丢，但可能会重复传输 3. Exactly once - 每一条消息肯定会被传输一次且仅传输一次，理想状态 1. At most once - 消息可能会丢，但绝不会重复传输
- 37. Leader Election算法 1. Leader Election 2. ISR Approach VS. Majority Vote 注： ISR，In-Sync Replicas简称 3. 某一个Partition所有Replication不工作
- 38. Broker/Controller思考 1. 选举Broker Leader最简单最直观的方案 2. 该选举Broker Leader的方案引入了哪些问题
- 39. Partition思考 1. Partition的数据结构，逻辑上/物理上的存储结构 2. Broker&Topic&Partition关系 3. Partition＆Consumer&Consumer Group关系 4. Partition Rebalance触发的条件及引发的问题
- 40. Replication Design –High Level 1. How to assign replicas of a partition to broker servers evenly? 2. For a given partition, how to propagate every message to all replicas?
- 41. Producer思考 1. Load balancing 2. Asynchronous send
- 42. Consumer思考 1. Push VS. Pull 2. Offset的归属问题及其可靠性的存储方案 3. 如何帮助减轻Broker设计的复杂度
- 43. Kafka3
- 44. Request Channel Log Manager Broker Internals Network Layer Acceptor Thread Processor Thread1 Processor Thread2 Processor Thread3 Processor ThreadN ClientConnections API Layer API Thread1 API Thread2 API Thread3 API ThreadN Request Purgatory Log Subsystem FileSystem Partition Log1 Partition Log2 Partition Log3 Partition LogN Replication Subsystem Replication Manager Replication Controller Replication Thread1 Replication Thread2 Replication Thread3 Replication ThreadN Replication Info Kafka Brokers ZookeeperZookeeper Kafka Log Replication Leadership Changes 1. 利用mmap和log index机制， broker定时或定量持久化消息 2. 网络架构优化
- 45. BrokerFailover Controller Broker State Affected BrokersBroker Path watch fired read available brokers determine set_p read ISR for partition in set_p determine new leader and ISR for partition in set_p write new ISR and Leader RPC: ISR/Leader assignment
- 46. 创建/删除Topic Controller Broker Path Affected BrokersTopic Path watch fired read created/deleted topics determine set_p read available broker list set AR as ISR, one replication in AR as Leader write new ISR and Leader RPC: ISR/Leader assignment LeaderAndISRRequest LeaderAndISRResponse Broker State
- 47. LeaderAndIsrRequest响应 highwatermark controller Epoch old? LeaderAndLsr Request leader_epoch old? partitionStateInfo 包含当前broker id Send Response N N N StaleControllerEpochCode Response Y StaleLeaderEpochCode Response partitionState log.warn Y Y 当前broker是否为 该partition leader makeFollowers makeLeaders IdleFetcher N Y
- 48. Replication工具 3. Preferred Replica Leader Election Tool 2. Replica Verification Tool 1. Topic Tool 4. Kafka Reassign Partitions Tool 5. State Change Log Merge Tool