DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine data

Editor's Notes

• #2: I’m Alan Gardner Here to talk about our use of Apache Kafka at Rocana
• #3: Platform engineer at Rocana Work on data ingest, storage and processing Distributed open-source systems: Hadoop, Kafka, Solr Systems programming work as well Work remotely from Ottawa, Canada This is my cat.
• #4: Working at Rocana is great: everybody is remote very smart, very nice people quarterly onsites
• #5: What is Rocana Ops? a platform for IT operations data designed for 10’s of thousands of servers in multiple data centers distill the entire organization’s IT infrastructure down to a single screen: “what’s wrong?” scalable collection framework - out of the box host data and app logs event data warehouse built on open source technologies and open schemas visualization, anomaly detection and machine learning to provide guided root cause analysis as opposed to a wall of graphs or pile of logs Apache Kafka is the “Enterprise Data Bus” Going to talk about why we chose Kafka in that role
• #6: To explain why we chose Kafka, I’m going to start with how Kafka works and why it’s designed the way it is.
• #7: Designed at LinkedIn to handle the explosion of different systems being created Jay Kreps blog post describes Kafka from first principles, including motivation Some of these images are cribbed from that post, where appropriate
• #8: LinkedIn’s problem: lots of front-end services lots of back-end services hooking them together produces this complex spaghetti of dependencies Front-end has to be highly available and low-latency if you write synchronously, you can only be as fast as your slowest backend service
• #10: Kafka acts as a central bus for data: every front-end service writes all events into Kafka backend services can take only the events they’re interested in data doesn’t live in Kafka forever Kafka is run as a utility within LinkedIn Solves one goal: centralized data bus, still need horizontal scale, durability
• #11: This is much better
• #13: Kafka is fundamentally a collection of logs events are only appended events are always consumed in the same order A single partition is a log: an ordered set of events Every event has an offset Partitions are the units of scale, like shards Log operations are constrained, so we can make them fast Example is sharding on users
• #14: Consuming and producing are completely decoupled: consumers maintain their own logical offset, representing the last event they consumed different consumers can consume at different rates producers continue to append new events in order events are retained until an expiry time, or max log size Kafka is not a durable long-term store Consumers can go offline for extended time or start from scratch and consume all available events Events are durably written and replicated
• #15: Kafka writes all data to disk, lots of good tricks: low-latency for recent data from the page cache data on the wire is the same as on disk no GC overhead for the page cache zero-copy ops
• #16: This is an overview of a typical Kafka system: multiple producers, brokers and consumers each broker has ownership of a set of partitions (it’s the primary) broker lists and partition assignment are stored in ZK consumers are using ZK to store offsets here, but that’s not the only way
• #18: Let’s revisit the Rocana architecture: thousands of agents writing into Kafka events are distributed across multiple partitions, written durably to disk multiple, separate consumers are decoupled from the producers and each other
• #19: Resource limits on producer machines: these machines are doing real work that’s important to the business our agent needs to quickly encode events and produce them batching is important to ensure efficiency latency to write to Kafka is still very low
• #20: Consumers don’t affect each other: each maintain their own offsets one consumer can be taken offline, can be slow, etc. with little impact upgrades are very easy a single consumer can even be rewound (theoretically) consumers can scale horizontally with the number of partitions
• #21: Kafka has critical mass within the industry: Cloudera, Hortonworks, MapR all support it Confluent has all the designers of Kafka working on a commercial stream processing platform
• #22: Those are all good things, but there are some sharp edges to watch out for.
• #23: Kingsbury tire fire slide Exactly once delivery is very hard Not all of our consumers are doing something idempotent You can play back the whole partition to find the last message which was written
• #24: Overview of a Rocana Event which would be published into Kafka: fixed fields and key-value pairs ID is a hash of an event fields, used for duplicate detection for durable sources we can use offset and inode, get 99% of the way for ephemeral sources we use arrival time + internal fields ID used for three things: assignment to a partition deduplication filter ID in Solr for idempotent inserts
• #25: Kafka “writes every message to disk” Defaults to fsyncing every 10k messages, or every 3 seconds (at most) ACK happens when a message is written but not fsynced OK, so I’ll replicate data across multiple machines
• #26: The default in Kafka is to continue making progress in the presence of node failures (AP): - unclean elections allow a replica which has not seen all writes to become the leader when the ISR shrinks to 0 - minimum ISR size is only 1 to accept writes by default - when a previously in-sync replica comes back, those records are lost - it can be disabled, see Jay’s blog for more discussion
• #27: Some things aren’t hard, but you need to look out for:
• #28: Data you put in Kafka really needs to have a schema Schemas really need to have an evolution strategy You probably want some notion of a schema registry Gwen’s post is great We use Avro, where the consumer has to know the writer schema tried to mitigate this with nullable fields, no luck
• #29: There isn’t any. No encryption on disk or in flight, no authentication: you can use stunnel you could encrypt each byte buffer and decrypt on the client side No authentication These will probably both be fixed in 0.9.0 this month
• #30: MirrorMaker is basically just a consumer/producer which pumps data between clusters: doesn’t preserve offsets, so consumers can’t fail over you can send events between two different sized clusters you can merge streams from two data centres
• #31: Kafka operations are pretty basic, it comes with a giant `bin` dir full of tools: CLI for rebalancing partitions and leaders leaders and partitions rebalance on node failure adding nodes requires reasignment Decomissioning nodes is a giant pain right now Tool for lagging consumer
• #32: Factors to consider when sizing a cluster: I/O throughput retention time frame (throughput over time)Partitions limit concurrency of consumers future growth (in terms of setting # of partitions) growing a cluster online is manual but possible in 0.8.2 growing number of partitions breaks consistent hashing! (http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/)
• #33: Jay Kreps has a blog post about this, using 3 commodity broker boxes with 6 cores, 6 spindles each: it’s a little weird, he only uses 6 partitions so he never exercises all the spindles in the cluster he batches small messages really aggressively (8k batches of 100 byte messages) his is on-premises, he hits 2.5M records/sec producing and consuming requiring 3 acks for every message halved throughput
• #34: I used a similar methodology on EC2 to get some sizing numbers: - Used 4k batch sizes, results were broadly similar (1k and 2k hurt perf) Over-provisioning partitions by 2x spindles doesn’t give benefit, but doesn’t slow down either Over-provisioning by 2x and adding 3x replication did cause slow down One partition actually hit 700k events/s, there may be coordination issues in the producer Synchronous acks were brutal, 10x performance hit, this is almost definitely due to AWS network latency Each node is ~$500/month At 250MB/sec, we’d only get ~18 hours of retention We’ve seen instances of only 12 hours of retention