ELK at LinkedIn - Kafka, scaling, lessons learned

Editor's Notes

• #3: 50+ % of site traffic come in via mobile
• #4: Many applications. Mobile frontend logs: average 2.4TB size (3+ billion docs).
• #5: Evaluated, used by some teams. Some Acquisitions use commercial solutions. Commercial solutions cost prohibitive.
• #8: Expect storage size to increase as we migrate to using doc_values
• #9: Leveraging open source stack. Large community. Leveraging common data transport. Rock solid, proven, dedicated support team.
• #10: Fast : a single Kafka broker can handle hundreds of megabytes of reads and writes per sec from thousands of clients. Scaleable: Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Durable: Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages w/o performance impact. Distributed by Design: Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
• #11: These numbers are from LinkedIn Kafka presentation at Apache Con 2015. Over 1100 brokers over 50+ clusters Over 32000 topics Over 350 thousand partitions, not including replication.
• #12: Log retention long enough for 3 days weekend. So we can re-index from Kafka if encounter issues. Dedicated log clusters to isolate traffic from other clusters. Two years ago, used our own kafka input plugin. Then switched to using KCC via pipe input for performance reason. Monitoring: . It’s important to monitor logstash nodes. LS has a bug where an error in any of its input, filter or output will stop the entire LS process. See metrics filter config file at end of these slides.
• #13: We’ve chosen to keep all of our clients local to the clusters and use a tiered architecture due to several major concerns. The primary concern is around the networking itself. Kafka enables multiple consumers to read the same topic, which means if we are reading remotely, we are copying messages over expensive inter-datacenter connections multiple times. We also have to handle problems like network partitioning in every client. Granted, you can have a partition even within a single datacenter, but it happens much more frequently when you are dealing with large distances. There’s also the concern of latency in connections – distance increases latency. Latency can cause interesting problems in client applications, and I like life to be boring. There are also security concerns around talking across datacenters. If we keep all of our clients local, we do not have to worry about ACL problems between the clients and the brokers (and Zookeeper as well). We can also deal with the problem of encrypting data in transit much more easily. This is one problem we have not worried about as much, but it is becoming a big concern now. The last concern is over resource usage. Everything at LinkedIn talks to Kafka, and a problem that takes out a production cluster is a major event. It could mean we have to shift traffic out of the datacenter until we resolve it, or it could result in inconsistent behavior in applications. Any application could overwhelm a cluster, but there are some, such as applications that run in Hadoop, that are more prone to this. By keeping those clients talking to a cluster that is separate from the front end, we mitigate resource contention.
• #14: For security reasons, data/logs generated in each DC stays there. Indexed by local ELK cluster. Aggregated views via Tribe nodes. All logstash use common filters to catch most common data leakage. How services log to Kafka - imposed common logging library. All services use common library, which automatically log to Kafka, WARN or above.
• #15: General architecture for each ELK cluster. Dedicated masters. Tribe client node (HTTP services).
• #16: Data - reliable transport, storing, queueing, consuming, indexing. Some data (java service logs for example) not in right format. Solutions to Data Kafka as transport, storage queue, backbone. More logstash instances, more Kafka partitions. Using KCC we can consume faster than ES can index. To increase indexing speed More ES nodes (horizontal). More shards (distribute work) Customized templates
• #17: Using local Kafka log clusters instead of aggregated metrics. Tribe to aggregate clusters. Use internal tool call Nurse to monitor and auto-remediate (restarts) hung/dead instances of LS and ES.
• #18: These numbers are estimate based on growth rate and plans. Beside logs, we have other application use cases internally.
• #20: This is how we use Logstash pipe input plugin to call out to Kafka Console Consumer. This currently give us the highest ingestion throughput.
• #21: It’s important to monitor logstash nodes. LS has a bug where an error in any of its input, filter or output will stop the entire LS process. You can use Logstash metrics filter to make sure that LS is still processing data. Sometime LS runs but no data goes through. This will let you know when that happens.