Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for Better Log Management
6 likes3,486 views
Loggly utilizes Apache Kafka for efficient log management, achieving high fault tolerance and real-time data processing on a massive scale with over 750 billion events logged. The technology enables log collection during network outages and provides flexibility through dynamically created queues, optimizing resource utilization and managing diverse customer logging behaviors. Ultimately, Kafka's deployment allows Loggly to focus on enhancing their service as the leading cloud-based log management solution.
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for Better Log Management
- 1. Why Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for Better Apache Log Storm Management Infrastructure Engineering Team June 2014 | Log management as a service Simplify Log Management
- 2. What Loggly Does World’s most popular cloud-based log management service § More than 5,000 customers § Near real-time indexing of events Distributed architecture, built on AWS Initial production services in 2011 § Loggly Generation 2 released in Sept 2013 | Log management as a service Simplify Log Management
- 3. Loggly: Addressing the first big data problem every company faces § Centralized logging and archival § Real-time processing, analysis and visualization § Monitoring, alerting and troubleshooting | Log management as a service Simplify Log Management
- 4. Agenda for this Presentation § The challenges of Log Management at scale § Overview of Loggly’s processing pipeline § Alternative technologies considered § Why we love Apache Kafka § How Kafka has added flexibility to our pipeline | Log management as a service Simplify Log Management
- 5. The Challenges of Log Management at Scale § Big data – >750 billion events logged to date – Sustained bursts of 100,000+ events per second – Data space measured in petabytes § Need for high fault tolerance § Near real-time indexing requirements § Time-series index management | Log management as a service Simplify Log Management
- 6. Log Management Processing Pipeline: Overview Load Balancing Kafka Stage 2 Loggly Custom Module | Log management as a service Simplify Log Management
- 7. Collectors Can Easily Outpace Downstream Processes Load Balancing Kafka Stage 2 Loggly Custom Module § Written in C++ § Designed to ingest massive data volumes § Need to collect regardless of what’s happening downstream | Log management as a service Simplify Log Management
- 8. Solution: Queue That’s External to Collector Load Balancing Kafka Stage 2 Loggly Custom Module § Based on Apache Kafka § Highly performant and reliable | Log management as a service Simplify Log Management
- 9. Alternate/ Supplementary Approaches Considered § Internal buffering in collectors – Added complexity § Cassandra – Not as good a queue as Kafka § Apache Storm – In initial Gen2 architecture, removed after launch | Log management as a service Simplify Log Management
- 10. The Secret to Log Management at Scale: Keep It Simple, Stupid Results: § Can process sustained rates of 100,000+ events per second per cluster § Average message 300 bytes | Log management as a service Simplify Log Management
- 11. Why We Love Kafka | Log management as a service Simplify Log Management
- 12. What Attracted Us in the First Place No single point of failure • Terabytes of data move through our Kafka cluster every day without losing a single event • We use age-based retention to purge old data on disks Low latency • 99.99999% of the time our data is coming from disk cache and RAM; only very rarely do we hit disk Performance • Crazy good! • We currently have a bunch of Kafka brokers running on m2.xlarge instances backed by provisioned IOPS. • One of consumer group (eight threads) which maps a log to a customer can process about 200,000 events per second draining from 192 partitions spread across three brokers Scalability • Ability to increase partition count per topic and downstream consumer threads provides flexibility to increase throughput when desired | Log management as a service Simplify Log Management
- 13. How Our Kafka Crush Has Deepened Distributed log collection • Local pods and collectors spread all over the Internet with local Kafka deployments to collect data from customers located all over world • Can collect logs even when we lose connectivity • When network comes back, Kafka sends the logs downstream to the rest of the pipeline More efficient, effective DevOps • Deploying Kafka throughout pipeline makes it easy to disable certain parts of system (for troubleshooting or upgrades) • No worrying that we will lose customer data • Example: Add support for new log type into our automatic parsing capabilities by turning off existing parser, deploying new one, and processing logs that Kafka has queued up Controlling resource utilization • Keep collectors as simple as possible for resilience and reliability reasons • Add intelligence into our pipelines using Kafka | Log management as a service Simplify Log Management
- 14. Resource Utilization Example: “Noisy Neighbors” | Log management as a service Simplify Log Management
- 15. “Noisy Neighbors” are Inherent to SaaS § Sending many times their “normal” level of logging volume, inadvertently or because their application is in big trouble § Routing logs to separate queue minimizes impact on other customers | Log management as a service Simplify Log Management
- 16. Kafka Queues Add Flexibility to Loggly Pipeline § Because Kafka topics are very cheap from a performance and overhead standpoint, we can create as many queues as we want § Scaled to the performance we want § Optimizing resource utilization across the system § Because they can be created dynamically, we can make business rules very flexible § Makes us confident that pipeline will scale as customer data volumes do | Log management as a service Simplify Log Management
- 17. Conclusion: Kafka Frees Our Development Team to Build Differentiating Features § Kafka deployment working without us thinking about it § Plenty of other things to do to keep our position as the world’s most popular cloud-based log management service! | Log management as a service Simplify Log Management
- 18. Does Log Management Sound Hard? It Should! Let us do the heavy lifting for you! Try Loggly FREE for 30 days About Us: Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success. Visit us at loggly.com or follow @loggly on Twitter. | Log management as a service Simplify Log Management
- 19. Did you like this presentation? Head over to our blog for more great content! Take me to the Loggly Blog | Log management as a service Simplify Log Management