Wayfair Use Case: The four R's of Metrics Delivery
3 likes1,169 views
Wayfair uses a sophisticated metrics delivery system leveraging a next-generation data pipeline with Kafka and Telegraf to monitor over 2000 VMs across multiple data centers. The architecture focuses on four key principles: resiliency, redundancy, routability, and retention, ensuring efficient and scalable metrics processing. The system is designed to handle substantial data volumes and spikes while maintaining high availability and fast query responses for critical operations.
Wayfair Use Case: The four R's of Metrics Delivery
- 1. May 15, 2018 The Four R’s of Metrics Delivery Jim Hagan
- 2. 2 A Little Background • Wayfair currently uses both Graphite and InfluxDB as a time series platform • We send a very diverse set of event trackers, timers, and other system metrics from over 2000 VMs running hundreds of applications. • Our systems are spread across three major data centers. The data is used by our developers, business stakeholders, by our internal alerting engine. • Most importantly our 24x7 Ops Monitoring Center is using this data to constantly analyze the vital signs of Wayfair’s IT infrastructure and storefront operations.
- 3. 3 A Little Background Our legacy data pipeline was an elaborate series of services relaying data from data center to data center over UDP (used to avoid blocking calls from the client). This configuration works very fast (and even supported replication) but lacks a number of the elements we are looking for in our metrics pipeline. Our next generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement...
- 4. 4 A Little Background The Four R’s: • Resiliency • Redundancy • Routability • Retention
- 5. 5 The Four R’s How we want to think of our our data pipeline... Application(s) Time Series Database Sweet Insights!
- 6. 6 The Four R’s How it really is (current generation)...
- 7. 7 The Four R’s (previous generation)...
- 8. 8 The Four R’s What requirements are driving that architecture… • Receive data from anywhere in simple text format (dynamic schemas) with a low overhead network protocol. • Handle in the range of 100 million data points per second (in the aggregate) • Blacklist, whitelist and re-direct into more than one stream to achieve scaling and other business objectives • Handle spikes of 3x to 5x average peak traffic • Buffer that raw data for a reasonable amount of time (say 24 hours) • Ingest data when and where we please • Make 99% of data available for time series queries within 30 seconds.
- 9. 9 The Four R’s Breaking those requirements down conceptually we arrive at the “four R’s” • Routability (keep, drop, redirect) • Retention (keep data in pre-digested, or final format for some time) • Resilience (survive network or DC failure, recover data after a DB failure, survive massive flood of data) • Redundancy (replication of raw data for failover and purpose built dbs)
- 10. 10 The Four R’s Conceptual architecture to support the four R’s LOAD BALANCE RECEIVE/ HANDLE BUFFER INGEST PERSIST Resilience Routability Redundancy Retention Redundancy Resilience Routability Retention Resilience Resilience Resilience Routability Routability
- 11. 11 The Four R’s Conceptual architecture to support the four R’s LOAD BALANCE RECEIVE/ HANDLE BUFFER INGEST PERSIST NGINX (UDP LB) >> UDP Telegraf Receiver >> UDP Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP
- 12. 12 The Four R’s NGINX (UDP LB) Telegraf Receiver (>> UDP, << TCP) Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP We use the UDP load balancing plugin for Nginx. This allows us to take very high data rates including 3 to 5 x spikes we referred to earlier and efficiently route them to an array of telegraf hosts. 3 udp load balancers can feed into dozens of telegraf hosts. In addition we can use different port designations to route traffic, so for example, let’s say we took all UDP traffic into port 8094 and sent it to one array of receivers and all that into 8095 into another array of receivers. This is giving us RESILIENCY and a means of top level ROUTABILITY.
- 13. 13 The Four R’s NGINX (UDP LB) Telegraf Receiver (>> UDP, << TCP) Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP We use several features of telegraf to perform basic traffic shaping… We use TAG and measurement filters to 1. drop certain metrics 2. keep certain metrics 3. route certain metrics to a specific Kafka topic 4. route to specific Kafka brokers We are getting both RESILIENCE and ROUTING in this layer.
- 14. 14 The Four R’s NGINX (UDP LB) Telegraf Receiver (>> UDP, << TCP) Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP We use several features of telegraf to perform basic traffic shaping… We use TAG and measurement filters to 1. drop certain metrics 2. keep certain metrics 3. route certain metrics to a specific Kafka topic 4. route to specific Kafka brokers We are getting both RESILIENCE and ROUTING in this layer.
- 15. 15 The Four R’s NGINX (UDP LB) Telegraf Receiver (>> UDP, << TCP) Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP The local Kafka layer gives us an immediate place to store the metrics coming into the system. No expensive processing needs to be applied to the data yet. We configure several different “mirroring” services to copy different topics from local Kafka instances to what we call “Aggregate” kafka instances. All of this communication is happening with minimal transformation. We are getting both RESILIENCE and ROUTING and RETENTION in this layer.
- 16. 16 The Four R’s: Kafka Local/Aggregate Mirroring Topology C1 Kafka (write-only) C1 Kafka (write-only) C1 Kafka (write-only) C3 Kafka (read-only) C3 Kafka (read-only) C3 Kafka (read-only) Seattle Boston Europe The C1 Clusters are used for write-only in the local data centers. I addition each data center has an “aggregate” cluster (C3) for consumption of the integrated data stream. So if I’m in Boston our Telegraf ingest layer can consume data from all three data centers. LOCAL AGGREGATE
- 17. 17 The Four R’s NGINX (UDP LB) Telegraf Receiver (>> UDP, << TCP) Kafka Local >> TCP Telegraf Ingest >> TCP InfluxDB >>TCP Kafka Aggregate >> TCP We have a second layer of Telegraf acting as a Kafka consumer. This allows us to subscribe to topics that we care about and route them to the DB of our choice. We can also deploy multiple layers of ingest and populate multiple databases. We are getting both RESILIENCE and ROUTING and REDUNDANCY in this layer.
- 18. 18 The Four R’s Some architectural recipes (Dynamic Topic Routing) NGINX (UDP LB) Kafka Topic A (receives metric A) Telegraf Ingest Consume Topic A InfluxDB for Topic A Kafka Topic B (receives metric B) Telegraf Ingest Consume Topic B InfluxDB for Topic B Telegraf Receiver Keep Metric A and Metric B Discard Metric C
- 19. 19 The Four R’s Some architectural recipes (Redundancy) InfluxDB for Data Science (Longer Retention) InfluxDB for Alerts (Short Retention) Telegraf Ingest (Topic A) Telegraf Ingest (Topic A) Kafka
- 20. 20 Monitoring It All (Load Balancer Layer)
- 21. 21 Monitoring It All (Telegraf Layers)
- 22. 22 Monitoring It All (Telegraf Layers)
- 23. 23 Monitoring It All (Kafka)