Innovate Everywhere: Choosing the Right Tools When Building Your SRE Toolchain
0 likes315 views
The document discusses the integration of SRE (Site Reliability Engineering) tools within organizations, emphasizing their necessity for enhancing service reliability and availability. Key topics include the importance of observability, shared ownership of tools among developers, and strategies for managing system reliability through automation and metrics analysis. It also details specific tools like Grafana, InfluxDB, and VictorOps that aid in incident management and performance monitoring.
Innovate Everywhere: Choosing the Right Tools When Building Your SRE Toolchain
- 1. Innovate Everywhere Choosing the right tools for your SRE toolchain @jasonhand
- 2. Leonard Gram Core Developer at Grafana Labs @xlson Margo Schaedel Developer Advocate at InfluxData @mschae16 Jason Hand DevOps at VictorOps @jasonhand
- 3. Expect To Learn ● Industry expectations around service reliability and availability (New Challenges) ● How to create simple and lightweight representations of your systems for everyone in the organization (Observability) ● SRE Toolchain (e.g.) Grafana, InfluxData and VictorOps work together through the entire (Lifecycle of an Incident)
- 4. @jasonhand “Reliability is the single most important feature we provide.” - Dan Jones, CTO VictorOps
- 5. Reliability & Availability @jasonhand What Do We Mean?
- 7. 7© 2017 FORRESTER. REPRODUCTION PROHIBITED.
- 11. DevOps SRE vs
- 12. DevOps Reduce organization silos Accept failure as normal Implement gradual change Leverage tooling & automation Measure everything jhand.co/2rLjEV7
- 13. SRE Shared ownership with developers by using the same tools and techniques across the stack Encourage moving quickly by reducing costs of failure Have a formula for balancing accidents and failures against new releases Encourages "automating this year's job away" and minimizing manual systems work to focus on efforts that bring long-term value to the system Believes that operations is a software problem, and defines prescriptive ways for measuring availability, uptime, outages, toil, etc. jhand.co/2rLjEV7
- 14. Shared ownership with developers by using the same tools and techniques across the stack Encourage moving quickly by reducing costs of failure Have a formula for balancing accidents and failures against new releases Encourages "automating this year's job away" and minimizing manual systems work to focus on efforts that bring long-term value to the system Believes that operations is a software problem, and defines prescriptive ways for measuring availability, uptime, outages, toil, etc. Reduce organization silos Accept failure as normal Implement gradual change Leverage tooling & automation Measure everything DevOps SRE jhand.co/2rLjEV7
- 19. Our First SRE Question... What keeps you up at night?
- 20. What Does the Data Mean? Even though you are collecting the data .. what does it actually mean? Collecting → Storing → Thresholds/triggers. Defining what needs action taken … not always the same for everyone. @jasonhand (events, logging, tracing, etc.)
- 21. You Are Here Specialized operations tools convergence between SRE and software engineering toolchains The tools SREs use at any given time will depend on where an organization is in their SRE journey.
- 22. How Do We DevOps/SRE? @jasonhand It Depends
- 23. SRE Toolchain
- 24. A transition story How a development team gained ownership of their services and learned more about observability and reliability. New responsibilities: ● On-call ● Deployment ● Reliability @xlson
- 25. The story continues... Users contact support about the service being unreliable. What they have: ● Telegraf & InfluxDb (InfluxData) for metrics ● Grafana, basic dashboard for requests / minute ● Victorops for on-call & incident management @xlson
- 26. The Problem An increase in requests to their systems, … but ... no increase in logins or purchases
- 27. The RED Method Tracking already: ● (Request) Rate Added metrics: ● (Request) Errors ● (Request) Duration @xlson
- 28. Client Side Failures After some more time they find that requests to their systems are failing client side without being properly terminated server side, which leads to requests timing out and erroring. They set up alarms to guard against failures in the future. @xlson
- 29. Setting Up An SRE Toolchain
- 30. ✓ ✓ ✓
- 31. Platform for All Metrics and Event Workloads @influxdb
- 32. telegraf & influxdb Choosing the right tools for your SRE toolchain @influxdb
- 33. What is telegraf ● Plugin-driven server agent for collecting & reporting metrics ● Input plugins to source a variety of metrics ○ directly from the system it’s running on ○ pull metrics from third party APIs ○ listen for metrics via a statsd and Kafka consumer services ● Output plugins to send metrics to other datastores, services, & message queues ○ InfluxDB, Graphite, OpenTSDB, Datadog, Librato, Kafka, MQTT, NSQ, and many others
- 34. What is influxDB • Time-series database built from the ground up to handle high write & query loads • Data store for timestamped data, (DevOps monitoring, application metrics, IoT sensor data, & real-time analytics) • Conserve space by configuring – to keep data for a defined length of time – to automatically expiring data – delete any unwanted data from the system • Offers a SQL-like query language for interacting with data • Dashboarding in popular projects like Grafana
- 35. What is Grafana ● An open source platform for beautiful analytics and monitoring ● Easy to use ● Over 40 Datasources ○ InfluxDB ○ Prometheus ○ Graphite ○ Elasticsearch ○ ...and many more @xlson Try Grafana on https://play.grafana.org/
- 45. React Respond
- 49. Learn & Improve
- 56. Thank You Leonard Gram Core Developer at Grafana Labs @xlson Margo Schaedel Developer Advocate at InfluxData @mschae16 Jason Hand DevOps at VictorOps @jasonhand