SRE Demystified - 07 - Practical Alerting
0 likes117 views
The document discusses practical alerting and monitoring within large systems, particularly using Borgmon at Google. It emphasizes the challenges associated with monitoring numerous components while maintaining low maintenance burdens, and details the architecture and data management strategies employed by Borgmon for effective alerting. Key features include white-box monitoring, standardized data formats, time-series storage, and rule evaluation for generating alerts based on error metrics.
SRE Demystified - 07 - Practical Alerting
- 1. SRE Demystified Practical Alerting ganesh@ganeshniyer.com ganesh.vigneswara@gmail.com, http://ganeshniyer.com Dr Ganesh Neelakanta Iyer
- 3. Monitoring • Monitoring a very large system is challenging for a couple of reasons: • The sheer number of components being analyzed • The need to maintain a reasonably low maintenance burden on the engineers responsible for the system • A large system should be designed to aggregate signals and prune outliers • We need monitoring systems that allow us to alert for high- level service objectives, but retain the granularity to inspect individual components as needed 3 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 4. Borgmon monitoring at Google • White-box monitoring • Instead of executing custom scripts to detect system failures, Borgmon relies on a common data exposition format • This enables mass data collection with low overheads and avoids the costs of subprocess execution and network connection setup • The data is used both for rendering charts and creating alerts, which are accomplished using simple arithmetic • To facilitate mass collection, the metrics format had to be standardized 4 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 5. Instrumentation of applications • Adding mapped variables for example • An example map-valued variable • Showing 25 HTTP 200 responses and 12 HTTP 500s: • http_responses map:code 200:25 404:0 500:12 5 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 6. Storage in the Time-Series Arena • A service is typically made up of many binaries running as many tasks, on many machines, in many clusters • Borgmon needs to keep all that data organized, while allowing flexible querying and slicing of that data • Borgmon stores all the data in an in-memory database, regularly checkpointed to disk • The data points have the form (timestamp, value), and are stored in chronological lists called time-series, and each time- series is named by a unique set of labels, of the form name=value. 6 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 7. Storage in the Time-Series Arena 7 A time-series for errors labeled by the original host each was collected from https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 8. Labels and Vectors • Time-series are stored as sequences of numbers and timestamps, which are referred to as vectors • Like vectors in linear algebra, these vectors are slices and cross-sections of the multidimensional matrix of data points in the arena • The name of a time-series is a labelset, because it’s implemented as a set of labels expressed as key=value pairs. One of these labels is the variable name itself, the key that appears on the varz page 8https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 9. Labels and Vectors • Example variable expression {var=http_requests,job=webserver,instance=host0:80,service=web,zone=us-west} 9 Label Value var The name of the variable job The name given to the type of server being monitored service A loosely defined collection of jobs that provide a service to users, either internal or external zone Location of the Borgmon that performed the collection of this variable https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 10. Rule Evaluation • The Borgmon program code, also known as Borgmon rules, consists of simple algebraic expressions that compute time-series from other time-series • Rules run in a parallel threadpool where possible, but are dependent on ordering when using previously defined rules as input • Aggregation is the cornerstone of rule evaluation in a distributed environment 10 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 11. Example Rule 11 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 12. Example Alert Rule • Creates an alert when the error ratio over 10 minutes exceeds 1% and the total number of errors exceeds 1 per second 12 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 13. Maintaining the configuration • Borgmon configuration separates the definition of the rules from the targets being monitored • Borgmon also supports language templates • The first class simply codifies the emergent schema of variables exported from a given library of code • Such templates exist for the HTTP server library, memory allocation, the storage client library • The second class templates are to manage the aggregation of data from a single-server task to the global service footprint 13 https://landing.google.com/sre/sre-book/chapters/practical-alerting/
- 14. References 14
- 15. Dr Ganesh Neelakanta Iyer ganesh@ganeshniyer.com ganesh.vigneswara@gmail.com