Beyond Nagios
1 like1,343 views
This document discusses moving beyond the limitations of Nagios for infrastructure monitoring. It summarizes that while Nagios is an industry standard, its configuration can be daunting and not user-friendly. It then describes common problems with Nagios like being overwhelmed by alerts, entering a "spiral of death" by adding more checks that lead to more alerts, and trying to improve coverage through more checks but resulting in the "trough of despair." The document recommends ways to improve the situation like measuring data, looking for patterns in alerts, putting alerts in context visually, and focusing on business impact.
Beyond Nagios
- 1. Beyond Nagios NYC DevOps 2011/07/21 Alexis Lê-Quôc - alq@datadoghq.com
- 2. Beyond Nagios NYC DevOps 2011/07/21 Alexis Lê-Quôc - alq@datadoghq.com
- 3. What I’m Going To Talk About • Super-quick Nagios summary • Monitoring/Alerting Pathologies • How to fix it
- 4. What Is • “Industry Standard in IT Infrastructure Monitoring” • For once it’s true... • Scheduler & Notification server
- 5. (+) Robust, Mature code-base (-) Configuration can be daunting (-) Not human-friendly
- 8. THE “OTHER” NAGIOS UI
- 9. Process alerts & Fix things Receive alerts Add more checks THE HAPPY START
- 10. Missed alerts Ignore Alerts Add more checks THE SPIRAL OF DEATH
- 11. Quality of life Few checks Few alerts More checks Too many alerts # of alerts FIGHT OR FLIGHT
- 12. Effective Checks n^2 Coverage Fault-tolerant Less urgency Few checks Few alerts Every host counts More checks Too many alerts Every host still counts Scale Complexity THE TROUGH OF DESPAIR
- 13. Effective Coverage Scale IF ONLY I ADDED MORE CHECKS...
- 14. Reset!
- 15. Way Out ‣Breathe! ‣Measure ‣Look for Patterns ‣Put Alerts in Context ‣Focus on the Business
- 16. Turn Nagios logs into structured data Analyze day | success_pct | warning_pct | error_pct | events ---------------------+-------------+-------------+-----------+-------- 2011-07-12 00:00:00 | 89 | 0| 2 | 9628 2011-07-13 00:00:00 | 90 | 0| 2 | 9210 2011-07-14 00:00:00 | 90 | 0| 2 | 9735 2011-07-15 00:00:00 | 89 | 0| 2 | 9531 MEASURE
- 17. day | success_pct | warning_pct | error_pct | events ---------------------+-------------+-------------+-----------+-------- 2011-07-12 00:00:00 | 89 | 0| 2 | 9628 2011-07-13 00:00:00 | 90 | 0| 2 | 9210 2011-07-14 00:00:00 | 90 | 0| 2 | 9735 2011-07-15 00:00:00 | 89 | 0| 2 | 9531 VISUALIZATION MATTERS
- 18. In Time Flapping LOOK FOR PATTERNS
- 19. PUT ALERTS IN CONTEXT https://app.datad0g.com/dash/dash/1000#/date_range/1310682467000.0-1310684267000.0
- 20. Ultimate (hard) question ‣Does this alert impact the business? ‣If so by how much? ‣Assumes that you track business metrics... ‣And they can be accessed programatically FOCUS ON THE BUSINESS
- 21. What applies to Nagios... Applies to other sources too etc...