OSMC 2015: Collectd Thresholds Plugin and Icinga by Florian Forster
0 likes166 views
The document discusses the features and capabilities of collectd, an open-source daemon for collecting and storing system metrics. It covers the use of thresholds and alerting mechanisms to notify users of metric status, including configuration examples for various plugins. Additionally, it highlights the importance of managing noisy and bursty metrics through hysteresis and failure notifications.
OSMC 2015: Collectd Thresholds Plugin and Icinga by Florian Forster
- 2. About me ● Florian "octo" Forster ● Open-source work since 2001 ● Started collectd in 2005
- 3. Agenda ● collectd ● Thresholds ● Alerting
- 4. Agenda ● collectd ● Thresholds ● Alerting
- 5. collectd ● Daemon ● collect metrics ● mangle / transport metrics ● store metrics (no retrieve)
- 6. collectd ● Open-source project ○ MIT and GPL licensed ● Platform independent ○ Linux, BSD, Solaris, AIX, HP-UX, … ○ Windows via SSC Serv (non-free)
- 7. collectd ● Agent based design ○ Runs on each host ● Extensible via plugins ○ Language bindings (Perl, Python, Java) ○ "exec" plugin, e.g. shell scripts
- 8. collectd ● 95+ "read" (input) plugins ○ System metrics (e.g. CPU, memory) ○ Application metrics (e.g. MySQL) ○ Other (Xeon Phi, SNMP, OneWire)
- 9. collectd ● 15+ "write" (output) plugins ○ Graphite ○ RRDtool ○ RRDCacheD ○ Riemann ○ MongoDB ○ HTTP (generic)
- 10. collectd # Input LoadPlugin cpu LoadPlugin memory LoadPlugin df <Plugin df> MountPoint "/" ValuesPercentage true </Plugin> # Output LoadPlugin network <Plugin network> Server "influxdb.example.com" </Plugin> Example configuration
- 11. collectd ● collectd's network plugin ○ Uses a binary protocol ○ UDP transport ○ 3rd party apps, e.g. InfluxDB, Prometheus → Grafana meets Monitoring (tomorrow 9:30, this room, German) → Prometheus (tomorrow 12:00, this room, English)
- 12. Agenda ● collectd ● Thresholds ● Alerting
- 13. Thresholds ● Implemented as a plugin ● Generates "notification" (aka. alert) ● Severities: FAILURE, WARNING, OKAY
- 15. Thresholds ● Load the Threshold plugin ● Configure thresholds ● Ensure "notifications" are handled
- 16. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 </Type> </Plugin> collectd.conf
- 17. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 </Type> </Plugin> collectd.conf
- 18. Thresholds ● Metrics must be selected by type ● They can be selected by host, plugin, plugin instance and type instance.
- 19. Thresholds <Plugin threshold> <Host "db.example.com"> <Plugin "cpu"> <Type "percent"> Instance "wait" WarningMax 40 </Type> </Plugin> </Host> </Plugin> collectd.conf
- 20. Thresholds ● WarningMin, FailureMin: Lower bound of acceptable values. ● WarningMax, FailureMax: Upper bound of acceptable values.
- 24. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 Persist true </Type> </Plugin> collectd.conf
- 26. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 Persist true PersistOK true </Type> </Plugin> collectd.conf
- 27. Warning Thresholds Warning! Warning! Warning! Warning! Warning! Warning! Warning! Warning! Ok Ok Ok
- 28. Warning Thresholds Warning! Never mind Help! It's happening again! Phew! Why does this keep happening?! Meh.
- 29. Thresholds ● Real metrics are noisy ● Murphy is not on our side ● First approach: Hysteresis
- 30. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 Hysteresis 5 </Type> </Plugin> collectd.conf
- 33. Thresholds ● Many metrics are bursty ● Especially latency ● Second approach: multiple failures
- 34. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 Hysteresis 5 Hits 6 </Type> </Plugin> collectd.conf
- 36. Thresholds ● "Interesting" metrics ● Create a FAILURE notification when metric disappears
- 37. Thresholds LoadPlugin threshold <Plugin threshold> <Type "temperature"> WarningMax 40 Interesting true </Type> </Plugin> collectd.conf
- 38. Agenda ● collectd ● Thresholds ● Alerting
- 39. Alerting ● "notifications" ● Severities: FAILURE, WARNING, OKAY
- 40. Alerting ● Support in some plugins: ● e.g. exec, network, logfile, syslog, write_riemann, write_sensu
- 41. Alerting LoadPlugin exec <Plugin exec> NotificationExec "user" "/usr/lib/collectd/notify.sh" </Plugin> Example: exec
- 42. Alerting /usr/lib/collectd/notify.sh <<EOF Severity: WARNING Time: 1447747649.961 Host: db.example.com Plugin: cpu Type: percent Data source "value" is currently 42. That is above the warning threshold of 40. EOF Example: exec notification format
- 43. Alerting ● Special notification plugins ● notify_desktop, notify_email
- 44. Alerting LoadPlugin "notify_email" <Plugin "notify_email"> From "collectd@example.com" Recipient "monitoring+collectd@example.com" SMTPServer "mail.example.com" SMTPUser "collectd" SMTPPassword "/!0sMcek3U" </Plugin> Example: notify_email
- 45. Alerting ● What's next? ● notify_nagios: https://collectd.org/bugs/1337 ● Writes service check results
- 46. Alerting LoadPlugin "notify_nagios" <Plugin "notify_nagios"> CommandFile "/usr/local/nagios/var/rw/nagios.cmd" </Plugin> Example: notify_nagios