OSDC 2018 | Hardware-level data-center monitoring with Prometheus by Conrad Hoffmann
0 likes348 views
This document discusses hardware-level data center monitoring using Prometheus. It outlines the speaker's data center which contains over 2,000 servers and 200 network devices. It then provides a brief introduction to Prometheus, highlighting its reliability, scalability, flexibility and ease of integration. Several Prometheus exporters are described that monitor nodes, network devices, and other systems, replacing tools like Nagios, Ganglia and Cacti. Methods for merging data from different sources are demonstrated. The transition to Prometheus monitoring is deemed successful due to the many available integrations and ease of developing new ones.
![Blackbox exporter - Smokeping replacement
2. Alert on target down and packet loss
ALERT SmokepingTargetDown
IF probe_success{job="smokeping"} == 0
FOR 2m
ALERT SmokepingTargetPacketLoss
IF 100*(1-avg_over_time(probe_success{job="smokeping"}[2m]))> 20](https://image.slidesharecdn.com/hardwareleveldatacentermonitoringwithprometheusconradhoffmann-180622115303/85/OSDC-2018-Hardware-level-data-center-monitoring-with-Prometheus-by-Conrad-Hoffmann-30-320.jpg)
OSDC 2018 | Hardware-level data-center monitoring with Prometheus by Conrad Hoffmann
- 1. Hardware-level data-center monitoring with Prometheus Conrad Hoffmann
- 2. Outline I. Our data-center II. Brief intro to Prometheus III. All my exporters IV. TL;DR & Soon™
- 3. Outline I. Our data-center II. Brief intro to Prometheus III. All my exporters IV. TL;DR & Soon™ AMS5
- 7. 2118 servers 56 racks 200 network devices
- 8. 2118 servers 56 racks 200 network devices 2 * 2 generic uplinks 3 AWS Direct Connect 3 Google X-Connect
- 9. Where we started... & NRPE Cloud Watch Cacti
- 10. What’s paging you at night? Collection Visualization Alerting Cacti ✔ ✔ ✔ CloudWatch ✔ ✔ ✔ Ganglia ✔ Graphite ✔ ✔ Icinga/Nagios ✔ ✔ ✔ Smokeping ✔ ✔ ✔ Statsd ✔
- 12. Outline I. Our data-center II. Brief intro to Prometheus III. All my exporters IV. TL;DR & Soon™ prometheus.io
- 13. The Promise of Prometheus Prometheus is a reliable, scalable, flexible monitoring and alerting system that is easy to integrate and focused on real time metrics.
- 14. Prometheus: reliability ● Pull-based (“scrape”) ● List of known targets ○ Can be dynamic, e.g. DNS or service discovery ● Built-in meta-monitoring ● Redundancy is easy
- 15. Prometheus: scalability ● Performant, efficient storage ● Scales well to available resources ● Easy to scale horizontally ● Federation
- 16. Prometheus: flexibility ● Multi-dimensional, label-based data model ● Each data point is defined by ○ A metric name ○ An arbitrary number of key-value pairs (labels) ○ A value ○ A timestamp (added by Prometheus) ● Data points with identical metric names and labels form a time series ● Powerful query language allows for easy aggregation based on labels
- 17. Prometheus: flexibility Target exposes: http_responses_total{backend="foo",code="2xx"} 804 http_responses_total{backend="foo",code="4xx"} 3170 http_responses_total{backend="bar",code="2xx"} 6637 http_responses_total{backend="bar",code="4xx"} 26 Possible query: sum(http_responses_total{backend="foo"})
- 18. Prometheus: ease of integration ● Data format is text based ● Scrapes are HTTP requests ● Many integrations exist already ● Excellent tooling/libraries to write new ones
- 19. Application Prometheus: ease of integration
- 20. Host node exporter Prometheus: ease of integration
- 21. Host SNMP exporter Router B Router A Prometheus: ease of integration Network
- 22. Host SNMP exporter Router B Router A Prometheus: ease of integration Network
- 23. Nomen est omen... ● Alerting ● Silencing ● Alert grouping & routing ● High availability Alertmanager
- 24. Displays data from many sources: ● Prometheus ● Graphite ● Influx ● OpenTSDB ● Elasticsearch ● MySQL/Postgres ● CloudWatch ● ... Grafana grafana.com
- 25. Outline I. Our data-center II. Brief intro to Prometheus III. All my exporters IV. TL;DR & Soon™ Now withProtips!
- 26. Node exporter ● Exports: OS- and hardware-level metrics for running systems ● Replaces: Ganglia, some Icinga/NRPE checks ● Noteworthy: ○ Comes with many collectors built-in ○ Use WMI exporter on Windows
- 27. Protip I Use the node exporter’s text file collector as an easy integration point for custom metrics! Examples: Chef data, RAID controller data, SMART data, cron jobs, ... node exporter script Text file Host
- 28. Blackbox exporter ● Exports: data about probes against endpoints that don’t support Prometheus natively (DNS, HTTP(S), ICMP, TCP) ● Replaces: Smokeping, some Icinga checks ● Noteworthy: ○ Monitor TLS certificate expiry :)
- 29. Blackbox exporter - Smokeping replacement 1. Send ICMP probe every five seconds
- 30. Blackbox exporter - Smokeping replacement 2. Alert on target down and packet loss ALERT SmokepingTargetDown IF probe_success{job="smokeping"} == 0 FOR 2m ALERT SmokepingTargetPacketLoss IF 100*(1-avg_over_time(probe_success{job="smokeping"}[2m]))> 20
- 31. Blackbox exporter - Smokeping replacement 3. Use Prometheus aggregation functions in Grafana
- 32. Blackbox exporter - Smokeping replacement
- 33. Protip II Scrape more, scrape faster! ● ~ 1M metrics ● > 5000 targets ● Mostly 10s scrape interval, some 5s, some longer ● 50 days retention time ● 250 GB storage ¯_(ツ)_/¯
- 34. SNMP exporter ● Exports: SNMP data from network devices ● Replaces: Cacti ● Noteworthy: ○ a pain to configure
- 35. SNMP exporter - Cacti replacement Once you have got the right SNMP config, alerts and nice graphs are easy!
- 36. SNMP exporter - Cacti replacement Cacti’s killer feature: the weathermap plugin! https://network-weathermap.com/
- 37. SNMP exporter - Cacti replacement There is a diagram panel type in Grafana, but… … we’re not quite there yet ¯_(ツ)_/¯
- 39. Protip III Build a dedicated long-term Prometheus server: ● Scrape only a few selected metrics ● Yank retention time way up ● Make backups (hot backups possible in Prometheus >2.1) Very useful data for estimating e.g. future bandwidth needs!
- 40. Collins exporter - Collins? ● https://tumblr.github.io/collins ● Infrastructure management / IPAM ● Server inventory, classification and lifecycle management
- 41. Collins exporter ● Exports: asset inventory data from Collins ● Replaces: a bunch of scripts ● Noteworthy: ○ https://github.com/soundcloud/collins_exporter
- 42. Collins exporter
- 43. Collins exporter ● Another candidate for long-term storage ● Valuable data for capacity planning
- 44. Protip IV Build your own integrations! Collins exporter: ● Written in Go ● 1 source file ● 264 lines total ¯_(ツ)_/¯
- 45. IPMI exporter ● Exports: IPMI data retrieved from BMCs ● Replaces: many Nagios/NRPE checks ● Noteworthy: ○ https://github.com/soundcloud/ipmi_exporter ○ Works regardless of hosts power state
- 46. IPMI exporter ● Mostly sensor data: temperature, fans, power consumption ● Mostly used for alerting: ○ Fans ○ Power supplies ○ Batteries
- 47. Protip V Make use of techniques to ingest non-numeric data!* ● Use labels to expose (semi-)static data of interest *...but do it with some caution! ipmi_bmc_info{firmware_revision="2.52",manufacturer_id="Dell_Inc"} 1
- 48. Protip V Make use of techniques to ingest non-numeric data!* ● Use labels and binary values to represent state *...but do it with some caution! collins_asset_state{tag="ABCD1234",state="Allocated"} collins_asset_state{tag="ABCD1234",state="Maintenance"} collins_asset_state{tag="ABCD1234",state="Unallocated"} 1 0 0
- 49. And now: merging data sources Example: BMC Firmware revisions of certain server types
- 50. And now: merging data sources Query: ipmi_bmc_info{firmware_revision!="2.52"} Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...}
- 51. And now: merging data sources Query: ipmi_bmc_info{firmware_revision!="2.52"} Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...} Query: collins_asset_details{nodeclass="app-2"} Result: collins_asset_details{ipmi_address="10.1.2.3",...}
- 52. And now: merging data sources Query: ipmi_bmc_info{firmware_revision!="2.52"} Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...} Query: collins_asset_details{nodeclass="app-2"} Result: collins_asset_details{ipmi_address="10.1.2.3",...} Query: label_replace(ipmi_bmc_info, "ipmi_address", "$1", "instance", "(.*)") Result: ipmi_bmc_info{firmware_revision="2.41",ipmi_address="10.1.2.3",...}
- 53. And now: merging data sources Query: collins_asset_details{nodeclass="app-2"} * on (ipmi_address) group_left(firmware_revision) label_replace(ipmi_bmc_info{firmware_revision!="2.52"}, "ipmi_address", "$1", "instance", "(.*)") Result: {firmware_revision="2.41",ipmi_address="10.1.2.3", nodeclass="app-2",primary_address="10.10.20.30",tag="ABCD1234"}
- 54. And now: merging data sources Query: collins_asset_details{nodeclass="app-2"} * on (ipmi_address) group_left(firmware_revision) label_replace(ipmi_bmc_info{firmware_revision!="2.52"}, "ipmi_address", "$1", "instance", "(.*)") * on (tag) group_left(status) (collins_asset_status == 1) Result: {firmware_revision="2.41",ipmi_address="10.1.2.3", nodeclass="app-2",primary_address="10.10.20.30",tag="ABCD1234",status="Allocated"}
- 55. Where we are now... & NRPE Cloud Watch Cacti ✘ ✘ ✘ ✘ ✘ ✘ ✘
- 56. Collection Visualization Alerting CloudWatch ✔ ✔ ✔ Graphite (✔) Prometheus ✔ Grafana ✔ Alertmanager ✔ What’s paging you at night?
- 57. What’s up with this CloudWatch thing? ● There is a CloudWatch exporter ● However, CloudWatch internal architecture is fundamentally incompatible with Prometheus ● Using CloudWatch as Grafana data source can incur costs
- 58. Outline I. Our data-center II. Brief intro to Prometheus III. All my exporters IV. TL;DR & Soon™
- 59. So, is it working? ● Yes
- 60. Was it worth it? ● Yes
- 61. Why was it worth it? ● Many integrations readily available ● New ones are easy to write ● Quality and quantity of monitoring has increased ● Monitoring and alerting has become much more consistent ● Easy to merge data sources for alerting or graphing This is true across the entire organization, not just infrastructure!
- 62. Soon: long term storage ● Not a primary concern for Prometheus ● Simple solution as explained ● Remote (read/)write interface ● Some features in Prometheus 2.0 to allow external solutions ○ Check out e.g. Thanos: https://github.com/improbable-eng/thanos
- 63. Soon: forging a standard? OpenMetrics working group ● https://github.com/RichiH/OpenMetrics
- 64. This is the end... Thank you!