Monitoring the Hashistack with Prometheus
0 likes1,451 views
The document discusses monitoring the HashiStack using Prometheus, an open-source monitoring and alerting system originally built by SoundCloud. It explains Prometheus’s data model, querying capabilities with PromQL, and the integration of HashiCorp products like Consul, Vault, and Terraform for metrics. The document outlines how to expose metrics, use exporters, and leverage mixins for alerts and dashboards in Grafana.
![Prometheus’ data model is very simple:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Timestamps are millisecond int64, values are float64
https://www.slideshare.net/Docker/monitoring-the-prometheus-way-julius-voltz-prometheus](https://image.slidesharecdn.com/monitoringthehashistackwithprometheus-180815100418/85/Monitoring-the-Hashistack-with-Prometheus-5-320.jpg)
![Can select vectors of values…
PromQL: http_requests_total{job=“nginx”, status=~“502”}[1m]
{job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} [30, 31, 32, 34]
{job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“500”} [4, 24, 56, 56]
{job=“nginx”, instances=“2.3.4.5:80”, path=“/home”, status=“500”} [76, 76, 76, 76]
{job=“nginx”, instances=“2.3.4.5:80”, path=“/setting”, status=“500”} [56, 106, 5, 96]
...](https://image.slidesharecdn.com/monitoringthehashistackwithprometheus-180815100418/85/Monitoring-the-Hashistack-with-Prometheus-8-320.jpg)
![And apply functions…
PromQL: rate(http_requests_total{job=“nginx”, status=~“502”}[1m])
{job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} 0.0666
{job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“500”} 0.866
{job=“nginx”, instances=“2.3.4.5:80”, path=“/home”, status=“500”} 0.0
{job=“nginx”, instances=“2.3.4.5:80”, path=“/settings”, status=“500”} 2.43
...](https://image.slidesharecdn.com/monitoringthehashistackwithprometheus-180815100418/85/Monitoring-the-Hashistack-with-Prometheus-9-320.jpg)
![And aggregate by a dimension…
PromQL: sum by (path) (rate(http_requests_total{job=“nginx”, status=~“502”}[1m]))
{path=“/home”} 0.0666
{path=“/settings”} 3.3
...](https://image.slidesharecdn.com/monitoringthehashistackwithprometheus-180815100418/85/Monitoring-the-Hashistack-with-Prometheus-10-320.jpg)
![Do binary operations…
PromQL: sum by (path) (rate(http_requests_total{job=“nginx”, status=~“502”}[1m]))
/
sum by (path) (rate(http_requests_total{job=“nginx”}[1m]))
{path=“/home”} 0.001
{path=“/settings”} 1.0
...](https://image.slidesharecdn.com/monitoringthehashistackwithprometheus-180815100418/85/Monitoring-the-Hashistack-with-Prometheus-11-320.jpg)
Monitoring the Hashistack with Prometheus
- 1. Monitoring the Hashistack with Prometheus Tom Wilkie @tom_wilkie August 2018 ❤
- 2. Prometheus ● A monitoring & alerting system. ● Inspired by Google’s BorgMon ● Originally built by SoundCloud in 2012 ● Open Source, now part of the CNCF ● Simple text-based metrics format ● Multidimensional datamodel ● Rich, concise query language
- 5. Prometheus’ data model is very simple: <identifier> → [ (t0, v0), (t1, v1), ... ] Timestamps are millisecond int64, values are float64 https://www.slideshare.net/Docker/monitoring-the-prometheus-way-julius-voltz-prometheus
- 6. Prometheus identifiers http_requests_total{job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“200”} http_requests_total{job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} http_requests_total{job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“200”} http_requests_total{job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“502”} Prometheus series selector http_requests_total{job=“nginx”, status=~“5..”}
- 7. Building queries usually starts with a selector PromQL: http_requests_total{job=“nginx”, status=~“5..”} {job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} 34 {job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“502”} 56 {job=“nginx”, instances=“2.3.4.5:80”, path=“/home”, status=“500”} 76 {job=“nginx”, instances=“2.3.4.5:80”, path=“/setting”, status=“502”} 96 ...
- 8. Can select vectors of values… PromQL: http_requests_total{job=“nginx”, status=~“502”}[1m] {job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} [30, 31, 32, 34] {job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“500”} [4, 24, 56, 56] {job=“nginx”, instances=“2.3.4.5:80”, path=“/home”, status=“500”} [76, 76, 76, 76] {job=“nginx”, instances=“2.3.4.5:80”, path=“/setting”, status=“500”} [56, 106, 5, 96] ...
- 9. And apply functions… PromQL: rate(http_requests_total{job=“nginx”, status=~“502”}[1m]) {job=“nginx”, instances=“1.2.3.4:80”, path=“/home”, status=“500”} 0.0666 {job=“nginx”, instances=“1.2.3.4:80”, path=“/settings”, status=“500”} 0.866 {job=“nginx”, instances=“2.3.4.5:80”, path=“/home”, status=“500”} 0.0 {job=“nginx”, instances=“2.3.4.5:80”, path=“/settings”, status=“500”} 2.43 ...
- 10. And aggregate by a dimension… PromQL: sum by (path) (rate(http_requests_total{job=“nginx”, status=~“502”}[1m])) {path=“/home”} 0.0666 {path=“/settings”} 3.3 ...
- 11. Do binary operations… PromQL: sum by (path) (rate(http_requests_total{job=“nginx”, status=~“502”}[1m])) / sum by (path) (rate(http_requests_total{job=“nginx”}[1m])) {path=“/home”} 0.001 {path=“/settings”} 1.0 ...
- 12. Hashistack ● Consul: service discovery, K/V store, service mesh… ● Vault: secret management and automation ● Terraform: infrastructure config as code
- 13. • All Hashicorp products use github.com/armon/go-metrics for metrics. • Exposes metrics in statsd, dogstatsd • Prometheus support added in 2015 • …but not plumbed through in most products (until recently)
- 14. Consul ● Prometheus support exposed in 1.1.0 (hashicorp/consul#4014, hashicorp/consul#4016) ● Exposed metrics are being improved (hashicorp/consul#4042)
- 15. Consul (II) ● Alternatively use the statsd_exporter with customer metrics mapping ● github.com/prometheus/ statsd_exporter
- 17. Consul (III) ● Still don’t get the “operational” metrics we want. ● Use the consul_exporter: github.com/prometheus/ consul_exporter
- 18. Consul Mixin ● Set of predefined alerts & dashboards for Prometheus / Grafana ● github.com/kausalco/ public/consul-mixin ● Users “Prometheus Mixin” format: ● Design Doc
- 19. Vault ● No Prometheus metrics yet (hashicorp/vault#2937) ● Again, use statsd_exporter ● BUT statsd metrics aren’t very “operational” ● Use vault_exporter ● github.com/grapeshot/ vault_exporter
- 20. Vault Mixin ● As per consul mixin, set of alerts and dashboard for Prometheus & Grafana ● github.com/grapeshot/ vault_exporter/vault-mixin
- 21. Terraform ● Terraform is a CLI tool - what does it mean to monitor that? ● I don’t have enough confidence to run it CI/CD.. ● But I do want to know if someone changes something and doesn’t apply it.
- 22. Terradiff • Runs on k8s • Pod containing: • git-sync (github.com/kubernetes/git-sync) • prom-run (github.com/tomwilkie/prom-run) terraform plan -detailed-exitcode https://www.weave.works/blog/provisioning-lifecycle-production-ready-kubernetes-cluster/
- 23. Hashistack ● Consul: service discovery, K/V store, service mesh… ● Vault: secret management and automation ● Terraform: infrastructure config as code
- 24. Questions?