Prometheus on Kubernetes
1 like215 views
This document discusses Prometheus on Kubernetes. It provides an overview of Prometheus and its ecosystem, including how it is used for service discovery on Kubernetes, collecting and storing metrics, ensuring high availability and scalability through sharding, and defining and alerting on service level agreements. It also covers instrumentation using exporters, the Prometheus query language PromQL, and components of the Prometheus ecosystem like Grafana and Alertmanager.
![PromQL
Funktionen
Request Rate pro Instanz:
rate(
http_count{handler="/ui/static"}[1m]
)
Aggregiert für jeden Service:
sum by (service) (
rate(
http_count{handler="/ui/static"}[1m]))
27](https://image.slidesharecdn.com/prometheus-on-kubernetes-cpetrausch-29-190503090614/85/Prometheus-on-Kubernetes-27-320.jpg)
![PromQL
Alert Rules
groups:
- name: website
rules:
- alert: HighErrorRate
expr: sum by (service)(rate(http_error{handler="/ui/static"}[1m])) > 0.5
for: 10m
labels:
severity: page
annotations:
summary: High request latency
30](https://image.slidesharecdn.com/prometheus-on-kubernetes-cpetrausch-29-190503090614/85/Prometheus-on-Kubernetes-30-320.jpg)
Prometheus on Kubernetes
- 1. Prometheus on Kubernetes Christoph Petrausch 29.4.2019
- 2. ● Golang ● Kubernetes Christoph Petrausch Linux Systems Engineer 2
- 5. 5 Service Discovery Pods, Nodes, Services A P I targets.yml A P I VMs DNS SRV, A, AAAA
- 7. 7 Metriken sammeln und speichern tsdb Target GET /metrics
- 13. 13
- 15. 15 Exporter Operating System Kubelet, Kube API, Kube Scheduler, etc.. Application Database mysql-exporter node-exporter
- 17. 17 Metrik Format # HELP Total Number of HTTP Requests # TYPE http_count counter http_count { } 731321 Dokumentation
- 18. 18 Metrik Format # HELP Total Number of HTTP Requests # TYPE http_count counter http_count { handler="/ui/static", instance="website-jas1kg1d-adjkm1", job="pods", service="website" } 731321 Labels
- 19. 19 Metrik Format # HELP node_disk_discard_time_seconds_total This is the total number of seconds spent by all discards. # TYPE node_disk_discard_time_seconds_total counter node_disk_discard_time_seconds_total{device="dm-0"} 0 node_disk_discard_time_seconds_total{device="dm-1"} 0 node_disk_discard_time_seconds_total{device="nvme0n1"} 0 node_disk_discard_time_seconds_total{device="sda"} 0 # HELP node_disk_discarded_sectors_total The total number of sectors discarded successfully. # TYPE node_disk_discarded_sectors_total counter node_disk_discarded_sectors_total{device="dm-0"} 0 node_disk_discarded_sectors_total{device="dm-1"} 0 node_disk_discarded_sectors_total{device="nvme0n1"} 0 node_disk_discarded_sectors_total{device="sda"} 0 # HELP node_cpu_seconds_total Seconds the cpus spent in each mode. # TYPE node_cpu_seconds_total counter node_cpu_seconds_total{cpu="0",mode="idle"} 100327.11 node_cpu_seconds_total{cpu="0",mode="iowait"} 167.2 node_cpu_seconds_total{cpu="0",mode="irq"} 1211.28 node_cpu_seconds_total{cpu="0",mode="nice"} 5762.09
- 20. Counter 20 Zeit
- 21. Gauge 21 Zeit
- 22. SLAs ● 99,9% aller Requests kürzer als 50ms ● 99,9% Verfügbarkeit ● 99,99% aller Request müssen erfolgreich sein 22
- 23. Long Tail 23
- 25. PromQL ● Angelehnt an SQL ● Aggregation über Labels ● Mathematische Funktionen ● Range und Offset Selektoren 25
- 26. PromQL Aggregationen Query: sum by (service)(http_count{handler="/ui/static"} ) Beispiel: http_count{handler="/ui/static",service="a",i="a"} 5 http_count{handler="/ui/static",service="a",i="y"} 5 http_count{handler="/ui/static",service="b",i="z"} 5 Resultat: {service="a"} =10, {service="b"} = 5 26
- 27. PromQL Funktionen Request Rate pro Instanz: rate( http_count{handler="/ui/static"}[1m] ) Aggregiert für jeden Service: sum by (service) ( rate( http_count{handler="/ui/static"}[1m])) 27
- 28. Histogram 28
- 29. PromQL Offset Differenz jetzt vs gestern: http_count – ( http_count offset 24h) 29
- 30. PromQL Alert Rules groups: - name: website rules: - alert: HighErrorRate expr: sum by (service)(rate(http_error{handler="/ui/static"}[1m])) > 0.5 for: 10m labels: severity: page annotations: summary: High request latency 30
- 31. PromQL Anwendung ● Grafana Dashboards ● Alert Rules ● Ad Hoc Queries ○ UI ○ API 31
- 33. Prometheus Ökosystem Alertmanager 33 ● Alert Routing ● Silences ● Inhibition ● Grouping
- 34. 34 1000 feet Architektur Pods, Nodes, Services A P I Target Target Target Target ... Grafana Alertmanager
- 35. Prometheus Offene Probleme 35 ● Langzeitspeicherung ● Aggregierung über mehrere Prometheus ● Deduplizierung bei Sharding
- 36. 36 Thanos T T T TSidecar Sidecar T T T TSidecar Sidecar Blob Storage Alertmanager Thanos Query Thanos Ruler Thanos Compact Grafana
- 38. Quellen 38