Scraping metrics for fun and profit
0 likes37 views
The document discusses the importance of observability in large, complex systems, focusing on the three pillars: metrics, logs, and traces. It highlights tools like Prometheus for metrics collection and Grafana for visualization, providing examples of configurations and custom exporters. Additionally, it emphasizes best practices in alerting and defining service-level agreements (SLAs) to monitor system health effectively.
![Confidential and Proprietary
PromQL
http_requests_total{job=”nginx", handler="/grafana"}[5m]
sum by (job) ( rate(http_requests_total[5m]) )
sum by (region) (bird_protocol_up{ip_version="ipv4",proto="BGP",env="$environment"})
https://prometheus.io/docs/prometheus/latest/querying/basics/](https://image.slidesharecdn.com/scrapingmetricsforfunandprofit1-240205134948-96823b04/85/Scraping-metrics-for-fun-and-profit-18-320.jpg)
Scraping metrics for fun and profit
- 1. Scraping metrics for fun and profit Bram Vogelaar @attachmentgenie
- 2. Confidential and Proprietary ~ ❯ whoami • Used to be a Molecular Biologist • Then became a Dev, now an Ops • Currently Cloud Engineer @ Seaplane.io • Amsterdam HUG organizer • Author of soon to be released LGTM book
- 4. Confidential and Proprietary Can you honestly answer this question? Can you figure out if your platform is in an error state?
- 5. Confidential and Proprietary Large complex systems will always be in a (somewhat) degraded state
- 6. Confidential and Proprietary 3 pillars of observability
- 7. Confidential and Proprietary 3 pillars of observability Metrics Logs Traces
- 8. Confidential and Proprietary Metrics ● Open-Source tool to do metrics collection and storage ● Since 2012/2015, by SoundCloud ● CNCF graduated project ● HTTP pull model ● PromQL DSL https://prometheus.io/
- 9. Confidential and Proprietary Metrics Ecosystem
- 10. Confidential and Proprietary Viewing (Node Exporter) Metrics
- 11. Confidential and Proprietary BYOD exporter / Beg, borrow, steal https://prometheus.io/docs/instrumenting/exporters/
- 12. Confidential and Proprietary Prometheus Configuration global: scrape_interval: 15s scrape_configs: - job_name: o11y static_configs: - targets: - localhost:9100 - localhost:3100 - localhost:3200
- 13. Confidential and Proprietary Prometheus/Metrics HA, Dynamic Scraping global: scrape_interval: 10s scrape_configs: - job_name: o11y static_configs: - targets: - localhost:9100 - localhost:3100 - localhost:3200 - job_name: o11y scrape_interval: 10s consul_sd_configs: - server: localhost:8500 tags: - metrics
- 14. Confidential and Proprietary Write your own exporter package main import ( "net/http" "github.com/prometheus/client_golang/prometheus/promhttp" ) func main() { http.Handle("/metrics", promhttp.Handler()) http.ListenAndServe(":2112", nil) }
- 15. Confidential and Proprietary Write your own exporter var pingCounter = prometheus.NewCounter( prometheus.CounterOpts{ Name: "ping_request_count", Help: "No of request handled by Ping handler", }, ) func ping(w http.ResponseWriter, req *http.Request) { pingCounter.Inc() fmt.Fprintf(w, "pong") } func main() { prometheus.MustRegister(pingCounter) …
- 16. Confidential and Proprietary Write your own exporter
- 17. Confidential and Proprietary Forcing labels / Decorating metrics global: scrape_interval: 10s external_labels: datacenter: "%{::trusted.extensions.pp_datacenter}" region: "%{::trusted.extensions.pp_region}" <- use ${IATA_CODES} env: "%{::trusted.extensions.pp_environment}"
- 18. Confidential and Proprietary PromQL http_requests_total{job=”nginx", handler="/grafana"}[5m] sum by (job) ( rate(http_requests_total[5m]) ) sum by (region) (bird_protocol_up{ip_version="ipv4",proto="BGP",env="$environment"}) https://prometheus.io/docs/prometheus/latest/querying/basics/
- 19. Confidential and Proprietary Tensegrity of observability Metrics Logs Traces Alerts Dashboards
- 20. Confidential and Proprietary Dashboards ● Grafana is a multi-platform open source analytics and interactive visualization web application ● Since 2014, by Torkel Ödegaard https://grafana.com/oss /grafana/
- 22. Confidential and Proprietary Zooming to what you need to see
- 23. Confidential and Proprietary What is going on here?
- 24. Confidential and Proprietary Annotate Graphs
- 25. Confidential and Proprietary Correlate metrics with events
- 26. Confidential and Proprietary Data should tell beautiful stories https://www.youtube.com/watch?v=hVimVzgtD6w https://www.youtube.com/watch?v=jbkSRLYSojo
- 27. Confidential and Proprietary Data should tell beautiful stories
- 28. Confidential and Proprietary Canvas Plugin
- 29. Confidential and Proprietary Correlate events with metrics
- 30. Confidential and Proprietary Instrument data flows
- 31. Confidential and Proprietary Correlate events with metrics Marketing Campaign ON
- 32. Confidential and Proprietary Don’t stare at dashboards
- 33. Confidential and Proprietary Alerts are PromQL queries too - alert: PrometheusJobMissing expr: absent(up{job="prometheus"}) for: 0m labels: severity: warning annotations: summary: Prometheus job missing (instance {{ $labels.instance }}) description: "A Prometheus job has disappearedn VALUE = {{ $value }}n LABELS = {{ $labels }}" https://awesome-prometheus-alerts.grep.to/
- 34. Confidential and Proprietary R.E.D & U.S.E U = Utilization, as canonically defined S = Concurrency E = Error Rate, as a throughput metric R = Request Throughput, in requests per second E = Request Error Rate, as either a throughput metric or a fraction of overall throughput D = Latency, Residence Time, or Response Time; all three are widely used
- 36. Confidential and Proprietary What we need to visualize $/€ revenue # of sales # of signups # of Api calls “(Ab)Normal” application use
- 37. Confidential and Proprietary SL{O,I,A} Service-Level Agreement (SLA) => Contract Service-Level Objective (SLO) => Wish/Hope Service-Level Indicator (SLI) => Observe
- 38. Confidential and Proprietary Only alert on SLO violation And definitely on HDD space usage Don’t alert on symptoms Do alert on the rate you are using the error budget