![promsd sidecar
"exporter_hosts": [
"10.0.0.1",
"10.0.0.2",
"10.0.0.3",
"10.0.0.4"
]
con๏ฌgly
fetch list of hosts
in a datacenter
1
promsd proxy
request /targets endpoint
for each host to get list
of available scrape targets
32
3
output all targets as a
๏ฌle service discovery
JSON ๏ฌle
4
Prometheus reads
the ๏ฌle and scrapes
the con๏ฌgured
targets.
{
"targets": [
โ10.0.0.1:9702โ,
โ10.0.0.2:9702โ
],
"labels": {
"__metrics_path__": โ/node_exporter_9100/metrics",
"job": โnode_exporterโ
}
},
{
"targets": [
โ10.0.0.1:9702โ,
โ10.0.0.2:9702โ
],
"labels": {
"__metrics_path__": "/varnishstat_exporter_19102/metrics",
"job": "varnishstat_exporter"
}
}
PromSD sidecar](https://image.slidesharecdn.com/monitoring-at-scale-migrating-to-prometheus-at-fastly-180813222714/85/Monitoring-at-scale-Migrating-to-Prometheus-at-Fastly-27-320.jpg)
Monitoring at scale: Migrating to Prometheus at Fastly
- 1. Monitoring at Scale Migrating to Prometheus at Fastly PROMCON 2018 | Marcus Barczak @ickymettle
- 5. How were we monitoring Fastly?
- 6. +
- 7. เน Operational overhead. เน Limited graphing functions. เน No alerting support, เน No real API for consuming metric data. Growing pains with Ganglia
- 8. aaS + +
- 9. เน Now supporting two systems. เน Where do I put my metrics? เน Still writing external plugins and agents. เน Monitoring treated as a "post-release" phase. Growing pains doubled
- 10. Scaling our infrastructure horizontally Required scaling our monitoring vertically
- 11. Third time lucky
- 12. เน Scale with our infrastructure growth, เน Be easy to deploy and operate. เน Engineer friendly instrumentation libraries. เน First class API support for data access. เน To reinvigorate our monitoring culture.โจ See: https://peter.bourgon.org/observability-the-hard-parts/ Project goals
- 13. ?
- 15. เน Build a proof of concept. เน Pair with pilot team to instrument their services. เน Iterate through the rest. เน Run both systems in parallel. เน Decommission SaaS system and Ganglia. Getting started
- 17. prometheus A prometheus B scrapes targets SJC scrapes targets
- 18. prometheus A prometheus B scrapes targets SJC scrapes targets prometheus A prometheus B scrapes targets JFK scrapes targets prometheus A prometheus B scrapes targets ATL scrapes targets
- 19. prometheus A prometheus B scrapes targets SJC scrapes targets prometheus A prometheus B scrapes targets JFK scrapes targets prometheus A prometheus B scrapes targets ATL scrapes targets GCP federator A federator B frontend stack
- 20. prometheus A prometheus B scrapes targets SJC scrapes targets prometheus A prometheus B scrapes targets JFK scrapes targets prometheus A prometheus B scrapes targets ATL scrapes targets GCP federator A federator B frontend stack Query Tra๏ฌc (TLS)
- 21. Prometheus Server Software Stack Ghost Tunnel TLS termination and auth. Service Discovery Sidecar Target con๏ฌguration Rules Loader Recording and Alert rules Prometheus
- 22. Prometheus Server Software Stack Ghost Tunnel TLS termination and auth. Service Discovery Sidecar Target con๏ฌguration Rules Loader Recording and Alert rules Prometheus Typical Server Software Stack Service Discovery Proxy Service discovery and TLS exporter proxy Exporters Built into services or sidecar
- 23. Build your own service discovery?
- 24. Fastly's infrastructure is bare metal hardware no cloud conveniences
- 25. เน Automatic discovery of targets. เน Self-service registration of exporter endpoints, เน TLS encryption for all exporter tra๏ฌc. เน Minimal exposure of exporter TCP ports. Service discovery requirements
- 26. Prometheus Server Software Stack Ghost Tunnel TLS termination and auth. PromSD Sidecar Target con๏ฌguration Prometheus Typical Server Software Stack PromSD Proxy Service discovery and TLS exporter proxy Exporters Built into services or sidecar generates con๏ฌg for prometheus scrapes proxied targets over TLS queries for available targets
- 27. promsd sidecar "exporter_hosts": [ "10.0.0.1", "10.0.0.2", "10.0.0.3", "10.0.0.4" ] con๏ฌgly fetch list of hosts in a datacenter 1 promsd proxy request /targets endpoint for each host to get list of available scrape targets 32 3 output all targets as a ๏ฌle service discovery JSON ๏ฌle 4 Prometheus reads the ๏ฌle and scrapes the con๏ฌgured targets. { "targets": [ โ10.0.0.1:9702โ, โ10.0.0.2:9702โ ], "labels": { "__metrics_path__": โ/node_exporter_9100/metrics", "job": โnode_exporterโ } }, { "targets": [ โ10.0.0.1:9702โ, โ10.0.0.2:9702โ ], "labels": { "__metrics_path__": "/varnishstat_exporter_19102/metrics", "job": "varnishstat_exporter" } } PromSD sidecar
- 28. promsd proxy fetch list of installed systemd services node_exporter process_exporter systemd "node_exporter": { "prometheus_properties": { "target": "127.0.0.1:9100" } }, โฆ "varnishstat_exporter": { "prometheus_properties": { "target": "127.0.0.1:19102" } } for each corresponding systemd service fetch the local exporter target address varnishstat_exporter 1 32 3 con๏ฌgly exposes an API used by prometheus and promsd sidecar /node_exporter_9100/metrics /varnish_exporter_19102/metrics /targetssidecar PromSD proxy
- 29. เน Really easy to leverage the ๏ฌle SD mechanism. เน New targets can be added with one line of con๏ฌg. เน TLS and authentication everywhere. เน Single exporter port open per host. It worked!
- 31. Prometheus at Scale at Fastly 114 Prometheus servers globally 28.4M time series 2.2M million samples/second
- 32. ... a few hours later
- 33. เน Engineers love it. เน Dashboard and alert quality have increased. เน PromQL enables some deep insights. เน Scaling linearly with our infrastructure growth. Prometheus wins
- 34. เน Metrics exploration without prior knowledge. เน Alertmanager's ๏ฌexibility. เน Federation and global views. เน Long term storage still an open question. Still some rough edges.
- 35. ๐