Prometheus Multi Tenancy
- 1. Multi-tenant Kubernetes observability with Prometheus robusta-dev Natan Yellin aantn Natan Yellin, robusta.dev
- 2. $ whoami Co-founder of robusta.dev Multi-cluster Kubernetes observability Add-on to Prometheus Substack newsletter: Why this Kubernetes thing? Natan Yellin aantn robusta-dev
- 3. How should I gather Prometheus metrics from all my tenants? Natan Yellin aantn robusta-dev
- 4. Assumptions Natan Yellin aantn Clusters Namespaces Virtual clusters (e.g. capsule, kamaji, vcluster) etc... 1. Many Kubernetes tenants 2. Tenants need some form of isolation 3. We want to monitor with Prometheus robusta-dev
- 5. What should I use? Natan Yellin aantn robusta-dev
- 6. In the beginning there was one Natan Yellin aantn robusta-dev
- 7. In the beginning there was one Natan Yellin aantn Simple No security isolation/RBAC No performance isolation If tenants are clusters, discovery is annoying Advantages: Disadvantages: "One team broke Prometheus for everyone else" robusta-dev
- 8. Then there were many Natan Yellin aantn robusta-dev
- 9. Then there were many Natan Yellin aantn Simple Security isolation Performance isolation Scalable? No unified queries No unified management More resources? Advantages: Major Disadvantage: Minor Disadvantages: "If you break it, it only breaks for your product line." robusta-dev
- 10. What we want Natan Yellin aantn Isolation Scalability Decentralized: Query all Prometheuses at once Centralized: robusta-dev
- 11. What else we want? Natan Yellin aantn Scalability Long term storage of metrics 1. 2. robusta-dev
- 12. Three approaches Natan Yellin aantn robusta-dev
- 13. Solve it outside Prometheus Natan Yellin aantn robusta-dev
- 14. Solve it outside Prometheus Natan Yellin aantn Doesn't touch Prometheus itself Delegates problem to other tool Queries need to address one Prometheus at a time Key advantages: Key disadvantage: robusta-dev
- 15. Multiple + Centralized (take 1) Natan Yellin aantn robusta-dev
- 16. Multiple + central (take 1) Natan Yellin aantn Reuses existing Prometheus Federated can do roll-up Federated can selectively scrape With roll-up/selective you can't actually query all Prometheuses Scaling Key advantages: Key disadvantages: robusta-dev
- 17. Natan Yellin aantn Disclaimer: Thanos has lots of options, I'm simplifying a little robusta-dev
- 18. Multiple + central (take 2) Natan Yellin aantn robusta-dev
- 19. Multiple Prometheuses + central Prometheus (take 2) Natan Yellin aantn Super scalable! Reuses existing Prometheus Very common solution, lots of tooling No RBAC built-in Key advantages: Key disadvantages: "Most mature option" - most people robusta-dev
- 20. One Prometheus to Rule them All Natan Yellin aantn robusta-dev
- 21. One Prometheus to Rule them All Natan Yellin aantn robusta-dev Cortex Grafana Mimir VictoriaMetrics TimescaleDB M3DB Options: ...
- 22. Grafana Mimir Natan Yellin aantn robusta-dev Native multi-tenancy! Backed by Grafana Complexity Key advantages: Key disadvantages:
- 23. Other useful tools Natan Yellin aantn Add prom-label-proxy to Thanos (and others) to enforce RBAC robusta-dev
- 24. Thank you! Natan Yellin aantn A special thank you to Shalom Cohen and Evgeny Uklist + Racoons team for providing inputs robusta-dev