How and why we got Prometheus working with Docker Swarm
2 likes881 views
The document discusses different approaches to integrating Prometheus monitoring with Docker Swarm. Version 1 used a "prom-wrapper" container but had issues with services and networks being dynamically added. Version 2 used Prometheus in federated mode with "prom-proxy" containers, but was too complex. Version 3 dynamically adds Prometheus to networks using the prom-wrapper, allowing monitoring of services in Swarm. The implementation is available on GitHub and allows using Weave Cloud's hosted Prometheus for Docker Swarm.
How and why we got Prometheus working with Docker Swarm
- 1. How and why we got Prometheus working with Docker Swarm Luke Marsden @lmarsden
- 2. Luke’s goal: Make Weave Cloud accessible to Docker Swarm users. Let’s get Prometheus working with Docker Swarm! Justin’s goal: Everyone kept asking me “how do I Prometheus?”
- 3. So, how to integrate? +
- 5. V1 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B
- 6. V1 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B prom-wrapper writes service discovery info to Prom config, starts Prom on the right Docker networks…
- 7. V1 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B
- 8. V1 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B Service C Network C
- 9. V1 architecture Service A prom- wrapper Network A Service B Network B Service C Network C swarm API
- 10. V1 architecture Service A prom- wrapper Network A Service B Network B Service C Network C swarm API Net A+B+C
- 11. Downsides to V1 design • Assumption: can’t dynamically add container to new networks • Therefore, need to restart Prometheus every time a service or network comes or goes • Services & networks can be re-deployed frequently, especially in a busy cluster + continuous delivery • Means Prometheus can miss scrapes • Losing data :-(
- 12. V2 architecture Service A Network A Service B Network B swarm API Net P prom- proxy prom- proxy Net P+A Net P+B prom- wrapper Service AService A Service BService B
- 13. V2 architecture Service A Network A Service B Network B Service C Network C swarm API Net P prom- proxy prom- proxy Net P+A Net P+B prom- wrapper Service AService A Service BService B Service CService C
- 14. V2 architecture Network A Network B Network C swarm API Net P prom- proxy prom- proxy prom- proxy Net P+A Net P+B Net P+C prom- wrapper Service AService A Service BService B Service CService C
- 15. What is prom-proxy? prom- proxy • socat? • No, needs to aggregate stats from multiple instances, concatenate them, discover them… • Write a custom proxy? • Hmm it’s starting to look a look like Prometheus… • Eureka! Use Prometheus in federated mode!
- 16. V2 architecture Service A Network A Service B Network B Service C Network C swarm API Net P Net P+A Net P+B Net P+C prom- wrapper Service AService A Service BService B Service CService C
- 17. Downsides to V2 design • It’s bloody complicated! • “You’re not going to do that, are you?” • Lots of moving parts • Oh well, let’s give it a go
- 19. Containers can be dynamically added to networks!
- 20. V3 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B
- 21. V3 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B prom-wrapper writes service discovery info to Prom config, starts Prom on the right Docker networks…
- 22. V3 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B
- 23. V3 architecture Service A prom- wrapper Network A Service B Network B swarm API Net A+B Service C Network C
- 24. V3 architecture Service A prom- wrapper Network A Service B Network B Service C Network C swarm API Net A+B+C prom-wrapper dynamically adds Prometheus to the right networks, rewrites config which Prom hot-reloads
- 25. So that’s what we did :-) • github.com/containersolutions/ prometheus-swarm-discovery • Now you can monitor Docker Swarm with Weave Cloud :-) • Demo!
- 26. Conclusion • Weave Cloud is an ops platform for app devs, help you go round the loop faster • Includes hosted scalable Prometheus, works with Swarm • Try the prototype at github.com/ containersolutions/prometheus-swarm-discovery • Try Weave Cloud at weave.works
- 27. Join the Weave user group for talks & trainings! • Continuous delivery • Network policy for security • Deep dive on Prometheus monitoring meetup.com/pro/Weave
- 28. and then we went to the pub!