Monitor all the thingz slideshare
Download as PPTX, PDF0 likes186 views
This document summarizes Soluto's transition to a new monitoring system. It discusses moving from a self-hosted Nagios installation to using Prometheus and ELK/Logz.io for time-series and log data. The new system provides self-service monitoring, scalability, and is open source. It monitors using a monitoring-as-code approach across technologies. Soluto also uses Mixpanel for business metrics. Next steps include more Prometheus adoption, distributing Icinga, and tighter coupling of monitoring with code.
Monitor all the thingz slideshare
- 1. Monitor all the thingz February 2018 @yaronidan @SolutoEng
- 3. Let’s think about it... ● Does monitoring = production? ● Can alerts suck less? ● Who carries the pager?
- 5. Our old monitoring system - https://www.ks-soft.net/hostmon.eng/
- 6. And then we grew… ● Ops became a bottleneck ● No monitoring for new tech ● SLA is now a thing
- 8. We realized we need a better solution Self service Scalable Open source with a wide community
- 11. Where do we keep our monitoring data? ● TSDB – Graphite
- 12. Where do we keep our monitoring data? ● TSDB – Graphite
- 13. Where do we keep our monitoring data? ● TSDB – Graphite/Prometheus ● ELK - powered by logz.io
- 14. Where do we keep our monitoring data? ● TSDB – Graphite/Prometheus ● ELK - powered by logz.io ● Any other data source
- 17. Let’s take a look at the engine ● Monitoring-as-code ● Scalable ● Modularity ● We can monitor anything we can think of
- 24. Who do we alert?
- 25. One person
- 27. Rotating on-call duty per team
- 28. Escalation
- 29. Did we fulfill what we set out to achieve? Self-service Scalable Open source with a wide community
- 30. Everybody contributes (Commit statistics for production Feb 20 - Jan 29) 2000commits in 343days
- 31. Everybody contributes (Commit statistics for production Feb 20 - Jan 29) 2000commits in 343days 5.8commits per day
- 32. Everybody contributes (Commit statistics for Feb 20 - Jan 29) 2000commits in 343days 5.8commits per day Contributed by73authors
- 33. Monitoring business ● We’re also using mixpanel for monitoring business metrics ● Different than traditional monitoring ● Gives us a broad view of the products condition
- 37. Back to where we started... ● Does monitroing = production? ● Can alerts suck less? ● Who carries the pager?
- 38. What’s next? ● More prometheus ● Distribute Icinga ● Tighter coupling with code
- 39. Questions?
- 42. Sources Monitoring blog post at Soluto’s engineering blog https://github.com/Soluto/nagios-plugins https://github.com/Soluto/check-redis-plugin https://github.com/Icinga/puppet-icinga2
- 43. Tips and Tricks Don’t let alert fatigue get you! Adjust alerts to only fire when disaster strikes Learn from outages - make sure alerts were firing, and if they don’t - make‘em! Choose a solution that allows versioning, preferable using monitoring-as-code Use templating for apps that share the same monitoring patterns Share the joy of holding a pager with your fellow developers Focus your monitoring efforts on metrics that can harm your business