Winning the metrics battle
- 1. Winning the metrics battle (finally)
- 2. Winning the metrics battle (finally) Simon Hildrew Nick Satterly Infrastructure Developer Monitoring Engineer The Guardian The Guardian
- 5. Total metrics 180,000 50,000 1,400 2,800
- 6. http://www.flickr.com/photos/ghostsigns/6676069121 5 minutes every 15 seconds http://www.flickr.com/photos/millynet/134071210
- 8. Physical screens Screensaver hacks 20 15 10 5 0
- 9. dev hack
- 11. metrics + dashboards = culture change
- 13. our approach Side project ➡ Prioritise Incremental upgrade ➡ Understand the real problem Use off the shelf tool ➡ Question the tools Pragmatic solution ➡ Be ambitious Done in a year ➡ Keep learning
- 14. Prioritise
- 16. a dedicated monitoring and metrics engineer
- 17. Understand the real problem
- 18. Urgent issue - current tool end of life
- 19. The story so far...
- 20. metrics were not helping us solve production outages
- 21. ballooning number of applications
- 22. but... difficult to instrument applications
- 23. T.T. Detect + T.T. Fix = T.T. Diagnose + T.T. Resolve
- 24. inaccessible tools http://www.flickr.com/photos/kdashy/2678539087
- 26. hypothesising & arguing easier than measuring http://www.flickr.com/photos/nouqraz/200049988
- 27. The ‘right’ thing • measure everything • measure frequently • measure each data point once • input and output must be open
- 32. Imposing a flawed model? http://www.flickr.com/photos/evansville/8953838/
- 33. Too difficult / no progress? http://www.flickr.com/photos/ginja_andy/4165849136/
- 34. Nagios • the “IBM” of monitoring tools • compromise over quantity and frequency of checks • < insert your criticism of nagios here >
- 35. Zabbix • metric collection tightly coupled to monitoring tool • confusing UI with poor visualisation • needed brute force to make limited API work
- 36. The ‘right’ thing • measure everything • measure frequently • measure each data point once • input and output must be open
- 38. don’t compromise
- 39. Be ambitious
- 40. http://www.flickr.com/photos/mugley/2961131550 Throw work away
- 41. Draw your dream
- 42. http://www.flickr.com/photos/sk8geek/7358702704 Get as far as you can
- 43. screens users db? alerting? Etsy dashboard message queue graphite SNMP? syslog? FITB ganglia api? network hosts applications
- 44. Develop missing pieces http://www.flickr.com/photos/kalexanderson/5969012589
- 45. screens users mongodb alerta elastic search Etsy dashboard message queue syslog SNMP graphite ganglia alerts alerts alerts FITB ganglia ganglia-api network hosts applications
- 48. rescale image??? Alerta https://github.com/guardian/alerta
- 49. Current stack • Ganglia • Guardian management https://github.com/guardian/guardian-management • FITB • Guardian ganglia-api https://github.com/guardian/ganglia-api • Graphite • Guardian alerta • Etsy dashboards https://github.com/guardian/alerta
- 50. Keep learning
- 51. we are not there yet
- 52. Watch the cultural changes
- 53. detecting
- 54. diagnosis
- 55. diagnosis
- 57. confirmation
- 58. #monitoringsucks
- 59. ➡ Prioritise ➡ Understand the real problem ➡ Question the tools ➡ Be ambitious ➡ Keep learning
- 60. tools can change culture
- 61. Thank you http://github.com/guardian http://gu.com/p/3ap5f Simon Hildrew Nick Satterly @sihil @nicksatterly simon.hildrew@guardian.co.uk nick.satterly@guardian.co.uk