Order from chaos: automating monitoring configuration
1 like235 views
Molly Duggan from Harvard's FAS research computing discusses the automation of monitoring configuration within a complex infrastructure including 100,000 CPU cores and 40 petabytes of storage. The project aimed to address issues with manual configuration and existing tools, leading to the development of 'hinoki', a command-line tool for managing Sensu configuration that allows for easier contributions and flexible integration. Outstanding issues remain, including the need for additional features and clarity in configuration management.
Order from chaos: automating monitoring configuration
- 1. Order from Chaos: Automating Monitoring Configuration Molly Duggan Harvard – FAS Research Computing https://rc.fas.harvard.edu
- 2. A Little Context ● 100,000 CPU cores on 3,000 nodes running 29 million jobs/year ● 40PB of storage on a variety of different systems ● 2 data centers ● 500+ lab groups with over 5500 users ● Cloud and VM infrastructure ○ Connected VMs for applications showing research data ○ Assorted DBs (researchers, museums) ○ Internal services (puppet, gitlab, etc) Everything that’s not compute is a snowflake!
- 4. Options ● Manual configuration through a dashboard with backups ○ Too much of a free-for-all ○ Not easy to see changes ○ Too hard to roll back updates ● Config Management ○ Old puppet version ○ Too much in one place ● Script against sensuctl ○ Not everything we wanted was implemented at the time we began this project (during the beta) ○ Asset packaging needs
- 5. Hinoki ● A tiny command-line tool to manage Sensu configuration ● Advantages for our shop: ○ Discrete repo with audit trail ○ Easy contribution for everyone on the team - just a git push ○ Flexibility moving forward ● Equal parts code and convention with CI/CD integration ● Import definitions via Sensu API ● Quick-start provisioning of an empty cluster ● Ship and package assets, add hash to configs ● pip install hinoki
- 6. Demo #1: Updating Settings
- 7. Demo #2: Initialize New Cluster
- 8. Outstanding Issues ● Add more features! ● We still repeat ourselves ● It can still be confusing to understand what you might be touching if you alter a check
- 9. Thank You! Molly Duggan Harvard – FAS Research Computing https://rc.fas.harvard.edu Github: @exitquote https://github.com/fasrc/hinoki