How Ansible Tower and Prometheus can help automate continuous deployments
- 1. Ansible Tower with Prometheus Experiences using ansible and prometheus in your CI/CD workflow
- 2. About This talk is about how we built a continuous deployment system for Deep-Impact. ● Spectra is a web application ● Multi-tenant ● Built with Clojure, Mango, Grape, MongoDB, ElasticSearch ● Runs on, AWS: EC2, Container services Darragh Grealish Site Reliability Engineer at 56K.Cloud DevOps in a Cloud Neutral way dg@56k.cloud https://www.56K.cloud/ What we do: ➔ Ansible, Docker Engineering ➔ CI/CD Improvements and Setup ➔ DevOps Bootstrapping ➔ CDN Consulting ➔ Wireless and Microwave Networking 56KCLOUND
- 3. Background What triggered the need: Deployments and Infrastructure managed by a one-click deployment service with Skyliner.io In spring they announce to shut down in July 15th ● Solely based on AWS ● Deploys one branch/repo per project ● Each project had a QA and Production setup ● Heavy use of EC2, Docker, ALB/ELB and cloudformation
- 5. What is Prometheus A open-source time-series based monitoring system, founded in 2012 by ex-Googlers working for Soundcloud Inspired by Google’s Borgman internal monitoring tool Uses time-series data as a data source for generating alerts, this makes it different than other tools out there Components: ● Prometheus ● Alert Manager ● Node Exporter (agent)
- 6. What is Prometheus Growing bigger, Github Project: Almost 4k commits, 11k stars, 1,258 forks, (178 issues, PR: 45 open / 1632 closed) 16,000 active installations of prometheus running behind Grafana, that report stats Grafana Talk, PromCon 2017 Munich
- 7. Ansible Tower A nice UI to see all your infrastructure with access control ● Schedule playbook, as Jobs ● Push Button Deploys* ● Dynamic Inventory from AWS ● Free Tier, up to 10 nodes (or inventory items) ● Using a pull model from source control, e.g Github ● Notifications: email, slack ● Access Control, with various integration, Google Auth, LDAP*
- 8. Deployment Workflow Deployment Stack: ● Ansible (/w tower) ● Prometheus ● Docker ● AWS EC2 ● AWS ECR (Register) ● AWS ELB (Load-balancer)
- 9. Demo ● We will add a new host to EC2 ● Ansible Tower will add the host to inventory ● Scheduled playbook will add the node_exporter to the new host and a prometheus target ● Monitoring will start ● An alert will fire when we kill the node_exporter Links: http://192.168.106.27:3000/dashboard/db/node-exporter-full?orgId=1&from=now-30m&to=now&refresh=1 m&var-node=52.209.114.9&var-port=9100 https://192.168.106.15/api/v1/inventories/70/ http://192.168.106.27:9093/#/alerts Cloud Guy, Trolls Movie, DreamWorks
- 10. Looking back ... Source: blog.intercom.com 3 ~ 6 months on
- 11. Looking back… Findings: 3~6 months on: ● One-Click deploys means less flexibility, but robustness ● Unattended automation isn’t easy, ○ Human intervention almost always required to address what state the infra is in ○ Possible mangement want the go-live decision ● Automating to reduce hosting costs , will create ● If your application can’t scale, your automation becomes more complex ○ Deterministic decisions ● Alerting on actionable data require you to gain application knowledge ● Graphing really helps, it visualizes the concern
- 12. Tooling: Ansible Tower Pros/Cons Pros: ● Simple UI for non-technical people ● One-click deployments (almost) ● Notifications; Slack messages look nice ● Scheduling Jobs (Playbook runs) ● Access Control Cons: ● Free tier of 10 nodes, hard to determine value in ● Can not be triggered vi Webhooks ● Addresses many CI/CD needs, but still needs a CI ● Secrets get split between your repository and ansible tower,
- 13. Tooling: Prometheus Pros/Cons Pros: ● Light, Quick setup ● Simple config file configuration ● Lots of integration, ● Very Open-Source , even the exporter is trying to standardize ● Large community CNCF member, Big : Digital Ocean, Soundcloud, Cloudflare, and now DB Cons: ● Early Days, version 1.x to 2.x lots of changes, It can break! ● Dataloss, only recently a backup strategy ● Alertmanager and Prometheus UI not consolidated ● Long-term data is not it’s streatch ● No access control (but no a focus)
- 14. Addressing the challenges: Making it dynamic Exploiting AWS Spot instance, 70% cheaper!! /w Block Statement ● Wrapping the ec2 module in a block statement and iterate over aws size type, Bid a set price: e.g 0.29~0.35 cent ● But with a condition: “--extra-vars=aws_spotinstance=true” as it can waste you more time ● ec2instances.info is great help for comparing
- 15. Making it dynamic Check in the playbook if your service is up, before adding it to monitoring and waking up people
- 16. Making it dynamic.. Use Tags, like everywhere, but not crazy ● Helps to maintain state ● Relabel your instances, use friendly names. ● Reference with environment and version “tag:Name=prod-elegant_cori_v1.0.16” ● Use instance filters in Ansible-Tower to consolidate your “tag:Platform=dreamliner” AWS EC2 console: list of instances
- 17. Thank you Questions My Details: Darragh Grealish Twitter: @grealish 56K.Cloud - DevOps Consulting and Services For: Ansible, Docker, Network Infrastructure dg@56k.cloud https://www.56K.cloud/ No clouds were harmed during the deployment of this talk :) Cloud Guy, Trolls Movie, DreamWorks
- 18. Backup 2 - check application state before monitor
- 19. Backup - looking up active targets in prometheus
- 20. Backup - Before prometheus reload with targets
- 21. Backup - Grafana node exporter
- 22. Backup
- 23. Tower API Inventory https://192.168.106.15/api/v1/inventories/67/hos ts/ ● We get a list of host ● Identify the hosts list ● Create a fact dict of hosts, ● Install prometheus exporters on these hosts ● Add the hosts into prometheus targets configuration ● Check for the targets to go green in the prometheus API
- 24. References: Sources that supported this talk/demo: https://prometheus.io/blog/2017/06/21/prometheus-20-alpha3-new-rule-format/