Safe and Fast Automation on AWS for Fun and Profit
0 likes138 views
The document discusses automation practices on AWS, highlighting methods for provisioning, incident management, and configuration management using tools such as Puppet and Terraform. It emphasizes the use of AWS Systems Manager for operational insights and automation, detailing features such as resource groups, parameter store, and security measures through IAM. Furthermore, it outlines security practices, execution control, and a practical use case for running commands in a Zookeeper cluster.
![Use case: zookeeper restart on a 5 node cluster
aws --region us-west-2 ssm send-command --document-name
"AWS-RunShellScript" --targets
'Key=tag:puppet:role::zookeeper,Values=cluster_name=test_generic-u
swest2-devc' --parameters
'{"workingDirectory":["/"],"executionTimeout":["100"],"commands":[
"zk_tool local check_cluster", "sudo service zookeeper restart",
"sleep 30", "zk_tool local check_cluster"]}' --timeout-seconds 300
--max-concurrency 2 --max-errors "1" --output text](https://image.slidesharecdn.com/safefastautomationonawsforfunprofit-191117230714/85/Safe-and-Fast-Automation-on-AWS-for-Fun-and-Profit-28-320.jpg)
Safe and Fast Automation on AWS for Fun and Profit
- 1. Safe and Fast Automation On AWS For Fun and Profit (?) Raghavendra Prabhu, Yelp me@rdprabhu.com / @randomsurfer
- 7. Automation in context of DevOps ➔ Provisioning ➔ Remediation ➔ Lifecycle Management ➔ Alerting /+ Actioning ➔ Incident Management ➔ Oncall handling ➔ Configuration Management ➔ Fleet actions ➔ … ➔ Anything that is done often and unsupervised
- 8. Automation at Yelp ● Ranges from Declarative to Imperative ○ Depending on how much is on fire ● Puppet and Terraform ● Fabric ● MCO ● Marley ● Taskerman ● Jenkins ● Clusterman ● SSH ● And many others..
- 9. Motivation ● More extensible and elegant than ssh/mco ● Lighter than taskerman ● More AWS aware and tighter integration ● Straddling the declarative and imperative ● Use cases ○ Chaos Reliability Engineering ○ A/B testing ○ Quick prototyping ● Better error handling ● Safer concurrency ● Secure!
- 11. AWS Systems Manager ● Boooring Enterprisey Name ● “Gain Operational Insights and Take Action on AWS Resources” ● “Systems Manager includes a unified interface that allows you to easily centralize operational data and automate tasks across your AWS resources.”
- 13. What SSM can do ● Resource groups ○ “Tag all resources easily and query” ● Actions ○ “Run something on tagged” ○ “Patch for security updates” ● Insights on Inventory ○ “Tell me something about resources that is not obvious” ● Parameter Store ○ “Let us parameterize our configuration” ● State Manager ○ “Let us manage resources based on state”
- 17. Run Command ● SSM service ● Run pre-defined or ad-hoc “commands” or flow-based automation ● Command = Target (Who) + Document (What) + Params (When) + How ● “Managed Instance” ● Parameterized “Document” ○ Predefined ○ Custom ● Security ● Free to use ● Invocation from CLI, SDK(boto et.al) or console.
- 21. Workflow 1. IAM instance profile on nodes a. Done in puppet (and never on console) b. Giant web of permissions c. Needed for invoker and executor and viewer. 2. Deploy SSM agent on nodes. a. Packages available for Ubuntu b. Setup a jenkins job and added to repo: amazon-ssm-agent c. Puppetized and deploy on zookeeper cluster. 3. Tag instances if not already a. Reused ubiquitous puppet::role tag 4. Fire! a. `aws ssm send-command`
- 22. Security ● Managed through IAM (Identity and Access Management) ○ Users, Groups, Roles, APIs. ● Cloudtrail-based auditing (sourcetype=aws:cloudtrail) ● Not dependent on SSH ● IAM instance profile for executor ○ Instance role defined in puppet ○ Attached to the host in terraform ● Configure User to read status ● Secret management through parameter store ● Highly granular access control ○ Adding approvers and SNS ○ Possible to separate roles of invoker, executor and viewer.
- 24. Security observations ● Multiple ways in documentation to achieve same thing ● Quick prototyping hard with IAMs and puppet ○ IAM Policy Simulator helps here. ○ Updating on console is disallowed (for good reasons). ● The recommended IAMs in docs are wide-open. ● Documentation on IAM workflow and IAMs at Yelp could improve ● Executor’s permissions ● Nature of agent ○ Open source
- 26. Execution Control ● Circuit breakers ○ max-errors (% or count) ○ timeout ● Concurrency (% or count) ○ Exponentially scaled (1, 2, 4, .. value) ● Targets ○ Instance IDs ○ Union or intersection of Tags ● Cancellation ● Maintenance Windows ● Command Timeout ● Integration
- 27. Output and Monitoring ● Audit logs with AWS cloudtrail ● Short output and return statuses through SSM ● Output storage in S3 ● Cloudwatch logs ● SNS notifications ○ Asynchronous nature ● AWS Console Cloudwatch Cloudtrail SNS
- 28. Use case: zookeeper restart on a 5 node cluster aws --region us-west-2 ssm send-command --document-name "AWS-RunShellScript" --targets 'Key=tag:puppet:role::zookeeper,Values=cluster_name=test_generic-u swest2-devc' --parameters '{"workingDirectory":["/"],"executionTimeout":["100"],"commands":[ "zk_tool local check_cluster", "sudo service zookeeper restart", "sleep 30", "zk_tool local check_cluster"]}' --timeout-seconds 300 --max-concurrency 2 --max-errors "1" --output text
- 30. Credits ➢ https://www.pexels.com/photo/person-holding-black-pen-1020325/ ➢ https://imgur.com/gallery/LVSJDgy ➢ https://www.pexels.com/photo/gray-steel-tubes-586019/ ➢ https://i.kym-cdn.com/photos/images/original/000/572/078/d6d.jpg ➢ https://aws.amazon.com/systems-manager/getting-started/ ➢ https://giphy.com/gifs/animation-cartoon-robot-8qrrHSsrK9xpknGVNF ➢ https://www.pexels.com/photo/amplifier-analogue-audio-blur-462439/ ➢ https://www.pexels.com/photo/black-and-white-business-chart-computer-241544/ ➢ https://media3.giphy.com/media/lXiRpzaKeG9nWR5eM/giphy.gif