Vagrant to-aws-flow
Download as PPTX, PDF0 likes8,993 views
Vagrant can be used to create consistent development environments that match production. This allows developers to develop locally yet still have access to all production dependencies. The presenter recommends using Vagrant to provision local development environments that match the OS used in production AWS environments. Developers can then use the same configuration management tools and processes locally and in production. Code is deployed from Vagrant environments to AWS, ensuring development and production remain consistent. A demo is provided to stand up identical local and AWS environments and deploy code between them.
Vagrant to-aws-flow
- 1. Vagrant to AWS Flow Delivered by Colin Johnson, BlueChipTek (@bluechiptek)
- 2. Who am I? • DevOps Engineer and Occasional Developer • Certifications: • AWS Certified Solutions Architect – Associate Level (Number 224) • AWS Certified SysOps Administrator – Associate Level (Number 2564) • Worked with a number of Bay Area companies: inPowered, Jive Software, Lyft, NetShelter, Wingz • Contributed VPC and Testing Support for boto_* Modules in SaltStack Project
- 3. Goals for this Presentation • Understand benefits and pain points of cloud, local and vagrant based development • Describe a development flow that combines vagrant and AWS to create a: • consistent environments for all developers • consistent environment from development to production • help organizations move fast – if they are not already doing this • integrate nearly flawlessly with AWS • Ease Development <-> Production Software Deployment
- 4. Why Vagrant to AWS Flow? • Benefits for Everyone: • Really Fast. Minutes to get a development environment setup. • Really Fast. No lag to test code changes. • Really Consistent. More similar to production than traditional “local” development. • Easy. Developers just type “vagrant ssh” • Developers • Develop Offline • Provision Infrastructure on Own • DevOps / Operations • No need to be involved in development or environment setup. • QA • Test Offline. • Automate Testing.
- 5. What does Vagrant do? • Simple Use Case: • Allows local development. • Offline? No Problem. • Ensures consistency. • You use a Mac, your co-worker uses Windows but you want to run your app on Ubuntu? No Problem. • Advanced Use Cases: • Allows local development of an entire “dev” or “qa” application. • Your application depends on MySQL, redis and an application developed by another group as well? No problem. Run them all locally. • Run tests. • Need to run tests quickly but don’t want to automate infrastructure? No problem. Vagrant up. Test. Vagrant destroy.
- 6. Experts at inPowered • Problem - developers loved AWS, but wanted to develop locally. • Developers couldn’t develop offline • Developers had to “share” a single Development environment • Developers were developing on OS X whereas production was Amazon Linux • Developers were struggling to access Redis/MySQL and other dependencies • Solution: • Vagrant + AWS Flow: • Develop in Vagrant, Ship to AWS! • Ubuntu 15.04 for both Vagrant and AWS • Same configuration management for both Vagrant and AWS • Vagrant: • offline development • per developer environments • all dependencies were run or “mocked” locally
- 7. Salt boto_* Module Development at Lyft • Project: • adding VPC support and testing to salt’s boto_* modules (https://github.com/saltstack/salt/) • Problems: • Needed to do testing of AWS automation (approximately 200+ tests), but building test environment load balancers, security groups, etc was taking too long as did calls • Preferred testing locally • Preferred testing with multiple Operating Systems • Solution: • Two Vagrant instances for Running Salt – Ubuntu 12.04 and 14.04 – fresh environment for every test • Moto* for mocking AWS calls (https://github.com/spulec/moto) • Tests could be run by simply typing “vagrant up” and results could be generated and reviewed
- 8. Vagrant Development vs Cloud Development • One Environment for each Developer on Cloud Platform • Administration burden – DevOps either has to Build or Automate • I used to spend 10 hours/week setting up environments for new Developers • Inefficient Spend • Connectivity and Security requirements extends to uncontrolled “Development” environments • Shared Development on Cloud Platform • Sharing? You must be kidding. • Who “owns” the server? Your feature, my configuration change and someone else’s change are all mixed together. Testing is going to be so fun! • AWS / Cloud Development does typically offer all dependencies and services.
- 9. Vagrant Development vs Cloud Development: Act I – Internet Access Required • Developer 1: • Going to be on vacation for the next week. Probably limited Internet access. • Manager: • No problem. Can you get the new “every metric for every user action feature” shipped on Monday? • Developer 1: • I’ll try, but I can’t really test my code in without Internet.
- 10. Vagrant Development vs Cloud Development: Act II – Access and Management Headaches • DevOps Engineer: • It looks like port 22 and port 80 in all our development environments are open. Google just crawled all our development sites. • Manager: • Do you know why all the ports are open? • DevOps Engineer: • The docs for the webserver project say “to ssh in or do web testing, go to the security group and open ports 22 and 80 to 0.0.0.0/0” • Manager: • Do you know why? • DevOps Engineer: • I think the Developers get tired of waiting for VPN credentials to be issued.
- 11. Vagrant Development vs Local Development • Match local and production environments • Local environment (OS X, Windows or *nix) may not match production • Faster Environment Standup • Run same configuration management as production! • Develop Offline / Complete infrastructure available • Run your app and all dependencies
- 12. Vagrant Dev vs Local Dev: Act I – Dev/Prod Inconsistency • Developer: • It worked on my machine. • DevOps: • So, it worked on OS X 10.9.2, nginx preview release 1.9.11 and MariaDB. • Developer: • Yes. I also had to install SciPy and NumPy. • DevOps: • Anything else? • Developer: • I think that is it. Make sure to use SciPy 0.17.
- 13. Vagrant Development vs Local Development Act II – Local Development Inconsistency • Developer 1: • It worked on my machine. OS X 10.9.2 with nginx 1.9.11 preview release… • Developer 2: • I’m running Windows. 10. • Developer 1: • Damn. Get a Mac. Everyone else uses them. • Developer 3: • You should use Linux. Same as production. Except for MariaDB instead of MySQL. But I don’t think that’s a problem.
- 14. Vagrant Development vs Local Development Act III – Running the Application • Developer 1: • I’ve been trying to get nginx 1.9.11 preview installed on my Mac for the last four hours • Developer 2: • Use homebrew. I found an article online somewhere. • Developer 2: • Here’s a link: http://www.robertmulley.com/tutorial/nginx-install-and-setup- mac-os-x-yosemite/ • Developer 1: • Link me for MongoDB, InfluxDB and Redis installs as well :-)
- 15. Vagrant to AWS Flow: • Standup Development in Minutes • Easy ssh Access • Easy file access • Run local mocks • No sharing environments!
- 16. Vagrant + AWS: How To • Use same OS in Vagrant as AWS • Separate and Make Modular Each “Concern” • Infrastructure Provisioning • Configuration Management • Software Deployment • Avoid or Be Aware of Platform Specific Technologies • Vagrant synced_folder = not available in AWS – you can still use for software deployment • AWS IAM Profile = not available in Vagrant/Virtualbox – you can still use for software deployment
- 17. Vagrant Setup • Infrastructure Provisioning: • Run “vagrant up” • Uses a Vagrantfile in root directory of project • Vagrantfile defines: • CPU, Memory • Operating System – use same as AWS! • A “Provisioner” – which runs Configuration Management • Configuration Management (aka “Provisioner”) • I prefer shell as “glue code” • Use same Configuration Management as AWS - Ansible, Chef, Puppet, Salt, etc. • Software Deployment • In Vagrant, use a “synced directory”– then changes made by developers are manifest in real-time near real-time
- 18. AWS Setup • Infrastructure Provisioning: • Run “cfn-create-stack” • Creates an Auto Scaling Group and Launch Configuration with user-data • Use same OS as Vagrant! • Launch Configuration calls Configuration Management • Configuration Management (aka “Provisioner”) • I prefer shell as “glue code” • Use same Configuration Management as Vagrant - Ansible, Chef, Puppet, Salt, etc. • Software Deployment • Use whatever you’d like…
- 19. Vagrant to AWS Flow - Diagram Whatever you’d like… - Ansible - Chef - Other - Puppet - SaltStack - Shell Script(s)? Just run same thing on both AWS and Vagrant! Vagrant AWS / CloudFormation Creates Instance(s) Software Deployment Infrastructure Provisioning Configuration Management Vagrant - “sync_dir” AWS Whatever you’d like: - AWS Code Deploy - Fabric - Git Push - Artifact and Sync
- 20. Demo In the next 15 minutes we will: 1. Standup local development infrastructure on Ubuntu. 1. And look at a Vagrantfile. 2. Standup identical* production infrastructure in AWS. 1. And look at a CloudFormation file. 3. Deploy code to AWS and run the same code in Dev/Prod. 1. The deployment tool is a bit of a hack, btw…