![Same Git experience
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
Cloning into 'aws-cli'...
Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (9900/9900), done.
Checking connectivity... done.
$ nano README.rst
$ git commit -am 'updated README'
[master 4fa0318] updated README
1 file changed, 1 insertion(+)
$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote:
To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
4dacd6d..4fa0318 master -> master](https://image.slidesharecdn.com/devopscontinuousintegrationanddeploymentonaws-puttingmoneybackintoyourmission-150409213728-conversio-180506140355/85/Devops-continuousintegration-and-deployment-onaws-puttingmoneybackintoyourmission-150409213728-conversion-gate01-12-320.jpg)
!["AWS::CloudFormation::Init" : { "config" : {
"packages" : {
"yum" : {
"mysql" : [],
"mysql-server" : [],
"httpd" : [],
"php" : [],
"php-mysql" : []
}
},
"sources" : {
"/var/www/html" :
"https://s3.amazonaws.com/my-builds/build-v4.zip"
}
}](https://image.slidesharecdn.com/devopscontinuousintegrationanddeploymentonaws-puttingmoneybackintoyourmission-150409213728-conversio-180506140355/85/Devops-continuousintegration-and-deployment-onaws-puttingmoneybackintoyourmission-150409213728-conversion-gate01-32-320.jpg)
Devops continuousintegration and deployment onaws puttingmoneybackintoyourmission-150409213728-conversion-gate01
- 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved DevOps, Continuous Integration & Deployment on AWS Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh Dave Beck, Technical Director, National Novel Writing Month @DaveB_NaNoWriMo
- 2. DevOps
- 3. What is DevOps? • « DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support » - theagileadmin.com
- 5. What is Continuous Integration? • Changes to code automatically deployed to mainline branch – After passing unit and mock tests • Makes changes to code, and deployments iterative, not monolithic • Bugs are detected quickly • Helps automate deployments • Allows rapid development and deployment
- 6. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER DEVELOPER PICK TASKS SUBMIT CODE SCHEDULE BUILD RECURRENT BUILDS CODE FETCHCODE QUALITY TESTS TEST RESULTS BUILD OUTPUT DOCS BINARIES & PACKAGES DEV FACING NOTIFICATIONS CLOUDFORMATION AMIS or CONTAINERS
- 9. AWS code services AWS CodeCommit Coming soon AWS CodePipeline Coming soon AWS CodeDeploy Launched Nov 2014
- 10. Cloud software development lifecycle 10/13/14 10 MonitorProvisionDeployTestBuildCode AWS Elastic Beanstalk AWS OpsWorks Amazon CloudWatch AWS CloudFormation ?
- 11. Source Control Private Git repositories hosted on Amazon S3 • Full Git compatibility (use with existing tools) • All the benefits of the cloud (scalable, durable, reliable, low pay as you go pricing) • No size limits on repositories (store binary files) • Online code tools with browse, edit, diff 10/13/14 11
- 12. Same Git experience $ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli Cloning into 'aws-cli'... Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done. Resolving deltas: 100% (9900/9900), done. Checking connectivity... done. $ nano README.rst $ git commit -am 'updated README' [master 4fa0318] updated README 1 file changed, 1 insertion(+) $ git push Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli 4dacd6d..4fa0318 master -> master
- 13. Continuous Delivery Customizable release automation, with integrated build and test • Model and visualize custom release workflow (source build beta gamma prod) • Automate builds, tests, and deployments • Enforce custom rules, approvals, and gates • Integrate with third-party and custom tools 10/13/14 13
- 15. Deployment Coordinate software updates to fleets of EC2 instances • Rolling updates for no downtime • Deployment health checks and easy rollback • Auto Scaling integration • Works with any application • Reuse existing setup tools (Bash, Powershell, Chef, Puppet…) 10/13/14 15
- 16. 9/25/14 Slides not intended for 16
- 17. Cloud software development lifecycle 10/13/14 17 MonitorProvisionDeployTestBuildCode AWS Elastic Beanstalk AWS OpsWorks CloudWatchCloudFormationCodeDeploy CodeCommit CodePipeline
- 19. PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES
- 20. ON-DEMAND PAY AS YOU GO ELASTIC
- 24. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
- 25. A lot of options… • Configuration Management Systems – Puppet – Chef – Saltstack • Deployment Frameworks – CodeDeploy – AWS Elastic Beanstalk – AWS OpsWorks – Ansible – Fabric – Capistrano • Infrastructure Management – CloudFormation • Containers – Amazon EC2 Container Service
- 26. Bake an Image Configure dynamically Time consuming configuration (startup time) Static configurations (less change management) Bootstrapping
- 27. Bake an Image Configure dynamically Continuous deployment (latest code) Environment specific (dev- test-prod) Bootstrapping
- 28. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved AWS CLOUDFORMATION STACK-BASED DEPLOYMENT SERVICE
- 31. { "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }
- 32. "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }
- 33. { "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }
- 34. CLOUDFORMATION TEMPLATE PROCEDURAL DEFINITION Create it programmatically KNOWN CONFIGURATION Store stack configuration in source control PARAMETER DRIVEN Dynamic and user-driven templates COLLABORATION Share templates with ease as just files
- 38. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION.
- 39. KEY = ITERATION
- 40. ITERATION = MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
- 41. 11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean number of hosts simultaneously receiving a deployment 30,000 Max number of hosts simultaneously receiving a deployment DEPLOYMENTS AT AMAZON.COM (in 2011)
- 44. METRICS @ETSY
- 45. METRICS @OBAMA FOR AMERICA
- 46. Metrics and Monitoring Options CloudWatch … and many more
- 50. INNOVATE
- 51. SPEED AND AGILITY Experiment Often Fail quickly at a low cost More Innovation Experiment Infrequently Failure is expensive Less Innovation “ON-PREMISE”
- 52. At a glance What is NaNoWriMo? • National • Novel • Writing • Month We are a community of novelists. Every November, hundreds of thousands of writers gather virtually and in-person to write 50,000 words each. (The Great Gatsby, by comparison, is 47,094 words.) Last year, 180,000 started new novels, and 40,000 finished.
- 53. It all started in 1999 Our founder, Chris Baty, challenged 20 over-caffeinated friends to write 50,000 words each in a single month. Six of them finished.
- 54. Explosive growth, and no plan to manage it
- 55. NaNoWriMo now NaNoWriMo Main: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6 Camp NaNoWriMo: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6 The Young Writers Program: Drupal 5, MySQL 5.6 NaNoWriMo Store: Drupal 5, MySQL 5.6
- 56. November in detail • Nov 1: ~500k sessions as people begin • All month: ~50k forum posts per day • Nov 30: Last chance to post word counts
- 57. Annual Site Crash • Every year from 2003 to 2011 the site would crash on November 1. • In 2012 we migrated the sites to Amazon Web Services. • The result: No more outages! • Naturally, this is how our participants reacted…
- 58. Outrage! They wanted the site to crash; it was considered a rite of passage. Oh well.
- 59. Keeping NaNoWriMo Online • Proxy servers (Varnish) on Debian 7.5 EC2 instances • Ruby on Rails on EC2 (Nginx) • Worker servers running Resque on EC2 • Support servers running Resque, Redis, Sphinx • MySQL on RDS instances • Domains and subdomains on Route 53 • Image assets on S3 instances via EBS • Deployment via Capistrano, Puppet
- 60. The Year in Instances
- 61. Next Steps: Goals To increase uptime for all services To make server management less time-consuming To reliably and quickly respond to user-driven events To reduce EC2 costs through autoscaling To create a faster and more dynamic email system To improve site search
- 62. Next Steps: Goodbye Drupal What: Porting Drupal sites to Ruby 2.2, Rails 4.2 Why: The old Drupal-based sites are slow and use more than their share of server resources.
- 63. Next Steps: Emailing via Amazon SES What: Migrating outgoing emails from PHPlist/Sendmail to SES Why: Event-driven emails on the sites are backgrounded to support servers, which has proved slow and unreliable. Mass emails rely on an old and very sluggish application which has limited layout tools and metrics.
- 64. Next Steps: Load Balancing What: Converting proxy servers to Elastic Load Balancers Why: We will benefit from better AWS dashboard tools, metrics, and notifications. It is part of our autoscaling strategy.
- 65. Next Steps: CodeDeploy What: Using CodeDeploy for deployment process Why: Currently deployment relies on many custom shell scripts and tends to be unreliable. We intend to move to stateless instance spin-ups as part of our autoscaling strategy.
- 66. Next Steps: Autoscaling What: Implement an Auto Scaling process Why: In the past we’ve had to predict our server needs ahead of time. If we underestimated, the sites would crash; if we overestimated, we weren’t using our small budget effectively. And the planning process consumed too much staff time.
- 67. Next Steps: Amazon SNS Notifications What: Increased use of Amazon SNS service Why: Last year SNS event messaging helped us respond to emerging crises very quickly. With our expanded AWS presence, we will link event notifications to all of the services we use.
- 68. Next Steps: On the Horizon What: Amazon CloudSearch; AWS Lambda Why: We are interested in easy-to-manage alternatives to Sphinx – CloudSearch is one option. Lambda looks intriguing for a variety of possible applications.
- 69. Potential future architecture Internet Load Balancer ELB EC2 instances Code servers EC2 instances Databases RDS DB instances Search CloudSearch instance Outgoing email Amazon SES service
- 70. SAN FRANCISCO