Massive Continuous Integration and Light-speed Iterations
- 1. BT3 Concurrent Session 11/8/2012 10:15 AM "Massive Continuous Integration and Light-speed Iterations" Presented by: Jesse Dowdle AtTask, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Jesse Dowdle AtTask, Inc. Senior manager of development for AtTask Inc. Jesse Dowdle directs the engineering efforts of nearly a dozen agile teams in the U.S. and overseas. Successful engineering in a fast-paced industry requires constant research in emerging technologies. Jesse drives thought leadership on continuous delivery, development best practices, and process evolution to help AtTask ship software at lightning speed. Happiest when he’s building great software, Jesse sees technology, process, and people as equally important prerequisites for success. A native of Utah, where he enjoys fast cars in the summer and snowboarding in winter, Jesse tweets at @dowdlemj and blogs.
- 3. dowdlemj@attask.com Massive Continuous Integration and Light-speed Iterations •Jesse Dowdle, Director of Development •@dowdlemj •linkedin.com/in/jessedowdle •www.continuousdeliverist.com
- 4. •Salt Lake City, Utah •Software •Project as a Service Management •Collaborative •Salt Work Management Lake City, Utah •Software •Project as a Service Management •Collaborative Work Management
- 5. •Salt Lake City, Utah •Software •Project as a Service Management •Collaborative Work Management Agile at AtTask 5 years Wall to wall 11 Dev Teams
- 6. How we did Scrum Sprint Sprint Sprint How we did Scrum 4 WEEKS Sprint Sprint Sprint
- 7. How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Sprint Sprint How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Sprint Public Release Sprint Public Release Public Release
- 8. How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Internal Internal Sprint Internal Public Release Internal Internal Sprint Internal Public Release Internal Internal Internal Public Release Internal Public Release How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Internal Internal Sprint Internal Public Release Internal Internal Sprint Internal Public Release Internal Internal
- 9. Then Scripted scenarios Slow, brittle Monthly Releases 3-5 days acceptance Manual administration Now Tight unit-like UI tests Fast, Robust Daily releases 30-45 minutes Sign-off and go
- 10. What is Continuous Integration? “Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.” -- Martin Fowler Components •Version •Build •Test Control System Server Runner •Test Suite •Notification Mechanism
- 11. Basic Integration •Version •Build •Test Control triggers new action Server retrieves latest version, compiles, starts Application Stack Runner executes tests against Application Stack •Results •Build are collected and sent by notification Server is cleaned and restored. Basic Integration
- 12. BASIC INTEGRATION BUILD SERVER VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
- 13. BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
- 14. BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION BUILD SERVER VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
- 15. Massive Test Automation •800 Unit tests •3300 API tests •2000 UI tests •55,000 hours per month •Release Acceptance + CI “True” Continuous Integration •How often do I integrate? •What tests tell me I’m integrated? •What must I know to release?
- 16. Pipeline Priorities •Performance (Cycle Time) •Scalability •Visibility •Accountability ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
- 17. Commit ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Commit Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
- 18. Selenium Grid Grid + AtTask Commit Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Selenium Grid Grid + AtTask Commit Install Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
- 19. Selenium Grid Grid + AtTask Commit Install Build Installer Build Tests Unit Integration 1/4 3/4 2/4 4/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 2/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY 3/4 4/4
- 20. Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY HOW TO MAKE A PAPER AIRPLANE OR, QUEUEING THEORY AS EXPRESSED BY A FOLDED PROJECTILE Tear Down
- 21. Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Tear Down Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 SERIAL VS PARALLEL Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 2/4 SERIAL VS PARALLEL 3/4 4/4 Tear Down
- 23. CYCLE TIME CI Vendors CruiseControl Jenkins (Hudson) TeamCity Go Team Foundation Server
- 24. Cloud Vendors Amazon Web Services OpenStack SauceLabs JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN
- 25. JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN JENKINS DYNAMIC SLAVE ALLOCATION
- 26. JENKINS DYNAMIC SLAVE ALLOCATION SLAVE JENKINS DYNAMIC SLAVE ALLOCATION
- 27. SLAVE SLAVE SLAVE SLAVE JENKINS DYNAMIC SLAVE ALLOCATION JENKINS DYNAMIC SLAVE ALLOCATION
- 28. Scalability •Taking •To the Test Suite to massive scale the cloud with Amazon Cloud Formations •Dynamic Selenium Grids •Dynamic AtTask Environments Divide and Conquer •Module separation will allow for scale •Test Suites should support sharding •Run tests with different dependencies in parallel
- 29. Divide and Conquer Selenium Grid Selenium IE Grid + AtTask Commit Selenium FF Install Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 Divide and Conquer Unit Integration 1/4 3/4 2/4 4/4 Tear Down
- 30. PARALLEL BY DEPENDENCIES In Memory Database Email PARALLEL BY DEPENDENCIES UI Only Customer Data
- 31. In Memory Database Email UI Only Customer Data PARALLEL BY DEPENDENCIES From 3 days to 30 minutes with Selenium 2
- 32. SELENIUM 1 (1800 TESTS) SOMETIMES THE TESTS NEED TO CHANGE SELENIUM 1 (1800 TESTS) 4 HOURS SLOW SOMETIMES THE TESTS NEED TO CHANGE
- 33. TRANSITIONING TO SELENIUM 2 TRANSITIONING TO SELENIUM 2
- 34. TRANSITIONING TO SELENIUM 2 SELENIUM 1 (1800 TESTS) 4 HOURS TRANSITIONING TO SELENIUM 2
- 35. SELENIUM 1 (1800 TESTS) 4 HOURS SELENIUM 2 (750 TESTS) 30 MINUTES TRANSITIONING TO SELENIUM 2 TRANSITIONING TO SELENIUM 2 WHERE DID THE TESTS GO?
- 36. 2.1 MINUTES TRANSITIONING TO SELENIUM 2 2.1 MINUTES TRANSITIONING TO SELENIUM 2 34 SECONDS
- 37. 2.1 MINUTES 34 SECONDS WITH GRID... TRANSITIONING TO SELENIUM 2 2.1 MINUTES 34 SECONDS WITH GRID... 8 SECONDS TRANSITIONING TO SELENIUM 2
- 38. 2.1 MINUTES 34 SECONDS WITH GRID... 8 SECONDS 2.4 SECONDS TRANSITIONING TO SELENIUM 2 JSON PARTLY CLOUDY WITH A CHANCE OF RAIN
- 39. JSON PARTLY CLOUDY WITH A CHANCE OF RAIN Scalability Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 Tear Down
- 40. Scalability Selenium Grid Grid + AtTask Commit Build Installer Build Tests SLAVE IT’S RAINING GRIDS
- 41. SLAVE IT’S RAINING GRIDS SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS
- 42. SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS
- 43. SLAVE SELENIUM GRID SLAVE SLAVE SELENIUM GRID SLAVE SLAVE IT’S RAINING GRIDS Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
- 44. Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
- 45. Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
- 46. Accountability •Claim / Blame test results using AtTask Accountability •Claim / Blame test results using AtTask
- 47. Accountability •Claim / Blame test results using AtTask CI as a Platform •On-Demand •Easy demo environments button for version control •Visibility •Other into production systems quality measurements •Static Analysis •Code Coverage
- 48. Staffing for Massive CI •Get the right skill set •Watch your integration points •Technological •Organizational •DevOps Adoption •Process / Tool alignment •Information •Integration •Scrum Radiators with existing tools board •Backlog •Integration with Release Management
- 49. CI EVOLUTION ITS A CONTINUOUS INVESTMENT What does it cost? •3 FTE •Two human, one machine(s) •Keeping •Efficient costs down machine sizes •Rock-solid •Elasticity tear down
- 50. Return on investment •5 days (40 hours) to certify one release... •10 engineers •$40 per hour •$20,000 •30 minutes to certify one release... •10 engineers •$40 per hour •$250 To release 12 times in a year... $240,000 TO CERTIFY $3000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI To release 24 times in a year... $480,000 TO CERTIFY $6000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI To release 100 times in a year... $2,000,000 TO CERTIFY $25000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI
- 51. 100 deployments annually $2,000,000 100 deployments annually $265,000 $2,000,000
- 52. 100 deployments annually $265,000 Where to go from here •Continuous Delivery •Application Lifecycle Management •Continuous Integration of Infrastructure •Open Source
- 53. Tools Manifest •AtTask •EC2 - contributions... wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin •CloudFormation •Pipeline View - •Description •GIT - - wiki.jenkins-ci.org/display/JENKINS/AWS+Cloudformation+Plugin github.com/JoelJ/PipelineDashboard Setter - github.com/JoelJ/DescriptionSetter /wiki.jenkins-ci.org/display/JENKINS/Git+Plugin •AtTask for Jenkins - Coming soon... Further Reading •“Continuous Integration” -- Martin Fowler - http://www.martinfowler.com/ articles/continuousIntegration.html •Extreme Programming Explained -- Kent Beck - http://amzn.to/c0VvTB •“Continuous Deployment in 5 Easy Steps” -- Eric Ries - http:// radar.oreilly.com/2009/03/continuous-deployment-5-eas.html •“Four Principles of Low Risk Software Releases” -- Jez Humble - http:// www.informit.com/articles/article.aspx?p=1833567 •“Continuous Delivery” -- Jez Humble & David Farley - http:// continuousdelivery.com/ •The Lean Startup -- Eric Ries - http://theleanstartup.com/ •Continuous Integration -- Paul Duvall - http://amzn.to/9Zx1l1
- 54. Thank you! •Jesse Dowdle, Sr Manager of Development •@dowdlemj •linkedin.com/in/jessedowdle •www.continuousdeliverist.com •David Tolley, Sr QA Automation Engineer •@davidtolley •linkedin.com/profile/view?id=81007074