DevOps: Sprinkle Dev, Sprinkle Ops, Let's make Cake, not Mud Pies
Download as PPTX, PDF0 likes410 views
The document discusses common pitfalls and challenges in Agile development, highlighting issues such as feature inversion, the reliance on manual testing, and the failures of 'Agile Waterfall' approaches. It advocates for Continuous Delivery and DevOps as solutions to improve deployment efficiency and software quality through automated testing and integrated workflows. Case studies illustrate the transition from manual processes to automated systems, emphasizing the importance of proper pipeline management and adjustable architecture for successful implementations.
DevOps: Sprinkle Dev, Sprinkle Ops, Let's make Cake, not Mud Pies
- 1. Sprinkle Dev, Sprinkle Ops, lets make Cake not Mud Pies Brian Paulsmeyer Agile Gravy: September 29, 2016
- 2. Agile Development Pitfalls … that warn us of mistakes in our agile implementation 10/4/2016 www.centricconsulting.com 1
- 3. Waterfall in 2 Week Iterations •No ordering of stories •Gold plating of features •“Agile Waterfall” Assumptions • Learning from usage of system won’t change features • “All features will come together” in last day of last sprint • “Pause the world for 3+ years for us to finish” • “Value Proposition: All or Nothing” • Architecture can be “known” and not change • Deployments are still hard •Unfortunate ending for “Agile Waterfall” • “We ran out of money, and have nothing in production.” 10/4/2016www.centricconsulting.com 2
- 4. Constant Redefining of “Definition of Done” •Done should simply mean done •Burndown/Burnup no longer reliable •Feature inversion occurs when time spent on defects and maintenance overtakes feature work 10/4/2016www.centricconsulting.com 3 20 20 20 18 17 17 16 16 15 13 0 0 0 2 3 3 4 4 5 7 1 2 3 4 5 6 7 8 9 10 FEATURE INVERSION Features Defects and Hidden Work
- 5. Only focusing on Unit Tests and Manual Testing •Unit tests • Won’t ensure business workflows • Only allows minor refactoring • Failure: Mars Climate Orbiter (1999) • Team A: Pounds for unit of force • Team B: Newtons for unit of force (1 lb = 4.45 N) • Disintegrated in Mars atmosphere! • Each “Unit”/Component worked in isolation 10/4/2016www.centricconsulting.com 4
- 6. Continuous Delivery and DevOps … simple concept, requires diligent implementation 10/4/2016 www.centricconsulting.com 5
- 7. No more “Throw It Over the Wall” 10/4/2016www.centricconsulting.com 6
- 8. DevOps enables Continuous Delivery •Team performs activities across disciplines •Apply principles from iteration 0 through delivery 10/4/2016www.centricconsulting.com 7 Regression UAT Integration Testing DevOps – Full Lifecycle Delivery Pipeline and Container Management Local Environments Builds Unit Tests Merge Procedures Log Interpretation Dependency Mgmt Dev Deployments Traditional DEV activities Traditional OPS activities Traditional QA activities Prod Deployments Performance Mgmt Monitoring QA Deployments Branch Mgmt Change Mgmt Log Management Configurations Integration Deployments Performance testing
- 9. What It Takes Whether it be security, regulatory, software limitations, etc., there are many reasons why organizations say CD is impossible 10/4/2016www.centricconsulting.com 8 Tooling Process Culture Architecture
- 10. Basic Definitions and Levels of Maturity Continuous… 10/4/2016www.centricconsulting.com 9 Automated Deployment to ProductionDeployment Manual Compile/Build, Manual Testing, Manual DeploymentPain Automated Deployment to Integration/Staging/Pre-Prod Deployment Toolset Constant across all Environments Delivery Automated Integration/Workflow Tests Automated Component Builds Automated Component Tests* Integration Automated Unit Tests * Many teams mistakenly consider Continuous Integration (CI) complete with only automated unit tests
- 11. Pipeline •Recipe for building, deploying, and testing the system •Grouping of discrete stages for a complete workflow •Coordinates disparate technologies •Provides feedback of system state across environments 10/4/2016www.centricconsulting.com 10 Business & Customer Develop Customer Deploy to Production Deploy to Staging Deploy to QA Build and Unit Test Automated Integration / Workflow Testing
- 12. Detailed Pipeline 10/4/2016www.centricconsulting.com 11 Business & Customer Backlog Consolidate Features and Hotfixes Iteration Planning Build Server“X” Stories Development 1 Active Story Per Developer 1 Commit Acceptance Test Customer Build Artifact Acceptance Test Environment QA Testing QA Environment Auto-Trigger Pipeline: Installs To AIT Environment Executes Smoke and Acceptance Tests (<10 min) Manual Trigger Pipeline: Installs To QA Environment Executes Smoke Tests UAT Testing Manual Trigger Pipeline: UAT Environment UAT Smoke & UAT Tests Production Manual Trigger Pipeline: Prod Environment Prod Smoke Tests UAT Environment
- 14. Pipeline Workflow Tools … to speed implementation 10/4/2016 www.centricconsulting.com 13
- 15. Pipeline Stage Implementation •Pipeline workflow steps trigger external tools 10/4/2016www.centricconsulting.com 14 Business & Customer Develop Customer Deploy to Production Deploy to Staging Deploy to QA Build and Unit Test Java .Net TypeScript PHP etc. Ant Maven MSBuild Make etc. JUnit Nunit etc. Bash PowerShell Chef/Puppet VMs Docker/CoreOS etc. Automated Integration / Workflow Testing Selenium Cucumber JBehave SpecFlow Codeception etc. Same Tools as Deploy to QA Only Configuration Differences Same Tools as Deploy to QA Only Configuration Differences Build Tool Jobs (ex. Jenkins)
- 16. Key Pipeline Coordination Tool Features •Pipeline Triggers • Automatic on code check-ins to source control • Manual • Programmatic API •Workflow • Parallel steps • Sequential steps • Failure-only steps • Graphical view of steps •Artifact Tracking • Traceability from check-in to final deployment • View of deployed version in each environment • History of deployments • Ability to trigger updates to agile tracking tools (ex. JIRA) •Environment Support • Same deploy scripts, allows configuration changes per environment 10/4/2016www.centricconsulting.com 15
- 17. Pipeline Workflow Tool : Jenkins •On-Site •Version 2.0 Provides Workflow View •Software Language Agnostic 10/4/2016www.centricconsulting.com 16
- 18. Pipeline Workflow Tool : ThoughtWorks Go •On-Site •Software Language Agnostic 10/4/2016www.centricconsulting.com 17
- 19. Pipeline Workflow Tool : Octopus Deploy •.Net 10/4/2016www.centricconsulting.com 18
- 20. Automated Testing … so every day can be deployment day 10/4/2016 www.centricconsulting.com 19
- 21. Automated Component/Integration/Workflow Testing •Opinions of why automated testing isn’t needed • “Manual testing is sufficient” • “Business only wants to spend money on features” • “Developers only want to code features” • “Our <insert technology> is too hard to test with automation” • “Automated tests run too slow” 10/4/2016www.centricconsulting.com 20 Best Case: Missed Deadline Worst Case: Catastrophic Failure in Production Result of skipping automated tests
- 22. Selenium WebDriver Direct Test using Selenium WebDriver WebDriver driver = new FirefoxDriver(); // Navigate to webpage driver.get("http://www.google.com"); // Find the Search Field (UI Locator) WebElement element = driver.findElement(By.name(“Search")); // Fill in the data element.sendKeys(“something interesting"); // Now submit the form. WebDriver will find the form that contains the element element.submit(); // Assert the test passed 10/4/2016www.centricconsulting.com 21 Selenium WebDriver Web Page Test Test Test Test
- 23. Page Object Model 10/4/2016www.centricconsulting.com 22 • Separate Test Code and UI Locators • Supports Don’t Repeat Yourself (DRY) for Element Locators • Less Technical Test Case SearchPage page = new SearchPage(selenium); // Enter Search information page.search(“something interesting"); // Now submit the form page.submit(); // Assert the test passed Page Object Model(s) Web Page Test Test Test Test Selenium WebDriver ServerAPI
- 24. BDD Testing • Gherkin (English) Text File • Step Definitions Map Between English Test and System • Business Readable Test Case Given user searches for “something interesting” When user performs the Search Then assert something is true 10/4/2016www.centricconsulting.com 23 Page Object Model(s) Web Page Step Definition Step Definition Step Definition Selenium WebDriver ServerAPITest Test Test Test
- 25. Capabilities … we need to be successful 10/4/2016 www.centricconsulting.com 24
- 26. Rapid Response to Changing Business •Need to be able to react to disruptive technology and business models •Stagnant architecture limits new technology adoption • Changes blocked by fear of breaking system • Inability to deploy new systems to production • Unable to adopt new security upgrades • Unable to adopt new software versions 10/4/2016www.centricconsulting.com 25
- 27. Drive Defects From Most Used Features “You can’t test quality in” At a minimum make sure defects aren’t in the standard user workflows 10/4/2016www.centricconsulting.com 26
- 28. Adjustable/Changeable Architecture •Magic Quadrant Architecture • “Best” choice today ≠ Best choice tomorrow • Failed technologies and companies in all parts of magic quadrant •Natural product End-of-Life requires new architecture •Continuous Delivery and DevOps allows fluid architecture changes 10/4/2016www.centricconsulting.com 27 (Object-Oriented Database)
- 29. Cadence •What if everyday was a normal day •What if those normal days were release days •What if those release days were multiple releases throughout the day •Low Risk On-Demand Releases 10/4/2016www.centricconsulting.com 28
- 30. Deployment Styles … for production deployment 10/4/2016 www.centricconsulting.com 29
- 31. Standard Deployment •Deployment Steps • Disable user access • Upgrade servers from V1 to V2 • Test deployment • Enable user access • Upgrade destabilizes running environment •Problem Mitigation • Re-Enable backup if possible • Developer troubleshooting requires customer downtime 10/4/2016www.centricconsulting.com 30
- 32. Blue Green Router / Load Balancer Blue-Green Deployment •Two Identical Production Environments • Blue • Green •One Active: Router Controlled •Deployment Steps (Green Active) • Deploy to Blue environment – No users affected • Perform smoke tests on Blue environment • Optional: Switch subset of users to Blue (similar to Canary releases) • Switch all users to Blue environment •Problem Mitigation • Don’t switch to 2nd environment • Developer troubleshooting has no customer impact 10/4/2016www.centricconsulting.com 31
- 33. Canary Router / Load Balancer Canary Deployment •One Production Environment •Deployment Steps • Deploy to 1 server (Canary) Optional: Provision and build new server from scratch • Router sends portion of users to Canary • Random • Internal users only • User-type based (ex. Test users) • Gradually upgrade servers to V2 • Gradually switch users to V2 servers •Problem Mitigation • Don’t switch additional users to Canary environment • Developer troubleshooting has no customer impact 10/4/2016www.centricconsulting.com 32
- 34. Case Studies … that show it can be done
- 35. Before Build Automation •23 Teams • 150 Team Members (Developers, Quality Engineers, Scrum Masters) • 100 Developers •Service Oriented Architecture • Managers, Engines, Data Access •Baseline End of PI6 (SAFe) • PI6 demo system setup • Two weeks to setup • Manual Installations – not repeatable • Individual Teams created “magic instructions” in isolation to install and run solution • Some teams unable to run full solution • Custom data loaders used to skip workflow steps • Developers unable to test full workflow • Movement of data through workflow steps untested till end of PI • Update of component versions caused “Day of Downtime” for developers to fix unexpected issues 10/4/2016www.centricconsulting.com 34
- 36. After Build Automation: Jenkins Easy Button Builds • Starting Point: a machine with Windows — that’s it! • The rest is automated: − Provisioning: verify/install tools takes ~50 minutes − Puppet Agent, PowerShell, 7-Zip, VNC − SQL Server, .Net, Visual Studio − Noticia Repono Client − Install Custom Software takes 5-10 minutes − Clean system from any previous runs − Install/Configure Base Components − Install System − Install and Run Smoke Tests At this point the system is ready to be used to do further testing, demos, etc. • Deploy cycle repeats ChunkyMonkey Vanilla
- 37. Definition of Done … that makes our customers happy 10/4/2016 www.centricconsulting.com 36
- 38. Continuous Delivery Definition of Done •Automated Deployments • Fully Automatic or Push-Button Deploys • Same Executable Scripts for all Environments • No Deployment Hero •Automated Testing • Unit • Integration • Full System •Always Shippable • No Temporary Code • No Untested Code • Feature Toggles •One Backlog • Production Bug Fixes • New Features 10/4/2016www.centricconsulting.com 37
- 39. About Me 10/4/2016www.centricconsulting.com 38 • Our highest priority is building lifelong relationships with clients based on trust, respect, and collaboration. • We invest in our talented team and support their well being by keeping them challenged and inspired. • Our localized company structure allows us to play very active roles in the lives of our families and community. This relationship-centric focus keeps us passionate, committed and motivated in all facets of our lives. That’s why our team is here to stay. Centric Consulting • 17 years as a software developer and architect • Full stack developer in Java, .Net, TypeScript (Angular 2), and PHP • DevOps tooling experience including VMs, Docker, Chef, and Puppet • Continuous Delivery and Automated Testing implementations • Experience across industries including FDA and SEC regulated Continue the Journey • http://centricconsulting.com/centric-st-louis-scaled-agile-workshop/ Thank You for Attending!
Editor's Notes
- #23: Selenium WebDriver is used to manipulate HTML Web Pages by providing the ability to find and manipulate elements on an HTML page. As seen in this example the driver uses a findElement method to identify the element on the HTML page. It is important to add ids into the HTML that can be used to simplify finding the HTML elements on the web page. By making minor changes in the development of the web page the page becomes much simpler to automate with Selenium. With this single test a small change to the web page would be managed by changing the findElement By Name string. With a small number of tests this can be managed without adding a large cost when a UI change occurs. However once hundreds and even thousands of tests are created with findElement searches spread throughout the tests the cost to make a single change to the UI causes a ripple effect in the tests. If the test suites are coded at this “low-level” then a single UI change becomes extremely expensive and the tests become brittle and hard to maintain. In addition “the language of the test” is technical. It is difficult to show the test to a business owner, project manager, or even technical people because the test objective is lost within the technical web driver details.
- #24: The Page Object Model pattern can be used to eliminate the duplication of the Find Element locators. In addition the pattern greatly simplifies the test code. However the test is still actual compile-able code. The pattern can also be used to interact with APIs and other parts of the system in addition to directly interacting with the WebDriver.
- #25: Behavior Driven Design can be used to test web sites, APIs, as well as other systems. The test cases are written in a language that matches the business. The underlying BDD engine (ex. Cucumber) is used to translate the English text into actions against the system. This allows the Gherkin tests to interact with the system as needed without adding complexity to the actual test case. BDD provides an explicit separation between the test cases and the actual implementation of the system. By writing the tests with a business-centered focus the tests are more stable than if written directly against the UI implementation elements.
- #31: Down-time “We’ll Be Back” Landing Page Zero Downtime Blue-Green Canary Releases