How to Solve Problems of Testing in Agile Environments
1 like1,841 views
The document discusses challenges with testing in agile environments and proposes solutions. It notes that while companies have adopted agile frameworks, testing is still not optimized and contributes to slowdowns. Poor test case design and provision of low quality test data means test teams cannot keep up with the rate of change while fully testing systems. The document advocates shifting testing left in the development cycle by using an end-to-end, requirements-driven approach to testing to deliver higher quality software faster and at lower cost.
How to Solve Problems of Testing in Agile Environments
- 1. World® ’16 How to Solve Problems of Testing in Agile Environments Jonathon Wright – Director of Digital Assurance – CA Technologies Huw Price – VP of Application Delivery – CA Technologies DO5X29E DEVOPS
- 2. 2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD For Informational Purposes Only Terms of this Presentation © 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
- 7. 7 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Cognitive Adoption – ‘Evolution, Over Revolution’ 1 – “The Battle for Competitive Advantage in the App Economy”, Oxford Economics, 2015 VALUE-DRIVEN TIME 1960s 1990s TODAY of enterprises believe that they need value-driven algorithmic business models to respond to disruption in the next 3 years. 78% Legacy is your Legacy (API) Digitalization of your Core (SOA) Digital (Cognitive Adoption)
- 8. 8 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD 78% of Executives in most B2C industries say their business will be disrupted by digital technologies in the next 12 months. 20% say their business will be massively disrupted. - Forrester State of Digital Business - Big Bang Disruptions - Larry Downes - RallyON Keynote
- 10. 10 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Our Focus… Dramatically Improve Testing in Digital Better. Value. Smarter. *National Institute of Standards & Technology $60B $2.5BPer Year of IT Budget for QA and Testing* Per Year on Testing Tools How About Focusing on Minimum Viable Quality?
- 11. 11 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Limited test resources and lack of Automation of all testing is still manual70% Source: Bloor Research, 2014 Test Data for setup & staging of time spent looking for50% Source: Bloor Research, 2014 Agile Requirements Designer, TDM and RA Build of various dev & testing Environments of critical dependencies are unavailable when dev and test teams need them 56% Source: voke Market Snapshot TM Report: Service Virtualization – January 2015 Is Testing still the Bottleneck? Digital Testing
- 18. 18 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD INNOVATION (INNOVATION, GROWTH & MATURITY) FAST INNOVATION SLOW INNOVATION DECOMPOSITIONLIBERATIONIMMUNIZATION Incubate Scale Enhance WHAT IS HAPPENING NOW? Ecologist DevScrum Master Therapist Researcher Designer DevOps Engineer Disciplined Agilist Maker Creative Infrastructure Engineer SDET BA Tester Product Owner PM Ops Support FAST FEEDBACK SLOW FEEDBACK WHAT HAPPENED ?WHAT’S HAPPENING NOW? WHAT WILL HAPPEN ? WHAT COULD HAPPEN ?
- 23. 23 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD COGNITIVE ADAPTIVE ADOPTION Manual Continuous Autonomics Lean Engineering Connected Intelligence BUSINESS VALUE LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 ? Optimized Organization for DesignOps Continuous Insight & Learning Value Stream Mapping Consistency & Collaboration Heroes & Heavy Lifting DIGITAL MATURITY § Insight-Driven (Predictive / Prescriptive) § NoOps organized (IoT-Ops/CX-Ops/Sec-Ops) § Self-healing end-to-end autonomic orchestration § Microcontainerization & Microservices enabled § Open Innovation & Connected Intelligence § Quantum Teleportation (Shift X) Where are you? Minimum Viable Quality § Silos, manual handovers, waterfall process § One release/year § Monolithic apps § Long term project/resource planning § Error prone dev/test/release processes Minimum Viable Product Minimum Viable Delivery Organization Change Methods and Skills Insight Beliefs, Values, Culture Value Behaviors Heuristics
- 24. 24 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Connected Intelligence Release Management Environment Management Application Management Network Management Infrastructure Management Repository Management Data Management Test Data LearnProbe CA App Experience Analytics Connected Intelligence Cognitive Adaptive Insight (CAI)
- 25. 25 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS CONTINUOUS DELIVERY MAINFRAME SECURITY API analyticsRelease management dashboard Access incident responseReal-time app performance Connected Intelligence Cognitive Adaptive Insight (CAI)
- 26. 26 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Design QA/TESTDEV PRODUCTIONPRE-PROD Specification CI/Build Functional testing UAT Integration testing Performance engineering Deploy to pre-prod Code commit SCM Design Spec Requirements Business Users TEST Data TEST Stub TEST Data User BA Product Manager Customer Users Customer Experience Customer Experience Customer Experience Customer Experience Design Business Value Business Value Business Value Improvement Innovation Intelligence Assessment Insight Learning Maintenance Operations Delivery Testing Support Customer Digital Assurance Value Insight Learn Learn Monitor Monitor TEST Stub REAL User s TEST Data REAL Data TEST Stub USER Case Business idea Value-Driven Delivery
- 27. 27 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Digital Assurance Pocketbook Paul Gerrard NEW exclusive CA sponsored Pocketbook authored by Paul Gerrard, eminent Test- Transformation consultant, thought leader, teacher and international conference speaker. Pre-order your FREE copy today from http://bit.ly/digital-assurance
- 29. 29 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Step 1 – Continuous Adaptive Testing In-Sprint everything Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint 3 Sprint N Sprint 3 Sprint N User Stories Dev and Manual Testing Automated Regression Testing 3 weeks 3 weeks 3 weeks 3 weeks Sprint 1 Sprint 1 Sprint 2 Release Release Release Release Highlights § No Sprint-lag effect § Release in 3 weeks § In-Sprint Automation
- 30. 30 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD So How to Achieve Speed, Quality and Coverage? Pick an existing application and model requirements for 3 Sprints Write Code and Test Auto Regression Release Model Code and Test Auto Regression Release Model Code and Test Auto Regression Release Model Code and Test Auto Regression Release User Stories Dev and Exploratory Manual Testing Automated Regression Testing 3 weeks 2 weeks 2 weeks 2 weeks Automated Release Key Points § Sprint Acceleration § Do not go Big-Bang § Start small, see results, then scale § Scale gradually in the enterprise
- 31. 31 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Step 2 – Lean Engineering Pick an existing application and model requirements for 3 Sprints Model Code and Test Auto Regression Release User Stories Dev and Exploratory Manual Testing Automated Regression Testing 2 weeks Automated Release Sprint 1 Read User Story and create basic Model Automatically generate basic manual tests Export them back into your repository (ALM, Rally, JIRA, etc.) Run test cycle as usual Sprint 2 Read User Story and Create Model with Negative and Alternate Scenarios Automatically generate manual tests with optimal coverage Export them back into your repository (ALM, Rally, JIRA, etc.) Run test cycle as usual Sprint 3 User Story is directly Modeled. Model is single source or truth Automatically generate manual and auto tests with optimal coverage Automatically perform impact analysis on any changes Run test cycle in accelerated mode
- 32. 32 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Scaling Modeling Across the Enterprise Example of a Roadmap across BUs Business Unit Project M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 BU 01 Project 1 Transition to Modeling completed after 3 Sprints and 1 extra month of results/benefits analysis, review and final BU and PMO approval Project 2 Modeling adoption completed in 2 Sprints Project 3 Modeling adoption completed in 1 Sprint BU 02 Project 1 Transition to Modeling completed, results/benefits analyzed, reviewed and approved by end of Sprint 3 Project 2 Modeling adoption completed still in 3 Sprints Project 3
- 35. 35 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Cognitive Adaptive Testing Existing Manual Test Cases Dynamic Optimized – Tests Agile Requirements Designer Unstructured Partial coverage Test bloat Low effectiveness Analog – difficult to maintain Cognitive Adaptive - Tests based on Insight from Connected Intelligence New Requirements and Business Processes Fewest test steps (reduce by > 30%) Greater coverage Higher effectiveness Fully digitized – easy to maintain Continuous optimization Highest effectiveness Reduction > 50% Release OperationsDesign Development
- 36. 36 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Modeling Maturity AMBIVALENT LEVEL 1 INITIALIZING LEVEL 2 AWARE LEVEL 3 MINDFUL LEVEL 4 EXPERT LEVEL 5 § Test data is copied from production § Extensive use of End to End testing § Test data is created manually by testers § PII in test data, risking breach of security Challenges of Level 1 stay with following improvements: § Some databases are subsetted for testing purposed to speed testing and reduce storage costs § PII is masked in SOME test databases § PII removed from test data bases Challenges of Level 2 reduce with following definitive steps: § Test data is created using a tool § PII removed from test data bases subsets § Test data is explicitly drawn out in test plans § TDM is recognized as a skill Level 3 processes get matured: § Test data can be reserved § Test Data coverage is understood and known § The right tests are run, resulting in complete testing § Test Data on Demand is widespread. § Centralized TDM team Everything in Level 4 + § Enterprise level Modeling policy and strategy fully adopted by the organization § Common Enterprise Model Repository is leveraged used by all teams § Models are the single source of truth for Biz/Dev/Test/Ops § Continuous Development Framework in place Most IT Organizations
- 37. 37 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD > COGNITIVE ADAPTIVE < RUNDEPLOYTESTBUILDPLAN INTAKE Requirements User Stories Release Plan MODEL Import User Stories to Automatically Create, Visualize and Optimize Tests Initiate Functional, Non- Functional and Security as Code FUNCTIONAL § Subset/Mask Test Data § Create/Reserve Test Data § Test Automation Library § Ensure Mobile Experience INSIGHT-DRIVEN CONFIG/DEPLOY § Provision Entire Stack § Confirm Configurations § Approve Changes § Successfully Deploy § Internal or External Cloud INTEGRATION Remove Constraints with Virtual Services § Mobile, Web, App Server, Middleware, Backend, MF § 3rd Party Systems / API’s CODE Develop and Commit Code, Scan Code, Version Control, Continuous Integration. Complete Build and Initiate Release AUTOMATION AUTOMATION AUTOMATION ü Speed/Time-to-Market ü Quality/Availability AUTOMATION ü Cost/Financial Mix ü Risk/Compliance ü Requirements ü Environments ü Data ü Automation Customer Experience NON-FUNCTIONAL § Performance Engineering § Simulate Backend Load § Test Outlier Conditions § Ensure Mobile Experience § Security / Penetration Common Goals “Big Picture” Issues DESIGN SHIFT-LEFT SHIFT-RIGHT OPS Node.Probe Node.Test TDD BDD ATDD BPMN Nimbus Visio Node.Data Node.Learn MEASURE/FEEDBACK § Customer Experience § Business Service View § Application View § Infrastructure View § Dynamic Capacity § Feedback loop
- 39. 39 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Erik Simmons Value-Driven Delivery Copyright © 2016 nuCognitive LLC. All Rights Reserved. NEW world exclusive CA sponsored Webinar with nuCognitive. Register now value-stream.net
- 44. 44 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Recommended Sessions SESSION # TITLE DATE/TIME DO5X29S Model-Based Testing for Achieving Maximum CD Pipeline Acceleration 11/16/2016 at 3:00 pm DO5X47S 'Shift-Right' - Rapid Evolution with DesignOps 11/16/2016 at 3.45 pm DO5X40S Case Study: Why Rabobank Was Inspired 11/17/2016 at 3.45 pm
- 46. 46 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD Stay connected at communities.ca.com Thank you.
- 47. @CAWORLD #CAWORLD © 2016 CA. All RIGHTS RESERVED.47 @CAWORLD #CAWORLD DevOps – Continuous Delivery For more information on DevOps – Continuous Delivery, please visit: http://cainc.to/PiTFpu