CWIN17 New-York / Drive continuous delivery with continous testing
2 likes322 views
Continuous testing is an approach that aims to shift testing left and automate testing across the entire software development lifecycle. It involves automatically generating test cases from requirements, simulating test environments, accessing test data on demand, and integrating open source testing tools. Continuous testing helps ensure code quality, application quality, pipeline automation, and understanding of customer experience throughout development and delivery. Most companies currently cannot fully achieve continuous testing due to barriers like lack of test automation, slow testing bottlenecks, and complex testing tools.
CWIN17 New-York / Drive continuous delivery with continous testing
- 1. 1 Drive Continuous Delivery With Continuous Testing Jeff Bradley Regional CTO Continuous Testing CA Technologies September 2017
- 2. © 2017 CA. Confidential. All rights reserved. Eliminating the barriers between ideas and outcomes? How Good are You at …
- 3. © 2017 CA. Confidential. All rights reserved. Your Software Factory Must be an Accelerator. But is it an Inhibitor? ProductConcept EFFICIENT, HIGH-QUALITY, PRODUCTIVE PIPELINE MORE CUSTOMER VALUE BUSINESS ALIGNMENT
- 4. © 2017 CA. Confidential. All rights reserved. 4 THE MODERN SOFTWARE FACTORY The capabilities you need, from mobile to mainframe, to deliver the experiences customers want. Agile DevOps Security MAXIMIZE APPLICATION PERFORMANCE MAKE SECURITY A COMPETITIVE ADVANTAGE CREATE AN AGILE BUSINESS BUILD BETTER APPS, FASTER
- 5. © 2017 CA. Confidential. All rights reserved. 5 Continuous Testing How do you build better apps, faster?
- 6. © 2017 CA. Confidential. All rights reserved. Why Many Companies Think They’re Agile… They moved some Dev projects from waterfall to agile They’re having daily standups They have a scrum master Product owner is part of the team They are all talking and walking agile… And are talking about Continuous Delivery BUT…
- 7. © 2017 CA. Confidential. All rights reserved. Testing is STILL a Bottleneck… Even in DevOps Shops http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf …of delays were occurring at the Test/QA stage of the cycle. “Where are the main hold-ups in the software production process?” 32% 16% 22% 63% 21% 23% 30% 63%A 2017 survey of self- proclaimed DevOps practitioners found that … 7
- 8. © 2017 CA. Confidential. All rights reserved. Why Quality is So Hard to Achieve? Clear evidence that you are unlikely to enjoy the full benefits of continuous delivery unless you modernize your testing tools and practices. § 75% highlight continuous testing as critical or important § 76% see Continuous Testing as critical or highly desirable to ‘shift-left’ testing activity from downstream back into development Traditional testing tools are often too complex and expensive to use Adoption of continuous testing is much easier with the right tools Continuous testing requires a major shift in mindset and culture agree or strongly agree 63% agree or strongly agree 79% agree or strongly agree 72% “Continuous Testing as a Digital Business Enabler”, 2017 “ 8
- 9. © 2017 CA. Confidential. All rights reserved. “Continuous Testing as a Digital Business Enabler”, 2017 Tangible Evidence of the incremental value continuous testing represents for “Leaders” more likely to have succeeded in left-shifting testing activity to requirements phase more confident in quality of output more confident in speed of delivery more likely to experience rapid revenue growth more likely to deliver 10x faster iterations more likely to reduce defects by 50%+ 2.5x 1.5x 1.9x 2.6x 2.4x 3.9x SPEED QUALITY ROI 9
- 10. © 2017 CA. Confidential. All rights reserved. Challenges to Achieving Continuous Delivery & Testing Requirements Development QA / Testing Operations Release 70% of all testing is still manual4 63% of testers admit they can’t test across all the different devices and OS versions5 50% of time spent looking for test data6 79% of teams face prohibitive restrictions, time limits or access fees on needed 3rd party services3 X X X 64% of total defect cost originate in the requirements analysis and design phase1. 50% of developers time is spent fixing and finding defects2 ? 80% of teams experience delays in development and QA due to unavailable dependencies3 ? 57% are dissatisfied with the time it takes to deploy new features7 ! ! Greater than 50% production outages from configuration management ! Ave. cost of downtime per minute = $4,700 53% percent of visits are abandoned if a mobile site takes more than three seconds to load IDEA Security 50% more time spent on security defects in lower-performing teams8 30% of teams only security scan once per year9 70xrequired manual pen test scan cost vs. automation10 X X
- 11. © 2017 CA. Confidential. All rights reserved. What is Continuous Testing? First need to understand what it’s not! It’s not… Test Automation It’s not… Continuous Integration It’s not… Unit Testing It’s not… done only in the Test environment It’s not… a QA-led activity 11
- 12. © 2017 CA. Confidential. All rights reserved. What is Continuous Testing? The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected
- 13. © 2017 CA. Confidential. All rights reserved. Key Elements of Continuous Testing Automatically generate test cases from requirements 1 2 Simulate test environment 3 Access test data on- demand & ensure PII compliance 4 Start multi-layer / backend “requests & response” testing prior to UI 5 Democratize performance testing Integrate open source testing tools6 7 Ensure comprehensive, cloud-based API testing 8 Built-in automated application security testing 9 Promote artifacts when tests pass by orchestrating & automating the pipeline 10 Harness application insight across the SDLC to improve user experience
- 14. © 2017 CA. Confidential. All rights reserved. 14 Most Customers Cannot Do Continuous Testing Today, But They Aspire To Get There….
- 15. © 2017 CA. Confidential. All rights reserved. If This is Your Current SDLC You are waterfall 15 Dev Test X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
- 16. © 2017 CA. Confidential. All rights reserved. If This is Your Current State 16 Sprint 1 Sprint 2 Sprint 3 X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations You are water-SCRUM-fall
- 17. © 2017 CA. Confidential. All rights reserved. If This is Your Current State 17 Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Manual Testing Automated Regression Testing X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 Release X WEEKS X WEEKS Sprint 3 You are not “really” agile
- 18. © 2017 CA. Confidential. All rights reserved. What Testing at the Speed of Agile Should Look Like! 18 No Sprint-lag effect Release in end of Sprint In-Sprint Test Automation In-Sprint everything to achieve “Continuous Testing” Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Exploratory Manual Testing Test Automated Scripting & Exectution X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 Release Sprint 3 ReleaseReleaseRelease HIGHLIGHTS
- 19. © 2017 CA. Confidential. All rights reserved. TESTING REALITY Dev Tools TOOLSET TEST “Ice Cream Cone” UI Tools Dev Tools TOOLSET TEST PYRAMID API Unit UI API Tools UI Tools TESTING GOAL API Tools API Unit UI 94% 3% 3% 5% 45% 50% Non-functional: static, performance, security, etc… *Actual Customer Example of Their Testing Strategy
- 20. © 2017 CA. Confidential. All rights reserved. Why are Traditional Testing Activities a Bottleneck? Enable Interfaces (real or virtual) Create Detailed Test Cases Execute Test Cases Identify Interfaces (internal and 3P) Automate Tests (usually Regression) Create Defects Fix Defects Define an Idea or Customer Needs Write Requirements Create Requirements Traceability Matrix Read Requirements Manual Processes Find/Create Test Data Testing hasn’t changed in the past 30 years Identify Test Data Requirements
- 21. © 2017 CA. Confidential. All rights reserved. But Continuous Testing Changes the Game… Enable Interfaces (real or virtual) Create Detailed Test Cases Execute Test Cases Identify Interfaces (internal and 3P) Automate Tests (usually Regression) Create Defects Fix Defects Define an Idea or Customer Needs MODEL Requirements Create Requirements Traceability Matrix Read Requirements Find/Create Test Data Identify Test Data Requirements NOW ALL AUTOMATED
- 22. © 2017 CA. Confidential. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING TEST EXECUTION MONITORING Continuous Integration Continuous Testing Continuous Delivery Automated Deployment WhatHow • Flowchart Diagram TEST DESIGN DEVELOPMENT
- 23. © 2017 CA. Confidential. All rights reserved. Simple View of Continuous Testing
- 24. © 2017 CA. Confidential. All rights reserved. Simple View of Continuous Testing MONITORING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow • Test-Driven Development • Acceptance Test-Driven Development REQUIREMENTS MODELING TEST EXECUTIONTEST DESIGN DEVELOPMENT • Test Automation Scripts • Test Data
- 25. © 2017 CA. Confidential. All rights reserved. Simple View of Continuous Testing MONITORING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow • Test Environments • Test Automation Engines • Test Frameworks REQUIREMENTS MODELING TEST EXECUTION TEST DESIGN DEVELOPMENT
- 26. © 2017 CA. Confidential. All rights reserved. Simple View of Continuous Testing MONITORING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow • Analyze • Learn • Update Model REQUIREMENTS MODELING TEST EXECUTION TEST DESIGN DEVELOPMENT
- 27. © 2017 CA. Confidential. All rights reserved. 4 Pillars of Continuous Testing Can the application code flow across environments without manual intervention? Did developers create the correct features? Are users perceiving value in the application delivered? Did developers create the application code correctly? Pipeline Automation Code Quality Customer Experience Application Quality Code Correct? Automated Flow? User Value? Features Correct?
- 28. © 2017 CA. Confidential. All rights reserved. Building On Top Of The Pillars DEV Validate Build Promotion Criteria Model Requirements (Story Level) Assign Definition of Done for Backlog Items Sprint Planning Gold QA Build TESTProvision QA Environment and Deploy Build Validate Build Configuration Validate Build Promotion Criteria Gold QA Build Received Auto Generate Virtual Services Definition and Test Data Definition Auto Generate Test Cases (manual and automated) Enhance Modeled Requirements (Feature Level) Gold Stage Build STAGE Start Full Application Monitoring Validate Build Promotion Criteria Gold Stage Build Received Auto Generate Virtual Services Definition and Test Data Definition Auto Generate Test Cases (manual and automated) Enhance Modeled Requirements (Epic Level) Gold Production Build PROD Validate Approvals for Production Promotion Gold Prod Build Received Auto Generate Prod Environment Configuration (for A/B and Canary) Auto Generate Deployment Paths (A/B and Canary) Model A/B Tests and Canary Builds (Pick User Base) CX Hypothesis Back to Sprint Backlog CONTINUOUS TESTING
- 29. © 2017 CA. Confidential. All rights reserved. Translate Requirements Into Visual Models DEV Validate All Backlog Items Have a Definition of Done Validate All Backlog Items Are Prioritized Regression Test Promotion Criteria Validation Process Test Code Build Process Test Trigger for BVTs Create, maintain and Run BVTs Regression Test Environment Provisioning Process Test Automated Code Scan Triggers Test Virtual Service and Data Provisioning Process Test-Driven Development Development Environment Validate Build Promotion Criteria Model Requirements (Story Level) Assign Definition of Done for Backlog Items Sprint Planning Create Virtual Services and Automated Synthetic Data Generation Automated Code Scans (Quality/Security), Code Commit Prioritize Backlog (Ideas and Hypothesis) Gold QA Build App and API Code Development and Automated Unit Tests Update Configuration and Provision Environment Create and Deploy Build. Run Automated BVT - Build Verification Tests Continuous Testing Tasks Continuous Testing Tasks DEV Test Automated Code Scan Triggers Test Automated Code Scan Triggers
- 30. © 2017 CA. Confidential. All rights reserved. Regression Test Promotion Criteria Validation Process Add Layers Into Visual Models (Automation, Test Data, Virtual Services) Regression Test Application Monitoring Process Configuration Regression Test Processes for: § Environment Provisioning § Build Deployment Regression Test Build Configuration Test Environment Validate Build Promotion Criteria Gold QA Build Received Auto Generate Virtual Services Definition and Test Data Definition Auto Generate Test Cases (manual and automated) Validate Build Configuration Provision QA Environment and Deploy Build Provision Virtual Services and Test Data Start Full Application Monitoring Enhance Modeled Requirements (Feature Level) Gold Stage Build Intelligent Test Automation: § Capture and Analyze Test Result Data § Correlate Results to Past Executions § Formulate Hypothesis for Improvement Automated Test Execution TEST Continuous Testing Tasks Regression Test Provisioning Processes for: § Virtual Services § Test Data TEST Regression Test Provisioning Processes for: §Virtual Services §Test Data
- 31. © 2017 CA. Confidential. All rights reserved. Translate Requirements Into Visual Models Regression Test Promotion Criteria Validation Process Regression Test Application Monitoring Process Configuration Regression Test Processes for: § Environment Provisioning § Build Deployment Regression Test Build Configuratio n Staging Environment Validate Build Promotion Criteria Gold Stage Build Received Auto Generate Virtual Services Definition and Test Data Definition Auto Generate Test Cases (manual and automated) Validate Build Configuration Provision Stage Environment and Deploy Build Provision Virtual Services and Test Data (external only) Start Full Application Monitoring Enhance Modeled Requirements (Epic Level) Gold Prod Build Intelligent Test Automation: § Capture and Analyze Test Result Data § Correlate Results to Past Executions § Formulate Hypothesis for Improvement Automated Test Execution (Functional/ Performance) STAGE Continuous Testing Tasks Regression Test Provisioning Processes for: § Virtual Services § Test Data STAGE Regression Test Provisioning Processes for: §Virtual Services §Test Data
- 32. © 2017 CA. Confidential. All rights reserved. Regression Test Application Monitoring Process Configuration Regression Test Build Deployment Process to Production Regression Test Processes for: § Environment Provisioning Production Environment Validate Approvals for Production Promotion Gold Prod Build Received Auto Generate Prod Environment Configuration (for A/B and Canary) Auto Generate Deployment Paths (A/B and Canary) Validate Build Configuration Update Prod Environment Configuration (A/B and Canary Builds as needed) Deploy Build to Production (automatically) Start Full Application Monitoring (IT and CX) Model A/B Tests and Canary Builds (Pick User Base) CX Hypothesis Back to Sprint Backlog § Test the Data Collection and Analysis Process § Test the New Parameters Automatically Added to the Hypothesis Formulation Cognitive Process Analyze CX Metrics and Formulate Initial Hypothesis Translate A/B Test Requirements and Canary Build User Base Into Visual Models Test Process to Obtain Go Live Approvals PROD Continuous Testing Tasks Regression Test Build Configuration PROD Regression Test Build Configuration
- 33. © 2017 CA. Confidential. All rights reserved. End-to-End Integrated Continuous Testing from CA Automatically generate test automation scripts from requirements 1 2 Simulate test environment 3 Access test data on- demand & ensure PII compliance 4 Start multi-layer / backend “requests & response” testing prior to UI 5 Democratize performance testing Integrate open source testing tools6 7 Ensure comprehensive, cloud-based API testing 8 Built-in automated application security testing 9 Promote artifacts when tests pass by orchestrating & automating the pipeline 10 Harness application insight across the SDLC to improve user experience Continuous Delivery Director
- 34. © 2017 CA. Confidential. All rights reserved. CA’s Track Record Helping our Customers IDEA Requirements Development QA / Testing Operations Release Reduce time needed to generate synthetic test data by 10X Faster in delivering test cases & automation1 50% Time reduction to replicate defects3 50% reduction in app dev/test cycle2 23% time to market improvement3 90% availability for testing platforms3 1 x 500 throughput increase in testing4 35% cost reduction with 3rd party access fees3 Decrease downtime from release failures by Reduce application deployment time almost 94% Increase weekly deployments by 300% Up to 50%greater app performance 5 Early load testing to prep for an actual 25 Million Users6 3 Month Improvement in new automation And to full in-sprint automation 50% 80% Security 1 in 3 Fortune 100 firms choose Veracode 3 Trillion Lines of code scanned 32M Security flaws fixed7 12,000 Developer remediation coaching sessions
- 35. © 2017 CA. Confidential. All rights reserved. 35 THANK YOU
- 36. © 2017 CA. Confidential. All rights reserved. 36 GM Financial has operations in more than 20 countries, providing auto finance solutions to approximately 16,000 dealers worldwide Connect and automate the entire app pipeline from development, to test, to production: CA Service Virtualization CA Test Data Manager CA Continuous Delivery Director CA Agile Requirements Designer CA Release Automation SOLUTION: CHALLENGE: Cycle time too long to support exponential expansion of funded loans and comprehensive audit and compliance requirements • 5x faster loan funding • 98% reduction in deployment time from 1-2 hours manually to 2-3 minutes • 80% increase in test coverage • 90% time savings in regression testing from 93 to 9 hours RESULTS: “We anticipate growth in the next year from 1,000 automated deployments a month to 4,000– 5,000 a month.” Matt McComas AVP of Critical App Infrastructure, GM Financial Continuous Testing How do you build better apps, faster?
- 37. © 2017 CA. Confidential. All rights reserved. Mission: Establish an end-to-end best practices testing solution supported by complimentary tools that integrates, automates, and stabilizes the application delivery pipeline • CA Release Automation • CA Service Virtualization • CA Test Data Manager • CA Continuous Delivery Director SOLUTION: CHALLENGE: • Limited test coverage • Manual functional testing • Unstable test environments • No test data management • No automated release process • $500,000 in cost savings • 38,000 hours saved in manual testing in just 6 months • 85% increase in test coverage • 90% time savings in regression testing from 40 hours to 1 Continuous Testing & Delivery How do you build better apps, faster? 113 116 56 123 80 87 150 223 251 238 15 7 6 5 0 50 100 150 200 250 300 COUNT OF SEVERITY 1, 2, & 3 INCIDENTS 37
- 38. © 2017 CA. Confidential. All rights reserved. • CA Agile Requirements Designer • CA Test Data Manager • CA Application Test • CA Release Automation SOLUTION: CHALLENGE: Struggling to keep pace with the demands of the business. Needed to deliver applications and updates more quickly, of higher quality and with fewer defects. Continuous Testing & Delivery How do you build better apps, faster? “CA Technologies offered a compelling solution with its integrated tools that provide seamless, end- to-end test management…. CA Technologies solutions were able to manage all the technologies used across our applications, which other solutions were unable to do.” “With CA Technologies solutions, we can identify & rectify defects more quickly and easily before they are released into production. This will help us improve the speed and quality of software delivery.” 38