Drive Continuous Delivery With Continuous Testing

Editor's Notes

• #2: Throughout the global economy, across all industries, companies are re-inventing themselves―to become better at sensing the next big thing their customers need, and finding ways to deliver it to them faster than the competition.   At CA, we’re fueling a big part of this revolution where ideas are the new currency. Our passion is helping you to champion your best ideas and navigate the obstacles across the application lifecycle to break through and produce real measurable outcomes. What’s slowing you down? What are your bottlenecks preventing you from getting your product, your software, out the door with exceptional quality? What’s keeping you from getting to market as fast as possible? Have you already predefined what “as fast as possible” is with how things have been that you can’t see what it could be? [next]
• #3: It’s done through Continuous Testing. Well define it later on but to serve as a compass to this discussion: We represent the Continuous Delivery division of CA We define Continuous Delivery as the ability to release _ any application at any time to any environment with zero touches But the caveat to that is, you can’t achieve Continuous Delivery, without Continuous Testing. Testing has to be a focus, just like everything else, because it’s become one of the most glaring bottlenecks in the SDLC. It’s hindering DevOps and threatening Agile. We have customers all the time tell us how agile they are. [next]
• #5: http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf The majority of delays were still occurring at the Test/QA stage of the SDLC Therefore, even when DevOps exists, companies are still experiencing a trade-off between quality and speed 29% of our DevOps practitioners told us that quality was being sacrificed to at least a degree, which means they were looking for speed. 43% are choosing to focus on quality as an area for improvement. Were they going too fast? [next]
• #6: https://www.ca.com/us/collateral/industry-analyst-report/continuous-testing-lessons-from-leading-practitioners.html CA commissioned Freeform Dynamics to conduct market research surrounding Continuous Testing. The came back with a great quote saying there’s, “Clear evidence…” They found that: The need for adoption exists 75% Organizations recognize the critical need to shift testing left with continuous testing practices. 72% say it requires a mind shift. They need champions and evangelists to convince the powers that be. And the need for technology exists 63% say the tools they use and know of wont get them there 79% say that if the right tools existed, continuous testing could be achievable [next]
• #7: Of the respondents that were surveyed, they weren’t oblivious to continuous testing. Everybody knows and sees testing as a critical piece. However, within the sample there was also evidence of those who were ahead of the pack. This survey also concluded, industry “Leaders” who are already focused on continuous testing, are more successful shifting their testing activities further to the left of the SDLC. Are two times more confident in both the speed and quality of deliverables to meet demands are 1.5 times more likely to deliver 10x faster iterations [next] Are 2.6 times more likely to reduce defects by 50% or more [next] And 3.9 times more likely to experience rapid revenue growth against their peers. There’s obviously more to their stories and this wouldn’t all be credited continuous testing, but it reaffirms, right along with other research that’s out there, that: being more agile, cultivating a Devops culture, pressing on towards Continuous delivery, keeping speed and quality at the forefront, you WILL become a leader and increase revenue. [next]
• #8: So before you start to define Continuous Testing yourself, this is what Continuous Testing is NOT. [next]
• #9: We define continuous testing as… “” This picture helps to illustrate our point. Quality is not an owned discipline of QA department. Just like how evolution of TQM (Total Quality Management) and Lean Manufacturing put quality checks between each stage of the production line in the 80’s and 90’s, Quality is EVERYONE’s responsibility and it should be imbedded at each stage of the SDLC. [next]
• #10: Once an idea is approved and requirements are written: Requirements are read An RTM, or requirements traceability matrix is created Test cases are created and tied to the requirements in the RTM Test data requirements are defined Internal and external interfaces necessary for the requirement to be fulfilled are identified Automation tests are created from the requirements and test cases Test data has to be found or created before tests can be run Internal and external interfaces needed for tests to actually run are made available or virtually enabled Finally… the test is executed, any defects found are logged and developers start the code fixing cycle Testing activities are still a bottleneck because the process hasn’t changed in 30 years! There’s been plenty of advancements in surrounding departments and disciplines like development, business analysis and operations. Everything you see on the blue line in the middle is the problem. What’s interesting, what REALLY matters is to provide quick feedback to the developer. So the intention is to get to the Execute Test Cases as fast as possible. But with the current process you have to go through ALL the activities in blue first, so you can get to where you really care – being able to give a developer a thumbs up or down. Depending on how agile your organization is (remember the agile adoption phases), you may still be working on the workload of the previous “sprint” when this new code hits QA. So, you’re already late, the developer has moved on and now has to go back and address defects. But arguably, everything in blue, these steps can’t just go away. And you’re right. [next]
• #11: Let’s just automate them instead. If we model our requirements well, if we are intentional in our design, and have the right tools in place, ..then we can shift all these efforts to the design phase and capture everything we need in the requirements. Outside CA, the other solutions on the market don’t address these manual aspects, their focus is only in making the pain less painful by increasing efficiency of those intermediate manual steps, as opposed to removing them from the equation altogether. [next]
• #12: When we look at the end-to-end we’re bringing quality to the attention of everyone. Continuous Testing how it’s accomplished. [next] The way we do this at CA the following… 1)      Starting with Intake – aka requirements – Agile Requirements Designer (ARD) provides the capability to align business requirements and remove ambiguity by creating models that allow for the automatic creation of test cases and test case automation from the defined models.  In addition, this capability allows for optimal and efficient test case coverage and can identify 100% coverage of the defined model.  The model based approach also allows you to deal with change impact – if a requirement changes, ARD can model down and enable adaptive test case design and automation.  ARD models requirements directly from user stories from ALM, JIRA and Agile Central (and several other sources). 2)      Once we have defined our test cases and user stories, Service Virtualisation (SV) helps accelerate the delivery of quality code by virtualized services that can be shared across the enterprise to simulate those unavailable systems, both internal and external and remove constraints from Dev and Test. SV is available as Code so can be integrated directly into the Development IDE, thus accelerating adoption.  SV also provides performance testing capability against these virtualized services. 3)      Following on from removing constraints with system access, next we may need access to Test Data.  Test Data Manager (TDM) provides automatic generation of synthetic test data, on demand - available across the enterprise and in a compliant manner to assist with regulations such as GDPR.  TDM also supports traditional data masking and sub-setting but builds upon this with its Profiling capability to ensure you also understand your potential exposure to PII data, as well as providing the capability for data portability across the enterprise with its vTDM (virtual data) capability – effectively removing test data constraints across the organization in a compliant manner. 4)      Once we have our Test cases modelled and aligned with our user stories and test systems available, along with the right test data, CA AppTest enables us to accelerate testing outside of the UI by allowing you to test against APIs – following workflows and scenarios built, full API testing can occur much earlier in the Dev lifecycle, even if the dependent system is not yet available through integration with CA SV. 5)      CA Blazemeter provides SaaS infrastructure capabilities to move Performance and Load testing much earlier in the lifecycle.  CA Blazemeter provides a Domain Specific Language so that Performance Testing can even occur at the Unit Testing phase within the developer IDE.  CA Blazemeter is also built upon open source technologies that protect your existing investment with support of JMeter, Selenium, Gatling, Taurus – and other common opens source load testing frameworks.  CA Blazemeter also provides SaaS based functional API testing capability.  The choice for CA AppTest or CA Blazemeter will in part be driven by your direction with regards to SaaS adoption in your organisation. 6)      As above 7)      As above 8)      In addition to being able to shift Performance Testing left within the SDLC, we provide capabilities with respect to Application Security with CA Veracode – direct integration into the IDE with the “Greenlight” capability – security unit testing - that not only identifies risk in near real time whilst coding but also identifies good behaviours that helps improve the capabilities of the team.  Coupled with Static and Dynamic code analysis to identify flaws and vulnerabilities, CA Veracode is also a SaaS based capability - that has so far scanned more than 3 trillion lines of code for our customers. 9)      Nearing the end of the flow – the ability to understand the lifecycle of a user story – Continuous Delivery Director (CDD) provides you with the insight needed to understand the flow of a user story across the SDLC – CDD provides insights into phase promotion, cycle time, idle time and enables trust and transparency for all relevant stakeholders into the lifecycle of the delivery of value to a customer. Not only is this a CD pipeline optimization tool, giving you fantastic visibility to manage all states of the SDLC, but it also ensures applications and code doesn’t get promoted until test cases are passed. It really is the hub of the activity, with it’s integration with the testing tools to kick each one of them off. 10)  Lastly with respect to continuous improvement – CA Digital Experience Insights (DXI) allows you to understand the real user experience of the value you are delivering – across Mobile, Watch and Web – DXI allows critical insight into how users are interacting with your applications via user journey flow visibility, crash analytics and operational data, ensuring you can continue to deliver a quality experience to your users.  We spin the wheel to emphasize that that you don’t just kick off automated test scripts when code hits QA. These are disciplines to be used continuously at each stage of the SDLC. Only then is it possible to achieve continuous testing, continuous deployment, continuous delivery, and record-breaking ROI and customer experience. [next]