![6
10
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Monitoring
Collaborative
Development
Continuous
Release and
Deployment
Continuous
Customer
Feedback
and
Optimization
Continuous
Business
Planning
Monitor
and Optimize
Develop
and Test
Release
and Deploy
Plan
and Measure
DevOps –
Continuous
innovation,
feedback and
improvements
Enables clients to seize market opportunities
and reduce time to customer feedback
Increasing adoption of DevOps
Extending Lean and Agile practices across the entire software value chain
Continuous
Testing
11
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers](https://image.slidesharecdn.com/w8-140714152244-phpapp02/85/Continuous-Testing-through-Service-Virtualization-8-320.jpg)
![7
12
Continuous Deployment and Release
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Dev UT
At least 1/3 of is testing!
13
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Test
scripts
Test
data
Test
configurations
Test
reports
Version
Control
Orchestration & Automation
Dev UT
At least 1/3 of is testing!](https://image.slidesharecdn.com/w8-140714152244-phpapp02/85/Continuous-Testing-through-Service-Virtualization-9-320.jpg)
Continuous Testing through Service Virtualization
- 1. W8 Test Techniques 5/7/2014 1:45:00 PM Continuous Testing through Service Virtualization Presented by: Allan Wagner IBM Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Allan Wagner IBM IBM-certified IT Specialist Allan (Al) Wagner is currently a technical manager and evangelist with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific focus on quality management, test automation, and service virtualization. In his fifteen years of practical IT field experience, Al has assisted, supported, mentored, and enabled teams with their IT application infrastructure, development, implementation, and operations challenges. A frequent conference speaker on software quality principles and techniques, Al also has authored or coauthored the IBM Service Virtualization for Dummies book and numerous papers on quality management and test automation.
- 3. 1 Continuous Testing through Service Virtualization @Al_Wagner awagner@ca.ibm.com Technical Manager IBM Session W8 Test Techniques Wednesday, May 7th, 1:45pm 1 Business stakeholder expectations Better – Cheaper - Faster In the past • Organizations looked to find a balance between speed, cost & risk. • Development teams were not as agile as they are today; testing never seems to have enough time. • Software quality suffered. In the past • Organizations are asking teams to release higher quality software sooner. • Development teams are becoming more and more agile; testing still never seems to have enough time. • Software quality continues to suffer.
- 4. 2 2 Business stakeholder expectations Better – Cheaper - Faster In the past • Organizations looked to find a balance between speed, cost & risk. • Development teams were not as agile as they are today; testing never seems to have enough time. • Software quality suffered. In the past • Organizations are asking teams to release higher quality software sooner. • Development teams are becoming more and more agile; testing still never seems to have enough time. • Software quality continues to suffer. What are organizations “actually“ doing to deliver higher quality software to market faster? 3 Adding more Testers is not a sustainable model Labor costs continue to increase Testers Testers Testers Development Team Testers
- 5. 3 4 Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before! external users internal users Application Layer Mainframe ERP Systems Custom Applications Databases Enterprise Service Bus Functionality Usability Performance/Scalability Reliability Consumability Before, testers designed tests to measure quality based on a number of attributes. 5 hybrid Network Latency Security/Privacy Availability Cost Data Integrity Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before! external users internal users private Business Partner Services public External 3rd Party Services Application Layer Mainframe ERP Systems Custom Applications Databases Enterprise Service Bus Functionality Usability Performance/Scalability Reliability Consumability Before, testers designed tests to measure quality based on a number of attributes. Now, there are many new things testers also need to consider.
- 6. 4 6 Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers Tester Setup test environment Developer Write & deliver code Nightly build(s) Compile, unit test, publish Developer Developer Developer Developer Developer “N” nightly builds are piling up 7 Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers Tester Setup test environment Developer Write & deliver code Nightly build(s) Compile, unit test, publish Developer Developer Developer Developer Developer “N” days to install & configure “N” nightly builds are piling up Quality GateBVT Multiple Test Environments FVT Perf Test Security Test SIT UAT I just spent 2 weeks setting this up & the build is no good? When agility stops at testing, the entire organization hits the wall
- 7. 5 8 Bottom line – current approaches are not helping! Today’s testing challenges still remain Development VelocityTesting Labor Test Lab Complexity Outsourcing labor is no longer a sustainable model as global wages are increasing Product and application complexity are driving test lab cost increases Test teams can no longer keep up with agile development Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration Web/ Internet Routing Service Third-party Services Portals Content Providers EJB Shared ServicesArchives Business Partners Messaging Services Traditional Testing Other costs Testing costs 9 What Testers really need! Validating today’s complex applications Production-like test environments which are: realistic & complete available on demand inexpensive to setup Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration App Under TestRouting Service Third-party Services Portals Content Providers EJB Shared Services Archives Business Partners Messaging Services Other costs Testing costs Automated tests which are: able to run unattended validating application integration, functionality, security & performance capable of validating business flows across multiple channels durable, maintainable & resilient to change Orchestration from build to production deployment which is: reliable repeatable traceable Better Faster Cheaper
- 8. 6 10 Accelerate software delivery – for faster time to value Balance speed, cost, quality and risk – for increased capacity to innovate Reduce time to customer feedback – for improved customer experience Continuous Monitoring Collaborative Development Continuous Release and Deployment Continuous Customer Feedback and Optimization Continuous Business Planning Monitor and Optimize Develop and Test Release and Deploy Plan and Measure DevOps – Continuous innovation, feedback and improvements Enables clients to seize market opportunities and reduce time to customer feedback Increasing adoption of DevOps Extending Lean and Agile practices across the entire software value chain Continuous Testing 11 All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers
- 9. 7 12 Continuous Deployment and Release All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers Collaborative Development The goal: create the build package Continuous Monitoring The goal: production deployment Continuous Testing simultaneously across multiple test environments Everything between dev & prod is a test stage Quality Gate Security FVT Performance SIT UAT Quality Gate BVT Dev UT At least 1/3 of is testing! 13 All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers Collaborative Development The goal: create the build package Continuous Monitoring The goal: production deployment Continuous Testing simultaneously across multiple test environments Everything between dev & prod is a test stage Quality Gate Security FVT Performance SIT UAT Quality Gate BVT Test scripts Test data Test configurations Test reports Version Control Orchestration & Automation Dev UT At least 1/3 of is testing!
- 10. 8 14 Reality check Which answer would you pick? What percentage of your time are you spending setting up, maintaining, and re-cycling test environments? 1. 0% to 20% 2. 21% to 30% 3. 31% to 40% 4. 41% to 50% 5. More than 50% 11% 18% 14% 43% 14% Responses from GRUC Webcast – October 22nd, 2013 15 Solving the Test Environment dilemma Making production-like test environments a reality Traditional test environments are both underutilized and insufficient – Environments are expensive: hardware, software, and labor – Tester and developer time wasted waiting for appropriate environment Agile test environments require four enabling capabilities Dynamic Infrastructure Test Data Management Deployment Automation Test Environments DevOps Foundation Service Virtualization
- 11. 9 16 Reality check Which answer would you pick? How does your organization deal with testing bottlenecks due to unavailable or missing software and services? 1. Manually write simulation stubs to emulate missing functionality. 2. Use commercial service virtualization solution. 3. Defer testing until later when the code is available to test. 4. De-scope testing of that functionality. 5. None of the above. 9% 9% 56% 6% 19% Responses from GRUC Webcast – October 22nd, 2013 17 What is Service Virtualization? Making the unavailable available for testing Virtual components simulate the behavior of a service or application during testing – System dependencies are a key challenge in setting up test environments – Unavailable/inaccessible services slow down the testing process – Costly 3rd party access fees can significantly impact testing availability Virtual components run on commodity hardware, private cloud, public cloud Each developer and tester can easily have their own test environment Developers and testers can continue to use current testing procedures and tools Streamline creation of production-like environments and negative testing Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration App Under TestRouting Service Third-party Services Portals Content Providers EJB Shared Services Archives Business Partners Messaging Services Databases Mainframe applications App Under Test Third-party Services Packaged apps, messaging services, etc. Virtual Components
- 12. 10 18 Virtual components simulate the “real world” behavior Category Detail Simple Hard-coded response returned for given input Non-deterministic One-of-n hard-coded responses Data driven Input and/or output data specified in external data source (Excel, file, database) Data model, stateful Input and/or output data kept in data model with complex relationships. Supports CRUD and other stateful behavior Behavioral Extends model-driven to provide pre-packaged functionality, e.g. shopping basket, real-time data feed, trading exchange, order matching What are virtual components? Simulating application functionality, behavior, & performance 19 Test Environment Volume Constraints Volume Constraints 2hr. access window Maintenance Constraints Database / JDBC Server/ Web Services Mainframe / MQ/CICS Smart Device / SD-IO MQ/CICS Web Services Smart Devices JDBC Virtual Environment Capture & Model System Under Test Listen / Capture Store Test Case Repository Creating virtual components Listen and capture
- 13. 11 20 Build virtual components from the design specs HelloService Turn specifications into virtual components Creating virtual components Author from scratch using design specifications 21 Continuous Testing improves software quality Actual Service/App Virtual Service/App Service Virtualization is an enabler for continuous testing Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion. Controlled integration helps isolate defects for faster resolution. Service virtualization enables continuous integration testing Using virtual components Enables testers to continuously test what they want, when they want, how they want
- 14. 12 22 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application 23 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Simulate the behavior and performance of the dependent components. Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service
- 15. 13 24 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Simulate the behavior and performance of the dependent components. Begin end to end testing earlier and continuously. Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service 25 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Simulate the behavior and performance of the dependent components. Begin end to end testing earlier and continuously. Introduce the real implementations as they become available. Mainframe Database ERP System Custom Application Virtual Service
- 16. 14 26 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer new 27 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays. Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application Virtual Service App Layer new
- 17. 15 28 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays. Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees. Quickly stand up test labs using virtual services to simulate dependencies which have not changed. Testers Business Partner Services 3rd Party Services Virtual Service Virtual Service App Layer Virtual Service new 29 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays. Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees. Quickly stand up test labs using virtual services to simulate dependencies which have not changed. Test applications end to end with confidence but at a lower cost and without delay. Testers Business Partner Services 3rd Party Services Virtual Service Virtual Service App Layer Virtual Service new
- 18. 16 30 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application Testers 31 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes. Create virtual components from development specifications. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application JSON RESTWSDL Virtual Service Virtual Service Virtual Service Testers
- 19. 17 32 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes. Create virtual components from development specifications. Record the functionality and behavior of the application using virtual services. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Testers rec 101101100101101100 33 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes. Create virtual components from development specifications. Record the functionality and behavior of the application using virtual services. Execute automated tests at any layer in the application using virtual services to simulate functionality under development. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application WSDL Virtual Service Virtual Service Virtual Service Testers 101101100 101101100
- 20. 18 34 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer new 35 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity. Deploy a virtual service to simulate the required functionality and behavior replacing the defective component. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newVirtual Service
- 21. 19 36 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity. Deploy a virtual service to simulate the required functionality and behavior replacing the defective component. Continue testing without delay. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newVirtual Service 37 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity. Deploy a virtual service to simulate the required functionality and behavior replacing the defective component. Continue testing without delay. Replace the virtual service with the updated source code without having to redeploy your entire application. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newfixed
- 22. 20 38 The ROI of Service Virtualization The results are real! 3 Year Benefits: • Increased project delivery capacity 100% (40 projects to 80 projects) • Avoided hiring three additional FTE's, saving almost $600k. • Reduced incidents found in production from 2.5 to .3 per project, saving $3 million • Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k • Reduced consulting and third party testing fees with better test automation, saving $408k • Eliminated the need for additional hardware and labor to set up test environments, saving $700k • Generated new revenue for the business – estimated at tens of millions! A major European bank implemented service virtualization and automated integration testing as part of an Agile transformation to enable continuous testing. As a result of IBM capabilities, their system integration test (SIT) process was reduced from 3 weeks down to four hours! The Total Economic Impact™ of IBM Rational Service Virtualization and Test Automation Solutions, a July 2013 commissioned study conducted by Forrester Consulting on behalf of IBM 39 Orchestration Continuous Testing InfoSphere Optim Test Data Management Rational Test Virtualization Server Rational Test Workbench • Capture service behavior and model virtual services • Test services and applications long before their user interfaces becomes available and do integration testing • Performance test, regression test, functional test & integration test • Virtualize heterogeneous hardware, software, and services to provide 24x7 testing capabilities • Reduce the cost of traditional testing environments • Build Virtual Services to simulate over 70+ protocols Rational Performance Test Server • Reuse Rational Test Workbench scripts to drive performance testing • Combine virtual services and the real implementations to performance test earlier • Probe for identification of system bottlenecks • Create referentially intact, “right-sized” test databases • Automate test result comparisons to identify hidden errors • Protect confidential data used in test, training & development • Refresh test environments faster shortening iterative testing cycles and accelerating time to market IBM’s continuous testing solution Rational Quality Manager • Track quality using a comprehensive and customizable test plan • Link test cases with requirements, defects, development work items and builds • Author and execute automated or manual test scripts • Provide up-to-date project quality status using customizable dashboards Test Data Management Service Virtualization Test Automation with integration to Collaborative Development & Test Completing the Continuous Testing story Service Virtualization is the enabler but it is only part of a Continuous Testing solution
- 23. 21 40 Orchestrate and automate the deployment of applications, middleware configuration, and database changes into development, test and production environments. Databases Internal Messages Third-party Services virtual components Simultaneously test across multiple test stages Dev QA Test Data Management Deploy Provision Test Combining continuous testing withOrchestration from build to production Reliable, repeatable, & traceable deployment automation Deployment Automationintegrated with Test Environments Dynamic Infrastructure Test using real world network conditions Service Virtualization Test Automation Deploy and refresh right size secure data Deploy what is ready, virtualize the rest Continuously test in production-like env. Network Virtualization 41 Continuing on your Service Virtualization journey Increasing awareness and accelerating learning Get your copy today! Service Virtualization For Dummies ibm.co/ServiceVirtualizationForDummies Visit Service Virtualization on ibm.com ibm.co/servicevirtualization
- 24. 22 42 Continuous testing as our planet gets smarter If you think testing is hard now, just wait – but don’t wait too long intelligent interconnectedinstrumented We are already on the road to the Internet of Things Sooner than later, someone is going to ask you to test it! 43
- 25. 23 44