Mobile Application Dev and QA Testing with Simulated Environments
- 1. T5 Session 4/16/2015 1:00 PM "Mobile Application Dev and QA Testing with Simulated Environments" Presented by: Wayne Ariola Parasoft Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 ·∙ sqeinfo@sqe.com ·∙ www.sqe.com
- 2. Wayne Ariola Parasoft Chief strategy officer Wayne Ariola leads the development and execution of Parasoft’s long-term strategy. Wayne has contributed to the design of core Parasoft technologies and has been awarded several patents for his inventions. He is a recognized leader on service virtualization, API quality, quality policy governance, and application security. Wayne brings more than twenty years of strategic consulting experience in the technology and software development industries. Prior to joining Parasoft, he helped co-found a start-up and consulted in the strategic change practice at PricewaterhouseCoopers.
- 3. 6/29/15 1 Paraso& Proprietary and Confiden1al 1 6/29/15 Mobile Applica,on Dev and QA Tes,ng with Simulated Environments Mobile DEV + TEST-‐ April 23rd 2015 Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2 Fixed Fixed Tes1ng is A Barrier to Speed Plan Develop Build Test Release 1. “Test” is Currently a Time-‐Boxed Event 2. With the “Need for Speed” Tes1ng Must be an in-‐line event Plan Develop Build Test Release Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test
- 4. 6/29/15 2 Paraso& Proprietary and Confiden1al 3 Paraso& Proprietary and Confiden1al 3 Next Genera1on So&ware Tes1ng Verification Validation Paraso& Proprietary and Confiden1al 4 Complexity Impacts End User Experience Increased complexity, means mul1ple points of performance failure Major ISP 3rd Party Cloud Services Content Networks WiFi Browsers Phones Tablets Mainframe DB Servers Web Services App Servers Web Servers Load Balancers Mobile Components Service VirtualizationMobile Network Simulation
- 5. 6/29/15 3 Paraso& Proprietary and Confiden1al 5 Complexity Impacts Process Velocity Mobile Network Simulation App Behavior Simulation Mainframe 3rd Party Services ERP Internal App Client Simulation Mobile Application Isolation Paraso& Proprietary and Confiden1al 6 Paraso& Proprietary and Confiden1al 6 Rapid Environment Access
- 6. 6/29/15 4 Paraso& Proprietary and Confiden1al 7 Paraso& Proprietary and Confiden1al 7 So&ware Drives Innova1on The Cost of Quality associated with software has shifted dramatically Software has shifted from process enabler to business differentiator SDLC Speed will be the difference between a first mover and a follower Paraso& Proprietary and Confiden1al 8 Paraso& Proprietary and Confiden1al 8 So&ware Failures = Headlines -$2.3 Bn -3.75% The day of the announcement companies lost an average of shareholder value So&ware failures make headline news— eroding customer confidence, shareholder value and brand equity
- 7. 6/29/15 5 Paraso& Proprietary and Confiden1al 9 Paraso& Proprietary and Confiden1al 9 So&ware Failures = Headlines 821,000 1,080,000 451,000 1,400,000 896,000 1,550,000 5,500,000 650,000 3,490,000 2,240,000 0 1,000,000 2,000,000 3,000,000 4,000,000 5,000,000 6,000,000 Cloud Services Airline Bank So&ware Retail 1st Event 2nd Event Market does not forget— news about failure increase and average of 167% Paraso& Proprietary and Confiden1al 10 So&ware Failures = Headlines Financial Airlines/Aero Government Media Internet/Tel Automo,ve Technology Retail
- 8. 6/29/15 6 Paraso& Proprietary and Confiden1al 11 Paraso& Proprietary and Confiden1al 11 The Cost of So&ware Quality -‐ Sony http://www.reuters.com/article/2011/04/26/us-sony-stoldendata-idUSTRE73P6WB20110426 http://www.digitaltrends.com/gaming/sony-fined-almost-400000-for-2011-playstation-security-breach/ MarketValue Event 15 Days 30 Days -22% -33% -30%-19% -11% -12% Cumulative Loss = $18B Paraso& Proprietary and Confiden1al 12 Paraso& Proprietary and Confiden1al 12 User Experience Counts
- 9. 6/29/15 7 Paraso& Proprietary and Confiden1al 13 Paraso& Proprietary and Confiden1al 13 Fixed Fixed Impact of Test in the Evolving SDLC Plan Develop Build Test Release 1. Test is Currently Time-‐Boxed Event 2. With the “Need for Speed” Tes1ng Must be an in-‐line event Plan Develop Build Test Release Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Paraso& Proprietary and Confiden1al 14 Paraso& Proprietary and Confiden1al 14 Tes1ng Considered a Bodleneck Basic Tasks Skilled Tasks Process Oriented Business Views Fully Automated Provisioning End-to-End Tasks Data Analytics Organic Systems Automa'on is Incremental Efficiency Software Quality Processes
- 10. 6/29/15 8 Paraso& Proprietary and Confiden1al 15 Paraso& Proprietary and Confiden1al 15 What’s the View from Your Seat? Ac1ons Strategy Tasks Process Risk Governance Business Porholio Applica1on Release Itera1on Code Paraso& Proprietary and Confiden1al 16 Paraso& Proprietary and Confiden1al 16 Process Intelligence Process Intelligence aggregates “siloed” informa1on throughout the SDLC and iden1fies business risks early and automa1cally § Correlate disparate data across: § Dependent teams § Dependent components § Infrastructure systems § Tes1ng prac1ces and tools § Apply advanced sta1s1cal analysis § From pre-‐build paderns § From Marketplace § With Open APIs § Advantages § Defect remedia1on § Excep1on-‐based no1fica1ons
- 11. 6/29/15 9 Paraso& Proprietary and Confiden1al 17 Paraso& Proprietary and Confiden1al 17 Transforming the Iden1fica1on of Risk From To Causal Observa1ons Probabilis1c Defect Documenta1on Most Likely Causes Only Structured Data Structured and Unstructured Filters Business Policies Ad Hoc Automated Tool Dependent Tool Independent Paraso& Proprietary and Confiden1al 18 Paraso& Proprietary and Confiden1al 18 Next Genera1on Test Construc1on Verification Validation
- 12. 6/29/15 10 Paraso& Proprietary and Confiden1al 19 Paraso& Proprietary and Confiden1al 19 The State of Con1nuous Con,nuous delivery Con,nuous development Paraso& Proprietary and Confiden1al 20 Paraso& Proprietary and Confiden1al 20 Con1nuous – what’s missing Con,nuous delivery Con,nuous development Con,nuous tes,ng
- 13. 6/29/15 11 Paraso& Proprietary and Confiden1al 21 Paraso& Proprietary and Confiden1al 21 Gelng from “Automated” to “Con1nuous” There is a big leap moving from “Automated” to “Con1nuous” § Business expecta1ons clearly defined § Business risks iden1fied § Per applica1on, team, release § Defects automa1cally priori1zed versus business drivers § No defect le& behind § Mi1gate business risks § Clear ownership and workflow for remedia1on § Collabora1ve risk reduc1on § Collapse the remedia1on cycle 1me § Feedback loop for defect preven1on § Padern defini1on § Defect preven1on prac1ces enhanced Paraso& Proprietary and Confiden1al 22 Paraso& Proprietary and Confiden1al 22 “Shi&-‐Le&”
- 14. 6/29/15 12 Paraso& Proprietary and Confiden1al 23 Paraso& Proprietary and Confiden1al 23 “Shi&-‐Le&” Paraso& Proprietary and Confiden1al 24 Paraso& Proprietary and Confiden1al 24 “Shi&-‐Le&”
- 15. 6/29/15 13 Paraso& Proprietary and Confiden1al 25 Paraso& Proprietary and Confiden1al 25 Con1nuous Tests § Logically componen1zed § Correlated with business requirements § Incremental § Repeatable § Determinis1c § Maintainable within a process § Process is prescrip1ve based on results Continuous Testing Paraso& Proprietary and Confiden1al 26 Paraso& Proprietary and Confiden1al 26 From Automated to Con1nuous Continuous testing accelerates the SDLC by managing quality expectations and actionable tasks Requirements Defined Policy Management Development Defect Preven,on Development Development Tes,ng Sta1c Analysis Unit/Component Peer Review Automated Tests Integra,on Tes,ng API/Service Tests Smoke Test Security Tests Automated Tests System Tes,ng Func1onal Tests Scenario Tests Performance Tests CI Build Defect Remedia1on Tasks Go Release Path No Go Business Decision Service Virtualiza,on – Test Environment Access
- 16. 6/29/15 14 Paraso& Proprietary and Confiden1al 27 Paraso& Proprietary and Confiden1al 27 30 Environment Access? The average number of dependent application associated with the System Under Test (SUT) BUT, Dev/QA only have “trustworthy” access to SIX of the applications 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Paraso& Proprietary and Confiden1al 28 Paraso& Proprietary and Confiden1al 28 Test Environment Configura1on The average percentage of time spent configuring the test environment.30% 50% The average percentage of the test plan able to complete once configured.
- 17. 6/29/15 15 Paraso& Proprietary and Confiden1al 29 Paraso& Proprietary and Confiden1al 29 Survey Results… When testing with a dependent application only need access to a small percentage of the applications functionality 20 % Paraso& Proprietary and Confiden1al 30 Paraso& Proprietary and Confiden1al 30 Lack of reliable access to a realis,c test environment hampers soVware quality Conclusion
- 18. 6/29/15 16 Paraso& Proprietary and Confiden1al 31 Paraso& Proprietary and Confiden1al 31 Service Virtualiza,on delivers a simulated dev / test environment allowing an organiza1on to test any1me or anywhere Service Virtualiza1on Paraso& Proprietary and Confiden1al 32 Paraso& Proprietary and Confiden1al 32 So, What are you Virtualizing? • Mainframe • SOA / Web services • Java • .NET languages • XML • WSDL • WADL • UDDI • WSIL • SOAP • PoX (Plain XML) • REST • JSON • BPEL • Web Applications • Mobile Interfaces • AJAX • JSP • JavaScript • HTML • CSS • WS-* • Standards • MTOM(XOP) / MIME / DIME • OAuth • TCP/IP • webMethods Broker • webMethods IS • HL7 • FTP • HTTP 1.0 • HTTP 1.1 • HTTPS • JMS • IBM WebSphere MQ • Sonic MQ • RMI • EJB • SMTP • Tibco Rendezvous • .NET WCF • ISO 8583 • EDI • SAP • custom AUT
- 19. 6/29/15 17 Paraso& Proprietary and Confiden1al 33 Paraso& Proprietary and Confiden1al 33 Stubs Vs Service Virtualiza1on § Stubs § Inside-‐out approach that disassociates a test case with a dependent systems § Bridle § Limited reuse § Sta1c response § Service Virtualiza1on § Creates an environment in which to run “rich” test scenarios § Virtualized assets represent real system behavior § All virtualized-‐assets are reusable § Virtualized assets are programmable and extensible Service Virtualization provides a complete environment for developing and testing versus complex, dependent systems Paraso& Proprietary and Confiden1al 34 Paraso& Proprietary and Confiden1al 34 Access to Dev / Test Systems Test Access Simple Difficult Configuration Complex Internal Service Internal Application 3rd Party Service ERP Mainframe External Application External Database Internal Database Cloud App Message Queue ESB Web Server
- 20. 6/29/15 18 Paraso& Proprietary and Confiden1al 35 Paraso& Proprietary and Confiden1al 35 Service Virtualiza1on & Dependencies Service Virtualiza1on Virtual Lab Configuration Access § Access § Dependent applica1ons difficult § Scheduling conflicts § High access fees § Geo-‐poli1cal boundaries § 3rd party or partner applica1ons § Configura1on § Complex to configure § No control § Limited variability § Consumes test 1me DifficultSimple EasyComplex Paraso& Proprietary and Confiden1al 36 Paraso& Proprietary and Confiden1al 36 The Total Cost of System Access Service Virtualization dramatically reduces the cost of delivering a dev / test environment Hardware Savings Access EfficiencyStaged w/ Virtual Test Lab Staged Application Instance System Behavior Cost § Only manage the func1onality you need § Only manage data you need § Low-‐Risk “disposable instance” that can be accessed at any1me § Drama1cally reduce test environment setup and management Service Virtualization
- 21. 6/29/15 19 Paraso& Proprietary and Confiden1al 37 Paraso& Proprietary and Confiden1al 37 Service Virtualiza1on Examples Mobile Applica1on development and extension Agile/Parallel development limited by system dependencies Capacity Constrained staged environments Limited access to mainframes, ERPs, or 3rd party systems Test data management for complex transac1ons Parasoft Virtualize is used throughout the SDLC to speed access to and reduce the costs of managing development and test environments Paraso& Proprietary and Confiden1al 38 Paraso& Proprietary and Confiden1al 38 The Test Environment Challenge Test environment access is outside the control of development and test leaving gaps in the process IT Operations Parallel development delays… Need simple, realis1c access to dependent components… Too much 1me wai1ng for access…Need reliable test data Need a realis1c test environment easy to maintain Scheduling Configuration Access Limits Dependent Applications Staged Assets 3rd Party Assets Virtual Environments Hyper Visor App App App
- 22. 6/29/15 20 Paraso& Proprietary and Confiden1al 39 Paraso& Proprietary and Confiden1al 39 The Test Environment Challenge Test environment access is outside the control of development and test leaving gaps in the process IT Operations Scheduling Configuration Access Limits Dependent Applications Staged Assets 3rd Party Assets Virtual Environments Hyper Visor App App App Create, Manage, Provision Paraso& Proprietary and Confiden1al 40 Paraso& Proprietary and Confiden1al 40 Environment – “Passive” or “Ac1ve” Paraso9’s architecture is built for “passive” or “ac've” environment modeling and maintenance PVA Endpoint or App Component Protocol Correlated Tests Associated Environments Message Payload Response Logic Performance Profile Test Data Management • Native Support • Extension API • Marketplace • Environment Manager • ALM APIs • CR/CD APIs • Marketplace • Environment Manager • Marketplace • Native Support • Extension API • Marketplace • Modeling UI • Log Files API • Marketplace • Native Support • APM API • Marketplace • Modeling UI • TDM API • Marketplace
- 23. 6/29/15 21 Paraso& Proprietary and Confiden1al 41 Paraso& Proprietary and Confiden1al 41 Value of Service Virtualiza1on § Increase Access to “Trustworthy” Test Environments § Provide a complete test environment § Provide access 24/7 § Flexibility provides a plahorm for extending test scenarios, coverage § Reduce Opera1onal Expenditures § Significantly reduce test environment configura1on 1me § Simplify test environment access § Remove the bodleneck of test environment setup § Reduce Capital Expenditures § So&ware assets can be run anywhere § Significantly reduce bandwidth demand § Reduce dependency on staged environments Service Virtualization drives better software quality while reducing CapEx and OpEx Paraso& Proprietary and Confiden1al 42 Paraso& Proprietary and Confiden1al 42
- 24. 6/29/15 22 Paraso& Proprietary and Confiden1al 43 Paraso& Proprietary and Confiden1al 43 Paraso& Service Virtualiza1on Wayne Ariola Chief Strategy Officer Parasoft Corporation Wayne.ariola@Parasoft.com Questions? Paraso& Proprietary and Confiden1al 44 Paraso& Proprietary and Confiden1al 44 Virtual Lab Management Strengths Weaknesses § “Low” Risk § Reduce CAPEX § Reduce Server Sprawl § Manage Environment Complexity § Uniformity for Global Teams § Opera1onal Efficiency § Must Have Access § Permissions to Control § Configura1on Permuta1ons § Test Data § Performance § Hardware Capacity Constraints
- 25. 6/29/15 23 Paraso& Proprietary and Confiden1al 45 Paraso& Proprietary and Confiden1al 45 The Value/ROI Drivers – Human Virtual Lab Service Virtualiza1on § Reduce OPEX § Eliminate scheduling delays associated with staged test labs § Eliminate ‘some’ configura1on delays § Eliminates configura1on 1me by delivering more granular ‘service’ versus system § Eliminates scheduling delays § Understand Environment Complexity § Gives a good view of inter-‐ rela1onship of applica1ons within geo/poli1cal control § Gives a complete view of all system dependencies § Opera1onal Efficiency § Reduces demand on test lab resources § Reduces delays of scheduling, setup and re-‐configura1on § Eliminates delays associated with § System configura1on § Data management § Performance configura1on § Setup and tear down Paraso& Proprietary and Confiden1al 46 Paraso& Proprietary and Confiden1al 46 The Value/ROI Drivers – Physical Virtual Lab Service Virtualiza1on § Reduce CAPEX and server sprawl § Versus physical test lab will significantly reduce CAPEX § Can reduce incremental CAPEX due to physical test capacity constraints § Can reduce demand on “virtualized” infrastructure resources § Eliminate physical test labs (Careful!) § Can eliminate physical test lab BUT be careful what you want to virtualize due to compliance and data privacy concerns § Eliminates the need for incremental increases in physical test labs or virtual lab capacity
- 26. 6/29/15 24 Paraso& Proprietary and Confiden1al 47 Paraso& Proprietary and Confiden1al 47 The Value/ROI Drivers – Project/Process Virtual Lab Service Virtualiza1on § Time to Market (Careful!) § Increases 1me to market by providing access to “produc1on-‐ like” environments within geo/ poli1cal control § Increases 1me to market by providing more complete access § Provides a net new environment earlier in the process § Access any1me or anywhere § Good access to completed and built applica1ons within control § Access to both complete and incomplete components § Manage and understand Environment § Good view of applica1ons managed within resource pool § Much broader view of total system interdependencies § Understanding of 3rd party impacts § Defect Reproduc1on § Captures great informa1on for applica1ons under control § Extends informa1on by including components outside of control § Beder Tes1ng § Increased code coverage § Increase test efficiency § Increased test coverage § Increased # and quality of tests § Increase test efficiency § Increased test coverage § Increased # and quality of tests