When is a project ready for Software Automation_NEW
- 1. When is a Project Ready for Software Automation? Michael Christesen Mat Krueper Ty Henson 1
- 2. Introduction • Michael Christesen • Mat Krueper • Ty Henson 2
- 3. Agenda Mike: • Automation Misconceptions • Quick Review of Manual vs Automation Concepts • Methodology and Automation • Tips for Successful Automation • An alternative way to measure the cost of Automation Mat: • Tips on selecting the right Automation tools • Key Performance Indicators for Automation • Return on Investment • Review of real-world gains through Automation 3
- 5. The Basics: Benefits of Manual Testing • Low Cost (Short term) • Effective at Finding Real User Issues • Flexible Graphic from leaseweblabs.com 5
- 6. The Basics: Difficulties of Manual Testing • (Some) Tests are Difficult to Run • (Some) Tests are Repetitive and Boring • Subject to Human Error • Repetitive – Each Test is Run by Hand 6
- 7. The Basics: Difficulties of Automation • Can be Expensive • Initially Time Intensive • Automated Tools have Limitations • Defect investigation can take longer Graphic from http://blog.valuelabs.com/ 7
- 8. Why would you Automate? • Historically, it can take 3-10 times longer to automate a test cases compared to running it manually. (Over the last 5 years, some tools have decreased this to just 1.5 times!) • So you need to execute the automated test over (at least) the same number of iterations to break even. • There are also hidden costs to consider: • Licensing • Infrastructure • Support • Training 8
- 9. The Basics: Benefits of Automation • Quick to Run • Cost Effective • Testers can focus on new features • Test results are available for all to see • Execute detailed, repetitive, data intensive tests – automatically! Graphic from sdtimes.com 9
- 10. The Basics: Automation and Methodology • Waterfall: • Automation is a NOT a side project…it’s a full time role! • Agile • You need to automate your Regression tests since the amount of Regression continually expands with each Sprint. • If a test is only performed once...don’t automate! • Includes: • Edge Cases • Exploratory tests Graphic from apicasystem.com 10
- 11. The Cost of Automation • Common Measurement: Most companies determine the cost of automation as the time (in hours) spent doing automation. • Alternative Measurement: Number of Bugs forgone... • Reason: The point of automation is to find more bugs running scripts. Bugs found are the value of automation. Maybe automation should be measured the same way? Graphic from Santexgroup.com 11
- 12. The Goal: Successful Automation • Suggestions: • Automate the ‘Right’ Test Cases • Test Early / Test Often • Define Quality Test Data • Design Automated Tests that can resist change • Selecting the right Automation Tool Graphic from qatestlab.com 12
- 13. Automate the ‘Right’ Test Cases • Repetitively Run Tests (aka Smoke or Acceptance tests) • Data intensive tests • Frequently Used Functionality • High Risk, Business-critical functionality • Tests that are difficult to run manually • Tests that are run on differing hardware/software platforms and configurations • Tests that take a lot of effort or time Graphic from qatestlab.com 13
- 14. Test Early / Test Often • Start as early in the process as possible • Could start with Automated Unit testing • Gradually build your Automation Suite • The more you test, the more bugs you find. • The earlier bugs are found the cheaper they are to fix! 14
- 15. Define Quality Test Data • With good data… • Writing and Maintaining Automated tests is easier. • With Data driven tests you can run more iterations and find more bugs! • Can extend existing Automated tests as new features are developed – without having to edit the actual automated test itself. • Stay proactive in updating/defining data. 15
- 16. Automation Approach • Keep Automated tests small and focused • Easy to debug • Easy to manage • Can be reused and shared • These can then be grouped into a larger Automated test suite organized by functional area. • With the right framework, users can create large/complex Automated tests that are also easy to maintain! Graphic from checkpointech.com 16
- 17. Design Automated Tests Resistant to Change • If possible, avoid using screen coordinates. • Componentize your test scripts • Ensure that your development team uses unique names for each control and object! 17
- 18. How long do Automated Tests Last? • At some point, the application will change and the script will break. • If you are using GUI Capture/Playback a major revision to the UI may break your tests. • If the script breaks before the automation effort was repaid, it is better off as a manual test. 18
- 19. Selecting the ‘‘Right” Automation Tool Can be used by business users No programming skills required Simple Automate 60-70% of core business processes in 90 days or less Rapid return on investment Fast When underlying applications change, automation continues to work Simple maintenance of your business process portfolio Resilient True end-to-end business process validation covering all major technologies and platforms Functional, performance, role-based and risk-based validation - 24/7/365 coverage Complete 19
- 20. Choose Technology That Covers Everything ERP CRM SRM SCM PLM Enterprise Portal SAP BusinessObjectsTM SAP HANA® Mobile EPM (BPC) Adobe Flex 20
- 21. KPIs for Investments in Automation Infrastructure Here are 5 KPIs that should be considered when evaluating the ROI of projects in automated business process validation: • Acceleration in project timelines (deployment time for new technologies) • Length of test cycles • Reduction in customer-impacting defects • Hours of reduction in staff effort • Team throughput (number of projects delivered with the same resources) 21
- 22. Proven Business Benefits Optimized Business Processes Efficient discovery, deeper understanding Shorter Project Timelines Faster innovation Higher Quality Execution Reduced technology risk, fewer defects Lower Costs Improved operational efficiency 22
- 23. IDC Worksoft Customer Survey and ROI Analysis Demographics $11.4M Net Present Value of Worksoft Certify® 1 Year Payback Period 23
- 24. Proven
- 26. Reduction in Average Hours of Effort per Organization Surveyed 54,844 Hours Saved Annually 26
- 27. Process Owner Subject Matter Experts (Business & Technical) System Administrator / BASIS Business Analyst Database Administrator (DBA) Staff for Success 27 Supporting Roles: Executive Sponsor Note: Dashed line indicates part-time resource Automation Engineer(s) Automation Specialists & Testers Project Owner Project Manager Business/ Functional Lead(s) 27
- 28. Recap: Benefits of Automation • Time saved when execution is automatic • Enables More exhaustive testing • Manual testers can focus on new functionality • Increased confidence in the application 28
- 29. Thank you! Learn more at CQAA.org Questions? Mike: Mike.Christesen@nuevista.com Mat: mkrueper@worksoft.com Ty: thenson@worksoft.com 29