Test Automation Strategies and Frameworks: What Should Your Team Do?
- 2. ! ! ! ! !!!!! ! !!!! Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that uses agile methods to accelerate the delivery of secure, reliable software. Gene is an experienced software developer who focuses on continuous integration, static code analysis, automation, and any tool he can find to do his work for him. For the past few years, he has been on a team that is bringing a continuous delivery process into the US Department of Defense, showing that higher quality software can be delivered quicker and with more security by using agile techniques. Gene Gotimer Coveros, Inc.
- 3. 6/25/15& 1& © Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategies and Frameworks What Should Your Team Do? Max Saperstone © Copyright 2015 Coveros, Inc. All rights reserved. Gene Gotimer Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that uses agile methods to accelerate the delivery of secure, reliable software. Gene is an experienced software developer who focuses on continuous integration, static code analysis, automation, and any tool he can find to do his work for him. For the past few years, he has been on a team that is bringing a continuous delivery process into the US Department of Defense, showing that higher quality software can be delivered quicker and with more security by using agile techniques.
- 4. 6/25/15& 2& © Copyright 2015 Coveros, Inc. All rights reserved. About Coveros • Coveros helps organizations accelerate the delivery of secure, reliable software • Our consulting services: – Agile software development – Application security – Software quality assurance – Software process improvement • Our key markets: – Financial services – Healthcare – Defense – Critical Infrastructure Development Capabilities © Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Planning
- 5. 6/25/15& 3& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Planning ● SMART ○ Specific (What to accomplish, why, who, where – specific reqs) ○ Measurable (How much/many – Quantifiable way to determine completeness) ○ Achievable (answers question of “How will this be accomplished?”) ○ Relevant (Valued/Worthwhile, consider timing/priorities) ○ Time Bound (target completion) ● Some typical objectives: ○ Testing Consistency/Repeatability ○ Unattended test execution ○ Regression defect identification ○ Frequent Test Execution ○ Software Quality Improvement Test Automation Objectives © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Planning ● Usability ● Accuracy ● Integrity ● Interoperability and Compatibility ● Maintainability ● Availability ● Performance Areas to consider for Test Objectives
- 6. 6/25/15& 4& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Planning ● What is going to be considered in scope for testing? ● What will be out of scope? ● At which level of testing will we increase or decrease our scope to cover these? ● What is in scope for automation vs manual testing Test Automation Scope © Copyright 2015 Coveros, Inc. All rights reserved. ● Primarily only done for non- functional requirements, smoke tests, business process ● Acceptance test-driven development Story testing ● Test-driven development Unit testing Test Automation Planning Test Types to Include
- 7. 6/25/15& 5& © Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Framework Selection © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Framework Selection ● Frameworks provide test infrastructure for automating various types and levels of tests. ○ Capabilities for common automation requirements ○ Complexity Abstraction ○ Extend Coverage ○ Reduce Maintenance ● Framework Features ○ Test Driver ○ Object Repository Management ○ Data Management ○ Recovery Mechanisms What is a Framework?
- 8. 6/25/15& 6& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Framework Selection ● Record/Playback (Linear) ● Modular / Structured ● Data Driven ● Keyword Driven ● Action Based ● Hybrids Automation Framework Types © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Framework Selection ● There is no ‘Silver Bullet’ ● Tool Evaluation ○ Motivation ○ Project Technical constraints ■ OS/Language ■ Development IDE ■ Project Architecture ○ Team skillsets/expertise ○ Budget ○ Best tool fit selection Test Automation Tool Selection
- 9. 6/25/15& 7& © Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy Limited Time + Initial Effort required = MoSCoW method ● What MUST be Automated? ● What SHOULD be Automated? ● What COULD be Automated? ● What WON’T be Automated? Selecting Tests for Automation
- 10. 6/25/15& 8& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy ● Value vs Effort evaluation ● Frequency of Use ○ Repetitive tests that run for multiple builds ○ Tedious Manual Tasks ● Features that historically problematic/prone to defects ● Tests that are highly subject to human error ● Tests that require multiple data sets ● Tests that run on several different HW/SW platforms and configurations ● Tests that take a lot of effort and time when doing manual testing Determining what to Automate © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy ● Should data start empty, and be cleaned each time? ● What data is needed in QA for automated testing? What about manual testing? ● How can we keep QA in a state with known test data? ● How do we handle multiple users/automated tests potentially changing and needed the same data? ● Where is it acceptable to use production data? ● Should production data be sanitized before being used? ● Should production data be analyzed before being used? Data Management
- 11. 6/25/15& 9& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy ● Frameworks should include take screenshots methods which can be taken on any failed steps ○ These can also be included for additional information ● Automated frameworks should provide readable reports, and be used for documentation ● Results from frameworks should be automated to provide traceability into requirements ○ Multiple tools provide hooks into test management systems ○ Public APIs can allow automated updates for tooling Traceability through Automated Testing © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy ● Your automated tests should be treated as code ● When created smartly, steps are simple to find, update, and reuse ● Create a separate class for each area of functionality ● Instantiate important classes or objects in your base class Base Functionality Help FunctionalityLogin Functionality Library Functionality Best Practices
- 12. 6/25/15& 10& © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Strategy ● You can't test all possible combinations of parameters ● Using parameterized inputs to a method allows for a simple expansion of test suites ○ This allows one test to run with multiple data sets ○ Generally, these can run in parallel ● Multiple existing frameworks support parameterized testing ● Take your smoke and quicker executing tests and expand on them using parameters Parameterized Testing © Copyright 2015 Coveros, Inc. All rights reserved. Test Automation Frameworks ● Abbot ● Anteater ● Canoo WebTest ● Chorus ● Concordion ● Crosscheck ● DejaGnu ● Dogtail ● EMOS ● ItiN ● ItsNat ● JBehave ● Maveryx ● MozUnit ● Phoronix Test Suite ● QAliber ● Rasta ● Robot ● Robotium ● TestNG ● Selendroid ● WAAT What Testing Framework Might Be Right For Me?
- 13. 6/25/15& 11& © Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved. Questions