Pekka_Aho_Complementing GUI Testing Scripts - Testing Assembly 2022.pdf
0 likes98 views
This document discusses how monkey testing can complement GUI testing scripts. It describes how monkey testing generates tests at runtime without scripts, testing randomly to find errors anywhere in the system. Monkey testing finds different failures than scripts and covers more code. The document also introduces TESTAR, an open source monkey testing tool that detects GUI states, derives possible actions, selects actions randomly or through machine learning, executes actions, and checks for failures through implicit oracles. TESTAR can be enhanced with capabilities like learning models of the application to systematically explore the GUI.
![15
• Oracles for suspicious text and outputs
• Easy to specify using a regular expression
.*[eE]rror.*|.*[eE]xcepti[o?]n.*](https://image.slidesharecdn.com/pekkaahocomplementingguitestingscripts-testingassembly2022-220926123428-06d10d26/85/Pekka_Aho_Complementing-GUI-Testing-Scripts-Testing-Assembly-2022-pdf-15-320.jpg)
Pekka_Aho_Complementing GUI Testing Scripts - Testing Assembly 2022.pdf
- 1. Complementing GUI Testing Scripts with Smart Monkey Testing Dr. Pekka Aho Open University of the Netherlands
- 2. Automated GUI testing scripts Traditionally GUI testing scripts are… • designed manually • captured with C&R tools, or • manually written test scripts • static until maintenance required • maintained manually • for regression testing • checking the correctness of a selected set of functionality and properties 2 Test step 1 Check detail A Check detail B Test step 2 Check detail C
- 3. Monkey testing In monkey testing, the tests are… • generated during run-time • no test scripts to maintain • dynamic, randomness often included • multiple executions could yield different results • closer to robustness testing than functional testing • trying to find incorrectness anywhere in the system under testing 3
- 4. Monkey testing complements scripts GUI testing scripts and monkey testing… • find different kind of failures • cover different parts of code 4 Tool New bugs Known bugs Selenium 2 high 1 high TESTAR 2 high 4 low - Tool % IC % LC % MC Espresso 43,9 43,4 45,9 TESTAR 41,0 40,7 40,8 combined 52,3 52,1 52,3
- 5. Monkey testing complements scripts 5 GUI testing scripts Monkey testing
- 6. Random monkey testing with only the default TESTAR test oracles can be surprisingly efficient…
- 7. Cap Gemini/ProRail • Academia/industry cooperation • Web application: Java based (JEE6) with 12,263 LOC • System for managing the assignment of train platforms • Existing test approach: manual with 100 test cases. • After a change in the application: all 100 test cases were executed as regression tests TESTAR Manual Preparation 44 hour 43 hour Testing (51 hour) 6 hour Post testing 5 hour 2 hour Critical faults 4 0 Functional coverage 80% 73%
- 8. How to make the monkey smarter The monkey testing tool can be enhanced with… • a memory (model) to remember where it has been and what it has done • pre-defined information about the SUT • how to reach in every part of the SUT (e.g., login) • SUT-specific test inputs or test oracles • improved action selection algorithms • AI or ML to learn how to reach a certain goal • using the memory (model) to systematically explore • self-coordinating parallel GUI exploration 8
- 9. Open source TESTAR tool • https://testar.org/ • https://github.com/TESTARtool/TESTAR_dev/ • Many ongoing research projects with industrial collaboration
- 10. TESTAR - How does it work 6. Wait for the GUI to update 5. Execute the selected action 4. Select an action to execute using an algorithm 3. Derive possible actions that the end user could do 2. Check the test oracles on the state to find failures 1. Detect the current state of the GUI
- 11. 1. Detect the current state of the GUI 11 TESTAR uses a widget tree as the state of the GUI, including all widgets + their properties. Requires programmatical access to the GUI information: • Accessibility API for Windows apps (UIAutomation) • Webdriver for web applications • Appium for mobile apps 1. Detect the current state of the GUI
- 12. Oracles Mechanism to determine whether a failure was found (or not) In TESTAR: • Oracles give verdicts about each state of the GUI 2. Check the test oracles on the state to find failures
- 13. What kind of errors can be found with TESTAR?
- 14. 100% Automated implicit oracles • Crashes (unexpected close of the SUT) • Hangs (SUT becomes unresponsive)
- 15. 15 • Oracles for suspicious text and outputs • Easy to specify using a regular expression .*[eE]rror.*|.*[eE]xcepti[o?]n.*
- 16. 16 TESTAR provides a Java interface for implementing other oracles, for example: • Checking results from a database or other external source • Triggering specific oracles in a specific GUI state • Implementing platform specific error checking • E.g. checking JavaScript console for error messages
- 17. 2. Derive actions • . • . • . 3. Derive possible actions that the end user could do • Click and type (text, ‘Find what’-field) • Click and type (text, ‘Replace with’-field) • Click (Find next) • Click (Replace) • Click (Replace all) • Click (Cancel) • Click (Match case) • Click (Wrap round) • (Close X is filtered, shown with a grey dot) By default, TESTAR derives the following GUI actions: • Left mouse click • Left mouse click and type text • Move scrollbar But other actions, like right mouse clicks, double clicks, drag and drop, etc, are also supported
- 18. 3. Select an action using an algorithm 4. Select an action to execute using an algorithm TESTAR supports various action selection algorithms: • Random selection • Random with various prioritization algorithms • Various machine learning algorithms (Q learning, SARSA, etc) • State model learning with systematic random-order GUI exploration TESTAR provides a Java interface to implement new action selection algorithms.
- 19. oracles • Click (OK) • Click(File) • Click(Edit) • Click(Format) • Click (View) • Click (Help) • lick (☐) 5. Execute the selected action 6. Wait for the GUI to update By default, TESTAR executes the GUI actions using system mouse and keyboard • Testing from the end user perspective, including the GUI • For Windows apps, TESTAR monitors the SUT process • For web and mobile apps, Webdriver and Appium are used
- 20. • email: pekka.aho@ou.nl • web: http://www.testar.org/ • github.com/TESTARtool/TESTAR_dev Any questions?