Automated Mobile UI Testing Fragility: An Exploratory Assessment Study on Android
1 like199 views
This document discusses automated mobile UI testing fragility in Android applications. It explores three ways of testing Android UIs: manual tests, automated unit tests, and automated instrumented tests. Challenges of Android UI testing include different device configurations and dynamic application swapping. GUI automated tests are considered fragile if they fail due to interface changes rather than functionality changes. The study examines an open-source Android app across 336 releases to analyze how many tests need modification for new releases and the causes of fragility. Up to 75% of code-based and 100% of image-based tests required adaptation. Seven fragility causes were characterized, with all causing failures. Future work areas include better characterizing and preventing fragility.
Automated Mobile UI Testing Fragility: An Exploratory Assessment Study on Android
- 1. Automated Mobile UI Testing Fragility An Exploratory Assessment Study on Android Riccardo Coppola, Emanuele Raffero, Marco Torchiano Politecnico di Torino, Dipartimento di Automatica e Informatica
- 2. Android UI Testing Three ways of testing Android User Interfaces: • Manual tests; • Automated Unit Tests; • Automated Instrumented Tests.
- 3. Android UI Testing - Challenges • Different configurations of display sizes, layouts, pixel densities; • Different input modalities; • Dynamic swapping of applications; • Fast evolution of the operative system, and therefore of the app features.
- 4. Android UI Testing - Fragility A GUI automated test is said to be fragile when: • It fails when the application evolves; • The failure is not due to the modification of the functionalities under test, but to changes in the interface arrangement and definition. When a fragility occurs, extra effort is needed to: • Verify that no regression has occurred; • Modify the failing tests to adapt them to the changed UI.
- 5. Research questions RQ1 How many tests have to be modified to adapt to subsequent releases of the same app? RQ2 Which are the intrinsic causes of the fragilities of UI tests?
- 6. Selected Application • Open-source project; • 120k LOC; • 336 releases; • Stable graphic interface.
- 7. Testing Tools: Code-Based Use of ids and content to identify UI elements.
- 8. Testing Tools: Image Recognition Use of captures to identify UI elements.
- 9. Test Suite Successful Authentication Send message Reply to a message Delete a message Add user account Delete user account Delete account data Restore account data Export account settings Import account settings v2.102 v2.995 v3.309 v3.993 v4.804 v5.010 6 releases 10 features
- 10. Code-based test suites: Evolution Test suite evolution Tests not failed
- 11. Image-recognition test suites: Evolution Test suite evolution Tests not failed
- 12. Fragilities Classification • Identifier Change; • Text Change;
- 13. Fragilities Classification • Deletion or Relocation; • Physical Buttons use;
- 14. • Graphics Change; • Activity Flow Change; • Execution Time Variability. Fragilities Classification
- 15. Fragilities Occurrence Causes of fragilities in code-based failed test cases: Cause v2.995 v3.993 v4.804 Text Change 0/2 2/2 3/6 Identifier Change 0/2 0/2 3/6 Deletion or Relocation 2/2 0/2 3/6 Physical Buttons use 0/2 0/2 3/6
- 16. Findings RQ1 How many tests have to be modified to adapt to subsequent releases of the same app? Up to 75% of code-based test scripts, and up to 100% of image recognition test scripts, had to be adapted.
- 17. Findings RQ2 Which are the intrinsic causes of the fragilities of UI tests? Seven potential causes of fragility have been characterized. All of them caused fragilities in the test suite developed.
- 18. Future Work • Test Fragility Characterization: – Survey of existing tools; – Frequency of tool adoption; – How much and often tests are changed; – Classification of fragility causes; – Frequency of occurrence of specific causes.
- 19. Future Work • Guidelines definition: – Set of DOs and DONTs; • Test Code Analysis; • Test Code Refactoring: – Set of refactoring recipes; – Plug-in for an IDE.
- 20. Questions?
- 21. Thank You