Does Refactoring of Test Smells Induce Fixing Flaky Tests?
0 likes565 views
The document explores the relationship between test smells and flaky tests in software development, addressing the causes of flakiness and potential refactoring strategies to mitigate it. Through empirical analysis, it finds that refactoring test smells can significantly reduce the occurrence of flaky tests. The study suggests a need for further investigation into the adoption of refactoring practices among developers and the development of automated tools for this purpose.
Does Refactoring of Test Smells Induce Fixing Flaky Tests?
- 1. Does Refactoring of Test Smells Induce Fixing Flaky Tests? Fabio Palomba, Andy Zaidman Delft University of Technology The Netherlands
- 2. Regression Testing Test cases form the first line of defense against the introduction of faults Pinto et al. “Understanding Myths and Realities of Test-Suite Evolution” FSE 2012
- 3. Regression Testing The entire team rely on them to decide whether to merge a pull request Gousios et al. “Work practices and challenges in pull-based development: The integrator’s perspective” ICSE 2015
- 4. Regression Testing Or even whether proceed with the deployment of the system Beller et al. “Ooops, my tests broke the build: An explorative analysis of Travis CI with GitHub” - MSR 2017
- 5. Developers’ productivity is dependent on both the ability to find real problems and the cost of diagnosing faults in a timely fashion Perez et al. “A Test-Suite Diagnosability Metric for Spectrum-based fault localization approaches” - ICSE 2017 Regression Testing
- 6. Vahabzadeh et al. “An Empirical Study of Bugs in Test Code” ICSME 2015 Even angels eat beans ! Test code can be affected by bugs as well One of the typical bugs affecting test suites is called “flakiness”
- 7. Flaky Tests Test cases that exhibit both a passing and failing result with the same code Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014
- 8. Flaky Tests Test cases that exhibit both a passing and failing result with the same code Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014 They might hide real bugs, increase maintenance costs, and reduce developers’ confidence
- 9. Flaky Tests Flaky Tests are relevant for practitioners and highly discussed on the web
- 10. Empirical Studies Likely causes behind test code flakiness
- 11. Empirical Studies Flaky Test Identification Likely causes behind test code flakiness Proposing solutions to fix flaky tests
- 12. Empirical Studies Flaky Test Identification They focus only on specific causes behind test code flakiness Likely causes behind test code flakiness Proposing solutions to fix flaky tests
- 13. Empirical Studies Flaky Test Identification Thus, only ad-hoc solutions are available Likely causes behind test code flakiness Proposing solutions to fix flaky tests
- 14. Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014 Problems faced by previous research only represent a part of the whole story A Deeper Analysis
- 15. Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014 Problems faced by previous research only represent a part of the whole story A Deeper Analysis A deeper analysis of possible fixing strategies of other root causes is still missing
- 16. Studying Test Smells Van Deursen et al. “Refactoring Test Code” XP 2001 Symptoms of poor design or implementation choices in test code
- 17. Studying Test Smells Symptoms of poor design or implementation choices in test code Van Deursen et al. “Refactoring Test Code” XP 2001 Resource Optimism is a test that makes optimistic assumption about the state or the existence of an external resource
- 18. Studying Test Smells Symptoms of poor design or implementation choices in test code Van Deursen et al. “Refactoring Test Code” XP 2001 Indirect Testing is a test that exercises different classes with respect to the corresponding production class
- 19. Studying Test Smells Symptoms of poor design or implementation choices in test code Van Deursen et al. “Refactoring Test Code” XP 2001 Test Run War is a test that allocates resources that are also used by other methods, possibly causing interferences
- 21. What are the causes of test flakiness? Research Questions ?
- 22. What are the causes of test flakiness? To what extent can flaky tests be explained by the presence of tests smells? Research Questions ?
- 23. What are the causes of test flakiness? To what extent can flaky tests be explained by the presence of tests smells? To what extent does refactoring of test smells help in removing flaky tests? Research Questions ?
- 24. Software Projects 18open-source systems randomly selected from Github
- 25. Detecting Test Smells Bavota et al. “Are Test Smells Really Harmful?” An Empirical Study EMSE Palomba et al. “On the Diffuseness of Test Smells in Automatically Generated Test Code” SBST 2016 The detector exploits the definition of the smells to detect them The detector has a precision of 88% and a recall of 100%
- 26. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 27. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 28. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 29. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 30. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 31. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 32. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 33. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 34. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 35. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 36. Detecting Flaky Tests If the output of a test method was different at least once We ran JUnit class ten times
- 37. Causes of Test Flakiness We manually linked each flaky test onto one of the 10 root causes defined in the taxonomy by Luo et al. Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014
- 38. Causes of Test Flakiness We manually linked each flaky test onto one of the 10 root causes defined in the taxonomy by Luo et al. Luo et al. “An Empirical Analysis of Flaky Tests” FSE 2014 JAVA LOG Source code of the test Exceptions thrown
- 39. Causes of Test Flakiness Async Wait A test method making an asynchronous call and that does not wait for the result of the call. 27%
- 40. The test method does not properly acquire or release one or more to its resources IO issue Causes of Test Flakiness 22%
- 41. Different threads interact in a non- desirable manner Concurrency Causes of Test Flakiness 17%
- 42. Causes of Test Flakiness 11% Test Ordering Network 10% Ordering of tests execution Network performance
- 43. Test Smells vs Flaky Tests Test Smells Flaky Tests
- 44. Test Smells vs Flaky Tests 61%
- 45. 61% Test Smells vs Flaky Tests How many of them are casual co-occurrences?
- 46. Test Smells vs Flaky Tests 61% We manually identified the flakiness-inducing test smells
- 47. Test Smells vs Flaky Tests 61% We manually identified the flakiness-inducing test smells A Resource Optimism was casually related to a test case if the flakiness was due to issues in the management fo external resources
- 48. Test Smells vs Flaky Tests 54%
- 49. Test Smells vs Flaky Tests 54% Resource Optimism IO issue Network
- 50. Test Smells vs Flaky Tests 54% Resource Optimism IO issue Network Indirect Testing Test Ordering
- 51. Test Smells vs Flaky Tests 54% Resource Optimism IO issue Network Indirect Testing Test Ordering Test Run War Concurrency
- 52. The Role of Refactoring We manually refactored according to the guidelines defined by Van Deursen et al. Van Deursen et al. “Refactoring Test Code” XP 2001
- 53. The Role of Refactoring We re-ran the flaky tests identification We re-ran the test smell detection
- 54. The Role of Refactoring 100% of the test code refactored did not present a test smell anymore
- 55. The Role of Refactoring 100% of the test code refactored did not present a flaky test anymore
- 56. The Role of Refactoring 54% of the total flaky tests were removed by means of refactoring
- 58. Future Research Agenda Extending the Study Understanding whether refactoring of test smells is actually adopted by developers as flaky test fixing strategy Tools for Automated Refactoring of Test Smells
- 59. Does Refactoring of Test Smells Induce Fixing Flaky Tests? Fabio Palomba, Andy Zaidman Delft University of Technology The Netherlands