On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study
1 like367 views
The document presents an empirical study on the prevalence of test smells in automatically generated test code, revealing that 83% of analyzed JUnit classes contain such smells. Key findings indicate that common test smells include 'assertion roulette' (54%) and 'test code duplication' (33%). The study concludes that while automated tools focus on maximizing code coverage, they often neglect code quality, emphasizing the need for future research into improving the generation of test code fixtures.
![Automatic Generation ofTest Code
[Arcuri and Fraser - SSBSE’14]
Effectiveness of Whole Suite
Test Case Generation
[Panichella et al. - ICSE’16]
Impact ofTest Case Summaries
on Bug Fixing Performance
[Shamshiri et al. - ASE’15]
Effectiveness of Generated
Test Cases on Effectiveness](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-3-320.jpg)
![Automatic Generation ofTest Code
[Rojas et al. - ISSTA’15]
Usability of Automatic
GenerationTools in Practice](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-4-320.jpg)
![Test Smells inTest Code
[Van Deursen et al. - XP 2001]
“Test Smells represent a set of a
poor design solutions to write tests ”
11test smells related to the way
developers write test fixtures
and test cases](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-6-320.jpg)
![Who cares aboutTest Smells?
Test Cases can be re-generated!
Developers modify and remove test code
Developers add tests when automatic tools leave
uncovered branches
Developers combine generated with manually written tests
[Rojas et al. - ISSTA’15]
Usability of Automatic GenerationTools in Practice
True
BUT](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-15-320.jpg)
![Empirical Study Design
8test smell types
“RefactoringTest Code”
[Van Deursen et al. - XP 2001]](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-17-320.jpg)
![Empirical Study Design
110software projects
8test smell types
[Fraser and Arcuri -TOSEM 2014]
“A Large Scale Evaluation of Automated Unit
Test Generation using Evosuite”
“RefactoringTest Code”
[Van Deursen et al. - XP 2001]](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-18-320.jpg)
![Empirical Study Design
110software projects
8test smell types
[Fraser and Arcuri -TOSEM 2014]
“A Large Scale Evaluation of Automated Unit
Test Generation using Evosuite”
“RefactoringTest Code”
[Van Deursen et al. - XP 2001]](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-19-320.jpg)
![Empirical Study Design
16,603JUnit classes
[Fraser and Arcuri -TOSEM 2014]
“A Large Scale Evaluation of Automated Unit
Test Generation using Evosuite”](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-20-320.jpg)
![Data Extraction
test_method1()
test_method2()
…
test_methodN()
Test Suite
[Bavota et al. - EMSE 2015]
“AreTest Smells Harmful? An Empirical Study”
Test Smell Detector](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-21-320.jpg)
![Data Extraction
test_method1()
test_method2()
…
test_methodN()
Test Suite
[Bavota et al. - EMSE 2015]
“AreTest Smells Harmful? An Empirical Study”
Test Smell Detector
75%
precision
100%
recall
Sample size: 378 JUnit classes](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-22-320.jpg)
![RQ2:WhichTest Smells Occur More Frequently
in Automatically GeneratedTest Classes?
Results of the Study
!
Assertion Roulette
[Panichella et al. - ICSE’16]
The Impact ofTest Case Summaries on Bug Fixing Performance
An Empirical Investigation
These problems have a huge impact on
developers’ ability to find faults](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-36-320.jpg)
![Summarizing
Lesson 1: Current implementations of search-based algorithms
for automatic test case generation do not consider code quality,
increasing the probability to introduce smells!
NB: Considering test code quality is important not only to
avoid the introduction of smells, but also because the
coverage can be increased!
[F. Palomba,A. Panichella,A. Zaidman, R. Oliveto,A. De Lucia - ISSTA’16]
AutomaticTest Case Generation:What IfTest Code Quality Matters?](https://image.slidesharecdn.com/sbst16-160526212019/85/On-the-Diffusion-of-Test-Smells-in-Automatically-Generated-Test-Code-An-Empirical-Study-44-320.jpg)
On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study
- 1. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Fabio Palomba*, Dario Di Nucci*,Annibale Panichella+, Rocco Oliveto°,Andrea De Lucia* * University of Salerno, Italy +Delft University ofTechnology,The Netherlands ° University of Molise, Italy SBST 2016 May, 16th 2016 Austin,TX, USA
- 2. Automatic Generation ofTest Code method1() method2() … methodN() Production Class test_method1() test_method2() … test_methodN() Test Suite
- 3. Automatic Generation ofTest Code [Arcuri and Fraser - SSBSE’14] Effectiveness of Whole Suite Test Case Generation [Panichella et al. - ICSE’16] Impact ofTest Case Summaries on Bug Fixing Performance [Shamshiri et al. - ASE’15] Effectiveness of Generated Test Cases on Effectiveness
- 4. Automatic Generation ofTest Code [Rojas et al. - ISSTA’15] Usability of Automatic GenerationTools in Practice
- 5. Automatic Generation ofTest Code What about the characteristics of test code produced by such tools?
- 6. Test Smells inTest Code [Van Deursen et al. - XP 2001] “Test Smells represent a set of a poor design solutions to write tests ” 11test smells related to the way developers write test fixtures and test cases
- 7. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm(); jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0); assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }
- 8. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm(); jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0); assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } The test method checks the production method isGround()
- 9. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm(); jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0); assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } But also the production method isVariable()
- 10. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm(); jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0); assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } This is an EagerTest, namely a test which checks more than one method of the class to be tested, making difficult the comprehension of the actual test target.
- 11. Test Smells inTest Code A test case is affected by a Resource Optimism when it makes assumptions about the state or the existence of external resources, providing a non-deterministic result that depend on the state of the resources. An Assertion Roulette comes from having a number of assertions in a test method that have no explanation. If an assertion fails, the identification of the assert that failed can be difficult.
- 12. Who cares aboutTest Smells? Test Cases can be re-generated!
- 13. Who cares aboutTest Smells? Test Cases can be re-generated! True
- 14. Who cares aboutTest Smells? Test Cases can be re-generated! True BUT
- 15. Who cares aboutTest Smells? Test Cases can be re-generated! Developers modify and remove test code Developers add tests when automatic tools leave uncovered branches Developers combine generated with manually written tests [Rojas et al. - ISSTA’15] Usability of Automatic GenerationTools in Practice True BUT
- 16. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Empirical Study Design
- 17. Empirical Study Design 8test smell types “RefactoringTest Code” [Van Deursen et al. - XP 2001]
- 18. Empirical Study Design 110software projects 8test smell types [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “RefactoringTest Code” [Van Deursen et al. - XP 2001]
- 19. Empirical Study Design 110software projects 8test smell types [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “RefactoringTest Code” [Van Deursen et al. - XP 2001]
- 20. Empirical Study Design 16,603JUnit classes [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite”
- 21. Data Extraction test_method1() test_method2() … test_methodN() Test Suite [Bavota et al. - EMSE 2015] “AreTest Smells Harmful? An Empirical Study” Test Smell Detector
- 22. Data Extraction test_method1() test_method2() … test_methodN() Test Suite [Bavota et al. - EMSE 2015] “AreTest Smells Harmful? An Empirical Study” Test Smell Detector 75% precision 100% recall Sample size: 378 JUnit classes
- 23. Research Questions RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? ?
- 24. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? ? Research Questions
- 25. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? RQ3:WhichTest Smells Co-OccurTogether? ? Research Questions
- 26. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? RQ3:WhichTest Smells Co-OccurTogether? RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? ? Research Questions
- 27. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Analysis of the Results
- 28. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !13,791smelly JUnit classes
- 29. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !83%of the JUnit classes analyzed
- 30. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !RQ1 Test Smells are highly diffused in the automatically generated test suites
- 31. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54%
- 32. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54% Test Code Duplication 33%
- 33. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54% Test Code Duplication EagerTest 33% 29%
- 34. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !public void test8 () throws Throwable { Document document0 = new Document(); assertNotNull(document0); document0.procText.add((Character) ”s”); String string0 = document0.stringify(); assertEquals (“s”, document0.stringify()); assertNotNull(string0); assertEquals(“s”, string0); } Assertion Roulette
- 35. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Assertion Roulette What is the behavior under test? Are the generated assertions valid?
- 36. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Assertion Roulette [Panichella et al. - ICSE’16] The Impact ofTest Case Summaries on Bug Fixing Performance An Empirical Investigation These problems have a huge impact on developers’ ability to find faults
- 37. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !public void test8 () throws Throwable { GenericProperties generic0 = new GenericProperties(); boolean boolean0 = generic0.isValidClassname(); … } Test Code Duplication public void test9 () throws Throwable { GenericProperties generic0 = new GenericProperties(); boolean boolean0 = generic0.isValidClassname(); … }
- 38. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Test Code Duplication This problem can be avoided by generating test fixtures!
- 39. Results of the Study ! RQ3:WhichTest Smells Co-OccurTogether? Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest
- 40. Results of the Study ! RQ3:WhichTest Smells Co-OccurTogether? Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest Automatic tools have as main goal that of maximize coverage, without considering test code quality
- 41. RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? Results of the Study !Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case!
- 42. RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? Results of the Study !Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case! The higher the LOC of the JUnit class, the higher the probability to produce a smelly test case!
- 43. Summarizing Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells!
- 44. Summarizing Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells! NB: Considering test code quality is important not only to avoid the introduction of smells, but also because the coverage can be increased! [F. Palomba,A. Panichella,A. Zaidman, R. Oliveto,A. De Lucia - ISSTA’16] AutomaticTest Case Generation:What IfTest Code Quality Matters?
- 45. Summarizing Lesson 2:Automatic test case generation tools do not produce text fixtures during their computation, and this implies the introduction of several code clones in the resulting JUnit classes. Future research should spend effort in the automatic generation of test fixtures!
- 46. From now on… ? Challenge 1: EvaluatingTest Smells inTest Cases automatically generated by other tools
- 47. From now on… ? Challenge 1: EvaluatingTest Smells inTest Cases automatically generated by other tools Challenge 2: Defining new algorithms able to solve the design problems analyzed (e.g., test fixtures).
- 48. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Fabio Palomba*, Dario Di Nucci*,Annibale Panichella+, Rocco Oliveto°,Andrea De Lucia* * University of Salerno, Italy +Delft University ofTechnology,The Netherlands ° University of Molise, Italy SBST 2016 May, 16th 2016 Austin,TX, USA