Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities
0 likes537 views
This study revisits the analysis of test smells in automatically generated tests conducted by Grano et al. The authors find: 1) Through manual analysis of 100 test suites, the prevalence of test smells is lower than what detection tools reported in the previous study. 2) Detection tools have high false positive and false negative rates when identifying test smells in generated tests. They incorrectly flagged tests as containing smells like Mystery Guest and Resource Optimism. 3) Detection tools failed to identify instances of smells like Sensitive Equality and Indirect Testing that were found during manual analysis. 4) Relying solely on detection tools can lead to incorrect conclusions about test quality. Human validation of tool warnings is important.
![Related Work
2
[Grano et al., JSS 2019]
JTExpert
Test Case Generation Tools
Test Smell Detection Tool from previous
work [EMSE 2015]
GPD
Grano Palomba Di Nucci](https://image.slidesharecdn.com/icsme2020-201003083341/85/Revisiting-Test-Smells-in-Automatically-Generated-Tests-Limitations-Pitfalls-and-Opportunities-2-320.jpg)
![Related Work
3
[Grano et al., JSS 2019]
Main Results
81%
GPD precision in detecting test
smells (100% recall)
The tests [by EvoSuite] are scented since the
beginning as "crossover and mutation operations
[…] do not change the structure of the tests
of the JUnit test suites by
EvoSuite contain test smells
88%
Threats To Validity
Warnings raised by GPD are not manually
validated
EvoSuite was misconfigured:
- Old search algorithm
- Tests and Assertions are not minimization
Mutation and crossover alter the test
structure by adding/removing statements
[Arcuri and Fraser, TSE 2012]](https://image.slidesharecdn.com/icsme2020-201003083341/85/Revisiting-Test-Smells-in-Automatically-Generated-Tests-Limitations-Pitfalls-and-Opportunities-3-320.jpg)
![RQ2: Accuracy of Smell Detection Tools
8
Large False Positive Rate for Assertion
Roulette and Eager Tests
TABLE IV: Detection performance of different automated test smell detection tools for test cases generated by EVOSUITE.
FPR denotes the False Positive Rate and FNR is the False Negative Rate. The best values are highlighted in grey colour.
Test smell
Tool used by Grano et al. [6] TSDETECT calibrated by Spadini et al. [2]
FPR FNR Precision Recall F-measure FPR FNR Precision Recall F-measure
Assertion Roulette 0.72 0.00 0.22 1.00 0.36 0.05 0.50 0.67 0.5 0.57
Eager Test 0.53 0.05 0.33 0.95 0.49 0.05 0.45 0.73 0.55 0.63
Mystery Guest 0.12 — — — — 0.03 — — — —
Sensitive Equality 0.00 0.67 1.00 0.33 0.50 0.00 0.67 1.00 0.33 0.50
Resource Optimism 0.02 — — — — 0.02 — — — —
Indirect Testing 0.00 1.00 — 0.00 — — — — — —
@Test(timeout = 4000)
public void test07() throws Throwable {
ScriptOrFnScope s0 = new ScriptOrFnScope((-806),
(ScriptOrFnScope) null);
ScriptOrFnScope s1 = new ScriptOrFnScope((-330), s0);
s1.preventMunging();
s1.munge();
assertNotSame(s0, s1);
}
Fig. 2: Example of false positive for the tool used by Grano
et al. for Eager Test
@Test(timeout = 4000)
public void test00() throws Throwable {
Show show0 = new Show();
File file0 = MockFile.createTempFile("...");
Mystery Guest and Resource Optimism. For these two
types of smells, both detection tools raise several warnings.
However, they are all false positives by definition, as our gold
standard does not contain any instances of such smells. The
detection tools both annotate test methods that contain specific
strings or objects, such as: “File”, “FileOutputStream”
“DB”, “HttpClient” as smelly; however, EVOSUITE sep-
arates the test code from environmental dependencies (e.g.,
external files) in a fully automated fashion through byte-
code instrumentation [43]. In particular, it uses two mech-
anisms: (1) mocking, and (2) customized test runners. For
one, classes that access the filesystem (e.g., java.io.File)
(GPD)
Large False Negative Rate for Sensitive
Equality and Indirect Testing
GPD
Low False Positive Rate
Large False Negative Rate for most of
the test smells
TsDetector](https://image.slidesharecdn.com/icsme2020-201003083341/85/Revisiting-Test-Smells-in-Automatically-Generated-Tests-Limitations-Pitfalls-and-Opportunities-8-320.jpg)
![Limitations of Test Smell Detection Tools
10
GPD and TsDetector fail to detect instances of Sensitive Equality
@Test(timeout = 4000)
public void test62() throws Throwable {
SubstringLabeler.Match substringLabeler_Match0 = new SubstringLabeler.Match();
String string0 = substringLabeler_Match0.toString();
assertEquals("Substring: [Atts: ]", string0);
}
public void test62() throws Throwable {
SubstringLabeler.Match substringLabeler_Match0 = new SubstringLabeler.Match();
assertEquals("Substring: [Atts: ]", substringLabeler_Match0.toString());
}
Test generated
by EvoSuite but
not detected by
the two tools
This test would
be detected](https://image.slidesharecdn.com/icsme2020-201003083341/85/Revisiting-Test-Smells-in-Automatically-Generated-Tests-Limitations-Pitfalls-and-Opportunities-10-320.jpg)
Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities
- 1. Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn 1
- 2. Related Work 2 [Grano et al., JSS 2019] JTExpert Test Case Generation Tools Test Smell Detection Tool from previous work [EMSE 2015] GPD Grano Palomba Di Nucci
- 3. Related Work 3 [Grano et al., JSS 2019] Main Results 81% GPD precision in detecting test smells (100% recall) The tests [by EvoSuite] are scented since the beginning as "crossover and mutation operations […] do not change the structure of the tests of the JUnit test suites by EvoSuite contain test smells 88% Threats To Validity Warnings raised by GPD are not manually validated EvoSuite was misconfigured: - Old search algorithm - Tests and Assertions are not minimization Mutation and crossover alter the test structure by adding/removing statements [Arcuri and Fraser, TSE 2012]
- 4. Time To Revisit These Results 4
- 5. Our Study • RQ1: How widespread are test smells in automatically generated tests? • RQ2: How accurate are automated tools in detecting code smells in automatically generated tests? • RQ3: How well do test smells reflects real problem in test suites? 5 Manually analysing generated tests rather then relying on detection tools Assessing smell detection accuracy based on the manual oracle
- 6. Manual Analysis 6 100 Java classes from SF110 The same classes used by Grano at al. 100 Generated Test Suites Validator 2 Validator 3 Validator 4Validator 1 Validator 3 Validator 4 Validator 2Validator 1 Cross- validated Oracle
- 7. RQ1: Distributions of Test Smells 7 Eager Test Assertion Roulette Indirect Testing Sensitive Equality Mystery Guest Resource Optimism % Smelly Test Suites 0 25 50 75 100 Our results based on a manually validated dataset Results by Grano et al. (based on automated tools warning)
- 8. RQ2: Accuracy of Smell Detection Tools 8 Large False Positive Rate for Assertion Roulette and Eager Tests TABLE IV: Detection performance of different automated test smell detection tools for test cases generated by EVOSUITE. FPR denotes the False Positive Rate and FNR is the False Negative Rate. The best values are highlighted in grey colour. Test smell Tool used by Grano et al. [6] TSDETECT calibrated by Spadini et al. [2] FPR FNR Precision Recall F-measure FPR FNR Precision Recall F-measure Assertion Roulette 0.72 0.00 0.22 1.00 0.36 0.05 0.50 0.67 0.5 0.57 Eager Test 0.53 0.05 0.33 0.95 0.49 0.05 0.45 0.73 0.55 0.63 Mystery Guest 0.12 — — — — 0.03 — — — — Sensitive Equality 0.00 0.67 1.00 0.33 0.50 0.00 0.67 1.00 0.33 0.50 Resource Optimism 0.02 — — — — 0.02 — — — — Indirect Testing 0.00 1.00 — 0.00 — — — — — — @Test(timeout = 4000) public void test07() throws Throwable { ScriptOrFnScope s0 = new ScriptOrFnScope((-806), (ScriptOrFnScope) null); ScriptOrFnScope s1 = new ScriptOrFnScope((-330), s0); s1.preventMunging(); s1.munge(); assertNotSame(s0, s1); } Fig. 2: Example of false positive for the tool used by Grano et al. for Eager Test @Test(timeout = 4000) public void test00() throws Throwable { Show show0 = new Show(); File file0 = MockFile.createTempFile("..."); Mystery Guest and Resource Optimism. For these two types of smells, both detection tools raise several warnings. However, they are all false positives by definition, as our gold standard does not contain any instances of such smells. The detection tools both annotate test methods that contain specific strings or objects, such as: “File”, “FileOutputStream” “DB”, “HttpClient” as smelly; however, EVOSUITE sep- arates the test code from environmental dependencies (e.g., external files) in a fully automated fashion through byte- code instrumentation [43]. In particular, it uses two mech- anisms: (1) mocking, and (2) customized test runners. For one, classes that access the filesystem (e.g., java.io.File) (GPD) Large False Negative Rate for Sensitive Equality and Indirect Testing GPD Low False Positive Rate Large False Negative Rate for most of the test smells TsDetector
- 9. Limitations of Test Smell Detection Tools 9 According to GPD warnings 12% of the JUnit test suites by EvoSuite contain Mystery Guest 2% of the JUnit test suites by EvoSuite contain Resource Optimism EvoSuite does not use external resources or files thanks to: • Sandbox and scaffolding • Automated mocks generation • The use a customized JUnit runner FALSE POSITIVES
- 10. Limitations of Test Smell Detection Tools 10 GPD and TsDetector fail to detect instances of Sensitive Equality @Test(timeout = 4000) public void test62() throws Throwable { SubstringLabeler.Match substringLabeler_Match0 = new SubstringLabeler.Match(); String string0 = substringLabeler_Match0.toString(); assertEquals("Substring: [Atts: ]", string0); } public void test62() throws Throwable { SubstringLabeler.Match substringLabeler_Match0 = new SubstringLabeler.Match(); assertEquals("Substring: [Atts: ]", substringLabeler_Match0.toString()); } Test generated by EvoSuite but not detected by the two tools This test would be detected
- 11. Discussion • In the paper we further discuss the limitations of test smell detection tools (GDP and TsDetector) with more examples • Our results disagree with the conclusions by Grano et al. • Only 80% 32% of generated tests contain test smells • Researchers should avoid self-assessing their test smell detection tools • The involvement of human participants (preferably in industrial contexts) is critical for improving the accuracy of detection tools 11
- 12. Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn 12