![/ 33
Needs of Java Multi-Fault Datasets
Existing Multi-Fault Extensions of Defects4J
• Zheng et al. [1] manually combined two faulty versions in Defects4J.
• Since the procedure is manual, only 46 multi-fault versions were
created.
• A public fork of Defects4J (djpaterson/defects4j) [2] also provides the
"d4j-combine” command.
•
[1] Zheng, Y., Wang, Z., Fan, X., Chen, X., Yang, Z.: Localizing multiple software faults based on evolution algorithm. J. Syst. Softw. 139, 107–123 (2018)
[2] Paterson, David (2019) Improvements to Test Case Prioritisation considering Efficiency and Effectiveness on Real Faults. PhD thesis, University of Sheffield.
12](https://image.slidesharecdn.com/ssbse-new-new-211011133731/85/Searching-for-Multi-Fault-Programs-in-Defects4J-19-320.jpg)
Searching for Multi-Fault Programs in Defects4J
- 1. Gabin An, Juyeon Yoon, and Shin Yoo, COINSE, KAIST, October 2021 Searching for Multi-Fault Programs in Defects4J Challenge Track, SSBSE 2021 1
- 3. / 33 • Defects4J is a collection of real-world Java faults from open-source projects. • Each fault in Defects4J is isolated and reproducible using a set of revealing test cases that expose the fault. Faulty Program Defects4J the Challenging Case 3
- 4. / 33 • Defects4J is a collection of real-world Java faults from open-source projects. • Each fault in Defects4J is isolated and reproducible using a set of revealing test cases that expose the fault. Faulty Program Defects4J the Challenging Case F F P P P P P F 3 Passing Failing Reveal
- 5. / 33 Defects4J and SFA • In many Fault Localization (FL) and Automated Program Repair (APR) studies, it is frequently assumed that the System Under Test (SUT) contains a single fault that is responsible for all observed test failures. Single Fault Assumption (SFA) P P P P P F F F 4
- 6. / 33 Defects4J and SFA • In many Fault Localization (FL) and Automated Program Repair (APR) studies, it is frequently assumed that the System Under Test (SUT) contains a single fault that is responsible for all observed test failures. Single Fault Assumption (SFA) P P P P P F F F F F F 4
- 7. / 33 Defects4J and SFA • The isolation of individual faults in Defects4J makes it compatible with the SFA. • Many FL and APR techniques have been evaluated using Defects4J, as it allows precision evaluation and easier comparison between different techniques. A faulty program in Defects4J F F P P P P P F FL APR 5
- 8. / 33 In reality, SFA may not hold. • In a realistic debugging scenario, SFA may not hold! • Multiple faults can coexist in a program. • Failing test cases may be caused by distinct root causes. F F P P P P P F 6
- 9. / 33 In reality, SFA may not hold. • In a realistic debugging scenario, SFA may not hold! • Multiple faults can coexist in a program. • Failing test cases may be caused by distinct root causes. F F P P P P P F 6
- 10. / 33 In reality, SFA may not hold. • In a realistic debugging scenario, SFA may not hold! • Multiple faults can coexist in a program. • Failing test cases may be caused by distinct root causes. F F P P P P P F 6
- 11. / 33 In reality, SFA may not hold. • In a realistic debugging scenario, SFA may not hold! • Multiple faults can coexist in a program. • Failing test cases may be caused by distinct root causes. F F P P P P P F 6
- 12. / 33 In reality, SFA may not hold. • In a realistic debugging scenario, SFA may not hold! • Multiple faults can coexist in a program. • Failing test cases may be caused by distinct root causes. F F P P P P P F 6
- 13. / 33 In reality, SFA may not hold. • When the SFA does not hold, many FL/APR techniques developed under SFA become less effective. • Failing test cases will act as noise to each other. F F P P P P P F FL APR ? 7 F F P P P P P F P P P P P
- 14. / 33 • To mitigate the problem, researchers have proposed many automated multi-fault debugging techniques. • Ex) Failure Clustering Multi-Fault Debugging Techniques F F P P P P P F P P P P P 8
- 15. / 33 • To mitigate the problem, researchers have proposed many automated multi-fault debugging techniques. • Ex) Failure Clustering Multi-Fault Debugging Techniques Root Cause #1 Root Cause #2 F F P P P P P F P P P P P 8
- 16. / 33 Multi-Fault Debugging Techniques • To mitigate the problem, researchers have proposed many automated multi-fault debugging techniques. • Ex) Failure Clustering F F P P P P P F Root Cause #1 Root Cause #2 P P P P P 9 FL APR
- 17. / 33 Multi-Fault Debugging Techniques • To mitigate the problem, researchers have proposed many automated multi-fault debugging techniques. • Ex) Failure Clustering F F P P P P P F Root Cause #1 Root Cause #2 P P P P P 10 FL APR
- 18. / 33 Needs of Java Multi-Fault Datasets • Automated multi-fault debugging techniques would significantly benefit from a reliable dataset of realistic multi-fault programs. • The majority of multi-fault debugging techniques are evaluated using only C faults. • Our challenging case, Defects4J, is rarely used in multi-fault debugging studies, although it is the most frequently used dataset in FL and APR. 11
- 19. / 33 Needs of Java Multi-Fault Datasets Existing Multi-Fault Extensions of Defects4J • Zheng et al. [1] manually combined two faulty versions in Defects4J. • Since the procedure is manual, only 46 multi-fault versions were created. • A public fork of Defects4J (djpaterson/defects4j) [2] also provides the "d4j-combine” command. • [1] Zheng, Y., Wang, Z., Fan, X., Chen, X., Yang, Z.: Localizing multiple software faults based on evolution algorithm. J. Syst. Softw. 139, 107–123 (2018) [2] Paterson, David (2019) Improvements to Test Case Prioritisation considering Efficiency and Effectiveness on Real Faults. PhD thesis, University of Sheffield. 12
- 20. / 33 • Ex) Needs of Java Multi-Fault Datasets Lang-41b 41 Lang-43b 43 43 43 13
- 21. / 33 • Ex) Needs of Java Multi-Fault Datasets Lang-41b 41 Lang-43b 43 43 43 13
- 22. / 33 • Ex) Needs of Java Multi-Fault Datasets Lang-41b 41 Lang-43b 43 43 43 This Defects4J-provided buggy patch of Lang-43 is applied to Lang-41b. 13
- 23. / 33 • Ex) Needs of Java Multi-Fault Datasets Lang-41b 41 Lang-43b 43 43 43 This Defects4J-provided buggy patch of Lang-43 is applied to Lang-41b. Faults are artificially combined. 13
- 24. / 33 Needs of Java Multi-Fault Datasets • Artificially combined faults are not real multiple faults that actually coexisted in a program. • Goal: we aim to construct a real-world multi-fault dataset by finding the programs that actually contain multiple faults. Real-World 14
- 25. Methodology 15
- 26. / 33 Searching for Multi-Fault Programs • Let us consider a faulty program in Defects4J. • This program may contain other faults that have been simply not exposed due to the absence of their revealing test cases. F F P P P P P F 16
- 27. / 33 Searching for Multi-Fault Programs • Let us consider a faulty program in Defects4J. • This program may contain other faults that have been simply not exposed due to the absence of their revealing test cases. F F P P P P P F 16
- 28. / 33 Searching for Multi-Fault Programs • Let us consider a faulty program in Defects4J. • This program may contain other faults that have been simply not exposed due to the absence of their revealing test cases. F F P P P P P F 16 F F
- 29. / 33 Searching for Multi-Fault Programs Example: Math-5 and Math-6 Math-6b 6 Searching for Multi-fault Programs in Defects4J 3 1 --- a/src/main/java/org/apache/commons/math3/ complex/Complex.java 2 +++ b/src/main/java/org/apache/commons/ math3/complex/Complex.java 3 @@ -304,7 +304 ,7 @@ @@ public Complex reciprocal () { 4 if (real == 0.0 && imaginary == 0.0) { 5 - return NaN; 6 + return INF; 7 } Listing 1.1. The developer patch for Math-5 1 public void testReciprocalZero () { 2 Assert. assertEquals(Complex.ZERO. reciprocal (), Complex.INF); 3 // Error message: junit. framework. AssertionFailedError : expected :<(NaN , NaN)> but was:<(Infinity , Infinity)> 4 } Listing 1.2. The fault-revealing test case of Math-5 Math-5b 5 Math-5 17
- 30. / 33 Searching for Multi-Fault Programs Example: Math-5 and Math-6 Math-6b 6 Searching for Multi-fault Programs in Defects4J 3 1 --- a/src/main/java/org/apache/commons/math3/ complex/Complex.java 2 +++ b/src/main/java/org/apache/commons/ math3/complex/Complex.java 3 @@ -304,7 +304 ,7 @@ @@ public Complex reciprocal () { 4 if (real == 0.0 && imaginary == 0.0) { 5 - return NaN; 6 + return INF; 7 } Listing 1.1. The developer patch for Math-5 1 public void testReciprocalZero () { 2 Assert. assertEquals(Complex.ZERO. reciprocal (), Complex.INF); 3 // Error message: junit. framework. AssertionFailedError : expected :<(NaN , NaN)> but was:<(Infinity , Infinity)> 4 } Listing 1.2. The fault-revealing test case of Math-5 Math-5b 5 5 Math-5 17
- 31. / 33 Searching for Multi-Fault Programs Example: Math-5 and Math-6 Math-6b 6 Searching for Multi-fault Programs in Defects4J 3 1 --- a/src/main/java/org/apache/commons/math3/ complex/Complex.java 2 +++ b/src/main/java/org/apache/commons/ math3/complex/Complex.java 3 @@ -304,7 +304 ,7 @@ @@ public Complex reciprocal () { 4 if (real == 0.0 && imaginary == 0.0) { 5 - return NaN; 6 + return INF; 7 } Listing 1.1. The developer patch for Math-5 1 public void testReciprocalZero () { 2 Assert. assertEquals(Complex.ZERO. reciprocal (), Complex.INF); 3 // Error message: junit. framework. AssertionFailedError : expected :<(NaN , NaN)> but was:<(Infinity , Infinity)> 4 } Listing 1.2. The fault-revealing test case of Math-5 4 G. An et al. 304 if (real == 0.0 && imaginary == 0.0) { 305 return NaN; // Math -5b 306 } Listing 1.3. In Math-6b, Complex.java (line 305) contains the fault Math-5 Math-5b 5 5 Math-5 17
- 32. / 33 • We detect a fault by automatically transplanting its fault-revealing test cases to other faulty versions. Detecting a Fault Using its Revealing Tests Math-5b F testReciprocalZero 5 public void testReciprocalZero() { Assert.assertEquals(Complex.ZERO.reciprocal(), Complex.INF); } 18
- 33. / 33 5 • We detect a fault by automatically transplanting its fault-revealing test cases to other faulty versions. Detecting a Fault Using its Revealing Tests Math-6b 6 F 1st F 28th …. Math-5b ? public void testReciprocalZero() { Assert.assertEquals(Complex.ZERO.reciprocal(), Complex.INF); } F testReciprocalZero 5 19
- 34. / 33 T testReciprocalZero 5 • We detect a fault by automatically transplanting its fault-revealing test cases to other faulty versions. Detecting a Fault Using its Revealing Tests Math-6b 6 F 1st F 28th …. Math-5b ? public void testReciprocalZero() { Assert.assertEquals(Complex.ZERO.reciprocal(), Complex.INF); } F testReciprocalZero 5 19
- 35. / 33 T testReciprocalZero 5 • We detect a fault by automatically transplanting its fault-revealing test cases to other faulty versions. Detecting a Fault Using its Revealing Tests Math-6b 6 F 1st F 28th …. Math-5b F 5 5 public void testReciprocalZero() { Assert.assertEquals(Complex.ZERO.reciprocal(), Complex.INF); } F testReciprocalZero 5 19
- 36. / 33 T testReciprocalZero 5 • We detect a fault by automatically transplanting its fault-revealing test cases to other faulty versions. Detecting a Fault Using its Revealing Tests Math-6b 6 F 1st F 28th …. Math-5b F 5 5 public void testReciprocalZero() { Assert.assertEquals(Complex.ZERO.reciprocal(), Complex.INF); } junit.framework.AssertionFailedError: expected : <(Infinity, Infinity)> but was : <(NaN, NaN)> F testReciprocalZero 5 The same error message! 19
- 37. Instead of artificially combining faults, Lang-41b 41 Real Lang-43b 43 Real 43 Injected Math-6b Real 6 Math-5b 5 Real Combining Faults Combining Tests (Ours) S O U R C E
- 38. F 5 Real Instead of artificially combining faults, we use fault-revealing test cases to detect the real co-existing faults. Lang-41b 41 Real Lang-43b 43 Real 43 Injected F Math-6b Real 6 Math-5b 5 Real Combining Faults Combining Tests (Ours) S O U R C E T E S T
- 39. / 33 How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… Math-5b Math-36b Math-7b 2013-08-31 2006-06-05 2013-03-22 2013-03-19 2013-03-17 2012-04-02 2012-02-14 22
- 40. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b Math-7b 23
- 41. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 F • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b Math-7b 23
- 42. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 F • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b Math-7b 5 F 23
- 43. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 F 5 F • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b Math-7b 5 F 23
- 44. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 F 5 F • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b 5 P Math-7b 5 F 23
- 45. / 33 T testReciprocalZero How to Search? Overview of Search Strategy Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 F 5 F • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. Math-5b 5 F Math-36b 5 P Stop Math-7b 5 F 23
- 46. / 33 T F How to Search? Overview of Search Strategy Math-106b Math-1b …… Math-35b Math-5b …… Math-36b Math-6b 6 • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. …… Math-7b Fixed Oldest Latest 24
- 47. / 33 T F How to Search? Overview of Search Strategy Math-106b Math-1b …… Math-35b Math-5b …… Math-36b Math-6b 6 • A faulty is considered to be revealed only if • All revealing test cases fail with the same error message as in the original faulty version. …… Math-7b Fixed Oldest Latest 24
- 48. / 33 Building Multi-Fault Subjects 25 N M … … 3 4 1 5 3 5 4 5 1 6 3 6 4 6 5 6 … … (N, M) ∈ E if N is revealed in BM (the faulty version of M). Search Results E for Math →
- 49. / 33 Building Multi-Fault Subjects • F(BM), the set of faults found in BM, is then defined as • F(BM)= {M} ∪ { N | (N,M) ∈ E } • ex) F(B6) = {6} ∪ {1,3,4,5} = {1,3,4,5,6} • If |F(BM)|>1, BM is a multi-fault subject. 26 N M … … 3 4 1 5 3 5 4 5 1 6 3 6 4 6 5 6 … …
- 50. / 33 Building Multi-Fault Subjects • Ex) 27 Math-6b 6 F 3 4 5 1
- 51. / 33 Building Multi-Fault Subjects • Ex) 27 Math-6b 6 F 3 4 5 1 3 4 5 F F F Newly Added! 1 F
- 52. Results 28
- 53. / 33 Multiple Fault Subjects • We apply our search strategy on 326 faults in Defects4J: Lang 1-65, Chart 1-26, Math 1-106, Time 1-27, Closure 1-106 • We found that 95.4% (=311/326) of the studied faulty versions contain multiple faults. 305 return NaN; // Math -5b 306 } Listing 1.3. In Math-6b, Complex.j Author Pro Multi-Fault Subjects 29
- 54. / 33 305 return NaN; // Math -5b 306 } Listing 1.3. In Math-6b, Complex.j Author Pro Multiple Fault Subjects • About 40% of faulty versions have ≥10 faults. • About 7% of faulty versions have ≥20 faults. • ex) Closure-90b contains 24 faults. Multi-Fault Subjects 311/326 ≃ 95% 30
- 55. / 33 305 return NaN; // Math -5b 306 } Listing 1.3. In Math-6b, Complex.j Author Pro Multiple Fault Subjects • About 40% of faulty versions have ≥10 faults. • About 7% of faulty versions have ≥20 faults. • ex) Closure-90b contains 24 faults. Multi-Fault Subjects 126/326 ≃ 40% 30
- 56. / 33 305 return NaN; // Math -5b 306 } Listing 1.3. In Math-6b, Complex.j Author Pro Multiple Fault Subjects • About 40% of faulty versions have ≥10 faults. • About 7% of faulty versions have ≥20 faults. • ex) Closure-90b contains 24 faults. Multi-Fault Subjects 22/326 ≃ 7% 30
- 57. / 33 Lifespan of Faults Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 5 Math-5b 5 Math-36b 5 Math-7b 5 2012-04-01 2012-02-14 Date of the buggy commit Bug Inducing Changes 31
- 58. / 33 Lifespan of Faults Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 5 Math-5b 5 Math-36b 5 Math-7b 5 2012-04-01 2012-02-14 Date of the buggy commit Bug Inducing Changes 31
- 59. / 33 Lifespan of Faults Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 5 Math-5b 5 Math-36b 5 Math-7b 5 2012-04-01 2012-02-14 Date of the buggy commit 2013-03-22 Date of the fixed commit Bug Inducing Changes 31
- 60. / 33 Lifespan of Faults Math-106b Math-1b Fixed Oldest Latest Math-35b …… Math-6b …… …… 5 5 Math-5b 5 Math-36b 5 Math-7b 5 2012-04-01 2012-02-14 Date of the buggy commit Approximated lifespan of faults >= 355 days 2013-03-22 Date of the fixed commit Bug Inducing Changes 31
- 61. / 33 Lifespan of Defects4J Faults • The probability of having multiple faults at any given time can be nontrivial. NaN; // Math -5b ath-6b, Complex.java (line 305) contains the fault Math-5 aulty versions in De- mber of faults (b) The sorted life span of faults in days (average=154, standard deviation=246) 32 1 year 2 years 3 years
- 62. / 33 Summary & Conclusion • We build a real-world Java multi-fault dataset extending Defects4J using a systematic search based on the revealing test transplantation. • We do not alter any program source code to inject faults but just add test cases to reveal the faults. • Our search results show that 311 out of 326 studied faulty versions in Defects4J contain multiple faults. • Full results are available at: https://github.com/coinse/Defects4J- multifault/tree/main/multiple_faults Gabin An Juyeon Yoon Shin Yoo 33