Can we predict the quality of spectrum-based fault localization?
0 likes315 views
The document discusses predicting the effectiveness of spectrum-based fault localization techniques. It proposes defining metrics to capture aspects of source code, test executions, test suites, and faults. A dataset of 341 instances with 70 variables is generated from Defects4J projects, classifying instances as "effective" or "ineffective" based on fault ranking. Analysis identifies the most influential metrics, finding a combination of static, dynamic, and test metrics can construct a prediction model with excellent discrimination, achieving an AUC of 0.86-0.88. The results suggest effectiveness depends more on code and test complexity than fault type/location, and entangled dynamic call graphs hinder localization.
Can we predict the quality of spectrum-based fault localization?
- 1. Can we predict the quality of spectrum-based fault localization? Mojdeh Golagha fortiss Lionel C. Briand University of Luxembourg, University of Ottawa Alexander Pretschner Technical University of Munich
- 2. Problem Spectrum-based Fault localization 25.10.2020 Why is the effectiveness of spectrum based fault localization techniques so unpredictable? 2 Source code Test Suite Block Test 1 Test 2 Test 3 Tarantula Ochiai DStar 1 ∞ ∞ 2 ∞ ∞ ∞ 3 ∞ „ Not widely applied in practice yet. „ Effectiveness varies greatly from case to case. „ New algorithms and ideas as well as adjustments to the test suites to improve effectiveness. „ Why is the effectiveness of these techniques so unpredictable? „ What are the factors that influence the effectiveness of fault localization? „ Can we accurately predict fault localization effectiveness?
- 3. Solution 25.10.2020 Based on precise hypotheses, we define metrics and assess their influence on fault localization effectiveness. 3 I. Define metrics. II. Generate a data set. III. Apply classification analysis to assess the influence of metrics. IV. Build the most accurate prediction model for effectiveness.
- 5. Metrics 25.10.2020 Metrics capture static aspects of the source code, dynamic aspects of test executions, properties of the test suite, and fault properties. 5
- 7. Data Set 25.10.2020 Our final data set has 341 instances and 70 variables. The “effective” label has been assigned to 193 of these instances. 7 Each observation is a faulty version extracted from Defects4J Method-level SBFL. • Rank of the faulty method • 1 to 10 -> “effective” • Otherwise “ineffective” Variables Class Labels
- 9. Analysis of Significance 25.10.2020 To assess the significance and impact of each metric individually. 9 Pre-analysis: Pearson’s Correlation to remove correlated metrics I. Odds ratio analysis in univariate logistic regression II. Average information gain
- 10. Multivariate Analysis 25.10.2020 To generate prediction models. One model for each group and one model for all groups combined. 10 I. Tree II. Logistic Regression III. SVM IV. Random Forest V. Adaboost
- 11. Multivariate Analysis 25.10.2020 Considering the StatDynaTest metrics, random forest and logistic regression yield the best, almost identical, results. 11 Logistic Regression Model Random Forest Model
- 12. Multivariate Analysis 25.10.2020 There are eight metrics in common among them: Static - % Methods with Nesting Depth>5, Mean # of Fields per Type, Dynamic - Mean Node Degree, Max Node Out Degree, Graph Diameter, Response for Class, Test - % Method Coverage, % Methods Covered in Failing Tests. 12
- 13. Results 25.10.202013 • A combination of only a few static, dynamic, and test metrics enables the construction of a prediction model with excellent discrimination power between levels of effectiveness: - eight metrics yielding an AUC of .86 (Logistic Regression on Selected-SDT metrics) - fifteen metrics yielding an AUC of .88 (Random Forest on StatDynaTest metrics) • A confidence factor that can be used to assess the potential effectiveness of fault localization. • The most influential metrics can also be used as a guide for corrective actions on code and test suites leading to more effective fault localization. • The effectiveness of fault localization depends more on the complexity of the code and test suite than on the fault type and location. • Selected dynamic metrics measure the degree to which the call graph is entangled. If the dynamic call graph of the tests is highly entangled and coupled, it is difficult to localize a fault, no matter where it happens.