An Evaluation of Effort-Aware Fine-Grained Just-in-Time Defect Prediction Methods
Download as PPTX, PDF0 likes7 views
The document evaluates various effort-aware fine-grained just-in-time (JIT) defect prediction methods, comparing supervised and unsupervised approaches across Java open-source projects. It finds that the CBS+ supervised method performed best but still exhibited low performance across all methods examined, with non-linear models providing slight improvements. Future work is suggested to explore additional predictive features and alternative methods.
![Controversy over effort-aware JIT SDP
SEAA 2022 in Gran Canaria 4
• Supervised vs. Unsupervised
• Supervised needs training data
to make a prediction
• e.g., EALR[10] , CBS+[23]
• Unsupervised needs no data
and utilizes heuristics
• e.g., LT[20], CCUM[21]
Question:
Which is better for Fine-grained JIT SDP?](https://image.slidesharecdn.com/smanevaluationofeffort-awarefine-grainedjust-in-timedefectpredictionmethods-220831084853-ebeb907f/85/An-Evaluation-of-Effort-Aware-Fine-Grained-Just-in-Time-Defect-Prediction-Methods-4-320.jpg)
![Datasets
• A replication package by Trautsch et al.[25]
• 38 Java OSS projects of Apache
• 35 out of 38 were used
• Features from a fine-grained software defect prediction study[15]
• e.g., Lines added, committer’s experience
• Linkages between commits and bug reports are recovered by 2 ways
• Ad-hoc
• Keyword search such as ”fix” in a commit message
• ITS
• Links from commits to BRs and manually validated
SEAA 2022 in Gran Canaria 6
Used for comparisons](https://image.slidesharecdn.com/smanevaluationofeffort-awarefine-grainedjust-in-timedefectpredictionmethods-220831084853-ebeb907f/85/An-Evaluation-of-Effort-Aware-Fine-Grained-Just-in-Time-Defect-Prediction-Methods-6-320.jpg)
An Evaluation of Effort-Aware Fine-Grained Just-in-Time Defect Prediction Methods
- 1. An Evaluation of Effort- Aware Fine-Grained Just-in- Time Defect Prediction Methods Sousuke Amasaki1, Hirohisa Aman2, Tomoyuki Yokogawa1 1Okayama Prefectural University 2Ehime University
- 2. Software defect prediction target gets finer for saving resource • Traditional Software Defect Prediction • Focus on modules (classes, files, methods) • Just-in-Time Software Defect Prediction • Focus on commits (a set of file changes) • Fine-grained JIT Software Defect Prediction • Focus on file changes in a commit SEAA 2022 in Gran Canaria 2 Buggy! commit Buggy! commit Buggy!
- 3. Projects still have little resource for SQA • Traditional SDP is a classification problem for all targets • Effort-aware SDP is a ranking problem under limited budget SEAA 2022 in Gran Canaria 3 Buggy! Buggy! 100% Buggy! Buggy! 90%! 80%! 70%! 60%! 50%! Check all ones predicted as buggy Actually check ones fit in a budget Buggy! Buggy! Buggy!
- 4. Controversy over effort-aware JIT SDP SEAA 2022 in Gran Canaria 4 • Supervised vs. Unsupervised • Supervised needs training data to make a prediction • e.g., EALR[10] , CBS+[23] • Unsupervised needs no data and utilizes heuristics • e.g., LT[20], CCUM[21] Question: Which is better for Fine-grained JIT SDP?
- 5. Commits in 1 months Experiment Design SEAA 2022 in Gran Canaria 5 Commits in 3 months Futur e Pa st Sliding Window for training data Time-sensitive approach Sliding Window for test data
- 6. Datasets • A replication package by Trautsch et al.[25] • 38 Java OSS projects of Apache • 35 out of 38 were used • Features from a fine-grained software defect prediction study[15] • e.g., Lines added, committer’s experience • Linkages between commits and bug reports are recovered by 2 ways • Ad-hoc • Keyword search such as ”fix” in a commit message • ITS • Links from commits to BRs and manually validated SEAA 2022 in Gran Canaria 6 Used for comparisons
- 7. JIT SDPs applied to Fine-grained data Unsupervised sorts commits according to risk scores • LT, AGE • CCUM Supervised predicts a risk score with features then sorts them • EALR • Train linear model to predict D • CBS+ • Train logistic model to classify commits • Group defective/non-defective • Sort commits in a group SEAA 2022 in Gran Canaria 7
- 8. Research Questions RQ1: What are the differences between using manually validated links and using ad-hoc links for evaluation in terms of effort-aware performance measures? RQ2: Which supervised or unsupervised methods are better than the others for fine-grained JIT defect prediction? RQ3: Does using non-linear learning models improve the performance of supervised methods? SEAA 2022 in Gran Canaria 8
- 9. Comparison Results for RQ2 SEAA 2022 in Gran Canaria 9 No clear differences Best but negligible and low-performance Best but negligible and low-performance Ranking measure (no resource limit) Best Answer to RQ2: CBS+ was the best but negligible and low-performance
- 10. Non-linear JIT SDPs for evaluation • Assumption • Using non-linear models for EALR and CBS+ may capture complex relations • NL JIT SDPs • EARF • Train Random Forests to predict D • CBS+RF • Train Random Forests to classify commits • Group defective/non-defective • Sort commits in a group SEAA 2022 in Gran Canaria 10
- 11. Comparison Results for RQ3 SEAA 2022 in Gran Canaria 11 Degradation Stay low-performance Stay low-performance Effective only on EALR Answer to RQ3: Non-linear model was a little help Effective only on EALR
- 12. Conclusion • Examined JIT SDP methods on Fine-grained JIT SDP with effort-aware measures • CBS+ (supervised) was the best one • All methods were low-performance • Non-linear model was a little help • Future Work • Additional features for prediction • Investigation of other methods SEAA 2022 in Gran Canaria 12
Editor's Notes
- #3: Software defect prediction, or SDP, is an active research topic for decades. Traditional SDP utilizes static code metrics such as cyclomatic number to characterize software modules. Therefore, the unit of prediction is file, class, or methods. Nowadays, software projects are formed around a repository that records software development activities. Just-in-Time software defect prediction, or JIT SDP, utilizes those records to characterize code changes. JIT SDP builds a defect prediction model utilizing all code changes of a project. The diversity of developers may make the code changes heterogeneous. More homogeneous sets of code changes may contribute to better defect prediction. Version control systems enabled us to attribute code changes to individual developers. Personalized defect prediction can be implemented now.
- #9: We thus set these research questions. RQ1. Then, we focused on two types developers, namely, active developers and the rest of the developers. The second research question is. RQ3.