Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
2 likes144 views
The document discusses the evaluation of various SZZ implementations for identifying bug-inducing changes in software development. It emphasizes the importance of understanding where bugs are introduced to improve resource allocation in testing and presents a developer-informed dataset for thorough analysis. The results indicate differing performances of SZZ variants and provide key lessons regarding the sensitivity of SZZ to history rewritings and the need for a comprehensive view of code changes.
Evaluating SZZ Implementations Through a Developer-informed Oracle (ICSE 2021)
- 1. Evaluating SZZ Implementations Through a Developer-informed Oracle Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano, Gabriele Bavota, Michele Lanza, Rocco Oliveto
- 2. Where do bugs come from?
- 3. Find out changes that can lead to a problem and avoid them in future Understanding where bugs are introduced allows toâŠ
- 4. Estimate how much a program is error-prone Understanding where bugs are introduced allows toâŠ
- 5. Better allocate resources in testing activities Understanding where bugs are introduced allows toâŠ
- 7. Step 1 SZZ in a nutshell bug report analysis
- 8. Step 1 (A) Bug-fixing commit (B) git blame (C) Buggy commit SZZ in a nutshell bug report analysis
- 9. Step 1 Step 2 Filtering of resulting commits SZZ in a nutshell (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 10. Step 1 bug-inducing commit Step 2 Step 3 SZZ in a nutshell Filtering of resulting commits (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 11. Different SZZ variants proposed
- 12. There is a problem
- 13. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016
- 14. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016 Small datasets used for evaluation
- 15. Evaluating and comparing the SZZ variants Da Costa et al. @ TSE 2016 Small datasets used for evaluation Validation manually performed by researchers
- 16. Define a dataset validated by the developers The way
- 17. fixes a search bug introduced by 2508e12 and fixes a typo in the README.md
- 19. Mining of commits 2011 2020
- 20. Heuristic approach 1 keyword-based filter AI-powered syntax analysis Duplicate commits removal
- 21. Heuristic approach 2 keyword-based filter AI-powered syntax analysis Duplicate commits removal
- 22. 3 Heuristic approach keyword-based filter AI-powered syntax analysis duplicate commits removal
- 24. Bug report data fixes #1740 quote pov-ray binary on windows this fixes a bug introduced by #3523741⊠URL Date when the issue is reported https://tracker.freecadweb.org/view.php?id=1740 Commit message
- 25. 19,6M 3,6k 1,9k Analyzed commits: Extracted commits: After manual validation:
- 27. 1,1k 129 Final number of commits: Commits with issue report:
- 28. How do different variants of SZZ perform in identifying bug-inducing changes?
- 29. B-SZZ Ćliwerski et al. @ MSR 2005
- 30. R-SZZ e L-SZZ B-SZZ AG-SZZ DJ-SZZ Ćliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Kim et al. @ ASE 2006 Davies et al. @ JSE 2013
- 31. R-SZZ e L-SZZ B-SZZ AG-SZZ MA-SZZ DJ-SZZ RA-SZZ Ćliwerski et al. @ MSR 2005 Williams and Spacco @ ISSTA 2008 Da Costa et al. @ TSE 2016 Kim et al. @ ASE 2006 Davies et al. @ JSE 2013 Neto et al. @ SANER 2018
- 33. Step 1 bug-inducing commit Step 2 Step 3 Our experiment Filtering of resulting commits (A) Bug-fixing commit (B) git blame (C) Buggy commit bug report analysis
- 35. Results 0.66 (R-SZZ) Precision Recall F1-score 0.72 (SZZ@UNL) 0.61 (R-SZZ) 0.09 (SZZ@UNL) 0.19 (SZZ@OPN) Java only 0.16 (SZZ@UNL)
- 37. What have we learned?
- 38. â The buggy line is not always impacted in the bug-fix â Lesson 1
- 39. â SZZ is sensible to history rewritings â Lesson 2
- 40. â Looking at the big picture in code changes â Lesson 3
- 41. Summary
- 42. Take a look at our SZZ implementation! https://github.com/grosa1/pyszz