MIning Software Repositories (MSR) 2010 presentation
- 1. Proceedings of the 2010 7th IEEE Working Conference on Mining Software Repositories, p.1-10 Predicting the Severity of a Reported Bug Ahmed Lamkanfi, Serge Demeyer | Emanuel Giger | Bart Goethals Ansymo | s.e.a.l. | ADReM
- 2. Proceedings of the 2010 7th IEEE Working Conference on Mining Software Repositories, p.1-10 Predicting the Severity of a Reported Bug Ahmed Lamkanfi, Serge Demeyer | Emanuel Giger | Bart Goethals Ansymo | s.e.a.l. | ADReM
- 5. Severity of a bug is important ✓ Critical factor in deciding how soon it needs to be fixed, i.e. when prioritizing bugs
- 7. Severity is techn ical
- 8. ✓ Severity varies: ➡ trivial, minor, normal major, critical and blocker ➡ clear guidelines exist to classify severity of bug reports
- 9. ✓ Severity varies: ➡ trivial, minor, normal major, critical and blocker ➡ clear guidelines exist to classify severity of bug reports ✓ Both a short and longer description of the problem
- 10. ✓ Severity varies: ➡ trivial, minor, normal major, critical and blocker ➡ clear guidelines exist to classify severity of bug reports ✓ Both a short and longer description of the problem ✓ Bugs are grouped according to products and components ➡ e.g.: plug-ins, bookmarks are components of product Firefox
- 11. Can we accurately predict the severity of a reported bug by analyzing its textual descriptions?
- 12. Can we accurately predict the severity of a reported bug by analyzing its textual descriptions? Also the following questions:
- 13. Can we accurately predict the severity of a reported bug by analyzing its textual descriptions? Also the following questions: Potential indicators?
- 14. Can we accurately predict the severity of a reported bug by analyzing its textual descriptions? Also the following questions: Potential indicators? Short versus long description?
- 15. Can we accurately predict the severity of a reported bug by analyzing its textual descriptions? Also the following questions: Potential indicators? Short versus long description? Per component versus cross-component?
- 16. Approach
- 17. We use text mining to classify bug reports • Bayesian classifier: based on the probabilistic occurrence of words • training and evaluation period • in first instance, per component
- 18. We use text mining to classify bug reports • Bayesian classifier: based on the probabilistic occurrence of words • training and evaluation period • in first instance, per component
- 19. We use text mining to classify bug reports • Bayesian classifier: based on the probabilistic occurrence of words • training and evaluation period • in first instance, per component
- 20. We use text mining to classify bug reports • Bayesian classifier: based on the probabilistic occurrence of words • training and evaluation period • in first instance, per component Non-severe bugs Severe bugs (trivial, minor) (major, critical, blocker)
- 21. We use text mining to classify bug reports • Bayesian classifier: based on the probabilistic occurrence of words • training and evaluation period • in first instance, per component Undecided Non-severe bugs Default Severe bugs (trivial, minor) (normal) (major, critical, blocker)
- 22. Evaluation of the approach: ✓ precision and recall: Cases drawn from the open-source community ✓ Mozilla, Eclipse and GNOME
- 23. Results
- 24. How does the basic approach perform? ➡ per component and using short description
- 25. How does the basic approach perform? ➡ per component and using short description Non-severe Severe component precision recall precision recall Mozilla: Layout 0.701 0.785 0.752 0.653 Mozilla: Bookmarks 0.692 0.703 0.698 0.687 Eclipse: UI 0.707 0.633 0.668 0.738 Eclipse: JDT-UI 0.653 0.714 0.685 0.621 GNOME: Calendar 0.828 0.783 0.794 0.837 GNOME:Contacts 0.767 0.706 0.728 0.785
- 26. What keywords are good indicators of severity?
- 27. What keywords are good indicators of severity? Component Non-severe Severe inconsist, favicon, credit, Fault, machin, reboot, Mozilla Firefox- General extra, consum, licens, reinstal, lockup, underlin, typo, inspector, seemingli, perman, titlebar instantli, segfault, compil deprec, style, runnabl, hang, freez, deadlock, Eclipse JDT UI system, cce, tvt35, thread, slow, anymor, whitespac, node, put, param memori, tick, jvm, adapt mnemon, outbox, typo, pad, deadlock, sigsegv, relat, GNOME Mailer follow, titl, high, caus, snapshot, segment, acceler, decod, reflec core, unexpectedli, build, loop
- 28. How does the approach perform when using the longer description?
- 29. How does the approach perform when using the longer description? Non-severe Severe component precision recall precision recall Mozilla: Layout 0.583 0.961 0.890 0.314 Mozilla: Bookmarks 0.536 0.963 0.820 0.166 Mozilla: Firefox 0.578 0.948 0.856 0.308 general Eclipse: UI 0.548 0.976 0.892 0.197 Eclipse: JDT-UI 0.547 0.973 0.881 0.195 Eclipse: JDT-Text 0.570 0.988 0.955 0.257
- 30. How does the approach perform when using the longer description? Non-severe Severe component precision recall precision recall Mozilla: Layout 0.583 0.961 0.890 0.314 Mozilla: Bookmarks 0.536 0.963 0.820 0.166 Mozilla: Firefox 0.578 0.948 0.856 0.308 general Eclipse: UI 0.548 0.976 0.892 0.197 Eclipse: JDT-UI 0.547 0.973 0.881 0.195 Eclipse: JDT-Text 0.570 0.988 0.955 0.257
- 31. How does the approach perform when combining bugs from different components?
- 32. How does the approach perform when combining bugs from different components? Non-severe Severe component precision recall precision recall Mozilla 0.704 0.750 0.733 0.685 Eclipse 0.693 0.553 0.628 0.755 GNOME 0.817 0.737 0.760 0.835
- 33. How does the approach perform when combining bugs from different components? Non-severe Severe component precision recall precision recall Mozilla 0.704 0.750 0.733 0.685 Eclipse 0.693 0.553 0.628 0.755 GNOME 0.817 0.737 0.760 0.835 Much larger training set necessary ✓± 2000 reports instead of ± 500 per severity!
- 34. Conclusions ✓ It is possible to predict the severity of a reported bug ✓ Short description better source for predictions ✓ Cross-component approach works, but requires more training samples