PROMISE 2011: "Detecting Bug Duplicate Reports through Locality of Reference"
Download as PPT, PDF0 likes1,223 views
The document discusses a study focused on detecting duplicate bug reports in software development, specifically analyzing a large dataset from the Firefox bug tracking system. It outlines the challenges of managing duplicate reports, existing methodologies for identifying them, and experimental results demonstrating the effectiveness of various approaches, achieving recall rates up to 53%. The study aims to improve search tools for better triaging of reports and emphasizes the importance of time intervals between reports in enhancing detection accuracy.
PROMISE 2011: "Detecting Bug Duplicate Reports through Locality of Reference"
- 1. Detecting Bug Duplicate Reports Through Locality of Reference Tomi Prifti, Sean Banerjee, Bojan Cukic Lane Department of CSEE West Virginia University Morgantown, WV, USA September 2011
- 2. Presentation Outline Introduction Goals Related Work Understanding the Firefox Repository Experimental Setup Results Summary
- 3. Introduction Bug tracking systems are essential for software maintenance and testing Developers and simple users can report failure occurrences Advantages: Users involved in error reporting Direct impact of software quality. Disadvantages: Large number of reports on daily basis. Significant effort to triage. Users may submit many duplicate reports.
- 4. A typical bug report
- 5. Goals Comprehensive empirical analysis of a large bug report dataset. Creation of a search tool Encourage users to search the repository Avoid duplicate report submission whenever possible Assisting with report triage Build a list of reports possibly describing the same problem Let a triager examines the suggested list
- 6. Related Work Providing Triagers with a Suggested List Provide a suggested list of similar bugs to triagers for examinations. Wang et. al. exploit NLP techniques and execution information Duplicate detection rate as high as 67%-93% Semi-automated Filtering Determine the type of the report (Duplicate or Primary). If the new report is classified as a duplicate filter it out. Jalbert et al. use text semantics and a graph clustering technique to predict duplicate status Filtered out only 8% of duplicate reports
- 7. Related Work Semi-automated Assignment Apply text categorization techniques to predict the developer that should work on the bug Cubranic et. al. apply supervised Bayesian learning. Correctly classify 30% of the reports Anvik et. al. uses a supervised machine learning algorithm. Precision rates of 57% and 64% for Firefox and Eclipse Improving Report Quality Duplicate reports are not considered harmful Bettenburg et al. developed a tool, called CUEZILLA, that measures the quality of bug reports in real time “ Steps to reproduce” and “Stack traces” are the most useful information in bug reports
- 8. Related Work Bugzilla Search Tool Bugzilla 4.0 released around February 2011 provides duplicate detection Tool performs a Boolean full text search on the title over the entire repository Generates a dozen or so reports that may match at least one of the search terms In some instance testing with the exact title from an existing report title did not return the report itself Unknown accuracy of reported matches
- 9. Firefox Repository Firefox releases: 1.0.5, 1.5, 2.0, 3.0, 3.5 and the current version 3.6 (as of June 2010). 65% of reports reside in groups of one. 90% of duplicates are distributed in small groups of 2-16 reports
- 10. Time Interval Between Reports Many bugs receive the first duplicate within the first few months of the original report.
- 11. Experimental Setup Tokenization - “Bag-of-Words” Stemming. Reducing words to their root Stop Words Removal Lucene API used for pre-processing Term Frequency/Inverse Document Frequency (TF/IDF) used for weighting words Cosine Similarity used for similarity measures Example of tokenizing, stemming and stop word removal Sending email is not functional. send email function
- 12. Experimental Procedure Start with initial 50% as historical information Group containing most recent primary or duplicate is on top of the initial list Build suggested list using IR techniques As experiment progresses historical repository increases Continue until reports classified as duplicate or primary If a bug is primary it is forwarded to the repository This may not be realistic as triagers may misjudge reports
- 13. Measuring Performance Performance of the bug search tool is measured by the recall rate N recalled refers to the number of duplicate reports correctly classified N total refers to the total number of duplicate reports
- 14. Approach methodology Reporters query the repository. Use “title” (summary) to compare reports. Four experiments: TF/IDF “ Sliding Window” - TF/IDF “ Sliding Window” - Group Centroids - TF/IDF “ Sliding Window” - Group Centroids The centroid is composed of all unique terms from all reports in the group and the sum of their frequencies in each report. The total frequency of each term is divided by the number of reports in the group.
- 15. Sliding Window Defined “ Sliding-Window” approach. Keep a window of fixed size n Sort all groups based on the time elapsed between the last report and the new incoming report. Select top n groups (2000 is optimal analysis shows 95% accuracy of duplicate being in this group) Apply IR techniques only on top n groups Build a short list of top m most similar reports to present to the triager/reporter
- 16. Experimental Results Our results demonstrate that Time-Window/Group Centroid and report summaries predict duplicate problem reports with a recall ratio of up to 53%.
- 17. Performance and Runtime Large variance in recall rate initially. Time window approach stabilizes, while TF/IDF degrades. Classification run time is faster for the Time Window approach. Additional report increases computation time in TF/IDF
- 18. Result Comparisons Group Approach Results Hiew, et-al Text analysis Recall rate ~50% Cubranic, et-al Bayesian learning Text categorization Correctly predicted ~30% duplicates Jalbert, et-al Text Similarity Clustering Recall rate ~51% List size 20 Wang, et-al NLP Execution Information 67-93% detection rate (43-72% with NLP) Wang, et-al Enhanced version of prior algorithm 17-31% improvement over state of art Our approach Time Window/Centroids ~53% recall rate
- 19. Threats to Validity Assumption that the ground truth on duplicates is correct The life cycle of a bug is ever changing Some reports often change state multiple times
- 20. Summary and Future Work SUMMARY Comprehensive study to analyze long term duplicate trends in a large, open source project. Improve search features in duplicate detection by providing a search list. Time interval between reports can be used to improve the search space. FUTURE WORK Compare with other projects (eg: Eclipse) to be able to generalize the approach. Effects on duplicate propagation caused by a user incorrectly selecting a report from the suggested list.
- 21. TF/IDF Compare vector representing a new report to every vector that is currently in the database. Vectors in the database are weighted using TF/IDF to emphasize rare words. The reports are ranked based on their cosine-similarity scores. Report ranking is used to build the suggested list presented to the user. Run time impacted as repository size grows.
- 22. Sliding Window - TF/IDF Apply time window to limit groups under consideration for search. Only the reports within 2,000 groups are considered. Reports are weighted using TF/IDF. Scoring and building of the suggested list same as TF/IDF
- 23. Sliding Window – Centroid Same time window. Reports from the 2,000 groups not immediately searched and weighted using TF/IDF. Centroid vector representing each group is used. Example: Summary 1 unable send email Summary 2 send email function Summary 3 send email after enter recipient The resulting centroid of the group is: 1.0 send, 0.33 unable, 1.0 mail, 0.33 function, 0.33 after, 0.33 enter, 0.33 recipient.
- 24. Sliding Window – Centroid – TD/IDF Uses centroid technique described before. Weight each term in centroids using TF/IDF weighting scheme.