Poster: Improving Bug Localization with Report Quality Dynamics and Query Reformulation
0 likes132 views
This document summarizes a study on improving bug localization through considering the quality of bug reports and reformulating bug report queries. The study analyzes 5,500 bug reports from eight projects and finds that existing bug localization techniques perform poorly when bug reports lack useful information or contain excessive stack traces. Preliminary findings suggest context-aware query reformulation may help address these limitations by improving the quality and relevance of the queries used.
![IMPROVING BUG LOCALIZATION WITH REPORT QUALITY
DYNAMICS AND QUERY REFORMULATION
{MD. MASUDUR RAHMAN AND CHANCHAL K. ROY} UNIVERSITY OF SASKATCHEWAN, COMPUTER SCIENCE
ABSTRACT
In this poster paper, we present a large empirical
study using 5,500 bug reports from eight systems
and replicating three existing studies. Our find-
ings empirically demonstrate how quality dy-
namics of bug reports affect the performances of
the contemporary IR-based bug localizations. Ex-
isting techniques do not perform well if (1) a bug
report lacks rich structured information such as
relevant program entity names, and (2) the bug
report contains excessive structured information
such as stack traces. Our preliminary findings
also suggest that context-aware query reformulations
might help overcome such limitations.
TERM WEIGHT CALCULATION
TF −IDF(t) =
∀d∈DRF
(1+log(ft,d))×log
|D|
nt
Vi = {Ci, Mi},
Ei = {Ci ↔ Mi} ∪ {Ci → Cj, Mi → Mj} | j = i − 1
V =
N
i=1
{Vi}, E =
N
i=1
{Ei}, GST = (V, E)
S(Vi) = (1−ψ)+ψ
j In(Vi)
S(Vj)
|Out(Vj)|
(0 ≤ ψ ≤ 1)
SCHEMATIC DIAGRAM OF THE EMPIRICAL STUDY
Bug report
collection
Bug report
clustering
Clustered
bug reports
Project
codebase
BLUiR
BugLocator
LOBSTER
Result analysis
+ answering RQs
Findings
and insights
CONTACT INFORMATION
Web www.usask.ca/∼masud.rahman
Email masud.rahman@usask.ca
Twitter @masud2336
Phone +1 (306) 241 9293
CONCLUSION & FUTURE RESEARCH
• A large-scale empirical study pointing out
that state-of-the-art IR-based techniques are
not robust to various types of bug reports.
• Quality of the bug report is a major factor.
• Appropriate reformulation of the report con-
tents is warranted prior to bug localization.
• Future research can develop techniques that
take bug report quality into consideration.
REFERENCES
[1] M. M. Rahman and C. K. Roy. Poster: Improv-
ing bug localization with report quality dynamics
and query reformulation. In Proc. ICSE-C, page 02,
Gothenburg, Sweden, May 2018.
RESEARCH QUESTIONS & ANSWERS
• RQ1: How do existing IR-based bug local-
ization techniques perform with the bug re-
ports containing excessive amount of struc-
tured information (e.g., stack traces)?
• RQ2: How do existing IR-based techniques
perform with the bug reports containing
neither program element names nor stack
traces (i.e., only unstructured regular texts)?
• RQ3: Does a single technique perform si-
multaneously well with the bug reports
from both groups?
Figure 1: MAP@K of (a) Baseline (Lucene), (b) BugLocator, (c) BLUiR, and (d) LOBSTER with bug reports containing
excessive structured information (e.g., stack traces)
Figure 2: Hit@10 of (a) Baseline (Lucene), (b) BugLoca-
tor, (c) BLUiR, and (d) LOBSTER with bug reports con-
taining only regular texts
Figure 3: MAP@10 of all four techniques with bug re-
ports containing (a) stack traces, (b) natural language
texts only, (c) program elements, and (d) all bug reports
TRACE GRAPH DEVELOPMENT
Table 1: A Noisy Bug Report
Title: should be able to cast “null"
Bug ID: 31637, Project: eclipse.jdt.debug
Description: When trying to debug an application the variables
tab is empty. Also when I try to inspect or display a variable,
I get following error logged in the eclipse log file:
java.lang.NullPointerException
at org.eclipse.jdt.internal.debug.core.
model.JDIValue.toString(JDIValue.java:362)
at org.eclipse.jdt.internal.debug.eval.ast.
instructions.Cast.execute(Cast.java:88)
at org.eclipse.jdt.internal.debug.eval.ast.engine.
Interpreter.execute(Interpreter.java:44)
at org.eclipse.jdt.internal.debug.eval.ast.engine.
........................................ (8 more).......................................
Cast access
InterpreterJDIValue
toString run
runEvaluation
doEvaluation
EvaluationThread
execute
JDIThread
Thread
EvaluationThread
toString
JDIValue
run
execute
Figure 4: Trace graph of stack traces in Table 1
QUERY REFORMULATION
Table 2: An Example of Query Reformulation
Technique Group Query Terms QE
Baseline
BRST
127 terms from Table 1 after
preprocessing, Bug ID# 31637,
eclipse.jdt.debug
53
Proposed NullPointerException + “Bug
should be able to cast null" +
{JDIValue toString execute
EvaluationThread run}
01
Baseline
BRP E
195 terms (after preprocessing) from
Bug ID# 15036, eclipse.jdt.core
27
Proposed {astvisitor post postvisit previsit pre
file post pre astnode visitor}
01
Baseline
BRNL
32 terms after preprocessing, Bug ID#
475855, eclipse.jdt.ui
30
Proposed Preprocessed report texts
+ {compliance create
preference add configuration
field dialog annotation}
01
Table 3: A Poor Bug Report
Title: [preferences] Mark Occurences Pref Page
Bug ID: 187316, Project: eclipse.jdt.ui
Description: There should be a link to the pref page
on which you can change the color. Namely: Gener-
al/Editors/Text Editors/Annotations. It’s a pain in
the a** to find the pref if you do not know Eclipse’s
preference structure well.](https://image.slidesharecdn.com/icse2018-poster-190521005954/85/Poster-Improving-Bug-Localization-with-Report-Quality-Dynamics-and-Query-Reformulation-1-320.jpg)
Poster: Improving Bug Localization with Report Quality Dynamics and Query Reformulation
- 1. IMPROVING BUG LOCALIZATION WITH REPORT QUALITY DYNAMICS AND QUERY REFORMULATION {MD. MASUDUR RAHMAN AND CHANCHAL K. ROY} UNIVERSITY OF SASKATCHEWAN, COMPUTER SCIENCE ABSTRACT In this poster paper, we present a large empirical study using 5,500 bug reports from eight systems and replicating three existing studies. Our find- ings empirically demonstrate how quality dy- namics of bug reports affect the performances of the contemporary IR-based bug localizations. Ex- isting techniques do not perform well if (1) a bug report lacks rich structured information such as relevant program entity names, and (2) the bug report contains excessive structured information such as stack traces. Our preliminary findings also suggest that context-aware query reformulations might help overcome such limitations. TERM WEIGHT CALCULATION TF −IDF(t) = ∀d∈DRF (1+log(ft,d))×log |D| nt Vi = {Ci, Mi}, Ei = {Ci ↔ Mi} ∪ {Ci → Cj, Mi → Mj} | j = i − 1 V = N i=1 {Vi}, E = N i=1 {Ei}, GST = (V, E) S(Vi) = (1−ψ)+ψ j In(Vi) S(Vj) |Out(Vj)| (0 ≤ ψ ≤ 1) SCHEMATIC DIAGRAM OF THE EMPIRICAL STUDY Bug report collection Bug report clustering Clustered bug reports Project codebase BLUiR BugLocator LOBSTER Result analysis + answering RQs Findings and insights CONTACT INFORMATION Web www.usask.ca/∼masud.rahman Email masud.rahman@usask.ca Twitter @masud2336 Phone +1 (306) 241 9293 CONCLUSION & FUTURE RESEARCH • A large-scale empirical study pointing out that state-of-the-art IR-based techniques are not robust to various types of bug reports. • Quality of the bug report is a major factor. • Appropriate reformulation of the report con- tents is warranted prior to bug localization. • Future research can develop techniques that take bug report quality into consideration. REFERENCES [1] M. M. Rahman and C. K. Roy. Poster: Improv- ing bug localization with report quality dynamics and query reformulation. In Proc. ICSE-C, page 02, Gothenburg, Sweden, May 2018. RESEARCH QUESTIONS & ANSWERS • RQ1: How do existing IR-based bug local- ization techniques perform with the bug re- ports containing excessive amount of struc- tured information (e.g., stack traces)? • RQ2: How do existing IR-based techniques perform with the bug reports containing neither program element names nor stack traces (i.e., only unstructured regular texts)? • RQ3: Does a single technique perform si- multaneously well with the bug reports from both groups? Figure 1: MAP@K of (a) Baseline (Lucene), (b) BugLocator, (c) BLUiR, and (d) LOBSTER with bug reports containing excessive structured information (e.g., stack traces) Figure 2: Hit@10 of (a) Baseline (Lucene), (b) BugLoca- tor, (c) BLUiR, and (d) LOBSTER with bug reports con- taining only regular texts Figure 3: MAP@10 of all four techniques with bug re- ports containing (a) stack traces, (b) natural language texts only, (c) program elements, and (d) all bug reports TRACE GRAPH DEVELOPMENT Table 1: A Noisy Bug Report Title: should be able to cast “null" Bug ID: 31637, Project: eclipse.jdt.debug Description: When trying to debug an application the variables tab is empty. Also when I try to inspect or display a variable, I get following error logged in the eclipse log file: java.lang.NullPointerException at org.eclipse.jdt.internal.debug.core. model.JDIValue.toString(JDIValue.java:362) at org.eclipse.jdt.internal.debug.eval.ast. instructions.Cast.execute(Cast.java:88) at org.eclipse.jdt.internal.debug.eval.ast.engine. Interpreter.execute(Interpreter.java:44) at org.eclipse.jdt.internal.debug.eval.ast.engine. ........................................ (8 more)....................................... Cast access InterpreterJDIValue toString run runEvaluation doEvaluation EvaluationThread execute JDIThread Thread EvaluationThread toString JDIValue run execute Figure 4: Trace graph of stack traces in Table 1 QUERY REFORMULATION Table 2: An Example of Query Reformulation Technique Group Query Terms QE Baseline BRST 127 terms from Table 1 after preprocessing, Bug ID# 31637, eclipse.jdt.debug 53 Proposed NullPointerException + “Bug should be able to cast null" + {JDIValue toString execute EvaluationThread run} 01 Baseline BRP E 195 terms (after preprocessing) from Bug ID# 15036, eclipse.jdt.core 27 Proposed {astvisitor post postvisit previsit pre file post pre astnode visitor} 01 Baseline BRNL 32 terms after preprocessing, Bug ID# 475855, eclipse.jdt.ui 30 Proposed Preprocessed report texts + {compliance create preference add configuration field dialog annotation} 01 Table 3: A Poor Bug Report Title: [preferences] Mark Occurences Pref Page Bug ID: 187316, Project: eclipse.jdt.ui Description: There should be a link to the pref page on which you can change the color. Namely: Gener- al/Editors/Text Editors/Annotations. It’s a pain in the a** to find the pref if you do not know Eclipse’s preference structure well.