A Bug Report Analysis and Search Tool (presentation for M.Sc. degree)

A Bug Report Analysis and Search Tool
M.Sc. Presentation
YguaratÃ£ Cerqueira Cavalcanti
yguarata@gmail.com
Advisor: Silvio Romero de Lemos Meira
Co-Advisor: Eduardo Santana de Almeida
Center for Informatics – Federal University of Pernambuco (UFPE)
http://www.cin.ufpe.br
Reuse in Software Engineering (RiSE)
http://www.rise.com.br
07/03/2009, Recife – Brazil
YguaratÃ£ Cavalcanti (UFPE/CIn, RiSE) A Bug Report Analysis and Search Tool 07/03/2009, Recife – Brazil 1 / 57

Summary
1 Introduction
M.Sc. Context, Motivation, Proposed solution
2 The Bug Report Duplication Problem: A Characterization Study
Deﬁnition, Planning and Operation, Results
3 BAST
Requirements, Architecture, Overview
4 Case Study
Deﬁnition, Planning, Analysis and interpretation
5 Experiment
6 Related Work
7 Conclusion
8 References

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

M.Sc. Context
Change management handles requests for:
new features
correction of errors
improvements
It drives the software maintenance and evolution

Motivation
Software maintenance and evolution are characterised by their huge
cost and slow speed of implementation
Sommerville says that it takes almost 90% of costs
Year Total costs Reference
2000 >90% Erlikh (2000)
1993 75% Eastwood (1993)
1990 >90% Moad (1990)
1990 60–70% Huff (1990)
1988 60–70% Port (1988)
1984 65–75% McKee (1984)
1981 >50% Lientz and Swanson (1981)
1979 67% Zelkowitz et al. (1979)
Table: Conducted studies about software maintenance costs (Koskinen, 2004).

Bug tracking activity
Bug reports management
Verify bug report validity
Analyze the impact of a bug report
Assign a developer
Help with development process in general
Bug reports Software artifact that describes some defect or enhancement;
Generally, bug report submitters are developers, users, or
testers
Bug trackers Bug trackers are used to manage, store and handle change
requests (also known as bug reports)

Bug trackers advantages
Traceability (developers, releases)
Fast identiﬁcation of problems
Metrics (errors per developers, to identify critical components, etc)
Comments
Project history
Examples: Mantis, Bugzilla, Trac, Jyra

A bug report example

A bug report example [2]

Issues coming from bug trackers
Dynamic assignment of bug reports (Anvik et al., 2006);
Change impact analysis and effort estimation of new bug reports
(Song et al., 2006);
Quality of bug report descriptions (Ko et al., 2006);
Software evolution traceability (Sandusky et al., 2004); and
Duplicate bug reports detection consists in avoiding the submission of
bug reports that describe the submitted issue (Hiew, 2006).

The bug report duplication problem
Characterized by the submission of two or more bug reports that describe
the same software issue
Overhead of rework to search and analyze bug reports
People take almost 5-15 minutes to perform search and analysis (Anvik
et al., 2005; Cavalcanti et al., 2008)
10% to 30% of a bug report repository are composed by duplicated bug
reports (Anvik et al., 2005; Runeson et al., 2007; Cavalcanti et al., 2008)
So, costs with
opening bug reports (5-15 minutes)
CCB analysis (5-15 minutes)
developer analysis (5-15 minutes)

Proposed solution
The proposed solution consists in a Web based application that enables
people involved with bug report search and analysis to perform such
tasks more effectively.

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Deﬁnition
The goal of this study was to analyze bug repositories and the activities for
searching and analyzing bug reports
with the purpose of understanding them with respect to the possible factors
that could impact on the duplication problem and their
consequences on software development
from the point of view of the researchers
in the context of software development projects
Questions
Q1: Do the projects have a considerable amount of duplicate bug reports?
Q2: Is the productivity being affected by the bug report duplication problem?
Q3: Is there a common vocabulary for bug report descriptions?
Q4: How are the relationships between master bug reports and duplicate bug
reports characterized?
Q5: Does the type of bug report inﬂuence the amount of duplicates?

Planning and operation
Projects and data selection
All bug reports till June/2008
Project LOC Staff size Bugs Life-time
Bugzilla 55K 340 12829 14
Eclipse 6.5M 352 130095 7
Epiphany 100K 19 10683 6
Evolution 1M 156 72646 11
Firefox 80K 514 60233 9
GCC 4.2M 285 35797 9
Thunderbird 310K 192 19204 8
Tomcat 200K 57 8293 8
Private Project 2M 21 7955 2
Performed at C.E.S.A.R. between June/2008 to August/2008

Results
Question 1: Do the analyzed projects have a considerable amount of
duplicate bug reports?
Metric Bugz. Eclip. Epiph. Evol. Firef. GCC Thund. Tomc. Private Proj. Mean SD
M1 % 23.32 19.44 31.52 43.24 38.39 17.68 49.10 8.24 21.59 28.1 13.4
Question 2: Is the submitters productivity being affected by the bug report
duplication problem?
M2 (min) 05-15 – 05-15 05-15 05-10 05-15 05-15 – 20-30 12.5 1.88
M4 bugs per day 71 722 59 403 334 198 106 46 145 231.5 222.1
Question 3: Is there a common vocabulary for bug report descriptions?
M5 % – 25 – – 22 – – – 35 31.2 9.5

Results [2]
Question 4: How are the relationships between master bug reports and
duplicate bug reports characterized?
One to one relation
bug123: bug3453
One to many relation
bug345: bug45345,
bug465, bug654
Figure: Bug reports grouping.

Results [3]
Question 5: Does the type of bug report inﬂuence the amount of duplicates?

Study summary
All the projects are being affected by the bug report duplication problem;
The productivity is being affected by the bug reports duplication problem;
It is not used a common vocabulary to describe the bug reports;
> 80% of the groups are composed by one-to-one grouping type;
The bug report duplication occur independently of the type of bug reports;
The number of LOC is not a factor for the duplication problem;
The size of the repository is not a factor for duplication;
Projects’ life-time is not a factor for duplication;
The staff size (developers) is not a factor for the duplication problem;
and
The proﬁle of the submitter is a determining factor for the submission of
duplicates: sporadic ≥ average ≥ frequent

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Requirements
Functional requirements
FR1 - Keyword-based search
FR2 - Rank search results based
on bug reports similarity rate
FR3 - Index bug reports from XML
ﬁles
FR4 - Index bug reports from
original database
FR5 - Extract useful information
from bug reports
Non-Functional requirements
NFR1 - Simple and intuitive ﬁlters
interface
NFR2 - Reports about bug
repository status
NFR3 - Integration with most
popular bug report tracking
systems
NFR4 - Log search queries and
user actions
NFR5 - Reasonable similarity rate
NFR6 - Web-based interface with
AJAX

Architecture

Overview

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Deﬁnition
Context. Performed in a real test cycle at a C.E.S.A.R. partner
between July and August 2008
Systematic process to test and open bug reports
Objectives. 1 Which can prevent more duplicate bug reports
2 To consider whether our tool decreases the time spent on
analysis of bug reports
Baseline tool. Internal tool where testers can search for bug reports using
SQL ﬁlters.
Null hypotheses
H0: µtime with BAST > µtime with baseline
µduplicates avoided with BAST < µduplicates avoided with baseline
Alternative hypotheses
H1: µtime with BAST < µtime with baseline
µduplicates avoided with BAST > µduplicates avoided with baseline

Planning
The tool was tested by the Bug Report Master
Responsible for the test cycle
Most experienced tester
Doubt should be saned with him
Case study design: Search and analysis being performed in:
1 step. Internal tool =⇒ BAST
2 step. BAST =⇒ Internal tool
Metrics (manual annotations):
Type of bug reports analyzed
Number of duplicate bug reports avoided
Time spent to analyze similar bug reports
Quantitative analysis: Descriptive statistics
It were analyzed 144 bug reports

Analysis and interpretation
Repository status

Analysis and interpretation [2]
Duplicates found

Time spent

Case study summary
Bug tracker status. More than 50% of duplicates
Duplicates found. Our tool can prevent more duplicates than the
baseline tool
Time spent. The bug report master saved time using our tool
Drawbacks
Case study design. Accommodation of the subject, in which he prefers
to use one tool instead of other.
Amount of bug reports in treatments. The amounts of bug reports that
were analyzed in each treatment were very different.
Lack of subjects. The number of subjects was not sufﬁcient to
generalize the case study results.

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Definition
The goal of this experiment was to analyze a tool to improve search and
analysis of bug reports
with the purpose of evaluating it with respect to its effectiveness and efficiency
on detection of duplicate bug reports and time saving
from the point of view of the researchers
in the context of software development projects
Questions
Q1 Is there a reduction on the number of duplicated bug reports
with the new tool adoption?
Q2 Is there a reduction on the time that submitters spend to perform
the search and analysis of bug reports with the tool adoption?
Q3 Did the submitters have difficulties to use the tool?

Deﬁnition [2]
Objects of study: BAST and Bugzilla.
Quality focus: Effectiveness and efﬁciency of the tool developed.
Context: The adoption of a tool developed to aid the bug report tracking
process, focusing on search and analysis of bug report to avoid
duplicates.
Experiment type: Off-line experiment (Wohlin et al., 2000)
Subjects: 18 Ph.D. and M.Sc. students from the Computer Science
department at Federal University of Pernambuco/Brazil
Performed distributed (no place restrictions)
Bug reports from Firefox open-source project

Planning
Subjects selection. Selected by convenience sampling (Wohlin et al.,
2000; Kitchenham and Pﬂeeger, 2002)
Instrumentation: 32 error descriptions concerning Firefox project
50% with defects that already have bug reports describing them in the
repository
50% with unique/not-reported defects
Guidelines to guide the experiment execution (FAQ)
Time-sheets to collect the time with search and analysis
Quantitative analysis: Descriptive statistics and hypothesis testing
[test-t (Wohlin et al., 2000)]
Qualitative analysis: Questionnaire

Planning [2]
Null hypothesis
H0: µtime with BAST > µtime with baseline
µduplicates avoided with BAST < µduplicates avoided with baseline
Alternative hypothesis
H1: µtime with BAST < µtime with baseline
µduplicates avoided with BAST > µduplicates avoided with baseline
Independent variables. The tool used (BAST or Bugzilla)
Dependent variables. (a) amount of duplicate bug reports and (b) the
time spent with search and analysis

Planning [3]
Experiment design

Analysis and interpretation
Descriptive statistics
Time spent on analysis Bug-reports avoided
BAST Bugzilla BAST Bugzilla
Mean 4.54 4.32 7.56 8.33
Maximum 6.84 9.56 13 12
Minimum 1.78 2.47 0 0
SD 1.49 1.91 3.5 3.2
Table: Descriptive statistics.

Descriptive statistics [2]
Figure: Box plot for time spent.

Descriptive statistics [3]
Figure: Box plot for duplicates avoided.

Hypothesis test
Time spent on analysis Duplicates avoided
t0 0.6292 -1.2466
Degrees of freedom 17 17
p-value 0.5376 0.2294
T distribution 2.11 2.11
Result (t0 > T) H0: not rejected H0: not rejected
Analysis of dependency
BAST time Bugzilla time BAST duplicates Bugzilla duplicates
Years of experience -0.13 -0.02 -0.19 0.18
Number of projects -0.11 0.37 -0.28 -0.025
Bug trackers used -0.16 0.35 -0.26 0.05

Qualitative analysis
BAST features. Seven (7) used the filter features provided by the tool.
BAST Usability. Only one mentioned some difficult to use the filters, and only
one subject had problem with ordering features.
BAST usefulness. Fifteen (15) subjects believe that the way as bug report
details are presented in BAST is useful for the analysis, more than Bugzilla.
Testimonials
“in fact, the way details are presented saves time to check them, since it is not
necessary to open extra tabs or windows to see the details”, and other wrote “it
became easier to identify the duplicate bug reports and navigate among the
details of the them”.

Validity Threats
Boredom
Lack of Historical Data
Environment
Subjects Knowledge on bug reports
Errors re-descriptions and ﬁctitious errors
Halo Effect
Internet Connection Constraints

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Related work
Automated Support for Classifying Software Failure Reports
(Podgurski et al., 2003)
Bug reports: Software failures automatically submitted
Technique: Supervised and unsupervised pattern classiﬁcation and
multivariate visualization
Testing: Batch runs
Dataset: GCC, Jikes, and JavaC
Assisted Detection of Duplicate Bug Reports (Hiew, 2006)
Bug reports: Natural language bug reports
Technique: Organize similar bug reports into centroids using TF-IDF
Testing: Batch runs
Dataset: Firefox, Eclipse, Apache, and Fedora Core
Results: Precision of 29% and recall of 50%

Related work [2]
Detection of Duplicate Defect Reports Using Natural Language
Processing (Runeson et al., 2007)
Technique: Natural Language Processing (NLP)
Testing: Batch runs and a tool
Dataset: Sony Ericsson Mobile Communications
Results: Recall of 40%
An Approach to Detecting Duplicate Bug Reports Using Natural
Language and Execution Information (Wang et al., 2008)
Technique: NLP and execution information
Testing: Batch runs
Dataset: Firefox and Eclipse
Results: Recall of 67%-93% at its best

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

Research contribution
A taxonomy for the bug repositories mining area
The state-of-the-art on mining bug repositories
A characterization of the bug report duplication problem
A tool to reduce the time spent with search and analysis of bug
reports
A case study to evaluate the tool proposed;
An experiment with 18 subjects to evaluate the tool

Papers
Cavalcanti, Y. C., Martins, A. C., de Almeida, E. S., and de Lemos Meira,
S. R. (2008a). Avoiding Duplicate CR reports in Open Source Software
Projects. In The 9th International Free Software Forum (IFSF’08), Porto
Alegre, Brazil.
Cavalcanti, Y. C., de Almeida, E. S., da Cunha, C. E. A., Pinto, E. R., and
Meira, S. R. L. (2008b). The Bug Report Duplication Problem: A
Characterization Study. Technical report, C.E.S.A.R and Federal
University of Pernambuco.
Papers for the Case Study and for the Experiment
And more two journal papers being written (characterization and thesis)

Future Work
Evolve from prototype
Information visualization
Alternative integration methods
Provide integration with other
tools
Search and raking techniques
Comments of a bug report
Number of informal references
Experiment replications

Outline
1 Introduction
3 BAST
4 Case Study
5 Experiment
6 Related Work
7 Conclusion
8 References

References I
Anvik, J., Hiew, L., and Murphy, G. C. (2005). Coping with an open bug
repository. In Proceedings of the 2005 OOPSLA workshop on Eclipse
technology eXchange, pages 35–39, New York, NY, USA. ACM Press.
Anvik, J., Hiew, L., and Murphy, G. C. (2006). Who should ﬁx this bug? In
Proceeding of the 28th International Conference on Software Engineering
(ICSE’06), pages 361–370, New York, NY, USA. ACM Press.
Cavalcanti, Y. C., Almeida, E. S., da Cunha, C. E. A., Pinto, E. R., and Meira,
S. R. L. (2008). The bug-report duplication problem: a characterization
study. Technical report, C.E.S.A.R and Federal University of Pernambuco.
Eastwood, A. (1993). Firm ﬁres shots at legacy systems. Computing Canada,
19(2), 17.
Erlikh, L. (2000). Leveraging legacy system dollars for e-business. IT
Professional, 2(3), 17–23.
Hiew, L. (2006). Assisted Detection of Duplicate Bug Reports. Master’s thesis,
The University of British Columbia.

References II
Huff, F. (1990). Information systems maintenance. The Business Quarterly,
(55), 30–32.
Kitchenham, B. and Pﬂeeger, S. L. (2002). Principles of survey research: part
5: populations and samples. SIGSOFT Software Engineering Notes, 27(5),
17–20.
Ko, A. J., Myers, B. A., and Chau, D. H. (2006). A linguistic analysis of how
people describe software problems. In Proceedings of the Visual
Languages and Human-Centric Computing (VLHCC’06), pages 127–134,
Washington, DC, USA. IEEE Computer Science.
Koskinen, J. (2004). Software maintenance costs.
http://www.cs.jyu.fi/~koskinen/smcosts.htm.
Lientz, B. P. and Swanson, E. B. (1981). Problems in application software
maintenance. Communications of the ACM, 24(11), 763–769.
McKee, J. R. (1984). Maintenance as a function of design. In AFIPS National
Conference Proceeding, volume 53, pages 187–1983.

References III
Moad, J. (1990). Maintaining the competitive edge. Datamation, 4(36), 61–62.
Podgurski, A., Leon, D., Francis, P., Masri, W., Minch, M., Sun, J., and Wang,
B. (2003). Automated support for classifying software failure reports. In
Proceedings of the 25th International Conference on Software Engineering
(ICSE’03), pages 465–475, Washington, DC, USA. IEEE Computer Society.
Port, O. (1988). The software trap – automate or else. Business Week,
9(3051), 142–154.
Runeson, P., Alexandersson, M., and Nyholm, O. (2007). Detection of
duplicate defect reports using natural language processing. In Proceedings
of the 29th International Conference on Software Engineering (ICSE’07),
pages 499–510. IEEE Computer Science Press.
Sandusky, R. J., Gasser, L., and Ripoche, G. (2004). Bug report networks:
Varieties, strategies, and impacts in a f/oss development community. In
Proceedings of the 1st International Workshop on Mining Software
Repositories (MSR’04), pages 80–84, University of Waterloo, Waterloo.

References IV
Sommerville, I. (2007). Software Engineering. Addison Wesley, 8 edition.
Song, Q., Shepperd, M. J., Cartwright, M., and Mair, C. (2006). Software
defect association mining and defect correction effort prediction. IEEE
Transactions on Software Engineering, 32(2), 69–82.
Wang, X., Zhang, L., Xie, T., Anvik, J., and Sun, J. (2008). An approach to
detecting duplicate bug reports using natural language and execution
information. In Proceedings of the 13th International Conference on
Software Engineering (ICSE’08), pages 461–470. ACM Press.
Wohlin, C., Runeson, P., Martin Höst, M. C. O., Regnell, B., and Wesslén, A.
(2000). Experimentation in Software Engineering: An Introduction. The
Kluwer Internation Series in Software Engineering. Kluwer Academic
Publishers, Norwell, Massachusets, USA.
Zelkowitz, M. V., Shaw, A. C., and Gannon, J. D. (1979). Principles of Software
Engineering and Design. Prentice Hall Professional Technical Reference.

A Bug Report Analysis and Search Tool
M.Sc. Presentation
YguaratÃ£ Cerqueira Cavalcanti
yguarata@gmail.com
Advisor: Silvio Romero de Lemos Meira
Co-Advisor: Eduardo Santana de Almeida
Center for Informatics – Federal University of Pernambuco (UFPE)
http://www.cin.ufpe.br
Reuse in Software Engineering (RiSE)
http://www.rise.com.br
07/03/2009, Recife – Brazil

A Bug Report Analysis and Search Tool (presentation for M.Sc. degree)

More Related Content

What's hot (20)

Similar to A Bug Report Analysis and Search Tool (presentation for M.Sc. degree) (20)

Recently uploaded (20)

A Bug Report Analysis and Search Tool (presentation for M.Sc. degree)