Defect localization

.
...... Defect Localization
Leo Arias
Canonical
September 24, 2013
Leo Arias (Canonical) Defect Localization September 24, 2013 1 / 26

Contents
...1 Goals
...2 Isolation
...3 Defect Localization problems
...4 Proposal
...5 Open discussion

Goals
Contents
...1 Goals
...2 Isolation
...4 Proposal

Goals
Goals of Test Automation
Improve quality.
Understand the system under test.
Reduce and not introduce risk.

Goals
Improve Quality
Tests enable us to specify the behavior in various scenarios
captured in a form that we can then execute.
Prevent bugs from being introduced.
Defect localization.

Goals
Defect Localization
Customer tests tell us that some behavior isn’t working as expected,
the Unit tests tell us why.
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails but no unit tests fail, it indicates a missing unit
test.

Isolation
Contents
...1 Goals
...2 Isolation
...4 Proposal

Isolation
Isolation
We should strive to isolate the system under test as much as possible
the parts we choose not to test.
Failure to isolate lead to context sensitivity because we end up testing
too much software all at once.

Isolation
Isolation, cont.

Defect Localization problems
Contents
...1 Goals
...2 Isolation
...4 Proposal

Dependencies

More Dependencies

Canonical Dependencies

Defect Localization
We should be able to pinpoint the bug quickly based on which test fails.
If a customer test fails on a project B because of a change in project A,
it indicates a missing customer test on project A.
(and a missing dependency test on project B).

Proposal
Contents
...1 Goals
...2 Isolation
...4 Proposal

Proposal
The problems
Projects publish an API or behavior without testing it from the point of
view of the consumers.
Projects start relying on an API or behavior even when it’s not properly
tested.
Projects change behaviors without notifying the other projects they
will aﬀect.

Proposal
Proposal
We should focus on defect localization and isolation while adding test
coverage.

Proposal
Short-term actions
When an automated test fails because of an error or a behavior change
on another project:
File a bug describing the problem and the projects affected.
<QA> Subscribe to the bug and follow the process to its
resolution.
Make sure all the people affected are notified.
<developers> Add automated customer tests with every fix to
prevent regressions.
<QA> Make sure no fix comes without a customer test. Check that
they are added on the right project.

Proposal
After the release...
<QA> Analyze the current API and GUI coverage.
<developers> Add isolated tests to fill the holes.
<developers> Make sure we only rely on APIs and behaviors with
automated customer tests.
<QA, developers> Write a test suite specific to test the
integration of all the projects on the different Ubuntu images and
devices.
<design, QA, developers> Start adding user tests since the
expected user experience is defined.
Establish an effective way for us to communicate between teams.

Open discussion
Open discussion
Thank you!
Creative Commons - atribución 3.0

Defect localization

More Related Content

Similar to Defect localization (20)

Recently uploaded (20)

Defect localization