Evolutionary Testing for Crash Reproduction

Evolutionary Testing for Crash
Reproduction
Annibale PanichellaMozhan Soltani Arie van Deursen

Cost of Bug Fixings
https://issues.apache.org/jira/browse/COLLECTIONS-70
Major Bug for
Apache Commons
Collections

Cost of Bug Fixings
Created on
June 2005
Major Bug for
Apache Commons
Collections

Cost of Bug Fixings
Created on
June 2005
Solved on
January 2006
Major Bug for
Apache Commons
Collections

Crash Replication
1) Inspect the stack trace / bug report
2) Analyse the code
3) Replicate the crash
4) Fix the problem

Related Work
Record-and-play approaches
S. Artzi et al., ESEC/FSE 2009.
Stored
Objects
Data from
Heap and Stack
@Test
public void test(){
boolean boolean0 = false;
String string0 = BooleanUtils.toStringOnOff(boolean0);
assertNotNull(string0);
assertEquals("off", string0);
}

Disadvantages:
1) Require software instrumentation
and special hardware deployment
2) Memory and run-time overhead:
• 10%-90% of memory overhead
• 30%-60% of runtime overhead
Record-and-play approaches
S. Artzi et al., ESEC/FSE 2009.
Related Work

Core dump-based approaches
J. Rößler et al., ICST 2013.
Stack Traces Core-dump
+
@Test
public void test(){
boolean boolean0 = false;
}
Related Work

Core dump-based approaches
J. Rößler et al., ICST 2013.
Disadvantages:
1) Requires core-dump at the time of
the crash in addition to stack
traces
2) Ability to replicate crashes
depends on the amount of core-
dump data available
Advantages:
1) No overhead due to system
monitoring
2) SBST Approaches
Related Work

Stack Trace based Crash Replication
N. Chen and Kim, TSE 2015.
STAR (Stack Traced based Automatic crash
Reproduction) uses backward Symbolic Execution
triggering crash preconditions
Symbolic Execution
public void test0() throws Throwable {
java.util.HashMap v1 = new java.util.HashMap();
java.util.HashMap v2 = new java.util.HashMap();
org.apache.commons.collections.map.TransformedMap v3 =
(org.apache.commons.collections.map.TransformedMap)
org.apache.commons.collections.map.TransformedMap.decorate((java.util.Map) v2,
(org.apache.commons.collections.Transformer) null,
(org.apache.commons.collections.Transformer) null);
v3.putAll((java.util.Map) v1);
}
Stack Trace
v2.size == 0
v2 == not null
v1 == not null
v2 instanceOf(Ljava/util/HashMap)
v1 instanceOf(Lorg/apache/commons/collections/map/TransformedMap)
java.lang.IllegalArgumentException:
org.apache.commons.collections.map.AbstractHashedMap.<init> (AbstractHashedMap.java:142)
org.apache.commons.collections.map.AbstractLinkedMap.<init> (AbstractLinkedMap.java:95)
org.apache.commons.collections.map.LinkedMap.<init> (LinkedMap.java:78)
org.apache.commons.collections.map.TransformedMap.transformMap (TransformedMap.java:153)
org.apache.commons.collections.map.TransformedMap.putAll (TransformedMap.java:190)
Preconditions
Test Case

Symbolic Execution
Disadvantages:
1) Crashes with environmental
dependencies (e.g, external ﬁles)
are not replicable
2) Path explosion
3) SMT solver limitations
Advantages:
1) Better than Randoop (random
testing)
2) Better than BugRedux

J. Xuan et al., ESEC/FSE 2015.
Mutation Analysis
@Test
public void test(){
Boolean boolean0 = false;
}
@Test
public void test(){
Boolean boolean0 = false;
String string0 = BooleanUtils.toStringOnOff(null);
}
MuCrash

Mutation Analysis
Disadvantages:
1) Leads to a large number of
unnecessary test cases
2) Crashes requiring method
sequences (not included in the
original test case) are not
reproduced
Advantages:
1) Replicate some crashes not
replicable by STAR
2) No solver is used

What about SBST Unit Test Tools?
Are they competitive if relying on Stack Traces only?

Why Unit Test Tools?
Stack Trace
Target Crash
Bug Name: ACC-48
Library: Apache Commons Collection
Exception Name Root Cause of the Exception

Why Unit Test Tools?
Target Crash
Bug Name: ACC-48
Library: Apache Commons Collection
Class Under Test
Stack Trace
Exception Name
Method Under Test
Line to reach

Test Case n
Test Case...
Test Case1
Test Case 2
Finale Test
Evaluation
Selection
Crossover
Mutation
Genetic
Algorithm
Search Strategy: one target (crash)
at a time approach

Fitness Function
Main Conditions to Satisfy
1) the line (statement) where the exception is thrown has to be covered
2) the target exception has to be thrown
3) the generated stack trace must be as similar to the original one as
possible.
Target Stack Trace
(2)
(1)
(3)
f(t) = 3 x line_coverage + 2 x exception_coverage + trace_similarity
(1) (2) (3)

Fitness Function
Target Stack Trace
(1) (2) (3)
1) line_coverage = approach_level + branch_distance

Fitness Function
Target Stack Trace
(1) (2) (3)
2) exception_coverage = 0 if the target exception in thrown; 1 otherwise

Fitness Function
Target Stack Trace
(1) (2) (3)
2) exception_coverage = 0 if the target exception in thrown; 1 otherwise
3) trace_similarity = cumulative differences with target trace elements
Trace Elements

Trace Similarity
Target Stack Trace
Two trace elements are equal iff:
1) same class name
2) same method name
3) same line
Generated Stack Trace

Trace Similarity
Target Stack Trace
1) same class name (distance = 0)
2) same method name
3) same line

Trace Similarity
Target Stack Trace
2) same method name (distance = 0)
3) same line

Trace Similarity
Target Stack Trace
3) same line (distance = |153 - 148| / (1+ |153 - 148| ) = 0.83)

Trace Similarity
Target Stack Trace
3) same line (distance = |153 - 148| / (1+ |153 - 148| ) = 0.83)
0.5
0.99
0.98
0.85
0.83
Tot = 4.15

Context: 10 real bugs from Apache Commons Collections
Experimented algorithms:
- EvoSuite + our ﬁtness function (30 independent runs)
- STAR (Symbolic execution)
- MuCrash (Mutation analysis)
Empirical Evaluation
Bug ID Version Exception Priority
ACC-4 2.0 NullPointer Major
ACC-35 2.0 UnsupportedOperation Major
ACC-48 3.1 IllegalArgument Major
ACC-53 3.1 ArrayIndexOutOfBound Major
ACC-77 3.1 IllegalState Major
ACC-104 3.1 ArrayIndexOutOfBound Major
ACC-331 3.2 NullPointer Minor
ACC-377 3.2 NullPointer Minor
Used in:

Results
Bug ID
% Successful
Replication
STAR MuCrash
ACC-4 30/30 YES YES
ACC-28 30/30 YES YES
ACC-53 28/30 YES NO
ACC-70 30/30 NO NO
ACC-77 30/30 YES NO
ACC-331 10/30 NO YES
ACC-377 0/30 NO NO

Results
Bug ID
% Successful
Replication
STAR MuCrash
ACC-4 30/30 YES YES
ACC-53 28/30 YES NO
ACC-70 30/30 NO NO
ACC-77 30/30 YES NO
ACC-377 0/30 NO NO
Our solution
replicated 8/10 bugs
STAR replicated 7/10
bugs
MuCrash replicated
6/10 bugs

Results
Bug ID
% Successful
Replication
STAR MuCrash
ACC-4 30/30 YES YES
ACC-53 28/30 YES NO
ACC-70 30/30 NO NO
ACC-77 30/30 YES NO
ACC-377 0/30 NO NO
Replicable by our
SBST solution only

ACC-70
Exception in thread "main" java.lang.NullPointerException at
org.apache.commons.collections.list.TreeList$TreeListIterator.previous (TreeList.java:841)
at java.util.Collections.get(Unknown Source)
at java.util.Collections.iteratorBinarySearch(Unknown Source)
at java.util.Collections.binarySearch(Unknown Source)
at utils.queue.QueueSorted.put(QueueSorted.java:51)
at framework.search.GraphSearch.solve(GraphSearch.java:53)
at search.informed.BestFirstSearch.solve(BestFirstSearch.java:20)
at Hlavni.main(Hlavni.java:66)
Target Stack Trace

ACC-70
TreeList treeList0 = new TreeList();
treeList0.add((Object) null);
TreeList.TreeListIterator treeList_TreeListIterator0 = new
TreeList.TreeListIterator(treeList0, 732);
// Undeclared exception!
treeList_TreeListIterator0.previous();
}
Target Stack Trace
Test generated by our solution

ACC-70
TreeList treeList0 = new TreeList();
treeList0.add((Object) null);
TreeList.TreeListIterator treeList_TreeListIterator0 = new
TreeList.TreeListIterator(treeList0, 732);
// Undeclared exception!
treeList_TreeListIterator0.previous();
}
Target Stack Trace
Test generated by our solution
public Object previous() {
...
if (next == null) {
next = parent.root.get(nextIndex - 1);
} else {
next = next.previous();
}
Object value = next.getValue();
...
}
}
Affected Code
if “parent” is null,
this code generates
an exception

Evolutionary Testing for Crash Reproduction

More Related Content

What's hot (20)

Similar to Evolutionary Testing for Crash Reproduction (20)

More from Annibale Panichella (20)

Recently uploaded (20)

Evolutionary Testing for Crash Reproduction