Recent Research in Search Based Software Testing

1 / 75CS Colloquium (Fall 2013), Ft. Collins, September 9
Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Recent Research on
Search Based Software Testing
Francisco Chicano

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
An optimization problem is a pair: P = (S,f) where:
S is a set of solutions (solution or search space)
f: S → R is an objective function to minimize or maximize
If our goal is to minimize the function we search for:
Global maximum
Local maximum
Global minimum
Local minimum
s ∈ S | f(s ) ≤ f(s), ∀s ∈ S
Optimization Problem
Search Software Testing SBSE

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Optimization Techniques
EXACT APPROXIMATED
AD HOC HEURISTICS METAHEURISTICS
Gradient
Lagrange multipliers
Based on Calculus
Dynamic Programming
Branch and Bound
Enumerative
SA
VNS
TS
Trajectory
EA
ACO
PSO
Population
Optimization Techniques

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Pseudocode of a simple EA
Three main steps: selection, reproduction, replacement
Variation operators ! Make the population to evolve
Recombination: exchange of features
Mutation: generation of new features
P = generateInitialPopulation ();
evaluate (P);
while not stoppingCondition () do
P´ = selectParents (P);
P´ = applyVariationOpterators (P´);
evaluate(P´);
P = selectNewPopulation (P,P´);
end while
return the best solution found
Evolutionary Algorithm

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Genetic Algorithms
•  Individuals
•  Recombination
"  One point
"  Two points
"  Uniform
•  Mutation → bit flips
0111000110
Binary Chromosome
0111000110
1100011001
000110 0111
011001 1100
1001 011010
1010 100101
0111010010 1 0
Evolutionary Algorithm

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
•  What is software testing?
–  It is the process of running a software product or a portion of it in
a controlled environment with a given input followed by the
collection and analysis of the output and/or other relevant
information of the execution.
•  What is the goal of software testing?
–  To find out errors in a portion or the complete software product
and/or to assure with a high probability that the software is
correct (according to the requirements).
Software Testing: Definition and Goal

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Software testing is important because…
60.000 M$ annually
(0,6% GDP) in USA
Software
errors
60 % of project
cost
Software Testing: Impact

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Classification of testing techniques (by goal)
–  Unit testing: test one module of the software.
–  Integration testing: test the interfaces between different modules in the software.
–  System testing: test the complete system.
–  Validation testing: test if the software system fulfills the requirements.
–  Acceptance testing: the client test whether the system is what s/he wants.
–  Regression testing: after a change in the software test whether a new error has
been introduced.
–  Stress testing: test the system under a high load
–  Load testing: test the response of the system under a normal load of work.
Software Testing: Classification

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
1.0, 2.3 Error!
Test case design Test case run Check of results
2.7, 5.4
JUnit, EasyMock,
XMLUnit, Jmeter,
Selenium
Automatic test
case generation
Software Testing: Automatization
Search
Techniques
Search Based
Software Testing

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Search Based Software Engineering
Search Based
Software Engineering
(SBSE)
Search Based
Software Testing

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Search Based Software Engineering
Special Issue on SBSE in Journal of Systems and Software (JSS)
Deadline: November 15th
http://neo.lcc.uma.es/mase/index.php/jss-sbse
Guest editors:
M. Harman & F. Chicano

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Our Research on SBSE
• Software Project Scheduling
• White-box Software Testing
• Testing of Concurrent Systems (based on Model Checking)
• Test Sequences for Functional Testing
• Test Suite Minimization in Regression Testing
• Software Product Lines Testing
• Prioritized Pairwise Combinatorial Interaction Testing
• Testing Complexity

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Problem Formulation Landscape Theory Decomposition SAT Transf. Results

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Test Suite Minimization
Given:
# A set of test cases T = {t1, t2, ..., tn}
# A set of program elements to be covered (e.g., branches) E= {e1, e2, ..., ek}
# A coverage matrix
Find a subset of tests X ⊆ T maximizing coverage and minimizing the testing cost
The single-objective version of this problem consists in finding a subset of
sts X ✓ T with minimum cost covering all the program elements. In formal
rms:
minimize cost(X) =
nX
i=1
ti2X
ci (2)
ubject to:
ej 2 E, 9ti 2 X such that element ej is covered by test ti, that is, mij = 1.
The multi-objective version of the TSMP does not impose the constraint of
ll coverage, but it defines the coverage as the second objective to optimize,
ading to a bi-objective problem. In short, the bi-objective TSMP consists in
nding a subset of tests X ✓ T having minimum cost and maximum coverage.
ormally:
minimize cost(X) =
nX
i=1
ti2X
ci (3)
maximize cov(X) = |{ej 2 E|9ti 2 X with mij = 1}| (4)
e1 e2 e3 ... ek
t1 1 0 1 … 1
t2 0 0 1 … 0
… … … … … …
tn 1 1 0 … 0
M=
3 Test Suite Minimization Problem
When a piece of software is modified, the new software is tested using
previous test cases in order to check if new errors were introduced. This
is known as regression testing. One problem related to regression testing
Test Suite Minimization Problem (TSMP). This problem is equivalent t
Minimal Hitting Set Problem which is NP-hard [17]. Let T = {t1, t2, · · ·
be a set of tests for a program where the cost of running test ti is ci an
E = {e1, e2, · · · , em} be a set of elements of the program that we want to
with the tests. After running all the tests T we find that each test can
several program elements. This information is stored in a matrix M = [m
dimension n ⇥ m that is defined as:
mij =
(
1 if element ej is covered by test ti
0 otherwise
The single-objective version of this problem consists in finding a subs
tests X ✓ T with minimum cost covering all the program elements. In fo
terms:
minimize cost(X) =
nX
i=1
ti2X
ci
subject to:
8ej 2 E, 9ti 2 X such that element ej is covered by test ti, that is, mi
Yoo & Harman

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• The set of solutions is the set of binary strings with length n
• Neighborhood used: one-change neighborhood
# Two solutions x and y are neighbors iff Hamming(x,y)=1
Binary Search Space
0111010010
0111010010
0111010000
0111010110
0111011010
0111000010
0110010010
0101010010
0011010010
1111010010
0111010011 0111110010

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• An elementary landscape is a landscape for which
where
• Grover’s wave equation
Elementary Landscapes: Characterizations
Linear relationship
Eigenvalue
Depend on the
problem/instance
def

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• If f is elementary, the average of f in any sphere and ball of any size around x is a
linear expression of f(x)!!!
Spheres around a Solution
H=1
H=2
H=3
Σf(y’) = λ1 f(x)
Σf(y’’) = λ2 f(x)
Σf(y’’’) = λ3 f(x)
n non-null
possible values
Sutton
Whitley
Langdon

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• What if the landscape is not elementary?
• Any landscape can be written as the sum of elementary landscapes
• There exists a set of eigenfunctions of Δ that form a basis of the
function space (Fourier basis)
Landscape Decomposition
X X X
e1
e2
Elementary functions
(from the Fourier basis)
Non-elementary function
f Elementary
components of f
f < e1,f > < e2,f >
< e2,f >
< e1,f >

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• The elementary landscape decomposition of
is
Elementary Landscape Decomposition of f
constant expression
Tests that cover ei
Tests in the solution that cover ei
Krawtchouk matrix
Computable in
O(nk)
µ1 + d ·
µ1 d ·
f(x) = cov(x) c · cost(x)
F. Chicano et al., SSBSE 2011

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• The elementary landscape decomposition of f2 is
Computable in
O(nk2)
Number of tests that cover ei or ei’
Number of tests in
the solution that
cover ei or ei’
Elementary Landscape Decomposition of f2

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• With the Elementary Landscape Decomposition (ELD) we can compute:
• With the ELD of f and f2 we can compute for any sphere and ball around a solution:
• Distribution of values around the average
Guarded Local Search
: the average : the standard deviation
Chebyshev inequality
At least 75% of the
samples are in the interval
f
Best Best
Apply local searchDon’t apply local search

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
• Steady state genetic algorithm: bit-flip (p=0.01), one-point crossover, elitist replacement
• GA (no local search)
• GLSr (guarded local search up to radius r)
• LSr (always local search in a ball of radius r)
• Instances from the Software-artifact Infrastructure Repository (SIR)
• printtokens
• printtokens2
• schedule
• schedule2
• totinfo
• replace
Guarded Local Search: Experimental Setting
n=100 test cases
Oracle cost c=1..5
100 independent runs
k=100-200 items to cover

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Guarded Local Search: Results
c=1 c=3 c=5
printtokens1
schedule
Time (secs.)

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
NP-hard Problems
In many papers we can read…
“Our optimization problem is NP-hard, and for this reason we use…
… which do not ensure an optimal solution but they are able to find
good solutions in a reasonable time.”
•  Metaheuristic techniques
•  Heuristic algorithms
•  Stochastic algorithms
As far as we know: no efficient (polynomial time) algorithm exists for solving
NP-hard problems
But we know inefficient algorithms (at least exponential time)

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
The SATisfiability Problem
Can we find an assignment of boolean values (true and false) to the variables
such that all the formulas are satisfied?
The first NP-complete problem (Stephen Cook, 1971)
If it can be solved efficiently (polynomial time) then P=NP
The known algorithms solve this problem in exponential time (worst case)
Nowadays, SAT solvers can solve instances with 500 000 boolean variables
This means a search space of 2500 000 ≈ 10150514
State-of-the-art algorithms in SAT

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
The SATisfiability Problem
My favourite
problem
SAT
instance
Use SAT
solvers
Optimal
solution
Main research question:
Can we use the advances of SAT solvers to
solve optimization algorithms up to optimality?
Test Suite
Minimization
Translation to SAT
Algorithms
MiniSAT+
Experimental
Results

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Outline
Original
TSM
Instance
PB
Constraints
SAT
Instance
F. Arito et al., SSBSE 2012

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Pseudo-Boolean Constraints
A Pseudo-Boolean (PB) constraint has the form:
der to use SAT solvers to solve optimization problems,
ate the constraints f(y)  B to Boolean formulas. To
f Pseudo-Boolean constraint plays a main role. A Pseudo
t is an inequality on a linear combination of Boolean va
nX
i=1
aixi B
2 {<, , =, 6=, >, }, ai, B 2 Z, and xi 2 {0, 1}. A PB co
isfied under an assignment if the sum of the coe cient
tisfies the relational operator with respect to B.
nstraints can be translated into SAT instances. The simpl
the PB constraint to an equivalent Boolean formula w
The main drawback of these approaches is that the num
grows exponentially with respect to the variables. In
2 2
...
fm(y)  Bm
er to use SAT solvers to solve optimization problems, we sti
te the constraints f(y)  B to Boolean formulas. To this a
Pseudo-Boolean constraint plays a main role. A Pseudo-Boolea
is an inequality on a linear combination of Boolean variables:
nX
i=1
aixi B
2 {<, , =, 6=, >, }, ai, B 2 Z, and xi 2 {0, 1}. A PB constraint
sfied under an assignment if the sum of the coe cients ai for
isfies the relational operator with respect to B.
nstraints can be translated into SAT instances. The simplest appr
the PB constraint to an equivalent Boolean formula with th
The main drawback of these approaches is that the number of
where
.
fm(y)  Bm
In order to use SAT solvers to solve opti
to translate the constraints f(y)  B to Bo
concept of Pseudo-Boolean constraint plays a m
constraint is an inequality on a linear combina
nX
i=1
aixi B
where 2 {<, , =, 6=, >, }, ai, B 2 Z, and x
to be satisfied under an assignment if the sum
xi = 1 satisfies the relational operator with
PB constraints can be translated into SAT in
translate the PB constraint to an equivalent
variables. The main drawback of these approac
generated grows exponentially with respect t
f2(y)  B2
...
fm(y)  Bm
order to use SAT solvers to solve optimization problems, we
slate the constraints f(y)  B to Boolean formulas. To this
of Pseudo-Boolean constraint plays a main role. A Pseudo-Boo
int is an inequality on a linear combination of Boolean variable
nX
i=1
aixi B
2 {<, , =, 6=, >, }, ai, B 2 Z, and xi 2 {0, 1}. A PB constra
atisfied under an assignment if the sum of the coe cients ai
satisfies the relational operator with respect to B.
constraints can be translated into SAT instances. The simplest ap
te the PB constraint to an equivalent Boolean formula with
es. The main drawback of these approaches is that the number
Can be translated to SAT instances (usually efficient)
Are a higher level formalism to specify a decision problem
Can be the input for MiniSAT+

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Translating Optimization to Decision Problems
Let us assume we want to minimize f(x)
B
f(x)  B
Check
B B B
f(x)  B
Check
f(x)  B
Check
f(x)  B
Check
no no no yes
Optimal solution found
The same can be done with multi-objective problems, but we need more
PB constraints
s of generality that we want to find a solution x⇤
2 X
, f(x⇤
)  f(x) for all the solutions x 2 X. This opti-
transformed in a series of decision problems in which
lution y 2 X for which the constraint f(y)  B holds,
nt integer values. This series of decision problems can
l (minimal) solution of the optimization problem. The
ows. We start with a value of B low enough for the
be. We solve the decision problem to check that it is
ter a loop in which the value of B is increased and the
n. The loop is repeated until the result is satisfiable.
e value of B is the minimal value of f in the search
the decision problem is an optimal solution of the
blem has several objective functions f1, f2, . . . , fm to
nstraint for each objective function:
f1(y)  B1
f2(y)  B2
.
ose without loss of generality that we want to find a solution x⇤
2 X
izes3
f, that is, f(x⇤
)  f(x) for all the solutions x 2 X. This opti-
oblem can be transformed in a series of decision problems in which
ve is to find a solution y 2 X for which the constraint f(y)  B holds,
Z takes di↵erent integer values. This series of decision problems can
find the optimal (minimal) solution of the optimization problem. The
ould be as follows. We start with a value of B low enough for the
o be unsatisfiabe. We solve the decision problem to check that it is
e. Then, we enter a loop in which the value of B is increased and the
s checked again. The loop is repeated until the result is satisfiable.
oop finishes, the value of B is the minimal value of f in the search
the solution to the decision problem is an optimal solution of the
n problem.
ptimization problem has several objective functions f1, f2, . . . , fm to
we need one constraint for each objective function:
f1(y)  B1
f2(y)  B2
...
that minimizes3
f, that is, f(x⇤
)  f(x) for all the solutions x 2 X
mization problem can be transformed in a series of decision problem
the objective is to find a solution y 2 X for which the constraint f(y)
where B 2 Z takes di↵erent integer values. This series of decision pr
be used to find the optimal (minimal) solution of the optimization pr
procedure could be as follows. We start with a value of B low eno
constraint to be unsatisfiabe. We solve the decision problem to chec
unsatisfiable. Then, we enter a loop in which the value of B is increa
constraint is checked again. The loop is repeated until the result is
Once the loop finishes, the value of B is the minimal value of f in
space and the solution to the decision problem is an optimal solu
optimization problem.
If the optimization problem has several objective functions f1, f2
minimize, we need one constraint for each objective function:
f1(y)  B1
f2(y)  B2
...
fm(y)  Bm
In order to use SAT solvers to solve optimization problems, w
…

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
PB Constraints for the TSM Problem
e1 e2 e3 ... em
t1 1 0 1 … 1
t2 0 0 1 … 0
… … … … … …
tn 1 1 0 … 0
M=
previous test cases in order to check if new errors were introduced. This
is known as regression testing. One problem related to regression testing
Test Suite Minimization Problem (TSMP). This problem is equivalent t
Minimal Hitting Set Problem which is NP-hard [17]. Let T = {t1, t2, · · ·
be a set of tests for a program where the cost of running test ti is ci an
E = {e1, e2, · · · , em} be a set of elements of the program that we want to
with the tests. After running all the tests T we find that each test can
several program elements. This information is stored in a matrix M = [m
dimension n ⇥ m that is defined as:
mij =
(
1 if element ej is covered by test ti
0 otherwise
The single-objective version of this problem consists in finding a subs
tests X ✓ T with minimum cost covering all the program elements. In fo
terms:
minimize cost(X) =
nX
i=1
ti2X
ci
subject to:
8ej 2 E, 9ti 2 X such that element ej is covered by test ti, that is, mi
The multi-objective version of the TSMP does not impose the constra
full coverage, but it defines the coverage as the second objective to opti
leading to a bi-objective problem. In short, the bi-objective TSMP consi
finding a subset of tests X ✓ T having minimum cost and maximum cove
Formally:
n
ase is not included. We also introduce m binary variables e
ch program element to cover. If ej = 1 then the correspondin
by one of the selected test cases and if ej = 0 the elem
y a selected test case.
lues of the ej variables are not independent of the ti variable
j must be 1 if and only if there exists a ti variable for whic
1. The dependence between both sets of variables can be wr
ing 2m PB constraints:
ej 
nX
i=1
mijti  n · ej 1  j  m.
n see that if the sum in the middle is zero (no test is co
j) then the variable ej = 0. However, if the sum is greater
ow we need to introduce a constraint related to each objectiv
o transform the optimization problem in a decision probl
in Section 2.2. These constraints are:
variables are not independent of the ti variables. A given
and only if there exists a ti variable for which mij = 1
ence between both sets of variables can be written with
nstraints:
nX
i=1
mijti  n · ej 1  j  m. (5)
the sum in the middle is zero (no test is covering the
ariable ej = 0. However, if the sum is greater than zero
introduce a constraint related to each objective function
the optimization problem in a decision problem, as we
. These constraints are:
nX
i=1
citi  B, (6)
mX
the following 2m PB constraints:
ej 
nX
i=1
mijti  n · ej 1  j 
We can see that if the sum in the middle is zero (no
element ej) then the variable ej = 0. However, if the sum
ej = 1. Now we need to introduce a constraint related to ea
in order to transform the optimization problem in a dec
described in Section 2.2. These constraints are:
nX
i=1
citi  B,
mX
j=1
ej P,
Cost Coverage

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Example
e
ough a small example how to model with PB con-
SMP according to the methodology above described.
E = {e1, e2, e3, e4} and M:
e1 e2 e3 e4
t1 1 0 1 0
t2 1 1 0 0
t3 0 0 1 0
t4 1 0 0 0
t5 1 0 0 1
t6 0 1 1 0
-obj TSMP we need to instantiate Eqs. (5), (6) and
 t1 + t2 + t4 + t5  4e1 (10)
 t2 + t6  4e2 (11)
 t1 + t3 + t6  4e3 (12)
 t5  4e4 (13)
t5 1 0 0 1
t6 0 1 1 0
If we want to solve the 2-obj TSMP we need to instantia
(7). The result is:
e1  t1 + t2 + t4 + t5  4e1
e2  t2 + t6  4e2
e3  t1 + t3 + t6  4e3
e4  t5  4e4
t1 + t2 + t3 + t4 + t5 + t6  B
e1 + e2 + e3 + e4 P
where P, B 2 N.
If we are otherwise interested in the 1-obj version the form
t1 + t2 + t4 + t5 1
t2 + t6 1
t1 + t3 + t6 1
t5 1
t1 + t2 + t3 + t4 + t5 + t6  B
(7). The result is:
e1  t1 + t2 + t4 + t5  4e1
e2  t2 + t6  4e2
e3  t1 + t3 + t6  4e3
e4  t5  4e4
t1 + t2 + t3 + t4 + t5 + t6  B
e1 + e2 + e3 + e4 P
where P, B 2 N.
If we are otherwise interested in the 1-obj version th
t1 + t2 + t4 + t5 1
t2 + t6 1
t1 + t3 + t6 1
t5 1
t1 + t2 + t3 + t4 + t5 + t6  B
e2  t2 + t6  4e2
e3  t1 + t3 + t6  4e3
e4  t5  4e4
t1 + t2 + t3 + t4 + t5 + t6  B
e1 + e2 + e3 + e4 P
where P, B 2 N.
If we are otherwise interested in the 1-obj version the for
t1 + t2 + t4 + t5 1
t2 + t6 1
t1 + t3 + t6 1
t5 1
t1 + t2 + t3 + t4 + t5 + t6  B
Bi-objective problem
Single-objective problem
(total coverage)
f(x)  B
e1  t1 + t2 + t4 + t5  6e1
e2  t2 + t6  6e2
e3  t1 + t3 + t6  6e3
e4  t5  6e4

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Algorithm for Solving the 2-obj TSM
Cost
Coverage
Total coverage
With coverage=|E| increase cost until success
Decrease cost and find
the maximum coverage
again
and again

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Instances from the Software-artifact Infrastructure Repository (SIR)
TSM Instances
The result is an instance with fewer test cases but having the same P
These transformed instances were solved using Algorithm 2. Table
size of the test suites with and without the reduction for each progr
observe a really great reduction in the number of test cases when t
approach is used.
Table 1: Details of the instances used in the experiments
Instance Original Size Reduced Size Elements to co
printtokens 4130 40
printtokens2 4115 28
replace 5542 215
schedule 2650 4
schedule2 2710 13
tcas 1608 5
totinfo 1052 21
In Table 2 we present the Pareto optimal set and the Pareto fr
t is an instance with fewer test cases but having the same Pareto front.
nsformed instances were solved using Algorithm 2. Table 1 shows the
e test suites with and without the reduction for each program. We can
really great reduction in the number of test cases when the previous
is used.
stance Original Size Reduced Size Elements to cover
rinttokens 4130 40 195
rinttokens2 4115 28 192
eplace 5542 215 208
chedule 2650 4 126
chedule2 2710 13 119
cas 1608 5 54
otinfo 1052 21 117
ble 2 we present the Pareto optimal set and the Pareto front for the
Tests
http://sir.unl.edu/portal/index.php !
Cost of each test: 1

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Pareto Front
80
82.5
85
87.5
90
92.5
95
97.5
100
0 1 2 3 4 5 6 7 8 9
coveragelevel
number of test cases
Pareto front
printtokens
printtokens2
replace
schedule
schedule2
tcas
totinfo
printtokens
printtokens2
replace
schedule
schedule2
tcas
totinfo
The result is an instance with fewer test cases but having the same Pareto front.
These transformed instances were solved using Algorithm 2. Table 1 shows the
size of the test suites with and without the reduction for each program. We can
observe a really great reduction in the number of test cases when the previous
approach is used.
Instance Original Size Reduced Size Elements to cover
printtokens 4130 40 195
printtokens2 4115 28 192
replace 5542 215 208
schedule 2650 4 126
schedule2 2710 13 119
tcas 1608 5 54
totinfo 1052 21 117
In Table 2 we present the Pareto optimal set and the Pareto front for the
instances described above. The columns “Tests” and “Elements” correspond to
the functions cost and cov of the 2-obj TSMP. The column “Coverage” is the
number of covered elements divided by the total number of elements. The optimal
solution for the 1-obj TSMP can be found in the lines with 100% coverage,
as explained in Section 3. It is not common to show the Pareto optimal set
or the Pareto front in numbers in the multi-objective literature because only
approximate Pareto fronts can be obtained for NP-hard problems. However, in
this case we obtain the exact Pareto fronts and optimal sets, so we think that this
The result is an instance with fewer test cases but having the same Pareto front.
approach is used.
replace 5542 215 208
schedule 2650 4 126
schedule2 2710 13 119
tcas 1608 5 54
totinfo 1052 21 117
In Table 2 we present the Pareto optimal set and the Pareto front for the
instances described above. The columns “Tests” and “Elements” correspond to
the functions cost and cov of the 2-obj TSMP. The column “Coverage” is the
number of covered elements divided by the total number of elements. The optimal
solution for the 1-obj TSMP can be found in the lines with 100% coverage,
as explained in Section 3. It is not common to show the Pareto optimal set
or the Pareto front in numbers in the multi-objective literature because only
approximate Pareto fronts can be obtained for NP-hard problems. However, in
this case we obtain the exact Pareto fronts and optimal sets, so we think that this
80
0 1 2 3
tcas
Fig. 1: Pareto fron
Table 3: Information about clauses-t
gorithm 2, average coverage and num
rithms for the instances from SIR.
Instance Ratio Algorithm 2
Original (s) Reduced (
printtokens 4.61 3400.74 2.1
printtokens2 4.61 3370.44 1.4
replace 4.62 1469272.00 345.6
schedule 2.19 492.38 0.2
schedule2 4.61 195.55 0.2
tcas 4.61 73.44 0.3
totinfo 4.53 181823.50 0.9
independent runs. However, the requ
printtokens, printtokens2 and re
rest of the programs. However, we sh
the result is an optimal solution, as
Tests Time

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Pareto Front Table 2: Pareto optimal set and Front for the instances of SIR.
Instance Elements Tests Coverage Solution
printtokens 195 5 100% (t2222, t2375, t3438, t4100, t4101)
194 4 99.48% (t1908, t2375, t4099, t4101)
192 3 98.46% (t1658, t2363, t4072)
190 2 97.43% (t1658, t3669)
186 1 95.38% (t2597)
printtokens2 192 4 100% (t2521, t2526, t4085, t4088)
190 3 98.95% (t457, t3717, t4098)
188 2 97.91% (t2190, t3282)
184 1 95.83% (t3717)
replace 208 8 100% (t306, t410, t653, t1279, t1301, t3134, t4057, t4328)
207 7 99.51% (t309, t358, t653, t776, t1279, t1795, t3248)
206 6 99.03% (t275, t290, t1279, t1938, t2723, t2785)
205 5 98.55% (t426, t1279, t1898, t2875, t3324)
203 4 97.59% (t298, t653, t3324, t5054)
200 3 96.15% (t2723, t2901, t3324)
195 2 93.75% (t358, t5387)
187 1 89.90% (t358)
schedule 126 3 100% (t1403, t1559, t1564)
124 2 98.41% (t1570, t1595)
122 1 96.82% (t1572)
schedule2 119 4 100% (t2226, t2458, t2462, t2681)
118 3 99.15% (t101, t1406, t2516)
117 2 98.31% (t2461, t2710)
116 1 97.47% (t1584)
tcas 54 4 100% (t5, t1191, t1229, t1608)
53 3 98.14% (t13, t25, t1581)
50 2 92.59% (t72, t1584)
44 1 81.48% (t217)
totinfo 117 5 100% (t62, t118, t218, t1000, t1038)
115 4 98.29% (t62, t118, t913, t1016)
113 3 96.58% (t65, t216, t913)
111 2 94.87% (t65, t919)
110 1 94.01% (t179)

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Reduction in the Number of Test Cases
Since we are considering cost 1 for the tests, we can apply an a priori reduction
in the original test suite
e1 e2 e3 ... em
t1 1 0 0 … 1
t2 1 0 1 … 1
… … … … … …
tn 1 1 0 … 0
Test t1 can be removedThe result is an instance with fewer test cases but having the same Pareto front.
approach is used.
replace 5542 215 208
schedule 2650 4 126
schedule2 2710 13 119
tcas 1608 5 54
totinfo 1052 21 117

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Results with the Reduction
The optimal Pareto Front for the reduced test suite can be found from 200 to
180 000 times faster
Table 3: Information about clauses-to-variab
gorithm 2, average coverage and number of te
rithms for the instances from SIR.
Instance Ratio Algorithm 2
Original (s) Reduced (s) Avg.
printtokens 4.61 3400.74 2.17 100
printtokens2 4.61 3370.44 1.43 100
replace 4.62 1469272.00 345.62 100
schedule 2.19 492.38 0.24 100
schedule2 4.61 195.55 0.27 100
tcas 4.61 73.44 0.33 100
totinfo 4.53 181823.50 0.96 100

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Background Testing SAT Transform. Results
R. Lopez-Herrejon et al., ICSM 2013

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Software Product Lines
A product line is a set of related products developed from a shared set of assets
•  The products have similar characteristics
•  The products have unique characteristics
Advantages
•  Support customization
•  Improves reuse
•  Reduce time to market
Product Lines are Pervasive
19
BMW Car Configurator
© RELH

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Software Product Lines
In Software Product Lines the product is Software
They are modelled using Feature Models

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Feature Models
Fig. 1. Graph Product Line Feature Model
Mandatory features Optional features
Exclusive-or
relations
Inclusive-or
relations
Cross-tree constraints
Graph Product Line Feature Model

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Testing of Software Product Lines
Fig. 1. Graph Product Line Feature Model
In a feature model, each feature (except the root) has one
parent feature and can have a set of child features. Notice
here that a child feature can only be included in a feature
combination of a valid product if its parent is included as
well. The root feature is always included. There are four kinds
of feature relationships: i) Mandatory features are depicted
with a ﬁlled circle. A mandatory feature is selected whenever
its respective parent feature is selected. For example, features
Driver and GraphType, ii) Optional features are depicted
with an empty circle. An optional feature may or may not
be selected if its respective parent feature is selected. An
example is feature Weight, iii) Exclusive-or relations are
depicted as empty arcs crossing over a set of lines connecting
a parent feature with its child features. They indicate that
exactly one of the features in the exclusive-or group must be
selected whenever the parent feature is selected. For example,
if feature Search is selected, then either feature DFS or
feature BFS must be selected, iv) Inclusive-or relations are
depicted as ﬁlled arcs crossing over a set of lines connecting
a parent feature with its child features. They indicate that
at least one of the features in the inclusive-or group must
be selected if the parent is selected. If for instance, feature
Algorithms is selected then at least one of the features Num,
CC, SCC, Cycle, Shortest, Prim, and Kruskal must
be selected. Besides the parent-child relations, features can
also relate across different branches of the feature model with
the so called Cross-Tree Constraints (CTC). Figure 1 shows
some of the CTCs of our feature model2
. For instance, Cycle
requires DFS means that whenever feature Cycle is
an
co
pr
an
co
by
m
va
al
w
of
of
{0
xi
co
w
bo
C
w
pr
ea
de
W
co
The GPL Feature Model is small: 73 distinct products
But the number of products grows exponentially with the
number of features…
… and testing each particular product is not viable

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Testing of SPLs: Combinatorial Interaction Testing
Assuming each features has been tested in isolation, most of the defects come
from the interaction between features
Combinatorial Interaction Testing consists in selecting the minimum number of
products that covers all t-wise interactions (t-wise coverage).
ZipMe
Compress Extract Checksum Adapt GZIP ArchCheck CRC
✔ ✔ ✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔ ✔
1
2
3
4
5
6
64 productsExample: 2-wise
ICPL
CASA

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Testing of SPLs: Multi-Objective Formulation
If we don’t have the resources to run all the tests, which one to choose?
Multi-objective formulation:
minimize the number of products
maximize the coverage (t-wise interactions)
The solution is not anymore a table of products, but a Pareto set
by the number of variables dj,k,l that
as:
1X
=1
fX
k=j+1
3X
l=0
dj,k,l (7)
rogram is composed of the goal (7)
(f 1) constraints given by (2) to (6)
he FM expressed with the inequalities
e number of variables of the program
). The solution to this zero-one linear
with the maximum coverage that can
ucts.
ALGORITHM
e for obtaining the optimal Pareto set
This algorithm takes as input the FM
l Pareto set. It starts by adding to the
always in the set: the empty solution
one arbitrary solution (with coverage
ons of the set of features). After that
h successive zero-one linear programs
creasing number of products starting
l model is solved using a extended
. This solver provides a test suite with
This solution is stored in the optimal
m stops when adding a new product to
crease the coverage. The result is the
EXPERIMENTS
s how the evaluation was carried out
sis. The experimental corpus of our
y a benchmark of 118 feature models,
ts ranges from 16 to 640 products, that
om the SPL Conqueror [16] and the
s. The objectives to optimize are the
//minisat.se/MiniSat+.html
want the algorithm to be as fast as possible. For comparison
these experiments were run in a cluster of 16 machines with
Intel Core2 Quad processors Q9400 at 2.66 GHz and 4 GB
running Ubuntu 12.04.1 LTS managed by the HT Condor 7.8.4
manager. Each experiment was executed in one core.
We computed the Pareto optimal front for each model.
Figure 2 shows this front for our running example GPL,
where the total coverage is obtained with 12 products, and
for every test suite size the obtained coverage is also optimal.
As our approach is able to compute the Pareto optimal front
for every feature model in our corpus, it makes no sense to
analyze the quality of the solutions. Instead, we consider more
interesting to study the scalability of our approach. For that,
we analyzed the execution time of the algorithm as a function
of the number of products represented by the feature model as
shown in Figure 3. In this ﬁgure we can observe a tendency:
the higher the number of products, the higher the execution
time. Although it cannot be clearly appreciated in the ﬁgure,
the execution time does not grow linearly with the number of
products, the growth is faster than linear.
Fig. 2. Pareto optimal front for our running example (GPL).
GPL
2-wise interactions

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Testing of SPLs: Approach
Original
SPLT
Instance
PB
Constraints
SAT
Instance

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Modelling SPLT using PseudoBoolean constraints
All the variables are boolean {0,1}
The values of the signature are:
–  00 (both unselected)
–  10 (only first selected)
–  01 (only second selected)
–  11 (both selected)
1 Variables
Variable Meaning
xp,i Presence of feature i in product p
cp,i,j,k Product p covers the pair (i, j) with signature k
di,j,k The pair (i, j) with signature k is covered by some product
k takes values 0, 1, 2 and 3.
2 Equations
For each p:
The equations related to the constraints of the model
For all p, i and j:
2cp,i,j,3  xp,i + xp,j  1 + cp,i,j,3 (1

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Equations of the model
–  For each product p
•  Constraints imposed by the Feature Model
–  For each product p and pair of features i and j
tions
ions related to the constraints of the model
i and j:
2cp,i,j,3  xp,i + xp,j  1 + cp,i,j,3
2cp,i,j,2  xp,i + (1 xp,j)  1 + cp,i,j,3
2cp,i,j,1  (1 xp,i) + xp,j  1 + cp,i,j,3
2cp,i,j,0  (1 xp,i) + (1 xp,j)  1 + cp,i,j,3
and k:

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Equations of the model (cont.)
–  For each pair of features i and j and signature k
•  n is the number of products
–  Objective: maximize coverage
2cp,i,j,0  (1 xp,i) + (1 xp,j)  1 + cp,i,j,3
nd k:
di,j,k 
X
p
cp,i,j,k  ndi,j,k
umber of products
max :
X
i,j,k
di,j,k
k:
di,j,k 
X
p
cp,i,j,k  ndi,j,k
mber of products
max :
X
i,j,k
di,j,k

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
 f:
,k,0 (2)
(3)
(4)
(5)
product.
when all
d dj,k,l,
k with
herwise.
he dj,k,l
lities for
Algorithm 1 Algorithm for obtaining the optimal Pareto set.
optimal set {;};
cov[0] 0;
cov[1] Cf
2 ;
sol arbitraryValidSolution(fm);
i 1;
while cov[i] 6= cov[i 1] do
optimal set optimal set [ {sol};
i i + 1;
m prepareMathModel(fm,i);
sol solveMathModel(m);
cov[i] |sol|;
end while
number of products required to test the SPL and the achieved

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Testing of SPLs: Results
of the redundant constraints can incr
adding more constraints could help th
a solution. We plan to study the right b
and augmenting constraints. Second,
feature models to further study the sca
ACKNOWLEDGEME
Funded by Austrian Science Fund
N15 and Lise Meitner Fellowship M
Ministry of Economy and Competitive
contract TIN2011-28194 and fellowsh
REFERENCES
[1] P. Zave, “Faq sheet on
http://www.research.att.com/ pamela/faq.h
[2] K. Pohl, G. Bockle, and F. J. van der L
Engineering: Foundations, Principles and
Experiments on 118 feature models taken from
SPLOT repository (http://www.splot-research.org)
SPL Conqueror (http://wwwiti.cs.uni-magdeburg.de/~nsiegmun/SPLConqueror/)
16 to 640 products
Intel Core2 Quad Q9400
2.66 GHz, 4 GB

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Problem Definition Proposal Results
J. Ferrer et al., GECCO 2012

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
The tester identifies the relevant test aspects (parameters) and defines
corresponding classes (parameter values)
A test case is a set of n values, one for each parameter
A kind of functional (black-box) testing
Combinatorial Interaction Testing

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
The coverage criterion will determine the degree of parameter interaction
The coverage criterion is defined by its strength t (t-wise)
In prioritized CIT, each t-tuple has a weight that measures the importance
Tool Support: CTE XL
Prioritized Combinatorial Interaction Testing

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Each Used Coverage (EUC)
Weight Coverage (WC)
P1
P2
P3
P4
P5
P6
P7
EUC = 3 / 7 = 0.43
WC = (0.20+0.25+0.15) / 0.9 = 0.66
Pair! Weight!
P1! 0.20!
P2! 0.25!
P3! 0.15!
P4! 0.10!
P5! 0.10!
P6! 0.05!
P7! 0.05!
∑ Pi! 0.9!
Coverage

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
# Access Method Operation Priv. EUC WC
1 Browser (with JavaScript) Edit Normal 0.12 0.30
2 Browser (with JavaScript) Edit Superuser 0.19 0.48
3 Browser (with JavaScript) Create Normal 0.27 0.60
4 Native Tool Create Superuser 0.38 0.71
5 Native Tool Edit Normal 0.50 0.80
6 Browser (with JavaScript) Delete Normal 0.58 0.88
7 Native Tool Delete Superuser 0.62 0.92
8 Browser (no JavaScript) Edit Normal 0.69 0.94
9 Browser (no JavaScript) Create Superuser 0.77 0.96
10 Database-Frontend Edit Normal 0.85 0.98
11 Database-Frontend Create Superuser 0.92 0.99
12 Browser (no JavaScript) Delete Superuser 0.96 0.99
13 Database-Frontend Delete Normal 1.00 1.00
30% weight
coverage with
one test case
With the weight
coverage we cover
most important
interactions of
components in the
first test cases
Coverage: example

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
60% weight
coverage with
only three
test cases
With the weight
coverage we cover
most important
interactions of
components in the
first test cases
Coverage: example

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
92% weight
coverage with
just seven
test cases
The six less
important test
cases just
suppose 8%
Coverage: example

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
GS is a constructive algorithm that reduces the problem step by step
It constructs the solution by generating the best test datum at a time
Proposal: Genetic Solver

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Browser (with JavaScript) Edit Edit Normal
Browser (with JavaScript) Normal
Browser (with JavaScript) Edit Normal
P1
New Test Datum
P2
P3
P1
P2
P3
P4
P5
Pn
P6
P7
P1
P2
P3
P4
P5
Pn
P6
P7
P4
P5
Pn
P6
P7
Set of ‘Remaining pairs’ New ‘Remaining pairs’Removing pairs
WC(RP) = 1
WC(TD) = 0.3
WC(RP’) = 0.7WC(RP) – WC(TD)= WC(RP’)
Proposal: Genetic Solver

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Distribution Description
D1 (equal weights) All classes have the same weight
D2 (50/50 split)
Half of the weight for each classification
are set to 0.9, the other half to 0.1
D3 (1/vmax2 split)
All weights of classes for a classification
are equal to 1/vmax2, where vmax is the
number of classes associated with the
classification.
D4 (random) Weights are randomly distributed
Set of benchmarks and distributions proposed by Bryce and Colbourn.
Scenario # Classes
S1
S2
S3
S4
S5
S6
S7
S8
Results: Experimental Evaluation

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
We compare 8 scenarios, 4 distributions, and different coverage values
–  Coverage values: 25%, 50%, 66%, 75%, 90%, 95%, and 99%
GS is the best in 6 out of 8 scenarios
GS is the best for all distributions
Scenario GS PPC PPS
S1 0 0 12
S2 8 18 0
S3 9 3 0
S4 14 9 1
S5 13 6 3
S6 24 1 0
S7 5 2 0
S8 19 6 -
Total 92! 45! 19!
Distribution! PPC! PPS!
D1-GS! 28↑! 10↓! 29↑! 8↓!
D2-GS! 26↑! 9↓! 42↑! 3↓!
D3-GS! 19↑! 10↓! 29↑! 8↓!
D4-GS! 22↑! 6↓! 41↑! 4↓!
Total! 95↑! 35↓! 141↑! 23↓!
Times one algorithm is better
than the others
Times a significant difference
between GS and the others exists
Results: Comparison with PPC and PPS (B&M)

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
We compared the algorithm focused on different coverage values
It is important to obtain the best results for intermediate values of coverage
The GS always performs better than the others for these coverage values
" PPS" PPC" GS
79%
21%
0%
50% Coverage
72%
16%
12%
75% Coverage
66%
14%
20%
100 %Coverage
Results: Comparison with PPC and PPS (B&M)

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Scenario GS DDA BDD
S1 2 2 2
S2 11 0 0
S3 6 1 0
S4 8 0 2
S5 7 3 0
S6 11 0 0
S7 3 0 1
S8 3 1 0
Totals 51! 7! 5!
Distribution! DDA! BDD!
D1-GS! 7↑! 7↓! 15↑! 5↓!
D2-GS! 10↑! 1↓! 16↑! 2↓!
D3-GS! 16↑! 0↓! 18↑! 1↓!
D4-GS! 16↑! 2↓! 22↑! 1↓!
Totals! 49↑! 10↓! 71↑! 9↓!
GS is the best in 7 out of 8 scenarios. It draws on the scenario S1.
GS is the best in 3 out of 4 distributions. It draws in D1 with DDA.
Times one algorithm is better
than the others
Times there exist significant
differences between the algorithms
Comparison among GS and the state-of-the-art algorithms:
Deterministic Density Algorithm (DDA): Bryce and Colbourn (2006)
Binary Decision Diagrams (BDD): Salecker et al. (2011)
Results: Comparison with DDA and BDD

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
79%
7% 14%
100% Coverage
" BDD" DDA" GS
50% Coverage
75%
21%
4%
75% Coverage
GS always performs better than the state-of-the-art algorithms
It is always better than the other algorithms for all scenarios and distributions for 50%
weight coverage.
Results: Comparison with DDA and BDD

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Background Proposal Results
J. Ferrer et al., Inf. & Soft. Tech. 2013

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Motivation
How difficult is to test the Software using
automatic test data generation?
Can we estimate the difficulty
analyzing the program?
This kind of measure would be useful
to estimate the testing costs

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
v(G)=E-N+2
One entry and exit node
v(G)=E-N+1
Strongly connected graph
McCabe’s Cyclomatic Complexity
What does it mean?
–  Number of linearly independent paths of the graph
–  Linearly independent paths find errors with high probability
–  The measure is an estimation of the cost of testing the code

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Other Measures
measure in use. An examination of the main metrics reveals that most of them confuse the complexity
of a program with its size. The underlying idea of these measures are that a program will be much more
di cult to work with than a second one if, for example, it is twice the size, has twice as many control paths
leading through it, or contains twice as many logical decisions. Unfortunately, these various ways in which
a program may increase in complexity tend to move in unison, making it di cult to identify the multiple
dimensions of complexity.
In this section we present the measures used in this study. In a first group we select the main measures
that we found in the literature:
• Lines of Code (LOC)
• Source Lines of Code (SLOC)
• Lines of Code Equivalent (LOCE)
• Total Number of Disjunctions (TNDj)
• Total Number of Conjunctions (TNCj)
• Total Number of Equalities (TNE)
• Total Number of Inequalities (TNI )
• Total Number of Decisions (TND)
• Number of Atomic Conditions per Decision (CpD)
• Nesting Degree (N )
• Halstead’s Complexity (HD)
• McCabe’s Cyclomatic Complexity (MC)
Let’s have a look at the measures that are directly based on source lines of code (in C-based languages).
The LOC measure is a count of the number of semicolons in a method, excluding those within comments and
string literals. The SLOC measure counts the source lines that contain executable statements, declarations,
and/or compiler directives. However, comments, and blank lines are excluded. The LOCE measure [31] is
• n1 = the number of distinct operators
• n2 = the number of distinct operands
• N1 = the total number of operators
• N2 = the total number of operands
From these values, six measures can be defined:
• Halstead Length (HL): N = N1 + N2
• Halstead Vocabulary (HV): n = n1 + n2
• Halstead Volume (HVL): V = N ⇤ log2 n
• Halstead Di culty (HD): HD = n1
2 ⇤ N2
n2
• Halstead Level (HLV): L = 1
HD
• Halstead E↵ort (HE): E = HD ⇤ V
• Halstead Time (HT): T = E
18
• Halstead Bugs (HB): B = V
3000
The most basic one is the Halstead Length, which simp
A small number of statements with a high Halstead Volume
quite complex. The Halstead Vocabulary gives a clue on th
highlights if a small number of operators are used repeatedl
operators are used, which will inevitably be more complex.
vocabulary to give a measure of the amount of code written.
the complexity based on the number of unique operators a
is to write and maintain. The Halstead Level is the invers
the program is prone to errors. The Halstead E↵ort attemp
take to recode a particular method. The Halstead Time is t
adequate to provide my desired level of confidence on it”.
complexity and di culty of discovering errors. Software c
McCabe are related to the di culty programmers experien
used in providing feedback to programmers about the com
managers about the resources that will be necessary to mai
Since McCabe proposed the cyclomatic complexity, it
concluded that one of the obvious intuitive weaknesses of
provision for distinguishing between programs which perfor
form massive amounts of computation, provided that they
noticed that cyclomatic complexity is the same for N nested
Moreover, we find the same weaknesses in the group of Halst
degree, which may increase the e↵ort required by the progra
Halstead’s weakness is a factor to consider that a nested sta
also studied the LOCE measure that takes into account wh
The proposed existing measures of decision complexity te
of a program control structure like McCabe’s complexity. Su
subprogram level, but metrics computed at those levels will d
the values of these metrics primarily depend upon the num
suggests that we can compute a size-independent measure o
of decisions within a program. In addition we have considere
The resulting expression takes into account the nesting deg
this assumption, we consider in this paper two measures de
• Density of Decisions (DD) = TND/LOC.
• Density of LOCE (DLOCE) = LOCE/LOC.
Finally, we present the dynamic measure used in the stud
measure, it is necessary to determine which kind of elemen
measures can be defined depending on the kind of element
defined as the percentage of statements (sentences) that are
which is the percentage of branches of the program that a
most of the related articles in the literature. We formally d
We have analyzed several measures as the total number of disjunction
(AND operator) that appear in the source code, these operators join a
(in)equalities is the number of times that the operator (! =) == is found
The total number of decisions and the number of atomic conditions per de
The nesting degree is the maximum number of control flow statements t
In the following paragraphs we describe the McCabe’s cyclomatic compl
measures in detail.
Halstead complexity measures are software metrics [14] introduced by
Halstead’s Metrics are based on arguments derived from common sense, i
The metrics are based on four easily measurable properties of the progra
• n1 = the number of distinct operators
• n2 = the number of distinct operands
• N1 = the total number of operators
• N2 = the total number of operands
From these values, six measures can be defined:
• Halstead Length (HL): N = N1 + N2
Legend

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
based on the piece of code shown in Figure 1. First, we compute the
piece of code, which can be seen in Figure 2. This CFG is composed of
Bs. Interpreted as a Markov chain, the basic blocks are the states, and
robabilities to move from one basic block to another. These probabilities
d to a concrete branch. For example, to move from BB1 to BB2 in our
< 2) must be true, then according to equations (2) to (10) the probability
P(y < 2) P(x < 0) ⇤ P(y < 2) = 1
2 + 1
2
1
2 ⇤ 1
2 = 3
4 = 0.75.
/* BB1 */
if (x < 0) || (y < 2)
{
/* BB2 */
y=5;
}
else
{
/* BB3 */
x=y-3;
while (y > 5) || (x > 5)
{
/* BB4 */
y=x-5;
}
/* BB5 */
x=x-3;
}
/* BB6 */
ode to illustrate the computation of Branch Coverage Expectation
transition probabilities, we build the transition matrix that represents
0
0.0 0.75 0.25 0.0 0.0 0.0
1
BB1
BB2 BB3
BB5
BB6
BB4
P(BB6,BB1)=1
P(BB2,BB6)=1
P(BB5,BB6)=1
P(BB3,BB5)=0.25
P(BB3,BB4)=0.75
P(BB4,BB4)=0.75
P(BB4,BB5)=0.25
P(BB1,BB3)=0.25
P(BB1,BB2)=0.75
Figure 2: The CFG and the probabilities used to build a Markov Chain of the piece o
exactly once the BB1 and BB6 in one run. In this way, the start and the end of t
a E[BBi] = 1. An example of the computation of the mathematical expectation is
E(BB2) = ⇡2
⇡1
= 0.1875
0.2500 = 0.75.
Table 1: Stationary probabilities and the frequency of appearance of the basic blocks of the p
Stationary Probabilities ⇡i Frequency of Appearance E
Our Proposal: Branch Coverage Expectation
P(c1&&c2) = P(c1) ⇤ P(c2), (7)
P(c1||c2) = P(c1) + P(c2) P(c1) ⇤ P(c2), (8)
P(¬c1) = 1 P(c1), (9)
P(a < b) =
1
2
, (10)
P(a  b) =
1
2
, (11)
P(a > b) =
1
2
, (12)
P(a b) =
1
2
, (13)
P(a == b) = q, (14)
P(a! = b) = 1 q, (15)
where c1 and c2 are conditions.
We establish a 1/2 probability when the operators are ordering relational operators (<, , >, ). Despite
that the actual probability in a random situation is not always 1/2, we have selected the value with the
lowest error rate. In the case of equalities and inequalities the probabilities are q and 1 q, respectively,
where q is a parameter of the measure and its value should be adjusted based on the experience. Satisfying
an equality is, in general, a hard task and, thus, q should be close to zero. This parameter could be highly
dependent on the data dependencies of the program. The quality of the complexity measure depends on a
good election for q. We delay to future work the thorough analysis of this parameter. Based on a previous
phase for setting parameters, we use q = 1/16 for the experimental analysis.
Then, once we have the CFG completed with the transition probabilities, the generation of the transition
matrix is automatic. This matrix relates the states and the probability to move from one to another. We
assume, without loss of generality, that there is only one entry and exit basic block in the code. Then,

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
BB1
BB2 BB3
BB5
BB6
BB4
P(BB6,BB1)=1
P(BB2,BB6)=1
P(BB5,BB6)=1
P(BB3,BB5)=0.25
P(BB3,BB4)=0.75
P(BB4,BB4)=0.75
P(BB4,BB5)=0.25
P(BB1,BB3)=0.25
P(BB1,BB2)=0.75
G and the probabilities used to build a Markov Chain of the piece of code of Figure 1
d BB6 in one run. In this way, the start and the end of the program always have
ple of the computation of the mathematical expectation is:
0.75.
bilities and the frequency of appearance of the basic blocks of the piece of code shown above.
Stationary Probabilities ⇡i Frequency of Appearance E[BBi]
0.2500 1.00
0.1875 0.75
0.0625 0.25
f every state in a Markov chain can be reached from every othe
hain is irreducible. For irreducible Markov chains having only p
istribution of the states q(t) tends to a given probability distributi
robability distribution ⇡ is called the stationary distribution and
near equations:
⇡T
P = ⇡T
,
⇡T
1 = 1.
.2. Definition of BCE
In our case the Markov model is built from the Control Flow G
tates of the Markov chain are the basic blocks of the program. A b
hat is executed sequentially with no interruption. It has one entry
nly the last instruction can be a jump. Whenever the first instru
est of the instructions are necessarily executed exactly once, in ord
Markov chain we must assign a value to the edges among vertic
ranches are computed according to the logical expressions that ap
efine this probability as follows:
BB1
BB2 BB3
BB5
BB6
BB4
P(BB6,BB1)=1
P(BB2,BB6)=1
P(BB5,BB6)=1
P(BB3,BB5)=0.25
P(BB3,BB4)=0.75
P(BB4,BB4)=0.75
P(BB4,BB5)=0.25
P(BB1,BB3)=0.25
P(BB1,BB2)=0.75
Figure 2: The CFG and the probabilities used to build a Markov Chain of the piece of code of Figure 1
exactly once the BB1 and BB6 in one run. In this way, the start and the end of the program always have
a E[BBi] = 1. An example of the computation of the mathematical expectation is:
E(BB2) = ⇡2
⇡1
= 0.1875
0.2500 = 0.75.
Table 1: Stationary probabilities and the frequency of appearance of the basic blocks of the piece of code shown above.
BB1 0.2500 1.00
BB2 0.1875 0.75
BB3 0.0625 0.25
BB4 0.1875 0.75
BB5 0.0625 0.25
BB6 0.2500 1.00
The stationary probability and the frequency of appearance of the BBs in a single execution of the piece
of code can be seen in Table 1. Now, we are able to compute the probability of appearance of a branch in
one single run. For example the expectation of traversing the branch BB3 BB4 is:
E[BB3, BB4] = E(BB3) ⇤ P34 = 1
4 ⇤ 3
4 = 3
16 = 0.1875.
In Figure 3 we show the mathematical expectations of traversing all the branches of the CFG of our
example in one single execution. So, finally we can compute the BCE by averaging the expectations of
traversing the branches which have a value lower than 1/2. We have excluded those values equals to 1/2
because both branches have the same value. In case all branches have the expectation of 1/2, then the BCE
is 1/2. In addition, a program with a Branch Coverage Expectation value of 1/2 would be the easiest one
to be tested. In this example the value of BCE is :
BCE = E[BB1,BB3]+E[BB3,BB4]+E[BB3,BB5]+E[BB4,BB5]+E[BB5,BB6]
5 =
1
4 + 3
16 + 1
16 + 3
16 + 1
4
5 = 3
16 = 0.1875.
ng parameters, we use q = 1/16 for the experiment
we have the CFG completed with the transition pr
matic. This matrix relates the states and the pro
ut loss of generality, that there is only one entry
tain a positive-recurrent irreducible Markov chain
c block (labelled as BB1) having probability 1. W
quency of appearance of each basic block in one s
y probability of a basic block is the probability of
y state. On the other hand, the frequency of appea
traversing the basic block in one single execution
E[BBi] =
⇡i
⇡1
,
mpleted with the transition probabilities, the generation of the transitio
elates the states and the probability to move from one to another. W
that there is only one entry and exit basic block in the code. Then
ent irreducible Markov chain we add a fictional link from the exit t
B1) having probability 1. We then compute the stationary probabilit
of each basic block in one single execution of the program (E[BBi]
c block is the probability of appearance in infinite program execution
hand, the frequency of appearance of a basic block is the mathematic
block in one single execution and is computed as:
E[BBi] =
⇡i
⇡1
, (16
ty of the entry basic block, BB1.
ng a branch (i, j) is computed from the frequency of appearance of th
ility to take the concrete branch from the previous basic block as:
E[BBi, BBj] = E[BBi] ⇤ Pij (17
overage Expectation (BCE) as the average of the values E[BBi, BB
Markov Chain
Compute
stationary
distribution
Expected BB executions in 1 run
Expected branch execution in 1 run

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
BCE =
1
|A|
X
(i,j)2A
E[BBi, BBj].
we analyze the new complexity measure over program artifacts,
on based on the piece of code shown in Figure 1. First, we c
his piece of code, which can be seen in Figure 2. This CFG is
BBs. Interpreted as a Markov chain, the basic blocks are the
E(BB2) = ⇡1
= 0.2500 = 0.75.
Table 1: Stationary probabilities and the frequency of appearance of the basic blocks of the piece of code shown above.
BB1 0.2500 1.00
BB2 0.1875 0.75
BB3 0.0625 0.25
BB4 0.1875 0.75
BB5 0.0625 0.25
BB6 0.2500 1.00
The stationary probability and the frequency of appearance of the BBs in a single execution of the pie
of code can be seen in Table 1. Now, we are able to compute the probability of appearance of a branch
one single run. For example the expectation of traversing the branch BB3 BB4 is:
E[BB3, BB4] = E(BB3) ⇤ P34 = 1
4 ⇤ 3
4 = 3
16 = 0.1875.
In Figure 3 we show the mathematical expectations of traversing all the branches of the CFG of o
example in one single execution. So, ﬁnally we can compute the BCE by averaging the expectations
traversing the branches which have a value lower than 1/2. We have excluded those values equals to 1
because both branches have the same value. In case all branches have the expectation of 1/2, then the BC
is 1/2. In addition, a program with a Branch Coverage Expectation value of 1/2 would be the easiest o
to be tested. In this example the value of BCE is :
BCE = E[BB1,BB3]+E[BB3,BB4]+E[BB3,BB5]+E[BB4,BB5]+E[BB5,BB6]
5 =
1
4 + 3
16 + 1
16 + 3
16 + 1
4
5 = 3
16 = 0.1875.
9
Branch Coverage Expectation
E[BBi, BBj] = E[BBi] ⇤ Pij
h Coverage Expectation (BCE) as the average of the values E[BBi
a program has a low value of BCE then a random test case gener
ber of test cases to obtain full branch coverage. The BCE is bounded
A be the set of edges with E[BBi, BBj] < 1/2:
A = {(i, j)|E[BBi, BBj] <
1
2
}.
7
Most difficult branches to cover BB1
BB2 BB3
BB5
BB6
BB4
P(BB6,BB1)=1
P(BB2,BB6)=1
P(BB5,BB6)=1
P(BB3,BB5)=0.25
P(BB3,BB4)=0.75
P(BB4,BB4)=0.75
P(BB4,BB5)=0.25
P(BB1,BB3)=0.25
P(BB1,BB2)=0.75
Figure 2: The CFG and the probabilities used to build a Markov Chain of the piece of c
exactly once the BB1 and BB6 in one run. In this way, the start and the end of the
a E[BBi] = 1. An example of the computation of the mathematical expectation is:
E(BB2) = ⇡2
⇡1
= 0.1875
0.2500 = 0.75.
Table 1: Stationary probabilities and the frequency of appearance of the basic blocks of the piec
Stationary Probabilities ⇡i Frequency of Appearance E[B
BB1 0.2500 1.00
BB2 0.1875 0.75
BB3 0.0625 0.25

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Table A.10: The correlation coe cients among all the measures analyzed in the benchmark 100%CP
HD MC LOCE N DD DLOCE BCE LOC SLOCTNDj TNCj TNE TNI TND CpD HL HV HVL HLV HE HT HB ES GA RND
HD - 0.796 0.786 -0.108 0.052 -0.035 0.285 0.932 0.853 0.742 0.731 0.644 0.639 0.799 0.454 0.870 0.842 0.864 1.0 0.920 0.920 0.864 0.070 -0.101 0.077
MC 0.796 - 0.965 0.266 0.519 0.408 0.025 0.805 0.962 0.925 0.934 0.829 0.811 0.985 0.524 0.976 0.969 0.977 -0.796 0.954 0.954 0.977 -0.150-0.226-0.074
LOCE 0.786 0.965 - 0.344 0.515 0.474 -0.038 0.796 0.974 0.884 0.882 0.822 0.789 0.976 0.501 0.945 0.938 0.945 -0.786 0.921 0.921 0.945 -0.186-0.251-0.133
N -0.108 0.266 0.344 - 0.765 0.877 -0.540 -0.207 0.180 0.235 0.240 0.311 0.234 0.276 0.136 0.138 0.127 0.139 0.108 0.089 0.089 0.139 -0.543-0.381-0.434
DD 0.052 0.519 0.515 0.765 - 0.912 -0.377 -0.043 0.405 0.449 0.489 0.485 0.437 0.538 0.283 0.368 0.367 0.372 -0.052 0.302 0.302 0.372 -0.439-0.304-0.311
DLOCE-0.035 0.408 0.474 0.877 0.912 - -0.485 -0.132 0.336 0.352 0.380 0.410 0.353 0.418 0.217 0.270 0.258 0.271 0.035 0.208 0.208 0.271 -0.504-0.345-0.397
BCE 0.285 0.025 -0.038-0.540-0.377 -0.485 - 0.307 0.081 0.065 0.008 -0.124 0.009 0.017 0.078 0.121 0.129 0.120 -0.285 0.159 0.159 0.120 0.510 0.375 0.534
LOC 0.932 0.805 0.796 -0.207-0.043 -0.132 0.307 - 0.879 0.753 0.730 0.634 0.646 0.810 0.419 0.891 0.892 0.890 -0.932 0.910 0.910 0.890 0.136 -0.053 0.120
SLOC 0.853 0.962 0.974 0.180 0.405 0.336 0.081 0.879 - 0.884 0.878 0.794 0.778 0.973 0.492 0.975 0.970 0.975 -0.853 0.960 0.960 0.975 -0.091-0.194-0.050
TNDj 0.742 0.925 0.884 0.235 0.449 0.352 0.065 0.753 0.884 - 0.773 0.813 0.719 0.897 0.515 0.919 0.908 0.919 -0.742 0.900 0.900 0.919 -0.119-0.175-0.036
TNCj 0.731 0.934 0.882 0.240 0.489 0.380 0.008 0.730 0.878 0.773 - 0.734 0.806 0.905 0.497 0.913 0.901 0.913 -0.731 0.895 0.895 0.913 -0.158-0.235-0.072
TNE 0.644 0.829 0.822 0.311 0.485 0.410 -0.124 0.634 0.794 0.813 0.734 - 0.618 0.822 0.435 0.798 0.785 0.797 -0.644 0.779 0.779 0.797 -0.272-0.279-0.207
TNI 0.639 0.811 0.789 0.234 0.437 0.353 0.009 0.646 0.778 0.719 0.806 0.618 - 0.799 0.439 0.794 0.791 0.795 -0.639 0.774 0.774 0.795 -0.121-0.201-0.095
TND 0.799 0.985 0.976 0.276 0.538 0.418 0.017 0.810 0.973 0.897 0.905 0.822 0.799 - 0.503 0.961 0.959 0.962 -0.799 0.935 0.935 0.962 -0.147-0.226-0.082
CpD 0.454 0.524 0.501 0.136 0.283 0.217 0.078 0.419 0.492 0.515 0.497 0.435 0.439 0.503 - 0.524 0.518 0.523 -0.454 0.514 0.514 0.523 -0.089-0.132 0.035
HL 0.870 0.976 0.945 0.138 0.368 0.270 0.121 0.891 0.975 0.919 0.913 0.798 0.794 0.961 0.524 - 0.991 1.0 -0.870 0.989 0.989 1.0 -0.071-0.180-0.012
HV 0.842 0.969 0.938 0.127 0.367 0.258 0.129 0.892 0.970 0.908 0.901 0.785 0.791 0.959 0.518 0.991 - 0.994 -0.842 0.971 0.971 0.994 -0.061-0.172-0.003
HVL 0.864 0.977 0.945 0.139 0.372 0.271 0.120 0.890 0.975 0.919 0.913 0.797 0.795 0.962 0.523 1.0 0.994 - -0.864 0.987 0.987 1.0 -0.072-0.181-0.011
HLV -1.0 -0.796-0.786 0.108 -0.052 0.035 -0.285 -0.932-0.853-0.742-0.731-0.644-0.639-0.799-0.454 -0.870-0.842-0.864 - -0.920-0.920-0.864 -0.070 0.101 -0.077
HE 0.920 0.954 0.921 0.089 0.302 0.208 0.159 0.910 0.960 0.900 0.895 0.779 0.774 0.935 0.514 0.989 0.971 0.987 -0.920 - 1.0 0.987 -0.046-0.168 0.006
HT 0.920 0.954 0.921 0.089 0.302 0.208 0.159 0.910 0.960 0.900 0.895 0.779 0.774 0.935 0.514 0.989 0.971 0.987 -0.920 1.0 - 0.987 -0.046-0.168 0.006
HB 0.864 0.977 0.945 0.139 0.372 0.271 0.120 0.890 0.975 0.919 0.913 0.797 0.795 0.962 0.523 1.0 0.994 1.0 -0.864 0.987 0.987 - -0.072-0.181-0.011
ES 0.070 -0.150-0.186-0.543-0.439 -0.504 0.510 0.136 -0.091-0.119-0.158-0.272-0.121-0.147-0.089 -0.071-0.061-0.072-0.070-0.046-0.046-0.072 - 0.365 0.445
GA -0.101-0.226-0.251-0.381-0.304 -0.345 0.375 -0.053-0.194-0.175-0.235-0.279-0.201-0.226-0.132 -0.180-0.172-0.181 0.101 -0.168-0.168-0.181 0.365 - 0.403
RND 0.077 -0.074-0.133-0.434-0.311 -0.397 0.534 0.120 -0.050-0.036-0.072-0.207-0.095-0.082 0.035 -0.012-0.003-0.011-0.077 0.006 0.006 -0.011 0.445 0.403 -
Table A.11: The correlation coe cients among all the measures analyzed in the benchmark ¬100%CP
HD MC LOCE N DD DLOCE BCE LOC SLOCTNDj TNCj TNE TNI TND CpD HL HV HVL HLV HE HT HB ES GA RND
HD - 0.698 0.359 -0.062 0.023 0.014 0.051 0.664 0.648 0.653 0.651 0.557 0.569 0.463 0.441 0.764 0.576 0.747 -1.0 0.872 0.872 0.747 0.069 0.067 0.079
MC 0.698 - 0.571 0.257 0.432 0.351 -0.142 0.472 0.667 0.936 0.937 0.803 0.827 0.718 0.671 0.782 0.762 0.786 -0.698 0.803 0.803 0.786 -0.177-0.168-0.173
LOCE 0.359 0.571 - 0.692 0.590 0.833 -0.461 0.414 0.717 0.435 0.432 0.479 0.485 0.814 0.086 0.564 0.503 0.560 -0.359 0.524 0.524 0.560 -0.461-0.452-0.476
N -0.062 0.257 0.692 - 0.708 0.870 -0.575 -0.160 0.190 0.163 0.161 0.229 0.220 0.502 -0.031 0.020 0.009 0.019 0.062 -0.007-0.007 0.019 -0.563-0.554-0.589
DD 0.023 0.432 0.590 0.708 - 0.774 -0.426 -0.178 0.280 0.306 0.304 0.385 0.372 0.723 0.026 0.089 0.056 0.087 -0.023 0.070 0.070 0.087 -0.476-0.473-0.497
DLOCE 0.014 0.351 0.833 0.870 0.774 - -0.556 -0.113 0.284 0.247 0.243 0.308 0.291 0.593 0.013 0.096 0.076 0.095 -0.014 0.073 0.073 0.095 -0.577-0.564-0.602
BCE 0.051 -0.142-0.461-0.575-0.426 -0.556 - 0.075 -0.143-0.078-0.079-0.200-0.138-0.318 0.080 -0.021-0.006-0.020-0.051 0.001 0.001 -0.020 0.714 0.698 0.732
LOC 0.664 0.472 0.414 -0.160-0.178 -0.113 0.075 - 0.857 0.398 0.397 0.386 0.406 0.494 0.144 0.906 0.821 0.901 -0.664 0.874 0.874 0.901 0.102 0.099 0.116
SLOC 0.648 0.667 0.717 0.190 0.280 0.284 -0.143 0.857 - 0.533 0.532 0.549 0.572 0.834 0.152 0.916 0.813 0.910 -0.648 0.875 0.875 0.910 -0.137-0.137-0.137
TNDj 0.653 0.936 0.435 0.163 0.306 0.247 -0.078 0.398 0.533 - 0.849 0.753 0.781 0.555 0.747 0.702 0.697 0.707 -0.653 0.731 0.731 0.707 -0.110-0.101-0.102
TNCj 0.651 0.937 0.432 0.161 0.304 0.243 -0.079 0.397 0.532 0.849 - 0.753 0.771 0.551 0.746 0.702 0.697 0.707 -0.651 0.731 0.731 0.707 -0.116-0.107-0.111
TNE 0.557 0.803 0.479 0.229 0.385 0.308 -0.200 0.386 0.549 0.753 0.753 - 0.623 0.600 0.544 0.633 0.619 0.636 -0.557 0.646 0.646 0.636 -0.278-0.270-0.270
TNI 0.569 0.827 0.485 0.220 0.372 0.291 -0.138 0.406 0.572 0.781 0.771 0.623 - 0.619 0.559 0.658 0.645 0.662 -0.569 0.671 0.671 0.662 -0.207-0.198-0.204
TND 0.463 0.718 0.814 0.502 0.723 0.593 -0.318 0.494 0.834 0.555 0.551 0.600 0.619 - 0.132 0.688 0.605 0.683 -0.463 0.648 0.648 0.683 -0.338-0.336-0.348
CpD 0.441 0.671 0.086 -0.031 0.026 0.013 0.080 0.144 0.152 0.747 0.746 0.544 0.559 0.132 - 0.394 0.436 0.402 -0.441 0.437 0.437 0.402 0.026 0.026 0.031
HL 0.764 0.782 0.564 0.020 0.089 0.096 -0.021 0.906 0.916 0.702 0.702 0.633 0.658 0.688 0.394 - 0.932 0.999 -0.764 0.980 0.980 0.999 -0.021-0.018-0.010
HV 0.576 0.762 0.503 0.009 0.056 0.076 -0.006 0.821 0.813 0.697 0.697 0.619 0.645 0.605 0.436 0.932 - 0.946 -0.576 0.874 0.874 0.946 -0.040-0.030-0.022
HVL 0.747 0.786 0.560 0.019 0.087 0.095 -0.020 0.901 0.910 0.707 0.707 0.636 0.662 0.683 0.402 0.999 0.946 - -0.747 0.974 0.974 1.0 -0.023-0.020-0.011
HLV -1.0 -0.698-0.359 0.062 -0.023 -0.014 -0.051 -0.664-0.648-0.653-0.651-0.557-0.569-0.463-0.441 -0.764-0.576-0.747 - -0.872-0.872-0.747 -0.069-0.067-0.079
HE 0.872 0.803 0.524 -0.007 0.070 0.073 0.001 0.874 0.875 0.731 0.731 0.646 0.671 0.648 0.437 0.980 0.874 0.974 -0.872 - 1.0 0.974 0.004 0.005 0.016
HT 0.872 0.803 0.524 -0.007 0.070 0.073 0.001 0.874 0.875 0.731 0.731 0.646 0.671 0.648 0.437 0.980 0.874 0.974 -0.872 1.0 - 0.974 0.004 0.005 0.016
HB 0.747 0.786 0.560 0.019 0.087 0.095 -0.020 0.901 0.910 0.707 0.707 0.636 0.662 0.683 0.402 0.999 0.946 1.0 -0.747 0.974 0.974 1.0 -0.023-0.020-0.011
ES 0.069 -0.177-0.461-0.563-0.476 -0.577 0.714 0.102 -0.137-0.110-0.116-0.278-0.207-0.338 0.026 -0.021-0.040-0.023-0.069 0.004 0.004 -0.023 - 0.954 0.940
GA 0.067 -0.168-0.452-0.554-0.473 -0.564 0.698 0.099 -0.137-0.101-0.107-0.270-0.198-0.336 0.026 -0.018-0.030-0.020-0.067 0.005 0.005 -0.020 0.954 - 0.950
RND 0.079 -0.173-0.476-0.589-0.497 -0.602 0.732 0.116 -0.137-0.102-0.111-0.270-0.204-0.348 0.031 -0.010-0.022-0.011-0.079 0.016 0.016 -0.011 0.940 0.950 -
Correlation Study with All the Measures
Study over
2600 programs

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Correlation with Cov. of an Automatic TD Gen.
Study over
2600 programs
subsection we have provided an overview of static measures that are part of our study. Now, we
measures that are similar and those that are di↵erent. In the next section we show the measures
ore correlated with the branch coverage, which is the way we measure the di culty of testing a
lation Between Coverage and Static Measures
previous section we showed the basic relationship among the static measures, in this section
the branch coverage in the study. The existing correlations between the branch coverage and
measures studied give us an idea of which static measures are useful to determine a priori the
of the automatic test data generation task. In this study we have applied three di↵erent test
ators, two based on evolutionary techniques (ES, GA) and one based on random testing (RND).
ationship between the most important static measures and the average branch coverage for all the algorithms. We
high value of correlation for each algorithm and benchmark.
100%CP ¬100%CP
ES GA RND ES GA RND
MC -0.150 -0.226 -0.074 -0.177 -0.168 -0.173
HD 0.070 -0.101 0.077 0.069 0.067 0.079
LOCE -0.186 -0.251 -0.133 -0.461 -0.452 -0.476
N -0.543 -0.381 -0.434 -0.563 -0.554 -0.589
DD -0.439 -0.304 -0.311 -0.476 -0.473 -0.497
DLOCE -0.504 -0.345 -0.397 -0.577 -0.564 -0.602
BCE 0.510 0.375 0.534 0.714 0.698 0.732
st question we should answer is if there exists a link between the coverage and the traditional
f code complexity: McCabe’s, Halstead’s, and LOCE. In Table 7 we show the correlation coe -
he most important static measures and the branch coverage obtained with three automatic test
ators. The correlations between Halstead’s Di culty and the coverage are very low, so the answer

Introduction
Test Suite
Minimization
Software
Product Lines
Pairwise
Prioritized CIT
Testing
Complexity
Approximated Behaviour of RND
complexity measures or other known static measures like the nesting degree.
ther use of the Branch Coverage Expectation
e detailed in Section 3 for each branch (BBi, BBj) the expected number of test cases
t is 1/E[BBi, BBj]. Then, given a number of test cases x, we can compute the number
ld be theoretically traversed if the tester execute x random test cases, according to th
f(x) =
⇢
(i, j)
1
E[BBi, BBj]
< x .
ks to this estimation, we propose a theoretical prediction about the behaviour of an au
erator based on random testing.
gure 9 we show a plot for a particular program with the expected theoretical behavi
experimental data obtained using the average branch coverage of the 30 independent e
generator for that program. The features of this test program are shown in Table 9. T
how that our theoretical prediction and the experimental data are very similar. Th
n is more optimistic because it does not take into account data dependencies. At th
gorithm, the experimental behaviour is better than the theoretical prediction, but in
erage (close to 90%), the behaviour of the RND test case generator is worse than ex
on for this behaviour could be the presence of data dependencies in the program,
d in the theoretical approach in order to keep it simple.
100
his estimation, we propose a theoretical prediction about the behaviour of an automatic test
based on random testing.
we show a plot for a particular program with the expected theoretical behaviour together
mental data obtained using the average branch coverage of the 30 independent executions of
tor for that program. The features of this test program are shown in Table 9. The resulting
at our theoretical prediction and the experimental data are very similar. The theoretical
ore optimistic because it does not take into account data dependencies. At the ﬁrst steps
m, the experimental behaviour is better than the theoretical prediction, but in the region of
close to 90%), the behaviour of the RND test case generator is worse than expected. One
this behaviour could be the presence of data dependencies in the program, which is not
he theoretical approach in order to keep it simple.
0 5 10 15 20 25 30
0
10
20
30
40
50
60
70
80
90
100
Number of Test Cases
BranchCoverage
Random Generator
Theoretical Prediction Approximated
number of TCs to
cover the branch

Thanks for your attention !!!
Recent Research on
Search Based software Testing

Recent Research in Search Based Software Testing

More Related Content

What's hot (20)

Similar to Recent Research in Search Based Software Testing (20)

More from jfrchicanog (20)

Recently uploaded (20)

Recent Research in Search Based Software Testing