![CONCEPTS, TERMINOLOGY, AND NOTATION 5
domain may satisfy the same criterion. Examples of a test-case selection criterion
include T = {0, 1, 2, 3}, T = { i, j, k |i = j = k and k 1 and k 10}, and
“T is a set of inputs that cause 60% of the statements in the program to be exercised
at least once during the test.”
Let D be the input domain of a given program P, and let OK(P, d), where d ∈ D,
be a predicate that assumes the value of TRUE if an execution of program P with
input d terminates and produces a correct result, and FALSE otherwise. Predicate
OK(P, d) can be shortened to OK(d) if the omission of P would not lead to confusion.
After we test-execute the program with input d, how can we tell if OK(d) is true?
Two assumptions can be made in this regard. One is that the program specification
is available to the tester. OK(d) is true if the program produced a result that satisfies
the specification. Another is the existence of an oracle, a device that can be used to
determine if the test result is correct. The target-practice equipment used in testing
the software that controls a computerized gunsight is a good example of an oracle. A
“hit” indicates that the test is successful, and a “miss” indicates otherwise. The main
difference between a specification and an oracle is that a specification can be studied
to see how to arrive at a correct result, or the reason why the test failed. An oracle
gives no clue whatsoever.
Let T be a test set: a subset of D used to test-execute a program. A test using T is
said to be successful if the program terminates and produces a correct result for every
test case in T . A successful test is to be denoted by the predicate SUCCESSFUL(T ).
To be more precise,
SUCCESSFUL (T) ≡ (∀t)T (OK(t))
The reader should not confuse a successful test execution with a successful pro-
gram test using test set T . The test using T fails if there exists a test case in T
that causes the program to produce an incorrect result [i.e., ¬SUCCESSFUL(T ) ≡
¬(∀t)T (OK(t)) ≡ (∃t)T (¬OK(t))]. The test using T is successful if and only if the
program executes correctly for all test cases in T .
Observe that not every component in a program is involved in program execution.
For instance, if Program 1.1 is executed with input i = j = k = 0, the assign-
ment statement match = 1 will not be involved. Therefore, if this statement is
faulty, it will not be reflected in the test result. This is one reason that a program can
be fortuitously correct, and therefore it is insufficient to test a program with just one
test case.
According to the IEEE glossary, a part of a program that causes it to produce an
incorrect result is called a fault in that program. A fault causes the program to fail
(i.e., to produce incorrect results) for certain inputs. We refer to an aggregate of such
inputs as a failure set, usually a small subset of the input domain.
In debug testing, the goal is to find faults and remove them to improve the reliability
of the program. Therefore, the test set should be constructed such that it maximizes
the probability and minimizes the cost of finding at least one fault during the test.
To be more precise, let us assume that we wish to test the program with a set of n
test cases: T = {t1, t2, . . . , tn}. What is the reason for using multiple test cases? It](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-21-320.jpg)
![6 CONCEPTS, NOTATION, AND PRINCIPLES
is because for all practical programs, a single test case will not cause all program
components to become involved in the test execution, and if there is a fault in a
component, it will not be reflected in the test result unless that component is involved
in the test execution.
Of course, one may argue that a single test case would suffice if the entire program
were considered as a component. How we choose to define a component for test-case
selection purposes, however, will affect our effectiveness in revealing faults. If the
granularity of component is too coarse, part of a component may not be involved in
test execution, and therefore a fault contained therein may not be reflected in the test
result even if that component is involved in the test execution. On the other hand, if
the granularity of the component is too fine, the number of test cases required and the
effort required to find them will become excessive. For all known unit-testing meth-
ods, the granularities of the component range from a statement (finest) to an execution
path (coarsest) in the source code, with one exception that we discuss in Section 7.2,
where the components to be scrutinized are operands and expressions in a statement.
For debug testing, we would like to reveal at least one fault in the test. To be more
precise, we would like to maximize the probability that at least one test case causes
the program to produce an incorrect result. Formally, we would like to maximize
p(¬OK(t1) ∨ ¬OK(t2) ∨ · · · ∨ ¬OK(tn)) = p((∃t)T (¬OK(t)))
= p(¬(∀t)T (OK(t)))
= 1 − p((∀t)T (OK(t)))
The question now is: What information can be used to construct such a test set?
It is well known that programmers tend to forget writing code to make sure that the
program does not do division by zero, does not delete an element from an empty queue,
does not traverse a linked list without checking for the end node, and so on. It may also
be known that the author of the program has a tendency to commit certain types of error
or the program is designed to perform certain functions that are particularly difficult
to implement. Such information can be used to find test cases for which the program is
particularly error-prone [i.e., the probability p(¬OK(t1) ∨ ¬OK(t2) · · · ∨ ¬OK(tn))
is high]. The common term for making use of such information is error guessing.
The essence of that technique is described in Section 3.4.
Other than the nature or whereabouts of possible latent faults, which are unknown
in general, the most important information that we can derive from the program and
use to construct a test set is the degree of similarity to which two inputs are processed
by the program. It can be exploited to enhance the effectiveness of a test set. To see
why that is so, suppose that we choose some test case, t1, to test the program first, and
we wish to select another test case, t2, to test the program further. What relationship
must hold between t1 and t2 so that the joint fault discovery probability is arguably
enhanced?
Formally, what we wish to optimize is p(¬OK(t1) ∨ ¬OK(t2)), the probability of
fault discovery by test-executing the program with t1 and t2. It turns out that this prob-
ability can be expressed in terms of the conditional probability p(OK(t2) | OK(t1)):](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-22-320.jpg)
![TWO PRINCIPLES OF TEST-CASE SELECTION 9
and therefore become handy under situations when no existing method is applicable.
For example, when a new software system is procured, the user organization often
needs to test it to see if it is reliable enough to pay the vendor and release the system
for production runs. If an operational profile is available, the obvious choice is to
perform operational testing as described in Section 4.2. Otherwise, test-case selection
becomes a problem, especially if the system is fairly large. Source code is generally
not available to the user to make use of the methods presented in Chapter 2, and
detailed design documents or specifications are not available to use the methods
presented in Chapter 3. Even if they are available, a typical user organization simply
does not have the time, resources, and technical capability to deploy the methods. In
that event, the two principles can be utilized to select test cases. The components to be
exercised could be the constituent subsystems, which can be recognized by reading
the system user manual. Two inputs are weakly coupled computationally if they cause
different subsystems to be executed in different sequences. Expert users should be
able to apply the two principles readily to achieve a high probability of fault detection.
In short, if one finds it difficult to use any existing method, use the two principles
instead.
Next, in practical application, we would like to be able to compare the cost-
effectiveness of test sets. In the literature, the effectiveness of a test set is measured
by the probability of discovering at least one fault in the test (see, e.g., [FHLS98]).
It is intuitively clear that we can increase the fault-discovery capability of a test
set simply by adding elements to it. If we carry this idea to the extreme, the test
set eventually would contain all possible inputs. At that point, a successful test
constitutes a direct proof of correctness, and the probability of fault discovery is
100%. The cost of performing the test, however, will become unacceptably high.
Therefore, the number of elements in a test set must figure prominently when we
compare the cost-effectiveness of a test set. We define the cost-effectiveness of a test
set to be the probability of revealing a fault during the test, divided by the number of
elements in the test set.
A test set is said to be optimal if it is constructed in accordance with the first and
second principles for test-case selection and if its size (i.e., the number of elements
contained therein) is minimal. The concept of path testing (i.e., to choose the execution
path as the component to be exercised during the test) is of particular interest in this
connection because every feasible execution path defines a subdomain in the input
domain, and the set of all subdomains so defined constitutes a partition of the input
domain (in a set-theoretical sense; i.e., each and every element in the domain belongs
to one and only one subdomain). Therefore, a test set consisting of one element from
each such subdomain is a good one because it will not only cause every component to
be exercised at least once during the test, but its constituent elements will be loosely
coupled as well. Unfortunately, path testing is impractical in general because most
programs in the real world contain loop constructs, and a loop construct expands into
a prohibitively large number of execution paths. Nevertheless, the idea of doing path
testing remains of special interest because many known test-case selection methods
can be viewed as an approximation of path testing, as we demonstrate later.](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-25-320.jpg)
![CLASSIFICATION OF TEST-CASE SELECTION METHODS 11
the specification, there is a computational fault. To be more precise, we can restate
the definitions as follow.
1. Computational fault. The program has a computational fault if (∃i)(∃ j)((Pi ⊃
Cj ∧ Si (x) = f j (x)).
2. Domain fault. The program has a domain fault if ¬(∀i)(∃ j)(Pi ⊃ Cj ).
In words, if the program specification says that the input domain should consist
of m subdomains X1, X2, . . . , Xm, the program should partition the input domain
into n subdomains D1, D2, . . . , Dn, where n must be greater than or equal to m
if the partition prescribed by the specification is compact. The partition created by
the program must satisfy the condition that every Di = {d | d ∈ D and Pi (d)} be
contained completely in some X j , X1 ∪ X2 ∪ . . . ∪ Xm = X, and D1 ∪ D2 ∪ . . . ∪
Dn = D. Otherwise, there is a domain fault.
If there is a subdomain Di created by the program that is contained completely in
some X j prescribed by the specification, then for every input in Di , the value com-
puted by Si must be equal to that computed by f j . Otherwise, there is a computation
fault.
It is possible that a program contains both domain and computational faults at the
same time. Nevertheless, the same element in the input domain cannot be involved in
both kinds of fault. If the program is faulty at a particular input, it is either of domain
or computational type, but not both.
Previously published methods of program-fault classification include that of
Goodenough and Gerhart [GOGE77], Howden [HOWD76], and White and Cohen
[WHCO80]. All three include one more type of fault, called a subcase fault or
missing-path fault, which occurs when the programmer fails to create a subdomain
required by the specification [i.e., if ¬(∀i)(∃ j)(Ci ⊂ Pj )]. Since such a fault also
manifests as a computational fault, we chose, for simplicity, not to identify it as a
fault of separate type.
In Chapters 2 and 3 we discuss test-case selection methods that are designed
particularly for revealing the domain faults. In such methods, the components to be
exercised are the boundaries of subdomains embodied by the predicates found in the
source code or program specification.
1.4 CLASSIFICATION OF TEST-CASE SELECTION METHODS
It was observed previously that when a program is being test-executed, not all con-
stituent components would be involved. If a faulty component is not involved, the fault
will not be reflected in the test result. A necessary condition, therefore, for revealing
all faults in a test is to construct the test set in such a way that every contributing
component in the program is involved (exercised) in at least one test execution!
What is a component in the statement above? It can be defined in many differ-
ent ways. For example, it can be a statement in the source code, a branch in the](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-27-320.jpg)
![12 CONCEPTS, NOTATION, AND PRINCIPLES
control-flow diagram, or a predicate in the program specification. The use of differ-
ent components leads to the development of different test-case selection methods. As
shown in Chapters 2 and 3, many test-case selection methods have been developed.
If the component used is to be identified from the source code, the resulting test-
case selection method is said to be code-based. The most familiar examples of such
a method are the statement test, in which the program is to be tested to the extent that
every statement in its source code is exercised at least during the test, and the branch
test, in which the program is to be tested to the extent that every branch in its control-
flow diagram is traversed at least once during the test. There are several others that
cannot be explained as simply. All the methods are discussed in detail in Chapter 2.
If the component used is to be identified from the program specification, the
resulting test-case selection method is said to be specification-based. Examples of the
components identifiable from a program specification include predicates, boundaries
of input/output variables, and subfunctions. Chapter 3 is devoted to a discussion of
such methods.
It is possible that certain components can be identified from either the source
code or the program specification. The component defined in the subfunction testing
method discussed in Chapter 3 is an example.
Since a component can be also a subdomain consisting of those and only those
inputs that cause that component to be exercised during the test, a test-case selection
method that implicitly or explicitly requires execution of certain components in
the program during the test can also be characterized as being subdomain-based
[FHLS98]. The test methods and all of their derivatives, discussed in Chapters 2 and
3, are therefore all subdomain-based.
Are there any test-case selection methods that are not subdomain-based? There are
at least two: random testing [DUNT84, CHYU94, BOSC03] and operational testing
[MUSA93, FHLS98]. The first, although interesting, is not discussed in this book
because its value has yet to be widely recognized. The second is important in that it
is frequently used in practice. Because it is neither code- nor specification-based, we
choose to discuss it in Section 4.2.
1.5 THE COST OF PROGRAM TESTING
Software testing involves the following tasks: (1) test-case selection; (2) test execu-
tion; (3) test-result analysis; and if it is debug testing, (4) fault removal and regression
testing.
For test-case selection, the tester first has to study a program to identify all input
variables (parameters) involved. Then, depending on the test-case selection method
used, the tester has to analyze the source code or program specification to identify all
the components to be exercised during the test. The result is often stated as a condition
or predicate, called the test-case selection criterion, such that any set of inputs that
satisfies the criterion is an acceptable test set. The tester then constructs the test cases
by finding a set of assignments to the input variables (parameters) that satisfies the test-
case selection criterion. This component of the cost is determined by the complexity
of the analysis required and the number of test cases needed to satisfy the criterion.](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-28-320.jpg)
![18 CODE-BASED TEST-CASE SELECTION METHODS
α
β
γ δ
ε
η
α: cin a b e;
w = b - a;
β: / w e;
p = a + w / 3;
u = f(p);
q = b - w / 3;
v = f(q);
γ: / !(u v);
b = q;
δ: / u v;
a = p;
ε: w = b - a;
η: / !(w e);
max = (a + b) / 2;
cout max endl;
Figure 2.1 Program graph of Program 2.1. (Adapted from [HUAN75].)
The left component / C may be omitted if C is always true. The program graph of
Program 2.1 is shown in Figure 2.1. This program is designed to find the abscissa
within the interval (a, b) at which a function f(x) assumes the maximum value. The
basic strategy used is that given a continuous function that has a maximum in the
interval (a, b), we can find the desired point on the x-axis by first dividing the interval](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-34-320.jpg)
![20 CODE-BASED TEST-CASE SELECTION METHODS
sequence of operations to be performed, as explained elsewhere). This renders the
test set less efficient in the sense that it takes more elements to achieve the same goal.
In practice, rarely can we afford to do a full statement test, simply because it
requires too many test cases. Software development contracts that require statement
testing often reduce the coverage requirement to 60% or less to make the cost of testing
more acceptable. The tester is always under pressure to meet the test requirements by
using as few test cases as possible. A minimal set of test cases can be constructed by
finding a minimal set of execution paths that has the property that every statement is
on some path in that set. Then find a test case to traverse each path therein. The test
set so constructed will not only have a minimal number of elements but will also have
loosely coupled elements (because different elements cause the program to traverse
different execution paths), and therefore will be more compact and effective.
Another point to be made in this connection: Although the subdomains defined
by the statements in a program are overlapping in nature, the elements of a required
test set for statement testing need not be selected from overlapping subdomains, as
suggested or implied in some literature (see, e.g., [FHLS98]).
What is described above is a proactive approach to test-case selection, meaning that
we look actively for elements in the input domain that satisfy the selection criterion.
It requires nontrivial analytical skill to do the analysis, and could be time consuming
unless software tools are available to facilitate its applications. If the program to be
tested is relatively small, the test-case designer may be able to find the majority of
required test cases informally. Also, some test cases may be available from different
sources, such as program designers or potential end users and there is no reason not
to make good use of them. In that case, the only thing left to be done is to determine
to what extent coverage has been achieved, and if the coverage is not complete, what
additional test cases are needed to complete the task. The answer can readily be found
by instrumenting the program with software counters at a number of strategic points
in the program, as explained in Chapter 7. The software instruments not only provide
accurate coverage information but also definitive clues about additional test cases
needed to complete the task. This is important because in practice the tester has not
only the responsibility to achieve the test coverage prescribed by the stakeholders
but also to produce verifiable evidence that the coverage required has indeed been
achieved. The software instruments do provide such evidence.
There is another way to determine if a particular statement in a program has been
exercised during the test. Given a program, create a mutant of that program by altering
one of its executable statements in some way. The program and the mutant are then
test-executed with test cases. Unless the mutant is logically equivalent to the original
program, it should produce a test result different from that of the original program
when a test case causes the altered statement to be exercised during the test. Thus,
to do the statement test, we can create a mutant with respect to every executable
statement in the program, and then test the program together with all mutants until
the program differentiates itself from all the mutants in the sense that it produces at
least one test result different from that of every mutant.
Compared to the method of instrumenting the program with counters, it is advan-
tageous, in that it will ask for additional test cases to reveal a fault if the program](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-36-320.jpg)
![HOWDEN’S AND McCABE’S METHODS 23
2.4 HOWDEN’S AND McCABE’S METHODS
The branch test described above can be seen as a method for choosing a sample of
execution paths to be exercised during the test. There are at least two other methods
for selecting a sample of paths to be tested.
Boundary–Interior Method
The Howden method, called boundary–interior testing [HOWD75], is designed to
circumvent the problem presented by a loop construct. A boundary test of a loop
construct causes it to be entered but not iterated. An interior test causes a loop
construct to be entered and iterated at least once. A boundary–interior test combines
these two to reduce the number of paths that need to be traversed during the test. If
an execution path is expressed in a regular expression, the path to be exercised in the
boundary–interior test is described by replacing every occurrence of the form α∗
with
+ α, where is the identity under the operation of concatenation. The examples
listed in Table 2.1 should clarify this definition.
In practice, the paths prescribed by this method may be infeasible because certain
types of loop construct, such as a “for” loop in C++ and other programming lan-
guages, has to iterate a fixed number of times every time it is executed. Leave such a
loop construct intact because it will not be expanded into many paths.
Semantically speaking, the significance of having a loop iterated zero and one
times can be explained as follows. A loop construct is usually employed in the source
code of a program to implement something that has to be defined recursively: for
example, a set D of data whose membership can be defined recursively in the form
1. d0 ∈ D. (initialization clause)
2. If d ∈ D and P(d), then f (d) is also an element of D. (inductive clause)
3. Those and only those obtainable by a finite number of applications of clauses
1 and 2 are the elements of D. (extremal clause)
Here P is some predicate and f is some function. In this typical recursive definition
scheme, the initialization clause is used to prescribe what is known or given, and the
inductive clause is used to specify how a new element can be generated from those
given. (The extremal clause is understood and usually omitted.) Obviously, set D is
defined correctly if the initialization and inductive clauses are stated correctly. When
TABLE 2.1 Examples
Paths in the Program Paths to Be Traversed in the Test
ab∗
c ac + abc
a(b + c)∗
d ad + abd + acd
ab∗
cd∗
e ace + abce + acde + abcde](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-39-320.jpg)
![24 CODE-BASED TEST-CASE SELECTION METHODS
D is used in a program, it will be implemented as a loop construct of the form
d := d0;
while P(d) do begin S; d := f (d) end;
where S is a program segment designed to make use of the elements of the set.
Obviously, a test execution without entering the loop will exercise the initialization
clause, and a test execution that iterates the loop only once will exercise the inductive
clause. Therefore, we may say that boundary–interior testing is an abbreviated form
of path testing. Instead of exercising all possible execution paths generated by a
loop construct, it is designed to circumvent the problem by exercising only those
paths that involve initialization clauses and inductive clauses of inductive definitions
implemented in the source code.
McCabe’s Method
The McCabe method is based on his complexity measure [MCCA76], which requires
that at least a maximal set of linearly independent paths in the program be traversed
during the test.
A graph is said to be strongly connected if there is a path from any node in the
graph to any other node. It can be shown [BERG62] that in a strongly connected
graph G = E, N, where E is the set of edges and N is the set of nodes in G,
there can be as many as v(G) elements in a set of linearly independent paths, where
v(G) = |E| − |N| + 1
The number v(G), also known as McCabe’s cyclomatic number, is a measure of
program complexity [MCCA76].
Here we speak of a program graph with one entry and one exit. It has the property
that every node can be reached from the entry, and every node can reach the exit. In
general, it is not strongly connected but can be made so by adding an edge from the
exit to the entry. For example, we can make the program graph in Figure 2.1 strongly
connected by adding an edge (dashed line), as depicted in Figure 2.3. Since there
are seven edges and five nodes in this graph, v(G) = 7 − 5 + 1 = 3 in this example.
Note that for an ordinary program graph without that added edge, the formula for
computing v(G) should be
v(G) = |E| − |N| + 2
Next, for any path in G, we can associate it with a 1 × |N| vector, where the
element on the ith column is an integer equal to the number of times the ith edge
is used in forming the path. Thus, if we arrange the edges in the graph above in the
order αβδεγη, the vector representation of the path αβγεη is 1 1 0 1 1 0 1 and
that of βγεβγε is 0 2 0 2 2 0 0. We write αβγεη = 1 1 0 1 1 0 1 and
βγεβγε = 0 2 0 2 2 0 0.](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-40-320.jpg)
![26 CODE-BASED TEST-CASE SELECTION METHODS
in common structurally, which means that the operations to be performed along
these two paths will have little in common. Now, if t1 and t2 are the inputs that
cause these two paths to be traversed, it implies that the distance between t1 and t2
[i.e., ␦(t1, t2) = p(OK(t2) | OK(t1)) − p(OK(t2)) as defined in Section 1.1] would be
greater than if the two paths are not linearly independent. The test cases in a basis
set are therefore as loosely coupled as possible as far as can be determined by their
dependencies. Thus, to test a program with a basis set is to exercise a finite subset of
execution paths in the program that yield a maximal probability of fault discovery.
To construct such a test set, we need to construct a set of linearly independent paths
first. We can start by putting any path in the set. We then add to this set another path
that is linearly independent of the existing paths in the set. According to graph theory,
we will have to terminate this process when the number of paths in the set is equal to
McCabe’s cyclomatic number because we will not be able to find an additional path
that is linearly independent of the existing paths. We then find an input for each path
in the set. The result is the test set desired.
Note that although v(G) is fixed by the graph structure, the membership of a basis
set is not unique. For example, {αβδεη, αβδεβγεη, αη} is also a basis set in G.
That is, in McCabe’s method, the set of paths to be traversed is not unique. The set of
paths to be traversed can be {αβδεη, αβδεβγεη, αη} instead of {αβδεη, αβγεη,
αη}, mentioned previously.
It is interesting to observe that v(G) has the following properties:
1. v(G) ≥ 1.
2. v(G) is the maximum number of linearly independent paths in G, and it is the
number of execution paths to be traversed during the test.
3. Inserting or deleting a node with an outdegree of 1 does not affect v(G).
4. G has only one path if v(G) = 1.
5. Inserting a new edge in G increases v(G) by 1.
6. v(G) depends only on the branching structure of the program represented
by G.
2.5 DATA-FLOW TESTING
Data-flow testing [LAKO83, GUGU02] is also an approximation of path testing. The
component that will be exercised during the test is a segment of feasible execution
path that starts from the point where a variable is defined and ends at the point where
that definition is used. It is important to exercise such segments of execution paths
because each is designed to compute some subfunction of the function implemented
by the program. If such a segment occurs in more than one execution path, only one
of them needs to be exercised during the test.
For the purpose of this discussion, the concept of data-flow analysis can be ex-
plained simply as follows. When a program is being executed, its component, such as
a statement, will act on the data or variables involved in three different ways: define,](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-42-320.jpg)
![36 CODE-BASED TEST-CASE SELECTION METHODS
all-path
all-du-path
all-use
all-c/some-p
all-c-use
all-p/some-c
all-p-use
branch
statement
all-def
Figure 2.5 Coverage relation among test-case selection criteria.
of the path segments in the rightmost column of Table 2.15. Hence, the test cases
required are x, 2, x, 3, and x, 5.
It is interesting to observe that the data-flow testing criteria discussed above are
related in some way. In fact, they are related by the coverage relation mentioned
elsewhere. One test-case selection criterion is said to cover another if satisfaction of
that criterion implies that of the other. Figure 2.5 shows the coverage relationship
among test-case selection criteria based on data flow and the structure of the source
code [FRWE88].
2.6 DOMAIN-STRATEGY TESTING
Recall that in the branch testing the components to be exercised during the test are the
branch predicates found in the source code, which define the line segments that form
the borders of subdomains created by the program. We select a test case arbitrarily
from each side of the border to see if the functions defined on the two adjacent
subdomains are computed correctly.
In this section we show that for each branch predicate, a trio of test cases, lo-
cated on or near the borderline it defines, can be chosen to test if the borderline is
implemented correctly by the program. The method, as it is outlined here, will work
only if the predicate is linear, representing a straight line in n-dimensional space.](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-52-320.jpg)
![DOMAIN-STRATEGY TESTING 37
This method, known as domain-strategy testing [WHCO80], is designed to detect
domain faults. It is based on geometrical analysis of the domain boundary defined
by the source code, exploiting the fact that points on or near the border are most
sensitive to domain faults. The method works only if the program has the following
properties:
1. The program contains only simple linear predicates of the form a1v1 + a2v2 +
· · · + akvk ROP C, where the vi’s are variables, the ai’s and C are constants,
and ROP is a relational operator.
2. The path predicate of every path in the program is composed of a conjunction
of such simple linear predicates.
3. Coincidental (fortuitous) correctness of the program will not occur for any test
case.
4. The path corresponding to each adjacent domain computes a different subfunc-
tion.
5. Functions defined in two adjacent subdomains yield different values for the
same test point near the border.
6. Any border defined by the program is linear, and if it is incorrect, the correct
border is also linear.
7. The input space is continuous rather than discrete.
The essence of domain-boundary geometrical analysis to be performed in test-case
selection can be stated as follows. For simplicity, we use examples in two-dimensional
space to illustrate the idea involved.
r Each border is a line segment in a k-dimensional space, which can be open or
closed, depending on the type of relational operator used in the predicate.
r A closed border segment of a domain is actually part of that domain and is
formed by a predicate consisting of a ≥, =, or ≤ relational operator.
r An open border segment of a domain forms part of the domain boundary but
does not constitute part of that domain and is formed by using a , , or =
relational operator.
For example, let domain A in a two-dimensional space be defined by the predicate
x ≥ 1, and let the domain immediately adjacent to it on the left be domain B. Domain
B is defined by the predicate x 1. The straight line described by x = 1 is the
border of these two domains. The line x = 1 is a closed border and a part of domain
A. To domain B, however, it is an open border and is not part of that domain. The
test points (cases) selected will be of two types, defined by their relative position
with respect to the given border. An on test point lies on the given border, while
an off test point is a small distance ε from, and lies on the open side of, the given
border.](https://image.slidesharecdn.com/2203934-250603022051-73e4e4d6/85/Software-Error-Detection-Through-Testing-And-Analysis-1st-Edition-J-C-Huang-53-320.jpg)
Software Error Detection Through Testing And Analysis 1st Edition J C Huang
- 1. Software Error Detection Through Testing And Analysis 1st Edition J C Huang download https://ebookbell.com/product/software-error-detection-through- testing-and-analysis-1st-edition-j-c-huang-4407868 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Humancentered And Errorresilient Systems Development Ifip Wg 132135 Joint Working Conference 6th International Conference On Humancentered Software Engineering Hcse 2016 And 8th International Conference On Human Error Safety And System Develop 1st Edition Cristian Bogdan https://ebookbell.com/product/humancentered-and-errorresilient- systems-development-ifip-wg-132135-joint-working-conference-6th- international-conference-on-humancentered-software-engineering- hcse-2016-and-8th-international-conference-on-human-error-safety-and- system-develop-1st-edition-cristian-bogdan-5607652 Software Engineering Barry W Boehms Lifetime Contributions To Software Development Management And Research Barry W Boehm https://ebookbell.com/product/software-engineering-barry-w-boehms- lifetime-contributions-to-software-development-management-and- research-barry-w-boehm-44886620 Software Architecture 16th European Conference Ecsa 2022 Prague Czech Republic September 1923 2022 Proceedings Ilias Gerostathopoulos https://ebookbell.com/product/software-architecture-16th-european- conference-ecsa-2022-prague-czech-republic- september-1923-2022-proceedings-ilias-gerostathopoulos-46090988 Software Engineering Research Management And Applications Roger Lee https://ebookbell.com/product/software-engineering-research- management-and-applications-roger-lee-46317156
- 3. Softwaredefined Radio For Engineers Illustrated Travis F Collinspu Di https://ebookbell.com/product/softwaredefined-radio-for-engineers- illustrated-travis-f-collinspu-di-46706858 Software Architecture For Web Developers An Introductory Guide For Developers Striving To Take The First Steps Toward Software Architecture Or Just Looking To Grow As Professionals 1st Edition Mihaela Roxana Ghidersa https://ebookbell.com/product/software-architecture-for-web- developers-an-introductory-guide-for-developers-striving-to-take-the- first-steps-toward-software-architecture-or-just-looking-to-grow-as- professionals-1st-edition-mihaela-roxana-ghidersa-46801836 Software Design By Example Greg Wilson https://ebookbell.com/product/software-design-by-example-greg- wilson-47254836 Software Design Patterns The Ultimate Guide 1st Edition Sufyan Bin Uzay https://ebookbell.com/product/software-design-patterns-the-ultimate- guide-1st-edition-sufyan-bin-uzay-47260458 Softwaredefined Networks A Systems Approach Larry Peterson https://ebookbell.com/product/softwaredefined-networks-a-systems- approach-larry-peterson-47322378
- 7. SOFTWARE ERROR DETECTION THROUGH TESTING AND ANALYSIS
- 9. SOFTWARE ERROR DETECTION THROUGH TESTING AND ANALYSIS J. C. Huang University of Houston A JOHN WILEY & SONS, INC., PUBLICATION
- 10. Copyright C 2009 by John Wiley Sons, Inc. All rights reserved. Published by John Wiley Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Cataloging-in-Publication Data: Huang, J. C., 1935– Software error detection through testing and analysis / J. C. Huang. p. cm. Includes bibliographical references and index. ISBN 978-0-470-40444-7 (cloth) 1. Computer software–Testing. 2. Computer software–Reliability. 3. Debugging in computer science. I. Title. QA76.76.T48H72 2009 005.14–dc22 2008045493 Printed in the United States of America 10 9 8 7 6 5 4 3 2 1
- 11. To my parents
- 13. CONTENTS Preface ix 1 Concepts, Notation, and Principles 1 1.1 Concepts, Terminology, and Notation 4 1.2 Two Principles of Test-Case Selection 8 1.3 Classification of Faults 10 1.4 Classification of Test-Case Selection Methods 11 1.5 The Cost of Program Testing 12 2 Code-Based Test-Case Selection Methods 14 2.1 Path Testing 16 2.2 Statement Testing 17 2.3 Branch Testing 21 2.4 Howden’s and McCabe’s Methods 23 2.5 Data-Flow Testing 26 2.6 Domain-Strategy Testing 36 2.7 Program Mutation and Fault Seeding 39 2.8 Discussion 46 Exercises 51 3 Specification-Based Test-Case Selection Methods 53 3.1 Subfunction Testing 55 3.2 Predicate Testing 68 3.3 Boundary-Value Analysis 70 3.4 Error Guessing 71 3.5 Discussion 72 Exercises 73 4 Software Testing Roundup 76 4.1 Ideal Test Sets 77 4.2 Operational Testing 80 4.3 Integration Testing 82 4.4 Testing Object-Oriented Programs 84 4.5 Regression Testing 88 vii
- 14. viii CONTENTS 4.6 Criteria for Stopping a Test 88 4.7 Choosing a Test-Case Selection Criterion 90 Exercises 93 5 Analysis of Symbolic Traces 94 5.1 Symbolic Trace and Program Graph 94 5.2 The Concept of a State Constraint 96 5.3 Rules for Moving and Simplifying Constraints 99 5.4 Rules for Moving and Simplifying Statements 110 5.5 Discussion 114 5.6 Supporting Software Tool 126 Exercises 131 6 Static Analysis 132 6.1 Data-Flow Anomaly Detection 134 6.2 Symbolic Evaluation (Execution) 137 6.3 Program Slicing 141 6.4 Code Inspection 146 6.5 Proving Programs Correct 152 Exercises 161 7 Program Instrumentation 163 7.1 Test-Coverage Measurement 164 7.2 Test-Case Effectiveness Assessment 165 7.3 Instrumenting Programs for Assertion Checking 166 7.4 Instrumenting Programs for Data-Flow-Anomaly Detection 169 7.5 Instrumenting Programs for Trace-Subprogram Generation 181 Exercises 192 Appendix A: Logico-Mathematical Background 194 Appendix B: Glossary 213 Appendix C: Questions for Self-Assessment 220 Bibliography 237 Index 253
- 15. PREFACE The ability to detect latent errors in a program is essential to improving program reliability. This book provides an in-depth review and discussion of the methods of software error detection using three different techniqus: testing, static analysis, and program instrumentation. In the discussion of each method, I describe the basic idea of the method, how it works, its strengths and weaknesses, and how it compares to related methods. I have writtent this book to serve both as a textbook for students and as a technical handbook for practitioners leading quality assurance efforts. If used as a text, the book is suitable for a one-semester graduate-level course on software testing and analysis or software quality assurance, or as a supplementary text for an advanced graduate course on software engineering. Some familiarity with the process of software quality assurance is assumed. This book provides no recipe for testing and no discussion of how a quality assurance process is to be set up and managed. In the first part of the book, I discuss test-case selection, which is the crux of problems in debug testing. Chapter 1 introduces the terms and notational conventions used in the book and establishes two principles which together provide a unified conceptual framework for the existing methods of test-case selection. These principles can also be used to guide the selection of test cases when no existing method is deemed applicable. In Chapters 2 and 3 I describe existing methods of test-case selection in two categories: Test cases can be selected based on the information extracted form the source code of the program as described in Chapter 2 or from the program specifications, as described in Chapter 3. In Chapter 4 I tidy up a few loose ends and suggest how to choose a method of test-case selection. I then proceed to discuss the techniques of static analysis and program instru- mentation in turn. Chapter 5 covers how the symbolic trace of an execution path can be analyzed to extract additional information about a test execution. In Chapter 6 I address static analysis, in which source code is examined systematically, manually or automatically, to find possible symptoms of programming errors. Finally, Chapter 7 covers program instrumentation, in which software instruments (i.e., additional program statements) are inserted into a program to extract information that may be used to detect errors or to facilitate the testing process. Because precision is necessary, I have made use throughout the book of concepts and notations developed in symbolic logic and mathematics. A review is included as Appendix A for those who may not be conversant with the subject. I note that many of the software error detection methods discussed in this book are not in common use. The reason for that is mainly economic. With few exceptions, ix
- 16. x PREFACE these methods cannot be put into practice without proper tool support. The cost of the tools required for complete automation is so high that it often rivals that of a major programming language compiler. Software vendors with products on the mass market can afford to build these tools, but there is no incentive for them to do so because under current law, vendors are not legally liable for the errors in their products. As a result, vendors, in effect, delegate the task of error detection to their customers, who provide that service free of charge (although vendors may incur costs in the form of customer dissatisfaction). Critical software systems being built for the military and industry would benefit from the use of these methods, but the high cost of necessary supporting tools often render them impractical, unless and until the cost of supporting tools somehow becomes justifiable. Neverthless, I believe that knowledge about these existing methods is useful and important to those who specialize in software quality assurance. I would like to take opportunity to thank anonymous reviewers for their comments; William E. Howden for his inspiration; Raymond T. Yeh, José Muñoz, and Hal Watt for giving me professional opportunities to gain practical experience in this field; and John L. Bear and Marc Garbey for giving me the time needed to complete the first draft of this book. Finally, my heartfelt thanks go to my daughter, Joyce, for her active and affectionate interest in my writing, and to my wife, Shih-wen, for her support and for allowing me to neglect her while getting this work done. J. C. Huang Houston
- 17. 1 Concepts, Notation, and Principles Given a computer program, how can we determine whether or not it will do exactly what it is intended to do? This question is not only intellectually challenging, but also of primary importance in practice. An ideal solution to this problem would be to develop certain techniques that can be used to construct a formal proof (or disproof) of the correctness of a program systematically. There has been considerable effort to develop such techniques, and many different techniques for proving program correctness have been reported. However, none of them has been developed to the point where it can be used readily in practice. There are several technical hurdles that prevent formal proof of correctness from becoming practical; chief among them is the need for a mechanical theorem prover. The basic approach taken in the development of these techniques is to translate the problem of proving program correctness into that of proving a certain statement to be a theorem (i.e., always true) in a formal system. The difficulty is that all known automatic theorem-proving techniques require an inordinate amount of computation to construct a proof. Furthermore, theorem proving is a computationally unsolvable problem. Therefore, like any other program written to solve such a problem, a theorem prover may halt if a solution is found. It may also continue to run without giving any clue as to whether it will take one more moment to find the solution, or whether it will take forever. The lack of a definitive upper bound of time required to complete a job severely limits its usefulness in practice. Until there is a major breakthrough in the field of mechanical theorem proving, which is not foreseen by the experts any time soon, verification of program correctness through formal proof will remain impractical. The technique is too costly to deploy, and the size of programs to which it is applicable is too small (relative to that of programs in common use). At present, a practical and more intuitive solution would be to test-execute the program with a number of test cases (input data) to see if it will do what it is intended to do. How do we go about testing a computer program for correctness? Perhaps the most direct and intuitive answer to this question is to perform an exhaustive test: that is, to test-execute the program for all possible input data (for which the program is expected to work correctly). If the program produces a correct result for every possible input, it obviously constitutes a direct proof that the program is correct. Unfortunately, it is in general impractical to do the exhaustive test for any nontrivial program simply because the number of possible inputs is prohibitively large. Software Error Detection through Testing and Analysis, By J. C. Huang Copyright C 2009 John Wiley Sons, Inc. 1
- 18. 2 CONCEPTS, NOTATION, AND PRINCIPLES To illustrate, consider the following C++ program. Program 1.1 main () { int i, j, k, match; cin i j k; cout i j k; if (i = 0 || j = 0 || k = 0 || i+j = k || j+k = i || k+i = j) match = 4; else if !(i == j || j == k || k == i) match = 3; else if (i != j || j != k || k != i) match = 2; else match = 1; cout match endl; } If, for an assignment of values to the input variables i, j, and k, the output variable match will assume a correct value upon execution of the program, we can assert that the program is correct for this particular test case. And if we can test the program for all possible assignments to i, j, and k, we will be able to determine its correctness. The difficulty here is that even for a small program like this, with only three input variables, the number of possible assignments to the values of those variables is prohibitively large. To see why this is so, recall that an ordinary integer variable in C++ can assume a value in the range −32,768 to +32,767 (i.e., 216 different values). Hence, there are 216 × 216 × 216 = 248 ≈ 256 × 1012 possible assignments to the input triple (i, j, k). Now suppose that this program can be test-executed at the rate of one test per microsecond on average, and suppose further that we do testing 24 hours a day, 7 days a week. It will take more than eight years for us to complete an exhaustive test for this program. Spending eight years to test a program like this is an unacceptably high expenditure under any circumstance! This example clearly indicates that an exhaustive test (i.e., a test using all possible input data) is impractical. It may be technically doable for some small programs, but it would never be economically justifiable for a real-world program. That being the case, we will have to settle for testing a program with a manageably small subset of its input domain. Given a program, then, how do we construct such a subset; that is, how do we select test cases? The answer would be different depending on why we are doing the test. For software developers, the primary reason for doing the test is to find errors so that they can be removed to improve the reliability of the program. In that case we say that the tester is doing debug testing. Since the main goal of debug testing is to find programming errors, or faults in the Institute of Electrical and Electronics
- 19. CONCEPTS, NOTATION, AND PRINCIPLES 3 Engineers (IEEE) terminology, the desired test cases would be those that have a high probability of revealing faults. Other than software developers, expert users of a software system may also have the need to do testing. For a user, the main purpose is to assess the reliability so that the responsible party can decide, among other things, whether or not to accept the software system and pay the vendor, or whether or not there is enough confidence in the correctness of the software system to start using it for a production run. In that case the test cases have to be selected based on what is available to the user, which often does not include the source code or program specification. Test-case selection therefore has to be done based on something else. Information available to the user for test-case selection includes the probability distribution of inputs being used in production runs (known as the operational profile) and the identity of inputs that may incur a high cost or result in a catastrophe if the program fails. Because it provides an important alternative to debug testing, possible use of an operational profile in test-case selection is explained further in Section 4.2. We discuss debug testing in Chapters 2 and 3. Chapter 4 is devoted to other aspects of testing that deserve our attention. Other than testing as discussed in Chapters 2 and 3, software faults can also be detected by means of analysis, as discussed in Chapters 5 through 7. When we test-execute a program with an input, the test result will be either correct or incorrect. If it is incorrect, we can unequivocally conclude that there is a fault in the program. If the result is correct, however, all that we can say with certainty is that the program will execute correctly for that particular input, which is not especially significant in that the program has so many possible inputs. The significance of a correct test result can be enhanced by analyzing the execution path traversed to determine the condition under which that path will be traversed and the exact nature of computation performed in the process. This is discussed in Chapter 5. We can also detect faults in a program by examining the source code systematically as discussed in Chapter 6. The analysis methods described therein are said to be static, in that no execution of the program is involved. Analysis can also be done dynamically, while the program is being executed, to facilitate detection of faults that become more obvious during execution time. In Chapter 7 we show how dynamic analysis can be done through the use of software instruments. For the benefit of those who are not theoretically oriented, some helpful logico- mathematical background material is presented in Appendix A. Like many others used in software engineering, many technical terms used in this book have more than one possible interpretation. To avoid possible misunderstanding, a glossary is included as Appendix B. For those who are serious about the material presented in this book, you may wish to work on the self-assessment questions posed in Appendix C. There are many known test-case selection methods. Understanding and compar- ison of those methods can be facilitated significantly by presenting all methods in a unified conceptual framework so that each method can be viewed as a particular instantiation of a generalized method. We develop such a conceptual framework in the remainder of the chapter.
- 20. 4 CONCEPTS, NOTATION, AND PRINCIPLES 1.1 CONCEPTS, TERMINOLOGY, AND NOTATION The input domain of a program is the set of all possible inputs for which the program is expected to work correctly. It is constrained by the hardware on which the program is to be executed, the operating system that controls the hardware, the programming environment in which the program is developed, and the intent of the creator of the program. If none of these constraints are given, the default will be assumed. For example, consider Program 1.1. The only constraint that we can derive from what is given is the fact that all variables in the program are of the type “short integer” in C++. The prevailing standard is to use 16 bits to represent such data in 2’s-complement notation, resulting in the permissible range −32,768 to 32,767 in decimal. The input domain therefore consists of all triples of 16-bit integers; that is, D = { x, y, z |x, y, and z are 16-bit integers} Input (data) are elements of the input domain, and a test case is an input used to perform a test execution. Thus, every test case is an input, but an input is not necessarily a test case in a particular test. The set of all test cases used in testing is called a test set. For example, 3, 5, 2 is a possible input (or test case) in Program 1.1, and in a particular test we might choose {1, 2, 3, 4, 5, 6, 0, 0, 5, 5, 0, 1, 3, 3, 3} as the test set. This notational convention for representing program inputs remains valid even if the program accepts an input repeatedly when run in an interactive mode (i.e., sequence of inputs instead of a single input). All we need to do is to say that the input domain is a product set instead of a simple set. For example, consider a program that reads the value of input variable x, which can assume a value from set X. If the function performed by the program is not influenced by the previous value of x, we can simply say that X is the input domain of the program. If the function performed by the program is dependent on the immediately preceding input, we can make the product set X · X = { x1, x2 |x1 ∈ X and x2 ∈ X} the input domain. In general, if the function performed by the program is dependent on n immediately preceding inputs, we can make the product set Xn+1 = { x1, x2, . . . , xn, xn+1 |xi ∈ X for all 1 ≤ i ≤ n + 1} the input domain. This is the property of a program with memory, often resulting from implementing the program with a finite-state machine model. The value of n is usually small and is related to the number of states in the finite-state machine. Do not confuse a program with memory with an interactive program (i.e., a program that has to be executed interactively). Readers should have no difficulty convincing themselves that an interactive program could be memoryless and that a program with memory does not have to be executed interactively. We shall now proceed to define some terms in program testing that might, at times, have a different meaning for different people. The composition of a test set is usually prescribed using a test-case selection criterion. Given such a criterion, any subset of the input domain that satisfies the criterion is a candidate. We say “any subset” because more than one subset in the input
- 21. CONCEPTS, TERMINOLOGY, AND NOTATION 5 domain may satisfy the same criterion. Examples of a test-case selection criterion include T = {0, 1, 2, 3}, T = { i, j, k |i = j = k and k 1 and k 10}, and “T is a set of inputs that cause 60% of the statements in the program to be exercised at least once during the test.” Let D be the input domain of a given program P, and let OK(P, d), where d ∈ D, be a predicate that assumes the value of TRUE if an execution of program P with input d terminates and produces a correct result, and FALSE otherwise. Predicate OK(P, d) can be shortened to OK(d) if the omission of P would not lead to confusion. After we test-execute the program with input d, how can we tell if OK(d) is true? Two assumptions can be made in this regard. One is that the program specification is available to the tester. OK(d) is true if the program produced a result that satisfies the specification. Another is the existence of an oracle, a device that can be used to determine if the test result is correct. The target-practice equipment used in testing the software that controls a computerized gunsight is a good example of an oracle. A “hit” indicates that the test is successful, and a “miss” indicates otherwise. The main difference between a specification and an oracle is that a specification can be studied to see how to arrive at a correct result, or the reason why the test failed. An oracle gives no clue whatsoever. Let T be a test set: a subset of D used to test-execute a program. A test using T is said to be successful if the program terminates and produces a correct result for every test case in T . A successful test is to be denoted by the predicate SUCCESSFUL(T ). To be more precise, SUCCESSFUL (T) ≡ (∀t)T (OK(t)) The reader should not confuse a successful test execution with a successful pro- gram test using test set T . The test using T fails if there exists a test case in T that causes the program to produce an incorrect result [i.e., ¬SUCCESSFUL(T ) ≡ ¬(∀t)T (OK(t)) ≡ (∃t)T (¬OK(t))]. The test using T is successful if and only if the program executes correctly for all test cases in T . Observe that not every component in a program is involved in program execution. For instance, if Program 1.1 is executed with input i = j = k = 0, the assign- ment statement match = 1 will not be involved. Therefore, if this statement is faulty, it will not be reflected in the test result. This is one reason that a program can be fortuitously correct, and therefore it is insufficient to test a program with just one test case. According to the IEEE glossary, a part of a program that causes it to produce an incorrect result is called a fault in that program. A fault causes the program to fail (i.e., to produce incorrect results) for certain inputs. We refer to an aggregate of such inputs as a failure set, usually a small subset of the input domain. In debug testing, the goal is to find faults and remove them to improve the reliability of the program. Therefore, the test set should be constructed such that it maximizes the probability and minimizes the cost of finding at least one fault during the test. To be more precise, let us assume that we wish to test the program with a set of n test cases: T = {t1, t2, . . . , tn}. What is the reason for using multiple test cases? It
- 22. 6 CONCEPTS, NOTATION, AND PRINCIPLES is because for all practical programs, a single test case will not cause all program components to become involved in the test execution, and if there is a fault in a component, it will not be reflected in the test result unless that component is involved in the test execution. Of course, one may argue that a single test case would suffice if the entire program were considered as a component. How we choose to define a component for test-case selection purposes, however, will affect our effectiveness in revealing faults. If the granularity of component is too coarse, part of a component may not be involved in test execution, and therefore a fault contained therein may not be reflected in the test result even if that component is involved in the test execution. On the other hand, if the granularity of the component is too fine, the number of test cases required and the effort required to find them will become excessive. For all known unit-testing meth- ods, the granularities of the component range from a statement (finest) to an execution path (coarsest) in the source code, with one exception that we discuss in Section 7.2, where the components to be scrutinized are operands and expressions in a statement. For debug testing, we would like to reveal at least one fault in the test. To be more precise, we would like to maximize the probability that at least one test case causes the program to produce an incorrect result. Formally, we would like to maximize p(¬OK(t1) ∨ ¬OK(t2) ∨ · · · ∨ ¬OK(tn)) = p((∃t)T (¬OK(t))) = p(¬(∀t)T (OK(t))) = 1 − p((∀t)T (OK(t))) The question now is: What information can be used to construct such a test set? It is well known that programmers tend to forget writing code to make sure that the program does not do division by zero, does not delete an element from an empty queue, does not traverse a linked list without checking for the end node, and so on. It may also be known that the author of the program has a tendency to commit certain types of error or the program is designed to perform certain functions that are particularly difficult to implement. Such information can be used to find test cases for which the program is particularly error-prone [i.e., the probability p(¬OK(t1) ∨ ¬OK(t2) · · · ∨ ¬OK(tn)) is high]. The common term for making use of such information is error guessing. The essence of that technique is described in Section 3.4. Other than the nature or whereabouts of possible latent faults, which are unknown in general, the most important information that we can derive from the program and use to construct a test set is the degree of similarity to which two inputs are processed by the program. It can be exploited to enhance the effectiveness of a test set. To see why that is so, suppose that we choose some test case, t1, to test the program first, and we wish to select another test case, t2, to test the program further. What relationship must hold between t1 and t2 so that the joint fault discovery probability is arguably enhanced? Formally, what we wish to optimize is p(¬OK(t1) ∨ ¬OK(t2)), the probability of fault discovery by test-executing the program with t1 and t2. It turns out that this prob- ability can be expressed in terms of the conditional probability p(OK(t2) | OK(t1)):
- 23. CONCEPTS, TERMINOLOGY, AND NOTATION 7 the probability that the program will execute correctly with input t2 given the fact that the program executed correctly with t1. To be exact, p(¬OK(t1) ∨ ¬OK(t2)) = p(¬(OK(t1) ∧ OK(t2))) = p(¬(OK(t2) ∧ OK(t1))) = 1 − p(OK(t2) ∧ OK(t1)) = 1 − p(OK(t2) | OK(t1))p(OK(t1)) This equation shows that if we can choose t2 to make the conditional probability p(OK(t2) | OK(t1)) smaller, we will be able to increase p(¬OK(t1) ∨ ¬OK(t2)), the probability of fault discovery. The value of p(OK(t2) | OK(t1)) depends on, among other factors, the degree of similarity of operations performed in execution. If the sequences of operations performed in test-executing the program using t1 and t2 are completely unrelated, it should be intuitively clear that p(OK(t2) | OK(t1)) = p(OK(t2)), that is, the fact that the program test-executed correctly with t1 does not influence the probability that the program will test-execute correctly with test case t2. Therefore, p(OK(t2) ∧ OK(t1)) = p(OK(t2))p(OK(t1)). On the other hand, if the sequences of operations performed are similar, then p(OK(t2) | OK(t1)) p(OK(t2)), that is, the probability that the program will execute correctly will become greater given that the program test-executes correctly with input t1. The magnitude of the difference in these two probabilities, denoted by ␦(t1, t2) = p(OK(t2) | OK(t1)) − p(OK(t2)) depends on, among other factors, the degree of commonality or similarity between the two sequences of operations performed by the program in response to inputs t1 and t2. For convenience we shall refer to ␦(t1, t2) henceforth as the (computational) cou- pling coefficient between test cases t1 and t2, and simply write ␦ if the identities of t1 and t2 are understood. The very basic problem of test-case selection can now be stated in terms of this coefficient simply as follows. Given a test case, find another that is as loosely coupled to the first as possible! Obviously, the value of this coefficient is in the range 0 ≤ ␦(t1, t2) ≤ 1 − p(OK(t2)), because if OK(t1) implies OK(t2), then p(OK(t2) | OK(t1)) = 1, and if the events OK(t1) and OK(t2) are completely independent, then p(OK(t2) | OK(t1)) = p(OK(t2)). The greater the value of ␦(t1, t2), the tighter the two inputs t1 and t2 are coupled, and therefore the lower the joint probability of fault discovery (through the use of test cases t1 and t2). Asymptotically, ␦(t1, t2) becomes zero when the events of successful tests with t1 and t2 are absolutely and completely independent, and ␦(t1, t2) becomes 1 − p(OK(t2)) = p(¬OK(t2)) when a successful test with t1 surely entails a successful test with t2.
- 24. 8 CONCEPTS, NOTATION, AND PRINCIPLES Perhaps a more direct way to explain the significance of the coupling coefficient ␦(t1, t2) is that p(¬OK(t1) ∨ ¬OK(t2)) = 1 − p(OK(t2) | OK(t1))p(OK(t1)) = 1 − (p(OK(t2) | OK(t1)) − p(OK(t2)) + p(OK(t2)))p(OK(t1)) = 1 − (␦(t1, t2) + p(OK(t2)))p(OK(t1)) = 1 − ␦(t1, t2)p(OK(t1)) − p(OK(t2))p(OK(t1)) The values of p(OK(t1)) and p(OK(t2)) are intrinsic to the program to be tested; their values are generally unknown and beyond the control of the tester. The tester, however, can select t1 and t2 with a reduced value of the coupling coefficient ␦(t1, t2), thereby increasing the fault-discovery probability p(¬OK(t1) ∨ ¬OK(t2)). How can we reduce the coupling coefficient ␦(t1, t2)? There are a number of ways to achieve that, as discussed in this book. One obvious way is to select t1 and t2 from different input subdomains, as explained in more detail later. 1.2 TWO PRINCIPLES OF TEST-CASE SELECTION Now we are in a position to state two principles. The first principle of test-case selection is that in choosing a new element for a test set being constructed, preference should be given to those candidates that are computationally as loosely coupled as possible to all the existing elements in the set. A fundamental problem then is: Given a program, how do we construct a test set according to this principle? An obvious answer to this question is to select test cases such that the program will perform a distinctly different sequence of operations for every element in the set. If the test cases are to be selected based on the source code, the most obvious candidates for the new element are those that will cause a different execution path to be traversed. Since almost all practical programs have a large number of possible execution paths, the next question is when to stop adding test cases to the test set. Since the purpose of using multiple test cases is to cause every component, however that is defined, to be exercised at least once during the test, the obvious answer is to stop when there are enough elements in the test set to cause every component to be exercised at least once during the test. Thus, the second principle of test-case selection is to include in the test set as many test cases as needed to cause every contributing component to be exercised at least once during the test. (Remark: Contributing here refers to the component that will make some difference to the computation performed by the program. For brevity henceforth, we omit this word whenever the term component is used in this context.) Note that the first principle guides us as to what to choose, and the second, as to when to stop choosing. These two principles are easy to understand and easy to apply,
- 25. TWO PRINCIPLES OF TEST-CASE SELECTION 9 and therefore become handy under situations when no existing method is applicable. For example, when a new software system is procured, the user organization often needs to test it to see if it is reliable enough to pay the vendor and release the system for production runs. If an operational profile is available, the obvious choice is to perform operational testing as described in Section 4.2. Otherwise, test-case selection becomes a problem, especially if the system is fairly large. Source code is generally not available to the user to make use of the methods presented in Chapter 2, and detailed design documents or specifications are not available to use the methods presented in Chapter 3. Even if they are available, a typical user organization simply does not have the time, resources, and technical capability to deploy the methods. In that event, the two principles can be utilized to select test cases. The components to be exercised could be the constituent subsystems, which can be recognized by reading the system user manual. Two inputs are weakly coupled computationally if they cause different subsystems to be executed in different sequences. Expert users should be able to apply the two principles readily to achieve a high probability of fault detection. In short, if one finds it difficult to use any existing method, use the two principles instead. Next, in practical application, we would like to be able to compare the cost- effectiveness of test sets. In the literature, the effectiveness of a test set is measured by the probability of discovering at least one fault in the test (see, e.g., [FHLS98]). It is intuitively clear that we can increase the fault-discovery capability of a test set simply by adding elements to it. If we carry this idea to the extreme, the test set eventually would contain all possible inputs. At that point, a successful test constitutes a direct proof of correctness, and the probability of fault discovery is 100%. The cost of performing the test, however, will become unacceptably high. Therefore, the number of elements in a test set must figure prominently when we compare the cost-effectiveness of a test set. We define the cost-effectiveness of a test set to be the probability of revealing a fault during the test, divided by the number of elements in the test set. A test set is said to be optimal if it is constructed in accordance with the first and second principles for test-case selection and if its size (i.e., the number of elements contained therein) is minimal. The concept of path testing (i.e., to choose the execution path as the component to be exercised during the test) is of particular interest in this connection because every feasible execution path defines a subdomain in the input domain, and the set of all subdomains so defined constitutes a partition of the input domain (in a set-theoretical sense; i.e., each and every element in the domain belongs to one and only one subdomain). Therefore, a test set consisting of one element from each such subdomain is a good one because it will not only cause every component to be exercised at least once during the test, but its constituent elements will be loosely coupled as well. Unfortunately, path testing is impractical in general because most programs in the real world contain loop constructs, and a loop construct expands into a prohibitively large number of execution paths. Nevertheless, the idea of doing path testing remains of special interest because many known test-case selection methods can be viewed as an approximation of path testing, as we demonstrate later.
- 26. 10 CONCEPTS, NOTATION, AND PRINCIPLES 1.3 CLASSIFICATION OF FAULTS In the preceding section we said that a test case should be selected from a subdomain or a subset of inputs that causes a component to be exercised during the test. Is there a better choice if there is more than one? Is there any way to improve the fault- detection probability by using more than one test case from each subdomain? The answer depends on the types of faults the test is designed to reveal. What follows is a fault classification scheme that we use throughout the book. In the abstract, the intended function of a program can be viewed as a function f of the nature f : X → Y. The definition of f is usually expressed as a set of subfunctions f1, f2, . . . , fm, where fi : Xi → Y (i.e., fi is a function f restricted to Xi for all 1 ≤ i ≤ m), X = X1 ∪ X2 ∪ · · · ∪ Xm, and fi = f j if i = j. We shall use f (x) to denote the value of f evaluated at x ∈ X, and suppose that each Xi can be described in the standard subset notation Xi = {x | x ∈ X ∧ Ci (x)}. Note that, above, we require the specification of f to be compact (i.e., fi = f j if i = j). This requirement makes it easier to construct the definition of a type of programming fault in the following. In practice, the specification of a program may not be compact (i.e., fi may be identical to f j for some i and j). Such a specification, however, can be made compact by merging Xi and X j . Let (P, S) denote a program written to implement the function f described above, where P is the condition imposed on the input and S is the sequence of statements to be executed. Furthermore, let D be the set of all possible inputs for the program. Set D is the computer-implemented version of set X mentioned above. No other constraints are imposed. The definition of set D, on the hand, will be constrained by programming language used and by the hardware on which the program will be executed. For example, if it is implemented as the short integers in C++, then D is a set of all integers representable by using 16 bits in 2’s-complement notation. The valid input domain (i.e., the set of all inputs for which the program is expected to work correctly) is seen to be the set {d | d ∈ D and P(d)}. The program should be composed of n paths: (P, S) = (P1, S1) + (P2, S2) + · · · + (Pn, Sn) Here (Pi , Si ) is a subprogram designed to compute some subfunction f j . P ≡ P1 ∨ P2 ∨ · · · ∨ Pn, and P is in general equal to T (true) unless the programmer places additional restrictions on the inputs. We shall use S(x) to denote the computation performed by an execution of S with x as the input. Two basic types of fault may be committed in constructing the program (P, S). The program created to satisfy a specification must partition its input domain into at least as many subdomains as that required by the specification, each of which must be contained completely in some subdomain prescribed by the specification. Otherwise, there is a domain fault. If there is an element in the input domain for which the program produces a result different from that prescribed by the specification, and the input is in a subdomain that is contained completely in a subdomain prescribed by
- 27. CLASSIFICATION OF TEST-CASE SELECTION METHODS 11 the specification, there is a computational fault. To be more precise, we can restate the definitions as follow. 1. Computational fault. The program has a computational fault if (∃i)(∃ j)((Pi ⊃ Cj ∧ Si (x) = f j (x)). 2. Domain fault. The program has a domain fault if ¬(∀i)(∃ j)(Pi ⊃ Cj ). In words, if the program specification says that the input domain should consist of m subdomains X1, X2, . . . , Xm, the program should partition the input domain into n subdomains D1, D2, . . . , Dn, where n must be greater than or equal to m if the partition prescribed by the specification is compact. The partition created by the program must satisfy the condition that every Di = {d | d ∈ D and Pi (d)} be contained completely in some X j , X1 ∪ X2 ∪ . . . ∪ Xm = X, and D1 ∪ D2 ∪ . . . ∪ Dn = D. Otherwise, there is a domain fault. If there is a subdomain Di created by the program that is contained completely in some X j prescribed by the specification, then for every input in Di , the value com- puted by Si must be equal to that computed by f j . Otherwise, there is a computation fault. It is possible that a program contains both domain and computational faults at the same time. Nevertheless, the same element in the input domain cannot be involved in both kinds of fault. If the program is faulty at a particular input, it is either of domain or computational type, but not both. Previously published methods of program-fault classification include that of Goodenough and Gerhart [GOGE77], Howden [HOWD76], and White and Cohen [WHCO80]. All three include one more type of fault, called a subcase fault or missing-path fault, which occurs when the programmer fails to create a subdomain required by the specification [i.e., if ¬(∀i)(∃ j)(Ci ⊂ Pj )]. Since such a fault also manifests as a computational fault, we chose, for simplicity, not to identify it as a fault of separate type. In Chapters 2 and 3 we discuss test-case selection methods that are designed particularly for revealing the domain faults. In such methods, the components to be exercised are the boundaries of subdomains embodied by the predicates found in the source code or program specification. 1.4 CLASSIFICATION OF TEST-CASE SELECTION METHODS It was observed previously that when a program is being test-executed, not all con- stituent components would be involved. If a faulty component is not involved, the fault will not be reflected in the test result. A necessary condition, therefore, for revealing all faults in a test is to construct the test set in such a way that every contributing component in the program is involved (exercised) in at least one test execution! What is a component in the statement above? It can be defined in many differ- ent ways. For example, it can be a statement in the source code, a branch in the
- 28. 12 CONCEPTS, NOTATION, AND PRINCIPLES control-flow diagram, or a predicate in the program specification. The use of differ- ent components leads to the development of different test-case selection methods. As shown in Chapters 2 and 3, many test-case selection methods have been developed. If the component used is to be identified from the source code, the resulting test- case selection method is said to be code-based. The most familiar examples of such a method are the statement test, in which the program is to be tested to the extent that every statement in its source code is exercised at least during the test, and the branch test, in which the program is to be tested to the extent that every branch in its control- flow diagram is traversed at least once during the test. There are several others that cannot be explained as simply. All the methods are discussed in detail in Chapter 2. If the component used is to be identified from the program specification, the resulting test-case selection method is said to be specification-based. Examples of the components identifiable from a program specification include predicates, boundaries of input/output variables, and subfunctions. Chapter 3 is devoted to a discussion of such methods. It is possible that certain components can be identified from either the source code or the program specification. The component defined in the subfunction testing method discussed in Chapter 3 is an example. Since a component can be also a subdomain consisting of those and only those inputs that cause that component to be exercised during the test, a test-case selection method that implicitly or explicitly requires execution of certain components in the program during the test can also be characterized as being subdomain-based [FHLS98]. The test methods and all of their derivatives, discussed in Chapters 2 and 3, are therefore all subdomain-based. Are there any test-case selection methods that are not subdomain-based? There are at least two: random testing [DUNT84, CHYU94, BOSC03] and operational testing [MUSA93, FHLS98]. The first, although interesting, is not discussed in this book because its value has yet to be widely recognized. The second is important in that it is frequently used in practice. Because it is neither code- nor specification-based, we choose to discuss it in Section 4.2. 1.5 THE COST OF PROGRAM TESTING Software testing involves the following tasks: (1) test-case selection; (2) test execu- tion; (3) test-result analysis; and if it is debug testing, (4) fault removal and regression testing. For test-case selection, the tester first has to study a program to identify all input variables (parameters) involved. Then, depending on the test-case selection method used, the tester has to analyze the source code or program specification to identify all the components to be exercised during the test. The result is often stated as a condition or predicate, called the test-case selection criterion, such that any set of inputs that satisfies the criterion is an acceptable test set. The tester then constructs the test cases by finding a set of assignments to the input variables (parameters) that satisfies the test- case selection criterion. This component of the cost is determined by the complexity of the analysis required and the number of test cases needed to satisfy the criterion.
- 29. THE COST OF PROGRAM TESTING 13 A commonly used test-case selection criterion is the statement test: testing the pro- gram to the extent that every statement in the source code is exercised at least once during the test. The critics say that this selection criterion is far too simplistic and in- effectual, yet it is still commonly used in practice, partly because the analysis required for test-case selection is relatively simple and can be automated to a great extent. The process of test-case selection is tedious, time consuming, and error prone. The most obvious way to reduce its cost is through automation. Unfortunately, some parts of that process are difficult to automate. If it is specification based, it requires analysis of text written in a natural language. If test cases satisfying a selection criterion are to be found automatically, it requires computational power close to that of a mechanical theorem prover. For operational testing, which we discuss in Section 4.2, the cost of test-case selection is minimal if the operational profile (i.e., the probability distribution of inputs to be used in production runs) is known. Even if the operational profile had to be constructed from scratch, the skill needed to do so is much less than that for debug testing. That is one of the reasons that many practitioners prefer operational testing. It should be pointed out, however, that a real operational profile may change in time, and the effort required to validate or to update an existing profile is nontrivial. Test execution is perhaps the part of the testing process that is most amenable to automation. In addition to the machine time and labor required to run the test, the cost of test execution includes that of writing the test harness (i.e., the additional nondeliverable code needed to produce an executable image of the program). The cost of test-result analysis depends largely on the availability of an oracle. If the correctness of test results has to be deduced from the specification, it may become tedious, time consuming, and error prone. It may also become difficult to describe the correctness of a test result if it consists of a large aggregate of data points, such as a graph of a photographic image. For that reason, the correctness of a program is not always unequivocally definable. A class of computer programs called real-time programs have hard time con- straints; that is, they not only have to produce results of correct values but have to produce them within prescribed time limits. It often requires an elaborate test harness to feed the test cases at the right time and to determine if correct results are produced in time. For that reason, a thorough test of a real-time software system is usually done under the control of an environment simulator. As the timing aspect of program execution is not addressed in this work, testing of real-time programs is beyond the scope of this book. Finally, it should be pointed out that, in practice, the ultimate value of a test method is not determined solely by the number of faults it is able to reveal or the probability that it will reveal at least one fault in its application. This is so because the possible economical consequence of a fault could range from nil to catastrophic, and the value of a program often starts to diminish beyond a certain point in time. A test method is therefore of little value in practice if the faults it is capable of revealing are mostly inconsequential, or if the amount of time it takes to complete the test is excessive.
- 30. 2 Code-Based Test-Case Selection Methods We start by discussing a family of test methods that can be used to do debug testing, and the test cases are to be selected based on the information that can be extracted from the source code. In debug testing, as explained in Chapter 1, we want to maximize the probability that at least one fault will be revealed by the test. That is, if we use a test set of n elements, say, T = {t1, t2, . . ., tn}, we want to maximize the probability p(¬OK(t1) ∨ ¬OK(t2) ∨ · · · ∨ ¬OK(tn)) = p((∃t)T (¬OK(t))) = p(¬(∀t)T (OK(t))) = 1 − p((∀t)T (OK(t))) How do we go about constructing such a test set? We can do it incrementally by letting T = {t1} first. If there is any information available to find a test case that has a high probability of revealing a fault in the program, make it t1. Otherwise, t1 can be chosen arbitrarily from the input domain. We then proceed to add another test case t2 to T so that the probability of fault discovery is maximal. That probability can be expressed as p(¬OK(t1) ∨ ¬OK(t2)) = p(¬(OK(t1) ∧ OK(t2))) = p(¬(OK(t2) ∧ OK(t1))) = 1 − p(OK(t2) ∧ OK(t1)) = 1 − p(OK(t2) | OK(t1))p(OK(t1)) As explained in Chapter 1, this can be achieved by finding another test case, t2, such that t1 and t2 are as loosely coupled as possible; that is, ␦(t1, t2), the coupling coefficient between t1 and t2, is minimal. The value of ␦(t1, t2) is a function of the degree of exclusiveness between the sequences of operations to be performed by the program in response to test cases t1 and t2. Software Error Detection through Testing and Analysis, By J. C. Huang Copyright C 2009 John Wiley Sons, Inc. 14
- 31. CODE-BASED TEST-CASE SELECTION METHODS 15 Now if we proceed to find t3, the third element for the test set, the fault discovery probability to maximize is p(¬OK(t1) ∨ ¬OK(t2) ∨ ¬OK(t3)) = p(¬(OK(t1) ∧ OK(t2) ∧ OK(t3))) = p(¬(OK(t3) ∧ OK(t2) ∧ OK(t1))) = 1 − p(OK(t3) ∧ OK(t2) ∧ OK(t1)) = 1 − p(OK(t3) | OK(t2) ∧ OK(t1))p(OK(t2) ∧ OK(t1)) Again, this probability can be maximized by minimizing the conditional probability p(OK(t3) | OK(t2) ∧ OK(t1)), that is, the probability that the program will execute correctly with t3 given that the program executed correctly with t1 and t2. Obviously, this can be accomplished by selecting t3 such that neither OK(t1) nor OK(t2) will have much, if any, impact on the probability p(OK(t3)). That is, t3 should be chosen such that both ␦(t1, t3) and ␦(t2, t3) are minimal. This process can be repeated to add inputs to the test set. In general, to add a new element to the test set T = {t1, t2, . . . , ti }, the (i + 1)th test case ti+1 is to be selected to maximize the probability p(¬OK(t1) ∨ · · · ∨ ¬OK(ti ) ∨ ¬OK(ti+1)) = p(¬(OK(t1) ∧ · · · ∧ OK(ti ) ∧ OK(ti+1))) = p(¬(OK(ti+1) ∧ OK(ti ) ∧ · · · ∧ OK(t1))) = 1 − p(OK(ti+1) ∧ OK(ti ) ∧ · · · ∧ OK(t1)) = 1 − p(OK(ti+1) | OK(ti ) ∧ · · · ∧ OK(t1))p(OK(ti ) ∧ · · · ∧ OK(t1)) This probability can be maximized by minimizing the conditional probability p(OK(ti+1) | OK(ti ) ∧ · · · ∧ OK(t1)), that is, by selecting ti+1 in such a way that ␦(t1, ti+1), ␦(t2, ti+1), . . . , ␦(ti , ti+1) are all minimal. In practice, there are several dif- ferent ways to do this, each of which led to the development of a different test-case selection method, discussed in this and the following chapters. In studying those methods, keep in mind that program testing is a very practical problem. As such, the value of a debug test method is predicated not only on its capability to reveal faults but also by the cost involved in using it. There are three components to the cost: the cost of finding test cases, the cost of performing test execution, and the cost of analyzing the test results. The number of test cases used has a direct impact on all three components. Therefore, we always strive to use as few test cases as possible. The amount of training and mental effort required to construct a test set is a major factor in determining the cost of test-set construction. Therefore, in developing a test method, whenever alternatives are available to accomplish a certain task, we invariably choose the one that is most cost-effective. In some studies, the cost of debug testing includes the cost of removing the faults detected as well. This
- 32. 16 CODE-BASED TEST-CASE SELECTION METHODS is rightly so, because the main purpose of debug testing is to improve the reliability of a software system by detecting and removing latent faults in the system. To facilitate understanding of the test-case selection methods discussed below, it is useful to think that every method for debug testing is developed as follows. First, a type of programming construct, such as a statement or branch predicate, is identified as the essential component of a program each component of which must be exercised during the test to reveal potential faults. Second, a test-case selection criterion is established to guide the construction of a test set. Third, an analysis method is devised to identify such constructs in a program and to select test cases from the input domain so that the resulting test set is reasonable in size and its elements are loosely coupled computationally. Those are the essential elements of every debug test method. 2.1 PATH TESTING In path testing, the component to be exercised is the execution path. The test-case selection criterion is defined to select test cases such that every feasible execution path in the program will be traversed at least once during the test. Path testing is interesting in that every feasible execution path defines a subdomain in the input domain, and the resulting set of subdomains constitutes a partition of the input domain. That is, the intersection of any two subdomains is an empty set, and the union of all subdomains is the input domain. In other words, every input belongs to one and only one subdomain, and the subdomains are not overlapping. If two test cases are selected from the same subdomain, they will be tightly coupled computationally because they will cause the program to test-execute along the same execution path and thus perform the same sequence of operations. On the other hand, if two test cases are selected from different subdomains of this sort, they will cause the program to test-execute along two different execution paths and thus perform different sequences of operations. Therefore, they will have a smaller coupling coefficient, as explained previously. Any subset of the input domain that contains one and only one element from each subdomain defined by the execution paths in the program will satisfy the test-case selection criterion, and the test cases will be loosely coupled. Path testing is not practical, however, because almost all real-world programs contain loop constructs, and each loop construct often expands into a very large number of feasible execution paths. Besides, it is costly to identify all subdomains, find an input from each, and determine the corresponding correct output the program is supposed to produce. Despite its impracticality, we choose to discuss path testing first because all test-case selection methods discussed in the remainder of the chapter can be viewed as an approximation of path testing. Originally, these methods were developed independently based on a variety of ideas as to what is important in a program. The two test-case selection principles developed in Section 1.2 provide a unified conceptual framework based on which of these methods can be described as different instantiations of a generalized method—path testing. Instead of exercising all execution paths, which is impractical in general, only a sample of them will be exercised in these methods. Basically, these methods differ in the ways in which the paths are sampled. These methods are made more practical by reducing the
- 33. STATEMENT TESTING 17 number of test cases to be used, conceivably at the cost of inevitable reduction in the fault-discovery capabilities. 2.2 STATEMENT TESTING In a statement test the program is to be tested to the extent that every executable statement in the program is exercised at least once during the test. The merit of a statement test can be explained simply as follows. In general, not every statement in the program is involved in a program execution. If a statement is faulty, and if it is not involved in a test execution, the fault will never be reflected in the test result. To increase the fault-discovery probability, therefore, we should use a test set that causes every statement in the program to be exercised at least once during the test. Consider the C++ program that follows. Program 2.1 Main() { float a, b, e, w, p, q, u, v; cin a b e; w = b - a; while (w e) { p = a + w / 3; u = f(p); q = b - w / 3; v = f(q); if (u v) a = p; else b = q; w = b - a; } max = (a + b) / 2; cout max endl; } To see the control structure of this program, it is useful to represent it with a directed graph, called a program graph, in which every edge is associated with a pair of the form / C, S, where C is the condition under which that edge will be traversed and S is the sequence of statements that will be executed in the process.1 1This graphic representation scheme makes it possible to represent a program, or its execution paths, by using a regular expression over its edge symbols if only syntactic structure is of interest, or a regular expression over the pairs of the form C, S if the computation performed by its execution paths is of interest. This cannot be achieved through the use of flowcharts.
- 34. 18 CODE-BASED TEST-CASE SELECTION METHODS α β γ δ ε η α: cin a b e; w = b - a; β: / w e; p = a + w / 3; u = f(p); q = b - w / 3; v = f(q); γ: / !(u v); b = q; δ: / u v; a = p; ε: w = b - a; η: / !(w e); max = (a + b) / 2; cout max endl; Figure 2.1 Program graph of Program 2.1. (Adapted from [HUAN75].) The left component / C may be omitted if C is always true. The program graph of Program 2.1 is shown in Figure 2.1. This program is designed to find the abscissa within the interval (a, b) at which a function f(x) assumes the maximum value. The basic strategy used is that given a continuous function that has a maximum in the interval (a, b), we can find the desired point on the x-axis by first dividing the interval
- 35. STATEMENT TESTING 19 2 1 3 4 5 6 7 x f(x) Figure 2.2 Function plot of f(x). into three equal parts. Then compare the values of the function at the dividing points a + w/3 and b − w/3, where w is the width of the interval being considered. If the value of the function at a + w/3 is less than that at b − w/3, the leftmost third of the interval is eliminated for further consideration; otherwise, the rightmost third is eliminated. This process is repeated until the width of the interval being considered becomes less than or equal to a predetermined small constant e. When that point is reached, the location at which the maximum of the function occurs can be taken as the center of the interval, (a + b)/2, with an error of less than e/2. Now suppose that we wish to test this program for three different test cases, and assume that the function f(x) can be plotted as shown in Figure 2.2. Let us first arbitrarily choose e to be equal to 0.1, and choose the interval (a, b) to be (3, 4), (5, 6), and (7, 8). Now suppose that the values of max for all three cases are found to be correct in the test. What can we say about the design of this test? Observe that if the values of function f(x) are monotonously decreasing in all three intervals chosen, the value of u will always be greater than v, as we can see from the function plot. Consequently, the statement a=p in the program will never be executed during the test. Thus if this statement is for some reason written erroneously as, say, a=q or b=p, we will never be able to discover the fault in a test using the three test cases mentioned above. This is so simply because this particular statement is not “exercised” during the test. The point to be made here is that our chances of discovering faults through program testing can be improved significantly if we select the test cases in such a way that every statement will be executed at least once. How do we construct a test set for statement testing? A simple answer to this question is to find, for each statement in a program, a test case that will cause that statement to be exercised during the test. In this way, each statement defines a subdomain (i.e., a subset of inputs that causes the statement to be exercised). Since an input that causes a statement to be exercised may cause other statements to be exercised also, the subdomains so defined are overlapping. Therefore, if we are to construct the test set simply by selecting one element from each such subdomain, the result may contain pairs of elements that are tightly coupled (i.e., causing the same
- 36. 20 CODE-BASED TEST-CASE SELECTION METHODS sequence of operations to be performed, as explained elsewhere). This renders the test set less efficient in the sense that it takes more elements to achieve the same goal. In practice, rarely can we afford to do a full statement test, simply because it requires too many test cases. Software development contracts that require statement testing often reduce the coverage requirement to 60% or less to make the cost of testing more acceptable. The tester is always under pressure to meet the test requirements by using as few test cases as possible. A minimal set of test cases can be constructed by finding a minimal set of execution paths that has the property that every statement is on some path in that set. Then find a test case to traverse each path therein. The test set so constructed will not only have a minimal number of elements but will also have loosely coupled elements (because different elements cause the program to traverse different execution paths), and therefore will be more compact and effective. Another point to be made in this connection: Although the subdomains defined by the statements in a program are overlapping in nature, the elements of a required test set for statement testing need not be selected from overlapping subdomains, as suggested or implied in some literature (see, e.g., [FHLS98]). What is described above is a proactive approach to test-case selection, meaning that we look actively for elements in the input domain that satisfy the selection criterion. It requires nontrivial analytical skill to do the analysis, and could be time consuming unless software tools are available to facilitate its applications. If the program to be tested is relatively small, the test-case designer may be able to find the majority of required test cases informally. Also, some test cases may be available from different sources, such as program designers or potential end users and there is no reason not to make good use of them. In that case, the only thing left to be done is to determine to what extent coverage has been achieved, and if the coverage is not complete, what additional test cases are needed to complete the task. The answer can readily be found by instrumenting the program with software counters at a number of strategic points in the program, as explained in Chapter 7. The software instruments not only provide accurate coverage information but also definitive clues about additional test cases needed to complete the task. This is important because in practice the tester has not only the responsibility to achieve the test coverage prescribed by the stakeholders but also to produce verifiable evidence that the coverage required has indeed been achieved. The software instruments do provide such evidence. There is another way to determine if a particular statement in a program has been exercised during the test. Given a program, create a mutant of that program by altering one of its executable statements in some way. The program and the mutant are then test-executed with test cases. Unless the mutant is logically equivalent to the original program, it should produce a test result different from that of the original program when a test case causes the altered statement to be exercised during the test. Thus, to do the statement test, we can create a mutant with respect to every executable statement in the program, and then test the program together with all mutants until the program differentiates itself from all the mutants in the sense that it produces at least one test result different from that of every mutant. Compared to the method of instrumenting the program with counters, it is advan- tageous, in that it will ask for additional test cases to reveal a fault if the program
- 37. BRANCH TESTING 21 happens to be fortuitously correct with respect to the test case used. Unfortunately, its cost-effectiveness is dismal because in general it requires a huge number of test executions to complete a job. We discuss this topic in more detail in Section 2.7. It must be emphasized here, however, that a statement test gives no assurance that the presence of a fault will definitely be reflected in the test result. This fact can be demonstrated using a simple example. For instance, if a statement in the program, say, x=x+y is somehow erroneously written as x=x-y, and if the test case used is such that it sets y = 0 prior to the execution of this statement, the test result certainly will not indicate the presence of this fault. This is so because although the statement is exercised during the test, the statement x=x-y is fortuitously correct at y = 0. The inadequacy of testing a program only to the extent that every statement is executed at least once is actually more fundamental than what we described above. There is a class of common programming faults that cannot be discovered in this way. For instance, a C++ programmer may mistakenly write if (B) s1; s2; instead of if (B) { s1; s2; } In this case the program produces correct results as long as the input data cause B to be true when this program segment is entered. The requirement of having every statement executed at least once is satisfied trivially in this case by using a test case that makes B true. Obviously, the fault will not be revealed by such a test case. The problem is that a program may contain paths from the entry to the exit (in its control flow) which need not be traversed in order to have every statement executed at least once. Since the present test requirement can be satisfied without having such paths traversed during the test, it is only natural that we will not be able to discover faults that occur on those paths. 2.3 BRANCH TESTING A practical solution to the problem just mentioned is to require that every edge or branch (we use these two terms interchangeably throughout) in the program graph be traversed at least once during the test. In accordance with this new test requirement, we will have to use a new test case that makes B false, in addition to the one that satisfies B, in order to have every branch in the program graph traversed at least once.
- 38. 22 CODE-BASED TEST-CASE SELECTION METHODS Hence, our chances of discovering the fault will be greatly improved, because the program will probably produce an erroneous result for the test case that makes B false. Observe that this new requirement of having every branch traversed at least once is more stringent than the earlier requirement of having every statement executed at least once. In fact, satisfaction of the new requirement implies satisfaction of the preceding one. This is so because every statement in the program is associated with some edge in the program graph. Thus, every statement has to be executed at least once in order to have every branch traversed at least once (provided that there is no inaccessible code in the program text). Satisfaction of the requirement stated previously, however, does not necessarily entail satisfaction of the new requirement. The question now is: How do we go about branch testing? The main task is to find a test set that will cause all branches in the program to be traversed at least once during the test. To find such a set of test cases: 1. Find S, a minimal set of paths from the entries to the exits in the program graph such that every branch is on some path in S. 2. Find a path predicate for each path in S. 3. Find a set of assignments to the input variables each of which satisfies a path predicate obtained in step 2. This set is the desired set of test cases. In step 1 it is useful to construct the program graph and use the method described in Appendix A to find a regular expression describing all the paths between the entry and the exit of the program. Find a minimal subset of paths from that regular expression such that every edge symbol in the program graph occurs at least once in the regular-expression representation of that subset. A set of inputs that will cause every path in that subset to be traversed at least once is the desired set of test cases. It is entirely possible that some paths so chosen may turn out to be infeasible. In that case it is necessary to seek an alternative solution. For example, consider Program 2.1. From its program graph shown in Figure 2.1, we can see that the set of all paths between the entry and the exit can be described by the regular expression α(β(δ + γ)ε)∗ η. It contains subsets αβδεβγεη, αβγεβδεη, (αβγεη + αβδεη), and others that consist of all edge symbols. Any of these can be chosen as the subset of paths to be traversed if it represents a feasible path. Just as in statement testing, test cases can be selected informally or obtained from other sources. Again, the program can be instrumented with software counters to monitor the coverage achieved as described in Chapter 7. All we need to do is to place a counter on every decision-to-decision path in the program. Examine the counter values after the test. If all are nonzero, it means that every branch in the program has been traversed at least once during the test. Otherwise, use the locations of the zero-count instruments as a guide to find additional test cases to complete the test.
- 39. HOWDEN’S AND McCABE’S METHODS 23 2.4 HOWDEN’S AND McCABE’S METHODS The branch test described above can be seen as a method for choosing a sample of execution paths to be exercised during the test. There are at least two other methods for selecting a sample of paths to be tested. Boundary–Interior Method The Howden method, called boundary–interior testing [HOWD75], is designed to circumvent the problem presented by a loop construct. A boundary test of a loop construct causes it to be entered but not iterated. An interior test causes a loop construct to be entered and iterated at least once. A boundary–interior test combines these two to reduce the number of paths that need to be traversed during the test. If an execution path is expressed in a regular expression, the path to be exercised in the boundary–interior test is described by replacing every occurrence of the form α∗ with + α, where is the identity under the operation of concatenation. The examples listed in Table 2.1 should clarify this definition. In practice, the paths prescribed by this method may be infeasible because certain types of loop construct, such as a “for” loop in C++ and other programming lan- guages, has to iterate a fixed number of times every time it is executed. Leave such a loop construct intact because it will not be expanded into many paths. Semantically speaking, the significance of having a loop iterated zero and one times can be explained as follows. A loop construct is usually employed in the source code of a program to implement something that has to be defined recursively: for example, a set D of data whose membership can be defined recursively in the form 1. d0 ∈ D. (initialization clause) 2. If d ∈ D and P(d), then f (d) is also an element of D. (inductive clause) 3. Those and only those obtainable by a finite number of applications of clauses 1 and 2 are the elements of D. (extremal clause) Here P is some predicate and f is some function. In this typical recursive definition scheme, the initialization clause is used to prescribe what is known or given, and the inductive clause is used to specify how a new element can be generated from those given. (The extremal clause is understood and usually omitted.) Obviously, set D is defined correctly if the initialization and inductive clauses are stated correctly. When TABLE 2.1 Examples Paths in the Program Paths to Be Traversed in the Test ab∗ c ac + abc a(b + c)∗ d ad + abd + acd ab∗ cd∗ e ace + abce + acde + abcde
- 40. 24 CODE-BASED TEST-CASE SELECTION METHODS D is used in a program, it will be implemented as a loop construct of the form d := d0; while P(d) do begin S; d := f (d) end; where S is a program segment designed to make use of the elements of the set. Obviously, a test execution without entering the loop will exercise the initialization clause, and a test execution that iterates the loop only once will exercise the inductive clause. Therefore, we may say that boundary–interior testing is an abbreviated form of path testing. Instead of exercising all possible execution paths generated by a loop construct, it is designed to circumvent the problem by exercising only those paths that involve initialization clauses and inductive clauses of inductive definitions implemented in the source code. McCabe’s Method The McCabe method is based on his complexity measure [MCCA76], which requires that at least a maximal set of linearly independent paths in the program be traversed during the test. A graph is said to be strongly connected if there is a path from any node in the graph to any other node. It can be shown [BERG62] that in a strongly connected graph G = E, N, where E is the set of edges and N is the set of nodes in G, there can be as many as v(G) elements in a set of linearly independent paths, where v(G) = |E| − |N| + 1 The number v(G), also known as McCabe’s cyclomatic number, is a measure of program complexity [MCCA76]. Here we speak of a program graph with one entry and one exit. It has the property that every node can be reached from the entry, and every node can reach the exit. In general, it is not strongly connected but can be made so by adding an edge from the exit to the entry. For example, we can make the program graph in Figure 2.1 strongly connected by adding an edge (dashed line), as depicted in Figure 2.3. Since there are seven edges and five nodes in this graph, v(G) = 7 − 5 + 1 = 3 in this example. Note that for an ordinary program graph without that added edge, the formula for computing v(G) should be v(G) = |E| − |N| + 2 Next, for any path in G, we can associate it with a 1 × |N| vector, where the element on the ith column is an integer equal to the number of times the ith edge is used in forming the path. Thus, if we arrange the edges in the graph above in the order αβδεγη, the vector representation of the path αβγεη is 1 1 0 1 1 0 1 and that of βγεβγε is 0 2 0 2 2 0 0. We write αβγεη = 1 1 0 1 1 0 1 and βγεβγε = 0 2 0 2 2 0 0.
- 41. HOWDEN’S AND McCABE’S METHODS 25 α β γ δ ε η μ Figure 2.3 Augmented program graph. A path is said to be a linear combination of others if its vector representation is equal to that formed by a linear combination of their vector representations. Thus, path βγεη is a linear combination of βγ and εη because βγεη = βγ + εη = 0 1 0 0 1 0 0 + 0 0 0 1 0 0 1 = 0 1 0 1 1 0 1 and path αη is a linear combination of αβδεη and βδε because αη = αβδεη − βδε = 1 1 1 1 0 0 1 − 0 1 1 1 0 0 0 = 1 0 0 0 0 0 1 A set of paths is said to be linearly independent if no path in the set is a linear combination of any other paths in the set. Thus, {αβδεη, αβγεη, αη} is linearly independent but {αβδεη, αβδεβδεη, αη} is not (because αβδεη + αβδεη − αβδεβδεη = αη). A basis set of paths is a maximal set of linearly independent paths. In graph G given above, since v(G) = 3, {αβδεη, αβγεη, αη} constitutes a basis set of paths. That is to say, in McCabe’s method, three paths, denoted by αβδεη, αβγεη, and αη, must be exercised during the test. The merit of exercising linearly independent paths in a program can be explained readily in the conceptual framework of this book. According to the definition given above, two paths in a program graph are linearly independent if they have little
- 42. 26 CODE-BASED TEST-CASE SELECTION METHODS in common structurally, which means that the operations to be performed along these two paths will have little in common. Now, if t1 and t2 are the inputs that cause these two paths to be traversed, it implies that the distance between t1 and t2 [i.e., ␦(t1, t2) = p(OK(t2) | OK(t1)) − p(OK(t2)) as defined in Section 1.1] would be greater than if the two paths are not linearly independent. The test cases in a basis set are therefore as loosely coupled as possible as far as can be determined by their dependencies. Thus, to test a program with a basis set is to exercise a finite subset of execution paths in the program that yield a maximal probability of fault discovery. To construct such a test set, we need to construct a set of linearly independent paths first. We can start by putting any path in the set. We then add to this set another path that is linearly independent of the existing paths in the set. According to graph theory, we will have to terminate this process when the number of paths in the set is equal to McCabe’s cyclomatic number because we will not be able to find an additional path that is linearly independent of the existing paths. We then find an input for each path in the set. The result is the test set desired. Note that although v(G) is fixed by the graph structure, the membership of a basis set is not unique. For example, {αβδεη, αβδεβγεη, αη} is also a basis set in G. That is, in McCabe’s method, the set of paths to be traversed is not unique. The set of paths to be traversed can be {αβδεη, αβδεβγεη, αη} instead of {αβδεη, αβγεη, αη}, mentioned previously. It is interesting to observe that v(G) has the following properties: 1. v(G) ≥ 1. 2. v(G) is the maximum number of linearly independent paths in G, and it is the number of execution paths to be traversed during the test. 3. Inserting or deleting a node with an outdegree of 1 does not affect v(G). 4. G has only one path if v(G) = 1. 5. Inserting a new edge in G increases v(G) by 1. 6. v(G) depends only on the branching structure of the program represented by G. 2.5 DATA-FLOW TESTING Data-flow testing [LAKO83, GUGU02] is also an approximation of path testing. The component that will be exercised during the test is a segment of feasible execution path that starts from the point where a variable is defined and ends at the point where that definition is used. It is important to exercise such segments of execution paths because each is designed to compute some subfunction of the function implemented by the program. If such a segment occurs in more than one execution path, only one of them needs to be exercised during the test. For the purpose of this discussion, the concept of data-flow analysis can be ex- plained simply as follows. When a program is being executed, its component, such as a statement, will act on the data or variables involved in three different ways: define,
- 43. DATA-FLOW TESTING 27 use (reference), or undefine. Furthermore, if a variable is used in a branch predicate, we say it is p-used; and if it is used in a statement that computes, we say it is c-used. To clarify the idea, let us consider Program 2.2. In this program, variables x and y are defined on line 2, variable z is defined on line 3, variable y is p-used on lines 4 and 5, variable x is c-used on line 6 while variable z is c-used first and then defined, variable y is c-used and then defined on line 7, variable x is c-used and then defined on line 8, and finally, variable z is c-used on line 9. Program 2.2 main() { 1 int x, y, z; 2 cin x y; 3 z = 1; 4 while (y != 0) { 5 if (y % 2 == 1) 6 z = z * x; 7 y = y / 2; 8 x = x * x; } 9 cout z endl; } This program computes xy by a binary decomposition of y for the integer y = 0, where x and y are both integers. It can be represented conveniently by the directed graph shown in Figure 2.4, in which every edge is associated with a branch predicate followed by a sequence of one or more statements. The program graph represents the control flow of the program. A branch in the graph will be traversed if the branch predicate is evaluated as being true when the control reaches the beginning node of that edge. A path in the program graph can be described by a string of edge symbols, such as αβδεη or αβγεη for short, or sequences of predicates and statements associated with the edges, such as αβδεη: cin x y; z = 1; / y != 0; / y % 2 == 1; z = z * x; y = y / 2; x = x * x; / !(y != 0); cout z endl;
- 44. 28 CODE-BASED TEST-CASE SELECTION METHODS α β γ δ ε η α: cin x y; z = 1; β: / y != 0; γ: / !(y % 2 == 1); δ: / y % 2 == 1; z = z * x; ε: y = y / 2; x = x * x; η: / !(y != 0); cout z endl; Figure 2.4 Program graph of Program 2.2. and αβγεη: cin x y; z = 1; / y != 0; / !(y % 2 == 1); y = y / 2; x = x * x; / !(y != 0); cout z endl;
- 45. DATA-FLOW TESTING 29 These sequences of branch predicates and statements, called symbolic traces, are useful in describing the execution paths of a program. They show not only how the control flow will traverse the program but also what will be done in the process. A path in a program graph is said to be definition-clear with respect to a variable, say x, if it begins at a point where x is defined and contains no statement that causes x to be undefined or redefined. For instance, the execution path between lines 2 and 5 in the following symbolic trace is definition-clear for variable z, and the path between lines 1 and 5 is definition-clear for variable x. Since for our purposes there is no need to be so precise as using the line numbers to indicate the beginning and end of a segment, it suffices to say that path αβδ is definition-clear for both variables x and z. The path between lines 5 and 9 (i.e., path δεη) is definition-clear for variable z. αβδεη: 1 cin x y; 2 z = 1; 3 / y != 0; 4 / y % 2 == 1; 5 z = z * x; 6 y = y / 2; 7 x = x * x; 8 / !(y != 0); 9 cout z endl; A path is loop-free if every edge on the path occurs only once. For example, path αβδεη is loop-free, but the paths αβδεβδεη and αβδεβγεη are not. A simple path is a path in which at most one node occurs twice (if the path is described in terms of nodes) or at most one edge occurs twice (if the path is described in terms of edges). Roughly speaking, a simple path in a program is a path that either does not form a loop, or if it does, it iterates the loop only once and then ends upon exit from the loop. Thus, paths αβδεη and αβδεβ are both simple paths, whereas αβδεβδ is not. A du path of a variable, say x, is a simple path that is definition-clear with respect to x. For example, on path αβδεη the paths formed by lines 12345 and 7345 are du paths with respect to variable x, and the paths formed by lines 2345 and 567345 are du paths with respect to variable z. The definitions, uses, and du paths that exist in Program 2.2 are summarized in Table 2.2. Note that αβγεη is also a du path for variable z defined in α, but is omitted because it represents an infeasible execution path. Now we are ready to enumerate a number of test-case selection criteria that can be formulated based on the data flow in a program. All-du-Path Testing All-du-path testing requires that every du path from every definition of every variable in the program to every use of that definition be traversed at least once during the test. The rationale behind this requirement is that some meaningful computation must have
- 46. 30 CODE-BASED TEST-CASE SELECTION METHODS TABLE 2.2 Variables in Program 2.2 and Their Data-Flow Attributes Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ, αβγεη δ δ, η δεβδ, δεη been performed in the interim, the correctness of which can be tested by exercising that path segment in the test. To do an all-du-path test for Program 2.2, therefore, is to find a small set of test cases that would traverse paths that include all path segments listed in the rightmost column of Table 2.2. It is obvious that those path segments would be included in some paths generated by iterating the loop in Program 2.2 zero, one, and two times, consisting of paths αη, αβδεη, αβγεη, αβδεβδεη, αβδεβγεη, αβγεβδεη, and αβδεβγεη. These are syntactic paths, meaning that they exit based on a syntactic analysis of the program. In general, not all of them are semantically feasible. We can verify their feasibility by analyzing their symbolic traces as shown in Section 5.3. The analysis therein shows that paths αβγεη, αβγεβγεη, and αβδεβγεη are infeasible. Therefore, only the remaining paths αη, αβδεη, αβδεβδεη, and αβγεβδεη could be used as candidate paths for test-case selection. To minimize the effort in selecting the execution paths to be traversed, we can consolidate the requirements prescribed by the rightmost column of Table 2.2 by dropping all path segments that are a subpath of another. For example, we can drop subpath βδ if αβδε is already a member because traversal of the latter implies traversal of the former. We then proceed to construct a table with one row for each path segment that needs to be exercised and one column for each candidate execution path, as illustrated in Table 2.3. A check mark indicates that the path segment listed in the row heading is covered (i.e., is a subpath of) the candidate execution path listed in the column heading. TABLE 2.3 Path Matrix for Table 2.2 Showing the Covering Execution Paths αη αβδεη αβδεβδεη αβγεβδεη αβδε √ √ αβγε √ εβδε √ √ εβγε αη √ εη √ √ √ δεβδ √ δεη √ √ √
- 47. DATA-FLOW TESTING 31 TABLE 2.4 Expanded Version of Table 2.3 αη αβδεη αβδεβδεη αβγεβδεη αβδεβγεβδεη αβδε √ √ αβγε √ εβδε √ √ εβγε √ αη √ εη √ √ √ δεβδ √ δεη √ √ √ We shall henceforth refer to Table 2.3 simply as a path matrix. This matrix indicates that the du path εβγε will not be covered by any candidate execution paths that we have chosen. An additional feasible execution path has to be chosen to cover this segment. As shown in Section 5.3, the path αβδεβγεβδεη turns out to be a feasible execution path that contains εβγε as a subpath. Thus, we expand the path matrix to become Table 2.4. Any candidate test-execution path is said to be indispensable if its removal from the matrix will leave some path segment uncovered. Obviously, every candidate path in Table 2.4 is indispensable. Indispensable candidate paths are the paths that must be exercised during the test. To do a du-path test for Program 2.2, therefore, is to test-execute the program along the paths listed in Table 2.4, which can be achieved by choosing y to be 0, 1, 2, 3, and 5, together with any normal value for x, as the test cases. To be more precise, a possible test set would be T = {x, 0, x, 1, x, 2, x, 3, x, 5} for any valid integer x. All-Use Testing All-use testing requires that at least one definition-clear path from every definition of every variable to every use of that definition be traversed during the test. Clearly, it is a watered-down version of all-du-path testing because the requirement of “all” paths is now replaced by “at least one.” We can use the same technique to find the execution paths that need to be exercised. For ease of comparison, the same table will be used to find the path segments that need to be exercised. Since the requirement is less stringent, the change will be indicated by crossing out the path segments no longer needed, as shown in Table 2.5. Remember that the solution is not unique. For example, the path segment crossed out in Table 2.5 could be αβδε instead. Next, we map the data in Table 2.5 into a path matrix to yield Table 2.6. Note that a test path is no longer needed, and the path segments covered by αβδεη will be covered by αβδεβδεη and αβγεβδεη. Table 2.6 shows that only three paths need to be exercised in an all-use test, and it can be done by using the test set T = {x, 0, x, 2, x, 3}, where x can be any valid integer.
- 48. 32 CODE-BASED TEST-CASE SELECTION METHODS TABLE 2.5 Table 2.2 Revised for All-Use Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.6 Path Matrix for Table 2.5 Showing the Covering Execution Paths αη αβδεη αβδεβδεη αβγεβδεη αβγεβδεη αβδε √ √ αβγε εβδε √ √ εβγε αη √ εη √ √ √ δεβδ √ δεη √ √ √ All-p-Use/Some-c-Use Testing All-p-use/some-c-use testing requires that at least one definition-clear path from every definition of every variable to every p-use (i.e., the definition is used in a predicate) of that definition be traversed during the test. If there is no p-use of that definition, replace “every p-use” in the sentence above with “at least one c-use” (i.e., the definition is used in a computation). The longest path segments in the rightmost column become shorter (Table 2.7). Thus, we reconstruct the path matrix to yield Table 2.8. Table 2.8 shows that only three paths, αη, αβγεβδεη, and αβδεβγεβδεη, need be traversed to exercise all the path segments listed in the rightmost column of Table 2.7. That can be achieved by using test set T = {x, 0, x, 2, x, 5}. All-c-Use/Some-p-Use Testing All-c-use/some-p-use testing requires that at least one definition-clear path from every definition of every variable to every c-use of that definition be traversed during the test. If there is no c-use of that definition, replace “every c-use” in the sentence above with “at least one p-use.” In the present example, the path segments that need to be exercised are identified in the rightmost column of Table 2.9. The path matrix of Table 2.10 shows that all path segments can be covered by three execution paths, αβδεβδεη, αβγεβδεη, and αβδεβγεβδεη. To do an all-c-use/some-p-use, therefore, is to use a test set {x, 2, x, 3, x, 5} to traverse these paths during the test.
- 49. DATA-FLOW TESTING 33 TABLE 2.7 Table 2.2 Revised for All-p-Use/Some-c-Use Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.8 Path Matrix for Table 2.7 Showing the Covering Execution Paths αη αβδεη αβδεβδεη αβγεβδεη αβδεβγεβδεη αβδ √ √ √ αβγ √ εβδ √ √ √ εβγ √ αη √ εη √ √ √ √ δεη √ √ √ √ TABLE 2.9 Table 2.2 Revised for All-c-Use/Some-p-Use Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε, εβδε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.10 Path Matrix for Table 2.9 Showing the Covering Execution Paths αβδεη αβδεβδεη αβγεβδεη αβδεβγεβδεη αβδε √ √ √ αβγε √ εβδε √ √ √ εβγε √ δεβδ √ δεη √ √ √ √
- 50. 34 CODE-BASED TEST-CASE SELECTION METHODS TABLE 2.11 Table 2.2 Revised for All-Definition Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.12 Path Matrix for Table 2.11 Showing the Covering Execution Paths αβδεη αβδεβδεη αβγεβδεη αβδ √ √ εβδ √ √ δεβδ √ All-Definition Testing All definition testing requires that for every definition of every variable in the program, at least one du path emanating from that definition be traversed at least once during the test. This requirement allows us to modify Table 2.2 as shown in Table 2.11. The du paths in Table 2.11 can be translated into the path matrix shown in Table 2.12, which shows that an all-definition test can be done by traversing just one path, αβδεβδεη, which can be done by test-executing the program with x, 3, x being any valid integer. All-p-Use Testing The requirement of all-p-use testing is derived from the all p-use/some c-use by drop- ping the “some c-use” requirement. Table 2.13 is obtained by making all necessary modifications to the rightmost column of Table 2.2. Based on Table 2.13, we con- struct the path matrix shown in Table 2.14. Table 2.14 shows that all path segments in the rightmost column of Table 2.13 can be exercised by traversing three paths: αη, αβγεβδεη, and αβδεβγεβδεη. That can be accomplished by using three test cases: x, 1, x, 2, and x, 5. All-c-Use Testing The criterion of all-c-use testing is derived from that of the all c-use/some p-use by dropping the “some-p-use” requirement. This criterion transforms Table 2.2 into Table 2.15. The rightmost column of Table 2.15 could be exercised by a number of execution paths as shown in Table 2.16, which shows that paths αβδεβδεη, αβγεβδεη, and αβδεβγεβδεη are essential and are sufficient to cover the rest
- 51. DATA-FLOW TESTING 35 TABLE 2.13 Table 2.2 Revised for All-p-Use Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.14 Path Matrix for Table 2.13 Showing the Covering Execution Paths αη αβδεη αβδεβδεη αβγεβδεη αβδεβγεβδεη αβδ √ √ √ αβγ √ εβδ √ √ √ εβγ √ αη √ εη √ √ √ √ δεη √ √ √ √ TABLE 2.15 Table 2.2 Revised for All-c-Use Testing Variable Defined in: p-Used in: c-Used in: du Paths: x α δ, ε αβδ, αβδε, αβγε ε δ, ε εβδ, εβγε y α β, γ, δ, η ε αβ, αβγ, αβδ, αη, αβδε, αβγε ε β, γ, δ, η ε εβ, εβγ, εβδ, εη, εβδε, εβγε z α δ, η αβδ δ δ, η δεβδ, δεη TABLE 2.16 Path Matrix for Table 2.15 Showing the Covering Execution Paths αβδεη αβδεβδεη αβγεβδεη αβδεβγεβδεη αβδε √ √ √ αβγε √ εβδε √ √ εβγε √ δεβδ √ δεη √ √ √
- 52. 36 CODE-BASED TEST-CASE SELECTION METHODS all-path all-du-path all-use all-c/some-p all-c-use all-p/some-c all-p-use branch statement all-def Figure 2.5 Coverage relation among test-case selection criteria. of the path segments in the rightmost column of Table 2.15. Hence, the test cases required are x, 2, x, 3, and x, 5. It is interesting to observe that the data-flow testing criteria discussed above are related in some way. In fact, they are related by the coverage relation mentioned elsewhere. One test-case selection criterion is said to cover another if satisfaction of that criterion implies that of the other. Figure 2.5 shows the coverage relationship among test-case selection criteria based on data flow and the structure of the source code [FRWE88]. 2.6 DOMAIN-STRATEGY TESTING Recall that in the branch testing the components to be exercised during the test are the branch predicates found in the source code, which define the line segments that form the borders of subdomains created by the program. We select a test case arbitrarily from each side of the border to see if the functions defined on the two adjacent subdomains are computed correctly. In this section we show that for each branch predicate, a trio of test cases, lo- cated on or near the borderline it defines, can be chosen to test if the borderline is implemented correctly by the program. The method, as it is outlined here, will work only if the predicate is linear, representing a straight line in n-dimensional space.
- 53. DOMAIN-STRATEGY TESTING 37 This method, known as domain-strategy testing [WHCO80], is designed to detect domain faults. It is based on geometrical analysis of the domain boundary defined by the source code, exploiting the fact that points on or near the border are most sensitive to domain faults. The method works only if the program has the following properties: 1. The program contains only simple linear predicates of the form a1v1 + a2v2 + · · · + akvk ROP C, where the vi’s are variables, the ai’s and C are constants, and ROP is a relational operator. 2. The path predicate of every path in the program is composed of a conjunction of such simple linear predicates. 3. Coincidental (fortuitous) correctness of the program will not occur for any test case. 4. The path corresponding to each adjacent domain computes a different subfunc- tion. 5. Functions defined in two adjacent subdomains yield different values for the same test point near the border. 6. Any border defined by the program is linear, and if it is incorrect, the correct border is also linear. 7. The input space is continuous rather than discrete. The essence of domain-boundary geometrical analysis to be performed in test-case selection can be stated as follows. For simplicity, we use examples in two-dimensional space to illustrate the idea involved. r Each border is a line segment in a k-dimensional space, which can be open or closed, depending on the type of relational operator used in the predicate. r A closed border segment of a domain is actually part of that domain and is formed by a predicate consisting of a ≥, =, or ≤ relational operator. r An open border segment of a domain forms part of the domain boundary but does not constitute part of that domain and is formed by using a , , or = relational operator. For example, let domain A in a two-dimensional space be defined by the predicate x ≥ 1, and let the domain immediately adjacent to it on the left be domain B. Domain B is defined by the predicate x 1. The straight line described by x = 1 is the border of these two domains. The line x = 1 is a closed border and a part of domain A. To domain B, however, it is an open border and is not part of that domain. The test points (cases) selected will be of two types, defined by their relative position with respect to the given border. An on test point lies on the given border, while an off test point is a small distance ε from, and lies on the open side of, the given border.
- 54. 38 CODE-BASED TEST-CASE SELECTION METHODS a b ε c Intended border Actual border Domain Di Domain Dj Figure 2.6 Test points for a border. Continuing the example given above, let ε be 0.01. Then with respect to a border defined by the predicate x = 1, test points such as 1.00, 2.00 or 1.00, 3.14 would be candidates for on test points, and 0.99, 2.14 or 0.99, −4.11 would be that for off test points. When testing a closed border of a domain, the on test points are in the domain being tested, and each off test point is in an adjacent domain. When testing an open border, each on test point is in an adjacent domain, while the off test points are in the domain being tested. This is indeed the case in the earlier example. In testing domain B with open border x = 1, the on points, such as 1.00, 2.00 and 1.00, 3.14, will be in domain A, and the off points, such as 0.99, 2.14 and 0.99, −4.11, will be in domain B itself. Three test points will be selected for each border segment in an on–off–on sequence, as depicted in Figure 2.6. The test will be successful if test points a and b are computed by the subfunction defined for domain Di, and test point c is computed by the subfunction defined for the neighboring domain Dj. This will be the case if the correct border is a line that intersects line segments ac and bc at any point except c. To verify that the actual border is identical to the border intended, we need to select test point c in such a way that its distance from the given border is ε, an arbitrarily small number. The strategy is reliable for all three types of domain faults depicted in Figure 2.7. The domain border may be placed erroneously in parallel below (Figure 2.7a) or above (Figure 2.7b) the intended (correct) one, or may intersect with it as shown in Figure 2.7c. Observe that in Figure 2.7a, fi (c) will be computed as f j (c); in Figure 2.7b, f j (a) and f j (b) will be computed as fi (a) and fi (b), respectively; and in Figure 2.7c, f j (b) will be computed as fi (b) instead. Since it is assumed that fi (p) = f j (p) for any point p near the border, all three types of domain faults can be detected by using this strategy. Recall that two important assumptions were made at the outset: (1) All path predicates are numerical and linear, and (2) functions defined in two adjacent subdo- mains yield different values for the same test point near the border. The percentage of real-world programs that satisfy assumption 1 probably varies widely in differ- ent application areas. Assumption 2 is contrary to requirements in most practical applications: The common requirement is to have the functions defined in adja- cent subdomains to produce approximately the same, if not identical, values near
- 55. Another Random Document on Scribd Without Any Related Topics
- 56. there was nothing to interfere with an undivided and close attention to any object of pursuit. The natural result of acquiring knowledge on these principles and from these causes was, that the knowledge was at last and best the mere lumber of memory, and the theme of vain prate and idle boasting, it was not food for the mind, it was not digested. There was scarcely a piece of music which Miss Henderson could not play at sight; but her style of playing was such as to weary rather than to fascinate; and to listen to the young lady's mechanical dexterity on the piano-forte, was called undergoing one of Miss Henderson's sonatas. There was the same hardness and absence of poetry also in her paintings. The outline was very correct, the colouring was accurate, the transcript complete, but there was no life in the living, no animation in the scenery. There was a provoking likeness in the portraits which she sometimes drew of her friends; and so proud was she of her skill in portrait-painting, that few of her acquaintance could keep their countenances safe from the harsh and wooden mockery of her pencil. Deriving a rich gratification to her vanity from her various accomplishments and miscellaneous acquirements, she fancied that her greatest happiness was in the pursuit of knowledge and the pleasures of science. Much did she despise the follies of the fashionable world, and very contemptuously did she regard the ignorant and half-educated part of the community, and that part in her judgment consisted of nearly all the world, her own self and one or two particular friends excepted. Into this select number Clara Rivolta was most graciously admitted. Miss Henderson, though gifted with a most ample and comfortable conceit of her own superior powers and acquirements, was still not backward, but rather liberal and dexterous in administering the delicious dose of flattery to those whom she honored with her notice and approbation, as being superior to the ordinary mass of mortals. Clara Rivolta received the homage paid to her mind and acquirements as the effusions of a warm heart and generous spirit. It is possible, however, to mistake heat of head for warmth of heart. This was a mistake into which Miss Henderson was perpetually falling, both as it related to herself and to others. Not only was the
- 57. young lady liberal in her praises of those whom she would condescend to flatter with the honor of her approbation, but she absolutely praised them at her own expense, expressing her high sense of their superiority to herself. But it should be added, that this kind of homage always expected a return with interest, and the language in which she praised her friends was only put forth as a model and specimen of that kind of homage which she should be best pleased to receive from her dear dear friends. To the vanity of intellect Miss Henderson added the vanity of sentiment. She had read something in books about the heart, and about sentiment and feeling, and so on; and she thought that there must be something fine in that concerning which so many fine words had been used. Thereupon, with that conceit she added sentimentality to the rest of her acquirements; and an acquirement in good truth it really was, seeing that it was by no means natural. Not the less fluently could the young lady discourse on that subject, because she knew nothing about it; but, on the other hand, she set herself up as a judge and censor-general on all her acquaintances and the world beside on the subject of sensibility of heart. She had enjoyed many opportunities of falling in love, and those which she had enjoyed she had not overlooked. Many and many a time was her heart lost, but never irrecoverably. Few were the gentlemen who thought it very prudent to venture to pay serious court to a young lady of lofty thoughts and lowly means. A very slight degree of notice was sufficient however to set if not her heart in flames, at least her tongue in motion to her confidential friends concerning sentiment and sensibility, and all that sort of thing. Such a companion as this was by no means fit for Clara Rivolta. But Mr. Martindale saw not the real character of the young lady, and Miss Henderson was wise enough to flatter the old gentleman into a conceit that she considered him as one of the few enlightened men of the age; and as Mr. Martindale himself was one of those oddities who think all the world blockheads but themselves, he was not displeased with that kind of homage which Miss Henderson paid
- 58. him: and as Mr. Martindale was one of the very few single gentlemen whom Miss Henderson had seen and had not fallen in love with, she was not quite so disagreeable to him as she was to many others. Mr. Martindale, therefore, tolerated the acquaintance with Clara; and as for Signora Rivolta, it appeared that Miss Henderson had sagacity enough to see that she was not to be imposed on or deceived by foolish talk, and therefore she avoided exposing herself to her. In person Miss Henderson was by no means disagreeable, she was rather pretty. There was it is true a little deficiency in height and a little redundancy in breadth; but still there was nothing remarkable one way or the other. She dressed in very good taste, and her ordinary manner was good. It is wicked, or at least very thoughtless, in young men to pay unmeaning attentions to any young lady, but especially to such very sentimental ones as Miss Henderson: frequently had she been rendered unhappy by this thoughtlessness. Now it is very silly for young men to boast of the hearts they win; and in winning such a heart as we are now speaking of there is certainly nothing to boast of, for any one was sure to succeed provided there was a vacancy. At the time of which we are writing, the fragrant Henry Augustus Tippetson was the favored and honored companion of Miss Henderson's walks; and it is difficult to say which was the prettiest animal of the two, Mr. Tippetson or his little white French dog. They were at one time always to be seen together at a certain hour of the day in the Green Park. They seemed to have a great fellow feeling, and both looked as spruce and neat as if they had both been dressed by the same valet. Mr. Tippetson, though something of a coxcomb, and considered to be vain of his person, still was so far diffident of himself as to use the assistance of his little quadruped companion to attract attention to himself. Often has he acknowledged, or rather boasted, that his little dog has been the means of bringing him into conversation with those whom otherwise he should not have had an opportunity of addressing; and oftentimes it has been supposed that it was Henry Augustus Tippetson's private opinion, that his little French dog was considered
- 59. by the ladies as a very pretty excuse for taking notice of the pretty owner of the same. Now it was the natural unsophisticated opinion of Clara Rivolta that Mr. Tippetson was an empty-headed, effeminate coxcomb, not worth notice, and absolutely incorrigible by any other discipline but that of time. But Miss Henderson had discovered, or fancied she had discovered, that Mr. Tippetson was not so great a coxcomb as he appeared to be. She acknowledged, indeed, that he was very attentive to his dress and his person; and very candidly did she make allowance for a little error in that respect, as he was but young, and she had heard it said that it is better to be too attentive in youth than too negligent in age in that respect. As for Mr. Tippetson's lisping, she was very sure that was perfectly natural and unavoidable. The use of perfumery was become absolutely necessary from the frequency of crowded apartments. As to the apparent diversity between the studying and the learned Miss Henderson, and the lounging, indolent, unreading habits of Mr. Tippetson, the difference was rather apparent than real, according to the young lady's own account of the matter: for though Mr. Tippetson was not at present much in the habit of reading, he had been formerly, and his mind was by no means unfurnished; he was a man of very great observation, and was constantly making remarks and observations on every thing he saw or heard. So that Miss Henderson was quite sure that when Clara came to be better acquainted with the young gentleman, she must think better of him. Thus it is that foolery is tolerated. Look at a coxcomb at a little distance, and observe his silly airs. The animal is absolutely nauseous, and his whole manner and style villanous and contemptible. But a more intimate acquaintance makes a discovery of some bearable qualities; and familiarity renders the odious less odious; and then it is thought that there are more qualities existing in him than have been discovered, because more have been discovered than were suspected. So foppery and foolery are tolerated from habit and intimacy.
- 60. This process of mind, from contempt to toleration, has been experienced by more disciplined minds than Clara's. No wonder that a young woman so unacquainted with human society should be led to sacrifice her better judgment to the plausibilities of so well- informed a person as Miss Henderson. Clara was far from perfection, though she was a most excellent and amiable creature, and was possessed of a tolerably sound judgment. She was accessible to flattery, and loved praise. It was not in her power or will to discriminate aright on that matter. Signora Rivolta had instructed and educated her daughter very much by the impulse of encouragement. That mode had produced many good effects, but it had its evils. Clara had become too susceptible of commendation, and her appetite was too strong to suffer her taste to be delicate. Thus there arose a kind of sentimental friendship between the two young ladies; in which intercourse of sentiment Miss Henderson had the advantage and the greatest power, not from superior strength of mind, or greater accuracy of discrimination, but because it had been her lot to enjoy a larger portion of experience or knowledge of human society. It might be imagined that a woman of such superior mind as Signora Rivolta, would have given to her only child, whose education she had by herself totally conducted, such information and such views of society and human nature, as to render her so well acquainted with life that she might not be a dupe of its ordinary deceptions. But this is not possible. Solitary education can never fit the mind for society; the social education must commence when the solitary has finished. Young people cannot understand the language of experience. Signora Rivolta might even have described with the utmost truth and philosophic accuracy the character of Miss Henderson, and might have given her child the strictest and most earnest injunctions to guard herself against its fascinations; and Clara might have been most attentive to the instruction, and desirous of obeying it, but when the character presented itself in real life she would not have recognised it.
- 62. CHAPTER VIII. Fair, gentle, sweet, Your wit makes wise things foolish. Shakspeare. The intimacy between Clara Rivolta and Miss Henderson continued for a time uninterrupted. The friends of the former were not aware of the character of Miss Henderson; and had they been, they would not have supposed that Clara so much admired and esteemed her as she really did. Old Mr. Martindale had paid very kind and friendly attention to the settlement of the affairs of the late Lord Martindale, and having spent as much time in town as he thought desirable, removed his establishment to the coast, that his family might enjoy the pleasures of a watering-place. So it came to pass that the female friends were parted: but though separated, they were not forgetful of each other. Miss Henderson wrote a most beautiful hand, so small, so clear, with letters so peculiarly well-turned, that whenever she put a letter into the post-office, she thought that the letter- sorter and the postman must pause to admire the beauty of the writing. Not one of her numerous acquaintance could condense so many words into the compass of one common-sized sheet of letter- paper. With such qualifications, no wonder that she seized every opportunity of writing letters. People always do with pleasure that which they think they do well. It would gratify us if we could present the public with a fac-simile of one of Miss Henderson's letters, but our publisher will not allow of it. Our readers must therefore be contented with the printed copy of one. It is as follows: How feeble, my ever dear Clara, is the power of language to express the emotions of the heart! My heart is the seat of ten thousand times ten thousand agitating and conflicting thoughts, painful recollections, gloomy forebodings, tender regrets, joyous
- 63. hopes. Oh, what is life without friendship! And how few, alas! who are worthy of the confidence of friendship. When I look upon the multitudes of people that pass and repass every day and every hour —when I see the common-place, every-day people of the world, and observe how careless and how contented they seem in the midst of their gross ignorance and stupidity, I cannot sometimes repress the almost impious wish that I were as stupid as they are. Is it not too true that increase of knowledge is increase of sorrow? The town is now empty, and yet pa's chapel is somehow well attended. Many people come from the city to hear him, and some young lawyers also from the inns of courts. You would be amused to see their vacant looks of admiration, almost amounting to astonishment, when pa gives one of his fine apostrophes, or his well-turned metaphors. Tippetson is still in town. I wonder what he can find to amuse him at this dull time of year. Pa would go to Brighton if he could find any tolerable substitute to fill his place in his absence, but the generality of preachers you know, my dear, are so dull, stupid, and common-place, that it is absolutely impossible for a person of any sense to sit and hear them; I am really astonished that the churches are so full as they are, there is scarcely one clergyman in twenty worth hearing. Tippetson says he is determined never to hear any body but pa. He was at the chapel yesterday, and sat directly opposite to us. I wish, my dear Clara, you could have seen him. He was so attentive, that he looked as if he was desirous of catching every syllable; and when any peculiarly fine and brilliant expression occurred, (you know pa's emphatic manner,) it was quite interesting to see how his countenance was lighted up with admiration. Pa makes it a point to preach quite as well when the fashionable people are out of town as when they are all here. Fashionable people! Ah, Clara, you do not know them so well as I do, and you need not wish to know them; so false, so vain, so hollow, so haughty! Mr. Martindale is the only person I ever saw who seems to understand them aright. What an advantage you have in the society of such an intelligent man! All his thoughts are wisdom, and all his sentences are oracles. Tippetson admires him prodigiously, and says that twenty such men in high life would
- 64. produce a complete revolution in the fashionable world. I heard Tippetson say, but he did not know that I heard him, that he intended to procure a little velvet paper book, bound in pink satin and with silver clasps, and that he should on one side of the leaves record the wise sayings of Mr. Martindale, and on the other the beautiful similes that occurred in pa's sermons. I am sure, my dear, you will laugh when I tell you what a blunder pa was likely to make last week. He wanted to go out of town for a few weeks, and endeavoured to find a gen gentleman to officiate for him at the chapel, and a friend of ours recommended a person of whom he had some slight knowledge, and pa saw him, and was just on the point of engaging him, when by some odd expression pa found out that he was an evangelical. It would have been the ruin of the chapel if the mistake had not been discovered in time to prevent any engagement. Poor pa made the best excuse he could; and here we all remain for want of a proper substitute to supply the chapel in our absence. When Tippetson heard of the blunder, he laughed outright, and said it was a pity that the gentleman had not been engaged, in order that he might convert some of the good folks at this end of the town. I assure you, my dear, that Tippetson is far from being a dull man; the fact is, he has a considerable degree of wit, but he is not like some people who are always endeavouring to shine in conversation, and to say brilliant things. Now do you know there is nothing so excessively disagreeable to me as a perpetual endeavour to shine in conversation. Tippetson really does say some good things sometimes; I am told that some of those clever articles in the newspapers which are ascribed to Sir William Curtis, are actually the production of Tippetson. Well, my dear, dear Clara, you see what a rambling style I am writing in; but I don't know how it is, when I have the pen in my hand it seems to communicate the perpetual motion to my fingers. Talking of perpetual motion, what an absurdity it is to think that it can ever be discovered! and yet I have heard people who think themselves very clever at mechanics talk as if it might be discovered; I once heard a very superior man say that the perpetual motion was one of nature's arcana. Oh, how pleasant it is to have a correspondent to whom one can write freely and fully on
- 65. any subject! How few are there like you, my dear friend, who have any interest in the pursuits of science and the discoveries of philosophy; and of those few who pretend to any relish for such things, there are not many who like you understand the subjects on which they converse: they are mere smatterers. You told me, I remember, that you found less literature and science in the world than you expected; and let me assure you, that is much rarer than even you imagine. The number of pretenders is very great; but real science is rare. I am afraid I shall tire you, my dear friend, but the truth is, I know not of any one to whom I can address myself so freely as to my dear Clara: but if I have trespassed too much on your valuable time by my poor unworthy scrawl, I can only cast myself on your mercy, or beg that you will punish me by an answer as long as my letter: punish did I say, I retract the unworthy expression, it would be no punishment to receive a copious epistle from a dear, intelligent, superior-minded friend. Your letters, my dear, are all instruction and wisdom. I could learn more from one of your letters than from a volume. I am almost ashamed of what I am going to say, I hesitate whether I shall acknowledge my sin; yet confession is one-half of repentance. I must acknowledge—will you ever forgive me? The fact is, I was so very naughty as to let Tippetson have a sight of your letter; and I am sure you will forgive me, if you can but imagine the admiration and delight with which he read it. I know you have too much strength of mind to be accessible to flattery, else I would not mention the affair to you; but the truth is, that he was so charmed with it that he begged me on his knees to let him have a copy of it, of course omitting names, and he was pleased to say that no pen was so worthy of the honor of transcribing it as mine: for the young man is pleased to compliment my hand-writing rather more than it deserves. There, now you are near the end of your labor of reading, and I am near the end of my pleasure in writing: for a pleasure it is to write to such a dear, kind, intelligent soul as my Clara Rivolta. Farewell; let me soon have another invaluable treasure in a letter from your intellectual pen to delight and instruct your faithful and sincere Rebecca Henderson.
- 66. This is the shortest we could find of the epistles of Miss Henderson to Clara Rivolta. The young lady to whom that and many more to the same purpose were addressed, thought that there was something extravagant in the style, but took it for granted that such was the style now in fashion, and therefore made allowances; but the worst of the matter was, that in making those allowances, she was led also to imitate the same style rather more than her good sense approved. As we have not so high an opinion of the superior excellence of Clara Rivolta's letters to Miss Henderson as Mr. Tippetson was pleased to express, we shall not favor our readers by sending any of them to the press. It is enough to state that the correspondence continued rather longer than Signora Rivolta would have approved had she been aware of its style and character. Those letters which Miss Henderson wrote to Clara when Mr. Tippetson had left town, and was gone she knew not whither, were of sad and sable aspect. Many and deep were the lamentations that there was nothing in the great metropolis worth living for, and yet there were several young gentlemen then in town whom Miss Henderson had once been dying for. Though Signora Rivolta did not think it necessary for the sake of her daughter's well-being, and for the purpose of preserving the purity of her mind, to insist upon seeing all letters which she might receive from or write to her female friends, yet the very frequent arrival of letters from Miss Henderson, and the very copious nature of them, judging from the time which Clara took to read them and reply to them, induced her mother to mention the subject, by way of hinting that such a very great intimacy and attachment to so new an acquaintance was hardly consistent with prudence and proper consideration. One day, when a very long communication had been received from the copiously-corresponding Rebecca Henderson, Signora Rivolta took occasion to say to Clara, I have no wish, my dear child, to interfere unnecessarily with your correspondence and with your friendships, but it has often struck me that your frequent and long letters to Miss Henderson are hardly
- 67. proper, considering how short a time you have been acquainted with that lady. I protest to you that I feel curious to know what is the subject of your correspondence. Is it literary, or scientific, or miscellaneous? Clara was rather confused, because she was well aware of the rigid and severe judgment of her mother, and she was nearly sure that such a correspondence would not altogether meet her approbation; she replied, There are some parts of the letters which treat of literature and science, but they are for the most part miscellaneous: they are a species of written conversation. There is very little conversation worth writing, replied her mother, and of course little worth reading; but, continued she with a smile, I must own that I should be gratified, if you would so far indulge my curiosity as to permit me to see one of Miss Henderson's letters, I will not dictate which, one will answer my purpose as well as another. Where the correspondence is so frequent and copious, it must display or form character, and I am interested to know what kind of a correspondent you have. I will show you any or all of the letters, replied Clara, who knew her mother's character of mind too well to attempt to elude her penetration; will you take the trouble to read this, which I have received this morning. Miss Henderson is a very kind friend, and perhaps she is disposed to be rather too flattering; but I can assure you, my dear mother, that, though I am pleased to have her good opinion, I am not rendered vain by her praises. It is her peculiar manner to compliment. Clara presented the letter, Signora Rivolta read it with great attention and with much seriousness; occasionally indeed she smiled, but that smile was presently checked. Clara watched her mother's countenance with great anxiety, observed its changes with much emotion, and was very much hurt and abashed by the look with
- 68. which her mother returned her the letter when she had read it through. My dear Clara, I must have some conversation with you on the subject of this correspondence. I cannot flatter you quite so adroitly as Miss Henderson does, but I have a much higher opinion of you than she has, for I do assure you that I would never have insulted your understanding so much as to send you such a ridiculous epistle as this. The language of this letter is foolish in the extreme. I hope your conversation was not in this style. You may well say that the letters are miscellaneous. Now my child, as I can only have your welfare in view, will you be kind enough so far to favor me with your confidence as to indulge me with a sight of some more of Miss Henderson's letters. Indeed, I acknowledge to you that I am anxious to see them all: from this specimen I could wish that you had never received any. Clara had much tenderness of feeling and great respect and reverence for her mother's superior understanding, and she felt very unpleasantly and painfully at the emotion with which her mother addressed her. Her color rose and fell, and the poor girl burst into tears. It was indeed truly mortifying, after having received such flattering homage from Miss Henderson, to be thus suddenly let down in her own judgment, and be thus brought to feel that she was not quite so superior to the rest of the world as she had been led to suppose herself. Signora Rivolta soothed her and spoke kindly to her. The letters were produced, for they had been carefully laid by as treasures of some value. And let not our readers judge harshly of the inexperienced mind of poor Clara; her very humility made her proud: for not positively thinking very highly of herself, when she found herself flattered and complimented by Miss Henderson, she was unduly exalted and gratified; and she began to fancy that she was indeed something extraordinary, to receive compliments from so accomplished a young lady. Every one of the letters did Signora Rivolta carefully peruse, having taken them to her own apartment for that purpose: it was a task
- 69. indeed of some difficulty, and attended with much weariness, but she felt anxious on her daughter's account, and would not relinquish her task till it was completed; and when it was completed, her astonishment was great indeed, that such nonsense could have ever been agreeable to her daughter; but she forgot the love of praise and the insinuations of flattery, how strong their influence is on young and inexperienced minds. When she had made an end of reading, she called Clara to her again, and after giving her own opinion of the character of Miss Henderson's mind, she said, Now, my dear child, I have one more request to make of you concerning these letters; that is, will you give me leave to destroy them? They will never be an honor to you; if you seek for praise, you must endeavour to procure it in a less equivocal shape than this. Here you are told in so many words almost, that your knowledge is most extensive and profound; that your taste is pure and perfect; that your strength of mind is superior to the rest of your sex; in short, that you possess every virtue and every excellence that can be attributed to a human being. Can you believe that Miss Henderson is silly enough to entertain such an opinion of you; and if she be not sincere, what can be her motive, but merely to indulge her own foolish inclination to talk or to write? I dare say that, if the fact could be ascertained, you would find that she has used the same kind of language to many others with whom she has corresponded; for her style seems to be that of a practised letter writer, who scribbles for her own gratification. Clara saw that there was truth and justice in these observations, and though she felt a considerable degree of reluctance, she could not refuse her mother's request; and the letters in all their fulness and interest and with all their fine compliments, were committed to what newspaper editors call the devouring element.
- 71. CHAPTER IX. Full many a lady I have eyed with best regard. Shakspeare. The sea-side is an excellent place for those who have nothing to do, and none but those can duly and rightly appreciate its advantages. To saunter about on the beach and listen to the roaring of the waters, and watch the tide rising or falling—to hear the rushing rattling of the pebbles that are rolled on the beach with every successive wave—to see the distant sail, now dark beneath a passing cloud, and then bright again as a leaf of silver from the light of the sun—to watch the sea-birds in their reeling, wheeling, staggering flight—to mark the little dabs of sea-weed in their grotesque variety, and to measure the progress of the tide by their disappearance on encroaching waves, or to measure how much the waters have receded, by observing how this, that, and the other weed are drier and farther from the reach of the wave—to notice the pretty wonder, and see the waving ringlets and fluttering bonnets of the little ones who are brought to breathe health and animation on the coast—to watch the sentimental looks of the solitary wanderer who comes there to breathe poetry—to see the flushed indications of a swelling heart in the looks of those who seem to hear in the sound of the rushing waters a voice from beloved and distant ones; these, and ten thousand other pretty occupations, banish from the mind all feeling of indolence, and make it fancy itself employed. And surely it is quite as well employed in seeing and feeling poetry as in reading it. To speak after the figurative and flowery style of Rebecca Henderson's pa, we might say that nature is all poetry, groves are her sonnets, gardens her madrigals, mountains her pindarics, and seas her epics. In walking by the sea-side, there is no thought of loss of time; for the sea being an emblem of eternity, banishes all
- 72. thought of time from the mind. Thus wandered there, day after day, our young friend, Clara Rivolta. Not the less interesting to us is this young lady, because in the simplicity and youngness of her experience, she has suffered herself to be carried away by the foolish and vain flatterings of an idle- minded, busy-tongued young woman, on whose mind knowledge has produced only its coarsest and grossest effects. To the sea-side did Clara betake herself the morning after the discovery of her foolish, sentimental correspondence with Miss Henderson. Many and painful were the efforts which she made to endeavour to think more soberly of herself, and to bring her thoughts and feelings to that steadiness and firmness which she could not but perceive and respect in her mother. It was not easy, it was not pleasant, to rouse herself from that delicious dream of self-complacency into which she had been lulled. We do not like to wake from a pleasant dream, even though we know it to be but a dream. Clara was also helping to deceive herself. She was indulging herself in the thought that she was far less censorious than she used to be, for she thought more favorably and judged more candidly of Mr. Tippetson than when she first saw him. But she forgot that a little of that candor and a little of that justice might perhaps be owing to the decided and flattering homage which that sweet-scented gentleman had paid her. No one can think very contemptibly of those who dexterously flatter; and there is a species of flattery which is very dexterous, which does not express itself in the bare words of common-place compliment or gross adulation, but which speaks in looks, tones, actions, and attentions. Mr. Tippetson had learned this art in perfection, and no wonder; for he had studied no other, and had found an interest and a pleasure in this. He had been despised and tolerated by a great number of persons and families. At first sight, all who came near him despised him, but upon better acquaintance they thought better of him; and as he had no feeling but for himself, he could when necessity required make himself very agreeable to most with whom he conversed. His art was to affect an almost exclusive interest for the person whom he addressed or conversed with. By this he had
- 73. rendered himself so very agreeable to Sir Gilbert Sampson, then to Miss Sampson, now Countess of Trimmerstone, that many observers thought at one time that he would have carried off the heiress; and very likely he would, had it not been for the title which was in the Martindale connexion. Even after the marriage of the young lady above-named, her lord and master was almost jealous of Tippetson, but when he became better acquainted with him he thought better of him. There was this also in the style of the young gentleman, that he never affected any superiority, but always spoke, if of himself at all, in terms of great humility and diffidence. It is indeed rather flattering and agreeable to us, when seeing one who at a little distance appears full of himself, very proud and conceited and contemptuous, we find when we come nearer to him, that he thinks very humbly of himself, and that he is towards us all respect, deference, and attention. The homage of those who seem constitutionally constructed to pay homage to every body affects and delights us not, but the homage of those who seem to expect homage is truly delightful. By this kind of art, Mr. Henry Augustus Tippetson rendered himself tolerable in spite of his foppery and exquisite affectations. We have introduced this gentleman again, because he is about to introduce himself; and though we would willingly let our characters speak for themselves, as we have said in the preceding volume, we cannot always trust them, for we know that they are all more or less hypocritical, and would put the best side outwards. Besides, there is wanting in narrative or written dialogue the countenance or expression, which the actor in a drama gives to the character. There may be much individuality in the characters of Shakspeare's dramas, but we question whether there be not somewhat less than most persons imagine. With all the individuality, however, that may be supposed to belong to them, we have very little doubt but that to different minds the names of Hamlet, Macbeth, Shylock, c. present an almost infinite variety of moral portrait; and there is great truth in the common language and philosophy in the common phraseology, of Kean's Richard, Garrick's Richard, Kemble's Coriolanus.
- 74. Kean's Richard and Garrick's Richard were no doubt different persons. Our characters, therefore, cannot speak definitely enough, if they only speak for themselves. But Mr. Henry Augustus Tippetson is coming. As Clara Rivolta was walking alone on the beach the morning after the destruction of Miss Henderson's pretty letters by the ruthless severity of Signora Rivolta, the young lady's mind was full of various and agitating thoughts, and she was meditating on the deceitfulness of the world and all that is therein; and then began she to think of Horatio Markham, whose considerate and kind attentions had greatly impressed her mind with a favorable idea both of herself and of him. She thought within herself, Is he also insincere, and must I believe nobody who speaks well of me? It was a pleasant and a soothing sight to have before her eyes the mighty ocean, and to see its rolling billows dashing with infinite monotony on the shore; and it was pleasant to her to let her spirit ride upon these mighty billows onward and onward to distant lands, and to imagine that there lived one in whose life she felt an interest. Whatever Miss Henderson might be, she was sure that Horatio Markham lacked not understanding and good sense; for she had observed that her mother had conversed with him very attentively, and apparently with great pleasure. Being disappointed in her friendship with Miss Henderson, her thoughts very naturally reverted to her incipient friendship with Markham. While engaged in these meditations, she was roused from her reverie by the sound of something plunging into the water, and presently after by the noise of the sharp, shrill yelping of a little dog that came shaking his white wet coat almost on Clara's dress. The master of the animal was at hand ready to apologise for the animal's want of decorum. The apology was accepted; the owner of the dog was Mr. Tippetson, he did not say that he threw the dog into the water on purpose to attract the young lady's attention. As Mr. Tippetson was personally acquainted with Clara, a conversation naturally sprung up on the occasion of this accidental meeting.
- 75. Clara hardly knew what to say, or what to avoid saying; for when the letters of Miss Henderson had been perused by Signora Rivolta, the frequent recurrence of the name of Tippetson gave occasion to that lady to make some remarks on the character of that gentleman by no means flattering either to him or to Clara's judgment. It was however impossible to behave rudely to the young gentleman. Some answer necessarily must be returned to his inquiries concerning Mr. Martindale, and the Colonel and Signora Rivolta. And as Mr. Tippetson very unceremoniously joined company with Clara, and pertinaciously walked by her side, and took all possible pains to make himself agreeable, it was impossible to get speedily rid of him. In the course of conversation mention was made of the name of Henderson. Clara bethought herself of the story of Mr. Tippetson supplicating on his knees for a copy of a letter addressed to her friend. Mr. Tippetson spoke of Miss Henderson not quite so flatteringly as Miss Henderson had spoken of him. Clara thought that ungrateful. Mr. Tippetson spoke of Mr. Henderson, but of him not quite in such high terms as the language used in Miss Henderson's letters had led Clara to expect. But there was nothing absolutely censorious in his expressions. Of Mr. Martindale he spoke in language of unmingled commendation, of Colonel Rivolta he spoke very flatteringly, and so also of Signora Rivolta. It was well for Clara that she had been so recently put upon her guard against flattery, or the ingenious homage of Mr. Tippetson might have intoxicated her. So dexterously did he manage, that, notwithstanding all that Signora Rivolta had said, and notwithstanding what she herself had seen, Clara could not help wishing that her mother had been better acquainted with Mr. Tippetson, as she felt assured that better acquaintance would produce better and more favorable thoughts of him. While Mr. Tippetson was keeping Clara in conversation almost against her will, there appeared at a distance on the beach old Mr. Martindale and Signora Rivolta. Clara would willingly have extricated herself from her companion, could it have been effected without any obvious and palpable effort. But she saw that it was absolutely
- 76. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com