Software Testing Introduction (Part 3)

LEVELS OF TESTING
• Software testing is generally carried out at
different levels.
• There are four such levels namely unit testing,
integration testing, system testing and
acceptance testing as shown in Figure 8.1.
• The first three levels of testing activities are done
by the testers and the last level of testing
(acceptance) is done by the customer(s)/user(s).
• Each level has specific testing objectives.
• For example, at the unit testing level,
independent units are tested using functional
and/or structural testing techniques.

• At the integration testing level, two or more units
are combined and testing is carried out to test the
integration related issues of various units.
• At the system testing level, the system is tested as
a whole and primarily functional testing
techniques are used to test the system.
• Non-functional requirements like performance,
reliability, usability, testability, etc. are also
tested at this level.
• Load/stress testing is also performed at this level.
• The last level i.e. acceptance testing, is done by
the customer(s)/user(s) for the purpose of
accepting the final product.

Unit Testing
• We develop software in parts / units and every
unit is expected to have a defined functionality.
• We may call it a component, module, procedure,
function, etc., which will have a purpose and may
be developed independently and simultaneously
• There are also problems with unit testing. How
can we run a unit independently?
• A unit may not be completely independent. It may
be calling a few units and also be called by one or
more units.
• We may have to write additional source code to
execute a unit.

• A unit X may call a unit Y and a unit Y may call
a unit A and a unit B as shown in Figure 8.2(a).
• To execute a unit Y independently, we may
have to write additional source code in a unit
Y which may handle the activities of a unit X
and the activities of a unit A and a unit B.
• The additional source code to handle the
activities of a unit X is called ‘driver’ and the
additional source code to handle the activities
of a unit A and a unit B is called ‘stub’.
• The complete additional source code which is
written for the design of stub and driver is
called scaffolding

Integration Testing
• A software program may have many units.
• We test units independently during unit testing
after writing the required stubs and drivers.
• When we combine two units, we may like to test
the interfaces amongst these units.
• We combine two or more units because they
share some relationship.
• This relationship is represented by an interface
and is known as coupling.
• The coupling is the measure of the degree of
interdependence between units.
• Two units with high coupling are strongly
connected and thus, dependent on each other.

• Two units with low coupling are weakly
connected and thus have low dependency on
each other.
• Hence, highly coupled units are heavily
dependent on other units and loosely coupled
units are comparatively less dependent on
other units as shown in Figure 8.3.

• Explain different strategies used for
integration testing

• A good design should have low coupling and thus
interfaces become very important.
• When interfaces are important, their testing will
also be important.
• In integration testing, we focus on the issues
related to interfaces amongst units.
• There are several integration strategies that
really have little basis in a rational methodology
and are given in Figure 8.4.
• Top down integration starts from the main unit
and keeps on adding all called units of the next
level.
• This portion should be tested thoroughly by
focusing on interface issues.

After completion of integration testing at this
level, add the next level of units and so on till we
reach the lowest level units (leaf units).
In this approach testing is conducted from main
module to sub module.
If the sub module is not developed a temporary
program called STUB is used for simulate the
submodule.

• There will not be any requirement of drivers
and only stubs will be designed.
• In bottom-up integration, we start from the
bottom, (i.e. from leaf units) and keep on
adding upper level units till we reach the top
(i.e. root node).
• There will not be any need of stubs.
• In this approach testing is conducted from sub
module to main module,if main module is not
developed a temporary program called
DRIVERS is used to simulate the main module.

• A sandwich strategy runs from top and
bottom concurrently, depending upon the
availability of units and may meet somewhere
in the middle.

• Each approach has its own advantages and
disadvantages.
• In practice, sandwich integration approach is
more popular.
• This can be started as and when two related units
are available.
• We may use any functional or structural testing
techniques to design test cases.
• Functional testing techniques are easy to
implement with a particular focus on the
interfaces and some structural testing techniques
may also be used.
• When a new unit is added as a part of integration
testing, then the software is considered as a
changed software.

• New paths are designed and new input(s) and
output(s) conditions may emerge and new
control logic may be invoked.
• These changes may also cause problems with
units that previously worked flawlessly.

System Testing
We perform system testing after the completion of
unit and integration testing.
We test complete software along with its expected
environment.
We generally use functional testing techniques,
although a few structural testing techniques may
also be used.
A system is defined as a combination of the
software, hardware and other associated parts that
together provide product features and solutions.
System testing ensures that each system function
works as expected and it also tests for non-
functional requirements like performance, security,
reliability, stress, load, etc.

This is the only phase of testing which tests both
functional and non-functional requirements of the
system.
A team of the testing persons does the system
testing under the supervision of a test team leader.
We also review all associated documents and
manuals of the software.
This verification activity is equally important and
may improve the quality of the final product.
Utmost care should be taken for the defects found
during the system testing phase.
A proper impact analysis should be done before
fixing the defect.

• Sometimes, if the system permits, instead of
fixing the defects, they are just documented
and mentioned as the known limitations.
• This may happen in a situation when fixing is
very time consuming or technically it is not
possible in the present design, etc.
• Progress of system testing also builds
confidence in the development team as this is
the first phase in which the complete product
is tested with a specific focus on the
customer’s expectations.
• After the completion of this phase, customers
are invited to test the software.

Acceptance Testing
• This is the extension of system testing.
• When the testing team feels that the product is ready
for the customer(s), they invite the customer(s) for
demonstration.
• After demonstration of the product, customer(s) may
like to use the product to assess their satisfaction and
confidence.
• This may range from adhoc usage to systematic well-
planned usage of the product.
• This type of usage is essential before accepting the
final product.
• The testing done for the purpose of accepting a
product is known as acceptance testing.
• This may be carried out by the customer(s) or persons
authorized by the customer(s).

• The venue may be the developer’s site or the
customer’s site depending on mutual
agreement.
• Generally, acceptance testing is carried out at
the customer’s site.
• Acceptance testing is carried out only when
the software is developed for a particular
customer(s).
• If we develop ‘standardised’ software for
anonymous users at large (like operating
systems, compilers, case tools, etc.), then
acceptance testing is not feasible.

DEBUGGING
• Whenever a software fails, we would like to
understand the reason(s) for such a failure.
• After knowing the reason(s), we may attempt to find
the solution and may make necessary changes in the
source code accordingly.
• These changes will hopefully remove the reason(s) for
that software failure.
• The process of identifying and correcting a software
error is known as debugging.
• It starts after receiving a failure report and completes
after ensuring that all corrections have been rightly
placed and the software does not fail with the same
set of input(s).
• The debugging is quite a difficult phase and may
become one of the reasons for the software delays.

Why Debugging is so Difficult?
Debugging is a difficult process.
This is probably due to human involvement and
their psychology.
Developers become uncomfortable after
receiving any request of debugging.
It is taken against their professional pride.
Shneiderman [SHNE80] has rightly commented
on the human aspect of debugging as:

“It is one of the most frustrating parts of
programming.
It has elements of problem solving or brain
teasers, coupled with the annoying recognition
that we have made a mistake.
Heightened anxiety and the unwillingness to
accept the possibility of errors, increase the
task difficulty.
Fortunately, there is a great sigh of relief and a
lessening of tension when the bug is ultimately

Debugging Process
• Debugging means detecting and removing
bugs from the programs.
• Whenever a program generates an
unexpected behavior, it is known as a failure
of the program.
• This failure may be mild, annoying, disturbing,
serious, extreme, catastrophic or infectious.
Depending on the type of failure, actions are
required to be taken.
• The debugging process starts after receiving a
failure report either from the testing team or
from users.

• The steps of the debugging process are
replication of the bug, understanding the
bug, locating the bug, fixing the bug and
retesting the program.
• These steps are explained below:

1. Replication of the bug:
• (i) The first step in fixing a bug is to replicate it.
• This means to recreate the undesired behaviour
under controlled conditions.
• The same set of input(s) should be given under
similar conditions to the program and the
program after execution, should produce a similar
unexpected behavior.
• If this happens, we are able to replicate a bug.
• In many cases, this is simple and straight forward.
• We execute the program on a particular input(s)
or we press a particular button on a particular
dialog, and the bug occurs.
• In other cases, replication may be very difficult.

• It may require many steps or in an interactive
program such as a game, it may require precise
timing. In worst cases, replication may be nearly
impossible.
• If we do not replicate the bug, how will we verify
the fix? Hence, failure to replicate a bug is a real
problem.
• If we cannot do it, any action, which cannot be
verified, has no meaning, howsoever important it
may be.
• Some of the reasons for non-replication of a bug
are:

1.The user incorrectly reported the problem.
2.The program has failed due to hardware problems
like memory overflow, poor network connectivity,
network congestion, non-availability of system
buses, deadlock conditions, etc.
3.The program has failed due to system software
problems. The reason may be the usage of a
different type of operating system, compilers,
device drivers, etc.
4.There may be any of the above-mentioned
reasons for the failure of the program, although
there is no inherent bug in the program for this
particular failure.

• Our effort should be to replicate the bug.
• If we cannot do so, it is advisable to keep the
matter pending till we are able to replicate it.
• There is no point in playing with the source
code for a situation which is not reproducible.

2. Understanding the bug
After replicating the bug, we may like to understand the
bug.
This means, we want to find the reason(s) for this failure.
There may be one or more reasons and is generally the
most time consuming activity.
We should understand the program very clearly for
understanding a bug.
If we are the designers and source code writers, there
may not be any problem for understanding the bug.
If not, then we may have serious problems.
If readability of the program is good and associated
documents are available, we may be able to manage
the problem.
If readability is not that good, (which happens in many
situations) and associated documents are not proper
and complete, the situation becomes very difficult and
complex.

• The worst cases are large programs written by many
persons over many years.
• These programs may not have consistency and may
become poorly readable over time due to various
maintenance activities.
• A debugger may be helpful for understanding the
program.
• A debugger inspects a program statement-wise and
may be able to show the dynamic behaviour of the
program using a breakpoint.
• The breakpoints are used to pause the program at
any time needed.

3. Locate the bug
We should first find the source code which causes the
incorrect behavior.
After knowing the incorrect behavior and its related
portion of the source code, we may find the portion
of the source code which is at fault.
Sometimes, it may be very easy to identify the
problematic source code (the second portion of the
source code) with manual inspection.
• Sometimes simple print statements may help us to
locate the sources of the bad behaviour.
• This simple way provides us the status of various
variables at different locations of the program with a
specific set of inputs.

4. Fix the bug and re-test the program
• After locating the bug, we may like to fix the bug.
• The fixing of a bug is a programming exercise
rather than a debugging activity.
• After making necessary changes in the source
code, we may have to re-test the source code in
order to ensure that the corrections have been
rightly done at right place.
• Every change may affect other portions of the
source code too.
• Hence an impact analysis is required to identify
the affected portion and that portion should also
be re-tested thoroughly.
• This re-testing activity is called regression testing
which is a very important activity of any
debugging process

Debugging Approaches
• There are many popular debugging
approaches, but success of any approach is
dependent upon the understanding of the
program.
• If the persons involved in debugging
understand the program correctly, they may
be able to detect and remove the bugs.

1. Trial and Error Method
(i) This approach is dependent on the ability
and experience of the debugging persons.
• After getting a failure report, it is analyzed
and the program is inspected.
• Based on experience and intelligence, and
also using the ‘hit and trial’ technique, the
bug is located and a solution is found.
• This is a slow approach and becomes
impractical in large programs.

Example
A calculator software is designed only to do
arithmetic operations
(addition,subtration,division and
multiplication) But the software produces a
bug when 5 is pressed.
Example : 5+2=10 5+3=11
which means 5 is performing functionality of 8
which is a bug.
So by trial and error method we can correct this
type of error but lot of time is wasted.
This approach will not work in large programs.

2. Backtracking
(ii) This can be used successfully in small programs.
• We start at the point where the program gives an
incorrect result such as an unexpected output is
printed.
• After analyzing the output, we trace backward the
source code manually until a cause of the failure is
found.
• The source code, from the statement where
symptoms of the failure is found, to the statement
where the cause of failure is found, is analyzed
properly.
• This technique brackets the locations of the bug in
the program.

Example
WAP to divide two numbers
int main().....................................................1
{....................................................................2
int a=4,b=0,div=0;........................................3
div=a/b; ……………………………….. ...4
}.....................................................................5
OUTPUT
Divide by zero excpetion

By using backtracking form line number 4,we
find that error exists at line number 3.
by mistake b is initialized to 0 which causes
divided by zero exception.
So we can correct it by assigning a valid value in
b.
Example : b=2

3. Brute Force
• This is the foremost common technique of
debugging however is that the least economical
method.
• During this approach, the program is loaded
with print statements to print the
intermediate values with the hope that a
number of the written values can facilitate to
spot the statement in error.
• This approach becomes a lot of systematic with
the utilization of a symbolic program (also known
as a source code debugger), as a result of
values of various variables will be simply
checked and breakpoints and watchpoints can
be easily set to check the values of variables
effortlessly.

Example
int main().....................................................1
{....................................................................2
int a=4,b=0,div=0;........................................3
printf(“a = %d b = %d”,a,b);
div=a/b; ……………………………….. ...4
printf(“%d”,div);
}.....................................................................5
OUTPUT
a=4 b=0
Divide by zero excpetion

4. Cause Elimination
(iv) Data related to error occurrence are organized
to isolate potential causes.
Alternatively, a list of all possible causes is
developed and tests are conducted to eliminate
each.
The cause is identified, properly fixed and re-tested
accordingly.

Example
int main()
{
int a,b,div;
printf(“Enter Value of a and b”);
Scanf(“%d%d”,&a,&b);
div=a/b;
}

Causes of Error
Test Case :
• a=10 b=2 output = 5
• a=5 b = 0 output = divide by zero exception
So with the help of print statement we can
eliminate the error

Modified Example
int main().....................................................1
{....................................................................2
int a,b,div;...................................................3
printf(“Enter Value of a and b value of b can’t be
0”);
Scanf(“%d%d”,&a,&b);
div=a/b; ……………………………….. ...4
}.....................................................................5

WHAT IS REGRESSION TESTING?
Regression Testing is a type of testing that is
done to verify that a code change in the
software does not impact the existing
functionality of the product.
This is to make sure the product works fine
with new functionality, bug fixes or any
change in the existing feature.
Previously executed test cases are re-
executed in order to verify the impact of
change.

• Regression Testing is a Software Testing type
in which test cases are re-executed in order
to check whether the previous functionality
of the application is working fine and the
new changes have not introduced any new
bugs.
• This test can be performed on a new build
when there is a significant change in the
original functionality that too even in a single
bug fix.

It serves several purposes like:
● Increases confidence in the correctness of the
modified program.
● Locates errors in the modified program.
● Preserves the quality and reliability of the
software.
● Ensures the software’s continued operation

Regression Testing Process
• Regression testing is a very costly process and
consumes a significant amount of resources.
The question is “how to reduce this cost?”
Whenever a failure is experienced, it is
reported to the software team.
• The team may like to debug the source code
to know the reason(s) for this failure.

• After identification of the reason(s), the
source code is modified and we generally do
not expect the same failure again.
• In order to ensure this correctness, we re-test
the source code with a focus on modified
portion(s) of the source code and also on
affected portion(s) of the source code due to
modifications.
• the existing test cases which were designed
for development testing and some of them
might have been used during development
testing.

• The existing test suite may be useful and may
reduce the cost of regression testing.
• As we all know, the size of the existing test
suite may be very large and it may not be
possible to execute all tests.
• The greatest challenge is to reduce the size of
the existing test suite for a particular failure.
• The various steps are shown in Figure 7.1.
Hence, test case selection for a failure is the
main key for regression testing.

Selection of Test Cases
• We want to use the existing test suite for regression
testing.
• How should we select an appropriate number of test
cases for a failure?
• The range is from “one test case” to “all test cases”.
• A ‘regression test cases’ selection technique may help
us to do this selection process.
• The effectiveness of the selection technique may
decide the selection of the most appropriate test
cases from the test suite.

• We consider a program given in Figure 7.2
along with its modified version where the
modification is in line 6 (replacing operator ‘*’
by ‘-‘).
• A test suite is also given in Table 7.2.

In this case, the modified line is line number 6 where ‘a*b’ is
replaced by ‘a-b’.
All four test cases of the test suite execute this modified line 6.
We may decide to execute all four tests for the modified
program.
If we do so, test case 2 with inputs a = 1 and b = 1 will
experience a ‘divide by zero’ problem, whereas others will not.
However, we may like to reduce the number of test cases for
the modified program.

• We may select all test cases which are executing the
modified line. Here, line number 6 is modified.
• All four test cases are executing the modified line (line
number 6) and hence are selected.
• There is no reduction in terms of the number of test cases.
• If we see the execution history, we find that test case 1 and
test case 2 have the same execution history.
• Similarly, test case 3 and test case 4 have the same
execution history.
• We choose any one test case of the same execution history
to avoid repetition.
• For execution history 1 (i.e. 1, 2, 3, 4, 5, 6, 7, 8, 10, 11), if we
select test case 1, the program will execute well, but if we
select test case 2, the program will experience ‘divide by
zero’ problem.

• Therefore, either test case 1 or test case 2 may have
to be selected.
• If we select test case 1, we miss the opportunity to
detect the fault that test case 2 detects.
• Minimization techniques may omit some test cases
that might expose fault(s) in the modified program.
• Hence, we should be very careful in the process of
minimization of test cases and always try to use safe
regression test selection technique (if at all it is
possible).
• A safe regression test selection technique should
select all test cases that can expose faults in the
modified program.

REGRESSION TEST CASES SELECTION
• Test suite design is an expensive process and
its size can grow quite large.
• Most of the times, running an entire test suite
is not possible as it requires a significant
amount of time to run all test cases.
• Many techniques are available for the
selection of test cases for the purpose of
regression testing

1. Select All Test Cases
• This is the simplest technique where we do not
want to take any risk.
• We want to run all test cases for any change in the
program.
• This is the safest technique, without any risk.
• A program may fail many times and every time we
will execute the entire test suite.
• This technique is practical only when the size of
the test suite is small.
• For any reasonable or large sized test suite, it
becomes impractical to execute all test cases.

2. Select Test Cases Randomly
• We may select test cases randomly to reduce the size
of the test suite.
• We decide how many test cases are required to be
selected depending upon time and available resources.
• When we decide the number, the same number of test
cases is selected randomly.
• If the number is large, we may get a good number of
test cases for execution and testing may be of some
use.
• But, if the number is small, testing may not be useful
at all.

Example
From previous example :
Test cases: 2 1 and 11
If we select test case 2 1, we miss the opportunity to
detect the fault that test case 2 detects which is divide by
zero exception.

3. Select Modification Traversing Test
Cases
• We select only those test cases that execute the
modified portion of the program and the portion
which is affected by the modification(s).
• Other test cases of the test suite are discarded.
Actually, we want to select all those test cases that
reveal faults in the modified program.
• These test cases are known as fault revealing test
cases.
• There is no effective technique by which we can find
fault revealing test cases for the modified program.
• This is the best selection approach, which we want,
but we do not have techniques for the same.
• Another lower objective

• may be to select those test cases that reveal the
difference in the output of the original program and
the modified program.
• These test cases are known as modification revealing
test cases.
• These test cases target that portion of the source
code which makes the output of the original program
and the modified program differ.
• Unfortunately, we do not have any effective technique
to do this. Therefore, it is difficult to find fault
revealing test cases and modification revealing test
cases.
• The reasonable objective is to select all those test
cases that traverse the modified source code and the
source code affected by modification(s).

Example
From previous example :
Test cases: 2 1 and 11
If we select test case 1 1, we can detect the fault which is
divide by zero exception.
But it is very difficult to choose only right test case
(specially when test cases are very large )which will give
the exception is very difficult.

MUTATION TESTING
• It is a popular technique to assess the
effectiveness of a test suite.
• We may have a large number of test cases for any
program.
• We neither have time nor resources to execute all
of them. We may select a few test cases using any
testing technique and prepare a test suite.
• How do we assess the effectiveness of a
selected test suite? Is this test suite adequate for
the program?
• If the test suite is not able to make the program
fail, there may be one of the following reasons:

1. The test suite is effective but hardly any
errors are there in the program. How will a
test when they are not there?
2. The test suite is not effective and could not
find any errors. Although there may be
errors, they could not be detected due to
poor selection of test suite. How will errors
be detected when the test suite is not
effective?
mutation testing may help us to assess the
effectiveness of a test suite and may also
enhance the test suite, if it is not adequate for a
program.

Mutation and Mutants
• The process of changing a program is known as
mutation.
• This change may be limited to one, two or very
few changes in the program.
• We prepare a copy of the program under test
and make a change in a statement of the
program.
• This changed version of the program is known as
a mutant of the original program.
• The behaviour of the mutant may be different
from the original program due to the introduction
of a change.
• However, the original program and mutant are
syntactically correct and should compile correctly.

• To mutate a program means to change aprogram.
• We generally make only one or two changes in
order to assess the effectiveness of the selected
test suite.
• We may make many mutants of a program by
making small changes in the program.
• Every mutant will have a different change in a
program.
• Consider a program to find the largest amongst
three numbers as given in Figure 3.11 and its two
mutants are given in Figure 4.7 and Figure 4.8.
• Every change of a program may give a different
output as compared to the original program.

• Many changes can be made in the program
given in Figure 3.11 till it is syntactically
correct.
• Mutant M1 is obtained by replacing the
operator ‘>’ of line number 11 by the operator
• ‘ =’. Mutant M2 is obtained by changing the
operator ‘>’ of line number 20 to operator ‘<’.
• These changes are simple changes.
• Only one change has been made in the
original program to obtain mutant M1 and
mutant M2.

• The mutants generated by making only one
change are known as first order mutants. We may
• obtain second order mutants by making two
simple changes in the program and third order
mutants by making three simple changes, and so
on.
• The second order mutant (M3) of the program
given in
• Figure 3.11 is obtained by making two changes in
the program and thus changing operator ‘>’ of
• line number 11 to operator ‘<’ and operator ‘>’ of
line number 20 to ‘ ’ as given in Figure 4.9.
• The second order mutants and above are called
higher order mutants.
• Generally, in practice, we prefer to use only first
order mutants in order to simplify the process of
mutation.

Mutation Operators
• Mutants are produced by applying mutant operators.
• An operator is essentially a grammatical rule that
changes a single expression to another expression.
• The changed expression should be grammatically
correct as per the used language.
• If the original expression is x + 1, and the mutant for
that expression is x + 2, that is considered as a lesser
change as compared to a mutant where the changed
expression is (y * 2) by changing both operands and
the operator.
• We may have a ranking scheme, where a first order
mutant is a single change to an expression, a second
order mutant is a mutation to a first order mutant, and
so on.
• Higher order mutants become difficult to manage,
control and trace.

Software Testing Introduction (Part 3)

More Related Content

Similar to Software Testing Introduction (Part 3) (20)

More from Thapar Institute (19)

Recently uploaded (20)

Software Testing Introduction (Part 3)