A MUTATION TESTING ANALYSIS AND REGRESSION TESTING

International Journal in Foundations of Computer Science & Technology (IJFCST), Vol.5, No.3, May 2015
DOI:10.5121/ijfcst.2015.5306 71
A MUTATION TESTING ANALYSIS AND REGRESSION
TESTING
Amit Singh1
and Khushbu babbar2
Department of Computer Engineering, Bhagwant University, Ajmer, Rajasthan
ABSTRACT
Software testing is a testing which conducted a test to provide information to client about the quality of the
product under test. Software testing can also provide an objective, independent view of the software to
allow the business to appreciate and understand the risks of software implementation. In this paper we
focused on two main software testing –mutation testing and mutation testing. Mutation testing is a
procedural testing method, i.e. we use the structure of the code to guide the test program, A mutation is a
little change in a program. Such changes are applied to model low level defects that obtain in the process
of coding systems. Ideally mutations should model low-level defect creation. Mutation testing is a process
of testing in which code is modified then mutated code is tested against test suites. The mutations used in
source code are planned to include in common programming errors. A good unit test typically detects the
program mutations and fails automatically. Mutation testing is used on many different platforms, including
Java, C++, C# and Ruby. Regression testing is a type of software testing that seeks to uncover
new software bugs, or regressions, in existing functional and non-functional areas of a system after
changes such as enhancements, patches or configuration changes, have been made to them. When defects
are found during testing, the defect got fixed and that part of the software started working as needed. But
there may be a case that the defects that fixed have introduced or uncovered a different defect in the
software. The way to detect these unexpected bugs and to fix them used regression testing. The main focus
of regression testing is to verify that changes in the software or program have not made any adverse side
effects and that the software still meets its need. Regression tests are done when there are any changes
made on software, because of modified functions.
KEYWORDS
Increment model, waterfall model, Software engineering, Software process model and Software.
1. INTRODUCTION
Software testing is the process of executing software in a controlled manner. When the final
product is delivered to the client it must be worked according to the requirements and needs of the
client. The term defect in software is the changes between the actual and expected results. There
are many different types of testing process, which and when conducted helps to eliminate the
defects in the program. Testing is the process of gathering information by making the
observations and comparing them to expectations. In our daily routine life, when we go out, for
shopping any product such as vegetable, clothes, pens, etc., we check them before purchasing that
its meet our requirements for our satisfaction and to get maximum gain. For example, when we
need to buy a mobile phone, we test the phone before actually purchasing it, i.e. it must be
worked in any condition. Testing is most important before using any software, hardware or any
other product. Testing not only used to fix the error in the code, but also to check whether the
program is work according to the given requirements and testing procedure. Software testing is a
testing which conducted a test to provide information to client about the quality of the product
under test. Software testing can also provide an objective, independent view of the software to

72
allow the business to appreciate and understand the risks of software implementation. The main
focus of software testing is to detect bugs in order to uncover and fix it. The scope of testing is to
execute code in various conditions and also code work as we aspects and fulfil all the needs and
produce output according to specifications. Testing must be done from the initial stages of the
software development.
Software testing not only detect an error it operate the controlled condition of the software to (1)
verify that it work correctly (2) to find an errors and (3) to validate the needs and requirements of
user and the system work accordingly.
1. Verifications the checking the software consistency by checking the results against the
requirements as already specified.
2. Error Detection: Testing should intentionally attempt to make things go wrong to determine if
things happen when they shouldn’t or things don’t happen when they should.
3. Validation checks the system correctness –i.e. the process of checking and fulfilling the needs
of the client.
A good test case is one that find the uncovered defects and fix them out. A successful test is one
that uncovers an error which not find yet. A good test should neither be too simple nor too
complex. To check if the system does what it is expected to do. Testing is done to check whether
the application meets the requirements and executed in any conditions.
This paper focused on two main types of software testing-Mutation Testing and Regression
Testing. In Section -2 and Section -3 summarizes mutation testing & regression testing and its
algorithm with their advantages and disadvantages, final discussion is concluded in Section-4.
2. MUTATION TESTING
Mutation testing is the process of changing, modifying or altering a program code, later retesting
a suit of correct unit tests against the mutated program. With the increase in demand for software,
increases the complexity of the design. As the complexity of software increases, need of testing
increases and customer satisfaction also. Mutation testing is a testing to check w the software
tester done the testing in a correct manner so that software works according to the needs of user.

73
Through mutation testing, we can find whether the set of testing methods used in the development
process were good and correct to ensure final product quality. Correctness of the testing schedule
can be determined by checking the behavior for each test conditions. If there are no defects
detected, then the program is appropriate or it passes the test. For testing the program we take set
of input values as a test condition so that it works according to needs and the corresponding
output values according to the specifications. Mutation testing is the test to check the consistency
and accuracy of a testing program to find the bugs in the program by using different test
conditions. Mutation testing is a fault-based testing strategy because the faults are created by
introducing a single error into the original program. There are many versions of mutation testing
like weak mutation, interface mutation and specification based mutation. The method was first
introduced by Richard Lipton in 1971, and there has been a huge interest since that
time. Mutation testing's working method is simple to use. In mutation testing we replace a logical
operator with its inverse. For example, operator "! =" is used in place of "= =." In some cases,
mutation involves rearranging lines to change the execution order or even deleting a few lines of
code. If the program is modified then test suite are executed against the code. The mutated code
passes or fails the unit test, depending on the testing condition. Then a written unit test must
detect mutated code errors, resulting in failure. A unit test that fails to detect code errors may
require a rewrite.
2.1 Mutation Testing Example
Mutation testing is the test in which operators get replaced with their reverse in the source code.
The result of applying one mutation operator to the program is called a mutant. If the test suite is
able to detect the change (i.e. one of the tests fails), then the mutant is said to be killed.
Let’s take example to find a maximum between two numbers:
function MAX(M<N:INTEGER)
return INTEGER is
begin
if M>N then
return M;
else
return N;
end if:
end MAX;
2.1.1 First test data set--M=1, N=2
1. The original function returns 2
2. Five mutants: replace”>“ operator in if statements by (>,<,<=or=)
3. Executing each mutant:
4. Adding test data M=2, N=1 will eliminate the latter live mutant, but the former live mutant
remains live because it is equivalent to the original function. No test data can eliminate it.
Mutants Outputs Comparison
if M>=N then 2 live
if M<N then 1 dead
if M<=N then 1 dead
if M=N then 2 live
if M< >N then 1 dead

74
2.2 Mutation Testing Algorithm
The following steps rely on the traditional mutation process:
1. Generate program test part.
2. Test part should run against the original program.
a. If the output is incorrect, the program must be changed and re-tested.
b. If the output is correct go to the next step ...
3. Construct mutants using a tool like Mothra.
4. Mutation testing will empower software testing process with trust and fidelity.
5. Each test part execute against each alive mutant.
a. If the output of the mutant differs from the output of the original program, the
mutant is considered incorrect and is killed.
6. Two kinds of mutants survive:
a. Functionally equivalent to the original program: Cannot be killed.
b. Killable: Test parts are insufficient to kill the mutant. New test cases must be
created.
7. The mutation score for a set of test cases is the percentage of non-equivalent mutants
killed by the test data.
8. Mutation Score = 100 * D / (N - E)
a. D = Dead mutants
b. N = Number of mutants
c. E = Number of equivalent mutants
9. The mutation score is 100%, if set of test parts of mutation is correct
Fig 2. Process of Mutation Testing
2.3 Advantages and disadvantages
2.3.1 Advantages:
Program code fault identification
1.Effective test case development
2.Detection of loopholes in test data
3.Improved software program quality
4.Elimination of code ambiguity

75
2.3.2 Disadvantages
1.Difficult implementation of complex mutations
2.Expensive and time-consuming
3.Requires skilled testers with programming knowledge
2.4. MUTATION OPERATORS
A variety of mutation operators were explored by researchers. Mutation operators examples used
for testing:
1. Statement deletion.
2. Change each Boolean expression with true and false.
3. Change each arithmetic operator with any other + with *, - and /.
4. Change each Boolean relation with any other > with >=, == and <=.
5. Change each variable with another variable declared in the same part (variable types
should be the same).
3. REGRESSION TESTING
Regression means errors that occur due to some action or activities in a system. In IT sector a
“regression” means the return of an error or bug. Regression testing is a type of testing that
emphasis on retesting at every change is made. Traditional regression tests are often partially
automated. The numbers of regression testing have been developed for the optimization. The
studies of the techniques suggest that regression testing can provide benefit to testers. Regression
testing attempts to mitigate two risks:
• Changes that was planned to fix an error failed.
• Sometime modification made some problems like introducing a new bug
Regression testing is the testing of test cases to check whether the old function of application is
working correctly and modification have not made any new bugs. This test can be applied on new
techniques when there is an important change in original functions or even a single bug fix.
Regression is the process of verification. It verifies that errors are fixed and the newly added
functions will not create any problem in previous working program or software. Testers perform
functional testing when new part of program is available for verification. The main focus of this
test is to verify the changes made in the existing functions of program and new added functions.
When test gets completed the software tester verify if the existing functions of previous software
is working as expected and new changes have not introduced any defect or bugs in function as
that was working before this changes. Regression testing is usually performed after verification of
changes or new functionality. The release of regression tests takes months to complete, regression
testing must be applied in the daily test cycle. Regression testing is applied when tester fix any
error or add new code for any new functions to the software. Regression testing compiles new
code with the previous code to provide the quality to the client and also it does not affect the
previous code. Testing team also check the changes done in last minutes in the system. In that
condition testing only affected application area in necessary condition to complete the testing
process on time with taking all major system changes and aspects.
3.1 Regression Testing Example
Example of regression testing with its process is explained below:
Let’s take example of college in which there are three main parts named Administration part,
Personal Information, and Faculty. Suppose error occurs in the Administration part, existing User

76
is not able to login with valid login details then error get occurred. Then Testing team pass the
bugs to the Development team to correct it and when development team fixes the Bug and again
give it to testing team than they checks that bugs get fixed and does not change the remaining
functions of other parts and also the functions of the same part. This process is known
as regression testing done by Software Testers.
1. Regression testing is required when there is an error in program.
2. Change in requirements and code is modified.
3. New feature is added to the software.
4. Bugs get removed and Defect gets fixed.
3.2 Regression Testing Algorithm
Let’s take P be a procedure or program, let P' be a improved version of P and let T be a test suite
for P. Regression testing consists of reusing T the new test cases are needed to effectively test
code or new functions added to or changed in producing A typical regression test proceeds as
follows:
1. Select T’ subset of T, a set of test cases to execute on P'.
2. Test P' with T’. Establish P’ correctness with respect to T’.
3. If needed, let’s take T" which is a set of new function for P'.
4. Test P' with T", establish to P”s correctness with respect to T”.
5. Create T’”, a new test suit and test history for P’, from T, T’,T”
Fig.3- Regression Testing process
4. CONCLUSIONS
Software testing is a method of executing a program for finding the software bugs or errors. There
are many types of software testing but in this paper we focused on two main type of testing and
the working of Mutation testing and Regression Testing. Mutation testing is a process of
producing small errors in the program to check that test suite picks them up. If test suite doesn't
pick up the any error then it’s lacking at somewhere then we have to add more test cases. In other
words, mutation testing is the testing which tests the test suite rather than application. Regression
testing is a test suite that is supposed to test as much function of program as possible. The idea is
to make a changes in program as required for fixing bug or for new functions, and regression

77
testing will try to catch any problems (or regressions) with any changes this is called regression
since the vast majority of tests have been added due to old bugs and if any problem found, we
have regressed to a old state (in which the problem once again exists).In other words, regression
testing tests the program.
REFERENCES
[1] A. S. Namin, J. H. Andrews, and D. J. Murdoch(2008), Sufficient mutation operators for measuring
test effectiveness. In Proc. ICSE, pages 351–360.
[2] A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G.Sayward(1979).” Mutation analysis”.
Technical report,G eorgia Institute of Technology.
[3] D. Schuler and A. Zeller(2009) “Javalanche: Efficient mutation testing for Java” In Proc. FSE, pages
297–298.
[4] J. H. Andrews, L. C. Briand, and Y. Labiche(2005).”Is mutation an appropriate tool for testing
experiments?” In Proc. ISE, pages 402–411.
[5] L. Zhang, S. S. Hou, J. J. Hu, T. Xie, and H. Mei(2010) ” Is operator-based mutant selection superior
to random mutant selection?” In Proc. ICSE, pages 435–444.
[6] M. B. Dwyer, and G. Rothermel.(2009)” Regression model checking.” In Proc. ICSM, pages 115–
124.
[7] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings,S. Sinha, S. A. Spoon, and A.
Gujarathi (2001) “Regression test selection for java software.” In Proc. OOPSLA, pages 312–326.
[8] M. Woodward and K. Halewood.(1988) “From weak to strong, dead oralive? an analysis of some
mutation testing issues”. In Proc. of the Second Workshop on Software Testing, Verification, and
Analysis,pages 152–158.
[9] P.Ammann and J.Offutt.”Introduction to Software Testing.” Cambridge University Press.
[10] P. E. Black, V. Okun, and Y. Yesha(2001), “Mutation of Model Checker Specifications for Test
Generation and Evaluation,” in Proceedings of the 1st Workshop on Mutation Analysis
(MUTATION’00), published in book form, as Mutation Testing for the New Century. San
Jose,California, pp. 14–20.
[11] P. G. Frankl, S. N. Weiss, and C. Hu(1997) “All-uses vs mutation testing:An experimental
comparison of effectiveness” JSS, 38(3):235–253.
[12] R. Abraham and M. Erwig(2009), “Mutation Operators for Spreadsheets,”IEEE Transactions on
Software engineering, vol. 35, no. 1, pp. 94–10.
[13] R. H. Carver(1993),“Mutation-Based Testing of Concurrent Programs,” in Proceedings of the IEEE
International Test Conference on Designing, Testing, and Diagnostics, Baltimore, Maryland, 17-
21,pp. 845–853.
[14] W. E. Howde(1982)” Weak mutation testing and completeness of test sets”.IEEE TSE, pages 371–
379.

A MUTATION TESTING ANALYSIS AND REGRESSION TESTING

More Related Content

Similar to A MUTATION TESTING ANALYSIS AND REGRESSION TESTING (20)

More from ijfcstjournal (20)

Recently uploaded (20)

A MUTATION TESTING ANALYSIS AND REGRESSION TESTING