![Problems of Search Based Software Testing 8
Usage of a single guidance Works on isolated functions
Coarse / Misleading Modern systems are object-oriented
void foo(int []a) {
int flag = 0;
for(int i=0; i<10; i++)
if(a[i] == 23)
flag = 1;
if(flag == 1) {
// target
}
}
DEI](https://image.slidesharecdn.com/dissertation-110223080040-phpapp02/85/Evolutionary-Testing-of-Stateful-Systems-a-Holistic-Approach-8-320.jpg)
![Some insights from our empirical evaluation… 22
«container classes are the de facto benchmark
for testing software with internal state»
[ Arcuri 2010 ]
DEI](https://image.slidesharecdn.com/dissertation-110223080040-phpapp02/85/Evolutionary-Testing-of-Stateful-Systems-a-Holistic-Approach-22-320.jpg)
Evolutionary Testing of Stateful Systems: a Holistic Approach
- 1. Evolutionary Testing of Stateful Systems: a Holistic Approach Matteo Miraz Advisor: prof. L. Baresi Febuary 18, 2011 Coadvisor: prof. P. L. Lanzi
- 2. Motivations 2 • Software systems permeate (almost) every aspect of our life • Software is buggy • In 2002 the costs related to software errors are estimated in 60 Billion USD 1999: NASA Mars Climate Orbiter ($125 million) 2011: iPhone Alarm DEI
- 3. Motivations 3 Testing is an effective technique to increase the quality of software Does not guarantee that the system is error-free Is extremely expensive, as it requires up to half of the entire development effort Stringent requirements on time-to-market limit the testing effort • We spoke with an important software consultant company, with major Italian banks as customers… • The good: they save up to half of the entire development effort • The bad: they do not test anything • The ugly: we had to explain them what testing is DEI
- 4. Related Work 4 • The research community proposed several ways to automatically generate tests • Symbolic Execution • Search-Based Software Testing DEI
- 5. Symbolic Execution 5 • The program is executed with symbolic values as input parameters (instead of concrete inputs) PC: a = 3 • Paths are associated with a PC: a = 3 & log10(b) = 7 Path Condition (PC) • A PC constraints symbolic inputs to traverse the path • It is created incrementally: at each condition a formula is PC: a = 3 & log10(b) = 7 added to the Path Condition Constraint solver • A constraint solver is used to find concrete inputs for PCs toTest(3, 10 000 000) DEI
- 6. Search Based Software Testing: a survey 6 • Goal: complete branch coverage • Paradigm “command and conquer” The program is analyzed to identify branches Each branch is considered separately from the others entry w = Math.log(b) if (a == 3) F T if (w == 7) F T // target exit DEI
- 7. Search Based Software Testing: a survey 7 Once a target has been selected • Identify dependent branches • Search for inputs able to reach the target • Fitness Function: Fitness(a = 0, b = 0) = • Approach Level + 1 + norm(| 0 – 3 |) = 1.003 Normalized distance entry w = Math.log(b) Parameters Fitness distance: | 0 – 3 | a=0, b=0 1.003 if (a == 3) Approach Level: 1 T a=1, b=0 1.002 F if (w == 7) a=3, b=0 0.999 Approach Level: 0 F T a=4, b=0 1.001 // target a=3, b=10 0.005 a=3, b=100 0.003 exit DEI
- 8. Problems of Search Based Software Testing 8 Usage of a single guidance Works on isolated functions Coarse / Misleading Modern systems are object-oriented void foo(int []a) { int flag = 0; for(int i=0; i<10; i++) if(a[i] == 23) flag = 1; if(flag == 1) { // target } } DEI
- 9. Targeting Stateful Systems 9 • We Target Stateful systems (e.g., Java classes) • Feature – State Loop • A feature might put objects in “particular ” states • A “particular” state might enables enable other features DEI
- 10. Our Approach: TestFul 10 An individual of the Evolutionary algorithm (a Test) is a sequence of constructor and method calls on the test cluster (i.e., the class under test + all the required dependencies) We use a multi-objective EA, guided by: • The coverage of tests • Tests with a high coverage stress more deeply the class under test and put objects in more interesting states • Compactness of tests • Unnecessary long tests waste computational resources during their evaluation DEI
- 11. Improvements 11 How can we improve How can we correctly the efficiency? reward tests? • We use efficiency • We use enhancement complementary techniques: coverage criteria: • Local Search • Control Flow Graph • Seeding • Data Flow Graph • Fitness Inheritance • Behavioral DEI
- 12. Local Search 12 • We hybridize the evolutionary algorithm with a step of local search • EA works at class level • Reaches complex state configurations • Local search works on methods • It focuses on the easiest missing element • It adopts a simpler search strategy (hill climbing) • Which element to improve? • One of the best elements • 5% of the population • 10% of the population DEI
- 13. Local Search 13 • At each generation of the Evolutionary Algorithm, we pick the { 5% / 10% / best } test and we target the easiest branch to reach among those not exercised yet target DEI
- 14. Local Search 14 • At each generation of the Evolutionary Algorithm, we pick the { 5% / 10% / best } test and we target the easiest branch to reach among those not exercised yet • We perform a local search by using a simple algorithm (hill climbing) • The guidance is provided by the following fitness function: • The result (if any) is merged in the evolutionary algorithm’s population DEI
- 15. Contribution of the Local Search 15 Simple Problem (Fraction) Complex Problem (Disjoint Set Fast) DEI
- 16. Seeding 16 • Problem: • Testful starts the evolution from a population with a poor quality. • Solution: • Run an inexpensive technique (random search) for a short time, and use its result as initial population DEI
- 17. Fitness Inheritance 17 • Problem: • Executing tests and collecting coverage is expensive • Solution: • Evaluate the “real” fitness only on a part of the population • Other individuals inherit the fitness of their parents • Which policy to select individuals to evaluate? • Uniform selection • Frontier selection: better tests are evaluate more often DEI
- 18. Improvements 18 How can we improve How can we correctly the efficiency? reward tests? • We use efficiency • We use enhancement complementary techniques: coverage criteria: • Local Search • Control Flow Graph • Seeding • Data Flow Graph • Fitness Inheritance • Behavioral DEI
- 19. Coverage of the Control-Flow Graph 19 Testful aims to maximize: The number of basic blocks executed (~ statement coverage) The number of branches exercised We compared our approach against jAutoTest: a Java Port of Bertrand Meyer’s Random Testing Approach Randoop: Michael Ernst’s taboo search etoc: Paolo Tonella’s Evolutionary Testing of Classes On a benchmark of classes from 10 min • Literature • Known software libraries • Independent testing benchmarks DEI
- 20. Coverage of the Data-Flow Graph 20 Fault-detection effectiveness of Statement and Branch coverage has been disputed • Criteria on the coverage of the Data-Flow Graph have been proposed • Fit well on Object-Oriented systems • Data dependency • Statements might define the value of variable (e.g., v = 10 ) • Statements might use the value of some variables (e.g. print(v) ) • P-Use if the use happens in a predicate (e.g. if(v == 3) ) • TestFul leverages Data-Flow information Extend the fitness function with all def-use and all def-puse coverage Improve Local Search and solve data-dependent problems (e.g., flag variable) • Tracking data-flow coverage is expensive! () DEI
- 21. Coverage of the Data-Flow Graph 21 Performed an extensive empirical evaluation between • Java Path Finder (symbolic execution) • Testful only guided by Control-Flow information The structural coverage remains high, in spite of the higher monitoring cost TestFul outperforms JPF (e.g., statement coverage: 89% vs. 35%) The data-flow coverage increases a little, and its standard deviation lower significantly DEI
- 22. Some insights from our empirical evaluation… 22 «container classes are the de facto benchmark for testing software with internal state» [ Arcuri 2010 ] DEI
- 23. Some insights from our empirical evaluation… 23 S. Mouchawrab, L. C. Briand, Y. Labiche, and M. Di Penta. Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments. IEEE Transactions on Software Engineering, 2010 DEI
- 24. Coverage of the Behavioral Model 24 White-Box coverage criteria judge tests by considering the covered code • If we execute the code that contains an error, we’ll likely spot it! • What if there is a high-level problem? (e.g., a feature is missing) Black-Box testing derives tests from the specification of the system • Adopts an outer perspective, is more resilient to high-level errors • Requires the specification of the system, often not available • We can both infer the behavioral model of the system, and reward tests according to their ability to thoroughly exercise the system DEI
- 25. Behavioral Coverage: Preliminary Results 25 DEI
- 26. Conclusions and Future Work 26 • Contributions: • Search-Based Software Testing • Holistic Approach • Efficiency Enhancement Techniques • Complementary Coverage Criteria • Extensive Empirical Validation • Most interesting research directions • Use more sources to seed the evolutionary algorithm • Consider coarse-grained tests (e.g. integration testing) • Consider other types of stateful systems (e.g., services) DEI
- 27. Thank you for the attention… Questions? DEI