Slides for a talk on search-based testing for Event-B models
Download as PPTX, PDF0 likes736 views
This document summarizes approaches for applying search-based testing techniques to generate tests from Event-B models. It discusses challenges like the lack of an explicit state space in Event-B. It presents two approaches - using genetic algorithms to generate test data for a given path, and using finite model learning techniques to approximate the Event-B model with a finite automaton. Finally, it outlines several opportunities for using search-based techniques like test suite minimization, test data generation, and combining ProB with metaheuristics like evolutionary algorithms.
![[Event-B in a Nutshell]](https://image.slidesharecdn.com/alin-stefanescu-slides-110520033942-phpapp02/85/Slides-for-a-talk-on-search-based-testing-for-Event-B-models-2-320.jpg)
Slides for a talk on search-based testing for Event-B models
- 1. 13th CREST Open Workshop12th-13th of May 2011, LondonSearch-Based Software Engineering for Model-Based TestingEvent-B in a NutshellTest Generation ApproachesSBT ChallengesTest Data GenerationFinite Model LearningConclusionsAlin Stefanescu - University of Pitesti, Romania*) joint work with colleagues from DEPLOY project
- 2. [Event-B in a Nutshell]
- 4. Inventor of the Z and B formal methods.
- 5. Z – developed in the 70s
- 6. B – developed in the 90s, successfully deployed in industry
- 7. Event-B – born with the 21st century
- 8. Evolution of B for system level specification
- 9. Developement supported by French and European projects: FP6 RODIN and FP7 DEPLOY
- 10. DEPLOY project (2008-2012) – funded by FP7 DEPLOY :: Industrial deployment of advanced systemengineering methods for high productivity and dependability using formal methods 4 industrial partners
- 11. Bosch, Siemens, SAP, SSF
- 12. 3 industrial service providers
- 15. Newcastle, Aabo, Düsseldorf, ETH Zurich,
- 17. Rodin platform for Event-BExtension of Eclipse IDE (Java-based)
- 18. Theorem proving as core technology
- 19. Many other Rodin Plug-ins
- 20. ProB:animation, consistency and model-checking
- 22. Decomposition
- 23. Modularisation
- 24. Team-work
- 25. Code generation
- 26. UML-B
- 27. etc.Event-B in a nutshellEvent-B modelState-transition model(like ASM, B, Z)
- 28. set theory as mathematical language
- 29. refinement as basic modeling approach
- 30. Contexts
- 32. constants
- 33. axioms
- 34. Machines
- 35. global variables
- 36. invariants
- 37. events that update the variables
- 38. Events
- 39. local parameters
- 40. guards
- 41. actionsITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})items : Powerset(ITEMS)
- 43. Test generation based on Event-BModel-Based Testing (MBT)Model-based testing (MBT) is a newlyintroduced topic in DEPLOYpriority topic for industrial partners like SAPchallenges due to the sheer size of the state space of real-life scenariosWeinvestigate search-based testing (SBT) techniques for Event-B.
- 44. Future MBT plugin in RODINEvent-B modelMBT Plug-inMBT Users{ Model-checking }{ Constraint-based }Extra test information{ Search-based }Test casesUniversity of Pitesti and University of DusseldorfTool developers:
- 45. Test generation from Event-BSBT Opportunity! 1. Generate a set of tests(sequence of events with concrete param.)2. Optimize test suite(according to some criteria) – if still neededEvent-B modelev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)........................ev3(..),ev7(..),...ev5(..)...............ev6(..),ev5(..),...ev8Global variables: var1, var2, var3, ...Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)...ev3(..),ev7(..),...ev5(..)
- 46. What is the explicit state spaceState Space of the Event-B model(3,4,{a,b},...)Abstract machineEvent-B modelev3(5)Global variables: var1, var2, var3, ...Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ............States given by the values of global variablesTransitions labeled by events with concrete parameters
- 47. Test generation from Event-BState Space of the Event-B model(3,4,{a,b},...)Approach 1:Explore the state space using the ProB model checkerstate space explosion mainly due to dataTry: guide the searchev3(5)......... SBT Opportunity!
- 48. Test generation from Event-B – part IIState Space of the Event-B model(3,4,{a,b},...)Approach 2:Explore state space ignoring the data(i.e. local parameters)Problem 1: still large state spaceThen: construct approximations of state space up to depth K using finite automata Try 1: using machine learning and static analysisTry 2: using evolutionary algorithms?Problem 2: infeasible sequencesTry 1: constraint solving for path feasibilityTry 2: test data generation with metaheuristics ev3......... SBT Opportunities!
- 50. Let’s take a look at some of specific challenges for Event-B...More details in:A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Towards Search-Based Testing for Event-B Models. To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.
- 51. No explicit state spaceFactEvent-B has no explicit states like the EFSMs no control state (as in EFSMs)ProblemLarge (possibly infinite) state spacetesting coverage criteria must be definedonly recent work addressing SBT for EFSMsPossible ideas:coverage of all events (or a given subset of them) or coverage of all test paths of length < Kmany other coverages possible, so industrial guidance is neededconsider the class of Event-B models with a special state variable (see industrial use cases from SAP, SSF, Bosch and UML-B models)
- 52. Non-numerical typesFactEvent-B is based on set theoryset relations, powersets, functions, set comprehensions, products, records, etc.Complex structured data (e.g. business domain)Problemfitness functions in literature mostly defined for numerical typesPossible solutionsdesign new fitness functions for set-based (non-numerical) typesefficient encoding of mixed non-numerical/numerical test data
- 53. Hierarchical modelsFactEvent-B supports different types of hierarchyrefinement from abstract to concrete levelsmodel decompositionmodularity most industrial models use some sort of hierarchy (due to size) Problemno much previous work on SBT addressing hierarchical modelsPossible ideas:adapt existing work on test selection for hierarchical state machinesuse the existing ProB model checker that can partially deal with hierarchy
- 54. Non-determinismFactEvent-B has different types of non-determinism:| or :∈ operators (e.g. x :∈ {item1, ..., item20})non-deterministic choice of the event to be executed when several enablednon-deterministic choice of parameters (ANY construct) non-deterministic initialisation of variables satisfying the set of invariantsProblemno much previous work on SBT addressing non-deterministismPossible ideas:devise fitness functions that improve the chance of choosing a given path in a non-deterministic model (under certain assumptions)make the non-determinism visible (model instrumentation)
- 56. Generating test data for a pathProblemGiven one path of events, provide the test data (event parameters) that enables the execution of the path.Approachgenetic algorithmsencoding of sets into binary genesmixed choromosomes (numerical and binary genes)More details in:I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Test Data Generation for Event-B Models using Genetic Algorithms. In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.
- 57. Test data generation with genetic algorithmsEvent-B modelEncoding of variables…01101001101010100101100111101itemsITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})items : Powerset(ITEMS)“Chromosome”Crossover0SelectionMutationFitness evaluationEnd?Fitness functionsSimulator (ProB)
- 58. Fitness functions for one pathfitness := approach level + normalized branch levelNew objective functions for set typesClassical Tracey’s objective for numerical types
- 59. Examples from the benchmark
- 60. Statistical resultsStatistical comparison of Genetic Algorithms (GA) and
- 61. Random Testing (RT)
- 62. on 18 paths covering 5 Event-B models
- 63. using statistical test like t-test and U-testAnd (of course) the winner is:GA performs significantly better than RT on most paths Note: We are currently evaluating constraint-solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).
- 65. Generating finite models from Event-BProblemThere is no explicit state space of an Event-B modelApproachFinite automata learning (adapted L* algorithm )Aproximation through cover automataK-bound on the length of executionsUse finite automata for conformance test generationMore details in:F. Ipate, I. Dinca, A. Stefanescu:Model Learning and Test Generation for Event-B using Cover Automata. Submitted to SEFM’11.
- 66. First experimentsSBT Opportunity?! Preliminary approachApproximation through cover automata for bound lIncremental -> fits very well with model refinementsMinimal finite automata Sometimes difficult to find counterexamples (to the approximation)Scales for medium size models:Boundlev3.........
- 67. [Conclusions]
- 68. Opportunities for Search-Based TechniquesTo wrap-up opportunities:Test suite minimisation with multi-objective optimisation
- 69. Test data generation for one path with search-based algorithms
- 70. Construct finite models with evolutionary algorithms
- 71. Combine ProB model-checker with meta-heuristics
- 72. Combine ProB constraint-solver with meta-heuristics
- 73. Experiment with different search algorithms (PSO, ACO, SA,...)To be answered until end of project (April 2012): Which of the above work good in practice?