Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
0 likes477 views
The document describes algorithms for generating effective test suites for mixed discrete-continuous controllers modeled in Stateflow. It introduces the challenges of testing cyber-physical systems with both discrete and continuous behaviors. It then presents six test generation algorithms, including ones based on input diversity, state/transition coverage, and output diversity/stability/continuity. An evaluation of these algorithms on three industrial case studies examines their fault detection abilities, how they compare to each other, and how test suite size impacts results. The best performing algorithms focused on maximizing differences between output signals.
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
- 1. .lusoftware veriïŹcation & validation VVS Effective Test Suites for ! Mixed Discrete-Continuous StateïŹow Controllers Reza Matinnejad Shiva Nejati Lionel Briand SnT Center, University of Luxembourg Thomas Bruckmann Delphi Automotive Systems, Luxembourg
- 2. Cyber Physical Systems (CPSs) Combination of computations (algorithms) and physical dynamics (differential equations) 2 Physical world Computation
- 3. Testing (Typical) Software 3 X = 10, Y = 30 Z = 20 Algorithms Fail Pass Z = 10
- 4. Testing (CPS) Software 4 Algorithms + Differential Equations Fail Z = 20 X = 10, Y = 30 S1(t) S2(t) S3(t) Pass Z = 20 S3(t) S1 t S2 t S3 t S3 t
- 5. Software Testing Challenges (CPS) âąâŻMixed discrete-continuous behavior (combination of algorithms and continuous dynamics) âąâŻInputs/outputs are signals (functions over time) âąâŻSimulation is inexpensive but not yet systematically automated âąâŻPartial test oracles 5
- 6. Generating effective test suites for Software used in ! Cyber-Physical Systems 6 Our Goal
- 7. Simulink/StateïŹow âąâŻA data ïŹow-driven block diagram language âąâŻIs widely used to develop Cyber Physical Systems âąâŻIs executable 7
- 8. StateïŹow âąâŻA Statechart dialect integrated into Simulink âąâŻCaptures the state-based behavior of CPS software âąâŻHas mixed discrete-continuous behavior 8
- 9. Generating effective test suites for mixed discrete-continuous StateïŹow controllers 9 Our Goal
- 10. Discrete Behavior What we typically think of software models 10 On Off On Off Speed < 10 Speed > 10
- 11. Discrete-Continuous Behavior What software models are actually being built using StateïŹow 11 On Off CtrlSig On Off Speed < 10 Speed > 10 t CtrlSig t CtrlSig
- 12. Generating effective test suites for mixed discrete-continuous StateïŹow controllers 12 Our Goal
- 13. Test Suite Effectiveness (1) âąâŻ Test suite size should be small because âąâŻ Test oracles cannot be fully automated âąâŻ Output signals need to be inspected by engineers 13 Model Simulation Input Signals Output Signal(s) S3 t S2 t S1 t S3 t S2 t S1 t Test Case 1 Test Case 2
- 14. Test Suite Effectiveness (2) âąâŻ Test suites should have a high fault revealing power âąâŻ Small deviations in outputs may not be recognized/important âąâŻ Test inputs that drastically impact the output signal shape are likely to have a higher fault revealing power 14 Test Output 1 TimeTime CtrlSig Faulty Model Output Correct Model Output Test Output 2
- 15. Test Generation Algorithms! ! 15 Our Approach
- 16. Test Generation Algorithms âąâŻ Input-based Test Generation: âąâŻ Input Diversity Algorithm âąâŻ Coverage-based Test Generation: âąâŻ State Coverage Algorithm âąâŻ Transition Coverage Algorithm âąâŻ Output-based Test Generation: âąâŻ Output Diversity Algorithm âąâŻ Failure-based Algorithm 16
- 17. Input Diversity âąâŻMaximizing distances among input signals 17 Test Case 1 Test Case 2 Input Signal 1 Input Signal 2 S1 t S1 t S2 t S2 t
- 19. Test Generation Algorithms âąâŻ Input-based Test Generation: âąâŻ Input Diversity Algorithm âąâŻ Coverage-based Test Generation: âąâŻ State Coverage Algorithm âąâŻ Transition Coverage Algorithm âąâŻ Output-based Test Generation: âąâŻ Output Diversity Algorithm âąâŻ Failure-based Algorithm 19
- 20. Structural Coverage âąâŻMaximizing the number of states/transitions covered 20 State Coverage Transition Coverage 1 4 2 3 1 4 2 3
- 21. Test Generation Algorithms âąâŻ Input-based Test Generation: âąâŻ Input Diversity Algorithm âąâŻ Coverage-based Test Generation: âąâŻ State Coverage Algorithm âąâŻ Transition Coverage Algorithm âąâŻ Output-based Test Generation: âąâŻ Output Diversity Algorithm âąâŻ Failure-based Algorithm 21
- 22. Output Diversity âąâŻMaximizing distances among output signals 22 Test Case 1 Test Case 2 Output Signal S3 t S3 t
- 23. Failure-based Test Generation 23 Instability Discontinuity 0.0 1.0 2.0 -1.0 -0.5 0.0 0.5 1.0 Time CtrlSigOutput âąâŻMaximizing the likelihood of presence of speciïŹc failure patterns in output signals 0.0 1.0 2.0 Time 0.0 0.25 0.50 0.75 1.0 CtrlSigOutput
- 24. We developed our failure-based test generation algorithm using! Meta-Heuristic Search 24
- 25. The Alternative Choice 25 Our ApproachExisting WorkTechnique Model Checking - Require precisely deïŹned oracles (user-speciïŹed assertions) - Have been largely applied to time-discrete models - State-explosion problem! - No need for automated test oracles - Applicable to time-continuous and non-linear models - Our algorithms are black-box randomized search: - non-memory intensive - can be parallelized
- 26. 26 Failure-based Test Generation using Meta-Heuristic Search Input Signals Slightly Modifying Each Input Signal Fitness Functions Capturing the Likelihood of Presence of Failure Patterns in the Output Signals Repeat Until maximum resources spent S Initial Candidate Solution Search Procedure R Tweak (S) if Fitness (R) > Fitness (S) S R Return S
- 27. Output Stability ! Fitness Function âąâŻSum of the differences of signal values for consecutive simulation steps 27 stability(sgo) = kP i=1 |sgo(i · t) sgo((i 1) · t)| 0.0 1.0 2.0 -1.0 -0.5 0.0 0.5 1.0 Time CtrlSigOutput
- 28. Output Continuity ! Fitness Function 28 âąâŻMaximum of the minimum left or right derivatives for all the simulation steps 0.0 1.0 2.0 Time 0.0 0.25 0.50 0.75 1.0 CtrlSigOutput continuity(sgo) = K 1 max i=1 (min(|LeftDer(sgo, i)|, |RightDer(sgo, i)|))
- 29. Comparing the! Test Generation Algorithms! ! 29 Evaluation
- 30. Research Questions âąâŻ RQ1 (Fault Revealing Ability) âąâŻ RQ2 (Fault Revealing Subsumption) âąâŻ RQ3 (Test Suite Size) 30
- 31. Experiment Setup âąâŻThree StateïŹow models: two industrial and one publicly available case study 31 75 (faulty models) * 100 (algorithm runs) *6 (generation algorithms) * 5 (different test suite sizes) = 225,000 test suites (in total) Test Suite (size=3,5, 10,25,50) { 1.Fault Seeding 2.Generation Algorithm SF Faulty SF {75 75
- 32. Research Question 1! Fault Revealing Ability How does the fault revealing ability of our proposed test generation algorithms compare with one another? 32
- 33. 1.0 0.0 0.5 Input Diversity Output Diversity Fault Revealing Rate RQ1: Fault Revealing Ability 33 1.⯠Output-based and coverage-based algorithms outperformed the input diversity algorithm 2.⯠Output-based algorithms outperformed the coverage-based algorithms 3.⯠Overall, output stability algorithm performed the best
- 34. Research Question 2! Fault Revealing Subsumption Is any of our generation algorithms subsumed by other algorithms? 34
- 35. RQ2: Fault Revealing Subsumption 35 âąâŻ For each of the 75 faulty models, we identiïŹed the best generation algorithm(s) for different test suite sizes (5, 10, 25, and 50) Fault 1 State Coverage Transition Coverage Output Diversity Output Stability Output Continuity Fault 2 Fault 3 Fault 4
- 36. RQ2: Fault Revealing Subsumption (2) 36 1.⯠The coverage-based algorithms found the least number of faults 2.⯠Coverage-based algorithms are subsumed by output diversity algorithm when the test suite size increases (size = 25 , 50)
- 37. Research Question 3! Test Suite Size What is the impact of the size of test suites generated by our generation algorithms on their fault revealing ability? 37
- 38. RQ3: Test Suite Size 38 1.⯠The fault revealing rates for output stability/continuity is very high for small test suites(size = 3,5) for Instability/Discontinuity failures 2.⯠For Other failures, the ability of output diversity in revealing failures rapidly increases as the test suite size increases DiscontinuityInstability Others 0.0 0.5 1.0 3 5 10 25 50 Test Suite Size FaultRevealingRateMean 3 5 10 25 50 3 5 10 25 50 Output Stability Ouput Continuity State Coverage Transition CoverageOutput Diversity
- 40. Lesson 1! Coverage-based algorithms are less effective than output-based algorithms âąâŻ The test cases resulting from state/transition coverage algorithms cover the faulty parts of the models âąâŻ 97% state coverage and 81% transition coverage âąâŻ Cover faulty parts for 73 (out of 75) fault-seeded models âąâŻHowever, they fail to generate output signals that are sufïŹciently distinct from the oracle signal, hence yielding a low fault revealing rate 40
- 41. Lesson 2! Combining Output-based Algorithms 41 âąâŻ We suggest to divide the test suite size budget between output-based algorithms: Output Continuity Output Stability Output Diversity
- 42. CoCoTest 42
- 43. .lusoftware veriïŹcation & validation VVS Effective Test Suites for ! Mixed Discrete-Continuous StateïŹow Controllers Reza Matinnejad (reza.matinnejad@uni.lu) Shiva Nejati Lionel Briand SnT Center, University of Luxembourg Thomas Bruckmann Delphi Automotive Systems, Luxembourg
- 44. Lesson 1! Combing Output-based Algorithms âąâŻ We suggest to divide the test suite size budget between output stability, output continuity, and output diversity: 1.⯠Allocate a small part of the test budget to output continuity 2.⯠Share the rest of the budget between output stability and output diversity, by giving output diversity a higher share 44
- 45. Input / Output Vectors 45 0 5 10 50 150 250 FuelLevelSensor FuelLevel 0 5 10 100.0 91.43 84.43 75.62 70.01 66.19 61.21 56.66 54.32 52.81 50 100 Time (s) Time (s)
- 46. Study subjects 46 Publicly Available Name No. of Inputs Hierarchical States Parallelism No. of States SCPC ASS No No 23 42 13 16 2 No 1 No GCS Yes 8 10 0 Yes No. of Transitions 25 53 27 âąâŻSCPC: Supercharger Clutch Position Controller âąâŻASS: Auto Start Stop Control âąâŻGCS: Guidance Control System
- 47. Fault Revealing Rate (FRR) 47 FRR(SF, TS) = ( 1 91ïŁżiïŁżq Ëdist(sgi, gi) > THR 0 81ïŁżiïŁżq Ëdist(sgi, gi) <= THR âąâŻ FRR based on gi, output of the fault-free model, sgi, output of the fault- seeded model, and a threshold THR: 1.⯠For continuous dynamic systems, the system output is acceptable when the deviation is small and not necessarily zero 2.⯠It is more likely that manual testers recognize a faulty output signal when the signal shape drastically differs from the oracle.
- 48. RQ3: Test Suite Size 48 1.⯠The fault revealing rates for output stability/continuity is very high for small test suites for Instability/Discontinuity 2.⯠For âOtherâ failures, the ability of OD in revealing failures rapidly increases as the test suite size increases Discontinuity SC TC OD OS OC * * + + -- Instability Others 0.0 0.5 1.0 3 5 10 25 50 * * * * + + + - - Test Suite Size FRRMean + - - 3 5 10 25 50 3 5 10 25 50 * * * - - - + + + - - - * * * + + + + + * * *- - -