Testing of Cyber-Physical Systems: Diversity-driven Strategies
0 likes377 views
Lionel Briand discusses strategies for testing cyber-physical systems using diversity-driven approaches. He outlines challenges in verifying controllers and decision-making components in cyber-physical systems due to large input spaces and expensive model execution. Briand proposes maximizing diversity of test cases to improve fault detection. He describes using diversity of input signals, output signals, and failure patterns to generate test cases. Search algorithms are used to find test cases that maximize diversity or reveal specific failure patterns. The strategies are shown to significantly outperform coverage-based and random testing on Simulink models.
![• Supercharger bypass flap controller
üFlap position is bounded within
[0..1]
üImplemented in MATLAB/Simulink
ü34 (sub-)blocks decomposed into 6
abstraction levels
Supercharger
Bypass Flap
Supercharger
Bypass Flap
Flap position = 0 (open) Flap position = 1 (closed)
Simple Example
16](https://image.slidesharecdn.com/cow57-lionel-180129145956/85/Testing-of-Cyber-Physical-Systems-Diversity-driven-Strategies-15-320.jpg)
![Open Loop Controllers
On
Off
CtrlSig
• Mixed discrete-continuous behavior:
Simulink stateflows
• No plant model: Much quicker simulation
time
• No feedback loop -> no automated oracle
• The main testing cost is the manual
analysis of output signals
• Goal: Minimize test suites
• Challenge: Test selection
• Entirely different approach to testing
echniques. We consider this criterion
ows have complex internal structures
ations, making them less amenable to
hey have rich time-continuous outputs.
following contributions:
lem of testing Stateflows with mixed
ehaviours. We propose two new test
output stability and output continuity
ting test inputs that are likely to pro-
uts exhibiting instability and disconti-
vely.
ox coverage and the blackbox output
eria to Stateflows, and evaluate their
or continuous behaviours. The former
aditional state and transition coverage
d the latter is defined based on the re-
s criterion [?].
ness of our newly proposed and the
ria by applying them to three Stateflow
wo industrial and one public domain.
RESULT.
(c) Engaging state of SCC -- mixed discrete-continuous behaviour
Disengaging
Engaged
[disengageReq]/time := 0
[time>5]
[time>5]
time + +;
OnMoving OnSlipping
OnCompleted
time + +;
ctrlSig := f(time)
Engaging
time + +;
ctrlSig := g(time)
time + +;
ctrlSig := 1.0
[¬(vehspd = 0)
time > 2]
[(vehspd = 0)
time > 3]
[time > 4]
Figure 1: Supercharge Clutch Controller (SCC) Stateflow.
transient states [?], engaging and disengaging, specifying that mov-
ing from the engaged to the disengaged state and vice versa takes
six milisec. Since this model is simplified, it does not show han-
dling of alterations of the clutch state during the transient states28](https://image.slidesharecdn.com/cow57-lionel-180129145956/85/Testing-of-Cyber-Physical-Systems-Diversity-driven-Strategies-24-320.jpg)
Testing of Cyber-Physical Systems: Diversity-driven Strategies
- 1. .lusoftware verification & validation VVS Testing of Cyber-Physical Systems: Diversity-driven Strategies Lionel Briand COW 57, London, UK
- 2. Cyber-Physical Systems • A system of collaborating computational elements controlling physical entities 2
- 3. Context • Projects on verification of cyber-physical systems, control systems, autonomous systems, … • Focus on safety, performance, resource usage … • Automotive, satellite, energy, manufacturing … 3
- 6. Development Process 6 Functional modeling: • Controllers • Plant • Decision Continuous and discrete Simulink models Model simulation and testing Architecture modelling • Structure • Behavior • Traceability System engineering modeling (SysML) Analysis: • Model execution and testing • Model-based testing • Traceability and change impact analysis • ... (partial) Code generation Deployed executables on target platform Hardware (Sensors ...) Analog simulators Testing (expensive) Hardware-in-the-Loop Stage Software-in-the-Loop Stage Model-in-the-Loop Stage
- 7. Simulink Models - Simulation • Simulation Models • heterogeneous 8 Software Plant Model Network Model • continuous behavior • are used for • algorithm design testing • comparing design options
- 9. Problem • How do we automatically verify and test CP functional models (e.g., controller, plant, decision) at MiL? • What types of requirements / properties do we check? • Commercial tools: • Cannot handle continuous operators, floating point models (e.g., SLDV, Reactis) • Based on model structural coverage: low fault detection • Can only handle linear systems and specific properties (Linear analysis toolbox) 10
- 10. Challenges • Limited work on verification and testing of controllers and decision-making components in CP systems • Space of test input signals is extremely large. • Model execution, especially when involving plant models, is extremely expensive. 11
- 11. More Challenges • Test oracles are not simple Boolean properties and easily known in advance – they involve analyzing changes in value over time (e.g., signal patterns) and assessing levels of risk. • Simulatable plant model of the physical environment is not always available or fully accurate and precise. 12
- 12. Diversity Strategies • Strategy: Maximize diversity of test cases • Assumption: “the more diverse the test cases the higher their fault revealing capacity” • Challenge: Define “diversity” in context, pair-wise similarity computation, test selection algorithm • Examples of early work • ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE 2010, Hemmati et al., Similarity-based test selection applied to model-based testing based on state models for control systems, selection with GA • Full control on test budget: Maximize fault detection for a given test suite size 13
- 15. • Supercharger bypass flap controller üFlap position is bounded within [0..1] üImplemented in MATLAB/Simulink ü34 (sub-)blocks decomposed into 6 abstraction levels Supercharger Bypass Flap Supercharger Bypass Flap Flap position = 0 (open) Flap position = 1 (closed) Simple Example 16
- 16. MiL Test Cases 17 Model Simulation Input Signals Output Signal(s) S3 t S2 t S1 t S3 t S2 t S1 t Test Case 1 Test Case 2
- 17. Initial Desired Value Final Desired Value time time Desired Value Actual Value T/2 T T/2 T Test Input Test Output Plant Model Controller (SUT) Desired value Error Actual value System output+ - MiL Testing of Controllers 18
- 18. Requirements and Test Objectives InitialDesired (ID) Desired ValueI (input) Actual Value (output) FinalDesired (FD) time T/2 T Smoothness Responsiveness Stability 20
- 19. 21 Test Generation Approach • We formalize controller’s requirements in terms of desired and actual outputs Smoothness • We rely on controller’s feedback to automate test oracles desired value actual value < Threshold Desired Value (Setpoint) Actual Value (feedback) System Output + - Control Signal Plant (environment)Controller
- 20. A Search-Based Test Approach Initial Desired (ID) FinalDesired(FD) Worst Case(s)? • Search directed by model execution feedback • Finding worst case inputs • Possible because of automated oracle (feedback loop) • Different worst cases for different requirements • Worst cases may or may not violate requirements 22
- 21. Initial Solution HeatMap Diagram 1. Exploration List of Critical RegionsDomain Expert Worst-Case Scenarios + Controller- plant model Objective Functions based on Requirements 2. Single-State Search time Desired Value Actual Value 0 1 2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Initial Desired Final Desired 23
- 22. Results • We found much worse scenarios during MiL testing than our partner had found so far • These scenarios are also run at the HiL level, where testing is much more expensive: MiL results -> test selection for HiL • But further research was needed: • Simulations are expensive • Configuration parameters 24
- 23. Final Solution + Controller Model (Simulink) Worst-Case Scenarios List of Critical PartitionsRegression Tree 1.Exploration with Dimensionality Reduction 2.Search with Surrogate Modeling Objective Functions Domain Expert Visualization of the 8-dimension space using regression treesDimensionality reduction to identify the significant variables (Elementary Effect Analysis) Surrogate modeling to predict the fitness function and speed up the search (Machine learning) 25
- 24. Open Loop Controllers On Off CtrlSig • Mixed discrete-continuous behavior: Simulink stateflows • No plant model: Much quicker simulation time • No feedback loop -> no automated oracle • The main testing cost is the manual analysis of output signals • Goal: Minimize test suites • Challenge: Test selection • Entirely different approach to testing echniques. We consider this criterion ows have complex internal structures ations, making them less amenable to hey have rich time-continuous outputs. following contributions: lem of testing Stateflows with mixed ehaviours. We propose two new test output stability and output continuity ting test inputs that are likely to pro- uts exhibiting instability and disconti- vely. ox coverage and the blackbox output eria to Stateflows, and evaluate their or continuous behaviours. The former aditional state and transition coverage d the latter is defined based on the re- s criterion [?]. ness of our newly proposed and the ria by applying them to three Stateflow wo industrial and one public domain. RESULT. (c) Engaging state of SCC -- mixed discrete-continuous behaviour Disengaging Engaged [disengageReq]/time := 0 [time>5] [time>5] time + +; OnMoving OnSlipping OnCompleted time + +; ctrlSig := f(time) Engaging time + +; ctrlSig := g(time) time + +; ctrlSig := 1.0 [¬(vehspd = 0) time > 2] [(vehspd = 0) time > 3] [time > 4] Figure 1: Supercharge Clutch Controller (SCC) Stateflow. transient states [?], engaging and disengaging, specifying that mov- ing from the engaged to the disengaged state and vice versa takes six milisec. Since this model is simplified, it does not show han- dling of alterations of the clutch state during the transient states28
- 25. Selection Strategies Based on Search • White-box structural coverage • State Coverage • Transition Coverage • Input signal diversity • Output signal diversity • Failure-Based selection criteria • Domain specific failure patterns • Output Stability • Output Continuity S3 t S3 t 29
- 26. System Output Input Signals Output Signals Controller Plant (environment) 30 instability failure pattern discontinuity failure pattern output diversity • We assume test oracles are manual Test Generation Approach • We rely on output signals to produce small test suites with high fault-revealing ability
- 27. Output Diversity -- Vector-Based 31 Output Time Output Signal 2 Output Signal 1 Normalized Euclidian Distance
- 28. 32 Output Diversity -- Feature-Based increasing (n) decreasing (n)constant-value (n, v) signal features derivative second derivative sign-derivative (s, n) extreme-derivatives 1-sided discontinuity discontinuity 1-sided continuity with strict local optimum value instant-value (v) constant (n) discontinuity with strict local optimum increasing C A B
- 29. 33 Output Diversity -- Feature-Based increasing (n) decreasing (n)constant-value (n, v) signal features derivative second derivative sign-derivative (s, n) extreme-derivatives 1-sided discontinuity discontinuity 1-sided continuity with strict local optimum value instant-value (v) constant (n) discontinuity with strict local optimum increasing C A B Similarity: To which extent any part of a signal is similar to a feature
- 30. Failure-based Test Generation 34 Instability Discontinuity 0.0 1.0 2.0 -1.0 -0.5 0.0 0.5 1.0 Time CtrlSigOutput • Search: Maximizing the likelihood of presence of specific failure patterns in output signals • Domain-specific failure patterns elicited from engineers 0.0 1.0 2.0 Time 0.0 0.25 0.50 0.75 1.0 CtrlSigOutput
- 31. Search • Whole test suite generation approach • Used when objective functions characterize the test suite • Optimize test objective for a given test suite size (budget for manual oracles) • Maximize the minimum distances of each output signal vector from the other output signal vectors • Adaptation of Simulated Annealing 35
- 32. Faulty Model Output 36 Correct Model Output Fault-Revealing Ability Covers the fault and Covers the fault but is Likely to reveal it is very unlikely to reveal it
- 33. Results • The test cases resulting from state/transition coverage algorithms cover the faulty parts of the models • However, they fail to generate output signals that are sufficiently distinct from expectations, hence yielding a low fault revealing rate • Diversity strategies significantly outperforms coverage-based and random testing • Output-based algorithms are much more effective, both based on diversity and failure patterns 37
- 34. Results • Feature-based diversity fares significantly better than vector-based diversity • Strategies based on failure patterns find different types of faults than diversity strategies • Existing commercial tools: Not effective at finding faults, not applicable to entire Simulink models, e.g., Simulink Design Verifier 38
- 37. Conclusions • Maximizing output diversity helps identify scenarios where the discrepancy between the produced and expected signal is large • Useful when test output signals are analyzed manually • Simulink models and their outputs are complex • Helps maximize fault detection within a fixed budget • Properly defining diversity was a challenge 41
- 38. Reflections on Diversity • Useful strategy when no precise guidance, directly related to the test objectives, is available for the search • In the general, the key issue is how to define diversity in an optimal way given the objectives • In practice, how diversity is defined also depends on what information is available at a reasonable cost • The time complexity of computing diversity is a major cost of the search – it must be accounted for 42
- 39. Acknowledgements • Shiva Nejati • Reza Matinnejad • Delphi Automotive Systems, Luxembourg 43
- 40. References • R. Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers: Scalable Search Using Surrogate Models”, IEEE/ACM ASE 2014 (Distinguished paper award) • R. Matinnejad et al., “Effective Test Suites for Mixed Discrete-Continuous Stateflow Controllers”, ACM ESEC/FSE 2015 (Distinguished paper award) • R. Matinnejad et al., “Automated Test Suite Generation for Time-continuous Simulink Models“, IEEE/ACM ICSE 2016 • R. Matinnejad et al., “Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior“, under minor revision with IEEE TSE 44
- 41. .lusoftware verification & validation VVS Testing of Cyber-Physical Systems: Diversity-driven Strategies Lionel Briand COW 57, London, UK