Automated Requirements-Based Testing for Medical Device Software.ppsx

0
Copyright 2019 – QA Systems GmbH www.qa-systems.cn
Automated
Requirements-Based Testing
for Medical Device Software

1
ISO 26262 SW Verification Phases
SW Architectural
Design
SW Unit
Design
Code
Implementation
SW Safety
Requirements
SW Unit
Verification
SW Integration
& Verification
Testing of the
Embedded SW
Safety measures properly implemented
Complies with unit design & fulfils ASIL SW requirements
No undesired functionality or functional safety properties
Safety measures properly implemented
Fulfils architectural design
Fulfils safety-related requirements in target environment
Part 6:
Product Development at the
software level
 Phases 9 – 11
 Tables 7 – 15
7
6
8 9
8
11
10
Configuration
& Calibration
Data

2
Requirements Verification Method
ISO 26262 Table 7 – Methods for software unit verification
ISO 26262 Table 10 – Methods for verification of software integration
ISO 26262 Table 14 – Methods for tests of the embedded software

3
Deriving Test Cases from Requirements
ISO 26262 Tables 8 – Methods for deriving test cases for software unit / integration testing
ISO 26262 Table 15 – Methods for deriving test cases for the test of the embedded software
ISO 26262 Table 11 – Methods for deriving test cases for software integration testing

4
Requirements Based Testing (RBT)
Requirements
• Decomposed
• Correct
• Complete
• Unambiguous
• Logically consistent
Tests
• Pre-conditions
• Inputs
• Expected behaviours
• Expected outputs
• Post-conditions
Unit
Design
Code
Unit
Test
Integration
Test
Architectural
Design
Requirements Coverage
% Requirements verified by tests
<-> Traceability to tests
Code Coverage
% code executed by tests
<->Traceability to requirements
<->Traceability to tests
Test Coverage
% tests executed & passing
<-> Traceability to requirements
Safety
Requirements
Embedded
Test
Configuration &
Calibration Data

5
Manual Test Generation
Test Cases crafted Manually from Requirements
Can be hard work! – Even with powerful test tools
Insufficiently validated requirements
 Decomposed, Correct, Complete, Unambiguous, Logically consistent
 High-reliance on structural code coverage & reverse engineering
Complexity of test vectors
 Pre-conditions, Inputs, Expected behaviours & outputs, post-conditions
Boundary of Low Level Requirements ≠ usable test case vectors
 Test Framework: Drivers, Dependencies & Datasets
Gaps & Overlaps
 Defensive programming, private/protected code etc
 Equivalence classes

6
Requirements Tests
Code
So…How to Automate?
Configuration &
Calibration Data

7
Tests
Generation from Requirements
Test Case Generation
Test Cases Generated from Requirements
Very limited capability from:
NL, SNL, PDL,
Use Case Scenarios
Mathematical specs
More capability from
Models (e.g. MBT with UML)
Requirements

8
Generation from Code
Test Cases Generated from Code
Test Vectors from path solving
 Intelligent optimisation
Full test framework
 Pre-conditions, Inputs,
Expected behaviours,
Expected outputs & post-conditions
Tests generated for
maintainability & traceability Code
Tests
AutoTest
Requirements

9
Why use Coverage & Traceability?
Standard Compliance – the 100% Picture
Bi-directional requirements traceability
All executable code is justified tested
Evidence of success is: passing tests + traceability
Helps Ensure Completeness
Changed requirements capture & validation
Just enough code changes
Test case design updates
RBT processes can be most effective when iterative
Code
Test
Req


10
AutoTest & Trace for ISO 26262
AutoTest

11
Copyright 2017 – QA Systems GmbH www.qa-systems.com
AutoTest Generation
Flexible application
 GUI or CLI invocation
 Complete suite of passing unit tests
 Additional test cases to fill gaps
 Black-box cluster integration test through public functions
 White-box unit isolation test of static functions
 Uses Cantata workspace preferences
Test cases exercise all paths through the code
 Entry-Point
 Statement
 Decision
 MC/DC (unique cause)
Test Cases are complete & maintainable for full control
 All required inputs: parameters + accessible data
 All expected outputs: parameters + accessed data + call-order
 Each test case path solving purpose explained

12
Build Run
Test
Exe
Instruments
AutoTest
Makefiles
Tests
Code
AutoTest Process
Code Copy
Generation
Report
Test Results
Automatic Test Generation
Automatic Test Execution

13
Example AutoTest Exercise
• 541 Source Files
• 807 C Functions
• 55,151 Executable LoC
• 4,901 McCabe total complexity
Source
Files
• 93% Fully executed
• 95% Fully executed
• 95%+ Fully executed
• 5,035 Total test cases
• 336,355 Total checks
Tested
Source
Files
Execution
36 minutes
Generation
2.03 hours

14
Traced requirements, test
status and code coverage
Test Information
.csv ReqIF Excel
Requirements
Requirements
Management Tool
Full bi-directional
requirements traceability
evidence
Drag and drop tracing of requirements
(text, diagrams, links) with test cases.
Generate tests
link to requirements
Test Tool
Requirements Trace Closes Loop

15
Easy Linking in Cantata Trace
Bi-directional drag and drop interface, immediately creates links on a server
Whole Test Scripts linked to Requirements
Individual Test Cases linked to Requirements

16
3 Part Automation
1 Automatic Test Vector Generation
Test case vectors from code exercising all paths (up to MC/DC coverage)
Sets input parameters & data throughout test execution
Checks expected vs actual data, input & output parameters and call order
3 Automated Traceability & Coverage Data Production
Complete Requirements imported/exported for testing
AutoTest cases generated with traceable descriptions
Test status, Requirements traceability & Structural coverage evidence
2 Automated Test Execution
Continuous integration build, run and reporting

17
Complete 3 Way Analysis
Requirements
Code
Tests
Requirements Coverage
See requirements
coverage in your
requirements
management & test tools
Use the same tool for all
trace data
Test Coverage
Run tests when not executed
(continuous integration and
testing helps a lot)
Fix tests when they fail
Code Coverage
When you have gaps, identify if the code is:
dead / redundant, unreachable, deactivated (not used in this context)
If not, then add a test and that needs to be traced to [new] requirements
Configuration &
Calibration Data

18
Learn More

19
Further Enhancements?
ISO 26262 Table 7 – Methods for software unit verification
ISO 26262 Table 10 – Methods for verification of software integration
ISO 26262 Table 14 – Methods for tests of the embedded software
In Development

20
Thank you

Automated Requirements-Based Testing for Medical Device Software.ppsx

More Related Content

Similar to Automated Requirements-Based Testing for Medical Device Software.ppsx (20)

Recently uploaded (20)

Automated Requirements-Based Testing for Medical Device Software.ppsx

Editor's Notes