Qualification of Eclipse-based Tools according to ISO 26262
2 likes1,934 views
This document discusses tool qualification according to ISO 26262. It provides an overview of ISO 26262 requirements for tool qualification, including determining a tool's Tool Confidence Level based on its potential errors and detection methods. It then describes how Validas developed a Tool Chain Analyzer to automatically analyze a tool chain and determine each tool's TCL. The document applies this approach to the Eclipse development environment and determines the TCA itself has TCL 1 if its textual export is reviewed, but would have TCL 3 without review. It concludes that tool users must qualify tools while tool providers should supply qualification information.
Qualification of Eclipse-based Tools according to ISO 26262
- 1. Validas AG Method for Qualification of Eclipse- based Tools according to ISO 26262 4.11.2010 Dr. Oscar Slotosch
- 2. Page 2 Validas AG Content ‣Motivation for Tool Qualification ‣ISO 26262 Requirements ‣Tool Chain Analysis ‣Application to Eclipse ‣Summary
- 3. Page 3 Validas AG Motivation for Tool Qualification ‣Development tools can have errors that - Cause errors in the product - Hide errors in the product ‣Both has to be avoided ‣Safety standards require to protect customer from them - IEC 65108 - ISO 26262 - DO178 B/C ‣Product verification is required ‣Tool confidence is required
- 4. Page 4 Validas AG 4Seite 19.11.2009 Validas AG ÷ 11 11 00
- 5. Page 5 Validas AG Content ‣Motivation for Tool Qualification ‣ISO 26262 Requirements ‣Tool Chain Analysis ‣Application to Eclipse ‣Summary
- 6. Page 6 Validas AG ISO 26262 Requirements on Tools Part 8, Chapter 11: Tool Qualification ‣ Analyze all used tools (the complete “Tool Chain”) - Use cases - Inputs/outputs ‣ Determine impact on safety of the product - TI1: No impact => Tool Confidence Level (TCL 1) - TI2: Impact: For all potential errors determine tool detection (TD) probability in the applied process • TD1: High => tool has TCL 1 • TD2: Medium => tool has TCL 2 • TD3: other => tool has TCL 3 ‣ For a given ASIL and TCL select the qualification methods: all “++” or an equivalent combination ‣ Make a “Confirmation Review” of - TCL classification - Qualification methods of the tools
- 7. Page 7 Validas AG Confidence from Use Sufficient and adequate data for the use of the tool with ‣Tool version and configuration ‣Comparable use cases ‣Systematic error recording ‣details of the period of use and relevant data on its use ‣the safeguards, avoidance measures or work-arounds for the known malfunctions, or detection measures for a corresponding erroneous output, if applicable The increased confidence from use argument shall only be valid for the considered version of the software tool In Eclipe there is a Usage Data Collector that uploads usage data But where is the download? And where are the malfunctions and safeguards?
- 8. Page 8 Validas AG Evaluation of the Development Process The development process applied for the development of the software tool shall comply with an appropriate standard (?!) ‣NOTE For open source developments some of the standards used by those communities can also be appropriate ‣This assessment covers the development of an adequate and relevant subset of the features of the software tool (Automotive SPICE, CMMI, ISO 15504, etc.) Where are the details, like tests?
- 9. Page 9 Validas AG Validation the validation measures shall demonstrate that the software tool fulfils its specified requirements ‣Tests for functional and non-functional aspects ‣the malfunctions and their corresponding erroneous outputs of the software tool occurring during validation shall be analysed together with information on their possible consequences and with measures to avoid or detect them ‣the reaction of the software tool to anomalous operating conditions shall be examined ‣Validation suites can be build
- 10. Page 10 Validas AG Content ‣Motivation for Tool Qualification ‣ISO 26262 Requirements ‣Tool Chain Analysis ‣Application to Eclipse ‣Summary
- 11. Page 11 Validas AG Tool Chain Analysis ‣Validas developed a method to determine the TCL automatically ‣Based on a simple but formal tool model with - Tools, use cases, artefacts - Data flow, control flow ‣Enriched by specification of - Errors - Detection and prevention - Probabilities ‣Tool: Tool Chain Analyzer
- 12. Page 12 Validas AG Tool Chain Analyzer
- 13. Page 13 Validas AG Results of a Simple Example All tools have TCL 3 (unchecked errors) Error Flow Control Flow Data Flow Artefacts Process Tool / Use Case make dcc lcc
- 14. Page 14 Validas AG Results of an Extended Example Make has TCL 1 (all errors checked with TD1) make dcc lcc
- 15. Page 15 Validas AG Tool Qualification Lessons ‣New standards require to analyze all tools in the process for “potential errors that affect the safety” ‣One tool can have different TCLs in different processes ‣Reduction of TCL in the process causes effort ‣A high TCL saves the effort for detecting the tool errors in development ‣Required information for TCL determination - Application (Product development) process (from the user) - Uses cases of tools (from user/supplier) - Potential errors (from supplier) - Error detection and prevention methods (from supplier) ‣Tool Chain Analysis automatically determines the TCL
- 16. Page 16 Validas AG Content ‣Motivation for Tool Qualification ‣ISO 26262 Requirements ‣Tool Chain Analysis ‣Application to Eclipse ‣Summary
- 17. Page 17 Validas AG Eclipse Applications ‣ Structure of Eclipse Applications: - Plugins - Bundles - Packages - Functions ‣ Potential Errors (in each part) - Exceptions - Assertions - Semantic Errors ‣ Error Detection - Catch - Stack Traces - Assertions - Tests
- 18. Page 18 Validas AG Example: Tool Chain Analyzer ‣ RCP Application ‣ Based on an ecore model with EMF generator ‣ Plugin architecture ‣ Based on ISO 26262 and formal semantics ‣ Use cases - TCL determination - Generation of documentation / explanations - Generation of graphical views ‣ Saftey Critical Errors: - Wrong Classification (TCL) ‣ Not critical: - Exceptions, Crashes, Dialogs, Persistency,..
- 19. Page 19 Validas AG Bundles and Dependencies ‣Base model (Tools): determined from the OSGI structures ‣TODO: potential errors and possible checks ‣TCL has method to analyze it‘s own dependecies and generates the base models ‣Could be generated from every eclipse tool architecture ‣Basis for error analysis ?
- 20. Page 20 Validas AG TCA Classification Information ‣ The TCA provides the following use cases - Textual export with potential errors • Wrong TCL ❖ Ignoring reachable checks ❖ Using unreachable checks • Wrong Conformance check of ASIL / Qualification - Graphical export (for debugging) with irrelevant errors - Determination of TCL within the tree view (for development) with irrelevant errors ‣ If textual export is reviewed (against the above errors) the TCA has TCL 1 ‣ If the export is not reviewed it would have TCL 3 - We could build a validation suite for TCL 3 (ASIL D) with • test automatization • our tests models • comparing the TCA results with its formal semantics and • a coverage measurement (EMMA) ‣ Since „confirmation review“ is required in ISO 26262 the TCA has TCL 1
- 21. Page 21 Validas AG Content ‣Motivation for Tool Qualification ‣ISO 26262 Requirements ‣Tool Chain Analysis ‣Application to Eclipse ‣Summary
- 22. Page 22 Validas AG Summary ‣ ISO 26262 requires to check all used tools for confidence ‣ Tool Confidence Level (TCL) depends on the application process ‣ Tool User - has to classify tools - can restrict to safety relevant functions - check the result of tools (manually or by redundancy) ‣ Tool provider/developer (of each plugin) should - provide information on use cases and tools - provide information on potential errors and checks - can NOT restrict to safety relevant functions - provide help for qualification of tools • Usage information on versions, configuration • Development process • Test cases / code coverage ‣ Tool Chain Analyzer has TCL 1, but requires manual review ‣ Validation with Coverage Measurement can reduce reviews
- 23. Page 23 Validas AG Arnulfstraße 27 80335 München www.validas.de info@validas.de Your partner for innovation in embedded quality Thank You !