Traceability Beyond Source Code: An Elusive Target?
4 likes1,106 views
This document discusses traceability beyond source code, which is an elusive target. It provides an overview and examples from industrial research projects on requirements-requirements, requirements-design, requirements-test cases traceability. The examples show challenges in capturing changes precisely and change rationales. Automated traceability approaches can reduce effort but may lack accuracy required for certification. Traceability is important for certification, change management and economic decision making.
![Requirements-Requirements
23
Requirements
• 160 Requirements!
• 9 change scenarios!
• 72 Requirements!
• 5 change scenarios!
[RE 2015, TSE 2015, ESEM 2014, ESEM 2013]](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-23-320.jpg)
![Requirements-Design
34
Archi. & Design
Requirements
[TOSEM 2014, IST 2012, FSE 2011, HASE 2011]](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-34-320.jpg)
![Requirements-Test Cases
58
Requirements
Test Cases
BodySense
[ISSTA 2015]](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-58-320.jpg)
![Regulations - Requirements
64
Requirements
Regulations
[RE 2014, MODELS 2014]
• New tax system under development
• System needs to be compliant with the
law and remain so over time](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-64-320.jpg)
![Example
66
Art. 105bis […]The commuting
expenses deduction (FD) is
defined as a function over the
distance between the principal
town of the municipality on
whose territory the taxpayer's
home is located and the place of
taxpayer’s work. The distance is
measured in units of distance
expressing the kilometric
distance between [principal]
towns. A ministerial regulation
provides these distances.
Interpretation + Traces](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-66-320.jpg)
![Natural Language Requirements
• [RE 2015] C. Arora et al., Change impact analysis for natural language requirements: An NLP
approach
• [TSE 2015] C. Arora et al., Automated Checking of Conformance to Requirements Templates using
Natural Language Processing
• [ESEM 2014] C. Arora et al., Improving requirements glossary construction via clustering
• [ESEM 2013] C. Arora et al., Automatic checking of conformance to requirements boilerplates via
text chunking
Requirements-Driven Testing
• [ISSTA 2015] C. Wang et al., Automatic Generation of System Test Cases from Use Case
Specifications
76](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-76-320.jpg)
![SysML Traceability and Safety Analysis
• [TOSEM 2014] L. Briand et al., Traceability and SysML design slices to support safety inspections:
A controlled experiment.
• [IST 2012] S. Nejati et al.A SysML-Based Approach to Traceability Management and Design Slicing
in Support of Safety Certification: Framework, Tool Support, and Case Studies
• [HASE 2011] M. Sabetzadeh et al., Using SysML for Modeling of Safety--Critical Software--
Hardware Interfaces: Guidelines and Industry Experience
• [FSE 2011] D. Falessi et al., SafeSlice: a model slicing and design safety inspection tool for SysML.
Legal Modeling and Analysis
• [MODELS 2014] G. Soltana et al., “ UML for Modeling Procedural Legal Rule”
• [RE 2014] M. Adedjouma et al., “Automated Detection and Resolution of Legal Cross References”
77](https://image.slidesharecdn.com/sstkeynote-150517100619-lva1-app6891/85/Traceability-Beyond-Source-Code-An-Elusive-Target-77-320.jpg)
Traceability Beyond Source Code: An Elusive Target?
- 1. .lusoftware verification & validation VVS Traceability Beyond Source Code: An Elusive Target? Lionel Briand Interdisciplinary Centre for Security, Reliability and Trust (SnT) University of Luxembourg SST, Florence - May 17, 2015
- 2. Acknowledgements • Shiva Nejati • Mehrdad Sabetzadeh • Fabrizio Pastore • Chetan Arora • Chunhui Wang • Ghanem Soltana • Davide Falessi 2
- 3. Outline • Introduction • Overview • Examples from industrial research projects • Reflections and conclusions 3
- 4. Motivations • Traceability research is source-code-centric • Certification (safety, privacy …) • Change management: Impact analysis, design rationale, regression testing … • Change management is a key challenge to certification • System-level activity 4
- 5. Challenges • Establishing and maintaining traces is typically expensive • Automation, in most cases, does not provide the level of accuracy required • The benefits of exploiting traces are still unclear in many contexts • Highly contextualized: A great deal of variation in development contexts entails a great deal of variation in traceability solutions • Targeted analysis of traces drives traceability solutions 5
- 6. Requirements • Hundreds or thousands of them • Higher-level requirements (usually from customers) decomposed into lower-level ones (analysts) • Some more critical than others • Constantly changing and evolving: A stronger argument for the economic benefits of traceability 6
- 7. Modeling • In many application domains where traceability is required, system and software modeling is a rising practice • Provisions in standards lead to modeling • IEC 61508 (meta-standard), DO-178B (Avionics), EN50129 (Railways), ISO 26262 (Automotive) • UML, SysML, Simulink, … 7
- 8. Economic Decision • Not just about trace “accuracy” … • Economic trade-off • Cost: Establishing and maintaining traces • Benefit: More accurate decisions, decrease in human effort • Decision science • Makes it hard to study, out of context, as it determines effort and benefits 8
- 9. Overview
- 10. Traceability at a Glance 10 Archi. & Design Requirements Test Cases Regulations Source Code Bulk of Research
- 11. Requirements-Source Code • Natural language • Hundreds or thousands of traces • Information Retrieval & Natural Language Processing • Coding conventions • Level of granularity? • Minimum accuracy for ensuring practicality? Few human studies … 11 Requirements Source Code
- 12. Traceability at a Glance 12 Archi. & Design Requirements Test Cases Regulations Source Code Change Impact
- 13. Requirements-Requirements • Mostly natural language • Sometimes structured (template) • Hundreds of traces • Domain terminology, concepts, and their relationships are key to discovering traces among requirements • Syntactic and semantic similarity measures 13 Requirements
- 14. Traceability at a Glance 14 Archi. & Design Requirements Test Cases Regulations Source Code Compliance with laws, regulations, standards
- 15. Standards-Requirements • Many standards, laws, and regulations • They must be interpreted in context • Compliance must be ensured • Critical systems: Risks and hazards • Requirements as mitigations • Subjectivity, residual risks 15 Requirements Regulations
- 16. Traceability at a Glance 16 Archi. & Design Requirements Test Cases Regulations Source Code Certification, change management
- 17. Requirements-Design • Capture the rationale of design decisions • Support evolution, avoid violating essential design decisions • Useful for impact analysis based on traces • What is a rationale? Level of granularity? • Design representation? 17 Archi. & Design Requirements
- 18. Traceability at a Glance 18 Archi. & Design Requirements Test Cases Regulations Source Code Certification Regression testing
- 19. Requirements-Test Cases • Requirements “coverage” required by standards • Normally many test cases per requirement • Thousands of traces • Regression testing • Precise impact analysis requires explicit test strategy and rationale • How were test cases derived from requirements? • Representation of requirements matters 19 Requirements Test Cases
- 20. Traceability at a Glance 20 Archi. & Design Requirements Test Cases Regulations Source Code Impact analysis, design conformance
- 21. Design-Source Code • Ideally, code should be generated from design models, e.g., controllers with Simulink • This would lead to “free” traceability • In practice, not always that simple … 21 Archi. & Design Source Code
- 22. Example Projects
- 23. Requirements-Requirements 23 Requirements • 160 Requirements! • 9 change scenarios! • 72 Requirements! • 5 change scenarios! [RE 2015, TSE 2015, ESEM 2014, ESEM 2013]
- 24. Example • R1: The mission operation controller shall transmit satellite status reports to the user help desk. • R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk. • R3: The mission operation controller shall transmit any detected anomalies with the user help desk. 24
- 25. Example • R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository. • R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk. • R3: The mission operation controller shall transmit any detected anomalies with the user help desk. 25
- 26. Challenge#1 - Capture Changes Precisely • R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository. • R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk. • R3: The mission operation controller shall transmit any detected anomalies with the user help desk. 26
- 27. Challenge#2 -" Capture Change Rationale • R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository. • R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk. • R3: The mission operation controller shall transmit any detected anomalies with the user help desk. 27
- 28. • R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository. • R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk. • R3: The mission operation controller shall transmit any detected anomalies with the user help desk. 28 Challenge#2 -" Change Rationale Rationales: R1: We want to globally rename “user help desk” R2: Avoid communication between “mission operation controller” and “user help desk” R3: We no longer want to “transmit satellite status reports” to “user help desk” but instead to “user document repository”
- 29. Solution Characteristics • Accounts for the phrasal structure of requirements 29 The mission operation controller shall transmit satellite status reports to the user help desk document repository. user help desk, Deleted user document repository, Added • Account for semantically-related phrases that are not exact matches and close syntactic variations
- 30. Approach 30 Rationale: Avoid communication between mission operation controller and user help desk. Propagation condition: mission operation controller AND user help desk AND transmit
- 31. RQ1 - Which similarity measures are best suited to our approach? • Experimented with 10 syntactic, 9 semantic measures, and all their pairwise combinations (109 combinations) 31
- 32. RQ2 - How should analysts use the sorted requirements list produced by our approach? 32
- 33. RQ3 - How effective is our approach? • Extra requirements traversed • Case-A between 1%-7% • Case-B between 6%-8% except one case • Number of impacted requirements missed: " 1 out of 106 33
- 34. Requirements-Design 34 Archi. & Design Requirements [TOSEM 2014, IST 2012, FSE 2011, HASE 2011]
- 35. Context • Context: Certification of safety-critical monitoring applications (fire and gas detection and emergency and process shutdown) in oil & gas industry • Certification: Assessing and discussing software requirements, design/architecture and implementation documents • Typically, many meetings taking place over 6 to 18 months 35
- 36. Observations • Analyzed 66 distinct certification issues: • Issues collected through observing certification meetings at different suppliers of maritime and energy systems • Meetings focused on requirements, architecture, and design documents 36 Expensive" to fix
- 37. Research Objective • Developing a model-based traceability methodology • Generate a sound and yet minimal design slice for a given safety requirement, to support safety inspections • Slices constructed based on traceability links established between safety requirements and design 37 Design Slice
- 38. Research Approach 38 Traceability Methodology to relate safety requirements to design Slicing Algorithm to extract a design slice relevant to a given safety requirement Model Driven Engineering (MDE) is the enabler
- 39. Modeling • System Modeling Language (SysML) • A subset of UML extended with system engineering diagrams • A standard for system engineering • Preliminary support for requirement analysis and built-in traceability mechanism 39
- 40. Is SysML enough? • Do we have proper guidelines for establishing traceability links between requirements and design? • SysML is only a notation and needs a methodology • Are the built-in SysML traceability links capable of addressing certification traceability issues? • New traceability links: Source and assumptions of sys. safety reqs. • We specialized the semantics of existing ones: Refine, decompose, derive … • Explicit and implicit links 40
- 41. Research Approach 41 Traceability Methodology to relate safety Requirements to design Slicing Algorithm to extract a design slice relevant to a given safety requirement
- 43. Traceability Information Model 43 Structural relations
- 46. Requirement to Design Traceability 46 • Mappings are documenting the design rationale! • Implications relations between phrases and block states and operations
- 47. Research Approach 47 Traceability Methodology to relate safety Requirements to design Slicing Algorithm to extract a design slice relevant to a given safety requirement
- 50. Slices 50 Fig. 11. The block and activity slices for the requirement in Fig. 8 extracted from the SysML design diagrams in Fig. 6. S. Nejati et al. / Information and Software Technology 54 (2012) 569–590
- 51. Slicing Algorithm • If a requirement holds over a design slice, it should also hold over the original design (soundness) • Proven analytically (formal proof) • If a requirement holds over the original design, then the design slice created for that requirement should conclusively satisfy that requirement (completeness) • Evaluated empirically (Case studies and experiments) 51
- 53. Case Study: SW/HW Interfaces 53 Interface Control Modules Hardware Communicates commands and data between control modules and hardware Goal: Practical guidelines to: (1) Capture the concurrent design of interfaces (2) Reduce the number and criticality of certification issues related to interfaces
- 54. Results • Created design models with traceability to requirements • One context diagram (BDD), One architecture diagram (IBD), One detailed structure diagram (BDD), One activity decomposition diagram (BDD), One overall activity diagram, 19 detailed activity diagrams • Created 65 traceability links for 30 safety-relevant requirements • Modeling effort was approximately 40 person-hours • Model Slicing • Extracted 34 block slices and 31 activity slices • Slicing reduced the number of block operations by 70% and the number of activity nodes by 50% 54
- 55. Controlled Experiment • Question: Do safety slices help find design issues? • Conducted in a laboratory setting with master students • Overall design • Seeded faults into the design • Incorrect behavior and structure • Divided the subjects into two groups • One group gets the design without slices • One group gets the design plus the relevant slices 55
- 56. Experiment Results • Slices show strong benefits in terms of: • Increasing the correctness of inspection decisions • Decreasing the proportion of uncertain decisions • Reducing the effort of inspections 56
- 57. Recent Similar Experiment • Do developers benefit from requirements traceability when evolving and maintaining a software system? Patrick Mäder, Alexander Egyed • Empirical Software Engineering (Springer), 2015 • Focus on program comprehension and maintenance • Tasks with and without traceability • Traceability led to 24% speed improvement and 50% better correctness 57
- 59. Context • Context: Automotive, sensor systems • Traceability between system requirements and test cases • Mandatory when software must undergo a certification process (e.g. ISO 26262) • Customers require such compliance • Use-case-centric development 59
- 60. Automated Test Generation • Restricted use case specifications: Structure, templates, restricted natural language (RUCM) • Domain modeling • Constraints • Automation combines Natural Language Processing and constraint solving • Automated test generation comes with traceability between use case flows and system test cases 60
- 61. 5 Specify Constraints OCL constraints Errors.size() = 0 ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Status <> null t > 0 and t < 50 Identify Constraints 4 Generate Abstract Test Cases 6 Generate Platform Specific Test Cases 7 Evaluate Completeness 3 Elicit Use Cases 1 Model the Domain 2 Domain Model Missing Entities List of Constraint descriptions Abstract Test Cases THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND RUCM Use Cases Platform Test Cases Mapping Table
- 62. Case Study Results • Rewrote 6 use case specifications of BodySense • 48 constraints to specify Effectiveness Applicability • Automatically generated test cases for 6 use cases • Specific test strategy (Rationale) • Approach covers more scenarios than manual testing: 100 versus 86 • Automated testing covers alternative flows not covered by manual testing
- 63. Discussion • Modeling effort reasonable after initial training • Main challenge is writing OCL constraints. • Test generation time took about 12 min per test cases, mostly due to constraint solving • Engineers miss important test scenarios because • Path analysis across multiple use case specifications is difficult • Regular use case specifications are less precise that RUCM
- 64. Regulations - Requirements 64 Requirements Regulations [RE 2014, MODELS 2014] • New tax system under development • System needs to be compliant with the law and remain so over time
- 65. Solution Overview 65 Test cases Actual software system Traces to Traces to Analyzable interpretation of the law Generates Results match? Impact of legal changes Simulates
- 66. Example 66 Art. 105bis […]The commuting expenses deduction (FD) is defined as a function over the distance between the principal town of the municipality on whose territory the taxpayer's home is located and the place of taxpayer’s work. The distance is measured in units of distance expressing the kilometric distance between [principal] towns. A ministerial regulation provides these distances. Interpretation + Traces
- 67. Example 67 The amount of the deduction is calculated as follows: If the distance exceeds 4 units but is less than 30 units, the deduction is € 99 per unit of distance. The first 4 units does not trigger any deduction and the deduction for a distance exceeding 30 units is limited to € 2,574. Interpretation + Traces
- 68. Discussion • We addressed the gap between legal experts and IT specialists • Models understandable by both legal experts and IT specialists • Modeling effort was considered reasonable given the life span of such eGovernment systems • Traceability to the law was considered a significant asset given frequent and complex changes in the law 68
- 69. Conclusions
- 70. Conclusions • From an economic standpoint, • the accuracy of trace recovery techniques cannot be interpreted out of context • what traceability information to capture is a trade-off • benefits depend on context • More human studies are required to assess cost-benefits • Design of such studies is not easy: baseline of comparison, comparable tasks, training, comparable skills … 70
- 71. Conclusions • Change impact analysis among requirements was surprisingly accurate • Change rationale needed to be captured • But this is expected to depend on requirements writing practice, e.g., precision and consistency • Accurate inter-requirements traces may require capturing tacit dependencies between domain concepts, e.g., domain model • What type of domain model do we need? Ontologies? • Can accuracy be improved through the use of NL templates? 71
- 72. Conclusions • Requirements-design traces require a precise design methodology, including practical mechanisms to capture design rationale and link it to requirements • Documenting design rationale cannot be automated, but can be facilitated • Questions, in each new context: • What is the right Modeling methodology? • What is the right trace granularity? • What information do traces need to carry? 72
- 74. Traceability is an economic decision
- 75. Context Matters!
- 76. Natural Language Requirements • [RE 2015] C. Arora et al., Change impact analysis for natural language requirements: An NLP approach • [TSE 2015] C. Arora et al., Automated Checking of Conformance to Requirements Templates using Natural Language Processing • [ESEM 2014] C. Arora et al., Improving requirements glossary construction via clustering • [ESEM 2013] C. Arora et al., Automatic checking of conformance to requirements boilerplates via text chunking Requirements-Driven Testing • [ISSTA 2015] C. Wang et al., Automatic Generation of System Test Cases from Use Case Specifications 76
- 77. SysML Traceability and Safety Analysis • [TOSEM 2014] L. Briand et al., Traceability and SysML design slices to support safety inspections: A controlled experiment. • [IST 2012] S. Nejati et al.A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety Certification: Framework, Tool Support, and Case Studies • [HASE 2011] M. Sabetzadeh et al., Using SysML for Modeling of Safety--Critical Software-- Hardware Interfaces: Guidelines and Industry Experience • [FSE 2011] D. Falessi et al., SafeSlice: a model slicing and design safety inspection tool for SysML. Legal Modeling and Analysis • [MODELS 2014] G. Soltana et al., “ UML for Modeling Procedural Legal Rule” • [RE 2014] M. Adedjouma et al., “Automated Detection and Resolution of Legal Cross References” 77
- 78. .lusoftware verification & validation VVS Traceability Beyond Source Code: An Elusive Target? Lionel Briand Interdisciplinary Centre for Security, Reliability and Trust (SnT) University of Luxembourg SST, Florence - May 17, 2015