Agile methods and safety critical software - Peter Gardner
Download as PPT, PDF7 likes6,466 views
The document discusses the compatibility of agile methods with safety-critical software development, focusing on certification criteria and evidence of agile benefits. It evaluates various agile frameworks, highlights necessary adaptations for safety-critical contexts, and summarizes empirical findings related to agile implementations. Ultimately, it concludes that while agile methods can be applied, they require significant modifications, particularly in documentation, traceability, and role independence to meet safety standards.
![What the Presentation is About
• (1) Can Agile methods be applied to
safety-critical software and the software still
be rigorously built and meet certification
criteria?
[compatibility].
• (2) What evidence is there for the benefits
of Agile methods, especially as regards
safety-critical software ?
[cost, quality, time etc]
Page 5](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-5-320.jpg)
![Views on Design,
Documentation, Traceability
• Stojanovic et. al. : “[According to] their proponents software code is the main
deliverable, while the role of system analysis, design and documentation in
software development and maintenance is de-emphasised and to some extent
ignored”.
• Chenu : “without any adaptation, XP’s set of by-the-book practices does not
provide the formalism and the proofs required for certification
inspections….documentation must be considered as part of the product”
• Chenu : “we’ve added the continuous traceability practice”
• ITEA Deliverable D.1 :“Agile development methods and practices as applied
for classical software development cannot identically be copied for embedded
real-time software development”.
• ITEA Deliverable D.1 : in large, complex systems “a predefined architecture is
needed…refactoring is not enough, a design is needed”.
• ITEA Deliverable D.1 : in safety-critical systems“emphasis is placed [on]
requirements traceability and thus extra documentation is required”.
Page 22](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-22-320.jpg)
![Views on Design,
Documentation, Traceability
• ITEA Deliverable D.2.12 : “developers of mission-critical airborne software are
heavily constrained by the RTCA DO-178B regulations. These regulations impose
strict rules regarding traceability and documentation that make it extremely hard
to employ an iterative software development process” [did they really mean
this?]
• ITEA Deliverable D.2.12 : “the mission-critical nature of this software has lead
to stringent procedures and plans that could specifically exclude the use of Agile
methods”
• Wils and Van Baelen : “Agility potential is inherently lower for DO-178B
compliant projects”
• Thomas : “beware Agile methods…dangerous where…the system is safety-
critical” [says use Z and SPARK etc]
• Ambler : would be “leery of applying Agile modelling to life-critical systems”.
• Somerville : “critical system engineering needs Agile approaches to
development”
• Somerville : “XP includes a set of good practices that have the potential to
contribute to critical systems engineering” but “some of these need to change
and new practices need to be included in the XP process”
Page 23](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-23-320.jpg)
![Empirical Evidence(2)
• My web search found two papers (which
were themselves surveys) :
• Abrahamsson [7] found 36 papers with
some evidence
• ITEA (Information Technology for European
Advancement) project deliverable D.1
contained three studies
Page 26](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-26-320.jpg)
![References (non-safety-critical)
[1] Modeling and Architectural Design in Agile Development Methodologies, Stojanovic, Dahanayake, Sol
http://www.emmsad.org/2003/Final%20Copy/27.pdf
[2] Agile in Embedded Software Development :
State-of the-Art Review in Literature and Practice ITEA Deliverable D.1
http://www.agile-itea.org/public/deliverables.php
[3] A Practical Guide to Seven Agile Methodologies, part 1, Rod Coffin and Derek Lane
http://www.devx.com/architect/Article/32761/1954
[4] A Practical Guide to Seven Agile Methodologies, part 2, Rod Coffin and Derek Lane
http://www.devx.com/architect/Article/32836/1954
[5] Empirical Findings in Agile Methods, Lindvall et. al.
http://www.cs.umd.edu/~mvz/pub/agile.pdf
[6] The Agile Unified Process, Scott Ambler
http://www.ambysoft.com/unifiedprocess/agileUP.html
[7] Pealing the Hype into Pieces : What do we Really Know About Agile in Research and Practice ?
Pekka Abrahamsson,VTT Technical Research Centre of Finland
http://www.agile-itea.org/public/papers/OLIO_abrahamsson.pdf
[8] Limitations of Agile Software Processes, Turk, France, Rump
http://www.agilealliance.org/system/article/file/1096/file.pdf
[9] The Cockburn Scale
http://en.wikipedia.org/wiki/Cockburn_Scale
[10] Manifesto Elicits Cynicism, Steven Rakitin Computer, December 2001
http://sunset.usc.edu/events/2002/arr/letters.pdf
Page 31](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-31-320.jpg)
![References (safety-critical)
[11] Agility and Lean for Avionics, Emmanuel Chenu
http://manu40k.free.fr/AgilityAndLeanForAvionics1.pdf
[12] ITEA Deliverable D.2.12 Towards An Agile Avionics Process, Wils and Van Baelen
http://www.agile-itea.org/public/deliverables/ITEA-AGILE-D2.12_v1.0.pdf
[13] Agility in the Avionics World, Wils and Van Baelen
https://lirias.kuleuven.be/bitstream/123456789/133945/1/2006-Sofia1.pdf
[14] Critical Software : The Need For A Radical Solution, Martyn Thomas
http://www.thomas-associates.co.uk/Critical%20Software.pdf
[15] Get Ready for Agile Methods, With Care, Barry Boehm, IEEE Computer 2002
http://www2.umassd.edu/swpi/xp/papers/r1064.pdf
[16] Extreme Programming for Critical Systems, Ian Sommerville
http://www.cs.st-andrews.ac.uk/~ifs/Talks/XPForCritSys.pdf
[17] Safety-Critical Applications Built via Agile Discipline, Nancy van Schooenderwoert
http://www.boston-spin.org/slides/boston_spin_slides_2008_09.pdf
Page 32](https://image.slidesharecdn.com/agilemethodsandsafetycriticalsoftware-100412105721-phpapp01/85/Agile-methods-and-safety-critical-software-Peter-Gardner-32-320.jpg)
Agile methods and safety critical software - Peter Gardner
- 1. AGILE METHODS AND SAFETY-CRITICAL SOFTWARE Are They Compatible ? Peter Gardner Technical Consultant
- 2. SILVER-ATENA’s Business • Software development principally in the rail and avionics sectors • Consultancy services • Provides skilled software engineering staff to work on clients’ premises • Full system development capability through other partner companies in the same group • Group offices in Malmesbury, Bangalore, Madrid and Munich Page 2
- 3. SILVER-ATENA Agile Club • Established August 2009 • Purpose : investigate the use of Agile methods in safety-critical software development • “Client” is the Open-DO wider community Page 3
- 4. SILVER-ATENA Agile Club • Average attendance 12 people, internal and external • Total attendance 32 : consultants, software engineers, project managers, quality department • Videos, Powerpoint, discussion Page 4
- 5. What the Presentation is About • (1) Can Agile methods be applied to safety-critical software and the software still be rigorously built and meet certification criteria? [compatibility]. • (2) What evidence is there for the benefits of Agile methods, especially as regards safety-critical software ? [cost, quality, time etc] Page 5
- 6. What the Presentation is About • For (1), this paper will look at, in particular, DO-178B and EN 50128, for the aviation and rail sectors respectively (level A/B, SIL 3/4) • For (2), this paper will summarise the results of a websearch looking for evidence of the benefits of Agile • I will also look at some opinions from people who work in the sector Page 6
- 7. Categories of Source Papers • Papers on Agile methods and non-safety- critical software (10) • Papers on Agile methods and safety-critical software (7) Page 7
- 8. Different Variants of Agile • XP • Adaptive Software Development • Crystal (family) • Dynamic Systems Development Method • Scrum • Agile Unified Process • Feature-Driven Development Page 8
- 9. Features of Agile Methods • Short release cycles • Incremental requirements development • Customer presence in project • Continuous integration • Customer determines functions in each release cycle • Generally for small teams (up to about 12) • More difficult to apply to larger teams • Adapt for each project • Multi-skilled team members (possibly) • Any team member can change anything Page 9
- 10. Features of Agile Methods • Pair-programming (XP) • Test-driven development • Retrospectives • Larger teams should have shorter cycles (!) • Refactoring • Less emphasis on design and documentation • Pair-programming (XP) • Self-organising teams • Project lasts as long as customer gives high-value requirements • Departments as service providers Page 10
- 11. Yes, mostly • Pass on 15 out of 20 • Maybe on 3 out of 20 • No, but not serious, for 1 out of 20 : anyone can change anything • But there is a real problem with one item : reduced emphasis on design and documentation Page 11
- 12. Features of Agile Methods • Short release cycles • Incremental requirements development • Customer presence in project • Continuous integration • Customer determines functions in each release cycle • Generally for small teams (up to about 12) • More difficult to apply to larger teams • Adapt for each project • Multi-skilled team members (possibly) • Any team member can change anything Page 12
- 13. Features of Agile Methods • Pair-programming (XP) • Test-driven development • Retrospectives • Larger teams should have shorter cycles • Refactoring • Less emphasis on design and documentation • Change company structure (ideally) (departmental to project structure) • Self-organising teams • Project lasts as long as customer gives high-value requirements • Departments as service providers Page 13
- 14. DO-178B Outputs(1) • Plan for Software Aspects of Certification • Software Development Plan • Software Verification Plan • Software Configuration Management Plan • Software Quality Assurance Plan • Software Requirements Standards • Software Design Standards • Software Code Standards • Software Requirements Data Page 14
- 15. DO-178B Outputs(2) • Software Design Description • Source Code • Executable Object Code • Software Verification Cases and Procedures • Software Verification Results • Problem Reports • Software Configuration Management Records • Software Quality Assurance Records • Software Accomplishment Summary Page 15
- 16. EN 50128 Outputs (part 1) • System safety plan • Software Quality Assurance Plan • Software Configuration Management Plan • Software Verification Plan • Software Integration Test Plan • Software /Hardware Integration Test Plan • Software Validation Plan • Software Maintenance Plan • Data Preparation Plan • Data Test Plan Page 16
- 17. EN 50128 Outputs (part 2) • Software Requirements Specification • System Safety Requirements Specification • Software Requirements Test Specification • Software Architecture Specification • Software Design Specification • Software Module Design Specification • Software Module Test Specification • Software Source Code Page 17
- 18. EN 50128 Outputs (part 3) • Software Module Test Report • Software Integration Test Report • Software Requirements Verification Report • Software Module Verification Report • Software Architecture and Design Verification Report • Software Source Code Verification Report • Software/Hardware Integration Test Report • Software Validation Report • Software Assessment Report • Software Change Records Page 18
- 19. Documents and Contents • Its not just a case of producing documentation • Much more importantly, one has to produce the contents of the documents i.e software development artefacts • For example, architectural design, low-level design, test cases etc Page 19
- 20. Traceability • Both DO-178B and EN 50128 require traceability through the lifecycle, yet : • Agile methods never even consider traceability Page 20
- 21. Independence of Roles • DO-178B and EN 50128 also require (for higher levels) some of the roles to be independent eg. implementor and tester • This constraint also has to be factored into Agile methods Page 21
- 22. Views on Design, Documentation, Traceability • Stojanovic et. al. : “[According to] their proponents software code is the main deliverable, while the role of system analysis, design and documentation in software development and maintenance is de-emphasised and to some extent ignored”. • Chenu : “without any adaptation, XP’s set of by-the-book practices does not provide the formalism and the proofs required for certification inspections….documentation must be considered as part of the product” • Chenu : “we’ve added the continuous traceability practice” • ITEA Deliverable D.1 :“Agile development methods and practices as applied for classical software development cannot identically be copied for embedded real-time software development”. • ITEA Deliverable D.1 : in large, complex systems “a predefined architecture is needed…refactoring is not enough, a design is needed”. • ITEA Deliverable D.1 : in safety-critical systems“emphasis is placed [on] requirements traceability and thus extra documentation is required”. Page 22
- 23. Views on Design, Documentation, Traceability • ITEA Deliverable D.2.12 : “developers of mission-critical airborne software are heavily constrained by the RTCA DO-178B regulations. These regulations impose strict rules regarding traceability and documentation that make it extremely hard to employ an iterative software development process” [did they really mean this?] • ITEA Deliverable D.2.12 : “the mission-critical nature of this software has lead to stringent procedures and plans that could specifically exclude the use of Agile methods” • Wils and Van Baelen : “Agility potential is inherently lower for DO-178B compliant projects” • Thomas : “beware Agile methods…dangerous where…the system is safety- critical” [says use Z and SPARK etc] • Ambler : would be “leery of applying Agile modelling to life-critical systems”. • Somerville : “critical system engineering needs Agile approaches to development” • Somerville : “XP includes a set of good practices that have the potential to contribute to critical systems engineering” but “some of these need to change and new practices need to be included in the XP process” Page 23
- 24. Views on Design, Documentation, Traceability • Van Schooenderwoert : “Agile doesn’t treat safety-critical software differently; all is maximum quality” • Turk et.al. : “formal specification, rigorous test coverage, and other formal analysis and evaluation techniques included in software engineering approaches provide better, but also more expensive, mechanisms to tackle the development of safety- or business-critical software” • Rakitin : “it seems to me that this is nothing more than an attempt to legitimise hacker behaviour” Page 24
- 25. Empirical Evidence(1) • We have seen some opinions • Is there any actual data ? Page 25
- 26. Empirical Evidence(2) • My web search found two papers (which were themselves surveys) : • Abrahamsson [7] found 36 papers with some evidence • ITEA (Information Technology for European Advancement) project deliverable D.1 contained three studies Page 26
- 27. Empirical Evidence(3) • Abrahamsson : online survey, 36 papers with empirical evidence : - a reduction in release time of 20% to 75% - a pre-release reduction in defects of 40% to 65% - a post-release reduction in defects of 30% to 40% - a productivity improvement of 15% to 70% Not known if any of these were safety-critical projects Page 27
- 28. Empirical Evidence(4) • ITEA (1) “has the adoption of Agile processes altered the quality of applications?” significantly better 36%, better 53%, unchanged 11%, somewhat worse 1% (2) “has the adoption of Agile processes altered the cost of development?” much less expensive 5%, less expensive 44%, unchanged 46%, more expensive 5% (3) “has the adoption of Agile processes altered the level of business satisfaction with the software ?” significantly better 26%, better 57%, unchanged 16%, somewhat worse 0%, much worse 1%. (4) “what proportion of projects are appropriate for Agile processes?” all 16%, most 50%, half 22%, some 7%, none 5% • Only two out of seven specific projects reported in the literature were safety- critical projects Page 28
- 29. Summary • I have examined whether Agile methods are compatible with the production of safety- critical software • I have presented evidence from the literature of the benefits of Agile methods, particularly as regards safety-critical software Page 29
- 30. Conclusions • Agile methods, in general, need to be adapted for use in the area of safety-critical software development • Certification requirements mean that necessary documentation must be incorporated into any Agile process selected. • Traceability and role independence must also be considered • The logical extension of Agile methods to the safety-critical world requires continuous certification and a colocated certification authority • Release cycles are likely to be longer in safety-critical projects but there is no reason not to move towards the concept of frequent release cycles with an onsite customer • We need to gather more evidence, using reliable, standard metrics • Silver-Atena’s Agile club would be willing to assist in this work Page 30
- 31. References (non-safety-critical) [1] Modeling and Architectural Design in Agile Development Methodologies, Stojanovic, Dahanayake, Sol http://www.emmsad.org/2003/Final%20Copy/27.pdf [2] Agile in Embedded Software Development : State-of the-Art Review in Literature and Practice ITEA Deliverable D.1 http://www.agile-itea.org/public/deliverables.php [3] A Practical Guide to Seven Agile Methodologies, part 1, Rod Coffin and Derek Lane http://www.devx.com/architect/Article/32761/1954 [4] A Practical Guide to Seven Agile Methodologies, part 2, Rod Coffin and Derek Lane http://www.devx.com/architect/Article/32836/1954 [5] Empirical Findings in Agile Methods, Lindvall et. al. http://www.cs.umd.edu/~mvz/pub/agile.pdf [6] The Agile Unified Process, Scott Ambler http://www.ambysoft.com/unifiedprocess/agileUP.html [7] Pealing the Hype into Pieces : What do we Really Know About Agile in Research and Practice ? Pekka Abrahamsson,VTT Technical Research Centre of Finland http://www.agile-itea.org/public/papers/OLIO_abrahamsson.pdf [8] Limitations of Agile Software Processes, Turk, France, Rump http://www.agilealliance.org/system/article/file/1096/file.pdf [9] The Cockburn Scale http://en.wikipedia.org/wiki/Cockburn_Scale [10] Manifesto Elicits Cynicism, Steven Rakitin Computer, December 2001 http://sunset.usc.edu/events/2002/arr/letters.pdf Page 31
- 32. References (safety-critical) [11] Agility and Lean for Avionics, Emmanuel Chenu http://manu40k.free.fr/AgilityAndLeanForAvionics1.pdf [12] ITEA Deliverable D.2.12 Towards An Agile Avionics Process, Wils and Van Baelen http://www.agile-itea.org/public/deliverables/ITEA-AGILE-D2.12_v1.0.pdf [13] Agility in the Avionics World, Wils and Van Baelen https://lirias.kuleuven.be/bitstream/123456789/133945/1/2006-Sofia1.pdf [14] Critical Software : The Need For A Radical Solution, Martyn Thomas http://www.thomas-associates.co.uk/Critical%20Software.pdf [15] Get Ready for Agile Methods, With Care, Barry Boehm, IEEE Computer 2002 http://www2.umassd.edu/swpi/xp/papers/r1064.pdf [16] Extreme Programming for Critical Systems, Ian Sommerville http://www.cs.st-andrews.ac.uk/~ifs/Talks/XPForCritSys.pdf [17] Safety-Critical Applications Built via Agile Discipline, Nancy van Schooenderwoert http://www.boston-spin.org/slides/boston_spin_slides_2008_09.pdf Page 32
- 33. Thank you for your attention !