![Thank You Please contact me if you have any questions Dale Sanders Intermountain Health Care www.ihc.com 801-408-2121 [email_address]](https://image.slidesharecdn.com/softwaresafetyv5-12768223647144-phpapp02/85/Concepts-in-Software-Safety-31-320.jpg)
Concepts in Software Safety
- 1. Principles of Software Safety Sanders, Ver 4 Copyright 2002 Guidelines For Clinical Information Systems And The Electronic Medical Record (EMR)
- 2. Overview Premise Background Brief history of software safety Examples of software safety incidents Analysis concepts Characteristics of a “safe software” environment Principles for clinical information systems and the Electronic Medical Record (EMR)
- 3. Premise Clinical Information Systems and Electronic Medical Records (EMR) are safety-critical, software-based systems Lives are at stake if the software is “unsafe” These systems should be designed, developed, and maintained in similar fashion to other safety-critical software systems Transportation, medical devices, nuclear power, weapon systems
- 4. My Background: I’ve given this topic some thought… Air Force Nuclear Weapons Safety Program: 1989-1995 Peacekeeper and Minuteman Intercontinental Ballistic Missile (ICBM) launch and guidance control software safety analysis TRW Independent Research and Development (IR&D): 1992-1995 “ Assessing Software Safety and Risk in Commercially Developed Software” “ Predicting Software Safety and Reliability Using Expert Systems” Army Tactical Nuclear Weapons Safety Office: 1993-1994 Pershing Missile Software Safety Assessment for Unauthorized Launch National Security Agency (NSA): 1993-1995 U.S. Nuclear Command and Control Risk Assessment Food and Drug Administration:1995 “ Applying 510k Principles to Computerized Patient Records” Intermountain Health Care: 1997-2004
- 5. Characteristics of Safe Software Software is “safe” if… It has features and procedures which ensure that it performs predictably under normal and abnormal conditions The likelihood of an undesirable event occurring in the execution of that software is minimized If an undesirable event does occur, the consequences are controlled and contained “ Software Safety and Reliability” --D. Herrman
- 6. Key Concepts Not all software should be developed and tested using the same methodology An EMR vs. A web page for posting minutes Adjust your methodology according to your risks No software-based, safety-critical system is 100% “safe” The question is: How safe is safe enough? Software safety is a component of system safety Software safety must be evaluated within the context of the system it operates, including the humans that interact with that system
- 7. Drivers of Software Safety History 1960s Nuclear Intercontinental Ballistic Missile Program Apollo Space Program 1970s Space Transportation System (Space Shuttle) Food and Drug Administration Department of Transportation Department of Energy 1980s and 1990s Rapid increase in software dependence within all the above Control of safety critical computer systems moves from hardware-based logic to software-based logic Complexity in all of these environments increases
- 8. A Few Notable Examples Patriot Missile system in Gulf War Abnormal condition: Continuous operation, no “reboot” Chrysler Automotive Jeep Sudden acceleration transitioning from Park to Drive Therac-25 Radiation Therapy Dose calculation algorithm error Washington D.C. Metro Central control system failure Radiology report failed to display Missed diagnosis, delayed treatment for cancer
- 9. The Role of Risk Assessment Frequency How often is this bad event likely to occur? Probability of an event occurring during a given time frame Consequence The business impact of that bad event If possible, it should be measured in dollars Not always possible Could be measured in lives, customers lost, etc. Risk Ideally, expressed as dollars lost per unit of time Risk = Frequency x Consequence
- 10. Risk Prevention: Software Safety Analysis Two basic types of safety analysis techniques Event based: “If we made a mistake in this software, could it lead to a patient safety incident? If so, how so and how severe?” Consequence based: “What would happen if we failed to retrieve all the radiology reports for this patient?”
- 11. Contributing Factors to Safety Risks *-- From the FAA *
- 12. Risk and Software Safety Frequency How often will we experience a Patient Safety related event that is attributable to a software error? It a subset of your Software Defect Rate, assuming you are tracking the number of “bugs” found in your software over a given period of time Consequence The clinical impact of that software error If possible, it should be measured in dollars If not dollars, some other meaningful unit of consequence Lawsuits, readmits, LOS, sentinel events, etc. Risk Ideally, expressed as dollars lost per unit of time
- 13. Root Causes of Software Safety Violations Requirements specification and communication Single largest source of errors Software executes “correctly” according to the understanding of the requirement, but the requirement was wrong within the scope of the system The requirement was simply misunderstood by the programmer Design and coding errors Second most common source of errors Poorly structured code Timing errors, incorrect queries, syntax errors, algorithm errors, results display errors, lack of self-tests, failed error handling
- 14. Root Causes of Software Safety Violations (cont) Computer hardware induced errors Not as common, but possible Hardware logic errors caused by overheating, power transients, radiation, magnetic fields Software change control process Changes to software introduces unanticipated errors Can be traced back to requirements and programming errors Failure of the configuration control process Inadequate testing Software functions properly in unit testing Software passes systems and integration testing, but should not because safety-critical test coverage is inadequate Can be traced back to requirements and programming errors
- 15. Specific Techniques for Software Safety Analysis All with their roots in hardware-based systems But they can be applied effectively to software Failure Modes Effects Analysis “ If this software fails to return the correct lab value, what is the impact?” Fault Tree Analysis “ What are all the events that could cause this software to incorrectly display lab values?” Fault Hazard Analysis Typically uses a Fault Tree Analysis Also considers human factors and operational procedures Common Cause Analysis Looks across fault trees for common roots Sneak Circuit Analysis “ Stray” code that inhibits desired functions or causes undesired functions to occur
- 16. General Software Safety Scenarios Software fails to perform a required function Function not executed or answer never returned Software performs a function that is not required or intended Wrong answer returned or issues wrong control instruction Software performs the right function, but at the wrong time or under inappropriate conditions Software timing or sequencing failure Parallel executions fail Synchronous or time-dependent executions fail Software fails to recognize a hazardous condition and react accordingly Or, the software recognizes the condition but reacts improperly
- 17. Data Safety Areas Validity checks fail (or do not exist) before acting upon safety critical data Illegal or out of range parameters Failure during initializing, clearing, or resetting critical data Validation failure of data addresses, pointers, indices, and variables Incorrect relationships established between files and records Detecting, handling, and/or correcting errors during data transfers Protecting data from being deleted or inadvertently overwritten
- 18. Creating a “Safe Software” Environment What would an auditor from the FAA look for at Boeing?
- 19. Auditing for Software Safety Is at least a high-level risk assessment conducted for software safety during the requirements and design phase? Is the software testing and quality assurance process risk adjusted? Are the test and development environments adequate for identifying safety risks before they appear in production? Is there an emphasis on human computer interfaces (HCI) and their relationship to safety risks? Is there a well-documented Safety Event Response process when software safety defects are discovered? Is there a robust root cause discovery and communication process after a safety event has occurred? Is there a software safety defect reporting and tracking system? Are there similar principles but different safety risk analysis processes for software developed internally vs. purchased?
- 20. More Audit Areas Is there an understanding and appreciation among the software development staff for safety risks? Is there a clear nomenclature for characterizing software safety risk scenarios? Is there a nomenclature for categorizing software defects based on safety risk? Is there a software safety governance and oversight body? Is there a well-documented software engineering process for safety critical applications? Is independent validation and verification of software part of the development methodology? Is safety critical software more tightly controlled for versioning and configuration? Is there a certification program for software engineers that are allowed to develop and work on safety critical software?
- 21. Applying This To Clinical Information Systems and EMR’s
- 22. Relating Patient Safety To EMR Software Safety Step 1: Define the general categories of patient safety risk scenarios, regardless of cause Step 2: Define the relationship between these general risk scenarios and the ability for the EMR or clinical information system to contribute to these scenarios Step 3: Use a software testing and safety tracking system to measure against these risk scenarios “ This function or module of software could contribute to a Moderate patient safety risk scenario. We should design and test accordingly.” “ This is a Severe software defect. It must be repaired immediately.”
- 23. Potential Categories of Patient Safety Risk Scenarios Type 1: Catastrophic Patient life is in grave danger. The probability for humans to recognize and intervene to mitigate this event is very low or non-existent. Intervention is required within seconds to prevent the loss of life. Type 2: Severe Patient health is in immediate danger. The probability for humans to recognize and intervene to mitigate this event is low, but possible. Intervention is required within minutes to prevent serious injury or degradation of patient health that could lead to the loss of life. Type 3: Moderate Patient health is at risk. However, the probability for humans to recognize and intervene to mitigate this event is probable. Intervention is required within hours or a few days to prevent a moderate degradation in patient health. Type 4: Minor Patient health is minimally at risk. The probability for humans to recognize and intervene to mitigate this event is high. Corrective action should occur within days or weeks to avoid any degradation in patient health.
- 24. Specific EMR Safety Risk Scenarios Errors in computerized protocols and decision support tools Invalid data posted to a patient record Valid data that is accidentally deleted Valid data that is not posted or not available Incomplete record Clinical data posted to the wrong patient record Right data, wrong patient Data that appears current and timely, but is not Their severity depends on the nature of the specific data or decision making context
- 25. When Developing and Testing… Software staff must ask: Does this software control or affect… Computerized protocols and decision support tools? Data that is posted to the EMR? Valid data that is not posted Incomplete record Clinical data posted to the wrong patient record Right data, wrong patient Timeliness of EMR data The deletion of EMR data? The performance or availability of the overall EMR? If so, the rigor of the software engineering process must increase accordingly.
- 26. Software Control vs. Safety Risk Does my software control any of these? If so, what is the probability that a defect could cause one of these scenarios? High Risk = Rigorous Design and Testing Catastrophic Severe Moderate Minor Computerized protocols and decision support tools Creating or updating data to the EMR Deleting data from the EMR Performance or availability of the overall EMR
- 27. Most to Least Safety Critical? GroupWise Transfusion Management HELP HELP2 Mysis Not necessarily the same as “Business Criticality”… For purposes of illustration… Increasing Safety Criticality and Software Engineering Rigor
- 28. For Illustration, Again… Software safety processes don’t apply Information System Business Criticality Data Sensitivity Safety Criticality Accudose 1 1 1 AGFA 1 1 1 Amicus 1 1 1 AS/400 Financial 1 1 4 Audit Log 2 4 4
- 29. In Conclusion There is growing need for software safety awareness in clinical information systems and EMR’s There are significant lessons learned from other industries We don’t have to reinvent the wheel To get started… Think like an FAA Software Safety Auditor Think like a patient Think like a physician
- 30. Acknowledgements Commercial aviation RTCA/DO-178B, Software Considerations in Airborne Systems and Equipment Certification European Committee for Electrotechnical Standards EN 51028, Software for Railway Control and Protection Systems Society of Automotive Engineers JA 1002 Software Safety and Reliability Program Standard U.S. FDA Center for Devices and Radiological Health Premarket Notification Submissions (510k) “ General Principles of Software Validation” U.S. FAA System Safety Handbook Appendix J: Software Safety “ Software Safety and Reliability” Debra S. Herrman “ Safeware: System Safety and Computers” Nancy G. Levison
- 31. Thank You Please contact me if you have any questions Dale Sanders Intermountain Health Care www.ihc.com 801-408-2121 [email_address]