9 Common Validation Errors EduQuest Advisory
2 likes927 views
The document analyzes 1,720 validation observations from a project to identify common validation errors. The top nine most frequent errors were: missing information, inconsistency, lack of detail, traceability issues, vague wording, unverifiable test results, GDP issues, incomplete testing, and ambiguity. Specifications, test scripts, validation plans, test plans, trace matrices, and test results were the most vulnerable documents. Identifying and addressing common errors can help companies prepare for FDA inspections.
9 Common Validation Errors EduQuest Advisory
- 1. The 9 Most Common Validation Errors: TABLE OF CONTENTS IDENTIFY FREQUENT DEFICIENCIES TO Applying Pareto Analysis to ACCELERAT E YOUR VALIDATION PROJECTS Common Validation Problems.... 2 Fig. 1: Top Finding Categories... 2 BY FRANK HOUSTON 13 Additional Observation Senior Validation Consultant, EduQuest, Inc. Categories .................................. 5 Identifying the Most Vulnerable Documents ................................ 8 Lessons Learned from 1,720 Observations Fig. 2: Top Document Types...... 8 What validation problems are you likely to see over and Fewer Validation Problems and over? When tackling complex validation challenges, you’ll Inspection Success Go Hand-in- save time, money and headaches when you know the most Hand........................................... 9 common problems and where to find them. About the Author ...................... 10 The following analysis is based on validation work EduQuest performed for a large FDA-regulated company over the past year. The goal was to bring the company’s software validation evidence up to the level of the U.S. FDA’s current expectations as well as those of the client’s own independent auditor. Our efforts yielded 1,720 observations. As part of a “lessons 1896 Urbana Pike, Suite 14 Hyattstown, MD 20871 USA learned” review, we grouped the observations into 22 (Near Washington, DC) different categories and, in the second part of our analysis, Phone: +1 (301) 874-6031 identified which documents most frequently contained the Email: Info@EduQuest.net observations. The results – in our experience – are typical www.EduQuest.net of the problems most companies face. © EduQuest 2010
- 2. Applying Pareto Analysis to Common Validation Problems Finds 9 Types of Deficiencies Through Pareto analysis of the categories of problems, we discovered that about 80% of our observations were clustered around nine types of deficiencies plotted on Figure 1. Our case was an exception to the 80/20 rule in that the top nine problem areas represented about 41% of our categories. FIGURE 1 Top Finding Categories 25.00% 90.00% % of Observations 80.00% 20.00% 70.00% Cumulative % 60.00% 15.00% 50.00% 10.00% 40.00% 30.00% 5.00% 20.00% 10.00% 0.00% 0.00% ilit y i ng cy us P . t.. . . a.. s.. . GD e. . ten uo ord ab tes or m Te ed big sis ce ew le te ne inf Am Tra on ab pl e gu of ng Inc r ifi om Va ck ssi ve La Inc Mi Un The most frequent deficiencies we found were: 1. Missing Information – Documents or records omitted fundamental information or content that should have been included. 2. Inconsistency – Documents contained statements inconsistent with other statements about the same topic THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 2
- 3. in the same document or in the same validation package. What’s more, no explanation or reason was given for the difference. We found that jargon, varying terminology, and contradictions in logic frequently caused these kinds of inconsistencies. 3. Lack of Needed Detail – This deficiency applied mostly to requirements documents. The requirements in the validation package did not adequately describe the characteristics of 1) data, 2) user interactions with Jargon and varying business processes, or 3) key processes internal to the software. terminology 4. Traceability – We found three frequent traceability frequently cause problems: validation The traceability matrix did not account for a inconsistencies. traceable specification or an observation step in a test script. The trace was broken. Either a requirement was barren (lacked decedents or a test) or one of the detailed requirements or test results was an orphan (lacked a parent somewhere in the requirement tree). The traceability matrix was incomplete. Requirement details were not explicitly numbered and traced to associated test steps. Requirements were not traced at a detailed level, so the reviewer needed to infer the detailed links between specifications and steps in a test script. 5. Vague Wording – Documents used generalities such as “in accordance to an approved procedure”, or “applicable regulatory requirements”, or “all associated THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 3
- 4. GxP and business processes”. In addition, documents used vague words such as “may”, “possibly”, “more or less”, and “approximately”. 6. Unverifiable Test Results – Expected results were not described sufficiently so that an independent reviewer could compare and verify actual results. The IEEE Standard for Software Test Documentation, Std. 829.1988, Clause 6.2.4 says you should, “...provide the exact value (with tolerances where appropriate) for each For executed required output or feature”. For executed scripts, actual scripts, actual results were not recorded or captured in a way that allowed an independent reviewer to compare them to results often were expected results. For example, “OK” was noted in the not recorded to actual-result column with no reference to a screen shot. allow comparison 7. GDP – We found three frequent Good Documentation Practice problems: with expected Hand-recorded data and testing evidence, such as results. test results, were presented in a way that could cause doubts about their authenticity (for example, cross-outs without initials, date, and reason). Data that confirmed a specific requirement was hard to find in the evidence provided (for example, a busy screen shot crammed with data). Handwritten corrections were made that changed the sense of a requirement or an expected test result, but no discrepancy report or change request was filed (for example, changing an expected result from indicator “Off” to “On”). In GDP, hand corrections are allowed without additional documentation only for obvious THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 4
- 5. typographical errors, such as dropped or transposed letters (for example, correcting “th” or “teh” to “the”). 8. Incomplete Testing – Test scripts did not fully or adequately test the associated requirement. 9. Ambiguity – Text could be interpreted more than one way, so it did not establish a single, unique requirement. The words “either” and “or” in a requirement are strong The words “either” clues the text is ambiguous. or “or” in a requirement are 13 Additional Observation Categories strong clues the Beyond these top nine categories of deficiencies, we text is ambiguous. identified 13 other categories from our observations. One might consider our category definitions were somewhat subjective, but for this sort of analysis the objectivity of the definitions was less important than consistency in classifying the observations. For this reason, we reviewed all the classifications several times before locking in the data for our lessons-learned pivot tables. Even so, we noted that between the “Ambiguous” and “Vague Wording” classifications, many observations could have fit in either one. To help you perform your own analysis, here are the additional categories of deficiencies we identified – ones that did not rise to the level of our most common findings but were still worth noting: Compound Requirement — Requirements that were not unique; that is, the requirement statement actually stipulated two or more system characteristics. (When the predicate of a requirement sentence contains “and” or a series of commas, or when the requirement is presented THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 5
- 6. as a compound sentence or series of bullets, it’s probably a compound requirement. This deficiency was often coupled with traceability problems.) For Your Information – Here we included comments on the potential to improve a document or process. The issue that generated the comment may or may not have had an impact on a determination of “substantial compliance.” We also included remarks on particularly good examples of documentation or development Risk assessment practice. indicated a need Incomplete Requirements – Our findings in this for requirements category fell into four subcategories: that were 1. The requirement in question implied another requirement, possibly complementary, that missing from the needed to be explicit to ensure verification. URS. 2. Regulatory impact analysis and risk assessment indicated a need for requirements that were missing from the User Requirement Specification (URS). 3. Requirements in a Software Requirements Specification (SRS), a System Design Specification (SDS) or a Configuration Specification (CS) were not sufficient to address the associated URS item. This deficiency was often associated with a broken trace. 4. System and business process analyses indicated the software had functionality that was used but had not been described in the URS. Rationale – Statements or assertions were made without supporting rationale or justification. Or, the THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 6
- 7. rationale or justification for a particular statement or assertion was not persuasive. Lack of Acceptance Criteria – Test and validation plans did not establish objective criteria based on the outcomes of various tasks in the validation process, such as vendor audit, testing, and problem resolution. The plans did not include criteria for assessing the seriousness of deviations as a basis for the overall evaluation and acceptance or rejection of the test and Test plans did not validation results. include objective Lack of Process for Resolving Deviations – A plan, criteria for protocol or script lacked a process for resolving deviations (for example, failure to meet expected test assessing the results, discovery of unanticipated behavior, or seriousness of deviations from Good Documentation Practices). deviations. Questionable Statement – A statement appeared to be inaccurate or incorrect. Redundant Requirement –The same requirement appeared more than once in a specification document. Topical Inconsistency – The text within a topic pertained to a different topic. Typo – Typographical errors were observed. Unsupported Deviation – The summary document omitted reporting on differences between planned activities and those that were actually carried out. Untestable Requirement – The requirement was not presented in objective, observable or measurable terms. In other words, the requirement did not describe a THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 7
- 8. system response or characteristic that a reasonable person could sense or measure. Violation – The text set up or highlighted a violation of procedures or regulations. Consider all these categories nothing more than suggestions or starting points for your own list of observations. As you gain experience, you’ll inevitably revise or cull out some categories and identify new ones. Identifying the Most Vulnerable Documents Taking the next step to document the lessons learned from this project, we then categorized the documents and records where we found the most frequent deficiencies. We discovered that about 85% of findings were concentrated in six key documentation areas, as shown in Figure 2. FIGURE 2 Top Document Types 25.00% 120.00% % of Observations 20.00% 100.00% Cumulative % 80.00% 15.00% 60.00% 10.00% 40.00% 5.00% 20.00% 0.00% 0.00% Te Pl an ste t Re tr ix T e e Ma n di t Ve ss. . 1 tio r ipt Va s t S s Va m D sults P A Pa .. st on S . .. la sse r t 1 Te ati on ti o . . dia ar.. Au nS ati sc ri . Re Sum . .. Tra s t P c n m or e i fic c nd l ida ec s me Sp l id Gx Te Sy THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 8
- 9. The top types of flawed documentation were: 1. Specifications (including User Requirements) 2. Test Scripts 3. Validation Plans 4. Test Plans 5. Trace Matrix Companies who 6. Test Results reduce the Although the exact order of problem areas may differ in frequency of these your organization, it’s likely these same six documentation deviations are areas will float to the top. From our experience, specification documents are usually the biggest pitfall for more likely to most companies. weather FDA inspections. Fewer Validation Problems and Inspection Success Go Hand-in-Hand After auditing many companies, large and small, and participating in countless remediation projects, EduQuest has found the results described above typical of companies worldwide. More importantly, we also have seen first-hand that companies who reduce the frequency of these problems with focused remediation efforts are much more likely to weather future FDA inspections. You can reasonably assume the same would be true if the frequency of such problems were low in the first place. We recommend you use these results and definitions to assess your own validation projects, or devise your own THE 9 MOST COMMON VALIDATION ERRORS WWW.EDUQUEST.NET 9
- 10. categories and charts to pinpoint your company’s most common problems. Either way, you’ll have a major head- start in better allocating validation resources and making needed improvements quickly. About the Author Melvin F. (Frank) Houston is a Senior Validation Consultant with EduQuest, Inc., a global team of FDA compliance experts based near Washington, D.C. Frank is a recognized authority on ISO 9000 Quality Standards and Quality System Regulation. He has particular expertise in the validation and auditing of computer-controlled systems in regulated industry. He also is considered an expert in software quality assurance and a champion of software/ system safety. Before joining EduQuest, Frank served 12 years with the U.S. FDA Center for Devices and Radiological Health (CDRH), where he influenced the formation and execution of FDA software policies for medical devices and Good Manufacturing Practices. He also develops instructional materials for EduQuest’s popular “FDA Auditing of 1896 Urbana Pike, Suite 14 Computerized Systems and Part 11” training course, Hyattstown, MD 20871 USA which is offered as an open enrollment course several times (Near Washington, DC) each year throughout the U.S. and Europe. The highly Phone: +1 (301) 874-6031 interactive course — which further explains FDA’s rules Fax: +1 (301) 874-6033 and expectations for systems validation — also is available Email: Info@EduQuest.net for on-site, on-demand delivery. For course details, email www.EduQuest.net MartinHeavner@EduQuest.net. © EduQuest 2010