Validation of qualitative lab test methods
Download as PPTX, PDF10 likes8,009 views
The document outlines the validation procedures for qualitative lab test methods, emphasizing the importance of establishing acceptance criteria for accuracy, precision, sensitivity, and specificity. It details various validation parameters including method comparison, reportable range, and reference range, along with standard calculations for accuracy and precision. Additionally, the document highlights requirements for FDA-cleared tests and non-FDA-cleared tests, and discusses quality control procedures in laboratory settings.
![•Method &number of samples:
• dividing the number of true positives by the sum of the number of
true positives plus the number of false negatives and multiplying by
100.
Diagnostic Sensitivity: = [TP ÷(TP + FN)] x 100
- If a person has a disease, how often will the test be positive (true
positive rate).
- The higher the sensitivity the lower the False Negative and vice
versa.](https://image.slidesharecdn.com/validationofqualitativelabtestmethodsdrmostafa-190321063244/85/Validation-of-qualitative-lab-test-methods-19-320.jpg)
![•Method & number of samples:
• Calculate by dividing the number of true negatives by the sum of
the number of true negatives plus the number of false positives
and multiplying by 100.
Diagnostic Specificity: [TN ÷(TN + FP)] x 100
-If a person does not have the disease, how often will the test be
negative ( true negative rate).
- The higher the specificity the lower the False Positive and vice
versa.](https://image.slidesharecdn.com/validationofqualitativelabtestmethodsdrmostafa-190321063244/85/Validation-of-qualitative-lab-test-methods-21-320.jpg)
Validation of qualitative lab test methods
- 1. Validation of Qualitative Lab Test Methods By: Dr Mostafa Mahmoud MD, Consultant Microbiologist Labs & Blood Banks Admin, Riyadh, MOH Associate Professor of Medical Microbiology & Immunology Faculty of Medicine – Ain Shams University
- 2. Procedures for validation • Each method validation/ verification study is a collection of experiments to assess performance and error in order to judge a method’s suitability for use in the laboratory. • Acceptability Criteria • the laboratory must establish acceptance criteria. • Parameters for accuracy, precision, sensitivity and specificity should include a confidence level of at least 90%, or meet the claims of the manufacturer.
- 3. Qualitative Methods: • Includes semi quantitative testing that use cut offs such as hepatitis testing and some molecular testing. • No values/concentrations are included in the patient report. • Test results are reported as positive/negative, normal/ borderline/abnormal, reactive/nonreactive, detected /not detected. •1. FDA cleared or approved methods (CLIA Standards). • Parameters to be validated: 1. Accuracy (correlation or comparison) 2. Precision (Replication) 3. Reportable range 4. Reference range (Normal values)
- 4. 1- Accuracy (correlation or comparison) • Demonstrates how close to the “true” value the new method can achieve. The materials (Samples) used: 1- Calibrators/controls, 2- Reference material, 3- Proficiency testing material with known values, 4- Samples tested by another lab using the same or similar method, 5- or by comparing results to an established comparative method.
- 5. Method & Number of samples: • Most sources recommend comparing at least 40 patient specimens. CLIA current guidance suggests a minimum of 20 samples. • A method comparison must be used for 5 days and for another 5 days if failed. • Document the results of the new method comparing the known values from the reference sources, another certified laboratory’s results or with results from the current method.
- 6. Calculations: • Calculate the percent of positive, negative and total accuracy by dividing observed results over known results multiplied by 100. • Example: New method = 19 positives, 20 negatives. Current method or reference material with known values = 20 positive, 20 negatives • Percent positive accuracy 19/20 X 100 = 95% • Percent negative accuracy 20/20 X 100 = 100% • Total accuracy 39/40 X 100 = 98%
- 8. 2- Precision (Reproducibility ): • Can the new method duplicate the same results? •Materials used: 1- patient samples are the first choice followed by 2- control material and reference solutions. Method & Number of samples: - 2 negative samples and 2 positive samples run in triplicate for 5 days will provide data for within-run and between-run components to estimate precision. •
- 9. Calculation: • Calculate the percent within-run (intra), between-run (inter) and total precision by dividing observed results over known results multiplied by 100. • See next tables.
- 10. ID Day 1 Day 2 Day 3 Pos sample Pos Pos Pos Pos Pos Pos Pos Pos Pos Pos sample Pos Pos Pos Neg Pos Pos Pos Pos Pos Neg sample Neg Neg Neg Neg Neg Neg Neg Neg Neg Neg sample Neg Neg Neg Neg Neg Neg Neg Neg Neg Within run % 12/12/x 100 = 100% 11/12 x 100 = 92% 12/12/x 100 = 100%
- 11. ID Day 4 Day 5 Between run % Pos sample Pos Pos Pos Pos Pos Pos 15/15x100 = 100% Pos sample Pos Pos Pos Neg Pos Pos 14/15x100 =93% Neg sample Neg Neg Neg Neg Neg Pos 14/15x100 =93% Neg sample Neg Neg Neg Neg Neg Neg 15/15x100 = 100% Within run % 12/12/x 100 = 100% 11/12 x 100 = 92% Total Precision: 58/60 x 100 = 96.7%
- 12. 3- Reportable Range: • The highest and lowest test values that can be analyzed while maintaining accuracy. • For tests without high or low values, define method criteria for a positive result. Method & Number of samples: • To verify reportable range, test at least 3-5 low and high positive samples once. • These samples can be combined with the accuracy/precision experiments. • Include both weak and strong positive samples.
- 13. • For methods depending on a cut-off value to determine positive results, testing positive specimens near the cut-off can serve as the cut-off validation. •CAP Requirement: • For qualitative tests that use a cut-off value to distinguish positive from negative, the cut-off value is established initially, and verified every 6 months thereafter. • N.B. This requirement does not apply to FDA-cleared/approved in vitro diagnostic assays that report the qualitative result based on a predefined cut-off value. • Applied to laboratory established threshold (cut-off value)
- 14. 4- Reference Range (Normal Values): • Provided by the manufacturer and verified by running known normal patients. • Manufacturer’s ranges or even published reference ranges in textbooks are used if the patients are similar. •Method & Number of samples: • The Reference Range can be verified by testing 20 known normal samples; if no more than 2 results fall outside the manufacturer/published range then that reference range can be considered to be verified.
- 15. • If the laboratory cannot reference the normal values, then the reference range will need to be established. This involves a selection of at least 120 reference samples for each group or subgroup that needs to be characterized.
- 16. 5- Sensitivity & Specificity: • CLIA does not require that these parameters to be verified.
- 17. Validation Summary for FDA-approved qualitative tests: • Clearly state the purpose of the verification, what platform/method and the number of samples for each experiment. Any discrepant results should be investigated and explained in the Summary. Test results that show sample problems such as contamination and degradation should not be used in the assessment but still listed with an explanation. • The Summary should also contain a Conclusion stating weather the study met the acceptance criteria or not and its suitability for us in the laboratory. • Additional samples to be added if results are outside acceptance. • If the results failed then test not to be implemented.
- 18. 2. Non-FDA Cleared tests • Qualitative methods developed in-House, non-FDA cleared methods and FDA-cleared methods modified by the laboratory. • - For Qualitative methods follow the instructions above for Accuracy, Precision, Reportable Range and Reference Range. •Qualitative Sensitivity is not done by analytic sensitivity but by diagnostic sensitivity. • Diagnostic sensitivity: The percent of subjects (patients) with the target condition whose test values are positive.
- 19. •Method &number of samples: • dividing the number of true positives by the sum of the number of true positives plus the number of false negatives and multiplying by 100. Diagnostic Sensitivity: = [TP ÷(TP + FN)] x 100 - If a person has a disease, how often will the test be positive (true positive rate). - The higher the sensitivity the lower the False Negative and vice versa.
- 20. •Qualitative Specificity • Qualitative sensitivity not assessed by Analytical Specificity but also by the Diagnostic Specificity. • Diagnostic Specificity: the percent of subjects (Persons) without the target condition whose test values are negative.
- 21. •Method & number of samples: • Calculate by dividing the number of true negatives by the sum of the number of true negatives plus the number of false positives and multiplying by 100. Diagnostic Specificity: [TN ÷(TN + FP)] x 100 -If a person does not have the disease, how often will the test be negative ( true negative rate). - The higher the specificity the lower the False Positive and vice versa.
- 22. Interference: • The laboratory must be aware of common interfering substances by referencing studies performed elsewhere (manufacturer or literature) or by performing studies. • Common blood interferences are; hemolysis, bilirubin, lipemia, preservatives and anticoagulants used in specimen collection.
- 23. CAP Accreditation requirements for validating laboratory tests FDA approved/cleared LDTs & modified FDA tests Accuracy & Precision (COM.40300, ph II) Verify manufacturer’s results Establish (= validation) Analytic sensitivity (LOD) (COM.40400, ph II) Verify: manufacturer or literature documentation OK Establish Analytic specificity (interferences) (COM.40500, ph II) Reference literature or manufacturer documentation Establish; studies by manufacturer or in literature OK Reportable range (AMR) (COM.40600, ph II) Verify* Establish* Reference range (COM.50000, ph II) Verify ** Establish**
- 24. Test method validation CBAHI standard LB.10 The laboratory develops a process for test method validation. LB.10.1 The laboratory implements policies and procedures on test method validation including: LB.10.1.1 Verification of accuracy/precision. LB.10.1.2 Verification of sensitivity (lower detection limit). LB.10.1.3 Verification of carryover acceptability. LB.10.1.4 Verification of the Analytic Measurement (AMR) LB.10.1.5 Approval of the method for clinical use (CRR??).
- 25. Test Method Validation Evidence of compliance to (LB 10) standard: 1- Document Reviews: - Policies, process and procedures on method validation. 2- Documented Evidences - Records surveyor-selected method confirms compliance with policies and procedures. 3- Staff Interview - Senior personnel are knowledgeable about the concept of method validation.
- 26. • LB.13 The laboratory has a system for instruments/methods correlation. LB.13.1 When the laboratory uses more than one method and/or instruments to test for a given analyte, the laboratory develops and implements policies and procedures on correlation to ensure the following: LB.13.1.1 The correlation studies are conducted every six months. LB.13.1.2 There is clear description of the correlation study. LB.13.1.3 There are clearly defined acceptance criteria. LB.13.1.4 There is a process for review and approval of the correlation results.
- 27. • Evidence of Compliance (LB.13): • Document Review - Policies, process and procedures on methods/instrument correlation. • Documented Evidences - Records surveyor-selected methods/instruments confirms compliance with policies and procedures. • Staff Interview - Senior personnel are knowledgeable about the purpose of methods/instrument correlation.
- 28. • LB.14 The laboratory has a system for controlling the quality of test methods. LB.14.1 The laboratory implements policies and procedures on quality control of test methods to satisfy the following: LB.14.1.1 Assignment of performance and review responsibility (control specimens are handled and tested in the same manner and by the same laboratory personnel testing patient samples). LB.14.1.2 Number and frequency of running controls. LB.14.1.3 Tolerance limits of controls results. LB.14.1.4 Corrective action to be taken in the event of unacceptable results. LB.14.2 The laboratory quality control system conforms to the manufacturer's instructions.
- 29. •Evidence of compliance to LB 14 • Document review: - Policies, process and procedures on controlling the quality of test methods. - Policies and procedures on controlling the quality of test methods confirm to the manufacturer instructions. • Documented Evidences: - Records surveyor-selected methods confirms compliance with policies and procedures. • Staff Interview: - Laboratory personnel understanding of the purpose of controlling the quality of test methods.