Quantitative methods of signal detection
Download as PPT, PDF5 likes1,756 views
This document discusses quantitative methods for signal detection in pharmacovigilance. It describes current methods like proportional reporting ratios (PRR) that measure disproportionate reporting of adverse drug reactions. Bayesian methods and log-linear models are presented as improvements. Longitudinal health records are discussed as an alternative that avoids some biases but is difficult to implement. Future directions may include using all available data on reports and incorporating pharmacokinetic properties of drugs.
![8
Disproportionality analysis (example)
. CNS drug for which the total No of reports is
400, of these 20 reports of diarrhoea
. All other products in the database (1 million
reports excluding reports involving drug A), of
these 50,000 reports of diarrhoea.
PRR = [20/400] / [50,000/1,000,000] = 1 (no
SDR)](https://image.slidesharecdn.com/quantitativemethodsofsignaldetection-151203222151-lva1-app6891/85/Quantitative-methods-of-signal-detection-8-320.jpg)
![9
Disproportionality analysis (example)
CNS drug for which the total No of reports is 400,
of these 40 reports of drowsiness
. All other products in the database (1 million
reports excluding reports involving drug A), of
these 25,000 reports of diarrhoea.
PRR = [40/400] / [25,000/1,000,000] = 4
(presence of a SDR)](https://image.slidesharecdn.com/quantitativemethodsofsignaldetection-151203222151-lva1-app6891/85/Quantitative-methods-of-signal-detection-9-320.jpg)
![28
Bibliography
Guideline on the use of statistical signal detection methods in EVDAS
http://www.ema.europa.eu/pdfs/human/phvwp/10646406enfin.pdf
Bate & Evans quantitaitve signal detection using ADR reporting. PDS
2009.
Hauben & Bate. Decision support methods for the detection of adverse
events in post-marketing data. Drug Discovery Today. April 2009.
Evaluation of Signal Detection Methods for use in Prospective Post
Licensure Medical Product Safety Surveillance [FDA Sentinel Initiative
Safety Signal Identification Contract] – Report
http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm149343.htm](https://image.slidesharecdn.com/quantitativemethodsofsignaldetection-151203222151-lva1-app6891/85/Quantitative-methods-of-signal-detection-28-320.jpg)
Quantitative methods of signal detection
- 1. An agency of the European Union Presented by: Name Surname Position/organisational descriptor Quantitative methods of signal detection François MAIGNEN
- 2. 2 Learning objectives •Current methods: Disproportionality analysis •Improvement of the current methods: Bayesian methods and (Bayesian) log-linear models •Comparison of the methods (limitations, stats vs clinical) •Star wars – a new hope? Longitudinal health records: better? Different and not devoid of biases •Future directions
- 3. 3 Measures of disproportionate reporting Most of the methods routinely used in pharmacovigilance (spontaneous reporting systems) databases are based on measures of disproportionate reporting (i.e. ROR, PRR, BCPNN, MGPS, etc …). Basically: “Observed vs Expected” analysis in a given database i.e. % of reports involving a given reaction for a given medicine compared to the % of reports involving this reaction on the whole database
- 4. 4 A spontaneous reporting system database SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N Event 3 n31 n32 … … … … … … n3N Event 4 n41 n42 … … … … … … n4N Event 5 n51 n52 … … … … … … n5N Event 6 n61 n62 … … … … … … n6N … … … … … … … … … … Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN
- 5. 5 Drug 1 All other medicinal products Total Event 1 a c All other reaction terms b d Total N = a + b + c + d c + d a + c a + b Proportional Reporting Ratio
- 6. 6 Proportional Reporting Ratio PRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN IN PRACTICAL TERMS? a/(a+b) = Proportion of reports involving a specific adverse event among all the reports involving DRUG A c/(c+d) = Proportion of reports involving THE SAME adverse event among all the reports of your database but DRUG A
- 7. 7 Proportional Reporting Ratio If the rate of reporting of AE for drug 1 is similar to the rate of reporting of this AE for all the other products of the database, the PRR will be equal to 1 (same proportion of reports involving the reaction for drug A than for the other drugs) … BUT … If the reaction is proportionately MORE reported with drug A than for the other products, the PRR will be increased (typically > 1). DIS-PROPORTIONALITY of reporting
- 8. 8 Disproportionality analysis (example) . CNS drug for which the total No of reports is 400, of these 20 reports of diarrhoea . All other products in the database (1 million reports excluding reports involving drug A), of these 50,000 reports of diarrhoea. PRR = [20/400] / [50,000/1,000,000] = 1 (no SDR)
- 9. 9 Disproportionality analysis (example) CNS drug for which the total No of reports is 400, of these 40 reports of drowsiness . All other products in the database (1 million reports excluding reports involving drug A), of these 25,000 reports of diarrhoea. PRR = [40/400] / [25,000/1,000,000] = 4 (presence of a SDR)
- 10. 10 Improvements of these methods •Considering possible confounding factors: stratification and log-linear models (ROR – see work from E. Van Puijenbroek) •Trying to circumvent low expected values or low case counts: Bayesian models (A. Bate & W. DuMouchel) •Other regression methods: LASSO and Bayesian logistic regressions (N. Noren, D. Madigan) •Clinical relevance not always clear or demonstrated •Some methods can be computationally demanding
- 11. 11 Bayesian methods BCPNN and MGPS rely on the same principle of conjugate prior distributions: •These methods will shrink the value of the measure of disproportionality using a Bayesian approach (prior based on existing dataset) •BCPNN: cell counts ~ Binomial dist., conjugate prior = beta •MGPS: cell counts ~ Poisson, conjugate prior = Gamma (mixture of Gammas).
- 12. 12 Bayesian methods Assume binomial y=7 successes, 20 trials. Non informative prior = Beta (2,4)
- 13. 13 Thresholds - ARBITRARY All these methods provide a ranking … Thresholds = arbitrary Trade-off between •Reviewing too many drug-event pairs (loss of operational benefit) •Missing some signals No ADR ADR
- 14. 14 Comparison of the methods Methodological difficulties No gold standard / no standardised reference method Imprecision of what constitutes a signal Retrospective vs prospective evaluation Importance of clinical judgement. The added value of clinical evaluation is currently unknown (if any).
- 15. 15 Comparison of methods 1 2 3 4 7 11 6 9 10 12 5 8 1 2 3 4 7 11 6 9 10 12 5 8 4 2 3 1 7 11 6 9 10 12 5 8 Meth.1 Meth.2 Meth.3 • Threshold 1: Meth. 2 = 5 true signals, meth. 1/3 = 4. • Threshold 1+2: Meth. 2=Meth.3 • First 5 signals: Meth. 1 ≠ Meth. 2 = 3. ADR No ADR
- 16. 16 Comparison of methods (PRR vs BCPNN)
- 17. 17 Performances of these methods Operational benefit (screening of large databases) Anecdotal evidence (in opposition to structured) of signals discovered thanks to the quantitative methods (recent examples incl. D:A:D and MI) Time benefit in some cases (Hochberg & EV study) NND ~ 7/15 (depending whether the study is retrospective or prospective) Idea: Quant. Methods + DMEs/TMEs
- 18. 18 New approaches to signal detection Deviation of Obs. vs Expect. distr. from a fitted distribution (Jim) Modelling of the hazard function of the time to onset (DSRU / François) hazard # mechanism Use of longitudinal databases (record linkage and electronic health records – OMOP / Noren / Callreus) ~ incidence rate ratio • Same patients different time windows (A. Bate) • Hospital records of different patients (T. Callreus)
- 19. 19 Merci Andrew
- 20. 20
- 21. 21 Interpretation / Limitations Patients prescribed a PPI are associated with acute pancreatitis in the month after the Prescription (ICdiff positive) But graph shows that these patients are general likely to have acute pancreatitis around the time of PPI prescribing (In agreement with confounding by indication)
- 23. 23
- 24. 24 Interpretation / limitations Bias associated with the hospitalisation (confirmed by later events which occurred remotely after the administration of the vaccine) Spurious / unexplained associations Rely heavily on temporal association (Post hoc ergo propter hoc)
- 25. 25 Longitudinal health records Powerful to detect some associations Even if reporting artefacts do not influence this method, not devoid of other biases (selection, protopathic, misclassification, etc …) Much more complicated to implement (very large datasets, confidentiality aspects, linkage of records and interoperability of databases, etc …)
- 26. 26 Future directions (?) •Performances in a prospective setting, added value of clinical judgement needs to be quantified •Duplicate detection •Other approaches to signal detection (use all the information included in the reports, PK/PD properties of the substances) •Surveillance networks (incl. spontaneous reporting)
- 27. 27 Key elements •PRR is a measure of disproportionality of reporting in a specific database (observed vs expected value computed on the whole database) •A SDR indicates a possible signal which needs to be medically confirmed •All other methods proceed from the same logic •Data mining on EHR will probably improve the detection of signals (organised method of collection) but is not free of biases and is resources demanding
- 28. 28 Bibliography Guideline on the use of statistical signal detection methods in EVDAS http://www.ema.europa.eu/pdfs/human/phvwp/10646406enfin.pdf Bate & Evans quantitaitve signal detection using ADR reporting. PDS 2009. Hauben & Bate. Decision support methods for the detection of adverse events in post-marketing data. Drug Discovery Today. April 2009. Evaluation of Signal Detection Methods for use in Prospective Post Licensure Medical Product Safety Surveillance [FDA Sentinel Initiative Safety Signal Identification Contract] – Report http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm149343.htm