David-Graham-HGML-presentation-20190424.pptx
- 1. Observational data, effectiveness, and real-world evidence: How do we get there from here? David J. Graham, MD, MPH Harry Guess Memorial Lecture Gillings School of Global Public Health University of North Carolina at Chapel Hill April 24, 2019 1
- 2. 2 Disclaimer The opinions expressed are my own and are not necessarily those of the US Food and Drug Administration or the Department of Health and Human Services No conflicts of interest to disclose
- 3. 3 Pub Med citations with “real-world evidence” in title or abstract, by year 0 50 100 150 200 250 300 350 2010 2011 2012 2013 2014 2015 2016 2017 2018 No. Publications Year
- 4. 4 How did we get here? • 2006: IOM report – The Future of Drug Safety • Identify ways to access health-related databases • Create a public-private partnership to support safety/efficacy • 2007: Food and Drug Administration Amendments Act • Section 905 – Establish a post-market risk identification and analysis system; link and analyze healthcare data from multiple sources (Sentinel Initiative) • 2010: Affordable Care Act • Creation of PCORI, charged with promoting comparative effectiveness research, to improve patient outcomes • 2016: 21st Century Cures Act • Section 3022 amends FD&C Act on use of RWE
- 5. 5 The 21st Century Cures Act’s RWE provision • Defines RWE: data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials • Requires FDA to establish a program to evaluate RWE • 1) To help support approval of new indications for approved drugs • 2) To help support or satisfy post-approval study requirements • Imposes timeline for FDA implementation • By Dec 2018, establish and implement an RWE framework • By Dec 2021, issue draft guidance describing • 1) Circumstances under which sponsors of drugs may rely on RWE • 2) Acceptable standards and methods for collecting and analyzing RWE • Section 3022 does not limit FDA’s use of RWE for other purposes or change the standards of evidence required under 505(c) and (d) of FD&C Act or 351(b) of the Public Health Service Act
- 6. https://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RealWorldEvidence/UCM627769.pdf • For drug and biological products • Outlines FDA’s plan to implement its RWE program • Multifaceted program • Internal processes • Guidance development • Stakeholder engagement • Demonstration projects
- 7. 7 Definitions from FDA’s RWE Framework • Real-world data (RWD) • Data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources • Real-world evidence (RWE) • Clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD
- 8. 8 Potential Sources of RWD • Medical claims • Electronic health records • International health care databases • Product or disease registries • Personal devices and health applications
- 9. 9 Potential uses of RWD in clinical studies • Randomized, interventional • Non-randomized, interventional • Non-randomized, non-interventional (observational) • Prospective data collection • Registry studies • Prospective cohort studies • Retrospective (existing data) • Cohort studies • Case-control studies
- 10. 10 Traditional RCTs not considered RWE • Research infrastructure is usually separate from routine clinical practice • More likely to have restrictive eligibility criteria designed to maximize detection of a drug effect • In addition to randomization, usually also double-blind • Use separate procedures and/or personnel to collect specified data using standardized procedures • Detailed case report forms that are separate from routine medical records • Protocol-driven scheduled monitoring • Efforts to ensure strict adherence to study procedures
- 11. 11 Scope of FDA’s RWE Program • Evaluates the potential use of RWE to support changes to labeling about drug product effectiveness, including: • Adding or modifying an indication, such as a change in dose, dose regimen, or route of administration • Adding a new population • Adding comparative effectiveness or safety information
- 12. 12 Criteria for evaluating the potential use of RWD/RWE in regulatory decision making • Are the RWD fit for use? • Can the study design used to generate RWE provide adequate scientific evidence to answer or help answer the regulatory question? • Does the study conduct meet FDA regulatory requirements?
- 13. 13 Fitness for use (1) • Assessing data reliability • Data accrual and structure • Missingness • Consistency over time (includes coding) • Coded data adequately represent intended underlying medical concepts • Quality control • Assessing data relevance • Captures clinical effectiveness outcomes that address specific regulatory questions • “Hard” outcomes vs. disease exacerbations/progression • Captures relevant data on exposure and covariates • Coding accuracy • International data?
- 14. 14 Fitness for use (2) Source: Duke-Margolis Center for Health Policy white paper, “A framework for regulatory use of real-world evidence,” 2017
- 15. 15 Adequacy of study design to answer regulatory question (1) • Randomized designs in routine clinical settings • Quality of data captured • Number of patients available • Variations in clinical practice • Bias control if blinding infeasible • Non-randomized, single arm with external RWD control • Is control population comparable • Lack of standardized or equivalent outcome measures • Variability in follow-up procedures
- 16. 16 Adequacy of study design to answer regulatory question (2) • Observational studies • FDA’s Pharmacoepidemiology Guidance (for safety) • Focus is on ability to draw reliable causal inference • Characteristics of data • Characteristics of study design and analysis • Active comparator • Unmeasured confounders • Measurement variability • Prespecified sensitivity analyses and statistical diagnostics • Transparency • Concern that studies can be conducted multiple times and in multiple databases, until desired result obtained
- 17. 17 FDA’s Pharmacoepidemiology Guidance • Appropriateness of data source • Pre-specified study protocol and statistical analysis plan • Selection of study population – explicit inclusion and exclusion criteria • Exposure ascertainment • Outcome ascertainment – validation, linkage • Confounding adjustment • Sensitivity analysis - robustness https://www.fda.gov/downloads/drugs/guidances/ucm243 537.pdf.
- 18. 18 Does study conduct meet FDA regulatory requirements • Informed consent • Oversight and monitoring • Guidance on use of electronic source data • Recommendations on capture, review, and retention of data • Guidance on use of EHRs in clinical studies • To ensure integrity of EHR data • Draft guidance on procedures to ensure that electronic records are trustworthy, reliable • Other potential future guidance documents
- 19. 19 Potential FDA draft guidance documents on RWD/RWE • How to assess whether RWD are fit for use to generate RWE in support of product effectiveness • Considerations for using RWD in RCTs for regulatory purposes. Including pragmatic design elements • Use of RWD to generate external control arms for single- arm trials • Observational study designs and how they might provide RWE to support effectiveness in regulatory decision making • Regulatory considerations raised by different study designs to generate RWE that are submitted to support product effectiveness
- 20. 20 A recent example of RWE Source: https://doi.org/10.1016/j.amjmed.2018.12.023
- 21. 21 Methods (1) • Medicare fee-for-service claims data, 2010-2015 • Inception cohort design • New users of warfarin or a NOAC • Standard NOAC doses only (73%-84%) • Age ≥ 65 • ≥ 6 months Medicare Parts A, B, D • Nonvalvular AF • No prior anticoagulant use • No diagnoses indicating valvular heart disease, VTE, or joint replacement in prior 6 months • Edoxaban use too low for study inclusion
- 22. 22 Methods (2) • PS matched warfarin to pooled NOAC users • Adjusted using stabilized IPTW generated from multinomial logistic regression model • Variables included in PS model (n=112) • Demographics • Cardiovascular risk factors • Bleeding risk factors • Chronic medical conditions • Rx medications • Metabolic inhibitors • Health care utilization & frailty • Prescriber characteristics • CHA2DS2-VASc and HAS-BLED scores
- 23. Dabigatran n=86,198 Rivaroxaban n=106,382 Apixaban n=73,039 Pooled NOACs n=265,626 Warfarin n=523,264 PS matching with replacement Warfarin Dabigatran Rivaroxaban Apixaban n=183,318 n=86,198 n=106,382 n=73,039 Warfarin Dabigatran Rivaroxaban Apixaban n=183,003 n=86,293 n=106,369 n=72,921 Multinomial logistic regression Stabilized IPTW Unweighted Weighted
- 24. 24 Cohort follow-up • On-therapy; 3-day gap allowance • Censoring for: • Disenrollment • Any outcome event • > 3-day gap in days supply • Switch to another oral anticoagulant • Dialysis or kidney transplant • Admission to NH, SNF, or hospice care • End of study period
- 25. 25 Outcomes • Thromboembolic stroke (PPV 88%-95%) • Intracranial hemorrhage (89%-97%) • Major extracranial bleeding* (87%) • Death (linked to Social Security) (95%) • 1st event or within 30 days of a 1° outcome * Hospitalized + requiring transfusion, resulting in death, or involving a critical extracranial site (modified ISTH definition)
- 26. 26 Analysis • Cox PH regression • For each outcome, a single regression model • Included independent variables for exposure to each of 4 study drugs • Generated HR (95% CI) for each pairwise comparison of interest (n=6) • NOAC vs. warfarin • NOAC vs. NOAC • All cohorts were simultaneously adjusted to the same standard and all subjects included in analysis • Sensitivity analyses: 14-day gap, study period post- apixaban approval, subset with 2+ Rxs, unweighted multivariable Cox, crude (unadjusted)
- 27. 27 Study design: inception cohort, time-to-event -183 d t0 Up to 5 years No oral anticoagulants No valvular heart disease No VTE, joint replacement Medical covariates, medication use Censor: Switch, therapy gap, NH, hospice, SNF, dialysis/transplant, outcome, study end Outcomes: Ischemic stroke, intracranial hemorrhage, major extracranial bleeding, death Not in hospital, NH, SNF, hospice Age ≥ 65
- 28. 28 Propensity score distributions (warfarin propensity scores only)
- 29. 29 Distribution of AT-NOAC IPTWs Cohort Mean Percentiles Min 50% 99% Max Dabigatran 300mg 1.00 0.41 0.97 1.81 4.25 Warfarin 1.00 0.23 0.96 1.93 4.10 Rivaroxaban 20mg 1.00 0.66 0.98 1.45 2.49 Apixaban 10mg 1.00 0.41 0.96 1.80 3.19
- 30. 30 Covariate balance across study cohorts: distribution of maximum standardized mean differences for each study covariate across six pairwise comparisons* SMD range Unweighted Unmatched Matched Weighted 0.00 – 0.01 1 109 0.02 – 0.04 18 3 0.05 – 0.09 44 0 0.10 – 0.19 38 0 ≥ 0.20 11 0 *D vs. W R vs. D R vs. W R vs. A A vs. W D vs. A
- 31. 31 Cohort sizes and event counts Warfarin n=183,318 Dabigatran n=86,198 Rivaroxaban n=106,389 Apixaban n=73,039 Events (n) Thromboembolic stroke 1,595 Intracranial hemorrhage 1,018 Major extracranial bleed 4,379 All-cause mortality 4,271
- 32. 32 Adjusted hazard ratios (95% CI) for NOAC vs. warfarin comparisons
- 33. 33 Adjusted hazard ratios (95% CI) for NOAC vs. NOAC comparisons
- 35. 35 Sensitivity analyses • 14 day gap allowance • Restricted to ≥ 2 dispensings • Restricted to post-apixaban approval • Multivariable regression • Crude • Adjusted • Post hoc (including most or all warfarin users) • Multivariable adjustment • Trimmed analyses • Interacted splines analyses
- 36. 36 Conclusions • NOACs have a more favorable benefit-harm balance than warfarin • Among NOACs, rivaroxaban has a less favorable benefit-harm balance than dabigatran or apixaban
- 37. 37 Does this study constitute RWE upon which regulatory decisions can be made for effectiveness and safety? More generally, could a claims-based observational study be used to support changes to labeling about a drug product’s effectiveness: New indication New population Comparative safety or effectiveness
- 38. 38 Substantial evidence (FD&C Act §505(d)) • Substantial evidence of effectiveness required for FDA approval • “Evidence consisting of adequate and well-controlled investigations…by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved, on the basis of which it could…be concluded…that the drug will have the effect it purports” • In certain situations, data from one adequate and well-controlled clinical investigation and confirmatory evidence may be sufficient to establish effectiveness, and may be considered to constitute substantial evidence
- 39. 39 Comparative Claims §201.57 (c)F(iii) (iii) Any statements comparing the safety or effectiveness of the drug with other agents for the same indication must, except for biological products, be supported by substantial evidence derived from adequate and well-controlled studies as defined in §314.126(b) of this chapter unless this requirement is waived under §201.58 or §314.126(c) of this chapter. For biological products, such statements must be supported by substantial evidence.
- 40. 40 “Adequate and well-controlled” studies (21CFR 314.126) • Prespecified protocol with description of objectives, study design, method of treatment assignment, methods to minimize bias, and methods of analysis • Design permits valid comparison with a control • Subjects have the disease or condition being studied • Method of assigning patients to treatment and control groups minimizes bias and is intended to assure comparability of groups for pertinent variables (e.g., severity/duration of disease, other drugs) • Adequate measures to minimize bias of subjects, observers, analysts • Methods of assessment of subjects’ response well-defined, reliable • Analysis is adequate to assess effects of drug
- 41. 41 Reality check • FDA just released its RWE framework—much work to be done, many questions to be resolved • Can observational studies qualify as “substantial evidence”? • Assuming the answer is “yes,” what are the necessary characteristics of an “adequate and well controlled” observational study? • How many AWC observational studies are needed to support an effectiveness claim? • Can multiple studies (some AWC others not), together, constitute substantial evidence? • Which product labels would be affected? • Where in label to place safety claims, effectiveness claims? • How should claims be phrased?
- 42. 42 Let’s play “devil’s advocate” (1) (Why might FDA question this study?) • Some results differed from those of pivotal RCTs • Protective effect for all-cause mortality • Protective effect for thromboembolic stroke • Differences in INR management between RCTs and community-based care • Matching warfarin with NOAC users excluded 65% of warfarin users • Impact on generalizability • Potential unmeasured confounding • Warfarin vs. NOAC • NOAC vs. NOAC? • Reliance on validated outcome algorithms • Should sample or all outcomes be confirmed? How? • Treatment persistence--bias and informative censoring
- 43. 43 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 20 40 60 80 100 120 140 160 180 200 Percent of Cohort Days after Reference Date Remaining Patients on Treatment (3-Day Gap Allowance) After Matching Warfarin to NOACs with Replacement Dabigatran 300mg (N=86,198) Rivaroxaban 20mg (N=106,389) Apixaban 10mg (N=73,039) Warfarin (N=182,722) On-treatment persistency
- 44. 44 Let’s play “devil’s advocate” (2) (Why else might FDA distrust observational studies?) • Unmeasured confounding (can’t be repeated enough) • Bias • Study integrity (planning, conduct, analysis) • Outcome ascertainment (how defined, sensitivity, PPV) • Concern that studies can be conducted multiple times and in multiple databases, until desired result obtained • Heterogeneity of results across different studies • False positives and false negatives
- 45. 45 Heterogeneity of treatment effects across databases (PS-stratified new user cohort studies) Source: Madigan et al. Am J Epidemiol 2013; 178(4):645-651 43% with I2 ≥ 75% 21% with “significant” positive and negative results
- 46. 46 Heterogeneity within the same database • 3rd generation oral contraceptives and VTE risk (GPRD) • Oral bisphosphonates and esophageal cancer (GPRD) • Lower extremity amputation risk with SGLT2 inhibitors vs. DPP4 inhibitors or sulfonylureas (MarketScan) SGLT2 inhibitors vs. DPP4 inhibitors Sulfonylureas Dawwas et al. 0.88 (0.65-1.15) 0.74 (0.57-0.90 Yang et al. 1.69 (1.20-2.33) 1.02 (0.69-1.55) Source: Dawwas et al. Diab Obes Metab 2018. doi: 10.1111/dom.13477 Yang et al. Diab Obes Metab 2019. doi: 10.1111/dom.13647
- 47. 47 Source: Anne Anderson (illustrator), Grimm’s Fairy Tales (1922) Rumpelstiltskin
- 48. 48 Hope diamond Diamond in the rough
- 49. 49
Editor's Notes
- #44: AC_2010_2015_Persistency_Matched_16Feb2017