Big data vs the RCT - Derek Angus - SSAI2017
Download as PPTX, PDF1 like342 views
The document describes a novel platform trial design called REMAP (Randomized, Embedded, Multifactorial, Adaptive Platform) that aims to efficiently test multiple interventions for critical illness. It utilizes a point-of-care embedded design within electronic health records to rapidly enroll patients and assign multifactorial intervention regimens based on a Bayesian statistical model. The model continuously updates probabilities of intervention effectiveness based on accumulating trial data and can trigger results when an intervention is found to be superior, equivalent, or inferior for a given patient subgroup. This allows the trial to efficiently evaluate and adapt multiple treatment options in a real-world intensive care setting.
![Z = g[ ]+ R[ ]+ Time[ ]+ Interv[ ]+ Inter,Shock[ ]+ Interv,Hypox[ ]+ Interv,Interv[ ]](https://image.slidesharecdn.com/ertnvneztb61ek9fofkc-signature-8173cdb1901b6b2a92f7345b405fb7be87b1fecb83bbc56c9eadcc8941de1aa8-poli-171101062227/85/Big-data-vs-the-RCT-Derek-Angus-SSAI2017-40-320.jpg)
Big data vs the RCT - Derek Angus - SSAI2017
- 1. CRISMA Center, Department of Critical Care Medicine Department of Health Policy and Management McGowan Institute for Regenerative Medicine Clinical and Translational Science Institute University of Pittsburgh Schools of the Health Sciences Derek C. Angus, MD, MPH, FRCP
- 2. CRISMA Center, Department of Critical Care Medicine Department of Health Policy and Management McGowan Institute for Regenerative Medicine Clinical and Translational Science Institute University of Pittsburgh Schools of the Health Sciences Derek C. Angus, MD, MPH, FRCP Big Data vs. the RCT?
- 3. C R I S M A Evidence-based medicine “Doctors must base what they do on randomized clinical trials (RCTs)” Archie Cochrane
- 4. C R I S M A How we know something works ... Patients with disease X
- 5. C R I S M A How we know something works ... Patients with disease X
- 6. C R I S M A How we know something works ... Patients with disease X
- 7. C R I S M A How we know something works ... Patients with disease X
- 8. C R I S M A How we know something works ... Patients with disease X
- 9. C R I S M A
- 10. C R I S M A The good news … •We now test many ideas with RCTs - 37,000 started in 2010 … -All FDA drug and device approvals •We now conduct RCTs very well - Methodologic conduct - Ethical oversight - Reporting
- 11. C R I S M A But … • RCTs are too narrow • Cherry-picked population; not the real world • RCTs are too broad • No data on treatment effects across patients • No comparative effectiveness • Rx A vs. B is not very helpful • What about A vs. B, vs. C, vs. D … etc. • Depending on whether I give E or F … • Having said all that, I just want the answer … • I don’t want my patient to be a guinea pig …
- 12. C R I S M A Clinical care Clinical research Parallel universes …
- 13. C R I S M A Enter the era of ‘Big Data’ • Integration of ‘deep’ personalized data • Causal inferences on optimal care • Broad – ‘real-world’ practice • Narrow – ‘personal’ estimates • Comparative – considers all options • Vanderbilt-IBM ‘BioVU’ initiative • ‘Live’ presentation of information at time of clinical decision-making • ‘Just-in-time’ cohort study in EHR • No guinea pigs • Longhurst et al. Health Affairs 2013
- 14. C R I S M A
- 15. Feature Leverage the EHR Low incremental costs Real-world ‘effectiveness’ Consider multiple therapies ‘Personalized’ estimates Offer ‘live’ tailored options Robust causal inference ‘Big Data’ Analytics ✔ ✗ ✔ ✔ ✔ ✔ ✔ Evidence Generator Report Card
- 16. C R I S M A Point-of-care (POC) Clinical Trials • A clinical moment in the EHR ‘alerts’ the clinical trial machinery • VA EHR • In-patient diabetics with poor glucose control • When physician placed insulin order in CPOE system … • Opportunity to randomize • Sliding scale • Weight-based algorithm • Fiore et al. Clinical Trials 2011 • Targeting the large ‘pragmatic’ trial arena • 2 thiazide diuretics in >13k high BP Veterans (NCT02185417) • 2 aspirin doses in 20k CVD patients (ADAPTABLE) (PCORI)
- 17. Feature Leverage the EHR Low incremental costs Real-world ‘effectiveness’ Consider multiple therapies ‘Personalized’ estimates Offer ‘live’ tailored options Robust causal inference ‘Big Data’ Analytics POC Trials ✔ ✗/✔ ✔ ✔ ✔ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ Evidence Generator Report Card
- 18. C R I S M A Platform Trials • Adaptive trials • Focus on disease, not a particular Rx • Multiple interventions (arms) • ‘Perpetual’ enrollment • Often based on Bayes’ theorem • Tailor choices over time Berry et al JAMA 2015 • Focus on pre-approval space • Emphasis on efficiency with (very) small sample sizes • Different therapies ‘graduate’ to next phase while trial continues Woodcock and Lavange NEJM 2017
- 19. C R I S M A
- 20. C R I S M A Response-adaptive randomization Rugo et al. NEJM 2016
- 21. C R I S M A The traditional RCT ... Patients with disease X At the start, 50% chance that A > B
- 22. C R I S M A The traditional RCT ... Patients with disease X At the end, >99% sure that A > B What about in the middle?
- 23. C R I S M A A planned trial of A vs. B in 400 patients The probability that A > B = 78% Start randomizing MORE patients to A than B … Alive Dead 40 20 No. of patients A B After 40 enrolled ….
- 24. C R I S M A After 80 patients … Now, the probability that A > B = 99.9% Stop the trial! Alive Dead 40 20 No. of patients A B
- 25. C R I S M A Caveats 1. If the ‘second’ 40 was flat or opposite direction … • Trial continues and the next ‘bet’ swings back closer to 50:50 2. When only 2 groups, power still driven by the smaller group • So, NOT very helpful if … • Single homogenous cohort • Two arms • But, becomes VERY interesting when … • Multiple arms • Multiple subgroups
- 26. B C Statistical modelRandomization rule Response-adaptive randomization A
- 27. B C Statistical modelRandomization rule Response-adaptive randomization A Odds weighted towards best RX
- 28. B C Statistical modelRandomization rule Response-adaptive randomization A D New arms activated
- 29. A B Statistical modelRandomization rule Response-adaptive randomization Or dropped D
- 30. C A Statistical modelRandomization rule Response-adaptive randomization Different weights for different patient groups D
- 31. Feature Leverage the EHR Low incremental costs Real-world ‘effectiveness’ Consider multiple therapies ‘Personalized’ estimates Offer ‘live’ tailored options Robust causal inference ‘Big Data’ Analytics POC Trials Platform Trials ✔ ✗/ ✔ ✗ ✗ ✔ ✔ ✔ ✔ ✗ ✔ ✔ ✔ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ Evidence Generator Report Card
- 32. Feature Leverage the EHR Low incremental costs Real-world ‘effectiveness’ Consider multiple therapies ‘Personalized’ estimates Offer ‘live’ tailored options Robust causal inference ‘Big Data’ Analytics POC Trials Platform Trials ??? ✔ ✗/✔ ✗ ✗ ✔ ✔ ✔ ✔ ✗ ✔ ✔ ✔ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Evidence Generator Report Card
- 33. C R I S M A A novel blend of ‘POC’ + platform designs •REMAP •Randomized •Embedded •Multifactorial •Adaptive •Platform trial REMAP ✔ ✔ ✔ ✔ ✔ ✔ ✔ Angus DC. JAMA 2015
- 34. C R I S M A • Funding • EU FP7 PREPARE WP 5 program (25M euro) • Australian NHMRC ‘OPTIMISE’ program ($6M) • New Zealand NHMRC ($2M) • Simultaneously test • Different anti-microbial strategies • Different host immunomodulation strategies • Different ventilation strategies • Separate RAR and stopping rules for multiple subgroups Angus DC. JAMA 2015
- 35. C R I S M A Angus DC. JAMA 2015 • Patients are preferentially assigned to best performing arm • Allocation is random, but NOT 50:50 • Odds of assignment proportional to odds of success • Not a guinea pig! • Embedded • ICU admission orders • Approved in Netherlands and New Zealand with delayed consent
- 36. C R I S M A REMAP-CAP elements • Domain – an area where a question is asked … • Domain #1 – choice of antibiotic • Domain #2 – whether to give steroids or not • Domain #3 – whether to extend macrolide or not • Domain #4 – choice of ventilator strategy • Domain #5 – oxygen titration strategy • Etc. …. • Intervention • Any option within a domain … • Regimen • Unique combination of interventions within a domain … • Stratum • Baseline subgroup • Ex. shock or not
- 37. Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn
- 38. Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domainsRegimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction
- 39. Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction
- 40. Z = g[ ]+ R[ ]+ Time[ ]+ Interv[ ]+ Inter,Shock[ ]+ Interv,Hypox[ ]+ Interv,Interv[ ]
- 41. Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domainsRegimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction
- 42. Statistical trigger Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domainsRegimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt External adaptations
- 43. Statistical trigger Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt External adaptations
- 44. Statistical trigger • Launch with initial weights • Update based on new probabilities Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization External adaptations
- 45. Statistical trigger • Launch with initial weights • Update based on new probabilities Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization External adaptations
- 46. Statistical trigger • Launch with initial weights • Update based on new probabilities Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization External adaptations
- 47. Statistical trigger • Launch with initial weights • Update based on new probabilities Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization External adaptations
- 48. Statistical trigger • Launch with initial weights • Update based on new probabilities Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization
- 49. Statistical trigger • Launch with initial weights • Update based on new probabilities Steering Committee can • Add strata, domains & interventions DSMB can • Request new external data be incorporated in priors • Overrule statistical triggers Multifactorial intervention assignments Regimen = set of domain-specific interventions Effect of an intervention is conditional upon • Stratum • Interventions within other domains Embedding Patient identification and enrollment • Tied to clinical ‘point-of-care’ Randomized interventions • Issued as ‘order set’ regimen EHR embedding • Screen and flag patient • Consent documentation • Generate regimen order set • Flag downstream states • Data collection Regimen Domain A Domain B Domain C #1 A1 B1 C1 #2 A1 B1 C2 #3 A1 B2 C1 #4 A1 B2 C2 #5 A2 B1 C1 ….. #n An Bn Cn Result declared when, within stratum, an intervention is • Superior >99% likely to be best • Equivalent >90% likely to be D<3% • Inferior <1% likely to be best Pre-specified architecture determined by • Choice of domains, strata, etc. • Choice of potential interactions Choices inform a Bayesian inference model • Pre-trial simulations evaluate performance Each external adaptation (ex. new domain) • Modify elements in Bayesian model • Re-simulate before ‘live’ deployment Pre-trial design and construction Patients Presumed severe CAP • Different strata (ex. shock or not) • Collected at sites • Managed at regional data centers • Merged at central statistical center Data collection Re-estimate Bayesian inference model with new data to update probabilities Update and adapt Response-adaptive randomization External adaptations
- 50. C R I S M A REMAP severe pneumonia … • Gets closer to individualized treatment decisions … • For example, should my patient receive IV hydrocortisone? • Depends on • Whether shock is present • How sick (hypoxic) the patient is • Whether underlying cause is viral or not • Whether an anti-viral is being administered • Whether other strategies are being used that may minimize lung injury and inflammation (protective vs. ultraprotective ventilation) • Separate probability estimate for each consideration … • Trial enrolls until a predefined level of certainty • As soon as one question hits threshold, answer is announced
- 51. C R I S M A Run the trial ‘in silico’ ahead of time … • Monte-Carlo simulations • Run 1,000s of times under different scenarios
- 52. ‘True’ mortality Average results from 1,000s of simulations 80 fewer deaths; higher power Scenario #1: 2 of 8 regimens are best
- 53. ‘True’ mortality Average results from 1,000s of simulations Similar power but 80 fewer deaths Scenario #1: 2 of 8 regimens are best
- 54. ‘True’ mortality Average results from 1,000s of simulations Scenario #2: One regimen is best
- 55. ‘True’ mortality Average results from 1,000s of simulations 94 fewer deaths; higher power Scenario #2: One regimen is best
- 56. C R I S M A Ok, but … • EHR data quality • Institutional commitment • Ethics • Statistics and design • Reporting and dissemination of results • Funding • Oversight • Integration with other clinical research programs
- 57. C R I S M A Conclusions • RCTs remain our strongest ‘truth’ finder • But, current RCT enterprise LETS US DOWN • ‘Big Data’ should not be cast as an alternative to the RCT • This is a false choice • Instead, the digital age enables novel RCTs designs • Smarter and safer
- 58. C R I S M A Self-learning healthcare is … fused care and research