Causal Inference from Pragmatic Trials.

@EpiEllie
Causal Inference from
Pragmatic Trials
Eleanor Murray, ScD
Department of Epidemiology
University of Michigan
May 8, 2019

Acknowledgements
Collaborators
 Ellen Caniglia, Sonja Swanson, Sonia Hernández-
Díaz, and Miguel Hernán
Funding
 This work was supported through a Patient-Centered
Outcomes Research Institute (PCORI) Award (ME-
1503-28119)
 All statements in this report, including its findings
and conclusions, are solely those of the authors and
do not necessarily represent the views of the Patient-
Centered Outcomes Research Institute (PCORI), its
Board of Governors or Methodology Committee.
Murray. 2019. Pragmatic randomized 2

Outline
I. What are pragmatic trials?
II. What are patient-centered
causal effects?
III. Can we estimate per-protocol
effects?
IV. Some guidelines for trial
design & analysis
3Murray. 2019. Pragmatic randomized

Part I: What are pragmatic
trials?

optimal for detecting treatment
effects
Randomization prevents confounding
…
But, results can have limited
applicability for clinical decision-
making
highly selected population
short duration & intermediate or
surrogate outcomes
comparators not clinically relevant

Solution: Pragmatic randomized
trials
Definition: A randomized trial
designed to assess real-world
effectiveness of interventions
Pragmatic randomized trials are
designed to ask clinically relevant,
generalizable, questions

Pragmatic randomized
trials
https://www.gopubmed.org/web/gopubmed/statistics/”Pragmatic+Clinical+Trials
+as+Topic”[mesh]

Trade-off: clinical relevance
vs bias
Long duration
+Allows a more relevant primary
outcome
–Increases chances of loss to follow-
up
–Increases chances of non-
adherence
Usual care as comparison
+Answers a more clinically relevant
question

Part II: What are patient-
centered causal effects?

Patient-centered causal
effects
 Definition: causal effects which
provide patients with usable
information for shared
decision-making
Murray EJ, et al. Journal of
Clinical Epidemiology. 2018;

Intention-to-treat & stratified
ITT are patient-centered
Murray. 2019. Pragmatic Randomized 11
Murray et al. 2018 J Clin
Trials, 103:10-21.

Non-inferiority results viewed with
extreme skepticism
“To me there has to be a point that they
developed this drug. Like what else is going
on with the drug?… It’s really not effective.
And not only is it equally effective than the
drug that has already been around, it’s
inconvenient”
Trials, 103:10-21. 12Murray. 2019. Pragmatic randomized

Superiority estimates are
patient-centered
Murray. 2019. Pragmatic Randomized 13Murray et al. 2018 J Clin
Trials, 103:10-21.

Patients understand absolute
measures
Murray. 2019. Pragmatic Randomized 14Murray et al. 2018 J Clin
Trials, 103:10-21.

Patients who expect to adhere may
change their choices based on per-
protocol effects
“It would depend on how critical
the case was. If I had serious
COPD and there was, both
parents had died of it, I would
say, ‘You know what? I am
committed to my health. I’m
committed to taking it as
prescribed.’ So I’d be willing to
try the new [less convenient]
drug.”
Trials, 103:10-21.

Per-protocol effects are patient-
centered when patients expect to
adhere
Trials, 103:10-21.

Part III: Per-protocol effects
are patient-centered, but can
we estimate them?

First, a closer look at the
intention-to-treat effect
The ITT answers the question
 How does the outcome change if
everyone is randomized to
exposure A vs exposure B?
Murray. 2019. Pragmatic Randomized

Intention-to-treat effect gives an
unbiased estimate of effect of
randomization
Randomization ensures no
confounding at baseline for
treatment assignment
Treatment happens after randomization
Loss to follow-up happens after
randomization
Post-randomization events are not
guaranteed to be unconfounded!

Effect of randomization is not really
an interesting effect
Often a lower bound on the effect of
treatment compared to placebo
Lower bound is insufficient for
adverse events or safety
When comparing active treatments,
ITT can vary towards or away from
the null
No real world, clinical, equivalent of
randomization

Effect of treatment is an interesting
effect
Relevant for real world, clinical,
decision making
Allows better risk assessment for
adverse events or safety
Interpretable for both placebo and
active / usual care comparators
21
Per-protocol effect is
the effect we really
want!
Murray. 2019. Pragmatic randomized

What is the per-protocol
effect?
A per-protocol effect answers the
question
How does the outcome change if
everyone is receives exposure A vs
exposure B?

This is what we want but we
can’t get it from data alone
All causal effects require that we
make assumptions that cannot
be verified in the data

What assumptions do we
need?
1. No unmeasured confounding: all
common causes of the treatment
and outcome are known and
measured in the data

Intention-to-treat effect has no
unmeasured confounding for
randomization

always requires adjustment for
confounding

need?
2. Positivity: non-zero probability of
all levels of treatment for all
individuals in our target population
(i.e. variability in exposure)

Randomization guarantees
positivity for the intention-to-
treat effect

Per-protocol effects may not
have positivity

need?
3. Consistency: our treatment levels
are clear and well-defined
 Are we asking a specific enough
question to get an answer we can
understand?

Why are well-defined
interventions important?
When there are multiple possible
‘interventions’ and we don’t specify
one, our answer is a weighted
average of all interventions but we
don’t know the weights

Why are well-defined
interventions important?
But, if the intervention is ill-defined
the confounding is probably also ill-
defined!

Recap: causal effect options
1.Intention-to-treat effects
 Effect of randomization to treatment
2.Per-protocol effects: effect of
treatment
 Effect of initiating treatment
 Effect of adhering to treatment protocol
 Effect of receiving point intervention,
among the ‘compliers’ (not necessarily
all adherers!)

But, per-protocol analyses
have a bad reputation
34

What is a per-protocol
analysis?
Common approaches
 censor when non-adherent,
don’t adjust for confounding
 add adherence to regression
model, adjust only for baseline
confounders
35
Common ≠ correct!

Per-protocol analyses tell us
how did trial outcomes differ
between those who did adhere
to, or recieved, assignment A
and those who did adhere to,
or receive, assignment B?

Per-protocol analyses in the
literature
Approach Description
1. “Modified ITT”  censor never initiators
2. “As-treated”  allow cross-over
 censor non-initiators or
discontinuers
3. “Per-protocol
population”
 censor if never initiate,
cross-over, or
discontinuation
4. Adherence
adjustment
 include adherence in
model for outcome
model
5. Instrumental
variables, aka
 compare outcome by
trial arm, and correct 37Murray. 2019. Pragmatic randomized

literature
discontinuers
3. “Per-protocol
population”
cross-over, or
discontinuation
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka

literature
2. “Per-protocol
population”
cross-over, or
discontinuation
3. “Per-protocol
population”
cross-over, or
discontinuation
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka

literature
2. “Per-protocol
population”
cross-over, or
discontinuation
discontinuers
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka
trial arm, and correct 40
Methods 1 to 3:
Censor without
adjustment

literature
2. “Per-protocol
population”
cross-over, or
discontinuation
discontinuers
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka
Methods 1 to 3:
Censor without
adjustment
Method 4:
Adjustment for
baseline confounding
only

Potential per-protocol
analyses
2. “Per-protocol
population”
cross-over, or
discontinuation
discontinuers
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka
Methods 1 to 3:
Censor without
adjustment
Method 4:
Adjustment for
only

But isn’t adherence intractably
confounded?
43
Coronary Drug Project. 1980, NEJM; 303: 1038-41.
0
5
10
15
20
25
30
Unadjusted Baseline adjusted
Non-adherers
Adherers
5-year mortality risk in CDP placebo arm

Effects are different from
analyses
Per-protocol effect tells us
“how would trial outcomes differ
if everyone adhered to
assignment A versus if everyone
adhered to assignment B”

Better per-protocol
analyses
discontinuers
3. “Per-protocol
population”
cross-over, or
discontinuation
4. Adherence
adjustment
model for outcome
model
5. Instrumental
variables, aka
Methods 1 to 3:
Censor without
adjustment
Method 4:
Adjustment for
only
Per-protocol
effect estimation
 censor if deviate from
protocol or include
adherence in outcome
model
 adjust for censoring or
time-varying
confounding

What do we need to adjust
for?
Hernan & Robins. 2017, NEJM 377:14
A.Random non-adherence
No confounding adjustment needed

How to choose analytic
approach?
A. Random: No adjustment needed
B. Adherence confounding by measured
covariates
 Adjustment required using any method

approach?
B. Measured covariates: adjustment using any method
C.Adherence confounding by measured
covariates and prior adherence
 G-methods required

approach?
B. Measured covariates: adjustment using any method
C. Measured covariates & adherence: g-methods
D.Adherence confounding by measured
covariates, prior adherence, and
unmeasured covariates
 Strong assumptions + structural nested 49Murray. 2019. Pragmatic randomized

A very quick intro to g-
methods
1. Inverse probability weighting
2. G-formula
3. G-estimation

G-methods generalize
estimation to treatment-
confounder feedback

G-methods are roughly similar
to …
Inverse probability
weighting
G-formula
G-estimation
Propensity
scores
Standardization
Instrumental
variables
≈
≈
≈

Revisiting the Coronary Drug
Project
-20
-10
0
10
20
Unadjusted Baseline adjustment Post-randomization IPW
Replication Cumulative
incidence model
Survival
model,
censoring
Survival model,
flexible dose-
response
Survival model,
quadratic
dose-response
Murray & Hernan. 2016, Clin Trials 13(4): 372-8.
Murray & Hernan. 2018, Trials 19: 158.

Estimating the per-protocol
effect
One approach: inverse probability
weighting
Step 1: define adherence to the protocol
Step 2: build inverse probability weights
for adherence and fit separately in each
trial arm
Step 3 [optional]: censor when non-
adherent
Step 4: fit an IP-weighted model for the
outcome given trial arm, baseline
covariates, and [optional] dose- 54Murray. 2019. Pragmatic randomized

Part V: Some guidelines for
patient-centered causal
effects from pragmatic trials

Start by considering the
goal(s)
 FDA approval
 Clinical decision-making
 Patient-clinician shared
decision-making
 Other
Take home: What causal
effect(s) do you need to know to
achieve this goal? 56Murray. 2019. Pragmatic randomized

Choose an analytic approach
that answers the causal
question
 Randomization to
treatment
 Initiation of
treatment among
‘compliers’
 Initiation of
treatment among
everyone
 Sustained
treatment
(protocol)
 Intention-to-treat &
stratified ITT
 Per-protocol with
instrumental
variables
 Per-protocol effect
estimation

Designing patient-centered
trials
 Patients mistrust
non-inferiority
 Stratified effects
are patient-
centered effects
 Patients assess
per-protocol
effects based on
reasons for non-
adherence
 Patients want to
 Choose superiority
goals
 Involve patients/
advocates in
choosing a priori
strata
 Include information
about why
participants don’t
adhere
 Report risks &58Murray. 2019. Pragmatic randomized

Finally, don’t forget about loss
to follow-up!
 Loss to follow-up can cause bias
away from the null even in the
intention-to-treat effect!
 Loss to follow-up is inherently
post-randomization, so requires
adjustment for post-randomization
confounding
Solutions:
 Inverse probability weighting
 G-formula
 Multiple imputation 59Murray. 2019. Pragmatic randomized

Draft pragmatic trial
guidelines
Choice of causal effect
1. Report estimates of both the intention-to-treat
effect and the per-protocol effect.
2. Report absolute risks and their differences, as well
as their ratios, for discrete outcomes.
3. Heterogeneity of treatment effects can be reported
using subgroup analyses that use the additive scale.
4. Pre-specify important prognostic factors &
maximum acceptable difference between groups;
adjust via standardization or inverse probability
weighting.
5. In sensitivity analyses, adjust for large imbalances in
any important prognostic factors, regardless of pre-
specification.
6. In survival analyses with competing events, specify
the intention-to-treat effect as the total effect of
treatment assignment; conduct sensitivity analyses

guidelines
Valid effect estimates in the presence of loss to follow-up
7. Collect post-randomization time-varying prognostic
factors that predict loss to follow-up, & use g-
methods to adjust.
Estimating the per-protocol effect of a point intervention
8. Per-protocol effects of point interventions can be
estimated using inverse probability weighting or
standardization.
9. Estimate bounds for the per-protocol effect of point
interventions when the instrumental conditions are
expected to hold for treatment assignment. Provide a
justification for the exclusion restriction.
10. When instrumental conditions and monotonicity hold,
discuss whether the effect in the “compliers” is of
interest. If so, estimate it & provide information on
the relative size and characteristics of the “compliers”

guidelines
Estimating the per-protocol effect for a sustained
intervention
11. Specify a priori a treatment protocol that
incorporates real world clinical decision-making.
When there is sufficient ambiguity about appropriate
strategies, more than 1 can be specified.
12. Collect sufficient data to determine adherence
throughout the follow-up, and to adjust for time-
varying prognostic factors that predict adherence.
13. Use g-methods to adjust for time-varying
confounders when there is treatment-confounder
feedback.

Where to get more
information
Some references:
 Proposed guidelines:
https://www.hsph.harvard.edu/causal/pragmatictrials/
 Patient-centered causal effects: Murray et al. 2018. J Clin Epi 103:10-21.
 Choosing a causal effect: Hernan & Scharfstein. 2018, Ann Intern Med;
168(7):515-6.
 Per-protocol effect estimation: Hernan & Robins. 2017, NEJM 377:14;
Lodi et al, 2016. AIDS; 30(17):2659-63.
 Placebo arm adherence analyses: Murray & Hernan. 2018, Trials 19:158;
Murray & Hernan. 2016, Clin Trials 13(4): 372-8.
 G-methods: Causal Inference, Hernan & Robins. Available online at:
https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/
Contact me:
@EpiEllie
ejmurray@bu.edu
https://github.com/eleanormurray

Causal Inference from Pragmatic Trials.

More Related Content

What's hot (20)

Similar to Causal Inference from Pragmatic Trials. (20)

Recently uploaded (20)

Causal Inference from Pragmatic Trials.

Editor's Notes