The implications of parameter independence in probabilistic sensitivity analysis: An empirical test

Providing Consultancy &
Research in Health Economics
The implications of parameter
independence in probabilistic
sensitivity analysis:
An empirical test
Matthew Taylor, YHEC

 Background
 The use of PSA in healthcare decision making
 Some preliminary tests
 Using an empirical model and various approaches to
correlation
 Some preliminary recommendations
 What can we tell researchers and decision makers?
Overview

 ICERs (or NMB, NHB etc.) tell us about benefits
and opportunity costs
 Decision makers are concerned with uncertainty
 One-way sensitivity analysis is of limited use
(likely to be many interactions that are not
immediately obvious)
 Hence, multi-way sensitivity analysis or PSA
 Used for value of information analysis to guide future
research
PSA in healthcare

Adalsteinsson E & Toumi M J Market Access & Health Policy 2013
Overview

 Base case ICER
 Price of drug, effectiveness of drug, other
 Size of scatter plot
 e.g. confidence intervals from evidence base
 Shape of scatter plot
 Interaction between parameters
Drivers of CEAC

 More often than not, parameters are varied
independently
 Whilst it is possible to apply covariance between
parameters, this is often ignored due to:
– Arbitrary decision
– Lack of understanding of ‘real’ interactions and
covariance
– Lack of available data to generate covariance
matrices
In HTA submissions

 “Only 1 of the 18 reviewed TAs considered the
incorporation of correlation and dependencies
between parameters”
Lanitis T, Muszbek N & Tichy E 2014
 “The assumption of no correlation can lead to
misleading probabilistic results and the
overestimation of uncertainty”
ibid
Arbitrary (?) decision

 “Only 1 of the 18 reviewed Tas considered the
incorporation of correlation and dependencies
between parameters”
Lanitis T, Muszbek N & Tichy E 2014
 “The assumption of no correlation can lead to
misleading probabilistic results and the
overestimation of uncertainty”
ibid
Arbitrary (?) decision

 Quite typical to see:
 Choleski decomposition for survival functions (e.g.
alpha and beta parameters)
 Simple correlations between progression-free survival
and overall survival
 Dirichlet for multiple outcomes
 More (very) rare to see correlations between:
 Utility and cost
 Effectiveness and safety
 Dose intensity and effectiveness (…and safety)
Lack of understanding

 Even if we had:
 One single data source
 …containing all relevant inputs to a model
 …over a long enough time period
 …and used Bayesian Markov chain Monte Carlo
(MCDC) methods or other techniques
 …the sample size would not be sufficient to
capture all permutations of correlation (models
may contain >100 input parameters)
Lack of data

 To build a simple model to explore the impact of
different types of parameter interactivity
 To develop some simple rules of thumb for
reviewers and decision-makers to consider
when interpreting PSA
 To understand the direction of biases
Aims

 2 parameters, A and B
 A: Mean = 5, SE = 3 (assume normal distribution)
 B: Mean = 7, SE = 4 (assume normal distribution)
 Outcome = A + B
Illustration

0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
=
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
M: 12.00
SD: (5.04)
Assuming Independent

0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
= M: 12.00
SD: (7.31)
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
+’ve correlation

0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
=
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
M: 12.00
SD: (5.04)
No correlation

0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+
= M: 12.00
SD: (0.98)
0
50
100
150
200
250
300
350
0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930
-’ve correlation

How does this apply to
CE models?
 Base case ICER
 Price of drug, effectiveness of drug, other
 Size of scatter plot
 e.g. confidence intervals from evidence base
 Shape of scatter plot
 Interaction between parameters

Example model
• Built an eight-state Markov model
• Applied costs and utilities to each state
• Then:
(i) Varied all parameters independently (i.e. no
correlation)
(ii) Varied costs together, utilities together and
probabilities together (“single multipliers”)
(iii)Varied all parameters together (i.e. 100%
correlation)

Example

Example
Independent sampling
in 144 parameters
leads to a cancelling
out effect
More pronounced for
effectiveness

Example (mean outcomes)
Mean
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic £28,135 £17,167 2.47 1.62 £10,968 0.85 £12,875 £6,069 n/a
100% indep. £28,277 £17,299 2.49 1.63 £10,978 0.86 £12,819 £6,150 99.40%
Partially corr. £28,045 £17,191 2.46 1.60 £10,854 0.86 £12,592 £6,386 82.70%
100% corr. £28,069 £17,476 2.47 1.60 £10,592 0.87 £12,121 £6,885 72.50%

Example (SDs)
Mean
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic £28,135 £17,167 2.47 1.62 £10,968 0.85 £12,875 £6,069 n/a
100% indep. £28,277 £17,299 2.49 1.63 £10,978 0.86 £12,819 £6,150 99.40%
Partially corr. £28,045 £17,191 2.46 1.60 £10,854 0.86 £12,592 £6,386 82.70%
100% corr. £28,069 £17,476 2.47 1.60 £10,592 0.87 £12,121 £6,885 72.50%
Standard deviation
Tx
cost
Cx
cost
Tx
QALY
Cx
QALY
Inc
costs
Inc
QALYs
ICER NMB CE
Deterministic n/a n/a n/a n/a n/a n/a n/a n/a n/a
100% indep. £1,856 £1,963 0.11 0.07 £1,964 0.09 n/a £2,316 n/a
Partially corr. £1,892 £2,238 0.41 0.23 £1,454 0.41 n/a £7,275 n/a
100% corr. £1,204 £3,072 0.52 0.07 £1,914 0.59 n/a £9,939 n/a

 Confidence intervals
 Number of health states
 Granularity of inputs
 Base case ICER/NMB (proximity to λ)
 Disease pathways
Other exploration

 Baseline (SE ~ 10% of mean)
 Smaller standard error (~5%)
 Larger standard error (~15%)
Degree of uncertainty

Small SE

Medium SE

Large SE

(How much does it matter?)
Small SE

Medium SE

Large SE

Range Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
Partially correlated 59.80% 54.50% 49.00%
100% correlated 61.70% 53.80% 50.80%
0%
20%
40%
60%
80%
100%
Small SE Baseline Large SE
100% independent Partially correlated 100% correlated

Range Small SE Baseline Large SE
100% independent 84.70% 69.40% 63.10%
100% correlated 61.70% 53.80% 50.80%
Range 24.90% 15.60% 14.10%
0%
10%
20%
30%
40%
50%
How much does
it matter?

Small SE
Different λ

Medium SE
Different λ

Large SE
Different λ

100% independent 84.70% 69.40% 63.10%
100% correlated 61.70% 53.80% 50.80%
Range 24.90% 15.60% 14.10%
0%
10%
20%
30%
40%
50%
100% independent 100.00% 99.66% 95.10%
100% correlated 88.90% 73.74% 66.80%
Range 11.10% 25.92% 28.30%
0%
10%
20%
30%
40%
50%
λ = £20,000
(ICER close to λ)
λ = £30,000
(ICER well below λ)

λ = £20,000
(ICER close to λ)
λ = £30,000
(ICER well below λ)
100% independent 84.70% 69.40% 63.10%
100% correlated 61.70% 53.80% 50.80%
0%
20%
40%
60%
80%
100%
100% independent 100.00% 99.66% 95.10%
100% correlated 88.90% 73.74% 66.80%
0%
20%
40%
60%
80%
100%

Number of health states
 Base case = 8 states
 Tested: 3 to 8

8 states

7 states

6 states

5 states

4 states

3 states

Health states in model
3 4 5 6 7 8
100% independent 57.80% 62.41% 63.56% 66.68% 67.13% 69.85%
Partially correlated 58.96% 54.45% 54.17% 52.52% 52.85% 53.31%
100% correlated 63.14% 57.99% 54.43% 54.59% 53.72% 53.85%
0%
20%
40%
60%
80%
3 4 5 6 7 8
%CE
Cancelling
out effect

Health states in model
Range
3 4 5 6 7 8
100% independent 57.80% 62.41% 63.56% 66.68% 67.13% 69.85%
Partially correlated 58.96% 54.45% 54.17% 52.52% 52.85% 53.31%
100% correlated 63.14% 57.99% 54.43% 54.59% 53.72% 53.85%
Range 5.34% 7.96% 9.39% 14.16% 14.28% 16.54%
0%
5%
10%
15%
20%
3 4 5 6 7 8

Granularity of costs
 Baseline = 1 single cost for each health
state
 Scenarios: 3, 5 and 10 components
 100% independent
 Partially correlated
 100% correlation

Sub-costs
%CE 1 3 5 10
100% independent 69.85% 72.13% 76.21% 83.48%
Partially correlated 53.31% 55.20% 56.12% 56.97%
100% correlated 53.85% 53.80% 53.91% 53.33%
0%
20%
40%
60%
80%
100%
1 3 5 10
Cancelling
out effect

Sub-costs
Range 1 3 5 10
100% independent 69.85% 72.13% 76.21% 83.48%
Partially correlated 53.31% 55.20% 56.12% 56.97%
100% correlated 53.85% 53.80% 53.91% 53.33%
Range 16.54% 18.33% 22.30% 30.15%
0%
10%
20%
30%
40%
1 3 5 10

Proximity to threshold
Cost of drug
%CE
£0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
Partially correlated 99.73% 96.64% 81.45% 52.31% 26.50% 10.30% 3.60% 1.70%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
0%
20%
40%
60%
80%
100%
£0 £5k £10k £15k £20k £25k £30k £35k
Always
CE Scatter
matters
50-50%
Scatter
matters
Never
CE

£0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
Range 1.13% 10.00% 25.92% 17.04% 29.80% 21.10% 10.80% 6.20%
NMB £16,069 £11,069 £6,069 £1,069 -£3,931 -£8,931 -£13,931 -£18,931
0%
10%
20%
30%
40%
50%
£0 £5k £10k £15k £20k £25k £30k £35k
Cost of drug
Range

£0 £5k £10k £15k £20k £25k £30k £35k
100% independent 100.00% 100.00% 99.66% 69.35% 4.40% 0.10% 0.00% 0.00%
100% correlated 98.87% 90.00% 73.74% 53.68% 34.20% 21.20% 10.80% 6.20%
Range 1.13% 10.00% 25.92% 17.04% 29.80% 21.10% 10.80% 6.20%
NMB £16,069 £11,069 £6,069 £1,069 -£3,931 -£8,931 -£13,931 -£18,931
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
-£25,000 -£20,000 -£15,000 -£10,000 -£5,000 £0 £5,000 £10,000 £15,000 £20,000
NMB
Range

 We already knew that parameter correlation
is important
 There is a perception that ignoring correlation
increases the uncertainty
 This can be misunderstood by decision-
makers
 Very often, ignoring correlation underestimates
the uncertainty
Summary

 We can assign covariance or other interaction
terms (e.g. using the Cholesky decomposition
method and other techniques)
 However, very unlikely that we will have suitable
data to quantify this adequately
 As a minimum, we should aim to understand the
likely direction of the consequences
What can we do?

What can we do?
Characteristic Consequence of ignoring correlation
More health states Likely to overstate confidence
Greater granularity of inputs Likely to overstate confidence
ICER proximity to λ
Likely to be a problem when fairly close
(but not very close) to threshold
other… etc…
other... etc…
But…

What can we do?
Too convoluted?

http://tinyurl.com/yhec-facebook
http://twitter.com/YHEC1
http://www.minerva-network.com/
http://tinyurl.com/YHEC-LinkedIn
Thank you
matthew.taylor@york.ac.uk
Telephone: +44 1904 323631
Website: www.yhec.co.uk

The implications of parameter independence in probabilistic sensitivity analysis: An empirical test

More Related Content

What's hot (20)

Similar to The implications of parameter independence in probabilistic sensitivity analysis: An empirical test (20)

More from cheweb1 (20)

Recently uploaded (20)

The implications of parameter independence in probabilistic sensitivity analysis: An empirical test