RSS local 2012 - Software challenges in meta-analysis

[Poster title]
[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4
1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]
Meta-analysis overview
A practical guide
Summary
Software and model selection challenges in
meta-analysis
MetaEasy, model assumptions and homogeneity
Evan Kontopantelis David Reeves
Health Sciences Primary Care Research Centre
University of Manchester
RSS Primary Care Study Group
Errol Street, 2 July 2012
Kontopantelis, Reeves Software and model selection challenges in meta-analysis

[Poster title]
A practical guide
Summary
Outline
1 Meta-analysis overview
The heterogeneity issue
More challenges
2 A practical guide
the MetaEasy add-in
metaeff & metaan
Methods and performance
ˆτ2 = 0
3 Summary

[Poster title]
A practical guide
Summary
More challenges
Heterogeneity
The big bad wolf
When the effect of the intervention varies signiﬁcantly from
one study to another.
It can be attributed to clinical and/or methodological
diversity.
Clinical: variability that arises from different populations,
interventions, outcomes and follow-up times.
Methodological: relates to differences in trial design and
quality.
Detecting quantifying and dealing with heterogeneity can
be very hard.

[Poster title]
A practical guide
Summary
More challenges
Absence of heterogeneity
Assumes that the
true effects of the
studies are all
equal and
deviations occur
because of
imprecision of
results.
Analysed with the
ﬁxed-effects
method.

[Poster title]
A practical guide
Summary
More challenges
Presence of heterogeneity
Assumes that
there is variation in
the size of the true
effect among
studies (in addition
to the imprecision
of results).
Analysed with
random-effects
methods.

[Poster title]
A practical guide
Summary
More challenges
Challenges with meta-analysis
Heterogeneity is common and the fixed-effect model is
under fire.
Methods are asymptotic: accuracy improves as studies
increase. But what if we only have a handful, as is usually
the case?
Almost all random-effects models (except Profile
Likelihood) do not take into account the uncertainty in ˆτ2.
Is this, practically, a problem?
DerSimonian-Laird is the most common method of
analysis, since it is easy to implement and widely available,
but is it the best?

[Poster title]
A practical guide
Summary
More challenges
Challenges with meta-analysis
...continued
Can be difﬁcult to organise since...
outcomes likely to have been disseminated using a variety
of statistical parameters
appropriate transformations to a common format required
tedious task, requiring at least some statistical adeptness
Parametric random-effects models assume that both the
effects and errors are normally distributed. Are methods
robust?
Sometimes heterogeneity is estimated to be zero,
especially when the number of studies is small. Good
news?

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Based on our original work...

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Organising
Data initially collected using data extraction forms.
A spreadsheet is the next logical step to summarise the
reported study outcomes and identify missing data.
Since in most cases MS Excel will be used we developed
an add-in that can help with most processes involved in
meta-analysis.
More useful when the need to combine differently reported
outcomes arises.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What it can do
Help with the data collection using pre-formatted
worksheets.
Its unique feature, which can be supplementary to other
meta-analysis software, is implementation of methods for
calculating effect sizes (& SEs) from different input types.
For each outcome of each study...
it identiﬁes which methods can be used
calculates an effect size and its standard error
selects the most precise method for each outcome

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What it can do
...continued
Creates a forest plot that summarises all the outcomes,
organised by study.
Uses a variety of standard and advanced meta-analysis
methods to calculate an overall effect.
a variety of options is available for selecting which
outcome(s) are to be meta-analysed from each study
Plots the results in a second forest plot.
Reports a variety of heterogeneity measures, including
Cochran’s Q, I2, HM
2
and ˆτ2 (and its estimated conﬁdence
interval under the Proﬁle Likelihood method).

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Advantages
Free (provided Microsoft Excel is available).
Easy to use and time saving.
Extracted data from each study are easily accessible, can
be quickly edited or corrected and analysis repeated.
Choice of many meta-analysis models, including some
advanced methods not currently available in other software
packages (e.g. Permutations, Proﬁle Likelihood, REML).
Unique forest plot that allows multiple outcomes per study.
Effect sizes and standard errors can be exported for use in
other meta-analysis software packages.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Installing
Latest version available from www.statanalysis.co.uk
Compatible with Excel 2003, 2007 and 2010.
Manual provided but also described in:
Kontopantelis E and Reeves D.
MetaEasy: A Meta-Analysis Add-In for Microsoft Excel.
Journal of Statistical Software, 30(7):1-25, 2009.
Play video clip

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Stata implementation
MetaEasy methods implemented in Stata under:
metaeff, which uses the different study input to provide
effect sizes and SEs
metaan, which meta-analyses the study effects with a
ﬁxed-effect or one of ﬁve available random-effects models
To install, type in Stata:
ssc install <command name>
help <command name>
Described in:
Kontopantelis E and Reeves D.
metaan: Random-effects meta-analysis.
The Stata Journal, 10(3):395-407, 2010.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Many random-effects methods
which to use?
DerSimonian-Laird (DL): Moment-based estimator of both
within and between-study variance.
Maximum Likelihood (ML): Improves the variance estimate
using iteration.
Restricted Maximum Likelihood (REML): an ML variation
that uses a likelihood function calculated from a
transformed set of data.
Proﬁle Likelihood (PL): A more advanced version of ML
that uses nested iterations for converging.
Permutations method (PE): Simulates the distribution of
the overall effect using the observed data.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Performance evaluation
our approach
Simulated various distributions for the true
effects:
Normal.
Skew-Normal.
Uniform.
Bimodal.
Created datasets of 10,000
meta-analyses for various numbers of
studies and different degrees of
heterogeneity, for each distribution.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Performance evaluation
our approach
Compared all methods in terms of:
Coverage, the rate of true negatives when the overall true
effect is zero.
Power, the rate of true positives when the true overall effect
is non-zero.
Conﬁdence Interval performance, a measure of how wide
the (estimated around the effect) CI is, compared to its true
width.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Homogeneity
Zero between study variance
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
zero between study variance
Coverage
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Power (25th centile)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Overall estimation performance

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Coverage performance
Small and large heterogeneity under various distributional assumptions
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
H²=1.18 − I²=15% (normal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
H²=1.18 − I²=15% (skew−normal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
H=1.18 − I=15% (bimodal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
H=1.18 − I=15% (uniform distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Coverage

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Power performance
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
CI performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
Confidence Interval performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Overall estimation performance (medians)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Overall estimation performance (medians)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Coverage by method
Large heterogeneity across various between-study variance distributions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Zero variance Normal Skew−normal
Bimodal Uniform
H=2.78 − I=64%
FE − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Coverage
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Coverage
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Coverage

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Power by method
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
FE − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Power (25th centile)
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Power (25th centile)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Power (25th centile)

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
CI performance by method
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
FE − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Overall estimation performance

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Which method then?
Within any given method, the results were consistent
across all types of distribution shape.
Therefore methods are highly robust against even severe
violations of the assumption of normality.
Choose PE if the priority is an accurate Type I error rate
(false positive).
But low power makes it a poor choice when control of the
Type II error rate (false negative) is also important and it
cannot be used with less than 6 studies.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Which method then?
For very small study numbers (≤5) only PL gives coverage
>90% and an acccurate CI.
PL has a ‘reasonable’ coverage in most situations,
especially for moderate and large heterogeneiry, giving it
an edge over other methods.
REML and DL perform similarly and better than PL only
when heterogeneity is low (I2 < 15%)
The computational complexity of REML is not justiﬁed.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Bring on the champagne?
Does not necessarily mean homogeneity.
Most methods use biased estimators and not uncommon
to get a negative ˆτ2 which is set to 0 by the model.
We identiﬁed a large percentage of cases where the
estimators failed to identify existing heterogeneity.
In our simulations, for 5 studies and I2 ≈ 29%:
30% of the meta-analyses were erroneously estimated to
be homogeneous under the DL method.
32% for REML and 48% for ML-PL.

[Poster title]
A practical guide
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What does it mean?
In these cases coverage was substandard and was over
10% lower than in cases where ˆτ2 > 0, on average.
The problem becomes less profound as the number of
studies and the level of heterogeneity increase.
Better estimators are needed.
There might be a large number of meta-analyses of
‘homogeneous’ studies which have reached a wrong
conclusion.

[Poster title]
A practical guide
Summary
What to take home
MetaEasy can help you organise your meta-analysis and
can be especially useful if you need to combine continuous
and binary outcomes.
Methods implemented in Stata under metaeff and metaan.
A zero ˆτ2 is a reason to worry. Heterogeneity might be
there but we cannot measure or account for in the model.
If ˆτ2 > 0, even if very small, use a random-effects model.
The DL method works reasonably well, under all
distributions, especially for low levels of heterogeneity.
Proﬁle likelihood, which takes into account the uncertaintly
in ˆτ2, works better when I2 ≥ 15%.

[Poster title]
Appendix
Thank you!
Key references
Comments, suggestions:
e.kontopantelis@manchester.ac.uk

[Poster title]
Appendix
Thank you!
Key references
References
Kontopantelis E, Reeves D.
MetaEasy: A Meta-Analysis Add-In for Microsoft Excel.
Journal of Statistical Software, 30(7):1-25, 2009.
metaan: Random-effects meta-analysis.
The Stata Journal, 10(3):395-407, 2010.
Performance of statistical methods for meta-analysis when
true study effects are non-normally distributed: A simulation
study.
Stat Methods Med Res, published online Dec 9 2010.

RSS local 2012 - Software challenges in meta-analysis

More Related Content

What's hot (18)

Viewers also liked (15)

Similar to RSS local 2012 - Software challenges in meta-analysis (20)

More from Evangelos Kontopantelis (20)

Recently uploaded (20)

RSS local 2012 - Software challenges in meta-analysis