Metaanalysis copy

AMANDEEP KAUR
JUNIOR RESIDENT
DEPARTMENT OF COMMUNITY MEDICINE
PGIMS, ROHTAK

 Introduction
▪ Types of review
▪ Definition
▪ Function of meta-analysis
 Conducting Meta-analysis
▪ Writing the research question and protocol
▪ Comprehensive search
▪ Selection of studies
▪ Appraisal (quality assessment) of studies
▪ Data abstraction
▪ Data analysis

 Effect size
 Presenting the findings – Forest plot
 Heterogeneity
 Dealing with heterogeneity
▪ Fixed and random effects model
▪ Meta-regression
▪ IPD analysis
 Strengths and Weaknesses of meta-analysis
 Software for meta-analysis

Information explosion
More than 1,00,000 articles are
published each year in more
than 20,000 journals.
Humanly impossible to read
through the articles
published in any field.
Publication bias
Concise summaries of literature
(Reviews) required, after
separating insignificant and
unsound from salient and
crucial.

Systematic review
Meta-analysis
Narrative review

“ review articles written by one or more experts based on a convenience
sample of studies with no description of the underlying methodology”
 Confuse ‘absence of proof’ of benefit as ‘proof of absence’ of benefit
 Do not statistically combine results from multiple studies
 Vote-counting

“ a review addressing a specific research question using explicit
methodology of collecting, selecting and appraising studies and,
whenever appropriate, synthesizing their results quantitatively”
 Has only qualitative or both qualitative and quantitative components
 Quantitative component is meta-analysis

“Quantitative approach for systematically combining
results of previous research to arrive at conclusions
about the body of research.”

 1952: Hans J. Eysenck concluded that there were no favorable effects
of psychotherapy, starting a raging debate which 25 years of evaluation research
and hundreds of studies failed to resolve
 1978: To proved Eysenck wrong, Gene V. Glass statistically
aggregated the findings of 375 psychotherapy outcome studies
Glass (and colleague Smith) concluded that psychotherapy did indeed
work.
Glass called the method “meta-analysis”

 Underpinning ideas can be identified earlier:
 K. Pearson (1904)
Averaged correlations for typhoid mortality after inoculation across 5 samples
 R. A. Fisher (1944)
Source of the idea of cumulating probability values
 W. G. Cochran (1953)
Discussed a method of averaging means across independent studies
Set out much of the statistical foundation for meta-analysis (e.g., Inverse variance
weighting and homogeneity testing)

 Identify heterogeneity in effects among multiple studies and, where
appropriate, provide summary measure
 Increase statistical power and precision to detect an effect
 Develop ,refine, and test hypothesis
 Reduce the subjectivity of study comparisons by using systematic and
explicit comparison procedure
 Identify data gap in the knowledge base and suggest direction for future
research
 Calculate sample size for future studies
 Analyses if and how previous studies have modified knowledge on a
certain topic

Writing the research question and a protocol
Comprehensive search
Selection of studies
Appraisal (quality assessment) of studies
Data abstraction
Data analysis

 Research question:
▪ P: the population of interest
▪ I: the intervention or exposure
▪ C: the comparison (in certain situations)
▪ O: the outcome of interest
 Protocol: specifying the –
▪ Research question
▪ Search methods
▪ Inclusion and exclusion criteria for studies
▪ Criteria for quality assessment (appraisal) of the studies
▪ Methods of data abstraction and synthesis

Cochrane Review of
magnesium
sulphate and other
anticonvulsants for
women with pre-eclampsia

 Hand searching – ‘gold-standard’ for published studies
The percent (or proportion) of (relevant) studies found in electronic databases
compared to hand searching is termed as ‘sensitivity’; and percent or proportion of
the yield that is relevant is called ‘specificity’.
 Computerized databases: Pubmed/Medline, EMBASE, Cochrane
Review/Trials Register
 Personal references, and emails
 Web, e.g. Google internet search engine (http://scholar.google.com)
 Conference programs
 Dissertations
 Review articles
 Government reports, bibliographies

 Explicit Inclusion and exclusion criteria
 Study designs: RCTs or CTs with a non-exercise control group
 Subjects: Females > 18 years of age
 Publication types: Journal articles, dissertations, & masters theses
 Languages: English
 Interventions: Bone mineral density assessed at femur, spine, and/or
radius
 Time Frame: Studies published & indexed between January 1966 and
December 1998

Non-randomized trials:
 Treatment allocation related to prognosis or pre-judgment of
appropriateness of treatment
Randomized trials:
 Inadequate randomization (e.g., alternating assignment)
 Lack of stratification on important factors
 Lack of or ineffective blinding
All trials:
 Patient drop-outs, patient switching arms
 Missing data
 Improper statistical analysis

 Quality scores developed by -
▪ Chalmers et al
▪ Jadad et al
 None is absolute best.
 Little is known about their relative merits and their association with
study outcomes.

 Reporting Bias
is a group of related biases potentially leading to over-representation of
significant or positive studies in systematic reviews
 Studies with significant positive findings -
 More likely to be published- Publication bias - over estimation of
treatment effects
 More likely to be published rapidly - Time lag bias
 More likely to be published in English - Language bias
 More likely to be published more than once - Multiple publication bias
 More likely to be cited by others - Citation bias

 Funnel Plot:
 Display the studies included in meta-analysis in a plot of effect size
against sample size (or some other measure of the extent to which the
findings could be affected by the play of chance).
 Egger’s Regression Test:
 Tests whether small studies tend to have larger effect sizes than would
be expected (implying that small studies with small effect sizes have
not been published).
 Begg’s rank correlation test

Symmetrical Funnel Plot
Showing No Publication Bias

An Asymmetric Funnel Plot
(indicative of publication bias)
(Region
of missing
studies)
-2 -1 0 1 2
Log Odds Ratio
Asymmetric plot –
•Publication bias
•Clinical heterogeneity
•Methodological heterogeneity

 Combine the results of larger studies only, which are less likely subject to
publication bias.
 File-drawer Method / Fail safe N: How many unpublished studies showing
a null result are required to change a ‘significant’ meta analysis result to a
‘non-significant’ one?
 ‘Trim and Fill’ method

An Asymmetric Funnel Plot
(indicating publication bias)
-2 -1 0 1 2
Log Odds Ratio
Trimmed
Filled
Estimated # missing
studies : 5

 At least two reviewers
 Sift and sift again
▪ The first sift – pre-screening - is to decide which studies to retrieve in full.
▪ The second sift – selection - is to look again at these studies and decide which
are to be included in your review
 Do not collect outcome data at the same time as eligibility information
▪ wasted time and effort - if study is excluded later on
▪ Results can sway decision
 Look out for duplicate publications

 Create a spreadsheet (Excel, or Open Office Calc)
 For each study, create the following columns:
 name of the study
 name of the author, year published
 number of participants who received intervention
 number of participants who were in control arm
 number who developed outcomes in intervention
 number who developed outcomes in control arm

22 studies to do meta analysis
Seven columns created
trial: trial identity code
trialnam: name of trial
year: year of the study
pop1: study population
deaths1: deaths in study
pop0: control population
deaths0: deaths in control

 Choice of metric :
▪ Original
▪ Standardized mean difference (Mean/Standard Deviation)
 Publication bias:
▪ Graphical methods
▪ Quantitative methods
 Choice of model/ heterogeneity:
▪ Fixed Effects
▪ Random Effects

35
Data Type Outcome Measures
Continuous Mean
Dichotomous (binary)
(displayed in 2x2 table)
Odds ratio (OR),
Risk ratio (RR),
Risk difference (RD)

 For continuous outcomes, the mean difference (effect size) is usually used
to compare treatment and control groups
 Effect sizes are standardized by the pooled estimate of the (common)
within-group variance
 For skewed continuous outcomes,
 values may be transformed (e.g. logarithmic), or
 the median may be used
36

Failure Success
Treatment a b
Control c d
Odds: Treatment: a/b, Control: c/d
Odds Ratio =
ad
bc
a /
b
c d

/
OR < 1 implies treatment effectiveness (protective)
OR > 1 indicative of treatment inferiority (risk)

 For the purposes of combining, analysis may be presented in
terms of log (OR), i.e. as a difference of log (Odds) of treatment
and control.
 Var[log (OR)] =
1 1 1 1
  
a b c d
 If any of the cell-counts is less than 5, use continuity correction
(add 0.5) before calculating OR.

a a b
/( 
)
c /( c 
d
)
RR =
 RR is also called the Risk Ratio
 It represents the probability of an event (failure) in the treatment
group relative to the probability of the same event in the control
group.
 RR is analyzed in log scale.
Var[log(RR)] =
1 1 1 1
 
a a 
b c c  d


RD =
c
c d
a
a b



 RD is the difference of two binomial probabilities, while
RR is the ratio.
Var(RD) =
,
p p 
p p
(1 ) (1 ) 1 1 2 2
m

n

where n=a+b, m=c+d, p1= a/n, p2=c/m

Failure Success Total
New Treatment 5 95 100
Control 10 90 100
Odds Ratio = (5/95) / (10/90) = 0.48
Risk Ratio = (5/100) / (10/100) = 0.50
(Recall OR  RR when probability is small. OR is generally more extreme (further from 1) than RR.)
Risk Difference = (5/100) - (10/100) = -0.05

 The effect size makes meta-analysis possible
“ratio of the frequency of the events in the intervention to that in the
control group.”
 Any standardized index can be an “effect size” (e.g., standardized mean
difference, correlation coefficient, odds-ratio) as long as it –
 Is comparable across studies (generally requires standardization)
 Represents the magnitude and direction of the relationship of interest
 Is independent of sample size
 Different meta-analyses may use different effect size indices

gHedges 
YExperimental  YControl
((NE  1)  SD2
E  (NC  1)SD2
C )) / (NTot  2)
 1
3
4(NE  NC )  9

 

 
Glass 
YExperimental YControl
SDControl
dCohen 
YExperimental  YControl
(SD2
E  SD2
C ) / 2

ES = 0.00
Control
Group
Intervention
Group
Overlapping
Distributions

Control
Group
Treatment
Group
ES = 0.40

Control
Group
Intervention
Condition
ES = 0.85

 The graphical display of results from individual studies on a common
scale is a “Forest plot”.
 Each study is represented by a black square and a horizontal line
(CI:95%).
 The area of the black square reflects the weight of the study / precision
of the study (roughly the sample size).
 A logarithmic scale should be used for plotting the Relative Risk /
Odds Ratio.
 Aggregate Effect size – displayed as a ‘diamond’.

The impact of fish oil consumption on Cardio-vascular diseases

 Look at the title of the forest plot, the intervention, outcome effect measure
of the investigation and the scale
 The names on the left are the authors of the primary studies included in
the MA
 The small squares represent the results of the individual trial results
 The size of each square represents the weight given to each study in the
meta-analysis
 The horizontal lines associated with each square represent the confidence
interval associated with each result
 The vertical line represents the line of no effect, i.e. where there is no
statistically significant difference between the treatment/intervention
group and the control group
 The pooled analysis is given a diamond shape. The horizontal width of the
diamond is the confidence interval

Effect of probiotics on the risk of antibiotic associated diarrhoea
D'Souza, A. L et al. BMJ 2002;324:1361

Effect of probiotics on the risk of antibiotic associated diarrhoea

Reviews usually bring together studies that were performed
 By different people
 In different settings
 In different countries
 On different people
 In different ways
 For different lengths of time
 To look at different outcomes
........… and these aren’t the only differences.
 Assessing combinability
Types of heterogeneity:
•Clinical heterogeneity
•Methodological
heterogeneity
•Statistical heterogeneity

 Test for existence of heterogeneity: have low power
▪ Cochrane’s Q – statistic based on chi-square test
▪ I2 statistic – scores heterogeneity between 0% and 100%
 25% - low heterogeneity
 50% - moderate
 75% - high
 Presence or absence of heterogeneity influences the subsequent method
of analysis:
▪ Fixed- effects model
▪ Random effect model
 Meta-regression: to over come heterogeneity

FIXED EFFECTS MODEL
• Conduct, if heterogeneity is absent
• Assumes the size of treatment effect be
same (fixed) across all studies &
variation due to chance
• Pooling: Mantel Haenszel OR
• Weight = 1/variance
= 1/SE2
• When heterogeneity exists we get:
• a pooled estimate which may give too
much weight to large studies,
• a confidence interval which is too
narrow,
• a P-value which is too small.
RANDOM EFFECTS
MODEL
• Conduct, if heterogeneity is present
• Assumes that the size of treatment
effect does vary between studies
• Der Simonian Laird method (DSL)
for Odds’ Ratio
• Weight = 1/variance
= 1/(SE2+ inter-trial variance)
• When heterogeneity exists we get:
• possibly a different pooled estimate
with a different interpretation,
• a wider confidence interval,
• a larger P-value

FIXED EFFECTS MODEL
• When heterogeneity does not exists:
• a pooled estimate which is correct,
• a confidence interval which is correct,
• a P-value which is correct.
RANDOM EFFECTS
MODEL
• When heterogeneity does not exist:
• a pooled estimate which is correct,
• a confidence interval which is too wide,
• a P-value which is too large
No universally accepted method for choosing.
A reasonable approach:
1. Decide whether the assumption of a fixed effects model is plausible. Could the
studies all be estimating the same effect? If not, consider a random effects model.
2. If fixed effects assumption is plausible, are the data compatible?
Graphical methods: forest plot, Galbraith plot.
Analytical methods: heterogeneity test, I2 statistic.
If assumption looks compatible with the data, use fixed effects, otherwise consider
random effects.

 The estimate of study results is the dependent variable and
one or more study-level variables are the independent
variables (predictors)
 Allows researchers to explore which types of patient-specific
factors or study design factors contribute to heterogeneity.
 Limited ability to identify important factors – struggles to
identify which patient features are related to the size of
treatment effect.

 Involves the central collection, checking and analysis of updated
Individual Patient Data
 Include all properly randomised trials, published and unpublished
 Include all patients in an intention-to-treat analysis
 Analysis stratified by trial
 IPD does not mean that all patients are combined into a single mega
trial; meta-analysis looks at the results within each study, and then
calculates a weighted average.
 Obtaining individual patient data from each of the trials is
challenging

 Collect raw data from related studies, whether or not the
studies collaborated at the design stage, exposures measures
and other covariates that can be applied uniformly across the
studies combined.
 The major advantage of a IPD over an MA is the use of
individual-based rather than group-based data.

 Comprehensive search strategy: multiple sources of information
 Explicit methodology: to ensure reproducibility and transparency
 Emphasis on all clinically important outcomes: related to efficacy, safety,
and tolerability of the interventions under consideration
 Limiting errors: two reviewers at all major steps; limits bias and improves
precision

70
 Good deal of effort
 Qualitative distinctions between studies not captured
 “Apples and oranges” criticism
 A good meta-analysis of badly designed studies will still result in bad
statistics.
 Selection bias
 Analysis of between study differences is co-relational
 Tend to look at ‘broad questions’ that may not be immediately
applicable to individual patients
 Simpson’s paradox (two smaller studies may point in one direction,
and the combination study in the opposite direction)

 Huge Checklist
[http://faculty.ucmerced.edu/wshadish/]
 Free Software:
 EpiMeta: from Epi Info
 Revman: from Cochrane Collaboration
 “meta” package in R for statistical computing
 Non-free
 meta module in STATA

 PRISMA Statement (formerly QUOROM) : Preferred Reporting Items
for Systematic Reviews and Meta-Analyses
 MOOSE Statement : proposal for reporting meta analyses of
observational studies in epidemiology

 Mantel-Haenszel methods have been shown to be more reliable when there are not
many data (small trials and not many of them). This is why they have been selected as
the principle method of meta-analysis in the Cochrane Collaboration. This method
(which can be used for OR, RR and RD) is the most appropriate for many Cochrane
reviews, and many Cochrane review groups use it as standard.
 Peto method performs well with sparse data and is then the best choice, but when
events are common there is usually no preference to use it over the other methods. It is
not a good idea to use the Peto method when the treatment effect is very large, as the
result may be misleading. This method is also unsuitable if there are large imbalances
in the size of groups within trials.
 Random effects model may be better when there is statistical heterogeneity between
the studies in your review (we will discuss this further in Module 13 on Heterogeneity).

Metaanalysis copy

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Metaanalysis copy (20)

More from Amandeep Kaur (12)

Recently uploaded (20)

Metaanalysis copy