The masking effect of measures of Disproportionality Analysis

The masking effect associated
with the measures of
disproportionality analysis

Presented by: François MAIGNEN
Position or Unit/Sector/Section/Team

An agency of the European Union

Context: spontaneous reporting
Spontaneous reports of adverse drug reactions contain several
suspected medicines (n) and several reactions (p)
These reports are entered in a database and transmitted to
EudraVigilance
Quantitative methods of signal detection rely on the principle of
disproportionality i.e. methods compute the proportion of a
given reaction for a given drug and COMPARE this proportion
to the proportion observed with this reaction for all other
products observed in the database
Idea: If increased = signal of disproportionate reporting
(potential signal) (highlights a reported association).
2

Quantitative methods
- Stem from classical methods used in epidemiology, measures
of association: OR, RR and work on exactly the same
principles.
- The content of the database is collapsed in a contingency table
- Measure of disproportionality: PRR, ROR and RRR
- Confidence measures: 95CI PRR, etc …
- Two methods of computation:
- Report level: One report will only be counted once (allocated to the
product and reaction of interest)
- Drug-event level: ALL nxp records in the reports will be used in the
computation
3
- IMPORTANTLY: SDR DOES NOT MEAN PRESENCE OF A SIGNAL

What is the MASKING effect associated with
these methods?
Effect which is poorly understood by which the information
contained in the database for a given MASKING product will
attenuate the strength of a REAL effect associated with
another MASKED product for which a disproportionality
analysis is conducted.
Therefore, we are facing two problems:
- Quantify the extend (and magnitude) of the masking effect
in the entire database (problem 1).

4

- For a given disproportionality analysis for a reaction (E) and
a product (P) identify the product which will induce the highest
masking effect to remove it from the analysis and (hopefully)
detect new (true) signals (problem 2).

Difficulties
Problem 1: Relies heavily on an (arbitrary) definition of what
constitutes a masking effect (or what is a masking product).
Problem 2: Much more important in terms of signal detection.
- If we were having an OBJECTIVE way to identify the highest
masking product, quantify the magnitude of its masking effect
- Then: try to detect NEW SIGNALS (TRUE EFFECTS) by
removing it from the analysis.
However: Little evidence so far of the benefits of removing a
masking product (or situations in which its removal is
beneficial).
5

Our approach
- Develop a mathematical algorithm for the quantitative
methods used in EudraVigilance (both PRR and 95CIPRR)
- Test and validate this algorithm from a statistical standpoint
- Test and validate this algorithm on real spontaneous reporting
databases
- Assess the practical implications of the implementation of this
algorithm (computational requirements, method of
computation, handling and allocation of reports, prevalence in
the database, terms affected by a masking effect, potential
consequences associated with masking removal [true / false
positive])
- Establish a practical approach to address the masking effect.
6

Masking effect of quantitative methods of
signal detection
• Effect first described by Larry Gould in 2003
• Well known effect poorly understood
• No current algorithm aimed at detecting, quantifying the
presence, direction and magnitude of a masking effect
• Potential important implications in terms of Public Health (real
signals might be missed)
• Decided to develop such algorithm in PROTECT WP 3.7
• F Maignen, JM Dogne (EMA/PRAC), M Hauben, E Hung (Pfizer),
L Van Holle (GSK vaccines)
7

Study aimed at developing, validating an
algorithm for measures of disproportionality
and their confidence interval
The study was conducted in five steps:
1. Development of an algorithm for the measures of
disproportionality (PRR, ROR and RRR)
2. Validation of algorithm in EudraVigilance (not submitted / not
presented)
3. Comparative analysis in two SRS databases (EV / Pfizer)
4. Development and validation of an algorithm for the confidence
intervals of measures of DA
5. Difference with subgroup analyses (not submitted / not
8
presented)

Masking effect
The masking is a collateral effect of quantitative methods of
signal detection which rely on disproportionality analysis by which
SDRs (corresponding to true signals) might be suppressed
(hidden) because of the presence of another product in the same
database.
•Danger: missing some signals or detecting some signals with
delay
Gould has first described the masking effect of disproportionality
analysis using the Relative Reporting Ratio (RRR).
Masking is incompletely understood. To date, there is no
algorithm to tackle its effect in an automated way.
9

MASKING EFFECT OF
MEASURES OF
DISPROPORTIONALITY
10

Presentation title (to edit, click View > Header and Footer)

Masking effect
New updated contingency table: masking product separated from
the background of the database

11

Masking ratio
The exact masking ratio (MR) is defined as the ratio of the
measures of disproportionality (DA) for A, without and with
product B in the database:

12

Computation of the MR at the DEC level
In a report containing n drugs and p events, each nxp drug-event
RECORD is treated independently.
Each RECORD is allocated to the corresponding cell of the table
(depending whether the record contains the product A, the
masking product B or the event of interest).
This method creates disjoint and independent sets. Masking is
constant for a given masking product B and event E.
The restriction applied to the database is easy to implement (with
the exclusion of the records involving the masking product from
the computation).
13

Mathematical expressions of the MR: example
of the RRR
By definition RRRA(withoutB) = RRRA *MRRRR
Therefore the masking ratio is the value by which the initial RRR A
will be multiplied after the removal of the masking product B.
The exact MR for the RRR (drug-event level) is equal to:

In general, in most of the large SRS databases the total number
of reports involving the masking product is much lower compared
to the total number of reports in the database (n 2. << n..)
14

Simplification of the MR for the RRR: identify
the HIGHEST masking product ONLY
In general n2. << n.. therefore MR for the RRR can be
approximated by:

•n21: number of reports involving the masking product B
•n.1: total number of reports involving event E.
Therefore, the masking:
•is mostly influenced by the proportion of reports involving the
masking product (B) for the reaction of interest to the total
number of report including this reaction in the entire database.
•Is reaction specific for a given (masking product) product B.

15

Masking induced by RRR computed at DEC
level

16

Computation of the MR at the report level (EV)
Each report is allocated to one cell of the contingency table and
only counted once. Pb: allocating the reports to the correct cell of
the contingency table.
It must take into account the handling of reports containing both
the product (A) and the masking product (B).
PRIORITY RULES: A > B > other products
•Reports containing product A -> Allocated to A
•Reports containing product B BUT NOT product A -> Allocated to
B (masking product)
•Reports neither containing A nor B -> Background of the
database (all other reports)
17

Computation of the MR at the REPORT LEVEL:
ALLOCATION RULES
%n2i number of reports containing B but not A
%n3i number of reports containing neither A and B
The computation of the masking ratio at the drug event level can
pose some practical issues concerning the computation of both
and %n3i.
CORRECT ALLOCATION? With this scenario the reports containing
both A and B are (and should be) allocated to A but in theory,
these reports could also be allocated to product B.

18

Issues associated with the computation of the
MR at the report level
• The computation of the masking ratio depends on the product
of interest A, the masking product B and the event under
consideration E
• COMPUTATIONAL DIFFICULTIES: COMPUTATIONALLY
DEMANDING approx. quadratic function of the number of
drug-event combinations in the database
• Limiting step: building and calculating the values in the
contingency table
• Need for simplification
19

Mathematical expression of the MR for the PRR
Likewise the MRPRR is equal to:

1) The MRPRR is drug A-masking drug B-event E specific (i.e.
%n21/%n31 and n2./n3.)
2) Therefore computationally demanding (build the contingency
tables)
3) Simplify the algorithm, make it masking drug B – event
specific and relax the allocation of reports containing both A
20 and B (double allocation).

Simplification of the algorithm for the other
measures of disproportionality (1)
The idea is to make the masking ratio REACTION SPECIFIC
(identical for a given masking product B for a given reaction E)
•Firstly, the total number of reports involving the masking
product B (n2.) will represent a very small subset of reports in the
entire database (compared to n3. or n..). n2. << n3.
•Secondly, the total number of reports involving the product A for
the reaction of interest (n11) would be low compared to the total
number of reports containing the reaction of interest (n 31 or n.1).
n11<< n31 or n11 ~ n11 + n31

21

Simplification of the algorithm for the other
measures of disproportionality (2)
Double allocation of the reports containing both products A and
B:
•Finally, the proportion of reports containing both the product (A)
of interest as well as the masking product (B) would also remain
low
•These reports could be allocated to both products to simplify the
computations.

22

Mathematical expressions of the MR

23


Influence of the (double) allocation and % of
reports containing both A and B: simulation
EudraVigilance does not provide a standardised setting to study
the influence of the allocation of reports containing both A and B
either to A or the B:
•Variable number of reports involving A and B and both. No
“extreme” circumstances
•Variable % of reports containing both A and B and uneven
distribution across the database
•Usually low % of reports containing both products
•Difficult to assess the effect of the size of the database
We have performed a simulation study aimed at circumventing
24
these methodological issues.

Simulation study
• Range of values for product A, for product B and for the overall
size of the database
• Range of values for the % of reports containing both A and B
• The number of reports in common was applied to the smallest
value of A or B and rounded to the nearest integer
• The number was deducted to B (reports allocated to A)
• The number was deducted to A (reports allocated to B)
• More than 2,000,000 contingency tables and 42 million
computations.
• PRRA, PRRB, L95CIPRRA, MR, unmasked PRRA, calculated PRRA (3
methods of allocation A, B, A + B), difference between
25
unmasked and calculated PRRA.

Two possible approaches for simulation (No
reports for A and B held constant or not)
No reports for A and B not held constant (approach chosen)
Product
A

Product
B

Product Product
A
B

Products
A
B

=

No reports for A and B held constant
Product
A

26

Product
B

Product Product
A
B

Products
A
B

Influence of the allocation of reports
containing both A and B: simulation

27


Allocation of reports to product A
• The calculated PRRA was identical to the unmasked PRRA
obtained after the removal of the masking.
• an increase in number of SDRs (No reports > 3 and a lower
bound of 95CI > 1) after removal of the masking effect. The
number of SDRs observed after removal of the masking effect
induced by B was 1,090,656 SDRs
• net gain of 52,589 SDRs or 4.9% of SDRs
• Masking increases # MR
• PRRA increases with % of reports containing both A and B.
28

Effect of the allocation of reports containing
both A and B

29

Evolution of PRRA: Allocation of reports
containing both A and B to product A

30

Masking: Allocation of reports containing both
A and B to product A

31

A and B to the MASKING product B
• The masking ratio loses its ability to predict the presence,
direction and magnitude of the masking effect: the unmasked
PRRA is only equal to the predicted one when the two products
have not reports in common
• Important loss of SDRs (after removal of the masking). The
number of SDRs observed after removal of the masking effect
of product B was 1,017,795 (corresponding to a net loss of
45,272 SDRs or 4.4% of the SDRs). The difference in SDRs
between the two methods of allocation (allocation to product A
vs allocation to B) is a net loss of 72,861 SDRs.
32

• Loss of the ability to predict the presence, direction and
magnitude of the masking effect

33

• Loss of SDRs (SDRs associated with low case counts 3 – 5
reports mostly affected).

34

Double allocation of the reports
• Reveals the same number of SDRs
• When less than 50% of reports contain both products, the
approximate MR provides a satisfactory estimate of the exact
MR.
• The size of the database mitigates the under and overestimation of the exact MR. The under or overestimation is
exclusively observed for small databases (i.e. number of
reports lower than 100,000)
• when the database reaches a size of 100,000 to 1,000,000
reports, the approximate MR consistently overestimates the
exact one (maximum twice its real value).
35

• When a high proportion and volume of reports contain both
products A and B compared to the number of reports in the
database (i.e. n31 or n32), the double allocation can lead to a
dramatic overestimation of the exact MR or to rare
computational issues (i.e. in 78,245 or 0.04% of the total
number of computations).
• Drug-event pairs involving a low number of reports were less
affected by these computational difficulties.
• In all cases, the approximate ratios consistently identified the
highest masking product (identified by the exact ratio).
36


37

Double allocation of the reports: correlation
between approx and exact MR

38

Masking function (2 variables)
• fct of n21/n.1 (x)
• And n2./n.. (y)
• Buffering zone
• n21/n.1 = 0.3 – 0.7
• MR = 2 when n21/n.1 = 0.5
• f(x,y) -> ∞ as n21/n.1 -> 1

39

Estimation of the masking effect
• Effect can be estimated using

40

Key messages part 1
• A masking ratio can be used to identify and quantify the
masking effect associated with the measures of
disproportionality.
• The method of computation (at the report or at the drug-event
level) has a dramatic effect on the masking mechanisms and
on the number of computations.
• Simple approximations to the above masking ratio are
demonstrated to be valid for large and diverse databases
provided that underlying assumptions on the size of the
database are verified: identification of highest masking
product
• For any event, the strongest masking effect is associated with
41
the drug with the highest number of records (or reports

COMPARATIVE ANALYSIS
IN TWO LARGE SRS
DATABASES
42


Analyses in EudraVigilance and Pfizer database
• Comparative analysis in two SRS databases: EudraVigilance
and Pfizer (based on hypothesis by L Gould that masking could
affect more Companies’ databases) Conducted in April 2011.
• Terms selected on the basis of:
• Seriousness: Set of MedDRA terms important to PhV (EU-ADR)
and DMEs (commonly reported to EV).
• Frequency of reporting to EV: Events rarely reported to
EudraVigilance (less than 100 reports) have also been included
in the study.
43

MedDRA terms included in study
• Type of products: Both NCEs (dopaminergic
agonists, antiretrovirals) and biologicals
(including mAb, vaccines, clotting factors,
etc …).
• Masking effect. Used the approximate MR
to quantify the masking induced by the
HIGHEST masking product.

44

Masking observed in EV
• 30,645 drug event combinations (DECs), 29,245 DECs EU-ADR
events, 1,400 DECs involved our additional set of events which
have been rarely reported in EV.
• Masking: Approximate MR > 1 for 18,599 masking drug-event
combinations (MECs) i.e. 61% of the DECs.
• MR > 1.1 for only 87 MECs (0.5% of MECs for which the MR is
above 1),
• MR > 1.5 for only 28 MECs (0.15%)
• MR > 2 for only 20 MECs (0.1%).
• All the drug-event combinations actually affected by an
important masking effect involved events rarely reported in EV
45

Highest masking
effect
• Induced by products for
which the reaction is
known
• “Carry-over” effect
(masking present induced
by products removed
from the market)

46

Removal of the masking effect
The removal of the masking effect has revealed 974 new signals
of disproportionate reporting (SDRs, defined by a new PRR above
or equal to 2).
Number of (SDRs) before the removal of the highest masking
product was 12,861 (i.e. 42% of the DECs included in our study)
Number after removal increased to 13,835 (i.e. increase by
approx. 3%).

47

DECs mostly
affected
• Mostly events rarely
reported
• Contains some signals
of Public health
importance (PML
natalizumab – known
in 2011)
• Mostly known signals,
handful of unknown
signals
48

DECs affected by masking

49


Nbre SDRs revealed
• Unclear relation between No
(or proportion) of SDRs
revealed and magnitude of
masking
• Nbre of SDRs revealed #
frequency of reporting of the
event to EV

50

Comparison with
Pfizer db
• Reveals structural
differences between the
two databases:
• Products for which the Cy
holds a license
• Influence of consumer /
non-serious reports?

52

Comparison with Pfizer db
Consequential masking more prevalent in Pfizer db than in EV
(confirms suspicions raised by L Gould)

53

Effect of removing the masking effect on the
ranking of SDRs
Provided that the % of reports that the two products have in
common remains low, the ranking of SDRs is marginally affected
by the removal of the masking effect induced by the HIGHEST
MASKING PRODUCT.

54

Key messages 2
• Our estimate of prevalence of significant masking showed that
the phenomenon may be rare.
• An important masking effect was consistently associated to
products known to induce the reaction.
• Masking mainly (but not only) affected events rarely reported
in our large spontaneous systems databases.
• Differences affecting important medical events were observed
between EudraVigilance and Pfizer database.
• The original ranking provided by the quantitative methods
included in our study was marginally affected by the removal
of the masking product.
55

MASKING EFFECT
ASSOCIATED WITH THE
CONFIDENCE INTERVALS
OF MEASURES OF
DISPROPORTIONALITY
56


Masking effect associated with L95CI
Similarly to the MR for measures of disproportionality, we have
defined a MR for the L95CI:

Which gives:

57

Masking effect associated with L95CI

58

Masking effect associated
with L95CI
Therefore:
• There is a direct mathematical /
statistical relationship between the
masking effect associated with the
measures of disproportionality and
their corresponding confidence
intervals.
• However, the “multiplication factor”
adds an element of complexity in this
relationship: no simplified algorithm.
59

Shape of the masking function
. The shape of the masking function differs according to the No of
reports involving the reaction of interest (n.1) and proportion of
reports involving A (n11/n.1) (LHS 10, 50%, RHS 50, 10%)

60

Masking: extent and comparison between the
PRR and Lower95CI
• Our simulation originally yielded 905,091 SDRs with the PRR
and 1,038,067 with its Lower95CI.
• The removal of the masking resulted in a gain of 77,036 SDRs
with the PRR (an additional 8.5% SDRs) and 68,900 SDRs with
the Lower95CI (an additional 6.6% SDRs).
• The removal of any effect (masking or revealing effect)
resulted in a net gain of approximately 5% new SDRs for both
methods.
• Any masking (approx. 60% DECs) and 30% affected by
masking > 10% (MR[CI] > 1.1)
61

Masking: extent and comparison between the
PRR and Lower95CI

62

PRR

Lower95CI

High overlap between the SDRs unravelled by
the masking between the two methods

63

PRR

Lower95CI

Relation between MR PRR and MRCI fct of n11.
(purple n11 = 1 to blue n11 = 10,000).
The masking ratio of the confidence interval is influenced by the
number of reports of the product (A) on which the
disproportionality analysis is conducted (different colour lines).
For an identical masking effect observed with the PRR, the
masking ratio associated with the corresponding lower bound of
the 95% confidence interval will decrease as the number of
reports containing product A increases.
Our simulation confirms that the masking product inducing the
highest masking effect on a given drug-event pair for the PRR
will also be the product inducing the highest masking effect for
the confidence interval.
64

Relation between MR PRR and MRCI

65

Relation between MR PRR and MRCI

66

Proportion of SDRs revealed

67

Key messages 3
• The quantification of the extent, direction and magnitude of
the masking effect associated with the confidence intervals
(CI) of measures of disproportionality (MD) can be automated.
• There is a direct relation between the masking associated with
the MD and their respective CI. Products inducing the highest
masking effect for the MD will induce the highest masking
effect for the CI.
• Removal of important masking is likely to remove a high
proportion of common drug-event pairs for the MD and their
CI.
68

What shall we retain from our studies?
• First algorithms tested and validated aimed at detecting,
quantifying the presence, direction and magnitude of a
masking effect (MD and CI). Computationally demanding.
• Confirm a lot of results obtained in the past empirically:
• Masking induced by products known to induce the reaction
• Higher prevalence of masking on SRS databases from Companies

• Prevalence of masking seems to be low in LARGE SRS
databases.
• Seems to affect (but not only) events rarely reported in the
database.
• However, not tested on smaller databases with different
69
pattern of products.

What shall we retain from our studies?
• Simplification of algorithm for MD might not be possible for CI
+++
• We did not VOLUNTARILY characterise the SDRs unravelled by
the removal of the highest masking effect.
• The real Public Health impact of removing a masking effect
needs to be further quantified using PROSPECTIVE studies
(methodological challenges incl. need to perform a blinded
adjudication of SDRs).
• The removal of the masking must be dictated by the rate of
true positive / false positive unravelled by the masking.
70

Perspectives / future work
• Three articles submitted to PDS so far
• Assessment of the Public Health impact of masking
• Influence of stratum specific in case of stratified analyses
• Possible comparison with other methods of adjustment
(logistic regression / ROR)
• Collaboration with University of Bordeaux (A Pariente) and
GSK (L Van Holle)

71

Conclusion
• Masking was so far poorly understood: major work and major
results
• First algorithm with a demonstrated link between the MD and
their CI
• Potential Public Health benefits (still to be demonstrated via
well designed prospective studies)
• Benefit in terms of new signals?
• Benefit in terms of time gained to detect a signal?
• Or both?

72

email@
francois.maignen@ema.europa.eu

#?

73

The masking effect of measures of Disproportionality Analysis

More Related Content

Similar to The masking effect of measures of Disproportionality Analysis (20)

More from Francois MAIGNEN (16)

Recently uploaded (20)

The masking effect of measures of Disproportionality Analysis