Introduction to Pharmacovigilance Signal Detection

Introduction to
Signal Detection

by Rodney L. Lemery MPH, PhD

Copyright  BioPharm Systems, Inc. 2011. All rights reserved.

Topics…
• Definitions
– Pharmacoepidemiology
– Pharmacovigilance
– Signal
– Signal Detection
• Signal Detection
– Qualitative
• Striking cases
• Periodic reviews
– Quantitative
• Analysis of disproportionality

Introduction to Signal Detection 2

Topics…
• Signal Detection (cont.)
– Disproportionality analysis tools
• Signal Prioritization
– WHO Triage
– MHRA Impact Analysis
– Categorization of priority
• Confirmed Signal
• Unconfirmed Signal
• Refuted Signal
• Signal Evaluation
– Causality
– Frequency
– Clinical implications
– Preventability


Acronyms and General
Definitions
• Prevalence – The total number of cases of a disease in
a given population at a specific time
• Incidence – The number of newly diagnosed cases
during a specific time period
• EMA – European Medicines Agency
• ADR – Adverse Drug Reaction
• APR – Adverse Product Reaction
• ICH – International Conference on Harmonization
• CIOMS – Council for International Organizations of
Medical Sciences


Definitions…

• Two pervasive definitions (Abenhaim, Moore,
& Begaud, 1999)
1. The collection and scientific evaluation of
adverse drug reactions (ADR), under normal
conditions of use for regulatory purpose.
−Restricts the concept to regulatory
Pharmacovigilance compliance only and only medicinal
products.
2. Watchfulness in guarding against danger from
products or providing for safety of the product
−Expansive beyond just regulations and
frames the construct for use in academia
and the sciences


Definitions…

•The application of epidemiologic
techniques used to study the effects of
Pharmacoepidemiology drugs in populations
−First mentioned in the early 1980’s
(Abenhaim, Moore, & Begaud, 1999)


Exercise 1

1. Please segregate into your
Define the small group as assigned by the
index card in your chair
following terms: 2. Using collaborative discussion,
please reach consensus for the
definition of the following terms:
Signal  “Signal”
 “Signal Detection”
3. Nominate a single
spokesperson from the group to
Signal Detection share the definition with the
larger audience.


Definitions…

•Much debate on the definition (we will use the
following):
•Information that arises from one or multiple sources,
which suggests a new potentially causal association, or
Signal a new aspect of a known association, between an
intervention and an event or set of related events, either
adverse or beneficial, that is judged to be of sufficient
likelihood to justify verificatory actions. (CIOMS, 2010
p.14)


Definitions

•Much debate on the definition (we will use the
Signal following):
•The act of looking for and/or identifying signals
Detection using event data from any source . (CIOMS, 2010
p.116)


Simplified Safety Signal Lifecycle

Signal Signal
Detection Prioritization

Signal
Evaluation
CIOMS (2010, p. 9)


Detailed Signal Management Process

Signal
Detection

Signal
Prioritization

Signal
Evaluation

11
Introduction to Signal Detection

Signal Detection…


Individual Case Safety Reports (ICSR)
• The accumulation of ICSR can occur from multiple
places according to the ICH E2D Guideline (ICH, 2003)
Sources of ICSR Description of Sources
Unsolicited sources Spontaneous reporting
Solicited sources Any organized collection of data
(outcomes research, clinical trials,
registries, surveys, billing databases
etc.)
Contractual agreements Inter-company exchange of safety data
Regulatory authority Any ICSR originating from the
regulatory authority submitted to a
company


Signal Detection…
Traditional Pharmacovigilance
Methods
· Individual case review
· Aggregate analysis
· Periodic reports

• Traditionally, signals are detected through the
assessment of individual case safety reports
(ICSR) in an individual or cumulative manner


Signal Detection…
• Qualitative review of ICSR
– Critical method of detection for events where the
background incidence is uncommon or rare and should
not be replaced by quantitative methods (Egberts,
Meyboom, & van Puijenbroek, 2002; Hopstadius,
Norény, Bate, & Edwards, 2008)
– May be implemented as a list of designated medical
events (DME) within a company
• Often manifest in systems as custom Standardized MedDRA
Queries (SMQ)
– FDA has a list of “Interesting PTs”
– EMA follows serious events identified by CIOMS V
• Results in the identification of “index” or “striking”
cases that can be monitored


Signal Detection…
• Qualitative review of ICSR (cont.)
– Periodic review of case series
• PSUR or equivalent
– Review of the aggregate tables displayed in
these periodic reports can provide indications of
potential “striking” events
• Specific sections such as the “Overall Safety
Evaluation” provide clinical context around the
identification of such signals and become an
important part of signal detection
(Waller, 2009 p. 67)


Exercise 2
small group as assigned by the
2. Using collaborative discussion
and the DSUR provided, please
DSUR AE determine the presence or
Summary Table absence of striking cases in the
table.
Review  Why did you or didn’t you
identify cases of interest?
share the results with the larger
audience.


Signal Detection…
• Quantitative assessment of ICSR
– Absolute counts
• Number of reports of an adverse event (AE) or
adverse product reaction (APR)

985
No APR All completed the 1 year
1000 duration of study resulting in
Participants 985 person-years
in an IND
study Steven’s Johnson
Syndrome (SJS)
15
Completed the 6 month mark
of study resulting in 7.5
person-years

Signal Detection…
• With a total of 992.5 person-years for the denominator of the
study incidence rate, we can calculate the incidence rate of SJS
in the IND population as 15/992.5 which is 0.0151 cases/992.5
person years
– A typical denominator is million person years so we must convert or
IND incidence rate to million person years
• In terms of million person years this becomes 15.1 cases/million person
years
• Using epidemiologic sources, we note that the estimated
background incidence rate of SJS is 1-6 cases/million person
years
• The incidence rate in the study is suspiciously higher than the
expected background rate therefore SJS is a signal


Signal Detection…
• Quantitative assessment of ICSR (cont.)
– Proportions
• Number of specific AE reports divided by total number of
reports for a given product
– Assume an IND study of 100 patients for a new
Anticonvulsant
– At the end of the study there are 450 ICSR for the product
– Within this 450 are 25 reports of “GI bleeding”
– The proportion would be 25/450 or ~6%
• NOTE: True “expected” proportions are not really possible to
determine from spontaneous reporting data
– However, highly prevalent product/event
combinations for given case series may
be enough to elevate the combination
to a signal

Exercise 3
Determine the
following with the 1. Please segregate into your
data distributed: index card in your chair
and the information provided,
Is there an please calculate the study
prevalence for “Peripheral
indication that neuropathy” and compare it to
“Peripheral the provided epidemiologic data
neuropathy” may  Has a signal been identified
in the data?
have a relationship 3. Nominate a single
to the spokesperson from the group to
consumption of share the results with the larger
audience.
“Alfenta”?


Disproportionality…
Data Mining Algorithms
· Disproportional reporting
ratios

• Once the ICSR database becomes large enough*, statistical techniques
(generally referred to as data mining) can be applied
– Usually on large datasets from regulatory agencies or public health
entities
– WHO: Vigibase
– FDA: AERS
– MCA: Sentinel (formally known as ADROIT)
– EMA: EudraVigilance
– Generally these techniques identify
disproportionate reporting ratios
*”Large Enough” is a function of product/event incidence in the population

“Large Enough”
• Implications on population size for sampling in signal
detection can be reduced to the following (CIOMS, 2010
p.31) :
Event Background Example Ease of Signal Detection
Incidence Incidence proving an Method
in Product of Event association
Takers (method)
Common Rare Phocomelia Easy ICSR or Periodic
due to Thalidomide (clinical observation) Review
Rare Rare Reye’s syndrome Less easy ICSR or Periodic
and Aspirin (clinical observation) Review

Common Common Cough Difficult Data Mining
and ACE inhibitors (large observational
trials/data)
Uncommon Common Breast carcinoma Very difficult Data Mining
to Rare and Hormone (large clinical trials)
Replacement
Therapies
Rare Common None known Virtually impossible Virtually impossible


Classical Versus Bayesian…
• The basis of disproportionality analysis is
either frequentist statistics or Bayesian
(Gajewski & Simon 2008)
– Classical (frequentist) statistics look at probabilities as long
term frequency with an assumption of repeatable
experiment or sampling methods and a “true” value for a
parameter
• TOTAL INFORMATION=Data from experimentation
– Bayesian statistics look at “true” probabilities as a function
incorporating prior beliefs or knowledge and is updated
• TOTAL INFORMATION=Historical Information + Data from
experimentation


Classical Versus Bayesian…

(Maggid, 2011)


Data Mining Algorithms…
• All measures calculated from a 2X2 Contingency Table
– Classical
• Proportional Rate Ratio (PRR)
• Reporting Odds Ratio (ROR)
• Relative Reporting Ratio (RRR)
– Bayesian
• Information Component
• MGPS/EBGM

Event of All other Events TOTAL
Interest
Product of Interest A B A+B
All other Products C D C+D

TOTAL A+C B+D A+B+C+D


• All measures of disproportionate reporting are
basically calculations of
OBSERVED/EXPECTED
• In PV, the EXPECTED data is also referred to
as the “background”
• What you include in the “background” is a point
of contention in the industry and no real rules
are present (Gogolak, 2003)


• Since the simple calculation is O/E, the
relationship between background and the
statistic of interest is inversely related:
– As the background increases the resulting
statistic decreases
• Large E results in small PRR
– As the background decreases the
resulting statistic increases
• Small E results in large PRR


Data Mining Algorithms
• In general, the frequentist or Bayesian methods will
perform similarly when there are five or more reports
of a particular product-event pair (CIOMS, 2010, pp.
59)
• Bayesian methods may provide a solution to “false
positive” indications in large datasets
• It is important to note that the literature does not
demonstrate consensus on cost/utility of various data
mining tools with respect to specificity and sensitivity
– (CIOMS, 2010, pp. 61)
• As such, company specific decisions
must be discussed as part of the
signal management process

Technical Solutions…
• Use of these databases requires that certain
assumptions be made
– Drugs used in the marketplace are used by a
representative sample of the greater population
• Any information derived from these databases should
be interpreted using the limitations of the data
contained therein (Edwards, 1999)
– Limited clinical quality of data
• USA allows reporting into the AERS system from anyone
(Health care provider {HCP} or not)
• EMEA only allows reporting by HCP thus typically more
complete clinical information


• Three main products are available for large data
mining opportunities
– Qscan by Drug Logic
– Empirica Signal by Oracle
– agSignals by Aris Global
– dsExplorer by Cerner
• In the spirit of full-disclosure, BioPharm Systems is a
Gold Partner with Oracle and the next slides are taken
from the Empirica Signal product
– Please note that except for very specific functional differences,
these software systems are designed to accomplish similar
tasks. The degree to which one is better than the other is not
discussed in this presentation and the use of slides is not an
endorsement of one product over another
– As already stated, the decision to evaluate
signals and the method used is a company
specific decision


• Empirica Signal can display disproportionality
results in a sector map fashion which allows for
visual assessment of signal strength.


• In this example of
“Ziconotide”, we see
that sector 2 is
“Vertigo” with a
disproportionality
ratio of 2.052
– This is a good example though
of a well know substantiated
concern with the product as it
is known to disturb motor
function and balance



• Sector 1 is “Tinnitus”
with a disproportionality
ratio of 4.194


• The software solutions available provide a
wonderful way to quickly and comprehensively
analyze marketed data reported to regulatory
agencies.
• These data are subject to the issues surrounding
the collection of the information in them
– Underreporting of serious events
• Changes the number of expected events
• “Weber Effect”: The peak reporting for events in a drug on
market occurs within the first 2 years of approval (Hartnell, &
Wilson, 2004) during the initial 5 year marketing period
– Over reporting of events of
non-interest (expected non-serious)


• False Causality attribution
– Signals ARE NOT CAUSAL INDICATIONS
– They are disproportionate reporting
indicators
• Mitigation of these limitations can occur if you
establish a signal prioritization method in your
company for dealing with the various signals
identified by manual or automated signal
detection


Signal Prioritization…
Triage of Outputs
· Interpret the signal in context of other
relevant sources, disease knowledge,
biologic plausibility, alternative
etiologies, etc.

Monitor
If signal is
indeterminate

NO
Impact Is signal
assessment and Need further NO
YES investigation refuted?
prioritization
Close out

YE
S
(if signal is refuted)


Signal Prioritization
• Prioritization of signals is still a very
controversial aspect of signal
management (Waller, 2010 p. 50)
• Two proposed methods are as follows:
– WHO – Triage (CIOMS, 2010 p. 88;
Lindquist, 2007)
– MHRA – Impact analysis (draft literature)


WHO - Triage
• Seriousness assessment
– Is the event serious or not?
• Unexpectedness
– Is the event expected or not?
• Disproportionality score
– Is the score high or not?
• Temporal displacement of score
– Has the disproportionality score increased over
time?


WHO - Triage
• Temporal occurrence
– Is the event occurring within the first few years of
launch
• Careful as Weber effect could contribute to
confounding here
• Multiple signaling countries
– Are more than one country seeing this
issue?
• Positive Rechallange
– Is there evidence of positive rechallange?
• Specialty list of terms of interest
– Is this a term of interest as identified by the
company

MHRA – Impact analysis…
• A calculated quantitative score based
on “Evidence” and “Public Health”
– Evidence Score
• Degree of disproportionality (PRR, IC etc)
• Strength of evidence
• Biologic Plausibility
– Public Health Score
• # of reported cases per year
• Expected health consequences
• Reporting rate in relationship to the
level of drug exposure


MHRA – Impact analysis
• Results in the following quantitative
categories:
– High
– Need to gather more information
– Low
– No action
• Additionally the MHRA is looking at a method to
incorporate the following in a prioritization
scheme:
– High profile product (media
attention)
– Risk perception by general
population
– Political obligations

• Regardless of the method used, every
signal should undergo this type
prioritization
– CIOMS (2010 p. 22) suggest that this effort
results in the following outcomes which I have
labeled in the following manner:
• Confirmed signal
– Results in the movement of the signal to the evaluation
process
• Unconfirmed signal
– Results in the monitoring of this event over time and regularly
re-prioritizing it based on new information
• Refuted signal
– Results in the closing of the signal
» Disease progressions
» Known issues etc.


• Regardless of the method used, care
should be taken in that this type of
assessment and analysis
should be an iterative process
when new information is
amassed


Exercise 4

and the information provided,
Signal please prioritize the various
Prioritization signals using one of the above
methods (WHO or MHRA)
share the findings with the larger
audience.


Signal Evaluation…
Signal Evaluation
· Individual case review
· Aggregate analysis
· Periodic reports
· Non-interventional studies
· Non-clinical studies
· Class analysis
· Other relevant information


Key Components to Signal
Evaluation…
• Once a signal is prioritized as
“Confirmed” or its equivalent, further
data must be gathered to further the
evaluation of the signal (Waller, 2010 p.
50, 51)
• Causality
– Does the provided evidence support a causal
relationship between the product and event?
• ICSR Causal Relationships
– Probable, possible, unlikely, unrelated, unassessable

Causality is more than this


Causality…
• Bradford-Hill (Shakir & Layton, 2002)
proposed the following categories should be
used to assess causal relationships found in
data
– Strength
• The stronger an association, the less likely it is
explained by other factors
– Consistency
• Multiple sources demonstrate similar
associations
– Temporality
• Is the event consistently treatment
emergent?
– Positive re or de challenge evidence


Causality…
• Bradford-Hill (cont.)
– Biologic Gradient
• Evidence of dose or duration related risk
– Dose-dependency or cumulative exposure over time
– Specificity
• Single product exposure results in event
– Presence of this leaves little doubt to causality
– A few adverse drug reactions in an of themselves are
syndromes
• Absence of this does not reflect a non-causal
relationship
– For example nicotine exposure and
lung cancers


Causality…
• Bradford-Hill (cont.)
– Plausibility
• Does the existing literature support this association?
– Coherence
• Is the association compatible with the existing literature and
knowledge?
– Experimental Evidence
• Is there data that demonstrates the event can be altered or
eliminated by some experimental regimen?
– Converging evidence between post marketed and clinical surveillance
– Analogy
• Are there existing alternative
explanations for the association
in the literature?
• Absence of these strengthens the causal likelihood
• Absence of any or all of these four criteria does not
indicate the absence of a causal relationship since
out data may be describing a newly
seen observation not part of the
literature used in these assessments

Causality…
• In general, the more of the criteria satisfied,
the stronger the causal relationship. The
assessment of how many and which criteria
are more important than others, is no simple
formula and judgment is required (Waller,
2010 p. 29)
• There are issues with the PV databases and
literature used in signal evaluation and one
must keep these in mind when applying the
Bradford-Hill criteria (Shakir &
Layton, 2002)


Evaluation…
• In addition to assessments of causality, the
concept of Frequency is important in signal
evaluation (Waller, 2010 p. 51)
– The question of frequency can be categorized
into the following measures of prevalence
• Very Common
– More than 1:10
• Common
– 1:10 to 1:100
• Uncommon
– 1:100 to 1:1000
• Rare
– 1:1000 to 1:10,000
• Very Rare
– Less than 1:10,000


Evaluation…
• In addition to assessments of causality and
frequency, the concept of Clinical
implications is important in signal evaluation
(Waller, 2010 p. 51)
– Clinical Implications can be rephrased to impact
on patient health
• Is the event life-threatening?
• Does its presence lead to a congenital anomaly?
• Does its presence lead to death?
• Does its presence lead to long term
disability or long term hospitalization?


Evaluation…
• In addition to assessments of causality,
frequency and clinical implications, the
concept of Preventability is important in
signal evaluation (Waller, 2010 p. 51)
– Preventability
• If an intervention could be applied at this stage, would
the event of interest or its outcomes be prevented?


Evaluation…
• To summarize, every confirmed signal is
evaluated in terms of the following:
– Causality
– Frequency
– Clinical implications
– Preventability
• Based on the suspected signal’s evaluation
outcome, you may choose to elevate it to the
status of a potential or identified
risk (CIOMS, 2010 p. 93, 94)


Evaluation
• This signal evaluation process is intimately
tied to risk identification and may result into
the feeding of the signal into the risk
mitigation/management planning processes
at your company.
• The information collected in the signal
evaluation phase will feed the safety
specification section of the risk management
plan.


Exercise 5

2. Using the provided clinical
information, is there evidence
Signal Evaluation that “Ascites” should be
elevated to a risk?
share the findings with the larger
audience.


Summary
1. Create a procedure to formalize signal
management in your organization
• Ensure your process includes signal detection,
prioritization and evaluation
2. Signal detection efforts should include both
qualitative and quantitative methods
Top Five 3. Signal prioritization should describe the objective
methods used to categorize the signals
Take- discovered through detection efforts
Aways 4. Signal evaluation should be used to document
and formally evaluate only those critical signals
(from prioritization methods)
5. Information collected from your formal evaluation
strategies can be used to seamlessly parse into
your risk management process


References
• Council for International Organizations of Medical Sciences (CIOMS). (2010). Practical Aspects of Signal
Detection in Pharmacovigilance. Report of CIOMS Working Group VIII, Geneva .
• Egberts, A.C.G., Meyboom, R.H.B., and van Puijenbroek, E.P. (2002). Use of Measures of Disproportionality in
Pharmacovigilance: Three Dutch Examples Drug Safety 25(6): 453-458
• Gajewski, B.J. and Simon, S.D. (2008). A One-Hour Training Seminar on Bayesian Statistics for Nursing
Graduate Students. The American Statistician, 62 (3)
• Hartnell, N.R. and Wilson, J.P.. (2004). Replication of the Weber Effect Using Postmarketing Adverse Event
Reports Voluntarily Submitted to the United States Food and Drug Administration. Pharmacotherapy. 24:743-
749
• Hopstadius, J., Norény, G.N., Bate, A., and Edwards, R.. (2008). Impact of Stratification on Adverse Drug
Reaction Surveillance. Drug Safety. 31(11)
• International Conference on Harmonisation (ICH). (2003). ICH Harmonised Tripartite Guideline. Post-approval
safety data management: Definitions and standards for expedited reporting E2D. Retrieved on July 26. 2011
from
http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E2D/Step4/E2D_Guideline.pdf
• Levitan, B., Yee, C. L., Russo, L., Bayney, R., Thomas, A. P., and Klincewicz, S. L. (2008). A Model for
Decision Support in Signal Triage. Drug Safety, 31 (9)
• Lindquist, M. (2007). Use of triage strategies in the WHO signal-detection process. Drug Safety, 30:635-7
• Maggid. (2011). Retrieved from http://actuary-info.blogspot.com/2011/05/homo-actuarius-bayesianes.html on
August 31, 2011
• Shakir, A.W., and Layton, D.. (2002). Causal Association in Pharmacovigilance and Pharmacoepidemiology:
Thoughts on the Application of the Austin Bradford-Hill Criteria. Drug Safety. 25 (6): 467-471
• Strom, B. L., and Kimmel, S. E. (2006). Textbook of Pharmacoepidemiology. Wiley, West Sussex, UK
• Waller, P. (2010). An Introduction to Pharmacovigilance. Wiley-Blackwell. Oxford, UK


Contact Information
Rodney has over 15 years experience in clinical research including
laboratory experimentation, clinical data management, clinical trial
design, dictionary coding and pharmacovigilance.

Rodney has worked for BioPharm Systems for eleven years now
serving in a variety of roles all related to the technical and/or clinical
implementations of software systems used in the clinical trial
process.

Prior to coming to BioPharm Systems Rodney worked at
pharmaceutical and technology companies in the Dictionary Coding,
Statistical Programming and Data Management areas.

In addition to his current work at BioPharm Systems, Rodney holds
an Associate faculty position at Walden University teaching Public
Health Informatics and other health information systems courses.

Rodney holds a Bachelor of Science in Genetic Engineering, a
Masters of Public Health in International Epidemiology and a Ph.D. in
Epidemiology focusing on Social Epidemiology


Introduction to Pharmacovigilance Signal Detection

More Related Content

What's hot (20)

Similar to Introduction to Pharmacovigilance Signal Detection (20)

More from Perficient (20)

Introduction to Pharmacovigilance Signal Detection