Mixed Effects Models - Signal Detection Theory

Week 13.1: Signal Detection Theory
! Signal Detection Theory
! Why Do We Need SDT?
! Sensitivity vs. Response Bias
! Implementation
! SDT & Other Independent Variables
! Logit vs. Probit

Tasks With Categorical Decisions
las gatos
(1) Grammatical
(4) Ungrammatical
The cop saw the
spy with the
binoculars.
• In analyzing these decisions, need to consider both overall
preference for certain categories & judgments of individual items

Study:
POTATO
SLEEP
RACCOON
WITCH
NAPKIN
BINDER

• Test:
• SLEEP
• POTATO
• BINDER
• WITCH
• RACCOON
• NAPKIN

• Test:
• SLEEP
• POTATO
• BINDER
• WITCH
• RACCOON
• NAPKIN
• In early memory experiments, all test probes were
previously studied items
• No way to distinguish a person who actually
remembers everything from a person who’s realized
these are ALL “old” items
Study:
POTATO
SLEEP
RACCOON
WITCH
NAPKIN
BINDER

• Test:
• SLEEP
• POTATO
• HEDGE
• BINDER
• SHELL
• RACCOON
• MONKEY
• OATH
• Adding “lure” items helps make the task less obvious
• But still have to interpret response to lures
• Did this person circle 50% of studied items because
they remember seeing those words … or because
they circled 50% of everything?
Study:
POTATO
SLEEP
RACCOON
WITCH
NAPKIN
BINDER

Signal Detection Theory
• For analyzing categorical judgments
• Part method for analyzing judgments
• Part theory about how people make
judgments
• Originally developed for
psychophysics
• Purpose:
• Better metric properties than ANOVA on
proportions (logistic regression has already
taken care of this)
• Distinguish sensitivity from response bias

Sensitivity vs. Response Bias
“If you’re
not sure,
guess C”
Knowing
which
answers are
C and
which aren't
Response
bias
Sensitivity

! Imagine asking groups of second-language
learners of English to judge grammaticality...

Grammatical condition
Ungrammatical cond.
80%
20%
80%
80%
ACCURACY
Without Intervention
People just judge 80% of sentences grammatical in both
conditions.
This is all response bias—no evidence that they are sensitive to
whether particular sentences are grammatical or not.
SAID
“GRAMMATICAL”

Ungrammatical cond.
80%
20%
80%
80%
ACCURACY
SAID
“GRAMMATICAL”
With Intervention
60%
60%
Ungrammatical cond.
60%
40%
Similarly, an intervention could shift response bias without
actually increasing sensitivity.

! Proportion accuracy would be misleading
! We want an analysis that tests both subjects’
sensitivity and their response bias
Ungrammatical cond.
80%
20%
80%
80%
ACCURACY
SAID
“GRAMMATICAL”
With Intervention
60%
60%
Ungrammatical cond.
60%
40%

! Comparison to “chance” get at a similar idea
! But, that assumes all responses equally likely
! Many experiments do balance frequency of
intended responses
! But even so, bias can differ for many reasons
– Relative frequency in experiment
– Prior frequency in the world (“no disease” less
common than “disease”)
– Motivational factors (e.g., one error “less bad”
than another)
– Not bad to have a response bias—we just need
to account for it in our analysis!

Sensitivity vs. Response Bias: Examples
• We present radiologists with 20 X-rays. Half
of the X-rays show lung disease and half show
healthy lungs. For each X-ray, the radiologist
has to judge whether lung disease is present.
• In this study, how can we define…
• Response bias?
• Sensitivity?

• We present radiologists with 20 X-rays. Half
of the X-rays show lung disease and half show
healthy lungs. For each X-ray, the radiologist
has to judge whether lung disease is present.
• Response bias?
• Overall propensity to judge that lung disease is present
• Sensitivity?
• Does the radiologist diagnose the
patient with lung disease more in the
cases where the patient actually has
lung disease?

• We are conducting a cross-cultural study of color
perception. Participant in a variety of cultures each
see 40 pairs of paint chips. For every pair, the
participant judges if the two chips are the same color
or different colors. In reality, 20 pairs are the same
color, and 20 pairs are different colors.
• Response bias?
• Sensitivity?

• We are conducting a cross-cultural study of color
perception. Participant in a variety of cultures each
see 40 pairs of paint chips. For every pair, the
participant judges if the two chips are the same color
or different colors. In reality, 20 pairs are the same
color, and 20 pairs are different colors.
• Response bias?
• Overall tendency to judge pairs as
the same
• Sensitivity?
• Do people judge pairs as the same
more when they are actually the
same?

• An I/O psychologist is interested in how
extracurricular activities influence the post-college
job search. Each research participant sees a series
of fictitious resumes and, for each resume, judges
whether they think the person merits hiring. The
researcher experimentally varies the number of
extracurricular activities listed on the resumes.
• Response bias?
• Sensitivity?

• An I/O psychologist is interested in how
extracurricular activities influence the post-college
job search. Each research participant sees a series
of fictitious resumes and, for each resume, judges
whether they think the person merits hiring. The
researcher experimentally varies the number of
extracurricular activities listed on the resumes.
• Response bias?
• Overall tendency to think
people merit hiring
• Sensitivity?
• Do extracurricular activities
increase hiring?

• We present undergraduates with a series of moral dilemmas
in which they have to imagine deciding between saving 1
person’s life and saving several people’s lives. The
dependent measure is how often people make the utilitarian
choice to save several people. Some scenarios are less
personal, and we hypothesize that people will make more
utilitarian choices in these scenarios.
• Response bias?
• Sensitivity?

• We present undergraduates with a series of moral dilemmas
in which they have to imagine deciding between saving 1
person’s life and saving several people’s lives. The
dependent measure is how often people make the utilitarian
choice to save several people. Some scenarios are less
personal, and we hypothesize that people will make more
utilitarian choices in these scenarios.
• Response bias?
• Overall frequency of utilitarian judgments
• Sensitivity?
• Do people make more of the utilitarian
judgments when the scenario is less
personal?

• We ask college students studying French to proofread a set of
40 French sentences, all of which contain a subject/verb
agreement error. The dependent measure is whether or not
the student judge the sentence as containing a subject/verb
agreement error (i.e., “error” or “no error”).
• Response bias?
• Sensitivity?

• We ask college students studying French to proofread a set of
40 French sentences, all of which contain a subject/verb
agreement error. The dependent measure is whether or not
the student judge the sentence as containing a subject/verb
agreement error (i.e., “error” or “no error”).
• Response bias?
• Sensitivity?
Trick question!! This is like the memory test that contains only “old”
items. Because the test only contains errors, there’s no way to tell
whether a participant’s response is driven by their general bias to
report errors or by noticing the error in this specific sentence. We
cannot separate response bias from sensitivity here. Unfortunately,
this limits the conclusions we can draw from this task.

Example Study:
Both the British and the French biologists
had been searching Malaysia and Indonesia
for the endangered monkeys.
Finally, the British spotted one of the
monkeys in Malaysia and planted a radio
tag on it.
Fraundorf, Watson, & Benjamin (2010)

The British scientists spotted the
endangered monkey and tagged it.
TRUE FALSE
Probe type = TRUE

The French scientists spotted the
endangered monkey and tagged it.
TRUE FALSE
Probe type = FALSE

SDT & Mixed Effects Models
! Traditional logistic regression model:
! Accuracy confounds sensitivity and
response bias
– Accuracy might differ across probe types
just because of bias to respond true
CORRECT MEMORY or
INCORRECT MEMORY?
Correct ~ 1 + ProbeType

! Signal detection model:
CORRECT MEMORY or
INCORRECT MEMORY?
JudgmentMade ~ 1 + ProbeType
JUDGED “TRUE” OR
JUDGED “FALSE”
JUDGED “GRAMMATICAL” OR
“UNGRAMMATICAL”

! More generally:
CORRECT MEMORY or
INCORRECT MEMORY?
glmer(JudgmentMade ~ 1 + StimulusCategory +
(1|RandomEffect), data=dataname,
family=binomial)

Respond
correctly
or
Respond
incorrectly?
True
statement
or
False
statement?

! SDT model:
Said
“TRUE”
=
Probe Type
is TRUE
Intercept
Baseline rate of responding TRUE.
Does item being true make you
more likely to say TRUE?
Overall
response
bias
Sensitivity
+
w/ effects coding…

! SDT model:
Said
“TRUE”
=
Probe Type
is TRUE
Intercept
Overall
response
bias
Sensitivity
+
Results

! More generally:
Responded
w/ category A
=
Stimulus
Type
Intercept
Baseline rate of “A” responses
Does item being in category “A”
make you more likely to judge as
”A”?
Overall
response
bias
Sensitivity
+
JudgmentMade ~ 1 + StimulusType

Now You Try It!
! bpd.csv
! Clinical trainees evaluating learning to diagnose
borderline personal disorder (BPD). Each
trainees sees 60 cases—half with BPD and half
without—and makes a diagnosis for each.
! Potentially relevant columns:
! JudgedBPD: Trainees’ judgment of BPD (1 yes, 0 no)
! HasBPD: Whether the person in the case actually has
BPD—as diagnosed by expert (“Y” or “N”)
! Accuracy: Was the trainees’ judgment correct? (1
yes, 0 no)

Now You Try It!
! If our memory experiment SDT analysis involved
a model formula like this:
! Can you run a SDT model on the bpd data?
! Tip 1: Apply effects coding (-0.5 and 0.5) to the
predictor variable!
! Tip 2: Should this be an lmer model or a glmer
model?
JudgmentMade ~ 1 + ProbeType + (1|Subject)

Now You Try It!
! If our memory experiment SDT analysis involved
a model formula like this:
! Can you run a SDT model on the bpd data?
! contrasts(bpd$HasBPD) <- c(-0.5, 0.5)
! model1 <- glmer(JudgedBPD ~
1 + HasBPD + (1|Trainee),
family=binomial, data=bpd)
JudgmentMade ~ 1 + ProbeType + (1|Subject)

Now You Try It!
Intercept: Overall tendency to judge people as having BPD or not
• Response bias (here, not significant)
HasBPD: Do we get more “has BPD” judgments when the person
actually has BPD?
• Sensitivity (significant!)

Now You Try It!
Our model of the random effects is that trainees differ only in their
intercept
• They diifer only in response bias … not in sensitivity
Can we also allow the sensitivity to be different for each trainee?

Now You Try It!
1 + HasBPD + (1 + HasBPD|Trainee),

Example Study:
Both the British and the French biologists
had been searching Malaysia and Indonesia
for the endangered monkeys.
Finally, the British spotted one of the
monkeys in Malaysia and planted a radio
tag on it.
Emphasized
or not?
Fraundorf, Watson, & Benjamin (2010)
We now have an additional
independent variable.

SDT & Other Independent Variables
! Signal detection model with another
independent variable:
my.model <- glmer(
JudgmentMade ~ 1 + ProbeType*Emphasis
+ (1|Trainee),
family=binomial,
data=memory)
JUDGED “TRUE” OR
JUDGED “FALSE”

! More generally…
my.model <- glmer(
JudgmentMade ~ 1 + StimulusType*OtherIV
+ (1|RandomEffect),
family=binomial,
data=mydata)

! SDT model:
Said
“TRUE”
=
Probe Type
is TRUE
Contrastive
Emphasis
Intercept
Emphasis x
TRUE
Does contrastive emphasis change
overall rate of saying TRUE?
Does emphasis especially increase
TRUE responses to true items?
Overall
response
bias
Overall
sensitivity
Effect on
bias
Effect on
sensitivity
+
+
+

! SDT model:
Said
“TRUE”
=
Probe Type
is TRUE
Contrastive
Emphasis
Intercept
Emphasis x
TRUE
Does contrastive emphasis change
overall rate of saying TRUE?
Does emphasis especially increase
TRUE responses to true items?
Overall
response
bias
Overall
sensitivity
Effect on
bias
Effect on
sensitivity
+
+
+
Results
+

! More generally…
Responded
w/ category A
=
Stimulus
Type
OtherIV
Intercept
Interaction
Baseline rate of “A” responses
Does item being in category “A”
make you more likely to judge as
“A”?
Does other independent variable
change overall rate of saying “A”?
Does other IV increase ability to
identify which category item is in?
Overall
response
bias
Overall
sensitivity
Effect on
bias
Effect on
sensitivity
+
+
+

Example 2: Ferreira & Dell (2000) Expt 6
• When & how do people avoid ambiguity in what they say?
• Task: Read sentences & repeat back from memory
• Ambiguous sentence start: “The coach knew you…”
– “The coach knew you since sophomore year.” (knowing you)
– “The coach knew you missed practice.” (knowing a fact)
• “The coach knew that you...”
• “that” is optional but clarifies it’s a knowing-a-fact sentence
• Dependent measure: Do people say “that” here?
• Are people sensitive to diff. from unambiguous case?:
• “The coach knew I...”
• Knowing-a-person sentence would be “The coach knew me.”
• Also vary whether instructions emphasize being clear

! SDT model:
Said
“that”
=
Ambiguity
Instructions
Intercept
Instructions
x Ambiguity
Baseline rate of including “that”
Do people say “that” more for you
(unambig.) than for I (ambig.)
Are people told to avoid ambiguity?
Do instructions especially increase
use of “that” for ambiguous items?
Overall
response
bias
Overall
sensitivity
Effect on
bias
Effect on
sensitivity
+
+
+

! SDT model:
Said
“that”
=
Ambiguity
Instructions
Intercept
Instructions
x Ambiguity
Baseline rate of including “that”
Do people say “that” more for you
(unambig.) than for I (ambig.)
Are people told to avoid ambiguity?
Do instructions especially increase
use of “that” for ambiguous items?
Overall
response
bias
Overall
sensitivity
Effect on
bias
Effect on
sensitivity
+
+
+
Results

Example 2: Ferreira & Dell (2000) Expt 6
• People NOT sensitive to whether what they’re saying is
grammatically ambiguous
• Effect of emphasizing clarity is that people just add extra
“that”s everywhere (whether actually needed or not)
• Case where a change in response bias tells us something
interesting about what people are doing
• Response bias is NOT just something we want to avoid /
get rid of
• Can be theoretically interesting
• Our measure of sensitivity in the SDT model is
independent of response bias, so OK to look at sensitivity
even if there is a response bias effect

Back to Our BPD Data…
! We’re concerned that there may be a Gender
bias in diagnoses of BPD (e.g., Bjorklund,
2009; Skodol & Bender, 2003)
! Can you test whether Gender affects response
bias and/or sensitivity in your model?
! Don’t forget to apply effects coding (-0.5 and 0.5) to
Gender
! Which gender do we think will get more BPD
diagonses?

! We’re concerned that there may be a Gender
bias in diagnoses of BPD (e.g., Bjorklund,
2009; Skodol & Bender, 2003)
! Can you test whether Gender affects response
bias and/or sensitivity in your model?
! contrasts(bpd$Gender) <- c(0.5, -0.5)
1 + HasBPD*Gender +
(1+HasBPD*Gender|Trainee),

Intercept: Overall tendency to judge people as having BPD or not
• Response bias (here, not significant)
HasBPD: Do we get more “has BPD” judgments when the person
actually has BPD?
• Sensitivity (significant!)
Gender: An effect of BPD on “has BPD” judgments, regardless of
whether the person has BPD
• This an effect of gender on response bias!
Gender:HasBPD: Is “has BPD” larger for one gender?
• No – no effect of gender on sensitivity

! Summary:
! No overall response bias to judge people as having
BPD or not
! Trainees have some ability to discern which people
have BPD and which don’t
! Overall bias to diagnosis more women with BPD, but
doesn’t affect sensitivity to the symptoms in making
the diagnosis

Logit and Probit
• How to link the binomial response
to the continuous model predictors?
• So far, we’ve been using the logit:
• Probit: Based on the cumulative
distribution function of the normal
p(recall)
1-p(recall)
[ ]
logit = log
d’ = CDF(recall) – CDF(1-recall)
Area under curve from -∞ up
to this point

Logit and Probit
• Extremely similar, but logit a little less sensitive to
extreme values
• Thus, will probably get qualitatively the same results
• Which to choose?
• Some literatures (SDT) use d’ units -> Probit model
• Otherwise, logit has a somewhat easier interpretation
• Odds / odds ratios

Probit
• To use the probit instead of the logit:
• model.Probit <- glmer(JudgedBPD ~
1 + HasBPD + (1 + HasBPD|Trainee),
data=bpd, family=binomial(link='probit'))
• (link='logit') is the same as the default model

Mixed Effects Models - Signal Detection Theory

More Related Content

What's hot (20)

Similar to Mixed Effects Models - Signal Detection Theory (20)

More from Scott Fraundorf (13)

Recently uploaded (20)

Mixed Effects Models - Signal Detection Theory