Cognitive Pre-testing

Joint Seminar of the Gender Task Force and the Tool Pool:
Cogni:ve Pretes:ng of
Cross-‐na:onally Comparable
Survey Instruments in a
Developing Country Context
with
applica*on
to
USAID
Feed
the
Future’s
Women’s
Empowerment
in
Agriculture
Index
Presented
at
the
Interna*onal
Food
Policy
Research
Ins*tute
Washington,
DC
9
May
2014
Kiersten
B.
Johnson,
PhD
Senior
Study
Director,
Westat

Objec:ves
• Understand
the
purpose
of
cogni*ve
interviewing/pretes*ng
• Become
familiar
with
the
cogni*ve
processes
that
are
assessed
• Recognize
when
it
is
necessary
to
implement
• Learn
what
methodological
decisions
can
be
made
to
tailor
the
pretes*ng
to
the
needs
and
characteris*cs
of
the
survey
• Obtain
a
sense
for
how
to
train
and
implement

Approach
• Overview
of
the
methodology
• Walk-‐through
of
the
steps
to
implement
• Demonstra*on
of
how
the
method
was
applied
for
the
Women’s
Empowerment
in
Agriculture
ques*onnaire
in
Haï*

Overview of the
Cogni:ve Interview
Method

Why cogni:ve pretes:ng?
We
want
to
systema*cally
iden*fy
and
analyze
sources
of
response
error
in
surveys,
and
use
that
informa*on
to
try
to
improve
the
quality
and
accuracy
of
our
survey
instruments.
Especially
important
with
new
or
revised
instruments/
ques*ons
Cri*cal
when
being
used
cross-‐
culturally
and/or
cross-‐
linguis*cally
Cogni:ve Interviewing?

Four Stages of Cogni:on Required for an
Accurate Response to a Survey Ques:on:
Cogni&ve
Stages
Cogni&ve
Stage
Defini&on
Problems
Causes
1.
Comprehension
Respondent
interprets
the
ques*on
Respondent
does
not
understand
Unknown
terms,
ambiguous
concepts,
long
and
overly
complex
2.
Retrieval
Respondent
searches
memory
for
relevant
informa*on
Respondent
does
not
remember/does
not
know
Recall
difficulty,
ques*ons
assume
respondent
has
informa*on
3.
Judgment
Respondent
evaluates
and/or
es*mates
response
Respondent
does
not
want
to
tell,
can’t
tell
Biased
or
sensi*ve,
es*ma*on
difficulty
4.
Response
Respondent
provides
informa*on
in
the
format
requested
Respondent
can’t
respond
in
the
format
requested
Incomplete
response
op*ons,
mul*ple
responses
necessary

What is cogni:ve pretes:ng?
• Qualita*ve
Cogni:ve Interviewing?
field
research
method
embedded
in
survey
interview
• Can
be
comprised
of:
• Direc*ons
how
people
interpret
the
ques*ons
difficul*es
people
had
construct
their
in
answers
answering
to
“think
aloud”
while
formula*ng
a
response
to
a
survey
ques*on
• Addi*onal
probes/ques*ons
integrated
into
dra_
instrument
• Interviewer
observa*ons
of
respondent
verbal
&
nonverbal
cues
• Intended
to:
• Help
determine
whether
the
ques*on
is
genera*ng
the
intended
informa*on
• Iden*fy
problems
related
to
ques*onnaire
design
Can
respondents
discuss
the
survey
ques3ons
with
a
reasonable
degree
of
validity?

Different Approaches
• Lab-‐based
Methodological Approaches
vs
field-‐based
• Thinking
aloud
vs.
probing
(vs.
a
combina*on
of
both)
• Concurrent
vs.
retrospec*ve
• Standardized
vs.
“ac*ve”
• Scripted,
semi-‐scripted,
or
en*rely
improvised
• Number
of
respondents
and
itera*ons
• Rela*vely
unskilled
data
collector
or
expert
inves*gator
• Analysis:
systema*c
review
of
interview
transcripts,
or
based
on
notes
Selec*on
of
methodological
approaches
is
likely
to
vary
according
to:
-‐ Type
and
amount
of
resources
available
(human,
temporal,
financial)
-‐ Nature
of
the
survey

Validity and Reliability of Cogni:ve Pretes:ng:
Poten:al Types of Error
• Iden*fica*on
of
“problems”
with
ques*ons
that
turn
out
not
to
be
“real”
during
survey
implementa*on
• difficult
to
measure;
limited
research
suggests
it’s
not
significant
problem
• Failure
to
iden*fy
problems
that
actually
exist
in
survey
instrument
design
and/or
administra*on
• almost
certain
to
occur,
but
possible
to
reduce
• Cogni*ve
interview
findings
may
be
inconsistent
when
interviews
are
implemented
by
independent
researchers
• not
necessarily
problema*c

Steps in
Implemen:ng a
Cogni:ve Pretest

Example: Steps in Implementa:on
• Develop
the
ques*onnaire
• Translate
the
ques*onnaire
• Iden*fy
candidate
ques*ons
for
assessment
• Select
methods;
develop
appropriate
probes
• Determine
number
and
selec*on
of
respondents
• Select
and
train
interviewers
• Prac*ce
interviews,
note-‐taking
and
audio
recording
• Implement
the
cogni*ve
interviews
• Summarize
field
notes;
possibly
use
so_ware
to
facilitate
data
analysis
• Write
report:
background,
data
and
methods,
results,
conclusions
&
recommenda*ons
• Revise
ques*ons
• Reiterate
METHOD-‐
OLOGICAL
CHOICES
EMBEDDED
HERE!

Cogni&ve
Stages
Cogni&ve
Stage
Defini&on
Problems
Causes
Compre-‐
hension
Respondent
interprets
the
ques*on
Respondent
does
not
understand
Unknown
terms,
ambiguous
concepts,
long
&
overly
complex
Retrieval
Respondent
searches
memory
for
relevant
informa*on
Respondent
does
not
remember/
does
not
know
Recall
difficulty,
ques*ons
assume
respondent
has
informa*on
Judgment
Respondent
evaluates
and/or
es*mates
response
Respondent
does
not
want
to
tell,
can’t
tell
Biased
or
sensi*ve,
es*ma*on
difficulty
Response
Respondent
provides
informa*on
in
format
requested
Respondent
can’t
respond
in
the
format
requested
Incomplete
response
op*ons,
mul*ple
responses
necessary
Probes
E.g.,
“Earlier
I
asked
you
who
contributes
most
to
decisions
regarding
a
new
purchase
of
different
items.
How
would
you
put
that
ques*on
in
your
own
words?
Is
this
ques*on
easy
or
difficult?
Why
is
it
difficult?”
E.g.,
“Many
people
find
it
difficult
to
recall
every
ac*vity
done
in
a
day.
How
well
do
you
remember
specific
ac*vi*es
you
were
doing
in
the
past
24
hours?”
“How
do
you
know
what
*me
you
woke
up
yesterday?”
E.g.,
“Do
you
think
that
any
other
women
you
know
may
be
afraid
to
answer
this
ques*on?
Why
do
you
think
they
might
be
afraid?”
E.g.,
“Earlier
I
asked
you
how
sa*sfied
are
you
with
your
available
*me
for
leisure
ac*vi*es
like
visi*ng
neighbors,
watching
TV,
listening
to
the
radio,
seeing
movies
or
doing
sports.
Was
this
ques*on
easy
or
difficult?
Why
was
it
difficult?”

Interviewer Training Content
• Understanding
what
cogni*ve
pretes*ng
is,
why
it
is
necessary
for
survey
ques*onnaire
development
• Knowledge
of
the
cogni*ve
processes
required
of
survey
respondents
• Full
understanding
of
the
inten*on
of
the
ques*ons
being
tested
• Knowledge
of
the
probes
to
be
used
in
cogni*ve
tes*ng
interviews
• Facility
with
in-‐depth
qualita*ve
interview
techniques
and
the
no*on
of
“narra*ves
as
data”
• Prac*cal
exercises
in
doing
cogni*ve
interviews,
to
include
cri*cal
group
discussion

Interviewer Training Content
Things
to
be
mindful
of
when
using
regular,
seasoned
survey
interviewers
to
implement
cogni*ve
pretes*ng:
• Revised
priori*es:
iden*fica*on
of
problems
of
understanding,
not
resolu*on
of
problems
of
understanding
• Need
to
administer
slowly
and
allow
*me
for
thoughkul
responses,
rather
than
proceed
expedi*ously
(cont.)

Materials for Training
• Slide
show
on
cogni*ve
tes*ng
and
why
it
is
necessary
• Ques*onnaire
to
be
tested
• A
detailed
ques*on-‐by-‐ques*on
interviewer’s
guide,
customized
for
the
cogni*ve
interview
• A
set
of
generic
possible
probes
for
use
in
the
interviews;
e.g.:
• “Why
did
you
answer
in
that
way?”
• “Tell
me
a
lille
bit
more”
• “Can
you
think
of
an
example
of
what
you
are
talking
about?”
• Signs
of
respondent
cogni*ve
difficul*es
• Long
silences
• Contradic*ons
• Reluctance
or
other
discomfort

Number of Interviews & Respondent Selec:on
• Number
Number of Interviews & Respondent Selec:on
of
interviews:
• Implement
for
each
country
>>
Within
each
country,
implement
for
each
language
comprising
10%
or
more
of
the
survey
sample
popula*on
>>
For
each
language,
implement
15
well-‐conducted
interviews*
(see
next
slide)
• Respondent
selec*on:
• Select
according
to
demographic
profile
of
eligible
respondents
to
the
survey
• Lab
vs
field
implementa*on
“The
palern
of
responses
across
a
limited
sample
of
respondents
provides
insight
into
the
common
palerns
of
understanding
and
interpreta*ons
of
people
who
are
typical
of
the
targeted
survey
respondents.”
-‐-‐
UNESCAP
2010

“For each language, implement 15 well-‐
conducted interviews*”
• No
empirically-‐grounded
conclusions
on
op*mal
sample
sizes;
typical
recommenda*ons
range
from
10-‐15
interviews
• Important
to
have
enough
respondents
so
all
relevant
aspects
of
a
ques*onnaire
are
tested
“The
point
is
not
to
obtain
sample
sizes
large
enough
to
supply
precision
in
sta*s*cal
es*mates.
Rather,
we
strive
to
interview
a
variety
of
individuals
who
will
be
useful
in
informing
our
decisions.
Sta*s*cians
strive
to
minimize
(error)
variance,
whereas
cogni*ve
interviewers
maximize
(subject)
variance.”
-‐-‐
Willis
2005

Interview team
• One
Team
interviewer
to
ask
the
respondent
ques*ons
• One
interviewer
to
operate
the
digital
recorder,
and
observe
&
take
notes
of
the
respondent’s
verbal
&
non-‐verbal
cues

Materials
Interview Materials
• Tailored
consent
form
• Customized
ques*onnaire
(hard
copy)
• Digital
recorder
• Pens/pencils/clipboard

Preparing the Respondent
• Informed
consent
to
record
• Explain
that
the
interview
is
evalua*ve,
and
we
will
be
asking
for
their
input
on
different
aspects
of
the
ques*ons
and
responses

Data Analysis
• Analysis
method
will
depend
on
type
of
data
• Fundamentally
qualita*ve;
coding
of
textual
responses
will
likely
be
needed
to
be
able
to
compare
responses
across
respondents
• Triangulate:
• Responses
to
closed-‐ended
CI
ques*ons
• Narra*ve
responses
to
probes
captured
in
text
• Interviewer
observa*ons
recorded
in
text
• Digital
recording
of
interview
• Applica*on
of
logic
to
interpret
findings
&
make
revisions
accordingly

Applica:on of
Cogni:ve Interview
Method:
Women’s
Empowerment in
Agriculture
Ques:onnaire in
Haï:

Cogni:ve Pretest of the Women’s Empowerment
in in Agriculture Agriculture Instrument Instrument in in HaïHaï:
:
• Step
1:
Decide
if
we
need
to
implement
cogni*ve
tes*ng.
• New
topic
being
addressed
• New
instrument
developed
for
assessing
women’s
empowerment
in
ag
• Cross-‐na*onal
comparability
of
ques*ons
and
responses
is
important
• Step
2:
Iden*fy
selected
ques*ons
that
respondents
may
find
cogni*vely
difficult.

Iden:fy Key Ques:ons (1)
Module
H2
• H2.01.
Recall
difficulty
• H2.01.
Recall
period
(last
12
months)
• H2.01.
Difficult
terms:
“in-‐kind
or
monetary
work
both
agriculture
and
other
wage
work”
• H2.02.
&
H2.03.
Comprehension:
“Input
into
decisions”
• H2.02.
&
H2.03.
Sensi*vity:
Input
into
decisions
Module
H3a
–
Access
to
Produc&ve
Capital
• H3.01a:
Household
items
owned:
concept
“household”,
difficult
items
(nonfarm
business
equipment,
Farm
equipment
(non-‐
mechanized),
Farm
equipment
(mechanized)
• H3.01b:
Number
of
items
owned:
number
recall
difficulty
• H3.02:
Owner
of
items:
judgment
difficulty
• H3.03:
Decide
to
sell:
judgment
difficulty
• H3.04:
Decide
to
give
away:
judgment
difficulty
• H3.05:
Decide
to
mortgage
or
rent
out:
judgment
difficulty,
difficult
terms
(mortgage,
rent
out)

Module
H3b
–
Access
to
Produc&ve
Capital
• H3.07:
Anyone
taken
loans
in
cash/kind:
Comprehension
(“Borrow
in-‐kind”),
recall
(last
12
months)
• H3.08:
Decision
to
borrow:
Judgment
• H3.09:
Decision
about
what
to
do
with
loans:
Judgment
• H3.10a:
Demand
for
extra
credit:
Comprehension
(condi*onal
sentence)
• H3.10b:
Reasons
for
lack
of
demand
for
extra
credit:
Judgment
• H3.11a:
Limita*ons
to
credit
supply:
Judgment
• H3.11b:
Reasons
for
limited
credit
supply:
Judgment,
sensi*vity
Module
H4
• H4.01:
Infrastructure:
Sensi*vity
• H4.02:
Proper
payment
of
wages
for
public
works:
Sensi*vity,
comprehension
• H4.03:
Protest
the
misbehavior
of
authori*es
or
elected
officials:
Sensi*vity
• H4.04:
Group
membership:
Comprehension
of
the
different
groups
involved
• H4.05:
Ac*ve
membership:
comprehension
Module
H5a
• H5.01:
Decision-‐maker:
Possible
sensi*vity
if
privacy
not
maintained
• H5.02:
Personal
decisions:
Possible
sensi*vity
if
privacy
not
maintained

Module
H5b
• H5.03:
Get
in
trouble:
comprehension;
Act
differently:
comprehension
(item
wording,
“Gevng
inputs
for
agricultural
produc*on”),
sensi*vity
• H5.04:
Others
think
poorly:
comprehension
(item
wording,
“think
poorly”),
sensi*vity
• H5.05:
Right
thing
to
do:
comprehension
(item
wording,
“right
thing
to
do”)
Module
H6
• 24
hour
recall
problems
(remembering
every
ac*vity,
*me
of
the
day)
• Variability
of
daily
schedules
• Comprehension:
“sa*sfied”
• PLUS…
dura*on
of
each
module,
debrief
interview
ques*ons

Selec:on of Methods
• Lab-‐based
vs
field-‐based
à
field-‐based
• Thinking
aloud
vs.
probing
(vs.
a
combina*on
of
both)
à
probing
• Rela*vely
unskilled
data
collector
or
expert
inves*gator
à
specially
trained
data
collector
• Concurrent
vs.
retrospec*ve
à
middle
road:
between
modules
• Standardized
vs.
“ac*ve”
à
standardized
• Scripted,
semi-‐scripted,
or
en*rely
improvised
à
scripted
• Number
of
respondents
à
12
HHs;
12
female
respondents
&
8
male
respondents
• Number
of
itera*ons
à
just
one
was
possible

Table
1.
Household
and
individual
samples
according
to
age
of
the
female
respondent
Age
of
female
respondent
Households
sampled
Dual-‐headed
household
Single
female-‐headed
household
18-‐35
4
2
36
and
older
4
2
Individuals
Sampled
Dual-‐headed
household
Single
female-‐headed
household
18-‐35
8
2
36
and
older
8
2

H2.01. Did you par:cipate in [ACTIVITY] in the past 12 months
(that is during the last [one/two] cropping seasons)?
• One
fi_h
of
the
20
respondents
reported
that
this
ques*on
was
difficult.
• In
some
cases,
there
were
comprehension
difficul*es;
respondents
generated
the
following
feedback
on
this
ques*on:
“I
haven’t
understood
the
ques*on,
that’s
why
it
seems
difficult
to
me.”
(Man,
age
49,
dual-‐headed
household)
“I
haven’t
understood
the
ques*on
well.”
(Woman,
age
67,
single-‐headed
household)

What does the term “in-‐kind work” mean for you?
• Only
three
respondents
approached
the
correct
meaning
• Outright
incorrect
responses:
• Don’t
know
• Incorrect
responses:
“own
account
work”
or
“work
to
earn
cash
or
something
else”
• Imprecise
responses
that
suggest
a
comprehension
problem:
“work
for
your
life,”
“work
to
obtain
a
benefit,”
“work
to
survive,”
etc.
• Two
respondents
indicated
the
item
may
be
sensi*ve
as
it
draws
alen*on
to
income-‐
genera*on
ac*vi*es,
which
respondents
may
consider
to
be
private
informa*on:
“The
ques*on
is
difficult
because
I
don’t
have
anything
to
make
a
living.”
(Man,
age
68,
dual-‐headed
household)
“(It’s
difficult)
because
I
don’t
want
to
tell
anyone.”
(Woman,
age
46,
single-‐headed
household)

Earlier
I
asked
you
if
you
do
what
you
do
regarding
different
ac*vi*es
so
that
others
don’t
think
poorly
of
you.
How
would
you
put
this
ques*on
in
your
own
words?
Is
this
ques*on
easy
or
difficult?
Why
was
it
difficult?

“Penser mal” has a range of meanings…
“To
bring
forward
a
cri*cism
that
does
not
please
you”
(Man,
age
33,
dual-‐headed
household)
“To
think
of
hur*ng
someone”
(Woman,
age
25,
dual-‐headed
household)
“It
means
destroying
the
life
of
a
person”
(Man,
age
32,
dual-‐headed
household)
“When
you
think
evil
of
a
person,
you
can
kill
that
person”
(Woman,
age
46,
single-‐headed
household)
“A
person
who
is
there
to
kill
someone”
(Woman,
age
67,
single-‐headed
household)

Time Use: Respondents were asked how they
knew what :me of the day it was.
• 50%
check
the
&me
using
a
cell
phone
they
carry
with
them
“With
my
phone,
I
check
it
every
*me
I
start
an
ac*vity”
(Man,
age
36,
dual-‐headed
household)
Nearly
half
rely
on
others:
“I
always
ask
the
neighbors
what
*me
is
it”
(Woman,
age
36,
dual-‐headed
household)
“I
get
the
*me
from
a
friend”
(Man,
age
40,
dual-‐headed
household)

References:
Key References
• Willis,
Gordon
B.
2005.
Cogni3ve
Interviewing:
A
Tool
for
Improving
Ques3onnaire
Design.
Thousand
Oaks,
CA:
Sage.
• Bealy,
Paul
C.
and
Gordon
B.
Willis.
2007.
“Research
Synthesis:
The
Prac*ce
of
Cogni*ve
Interviewing.”
Public
Opinion
Quarterly
71(2):287-‐311.
• UNESCAP.
2010.
Guidelines
for
Cogni3ve
and
Pilot
Tes3ng
of
Ques3ons
for
Use
in
Surveys.
Sta*s*cs
Division,
Economic
and
Social
Commission
for
Asia
Pacific
Region.
• Johnson,
Kiersten
B.
&
Diego-‐Rosell,
Pablo.
2014.
Assessing
the
Cogni*ve
Validity
of
the
Women’s
Empowerment
in
Agriculture
Index
Instrument
in
the
Hai*
Mul*-‐Sectoral
Baseline
Survey.
Manuscript
under
prepara3on
for
submission
to
peer-‐reviewed
publica3on.
• Blair,
J.,
Conrad,
F.,
Ackermann,
A.
and
Claxton,
G.
2006.
“The
Effect
of
Sample
Size
on
Cogni*ve
Interview
Findings.”
Paper
presented
at
the
American
Associa3on
for
Public
Opinion
Research
Conference.
Montreal,
Canada.
• Conrad,
Frederick
G.
and
Johnny
Blair.
2009.
“Sources
of
Error
in
Cogni*ve
Interviews.”
Public
Opinion
Quarterly
73(1):32-‐55.

Photo credits
• hlp://blogs.agu.org,
12/2012
(slide
#1)
• Chris
Blalman
(slide
#4)
• Dinesh
Maneer
Photography
(slide
#10)
• Doug
Satre
(slide
#22)
• hlp://church*mes.co.uk
(slide
#34)
Credits

Cognitive Pre-testing

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Cognitive Pre-testing (20)

More from IFPRI Gender (20)

Recently uploaded (20)

Cognitive Pre-testing