Questionnaire design & basic of survey

Building Capacity in Evaluating Outcomes
Unit 5: Collecting data
1
Questionnai
re

35.3
• Statistics is a tool for converting data into
information:
Data Statistics Information
But where then does data come from? How is it
gathered? How do we ensure its accurate? Is the
data reliable? Is it representative of the population
from which it was drawn?
Questionnai
re

4
Evaluation
questions
Indicators:
Evidence that
answers your
questions
Sources of
information:
program records,
individuals,
public
METHODS
Who
What
Questionnai
re

5
• Existing information
• People
• Pictorial records and observations
Questionnai
re

6
Quantitative: numbers breadth generalizability
Qualitative: words depth specific
Remember, "Not everything that counts can be counted."
Questionnai
re

7
Quantitative Qualitative
Surveys
Questionnaires
Focus groups
Tests Unstructured
interviews
Existing databases Unstructured
observations
Questionnai
re

8
Mixed methods for one program
• Log of activities and participation
• Self-administered questionnaires
completed after each workshop
• In-depth interviews with key
informants
• Observation of workshops
• Survey of participants
Questionnai
re

9
• Validity: Are you measuring what you think
you are measuring?
• Reliability: if something was measured
again using the same instrument, would it
produce the same (or nearly the same)
results?
Questionnai
re

10
What do these words mean relative to
your evaluation information?
How can you help ensure that your
evaluation data are trustworthy and
credible?
Questionnai
re

11
• Survey
• Case study
• Interview
• Observation
• Group assessment
• Expert or peer
reviews
• Portfolio reviews
• Testimonials
• Tests
• Photographs,
videotapes, slides
• Diaries, journals,
logs
• Document review
and analysis
Questionnai
re

12
 The purpose of your evaluation − Will the
method allow you to gather information that
can be analyzed and presented in a way
that will be credible and useful to you and
others?
 The respondents − What is the most
appropriate method, considering how the
respondents can best be reached, how they
might best respond, literacy, cultural
considerations, etc.?Questionnai
re

13
Consider…
• Resources available. Time, money, and staff to design,
implement, and analyze the information. What can you
afford?
• Type of information you need. Numbers, percent,
comparisons, stories, examples, etc.
• Advantages and disadvantages of each method.
• The need for credible and authentic evidence.
• The value of using multiple methods.
• The importance of ensuring cultural appropriateness.
Questionnai
re

14
UTILITY
Will the data sources and
collection methods serve the
information needs of your primary
users?
Questionnai
re

15
FEASIBILITY
Are your sources and methods
practical and efficient?
Do you have the capacity, time, and
resources?
Are your methods non-intrusive and
non-disruptive?
Questionnai
re

16
PROPRIETY
Are your methods respectful, legal,
ethical, and appropriate?
Does your approach protect and
respect the welfare of all those
involved or affected?
Questionnai
re

17
ACCURACY
Are your methods technically adequate to:
• answer your questions?
• measure what you intend to measure?
• reveal credible and trustworthy information?
• convey important information?
Questionnai
re

18
There is no one right method of collecting
data.
Each has a purpose, advantages, and
challenges.
The goal is to obtain trustworthy, authentic,
and credible evidence.
Often, a mix of methods is preferable.
Questionnai
re

19
• How appropriate is the method given the
culture of the respondent/the setting?
• Culture differences: nationality, ethnicity,
religion, region, gender, age, abilities, class,
economic status, language, sexual
orientation, physical characteristics,
organizational affiliation
Questionnai
re

20
Things to consider:
• Literacy level
• Tradition of reading, writing
• Setting
• Not best choice for people with oral tradition
• Translation (more than just literal translation)
• How cultural traits affect response – response sets
• How to sequence the questions
Questionnai
re

21
Things to consider:
• Preferred by people with
an oral culture
• Language level proficiency;
verbal skill proficiency
• Politeness – responding to authority (thinking it‟s
unacceptable to say “no”), nodding, smiling,
agreeing
• Need to have someone present
• Relationship/position of interviewer
• May be seen as interrogation
• Direct questioning may be seen as impolite,
threatening, or confrontational
Questionnai
re

22
A structured way to collect information
using questionnaires. Surveys are
typically conducted through:
Hand to hand or face to face
Mail (electronic or surface)
Phone
Internet.
Questionnai
re

23
Surveys are used…
• To collect standardized information
from large numbers of individuals
• When face-to-face meetings are
inadvisable
• When privacy is important or
independent opinions and responses
are needed
Questionnai
re

24
1. Decide who should be involved in the process.
2. Define survey content.
3. Identify your respondents.
4. Decide on the survey method.
5. Develop the questionnaire.
6. Pilot test the questionnaire and other materials.
7. Think about analysis.
8. Communicate about your survey and its results.
9. Develop a budget, timeline, and management
process.
Questionnai
re

How survey work to produce statistics
Respondent
answers to
quenstions
Inference
Characteristics of a
respondent Statistical
computing
Characteristics of
the sample
Characteristics of
the population
Inference

A survey from a process perspective
Define research objectives
Choose mode of
collection
Construct and
pretest a
questionnaire
Choose
sampling frame
Design and
select sample
Recruit and
measure sample
Code and edit data
Make postsurvey adjustments
Perform analysis
Questionnaire

The life cycle of a survey from a design
perspective
Construct
Measurement
Response
Edited
Response
Survey statistics
Target Population
Sampling Frame
Sample
Respondent
Postsurvey Adjustments
The Measurement dimension describes
what data are to be collected about the
observational units in the sample
The Representational dimention
concerns what population are
described by the survey
What
is the
survey
about?
Who
is the
survey
about?

The measurement dimension
Constructs are the elements of information that are
sought by the researcher :
How many incidents of crimes with victims there were in the last year;
The consumption of coffee in the last month;
The degree of knowledge of mathematics of childrens…
Measurements are ways to gather information about
constructs :
Questions posed to a respondent (“During the last 6 month, did you call the police
to report something that happened to you that you thought was a crime?”)
NB: the critical task for maesurement is to design questions that produce answers
reflecting perfectly the construct we are trying to measure.
Response could be produced in a variety of means
But in general the nature of the response is determined by the
nature of the measurement
Editing of data may examine the full distribution of
answers and look for atypical patterns of responses
Edited responses are the data from wich inference is made about the values of
the construct for an individual respondent
Construct
Measurement
Response
Edited
Response

The representational dimension
The target population is the set of unit to be studied
The adult population living in households in 2009;
The frame population if the set of target population members
that has chance to be selected into the survey sample :
In a simple case it is a list of all units in the target population, but sometimes it is
a set of units imperfectly linked to population members.
i.e. a list of telephone numbers when the target population is the adult population
The sample is the group from wich measurement will be
sought. In many case it is a very small fraction of the the sampling frame
Postsurvey adjustments consist on weighting up the
underrepresented groups in order to improve the survey
estimate
Because of mismatches of the sampling frame and the target population
(coverage problems) statistics based on the respondents can differ from
caracteristics of the target population. Examination of non response patterns may
suggest an underrepresentation of some groupes relative to the sampling frame
Target Population
Sample
Respondent
Postsurvey
Adjustments
Sampling Frame
Respondents are the elements successfully measures.
Non respondents is the complement

Evaluating survey questions:
Are the answers good measures of the
intended construct?
Example of methods that can be used
to evaluate draft survey questions
 Expert reviews
The substantive expert review the wording, the
order and the structure of questions, the
response alternatives etc.
A small number of target population participate in a
systematic discussion about the survey topic. The
researcher learn about the nomenclature of the concept,
the common perspective taken by the target population
on key issues etc…
 Focus groups
 Questionnaire pretest Researcher test how questions are read
and answered. A behaviour coding is
often used
Questionnaire

32
The proportion of people who respond: divide the
number of returned surveys by the total number of
surveys distributed.
Example: If you distribute 50 questionnaires and
you get 25 questionnaires back, your response rate
is 50%.
Questionnai
re

33
# that answered
# you contacted
Response rate =
Questionnai
re

34
• High response rate promotes confidence in
results.
• Lower response rate increases the likelihood of
biased results.
Questionnai
re

35
• There is no standard response rate. “The
higher, the better.” Anything under 60% is a
warning.
• Why is high return important? It‟s the only
way to know if results are representative.
• Address low response. How are people
who didn‟t respond different from those
who did? Only describe your results in
terms of who did respond.
Questionnai
re

36
• Generate positive publicity for your survey.
• Over sample.
• Ensure that respondents see the value of
participating.
• Use a combination of methods.
• Make (multiple) follow-up contacts.
• Provide incentives.
• Provide 1st class postage/return postage.
• Set return deadlines.
• Make the survey easy to complete.
Questionnai
re

37
Use language that is suggestive
rather than decisive.
Examples: “The data suggests” vs.
“These data show”; “It appears” vs.
“We can conclude”
• Don‟t generalize findings to the
entire group.
• Clearly describe who the data
represents.
Questionnai
re

39
1. Decide what information you need.
2. Determine sample – respondents.
3. Develop accurate, user-friendly
questionnaire.
4. Develop plan for distribution, return,
and follow-up.
5. Provide clear instructions and a
good cover letter.
6. Pilot test.
Questionnai
re

40
• Be specific
• Need to know Vs. would like to know
• Check to see if information exists
elsewhere
• What do you want to be able to say:
counts, percentages, relationships,
narratives
Questionnai
re

41
• Who will complete the questionnaire?
• What do you know about their
preferences, abilities, and cultural
characteristics that may affect the way
they respond?
Questionnai
re

42
• Make sure questions cover information needed.
• Word questions carefully.
• Consider cultural nuances.
• Sequence questions appropriately.
• Attend to formatting.
Questionnai
re

43
• Write clear, complete directions.
• Review to see if it is user-friendly; consider the
respondent.
• Make the questionnaire attractive.
• Work as a team.
• Plan on writing several draft questionnaires.
Questionnai
re

44
Distribution: when, where
• At meetings, sites, through mail, email, internet
Return: when, where
• Return to individual, collection box
• Return envelope addressed/stamped
• Return envelope addressed only
Follow-up
Questionnai
re

45
• Purpose of questionnaire –
how information will be used
• Why they are being asked to fill it out
• Importance of their response
• How and when to respond
• Whether response will be anonymous or
confidential
• Your appreciation
• Promise results, if appropriate
• Signature − sponsorship
Questionnai
re

46
• Always
• With people as similar to respondents as possible
• Do they understand the questions? The instructions?
• Do questions mean same thing to all?
• Do questions elicit the information you want?
• How long does it take?
• Revise as necessary
Questionnai
re

47
• Knowledge − what people know, how well they
understand something
• Beliefs − attitudes, opinions
• Behaviors − what people do
• Attributes/Demographics − what people are and
what people have
Questionnai
re

48
Impact of divorce on children
As a result of this program, to what extent do you
understand the following about children and divorce:
Not well Somewhat Very well Already knew
a. Stages of grief 1 2 3 4
b. Self-blame or guilt 1 2 3 4
c. The desire for
parents to reunite
1 2 3 4
Questionnai
re

49
Communication skills
List three communications techniques you learned in
this course that you have used with your children:
1.________________________________
2.________________________________
3.________________________________
Questionnai
re

50
As a result of this course, to what extent do you feel
that your attitude has changed about:
a. Discussing your children with your ex
not at all / somewhat / a great deal
b. Allowing your former in-laws to see your children
not at all / somewhat / a great deal
Questionnai
re

51
How visitation disputes are handled
1. Describe how you and your ex-spouse handled
visitation disagreements before the course.
2. Describe how you and your ex-spouse have handled
visitation disagreements
since the workshop.
Questionnai
re

52
Demographic characteristics − age, education,
occupation, or income
• Where do you currently live?
• How many children do you have?
• What is your age?
• How many years have you been employed at your
current job?
Questionnai
re

53
• Open-ended questions − allow respondents to
provide their own answers
• Closed-ended questions − list answers and
respondents select either one or multiple
responses
Questionnai
re

54
• Do not provide any specific responses from which
the participant would choose.
• Allow respondents to express their own ideas and
opinions.
Questionnai
re

55
Pros:
• Can get
unintended or
unanticipated
results
• Wide variety of
answers
• Answers in
participants‟
“voices”
Cons:
• More difficult to
answer
• May be harder to
categorize for
interpretation
• More difficult for
people who don‟t
write much
Questionnai
re

56
Examples:
What communication skills did you
learn in this workshop that you will
use with your children?
What benefits do you receive from
this organization?
Questionnai
re

57
• Provide specific answers from
which the participant must
choose.
• Sometimes called “forced
choice.”
• Response possibilities include:
one best answer, multiple
responses, rating, or ranking
scale.
Questionnai
re

58
Pros:
• Easy to analyze
responses
• Stimulates recall
Cons:
• Chance of none of
the choices being
appropriate
• Biases response to
what you‟re looking
for
• Misses unintended
outcomes
Questionnai
re

59
Example − one best answer:
What does the word “nutrition” mean to
you? (Circle one number.)
1 Getting enough vitamins
2 The food you eat and how your body
uses it
3 Having to eat foods I don‟t like
4 Having good health
Questionnai
re

60
Example − multiple responses:
Of the communication skills taught in
this workshop, which will you use
with your children? (Check all that
apply.)
___active listening
___acknowledge feelings
___ask more open-ended questions
___provide one-on-one time for discussion
___negotiation
___other_____________________Questionnai
re

61
Example − rating scale
To what extent do you agree or disagree with the
new Speeding Monitoring System „‟ SAHER”” ?
(Circle one.)
1 Strongly disagree
2 Mildly disagree
3 Neither agree or disagree
4 Mildly agree
5 Strongly agree
Questionnai
re

62
• The particular people for whom the
questionnaire is being designed
• The particular purpose of the
questionnaire
• How questions will be placed in
relation to each other in the
questionnaire
Questionnai
re

63
• Match vocabulary and reading skills of
your respondents.
• Are any words confusing?
• Do any words have a double meaning?
• Avoid the use of abbreviations and
jargon.
Questionnai
re

64
• Avoid jargon or technical language.
Jargon:
What kind of post-litigation concerns have
you and your ex-spouse had?
Better:
Since having your visitation rights set by a
judge, what other concerns have you and
your ex-spouse had about visitation?
Questionnai
re

65
• Avoid vague questions and answers.
• Avoid ambiguous words or phrases.
• Avoid questions that may be too
specific.
• Avoid making assumptions.
Questionnai
re

66
Vague:
How will this seminar
help you?
Questionnai
re

67
Poor spacing and
logic:
Children‟s Ages
0−1
1−3
3−6
7−12
13−18
Better spacing, logic, and
mutually exclusive:
Children‟s Ages
under 1 year of age
1−3 years of age
4−6 years of age
7−9 years of age
10−12 years of age
Questionnai
re

68
Vague:
How often did you attend a workshop for self-
improvement during the past year?
a. Never
b. Rarely
c. Several times
d. Many times
Questionnai
re

69
Better:
How often did you attend a workshop for self-
improvement during the past year?
a. Not at all
b. One to two times
c. Three to five times
d. More than five times
Questionnai
re

70
• Ordered options to gauge difference of
opinion.
• Keep the order of choices the same
throughout the form.
• Odd number of options allows people to
select a middle option.
• Even number forces respondents to
take sides.
• Simpler is better.
Questionnai
re

71
Category scales
Numeric scales
Semantic differentials
Questionnai
re

72
• Use words or phrases to express a range of
choices.
• The number of categories depends on the amount
of differentiation.
• Three, four, or five categories are most common.
Questionnai
re

73
• Balance the scale with an equal number of
positive and negative options.
• “No opinion” or “uncertain” are not part of a scale.
They are usually placed off to the side or in a
separate column.
• All choices should refer to the same
thing/concept.
Questionnai
re

74
Poor:
__Not worth my time
__Slightly interested
__Moderately
interested
__Very interested
Better:
__Not at all
interested
__Slightly interested
__Moderately
interested
__Very interested
Left column includes two concepts –
“worth” and “interest level.”
Questionnai
re

75
Not much
Some
A great deal
A little
Some
A lot
Not much
Little
Somewhat
Much
A great deal
Questionnai
re

76
Never
Seldom
Often
Always
Extremely poor
Below average
Average
Above average
Excellent
Questionnai
re

77
Strongly
disagree
Disagree
Agree
Strongly
agree
Uncertain
Disagree
Neither agree
nor
disagree
Agree
Completely disagree
Mostly disagree
Slightly disagree
Slightly agree
Mostly agree
Completely agree
Questionnai
re

78
• Overall appearance
• Length of the questionnaire
• Order of questions
• Demographic data collection
Questionnai
re

79
• Use an easy-to-read typeface.
• Leave plenty of white space.
• Separate different components of a
questionnaire by using different type styles.
• Use arrows to show respondents where to go.
Questionnai
re

80
• Start with the easiest questions −
avoid controversial topics.
• Address important topics early.
• Move from specific questions to
general questions.
• Move from closed-ended to open-
ended questions.
Questionnai
re

81
• Only include questions about
demographic data that you will use.
• You may want to preface demographic
questions with the purpose for
collecting the information.
• You may need to state that providing
this information is optional and/or
explain how it affects program
eligibility.
Questionnai
re

82
Age
Gender
Ethnicity
Marital status
Family size
Occupation
Education
Employment status
Residence
Previous contact with
organization
Prior knowledge of
topic
First-time participant
vs. repeats
How you learned
about the program
Questionnai
re

83
ALWAYS
ALWAYS
ALWAYS
Questionnai
re

84
• Does each question measure what it is
supposed to measure?
• Are all the words understood?
• Are questions interpreted in the same
way by all respondents?
• Are all response options appropriate?
• Is there an answer that applies to each
respondent?
Questionnai
re

85
• Are the answers respondents can choose
from correct? Are some responses missing?
• Does the questionnaire create a positive
impression - does it motivate people to
answer it?
• Does any aspect of the questionnaire
suggest bias?
• Do respondents follow the directions?
• Is the cover letter clear?
Questionnai
re

86
1. Select reviewers who are similar to
the respondents and who will be
critical.
(Also ask your colleagues to review
it.)
2. Ask them to complete the
questionnaire as if it were “for real.”
3. Obtain feedback on the form and
content of the questionnaire and the
cover letter. Was anything confusing,
difficult to answer, de-motivating?Questionnai
re

87
4. Assess whether the questions
produce the information you need.
5. Try the tabulation and analysis
procedures.
6. Revise.
7. If necessary, repeat these steps to
pre-test the revised version.
Questionnai
re

88
• A quality questionnaire is almost
never written in one sitting.
• A quality questionnaire goes through
multiple revisions (maybe a dozen!)
before it is ready.
• Remember – a list of questions is
just the starting point. There are
many factors that affect response.
Questionnai
re

89
When will data be collected?
• Before and after the program
• At one time
• At various times during the course of the
program
• Continuously through the program
• Over time − longitudinally
Questionnai
re

90
• Purpose and importance of the
survey
• Survey sponsor − use letterhead
• Why the respondent was selected to
participate
• Benefit(s) of completing survey
• Assurance of anonymity or
confidentiality
Questionnai
re

91
• How results will be used
• Instruction for returning the
survey
• When to respond
• How to obtain survey results
• Contact information
Questionnai
re

92
• Personalize the letter in salutation or
signature
• Hand-sign the letter
• Express appreciation for their
participation
• Include pre-addressed, stamped
return envelope
Questionnai
re

Questionnaire design & basic of survey

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Questionnaire design & basic of survey (20)

More from Kaimrc_Rss_Jd (20)

Recently uploaded (20)

Questionnaire design & basic of survey