How to design_a_questionnaire
0 likes191 views
This document provides guidance on designing questionnaires for surveys. It discusses important considerations for questionnaire design such as determining what questions to ask, wording questions precisely and avoiding bias, response formats, length, question order, and piloting the questionnaire. The key steps in designing a questionnaire are to maximize the response rate, obtain accurate relevant information, and pretest the questionnaire on a small sample to refine it.
![STUDENTBMJ VOLUME 9 JUNE 2001 studentbmj.com188
Education
“Small group teaching should not be
abolished,” the statement should be
rephrased as, “Small group teaching
should continue.” Double negatives
should always be avoided.
Ask precise questions
Questions may be ambiguous because a
word or term may have a different mean-
ing. For example, if we ask students to
rate their interest in “medicine,” this term
might mean “general medicine” (as
opposed to general surgery) to some, but
inclusive of all clinical specialties (as
opposed to professions outside medicine)
to others.
Another source of ambiguity is a fail-
ure to specify a frame of reference. For
example, in the question, “How often did
you borrow books from your library?” the
time reference is missing. It might be
rephrased as, “How many books have you
borrowed from the library within the past
six months altogether?”
Ensure those you ask have the
necessary knowledge
For example, in a survey of university lec-
turers on recent changes in higher edu-
cation, the question, “Do you agree with
the recommendations in the Dearing
report on higher education?” is unsatis-
factory for several reasons. Not only does
it ask for several pieces of information at
the same time as there are several recom-
mendations in the report, the question
also assumes that all lecturers know about
the relevant recommendations.
Level of details
It is important to ask for the exact level of
details required. On the one hand, you
might not be able to fulfil the purposes of
the survey if you omit to ask essential
details. On the other hand, it is important
to avoid unnecessary details. People are
less inclined to complete long question-
naires. This is particularly important for
confidential sensitive information, such
as personal financial matters or marital
relationship issues.
Sensitive issues
It is often difficult to obtain truthful
answers to sensitive questions. Clearly,
the question, “Have you ever copied
other students’ answers in a degree
exam?” is likely to produce either no
response or negative responses. Less
direct approaches have been suggested.2
Firstly, the casual approach: “By the way,
do you happen to have copied other stu-
dents’ answers in a degree exam?” may be
used as a last part of another decoy ques-
tion. Secondly, the numbered card
approach: “Please tick one or more of the
following items which correspond to how
you have answered degree examination
questions in the past.” In the list of items,
include “copy from other students” as
one of many items. Thirdly, the everybody
approach: “As we all know, most medical
students have copied other students’
answers in degree exams. Do you happen
to be one of them?” Fourthly, other people
approach. This approach was used in the
recent medical student survey.3
In this
survey, students were given the scenario,
“John copies answers in a degree exam
from Jean.” They were then asked, “Do
you feel John is wrong, what penalty
should be imposed for John, and have
you done or would you consider doing
the above?”
Minimise bias
People tend to answer questions in a way
they perceive to be socially desired or
expected by the questioner and they
often look for clues in the questions.
Many apparently neutral questions can
potentially lead to bias. For example, in
the question, “Within the past month,
how many lectures have you missed due
to your evening job?” students may per-
ceive the desired responses to be “never”
to the first question. This question could
be rephrased as, “Within the past month,
how many times did your evening job
commitment clash with lectures? How
many times did you give priority to your
evening job?”
Take another example. The question,
“Please rate how useful the following text-
Box 2: Types of closed (forced choice) format
Choice of categories
For example, “What is your marital status?”
[ ] Single
[ ] Married
[ ] Divorced
[ ] Widowed
Likert style scale
For example, “Statistics is an interesting subject”
Strongly disagree Disagree Cannot decide Agree Strongly agree
Differential scales
For example, “How would you rate the presentation?”
Extremely interesting 1 2 3 4 5 6 7 8 9 10 Extremely dull
It may also be diagrammatic:
Extremely interesting Extremely dull
Checklists
For example, “Circle the clinical specialties you are particularly interested in”
General medicine Obstetrics and gynaecology
General surgery Orthopaedics
Ophthalmology Accident and emergency
Paediatrics General practice
Ranking
For example, “Please rank your interests in the following specialties”
(1= most interesting, 8=least interesting)
General medicine Obstetrics and gynaecology
General surgery Orthopaedics
Ophthalmology Accident and emergency
Paediatrics General practice](https://image.slidesharecdn.com/howtodesignaquestionnaire-161108132937/85/How-to-design_a_questionnaire-2-320.jpg)
How to design_a_questionnaire
- 1. STUDENTBMJ VOLUME 9 JUNE 2001 studentbmj.com 187 Education As discussed in a previous issue a survey involves directly collecting information from people (or sometimes organisa- tions) whom we are interested in.1 The types of information will take account of the people’s or organisations’ level of knowledge, attitude, personalities, beliefs, or preferences. Questionnaires are wide- ly used to collect such information. Well designed questionnaires are highly struc- tured to allow the same types of informa- tion to be collected from a large number of people in the same way and for data to be analysed quantitatively and systemati- cally. Questionnaires are best used for collecting factual data and appropriate questionnaire design is essential to ensure that we obtain valid responses to our questions. Objectives in designing questionnaires There are two main objectives in design- ing a questionnaire: q To maximise the proportion of subjects answering our questionnaire—that is, the response rate. q To obtain accurate relevant informa- tion for our survey. To maximise our response rate, we have to consider carefully how we administer the questionnaire, establish rapport, explain the purpose of the survey, and remind those who have not responded. The length of the questionnaire should be appropriate. In order to obtain accurate relevant information, we have to give some thought to what questions we ask, how we ask them, the order we ask them in, and the general layout of the questionnaire. Deciding what to ask As discussed in last month’s issue, there are three potential types of information: q Information we are primarily interested in—that is, dependent variables. q Information which might explain the dependent variables—that is, independ- ent variables. q Other factors related to both depend- ent and independent factors which may distort the results and have to be adjusted for—that is, confounding vari- ables. Let us take as an example a national survey to find out students’ factors pre- dicting the level of certain knowledge, skills, and attitudes at the end of their undergraduate medical course. The dependent factors include the students’ level of relevant knowledge, skills, and attitudes. The independent factors might include students’ learning styles, GCSE and A level grades, socioeconomic status, ethnicity, etc. Confounding variables might include the types and quality of teaching in each medical school. Sometimes, additional questions are used to detect the consistency of the sub- ject’s responses. For example, there may be a tendency for some to tick either “agree” or “disagree” to all the questions. Additional contradictory statements may be used to detect such tendencies. Wording of individual questions The way questions are phrased is impor- tant and there are some general rules for constructing good questions in a ques- tionnaire. Use short and simple sentences Short, simple sentences are generally less confusing and ambiguous than long, complex ones. As a rule of thumb, most sentences should contain one or two clauses. Sentences with more than three clauses should be rephrased. Ask for only one piece of information at a time For example, “Please rate the lecture in terms of its content and presentation” asks for two pieces of information at the same time. It should be divided into two parts: “Please rate the lecture in terms of (a) its content, (b) its presentation.” Avoid negatives if possible Negatives should be used only sparingly. For example, instead of asking students whether they agree with the statement, How to design a questionnaire Wai-Ching Leung has some practical advice on questionnaires Box 1: Advantages of open or closed format Open format Allows exploration of the range of possi- ble themes arising from an issue Can be used even if a comprehensive range of alternative choices cannot be compiled Closed—that is, forced choice—format Easy and quick to fill in Minimise discrimination against the less literate (in self administered question- naire) or the less articulate (in interview questionnaire) Easy to code, record, and analyse results quantitatively Easy to report results
- 2. STUDENTBMJ VOLUME 9 JUNE 2001 studentbmj.com188 Education “Small group teaching should not be abolished,” the statement should be rephrased as, “Small group teaching should continue.” Double negatives should always be avoided. Ask precise questions Questions may be ambiguous because a word or term may have a different mean- ing. For example, if we ask students to rate their interest in “medicine,” this term might mean “general medicine” (as opposed to general surgery) to some, but inclusive of all clinical specialties (as opposed to professions outside medicine) to others. Another source of ambiguity is a fail- ure to specify a frame of reference. For example, in the question, “How often did you borrow books from your library?” the time reference is missing. It might be rephrased as, “How many books have you borrowed from the library within the past six months altogether?” Ensure those you ask have the necessary knowledge For example, in a survey of university lec- turers on recent changes in higher edu- cation, the question, “Do you agree with the recommendations in the Dearing report on higher education?” is unsatis- factory for several reasons. Not only does it ask for several pieces of information at the same time as there are several recom- mendations in the report, the question also assumes that all lecturers know about the relevant recommendations. Level of details It is important to ask for the exact level of details required. On the one hand, you might not be able to fulfil the purposes of the survey if you omit to ask essential details. On the other hand, it is important to avoid unnecessary details. People are less inclined to complete long question- naires. This is particularly important for confidential sensitive information, such as personal financial matters or marital relationship issues. Sensitive issues It is often difficult to obtain truthful answers to sensitive questions. Clearly, the question, “Have you ever copied other students’ answers in a degree exam?” is likely to produce either no response or negative responses. Less direct approaches have been suggested.2 Firstly, the casual approach: “By the way, do you happen to have copied other stu- dents’ answers in a degree exam?” may be used as a last part of another decoy ques- tion. Secondly, the numbered card approach: “Please tick one or more of the following items which correspond to how you have answered degree examination questions in the past.” In the list of items, include “copy from other students” as one of many items. Thirdly, the everybody approach: “As we all know, most medical students have copied other students’ answers in degree exams. Do you happen to be one of them?” Fourthly, other people approach. This approach was used in the recent medical student survey.3 In this survey, students were given the scenario, “John copies answers in a degree exam from Jean.” They were then asked, “Do you feel John is wrong, what penalty should be imposed for John, and have you done or would you consider doing the above?” Minimise bias People tend to answer questions in a way they perceive to be socially desired or expected by the questioner and they often look for clues in the questions. Many apparently neutral questions can potentially lead to bias. For example, in the question, “Within the past month, how many lectures have you missed due to your evening job?” students may per- ceive the desired responses to be “never” to the first question. This question could be rephrased as, “Within the past month, how many times did your evening job commitment clash with lectures? How many times did you give priority to your evening job?” Take another example. The question, “Please rate how useful the following text- Box 2: Types of closed (forced choice) format Choice of categories For example, “What is your marital status?” [ ] Single [ ] Married [ ] Divorced [ ] Widowed Likert style scale For example, “Statistics is an interesting subject” Strongly disagree Disagree Cannot decide Agree Strongly agree Differential scales For example, “How would you rate the presentation?” Extremely interesting 1 2 3 4 5 6 7 8 9 10 Extremely dull It may also be diagrammatic: Extremely interesting Extremely dull Checklists For example, “Circle the clinical specialties you are particularly interested in” General medicine Obstetrics and gynaecology General surgery Orthopaedics Ophthalmology Accident and emergency Paediatrics General practice Ranking For example, “Please rank your interests in the following specialties” (1= most interesting, 8=least interesting) General medicine Obstetrics and gynaecology General surgery Orthopaedics Ophthalmology Accident and emergency Paediatrics General practice
- 3. STUDENTBMJ VOLUME 9 JUNE 2001 studentbmj.com 189 Education books are. Please also state whether they are included in your lecturer’s recom- mended reading list?” There is a risk that the students may perceive that they should rate books recommended by lec- turers more favourably than those not recommended by their lecturers. This risk may be minimised by putting the second question later on in the questionnaire. Format of responses The responses can be in open or closed formats. In an open ended question, the respondents can formulate their own answers. In closed format, respondents are forced to choose between several given options. The advantages of each of these formats are shown in box 1. It is possible to use a mixture of the two for- mats—for example, give a list of options, with the final option of “other” followed by a space for respondents to fill in other alternatives. There are several forced choice for- mats. These are shown in box 2. Out of these formats, ranking is probably least frequently used, as the responses are rela- tively difficult to record and analyse. Length of questionnaire There are no universal agreements about the optimal length of questionnaires. It probably depends on the type of respon- dents. However, short simple question- naires usually attract higher response rates than long complex ones. In a BMJ survey of stroke survivors both the response rate and the proportion of com- pleted forms were higher for a shorter questionnaire (six questions with a visual analogue scale) compared with a longer and more complex questionnaire (with 34 questions).4 Arranging the questions The order of the questions is also impor- tant. Some general rules are: q Go from general to particular. q Go from easy to difficult. q Go from factual to abstract. q Start with closed format questions. q Start with questions relevant to the main subject. q Do not start with demographic and personal questions. It is useful to use a variety of question format as shown in box 2 to maintain the respondents’ interest. When a series of semantic differential scales are used, it may be a good idea to mix positive nega- tive—for example, interesting to dull—with negative positive—for example, useless to useful—scales. This might make the respondents think more and avoid the tendency to tick the same response for every question. Filter questions are useful to ensure that respondents answer only relevant questions. However, avoid a highly com- plex filter regime. Make good use of arrows and boxes to clarify the filter regime. Introduction, personalised letter, and ending It seems a good idea to have either a per- sonalised covering letter or at least an introduction explaining briefly the pur- pose of the survey, the importance of the respondents’ participation, who is responsible for the survey, and a state- ment guaranteeing confidentiality.5 A personalised letter can be easily generat- ed using mail-merge on a word proces- sor. It is also important to thank the respondent at the end of the question- naire. How to administer the questionnaires There are several ways of administering questionnaires. They may be self admin- istered or read out by interviewers. Self administered questionnaires may be sent by post, email, or electronically online. Interview administered questionnaires may be by telephone or face to face. Advantages of self administered questionnaires include: q Cheap and easy to administer. q Preserve confidentiality. q Can be completed at respondent’s con- venience. q Can be administered in a standard manner. Advantages of interview administered questionnaires include: q Allow participation by illiterate people. q Allow clarification of ambiguity. The exact method of administration also depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews. Piloting and evaluation of questionnaires Given the complexity of designing a questionnaire, it is impossible even for the experts to get it right the first time round. Questionnaires must be pretest- ed—that is, piloted—on a small sample of people characteristic of those in the sur- vey. In a small survey, there might be only pretesting of the drafted questionnaire. In a large survey, there may be three phases of piloting. In the first phase we might ask each respondent in great detail about a limited number of questions: effects of different wordings, what they have in mind when they give a particular answer, how they understand a particular word, etc. In the second phase the whole questionnaire is administered by inter- viewers. Analysis of the responses and the interviewers’ comments are used to improve the questionnaire. Ideally, there should be sufficient variations in respons- es among respondents; each question should measure different qualities—that is, the responses between any two items should not be very strongly correlated— and the non-response rate should be low. In the third phase the pilot test is pol- ished to improve the question order, filter questions, and layout. Conclusions Questionnaires must be carefully designed to yield valid information. Meticulous attention must be paid to ensure that individual questions are rele- vant, appropriate, intelligible, precise, and unbiased. The order of the questions must be carefully arranged, and the layout of the questionnaire must be clear. It is wise to draft a clear personalised covering let- ter. Questionnaires must first be piloted and evaluated before the actual survey. Further reading: Abramson JH, Abramson ZH. Survey methods in community medicine. Edinburgh: Churchill Livingstone, 1999. Bowling A. Research methods in health: investigating health and health services. Buckingham: Open University Press, 1997. Wai-Ching Leung lecturer in public health medicine, University of East Anglia w-c.leung@uea.ac.uk 1 Leung WC. How to conduct a survey. StudentBMJ 2001;9:143-5. (May.) 2 Barton JA. Asking the embarrassing question. Public Opinion Quarterly 1958;22:67-8. 3 Rennie SC, Crosby JR. Are “tomorrow’s doctors” hon- est? Questionnaire study exploring medical students’ attitudes and reported behaviour in academic miscon- duct. studentBMJ 2001;9:67-8.(March.) 4 Dorman PJ, Slattery J, Farrell B, Dennis MS, Sandercock PAG. A randomised comparison of the EuroQol and Short Form-36 after stroke. BMJ 1997;315:461. 5 Bissett AF. Designing a questionnaire: Send a person- al covering letter. BMJ 1994;308:202-3.