Survey design workshop

1. Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 1 February, 2011

2. Outline Objectives

3. Introductions

4. Logins & Resources

5. Research methods

6. Questionnaire design

7. Levels of measurement

8. Sampling

9. Evaluation

10. Objective 1 Understand the importance of a rigorous , step-by-step process in planning, developing & implementing research questionnaires

11. Objective 2 Consider the pros and cons of common methods for survey administration Face-to-face interview

12. Telephone survey

13. Mail survey

14. Internet/mobile survey

15. Objective 3 Examine nuts & bolts of questionnaire design e.g., Question style,

16. Response formats,

17. Layout, and

18. Pre- and pilot testing

19. Objective 4 Consider implementation issues Sampling methods

20. Sample size

21. Objective 5 Critically review example surveys. Existing examples

22. Student in-progress examples with a view towards planning, drafting, and/or revising of an initial draft (pilot) survey.

23. Resources Survey Design Workshop Notes (Wikiversity)

24. Readings

25. Books about surveys design and survey research (check library)

26. Introductions Introductions

27. Types of Research ( Research Methods ) There are 3 main research methods: Experimental

28. Quasi-experimental

29. Non-experimental

30. Types of Research - Experimental Characterised by: Random assignment

31. Control over extraneous variables

32. Types of Research - Quasi-experimental Characterised by: Non-random assignment

33. Control over some extraneous variables

34. Groups are “naturally occuring”

35. Types of Research - Non-experimental Characterised by: No “groups” or “conditions” are created or used (i.e., no full experimental or quasi-experimental groups)

36. Minimal control over extraneous variables

37. Survey Research Characteristics Surveys are widely used in non-experimental social science research.

38. Often use interviews or questionnaires .

39. Involve real-world samples .

40. Often quantitative , but can be qualitative .

41. History of Survey Research Survey research methodology was initially developed in the 1940's – 1960's.

42. Since the 1980's, theories and principles evolved to create a unified perspective on the design, conduct, and evaluation of surveys.

43. 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4) Systematic : follows a specific set of rules, a formal and orderly logic of operations

44. Impartial : selects units of the population without prejudice or preference

45. Representative : includes units that together are representative of the problem under study and the population affected by it

46. Theory-based : operations are guided by relevant principles of human behaviour and by mathematical laws of probability (chance).

47. Quantitative : assigns numerical values to nonnumerical character

48. Self-monitoring : procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

49. Contemporary : it is current, more than historical, fact-finding

50. Replicable : other people using the same methods in the same ways can get essentially the same results 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

51. Advantages of Survey-Based Research Ecological validity

52. Access to wide range of participants

53. Potentially large amounts of data

54. May be more ethical (than experiments)

55. Disadvantages of Survey-Based Research Lack of control -> less internal validity

56. Data may be 'superficial'

57. Can be costly to obtain representative data

58. Self-report data only

59. Potentially low compliance rates

60. Example Surveys Unit Satisfaction Survey Community Library Use

61. The research process 1. Establish need for info/ research 2. Problem definition/ Hypotheses 3. Research design 4. Sampling/ Data collection 5. Data analysis 6. Reporting

62. The research process

63. Survey construction: Overview What is a survey?

64. Types of questionnaires

65. Questionnaire development

66. Writing questions

67. Types of questions

68. Response formats  LOM

69. Survey formatting

70. A standardised stimulus

71. A measuring instrument

72. A way of converting fuzzy psychological stuff into hard data for analysis What is a survey?

73. Types of surveys Types of surveys Self - administered Interview - administered Postal Delivered and collected Telephone Face to face structured interview Web-based

74. Questionnaire development 1. Formulate generic questionnaire 2. Expand the questionnaire Turn into separate sections based on study objectives. Draft qs & response formats 4. Finalise questionnaire & implement Question order & funnel qs 3. Pre-test, pilot test, & redraft

75. Formulate Generic Questionnaire Turn objectives into sections of the survey

76. Ensure all questions relate to research objectives

77. For explanatory objectives or hypotheses ensure both dependent and independent variables exist

78. Cover Letter & Ethics Statement Introduction or cover letter: Outline details of research project and allow for ethical informed consent.

79. Few will read it without good prompting and being easy-to-read

80. Instructions Provides consistency - helps to ensure standard conditions across different administrations

81. Explain how to do the survey in a user-friendly manner

82. Example: Life Effectiveness Questionnaire

83. Instructions: Example

84. Cover letter / ethics statement checklist Outline details of research project e.g.,: Who are you? Are you bona fide?

85. Purpose of survey?

86. What's involved?

87. Explain any risks/costs/rewards

88. How will results be used?

89. Human ethics approval #

90. How is consent given / not given?

91. Voluntary - can choose not to continue anytime

92. More info: Complaints, how to obtain results, contact details etc.

93. Font (type, size)

94. No. of pages

95. Margins

96. Double vs. single-siding

97. Colour, etc. Layout

98. Demographics single section, usually at beginning or end of questionnaire

99. only include relevant questions Layout

100. Space for comments?

101. Indicate the end

102. Say thanks! Layout

103. Flow/Structure Logical order of questions (use sections)

104. Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)

105. Use funneling/branching questions to move respondents through survey

106. Start off with easy to answer and engaging questions

107. More controversial questions in middle section

108. Expanding the Survey

109. Personal Information Generally, researchers put personal information at beginning of survey (if required). However, this may put off respondents, so also consider uncluding at towards end.

110. Consider response format e.g, Income in categories or ranges

111. Personal Information More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone

112. Show cards for responses may help for face to face interviews

113. Pre- & pilot-testing Pre-test items on convenient others - ask for feedback

114. Revise items e.g., Which don’t apply to everybody

115. Are redundant

116. Are misunderstood

117. Are non-completed Reconsider ordering & layout

118. Pilot test on a small sample from the target population, analyse, & revise

119. Types of Questions Be able to justify and defend your choices ...

120. Writing questions - Dos Define target constructs

121. Check related research & questionnaires

122. Draft items (for important, fuzzy constructs aim to have multiple indicators)

123. Pre-test & revise

124. Writing questions - Dos Focus directly on topic/issue

125. Be clear

126. Be brief

127. Avoid big words

128. Use simple and correct grammar

129. Writing questions – Don'ts Inapplicable – must apply to all respondents Over-demanding – e.g., recall of detail or time-consuming, unnecessary questioning Ambiguous – meaning must be clear to all respondents Double negative – e.g., Do you not disapprove of tax reforms?

130. Writing questions – Don'ts Double-barrelled - e.g., “Do you think speed limits should be lowered for cars & trucks?” Leading - e.g., “don’t you see some danger in the new policy?” Loaded – e.g., “Do you advocate a lower speed limit to save human lives?” vs “Does traffic safety require a lower speed limit?”

131. Response biases Social desirability

132. Acquiescence yea- and nay-saying Self-serving bias

133. Order effects

134. Demand characteristics Interview High demand characteristics

135. Can elicit richer information Questionnaire Lower demand characteristics

136. Information may be less rich

137. Types of Questions

138. Accuracy of recall decreases over time

139. Objective questions A verifiably true answer (i.e., factual information) exists for each unit.

140. The question could be accurately answered by an observer. e.g., How times during 2009 did you visit a G.P.? ______

141. Subjective questions Asks about fuzzy personal perceptions.

142. There is no “true”, factual answer.

143. Many possible answers per unit.

144. Can't be accurately answered by an observer. e.g., Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received? perfectly very well reasonably poorly not at all

145. Objective vs. subjective questions Both types of questions may be appropriate; depends on the purpose of the study.

146. One criticism of this distinction: There is no such thing as “objective” and that all responses are subjective.

147. Open-ended Questions Rich information can be gathered

148. Useful for descriptive, exploratory work

149. Difficult and subjective to analyse

150. Time consuming

151. Open-ended questions Rich information can be gathered

152. Useful for descriptive, exploratory work

153. Difficult and subjective to analyse

154. Time consuming

155. Open-ended questions: Examples What are the main issues you are currently facing in your life? How many hours did you spend doing university study last week? _________

156. Closed-ended questions Important information may be lost forever

157. Useful for hypothesis testing

158. Easy and objective to analyse

159. Time efficient

160. Levels of Measurement = Type of Data Stevens (1946)

161. Levels of measurement N ominal / Categorical

162. O rdinal

163. I nterval

164. R atio

165. Discrete vs. continuous Discrete - - - - - - - - - - Continuous ___________

166. Each level has the properties of the preceding levels, plus something more!

167. Categorical / nominal Conveys a category label

168. (Arbitrary) assignment of #s to categories e.g. Gender No useful information, except as labels

169. Categorical / nomimal example: Phrenological labels

170. Ordinal / ranked scale Conveys order , but not distance e.g. in a race, 1st, 2nd, 3rd, etc. or ranking of favourites or preferences

171. Ordinal / ranked example: Ranked importance Rank the following aspects of the university according to what is most important to you (1 = most important through to 5 = least important)  Quality of the teaching and education  Quality of the social life  Quality of the campus  Quality of the administration  Quality of the university's reputation

172. Interval scale Conveys order & distance

173. 0 is arbitrary e.g., temperature (degrees C) Usually treat as continuous for > 5 intervals

174. Interval example: 8 point Likert scale

175. Ratio scale Conveys order & distance

176. Continuous, with a meaningful 0 point e.g. height, age, weight, time, number of times an event has occurred Ratio statements can be made e.g. X is twice as old (or high or heavy) as Y

177. Ratio scale: Time

178. Why do levels of measurement matter? Different analytical procedures are used for different levels of data. More powerful statistics can be applied to higher levels

179. Levels of measurement: Revision question Fill in all cells Descriptive Statistics Ratio Interval Ordinal / Rank Nominal / Categorical Graphs Examples Prop-erties Level

180. Closed-ended rating scales Dichotomous

181. Multichotomous

182. Verbal frquency scale

183. The list (multiple response)

184. Ranking

185. Likert scale

186. Graphical rating scale

187. Semantic differential

188. Non-verbal (idiographic)

189. Dichotomous 2 response options e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No Provides the simplest type of quantification

190. Multichotomous How many hours did you spend doing university study last week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours

191. Multichotomous More than two possible answers e.g., What type of attractions in your current trip to Canberra most appeal to you? __ historic buildings __ museum/art galleries __ parks and gardens

192. Verbal frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

193. The list (multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical

194. The list (multiple-response) What are the main issues that you are currently facing in your life? (tick all that apply) __ financial __ physical / health __ academic __ employment / unemployment __ relationships __ other (please specify)

195. Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for visiting Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing

196. Likert scale Measures strength of feeling or perception. Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you) 1 2 3 4 5 strongly disagree neutral agree strongly disagree agree 1 2 3 4 5 strongly agree neutral disagree strongly agree disagree

197. Graphical rating scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

198. What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active Semantic differential

199. Non-verbal scale Point to the face that shows how you feel about what happened to the toy. Also called an idiographic scale .

200. Verbal frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

201. Sensitivity & reliability Scale should be sensitive yet reliable.

202. Watch out for too few or too many options.

203. How many response options? Minimum = 2

204. Average = 3 to 9

205. Maximum = 10? Basic guide: 7 +/- 2 Number of response options?

206. AGREEMENT ABOUT SOMETHING 2-Categories DISAGREE AGREE 3-Categories DISAGREE NEUTRAL AGREE 4-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE AGREE AGREE 5-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE NEUTRAL AGREE AGREE Number of response options? Likert scale example

207. Watch out for too many or too few response options “ Capital punishment should be reintroduced for serious crimes ” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

208. Example: How could this question be improved? How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over

209. Are you satisfied with your marriage and your job? __________________________ Example: How could this question be improved?

210. You didn’t think the food was very good, did you? _____ Yes _____ No Example: How could this question be improved?

211. Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No Example: How could this question be improved?

212. What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth Example: How could this question be improved?

213. Comparison of data collection methods Alreck & Settle (1995; 32) Alreck and Settle (1995:32)

214. Finalise Questionnaire Draft Questions need to be exhaustive and mutually exclusive Include ‘other (please specify)’

215. Ensure categories do not overlap

216. Finalise Questionnaire Draft Length Try to keep them as short as possible

217. Only ask questions that relate to objectives

218. Tricks? Font size/double sided photocopying/numbering sections

219. Maximising Response Rate Layout and design is key

220. Respondent’s level of interest

221. Colour of paper

222. Accompanying letter / introduction

223. Mail surveys - self-addressed stamped return envelope

224. Rewards

225. Reminders or follow up calls

226. Pre-testing and Pilot testing Pre-test – try out on convenient others & revise

227. Pilot test – try out on a small sample from the target population & revise

228. Be assertive and interactive about seeking feedback – ask questions & observe

229. “The customer is always right.”

230. Pre-test & Revise Pre-test items and ask for feedback

231. Revise: items which don’t apply to everybody

232. redundancy

233. skewed response items

234. misinterpreted items

235. non-completed items Reconsider ordering & layout

236. LUNCH BREAK

237. Survey Design Critique In pairs, look through the example questionnaires, and highlight aspects which: could be improved

238. are particularly good

239. you would like to ask about

240. Examples Examine questionnaire examples

241. Examine structure, design issues and question styles

242. Note cover page and details provided

243. Sampling

244. Sampling: Overview Sampling terminology

245. What is sampling?

246. Why sample?

247. Sampling methods

248. Example: Shere Hite’s survey

249. Sampling terminology Target population To whom you wish to generalise Sampling frame Those who have a chance to be selected Sample Those who were selected and responsed Representativeness The extent to which the sample is a good indicator of the target population

250. What is sampling? “Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim (2006)

251. Why sample? Reduces cost, time, sample size etc.

252. If the sample is representative, the sample data allows inferences to be drawn about the total population.

253. Representativeness of a sample depends on: Adequacy of sampling frame

254. Sampling method

255. Adequacy of sample size

256. Response rate – both the % & representativeness of people in sample who actually complete survey It is better to have a small, representative sample than a large, unrepresentative sample.

257. Sampling methods Probability sampling Random

258. Systematic

259. Cluster Multi-Stage Cluster Non-probability sampling Quota

260. Convenience

261. Snowball

262. Random/probability sampling Each unit has an equal chance of selection

263. Selection occurs entirely by random chance

264. Also called representative sampling

265. Simple random sampling Everyone in the target population has an equal chance of selection

266. Useful if clear study area or population is identified

267. Similar to a lottery: List of names are assigned #s and randomly select #s of respondents

268. Randomly select # through table of random #s or by computer

269. Systematic random sampling Selecting without first numbering

270. Respondents (units) selected from a list/file.

271. Useful when survey population is similar e.g. List of students

272. Select sample at regular intervals from the population e.g., every 5 th person on a list, starting at a random number between 1 and 5

273. Stratified random sampling Sub-divide population into strata (e.g., by gender, age, or location)

274. Then random selection from within each stratum

275. Improves representativeness

276. e.g., Telephone interviews using post-code strata

277. Non-random / non-probability Also called purposive or judgemental sampling

278. Useful for exploratory research and case study research

279. Able to get large sample size quickly

280. Limitations include potential bias and non-representativeness

281. Convenience sampling Sampling is by convenience rather than randomly

282. Due to time/financial constraints

283. e.g. surveying all those at a tourist attraction over one weekend

284. Purposive sampling Respondents selected for a particular purpose e.g., because they may be “typical” respondents e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra

285. e.g., Frequent flyers to contact regarding service quality in an airline setting

286. Snowball sampling Useful for difficult to access populations e.g., illegal immigratnts, drug users

287. Respondents recommend other respondents

288. e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

289. Sampling process Identify target population and sampling frame

290. Select sampling method

291. Calculate sample size for desired power.

292. Maximise return rate

293. Summary of sampling strategy Identify target population and sampling frame

294. Selection sampling method

295. Calculate required sample size

297. Sampling Example: Shere Hite ‘American Sexology’

298. Hite's survey of American male-female relations (early 1980's) Shere Hite ‘doyenne of sex polls’

299. Media furors & worldwide attention

300. 127-item questionnaire about marriage & relations between sexes

301. Sample: 4500 USA women, 14 to 85 years

302. Conclusion: Society and men need to change to improve lives of women

303. Some of Hite’s findings about American women.... Only 13% married for 2+years were still in love

304. 70% married for 5+ years were having affairs... usually more for 'emotional closeness’ than sex

305. 76% of these women did not feel guilty 87% had a closer female friend than husband

306. 98% wanted “basic changes” to love relationships

307. 84% were emotionally unsatisfied

308. 95% reported emotional & psychological harassment from their men

309. Some of the critical comments.... She goes in with prejudice & comes out with a statistic. The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes. Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.

310. Hite's response rate & selection bias 100,000 questionnaires were sent to a variety of women’s groups (feminist organisations, church groups, garden clubs etc.)

311. 4,500 replied (4.5% return rate)

312. “ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Hite's response rate & selection bias

313. Sample size – it's not how big, it's how representative

314. Objectivity – watch out for manipulating the survey questions and results interpretation to suit your personal conjectures Lessons from Hite's male-female relations survey

315. Measurement Error

316. Measurement error Observed score = true score + measurement error Measurement error = systematic error + random error Any deviation from the true value caused by the measurement procedure.

317. Sources of measurement error Non-sampling (e.g., unreliable or invalid tests) Sampling (e.g., non-rep. sample) Personal bias (e.g., researcher favours a hypothesis) Paradigm (e.g., Western focus on individualism)

318. To minimise measurement error Use well designed measures: Multiple indicators

319. Sensitive to target constructs

320. Clear wording on questions/instructions

321. Reduce demand effects: Train interviewers

322. Use standard protocol To minimise measurement error

323. To minimise measurement error Maximise response rate: Pre-survey contact

324. Minimise length / time / hassle

325. Offer rewards / incentives

326. Coloured paper

327. Call backs / reminders

328. Ensure administrative accuracy Set up efficient coding, with well-labelled variables

329. Check data To minimise measurement error

330. Summary of sampling strategy Identify target population and sampling frame

331. Selection sampling method

332. Calculate required sample size

334. Sampling Task A research project's aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”. What is the research population?

335. How might you get hold of a sample frame?

336. What sampling technique would you use?

337. Confidence Levels / Margins of Error Relates to representativeness of a sample of a target population - to what extent can we be confident about the results?

338. Gives the estimated range of values into which we expect other samples to fall say 95% of the time

339. Social sciences = at least 90% or above, preferably 95%

340. Example 95% Confidence Level Graphs

341. Example 95% Confidence Level Table

342. Confidence Level Example Survey of visitors to Canberra between June 1 - August 31, 2004 = 10,000 visitors Want to work with 95% confidence level and 3% margin of error How many to survey? Answer: Sample size = 964 people Q: What are your favourite attractions?

343. Following Results + / - 3% error @ 95% confidence level

344. Confidence Intervals / Margins of Error One time out of 20, we expect the answers may be greater than +/- 3% Establish the confidence level, margin of error you want to work with

345. Identify the number of surveys to be done

346. Once you have completed that number, do not do any additional surveys HOWEVER….

347. Confidence Intervals / Margins of Error Sometimes we do surveys and we do not know how many will be returned until later, as with postal surveys

348. Thus you have to calculate the margin of error afterwards…. Count up # of returned surveys

349. Identify target population and confidence level

350. Calculate margin of error

351. Confidence Intervals / Margins of Error All this assumes you know your target population and can get a sample frame If not, a non-random sampling technique is best Also consider whether you want to report results for sub-groups – the margins of error will be wider Consider stratified random sampling

352. Summary - 1 Survey research has developed into a popular research method since the 1940's.

353. A survey is a standardised stimulus designed to convert fuzzy psychological phenomenon into hard data.

354. Summary - 2 Survey development - types of questions and response formats.

355. Sampling - probability & non-prob.

356. Levels of measurement & parametric / non-parametric stats

357. Ethical considerations

358. Sources of measurement error

359. Evaluation Please complete the workshop evaluations

360. References Alreck, P. & Settle, R. (1995). The survey research handbook (2 nd ed.). New York: Irwin. Stevens, S.S. (1946). On the theory of scales of measurement. Science , 103 , 677-680. Trochim, W. M. K. (2006). Sampling . In Research Methods Knowledge Base . Wikipedia (2009). Shere Hite - Methodology . See also Readings

361. Open Office Impress This presentation was made using Open Office Impress.

362. Free and open source software.

363. http://www.openoffice.org/product/impress.html

Survey design workshop

More Related Content

What's hot (20)

Similar to Survey design workshop (20)

More from James Neill (20)

Recently uploaded (20)

Survey design workshop

Editor's Notes