Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Download as PPTX, PDF1 like449 views
The document discusses the significance of effective survey design and execution, emphasizing that trust and perceived effort versus reward are crucial for obtaining responses. It outlines a systematic process for surveys, highlighting essential steps such as goal establishment, question formulation, and data analysis to drive decision-making. Key considerations include question validity and the potential for various errors that can impact survey results.
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
- 1. Surveys that work An introduction to the Survey Octopus and Total Survey Error Caroline Jarrett @cjforms #surveysthatwork2022
- 2. What would you do for a dollar? Caroline Jarrett @cjforms (CC) BY SA-4.0 2
- 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 3 $1 in the envelope beats $10 guaranteed later Caroline Jarrett @cjforms (CC) BY SA-4.0 3
- 4. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Response Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 4
- 5. Would you answer this survey? 5 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6 I’m the forms specialist Image credit: Flickr, taxrebate.org.uk 6
- 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7 Why do people answer questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 7
- 8. I wrote a book It seemed easier than continuing to answer lots of survey questions https://rosenfeldmedia.com/books/surveys-that-work/ Caroline Jarrett @cjforms (CC) BY SA-4.0 8
- 9. Let’s think about these topics today “Please have a look at this survey” “Tell me whether this is a good question” “Is this statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 9
- 10. The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons. Caroline Jarrett @cjforms (CC) BY SA-4.0
- 11. I change the definition a bit systematic method becomes process gathering information becomes ask questions entities become people quantitative descriptors become numbers attributes of the larger population become make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
- 12. The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
- 13. Caroline Jarrett @cjforms (CC) BY SA-4.0 13 Let’s rearrange that somewhat The survey is a process for getting answers to questions To make decisions From people getting numbers
- 14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14 Start with why and who; end with the number The survey is a process for getting answers to questions Why you want to ask Who you want to ask The number
- 15. Caroline Jarrett @cjforms (CC) BY SA-4.0 15 A survey is a quantitative method
- 16. Key Point A survey is a quantitative method The result of a survey is a number Caroline Jarrett @cjforms (CC) BY SA-4.0
- 17. Christian Rohrer mapped many more methods https://www.nngroup.com/articles/which-ux-research-methods/
- 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 18 Survey methodologists interview first
- 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19 Establish your goals for the survey What do you want to know? Why do you want to know? What decision will you make based on these answers? What number do you need for the decision?
- 20. Key Point The aim of a survey is to get a number that will help with a decision Caroline Jarrett @cjforms (CC) BY SA-4.0
- 21. Caroline Jarrett @cjforms (CC) BY SA-4.0 21 Let’s have a good look at that process The survey is a process for getting answers to questions Why you want to ask Who you want to ask The number
- 22. Caroline Jarrett @cjforms (CC) BY SA-4.0 22 The Survey Octopus has things to think about Why you want to ask Who you want to ask The number
- 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23 The topics are all somewhat connected 23
- 24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24 I made a process from the things to think about Goals Questions Questionnaire Response Sample Fieldwork Response Reports 24
- 25. Here are the 7 steps as a linear process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean and analyse the data Present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Reports Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with Caroline Jarrett @cjforms (CC) BY SA-4.0
- 26. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “Is this statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 26
- 27. Caroline Jarrett @cjforms (CC) BY SA-4.0 27 Is this a good question? Would you recommend us to a friend or family member? 27
- 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28 “Recommend to friend/family” can be OK A shop selling clothes What do you want to know? Whether they will recommend Why do you want to know? To compare numbers over time What number do you need to make a decision? Some good reasons (investigate and fix problems) Some bad ones (punish staff) 28
- 29. Caroline Jarrett @cjforms (CC) BY SA-4.0 29 “Recommend to friend/family” can be very weird A shop selling clothes A hospital ward What do you want to know? Whether they will recommend ???? Why do you want to know? To compare numbers over time Because someone said they had to ask What number do you need to make a decision? Some good reasons (investigate and fix problems) Some bad ones (punish staff) ???? What decision?
- 30. Questions can go wrong in other ways Caroline Jarrett @cjforms (CC) BY SA-4.0
- 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 31 Make sure that you test your questionnaire
- 32. Key Point To find out whether a question is a good one, test it with people who will answer it Caroline Jarrett @cjforms (CC) BY SA-4.0
- 33. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “Is it statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 33
- 34. What type of significance do you need? • A result that is statistically significant is one that is mathematically unlikely to be the result of chance • A result that is significant in practice is one that is meaningful in the real world Caroline Jarrett @cjforms (CC) BY SA-4.0 34
- 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35 If you ask the wrong questions, you’ll fail at validity (Lack of) validity Sampling error 35
- 36. Key Point Asking one person the right question gets better results than asking 10,000 people the wrong question Caroline Jarrett @cjforms (CC) BY SA-4.0
- 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 Many statisticians aren’t keen, either Scientists rise up against statistical significance https://www.nature.com/articles/d41586-019-00857-9 37
- 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38 Let’s look at another crucial error Non-response error 38
- 39. Non-response error happens when the people who do not respond are different to the people who do respond in a way that affects your decision Caroline Jarrett @cjforms (CC) BY SA-4.0 39
- 40. Jane Matthews told me a story • 20 people attend a workshop; they all seem to enjoy it • Only get 3 or 4 back from a web survey “If we rely on those responses, we might be at risk of making bad decisions” • Now changing to phoning half the people Credit: https://janematthews.com/
- 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 41 There are errors all around the Survey Octopus (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error 41
- 42. Key Point Significance in practice comes from making good decisions throughout the survey Caroline Jarrett @cjforms (CC) BY SA-4.0
- 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use Who you want to ask The list that you sample from The sample you ask The ones who respond The ones whose responses you can use The number (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error
- 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 44 This version uses terms from survey methodology Construct Measurement Response Edited response Sampling frame Survey statistic Representation Sample Respondents Post-survey adjustments (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error 44
- 45. Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley. 45
- 46. Caroline Jarrett Twitter @cjforms http://www.effortmark.co.uk/blog carolinej@effortmark.co.uk Caroline Jarrett @cjforms (CC) BY SA-4.0 46
Editor's Notes
- #5: People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- #6: This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
- #24: The octopus again; we've looked at 6 of the 8 tentacles.
- #25: The octopus again; we've looked at 6 of the 8 tentacles.
- #28: If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
- #36: The octopus again; we've looked at 6 of the 8 tentacles.
- #39: The octopus again; we've looked at 6 of the 8 tentacles.
- #42: The octopus again; we've looked at 6 of the 8 tentacles.
- #44: This is a more conventional way of looking at the octopus tentacles
- #45: This slide translates the issues into the technical terms used by survey methodologists