Survey (Primer on Questions, Sampling + Case Study)
1 like673 views
The document discusses survey research methodology, outlining various sampling techniques such as probability and non-probability sampling, along with important design considerations for surveys. It highlights the significance of question formulation and the risks associated with misrepresentative sampling, referencing the critical failure of polling during the 1948 U.S. presidential election. The document emphasizes the necessity for rigorous survey design to enhance the accuracy and reliability of public opinion research.
Survey (Primer on Questions, Sampling + Case Study)
- 2. primer nuts & bolts case study
- 3. By a small sample we may judge the whole piece. Miguel de Cervantes (1547-1616)
- 4. Evolutionary Lack sharp corners & sudden turns.
- 6. Answers reveal relationships to something they measure. designing questions to be good measures
- 7. Qualitative Intent Bubbles I want to see the world through the eyes of my respondents. I want to describe the context in a lot of detail.I want to show how social change occurs. I’m interested in how things come to be. I want to makesure others can repeat my findings. I really spend a lot of time wondering how to measure things. I wonder how smallpatterns generalize to big patterns. I want to know what causes something else. Quantitative I really want my research approach to be flexible and ableto change.
- 8. A standardized stimulus designed to convert fuzzy phenomenon into data. (Neill. 2014) Study of a population through observation of its members. "The survey is a systematic method for gathering information from entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.” (Groves et al. 2004, p.4)
- 9. Research Design Matrix MixedMethod Quantitative Qualitative Explore Theorize Validate Optimize Constructivist Interpretive Big Picture Objectivist Positivist Granular Depth of Insight, Small Sample, Longer Engagement, Personal Broadness of Phenomena, Large Sample, Shorter Engagement, General
- 10. No Yes I know a lot about the topic I need to summarize thefindings in numerical format I don’t need a lot of detail and nuance I need to generalize the resultsto the population I need to predict the likelihood of a certainthing happening I wantto measureincremental change I don’t reallyknow much about the topic I don’t need to summarize thefindings in numerical format I need to communicate a lot of detail and nuance I’m interested in casestudy information I’m not planning onpredicting the likelihood of anything I don’t need to measure delta Is survey the right research method?
- 11. Mutually exclusive and exhaustive § No overlap between answers § All possible answers No double-barreled questions § Wrong: “Is your work permanent or casual? “ § Right: “What is your opinion of our response time?” Focus on opinions or beliefs of participants § Wrong: “How usable is our Web site?” § Right: “What are users’ opinions of the site’s look and feel?” Close-ended is best § Cuts down on analysis time Demographic questions at the end Survey Question Do’s
- 12. Survey Question Don’ts Inapplicable: does not apply to all respondents § How many years can a worker of average health be employed in your trade? Over-demanding: e.g., recall of time-consuming details § What are the general physical, intellectual and moral conditions of life of the working men and women employed in your trade? Ambiguous: meaning must be clear to all respondents § Describe wage increases during so-called prosperity periods. Leading § Does your employer or his representative resort to trickery in order to defraud you of your earnings? Loaded § Have you ever known any rank and file workers who could retire from employment at the age of 50 and live on the money earned by them as wage workers.
- 13. Probability Sampling Simple random sampling: Choose the people to survey randomly without any order or preference, like taking an email list and choosing 10% of them in no specific order and without any preference. More scientific.Randomnessensureshigher chances of representationinthetargetpopulation. Systematic Random Sampling: More organized version of random sampling, where you take an email list and choose, say, every 3rd on the list. Stratified Sampling: this is when you first divide the target population into smaller groups, like separating email addresses according to how much each one of them earns or their gender or education level. Then in each group you systematically choose whom to talk to, for example every 5th person on the list. Cluster Sampling: here you divide the target population into groups that seem natural, which almost always means based on geographical spread. After having done that, you can perform a simple random sampling within each cluster.
- 14. Non- Probability Sampling Judgment: this is when the person conducting the survey uses personal judgment or experience to choose the people to interview, like standing outside a supermarket and choosing to talk only to the people that look respectable. Often done for the convenience, subjects who are easily accessible or require the least expense.Use this sampling when you not looking for the exact truth, but just a decent approximation of it. Useful asa way of testing or gauging the marketin a very simple way. Quota: Cut the group you want to talk to into smaller groups according to gender, income, or some other criteria, just like in stratified sampling. Then you choose whom to talk to according to your judgment of which ones are more representative, or simply who is easier and least costly to reach. Snowball: Asking people whom you have just surveyed to refer you to other related people. Economical way of sampling a group that is difficult to locate, like people with a certain illness or uncommon hobby, preference, or interest, as those people will usually have a network.
- 15. Is survey a numbers game? Your subject population informs the game. If the population is large but homogenous, your sample can be small. If the population is very diverse, you need a larger sample. So it’s not just about the absolute number of people surveyed. What's more important is the mapping of the target population correctly, and then sampling accordingly.
- 17. #EpicFail “That’s not what I heard.”
- 18. 1948 US Election Survey George Gallup introduced the world to survey sampling by predicting the result of a previous election. Gallup was a respected figure in the field. He not only pioneered polling firms but advanced social innovation – feeling the pulse of the public on issues that affected them. Telephone polls held before the elections consistently showed Dewey as the leading candidate. Other polling firms such as Crossley and Roper predicted the same result. Insteadof facilitatinga transitionin government,the polls had misledthe presidentialcandidatesandall other politicians,the Washingtonbureaucrats,themedia,and the public.
- 19. Critical Errors Polling firms ended their polling two weeks before Election Day. Therefore, they missed the importance of measuring preferences just before the election. Pollsters had no certain way of deciding who would stay home on Election Day and who would go and cast a vote. The polling firms did not put much premium on the undecided voters, and therefore miscalculated their margins of error. The pollsters’ use of quota sampling that biased responses towards Truman. The Social Science ResearchCouncil committeeconcludedthat the 1948 polls werenot up to the standardsofscience.
- 20. Legacy It brought the academe and social scientists into a field they previously did not pay attention to. This brought rigor into survey as a method for gathering information on public opinion. Survey design became more sensitive to errors of sampling and brought attention to the importance of designing questions that elicit more accurate indicators of their target population. The academe developed a more strategic method for survey sampling – probability sampling, to lessen errors of underrepresentation or misrepresentation. The Gallup Poll of the 1948 US PresidentialElectionsprovedto be a watershedeventthat revolutionizedthescience of survey.
- 21. Up to this point, we in the field of public opinion research have had tocarry the ball ourselves with little help,but withplenty of criticism, from the social scientists. (Gallup 1948) The moreI worked on this report, the moreI felt the debt that weowe to these men who have been willing to risk their own money in trying to learnsomething about American political behavior… And wehave a responsibility, wein the universities, to do our best to help improve these techniques…. BecauseI believe in this work as aninstrument of democracy. (Stouffer 1948.p 214.)
- 22. A famous sampling mistake #TurningPoint “…predicting behavior on the basisof knowledge of attitudeis a very hazardousventure.” (Dinerman.1948)
- 23. If weknew what it was we were doing, it would not be called research,would it?
- 24. Babbie, Earl. (2011) The Practice of Social Research. Belmont CA: Wadsworth. Creative Research Systems (2009). Survey design: How to beginyour survey project. Fowler, F. J.,Jr. (2002). Designing questions to be good measures. In InF. J. Fowler, Survey research methods (3rd ed.) (pp. 76-103). Thousand Oaks, CA: Sage. Groves, Robert M.; Fowler, Floyd J.;Couper, Mick P.; Lepkowski, James M.; Singer,Eleanor & Tourangeau, Roger (2004). Survey methodology. Hoboken,NJ: JohnWiley & Sons. Howitt & Cramer (2011): Chapter 33 - The analysis of a questionnaire / survey project (pp. 407-415) Lindloff, Thomas & Taylor, Bryan. (2011). Qualitative Communication Research Methods. Thousand Oaks, CA: Sage. Nardi, P. (2006). Developing a questionnaire (Ch. 4). In Doing surveyresearch: Aguide to quantitative methods (2nd. ed.) (pp. 66-106). Boston,MA: Pearson. Trochim, W. M. K. (2006). Survey research. Zetterberg, Hans (2004). US Election 1948: The First Great Controversy aboutPolls, Media, and Social Science. Paper presented at the WAPOR regional conference on"Elections, News Media and Public Opinion"in Pamplona, Spain,November, 24-26, 2004. http://www.zetterberg.org/Lectures/l041115.htm References