How non-native English speakers respond to English surveys
1 like635 views
The document discusses the challenges non-native English speakers face when responding to English surveys, highlighting issues such as response tendencies, data quality measures, and specific errors made by ESL (English as a Second Language) speakers. It emphasizes the need for survey design that accommodates language differences and provides recommendations for improving data quality, including avoiding cultural references and ensuring questions are clear and concise. The document is intended for professionals involved in survey research and data quality assessment.
How non-native English speakers respond to English surveys
- 1. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and PollingPlease tweet! @LoveStats @PeanutLabsMediaPlease tweet! @LoveStats @PeanutLabsMedia I. Can’t. Understand. You: How non-native English speakers respond to English surveys Presented by Annie Pettit Chief Research Officer at Peanut Labs DIY Sample and Polling
- 2. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling 20% of US residents don’t speak English at home 9% of US residents don’t speak English well
- 3. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Survey •12 minutes, US, Census Rep •Driving, shopping, electronics, eating out, working
- 4. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling N=1212 N=228 N=204 Method Simple Language
- 5. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling DATA QUALITY
- 6. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Native Speakers Are Click Happy % of respondents choosing many answer options 0% 5% 10% 15% 20% Q8 Q11 Q16 Q24 Q26 Q30 Somewhat Good Very Good Extremely Good
- 7. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Speakers Are Click Averse % of respondents choosing minimal answer options 0% 20% 40% 60% Q8 Q11 Q16 Q24 Q26 Q30 Somewhat Good Very Good Extremely Good
- 8. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Speakers Choose Red Herrings Sounds familiar? Looks familiar? 0% 2% 4% 6% 8% 10% Somewhat Good Very Good Extremely Good
- 9. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Speakers Don’t Follow Instructions Failure rate on “Choose 3 items” 0% 20% 40% 60% 80% Choose 3 Choose 3 Somewhat Good Very Good Extremely Good
- 10. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Speakers Choose None of Above % choosing many None of Above answers 0% 4% 8% 12% 1 2 3 Somewhat Good Very Good Extremely Good
- 11. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Provide Shorter Verbatims % providing shortest responses 0% 5% 10% 15% 20% 25% 30% 1 2 3 4 5 Series1 Series3 Series5 Blank % in Q1 % in Q2 % in Q3 % in Q4 Answer Shortest Answers Longest Answers
- 12. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL are really fast or really slow % in Slowest to Fastest Deciles of Responses 0% 5% 10% 15% 20% 25% 1 2 3 4 5 6 7 8 9 10 Poor or Somewhat Good Very Good Extremely Good Slowest Decile Fastest Decile
- 13. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling ESL Responders Make More Errors 0% 5% 10% 15% 20% 25% 30% 1 2 3 4 5 6 7 8 9 10 11 12 Total Number of Errors Somewhat Good Very Good Extremely Good 0 1 2 3 4 5 6 7 8 9 10 11 Number of errors made
- 14. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling RESULT OF THESE DATA QUALITY MEASURES
- 15. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Higher % of ESL People Cut Are their opinions less important? Is it fair to cut them? 0% 2% 4% 6% 8% 10% 12% 14% 1 2 3 Somewhat good English Good English Excellent English
- 16. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Honest ESL People Get Cut Any final comments on this survey? • great and interesting survey • Have nothing for u guy, sorry im dumb lol lmao XD • i 'm very active • I enjoyed this survey • it was a good survey • it was very great to fill all ur surveys • very nice survey, not enough points • Well nothing to share now but their are something friendly way of living now. What 3 things do you do with your phone? • 1..To call someone 2..To know what is going on around the world. 3..to text my friends • coupons, texts, facebook • i phone, awesome, can download games and I used iPhone and mobile • internet and check emails and talk and text • Looking information on the internet. Call the doctors. • Play games, Watch Youtube, and Watch read, Talk Text Listen to music • text. chatting with friends and talking on the phone
- 17. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Most Language Discriminatory Variables Beta Sig Under Click V1 -.283 .000 Contradict V1 -.150 .002 Can’t Rank -.137 .006 Lies -.151 .011 Under Click V2 -.132 .030 Eliminate based on language
- 18. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Eliminate based on quality Least Language Discriminatory Variables Beta Sig. Over Click V1 .004 .944 Over Click V2 -.005 .927 Straightlining V1 -.005 .926 Over Click V3 .005 .923 Don’t Follow Instructions -.008 .903 Verbatim V1 .011 .838 Invalid Phone Number .015 .756 Contradict V2 -.019 .704
- 19. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling CONCLUSIONS
- 20. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Recommendations •Choose less language discriminative data quality questions • Overclicking • Straightlining • Follow instructions • Verbatim quality • Contradictions • Red herrings • None of the above
- 21. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Recommendations • Avoid cultural references, slang, norms • Consider forcing minimum responses • Keep questions short • Create white space Have you seen any advertising for this brand? Please consider TV ads, billboards, internet ads, and other ads. “Costs an arm and a leg”
- 22. Please tweet! @PeanutLabsMedia @LoveStats DIY Sample and Polling Thank you! Annie Pettit Chief Research Officer annie@peanutlabs.com ca.linkedin.com/in/AnniePettit/ facebook.com/AnniePettit twitter.com/LoveStats Jonathan Cheriff Director of Sales & Marketing jonathan.cheriff@peanutlabs.com Find PeanutLabs on LinkedIn Facebook Twitter YouTube DIY SURVEY PROGRAMMING DIY POLLINGDIY SAMPLE