Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
2 likes386 views
The document discusses using generative AI to assess the quality of open-ended survey responses. It outlines current challenges with open-ended responses, how generative AI can help through natural language processing, and best practices for implementation. Specifically, it provides an example of using ChatGPT to analyze sample open-ended responses and discusses setting up a Google Sheet to automate the process at scale and continuously improve the model through iterative training.
Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
- 1. Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys NewMR Webinar June 2023 Karine Pepin 2CV Research
- 2. LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE Using Generative AI to Assess the Quality of Open- Ended Responses in Surveys By Karine Pepin SVP
- 3. 3 Current Challenges with Open-Ended Responses How Generative AI can help Best practices
- 4. 4 Checks of Data Integrity Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In Partnership with a Council of Industry Leader TIME SPENT Pre- survey In-survey QC checks (Automated) Post-survey checks (mostly manual) o Geo-Location Tracking o Digital Device Fingerprinting o IP-Deduplication o Bots o Bot Script o Click-Farm o Form Filler Check o Instructional Manipulation Check o Low Incidence Check o Speeder/Racer o Straightlining o Red Herring/Trap Question o Contradictory Answers o Open-End Response Validation Survey o Data Discrepancy Checks o Open-End Response Review o Benchmark Comparison o MaxDiff Question Check
- 5. 5 In-survey validation: What it does Type of issues flagged in survey • Copy and paste • Profanity • Proper grammar • Duplicate response • Gibberish Be D add is do ha sh he be sh he be can ha DJ ha sh ha sh ha can be sh vs CZ sh ha en be xxx Amazon all very good and very nice debit cards in very nice and very good information technology they proide beat service in short best of time because I feel like you are doing well da*n I have no problem getting to the money 5
- 6. 6 In-survey validation: What it doesn’t do Thinking back to the last purchase you made using this type of payment, where was the purchase made and why did you choose this specific form of payment? In his inaugural address, Donald J. Trump maintained the “Make America Great Again” campaign mantra that helped sweep him into office. And he took the theme of “transfer of power” a step further, saying “we are transferring power from Washington I’m just making More money and I’m Saving up even more No thanks I don’t want to answer this question again I just got a call from the lord and I am in your high school. Tesla company is growing very fast and they make the best electric vehicles in the United States. Very much I appreciate you feeling today I am Good night too much for your help today to get a good one to come over and watch
- 7. 7 Open-Ended Response: What we need OpenAI's natural language processing models are designed to understand the context in which language is used allowing them to analyze the meaning and intent behind open-ended responses.
- 8. 8 Here’s a simple example using ChatGPT
- 9. 9 Limitations with ChatGPT Not possible to provide additional training data to ChatGPT directly Using GPT via Google Apps Script solves these challenges Cannot be integrated directly into Google Sheets/ Excel which allows you allows you to analyze more than one open-ended at the time making it hard to work with large datasets. Doesn’t get smarter: Doesn’t scale
- 10. 10 Automation Process Automate Use Google Apps Script to build a custom VERIFY_RESPONSE() formula that calls the OpenAI Chat completion API to receive a YES/NO assessment along with a reason Train Provide open-ended responses as training data to help the model learn to generate a more accurate assessment to similar open-ended responses Iterate Run open-ended responses in batches in order improve on your training
- 11. 11 Google Sheet Set-Up: Settings
- 12. 12 Google Sheet Set-Up: Training Data
- 13. 13 Google Sheet Set-Up: Survey Question
- 14. 14 Google Sheet Set-Up: Open-Ended Response Input/Output
- 15. 15 Google Sheet Set-Up: Open-Ended Response Input/Output
- 16. 16 Google Sheet Set-Up: Open-Ended Response Input/Output
- 17. 17 Google Sheet Set-Up: Double API Call for Other Languages
- 18. 18 How Does it Perform Is it better than a human? No. Is it better than doing nothing Yes! Is it faster than a human? Yes!
- 20. 20 Best Practices Use generic OE questions designed for QC only Data Confidentiality Create a bank of OE questions so the model doesn’t have to be train each time Efficiency Use more than one OE questions to make the final assessment Accuracy Provide additional training data when new countries, languages, or audiences are added Tuning Review rejected responses carefully. Spot check accepted responses. Validation The API could be integrated directly into the survey script, but exercise caution! Survey Integration
- 22. Q & A Ray Poynter NewMR Karine Pepin 2CV Research
- 23. Sponsors