What’s really going on inside panels?
And does it matter?
________
NORSTAT SUMMER LOUNGE
June 2014
Dan Kvistbo
SORT OF AN AGENDA
 HOW FAR WE ARE WHERE WE ARE GOING 
THE BEGINNING
BARRIERS ALL BUT GONE
1999-2001
_____
• Toluna
• GMI
• Ciao!
• Lightspeed
• Zapera
• Research Now
• etc.
_____
LOW COST | HIGH SPEED
LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.
FOCUS ON BENEFITS | DATA ASSUMED REP2
RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE
CASE IN POINT | P&G
P&G | 1999: 15%  2000: 40%  2001: 50%
”To save money and expedite
market research surveys”...
_____
”[...] because surveys are
completed 75% faster over the
Web-and at half the cost of
conventional methods [...]”
TROUBLED WATERS | 2003‘ish - 2006 ()
CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT
“Two surveys a week apart yielded
different recommendations…
_____
I never thought I was trading data
quality for cost savings”
A WOMAN WITH A BUDGET
Kim Dedeker, Procter
& Gamble's VP-Global
Consumer & Market
Knowledge, 2006
(Now Chairman, the
Americas at Kantar)
WHAT HAD GONE WRONG?
PART (I) OF THE TALE
“...clients, have put many
pressures on the research
suppliers […]”
_____
”We’ve focused on cost and
speed and an unintended
consequence was erosion in
quality. Now, we’re starting to feel
the effects”…
Kim Dedeker
PART (II) OF THE TALE
• Heavy survey taking
• Panel conditioning
• Declining cooperation
• Increase in demand
• Persistent mode differences
• Long and complex questionnaires
• Multi panel membership
• The 0.25% doing 30+%
• ‘Yes we can’
• Etc.
_____
(Source: Reg Baker, Market Strategies International)
MR INDUSTRY RESPONSE?
”It’s like the whole in the ozone
layer. Everyone knows it’s a
growing problem.”
_____
”But they just ignore it and go on to
the next project.”…
(S. Morwood, VP Worldwide Market Research for
IBM. AdAge oct. 2006)
TRUTH BE TOLD
ESOMAR 25
2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 –
The Grand Mean Project
ARF ORQC
”Million dollar” FoQ project
ISO 26362
AAPOR Report on Online Panels
FoQ 2 (Reduce the Risk)
Report on the AAPOR task force on non-probability sampling
ESOMAR/GRBN (draft) Guideline For Online Sample Quality
*(Consultation draft)
<
26 28
 HALFWAY THROUGH! 
BUT IN ESSENCE
“75 years of basing quantitative market research
on a paradigm of projecting the results of
samples onto populations via assumptions of
random probability sampling, has been replaced
by a willingness to use online access panels,
which do not approximate to this model.”
_____
- Ray Poynter
(Handbook of Online and Social Media Research, 2010)
AND DESPITE THE EFFORTS
Lemon market | Conditions:
_____
• Asymmetry of information (buyer vs. seller)
• Incentive for the seller to pass off a low-quality
product as a higher-quality one
• Sellers with a bettter product have no way of
demonstrating this credibly to buyers
• Deficiency of effective quality assurances/guarantees
_____
http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased)
Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism
- George A. Akerlof
The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500
BLACK BOX
White-box testing
_____
a method of testing [software] that tests
internal structures or workings [of an
application], as opposed to [its] functionality.
(http://en.wikipedia.org/wiki/White-box_testing)
SO... WHAT IS GOING ON INSIDE?
 CLOSER...
DIRTY LITTLE SECRETS OF ONLINE PANELS
ISSUE #1 CONDUCT
_____
N = 1000, whatever it takes
ISSUE #2 RECRUITMENT
ISSUE #3 RESPONSE RATES
Human Interaction?
Trust that the survey is legitimate?
Time involvement with each respondent?
Attention given to each respondent?
Respondent control over access?
Respondent control over whether to respond?






CONSEQUENCE
The concept of TSE
gets a whole new
meaning…
• “Total” sampling error
• “Total” coverage error
• “Total” non-response error
• Etc.
IS IT EITHER OR?
A Data
Q sample
A convenience
sample of a
convenience
sample
Random
probability
sample
Convenience
sample
A quota sample
of a random
sample
Yes, you are
”...trading data
quality for cost
savings...”
TO CONCLUDE
“If you pay peanuts for your sample, chances are
you're getting monkeys”
(Gordon Morris Global Insights Manager at Sony Ericsson)
”I don’t want it done scientifically, I just want it
done right!”
(Client, Ron Sellers)
“Sometimes close counts in more than
horseshoes”
(Reg Baker)
LOOKING FORWARD
A few words on the future of online research
1. It’s (increasingly) mobile
2. It moves beyond self-reported attitudes,
emotions and behaviours to merge survey data and
behavioral data
3. Success depends as much on buyers
acknowledging their responsibility (ask better
questions and adapt to the realm of mobile!) as on
sample providers to deliver the appropriate sample
& platform
THANK YOU
Or google
...there’s only one of me
_____
DAN KVISTBO
dan.kvistbo@norstat.dk
+45 27772016
@Kvistbo
Questions?

More Related Content

PPT
iLiftTV executive summary 2013 full size
PDF
Natürlich Online
PPTX
It's the end of the world as we know it
PDF
Building Interactive Systems for Social Good [Job Talk]
PPTX
Brian Fabo
PPTX
Powering Supply Chain Transformation Through Analytics Innovation
PDF
Genesi di una tecnologia, dalla ricerca all'industria...
PDF
Andrea Mocci: Beautiful Design, Beautiful Coding at I T.A.K.E. Unconference 2015
iLiftTV executive summary 2013 full size
Natürlich Online
It's the end of the world as we know it
Building Interactive Systems for Social Good [Job Talk]
Brian Fabo
Powering Supply Chain Transformation Through Analytics Innovation
Genesi di una tecnologia, dalla ricerca all'industria...
Andrea Mocci: Beautiful Design, Beautiful Coding at I T.A.K.E. Unconference 2015

Similar to Whats really going on inside panels and does it matter (20)

PPTX
Software Sustainability: Better Software Better Science
PDF
Continuous Delivery Sounds Great but it Won't Work Here
PPT
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
PPTX
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
PDF
Ux event ut lab slideshare - v2
PDF
ROAD POTHOLE DETECTION USING YOLOV4 DARKNET
PPT
Perceived Usability, Attractiveness and Intuitiveness of Responsive Mobile To...
PPTX
COBWEB Summit at the OGC TC Dublin, 2016
PDF
40120130405011
PDF
IRJET- Improved Model for Big Data Analytics using Dynamic Multi-Swarm Op...
PDF
Intelligent Traffic Light Control System
PDF
Analysis random org nist2005
PPTX
Orbit Foundations of Responsible Research and Innovation
PDF
Utilization of Super Pixel Based Microarray Image Segmentation
PDF
Mr1480.ch4
PDF
TestCon2018 - Next Generation Testing in the Age of Machines
PDF
IRJET- A Real Time Yolo Human Detection in Flood Affected Areas based on Vide...
PDF
Banner-blindness in on-line advertising
PDF
STI Summit 2011 - Visual analytics and linked data
PDF
SamplingMethodinResearchMethodologyHowtoChooseaSamplingTechniqueforResearch.pdf
Software Sustainability: Better Software Better Science
Continuous Delivery Sounds Great but it Won't Work Here
Scott Edmunds @ Balti & Bioinformatics: New Models in Open Data Publishing
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
Ux event ut lab slideshare - v2
ROAD POTHOLE DETECTION USING YOLOV4 DARKNET
Perceived Usability, Attractiveness and Intuitiveness of Responsive Mobile To...
COBWEB Summit at the OGC TC Dublin, 2016
40120130405011
IRJET- Improved Model for Big Data Analytics using Dynamic Multi-Swarm Op...
Intelligent Traffic Light Control System
Analysis random org nist2005
Orbit Foundations of Responsible Research and Innovation
Utilization of Super Pixel Based Microarray Image Segmentation
Mr1480.ch4
TestCon2018 - Next Generation Testing in the Age of Machines
IRJET- A Real Time Yolo Human Detection in Flood Affected Areas based on Vide...
Banner-blindness in on-line advertising
STI Summit 2011 - Visual analytics and linked data
SamplingMethodinResearchMethodologyHowtoChooseaSamplingTechniqueforResearch.pdf
Ad

Recently uploaded (20)

PDF
SlidesGDGoCxRAIS about Google Dialogflow and NotebookLM.pdf
PDF
Session 1 (Week 1)fghjmgfdsfgthyjkhfdsadfghjkhgfdsa
PPTX
Mathew Digital SEO Checklist Guidlines 2025
PPTX
newyork.pptxirantrafgshenepalchinachinane
PPTX
Cyber Hygine IN organizations in MSME or
PDF
The Ikigai Template _ Recalibrate How You Spend Your Time.pdf
PPTX
Database Information System - Management Information System
PPTX
IPCNA VIRTUAL CLASSES INTERMEDIATE 6 PROJECT.pptx
PDF
Understand the Gitlab_presentation_task.pdf
PDF
BIOCHEM CH2 OVERVIEW OF MICROBIOLOGY.pdf
PDF
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
PDF
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
PPTX
Reading as a good Form of Recreation
PPT
12 Things That Make People Trust a Website Instantly
PDF
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
PDF
Top 8 Trusted Sources to Buy Verified Cash App Accounts.pdf
PDF
Course Overview and Agenda cloud security
PPTX
AI_Cyberattack_Solutions AI AI AI AI .pptx
PPTX
TITLE DEFENSE entitle the impact of social media on education
PPT
415456121-Jiwratrwecdtwfdsfwgdwedvwe dbwsdjsadca-EVN.ppt
SlidesGDGoCxRAIS about Google Dialogflow and NotebookLM.pdf
Session 1 (Week 1)fghjmgfdsfgthyjkhfdsadfghjkhgfdsa
Mathew Digital SEO Checklist Guidlines 2025
newyork.pptxirantrafgshenepalchinachinane
Cyber Hygine IN organizations in MSME or
The Ikigai Template _ Recalibrate How You Spend Your Time.pdf
Database Information System - Management Information System
IPCNA VIRTUAL CLASSES INTERMEDIATE 6 PROJECT.pptx
Understand the Gitlab_presentation_task.pdf
BIOCHEM CH2 OVERVIEW OF MICROBIOLOGY.pdf
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
Reading as a good Form of Recreation
12 Things That Make People Trust a Website Instantly
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
Top 8 Trusted Sources to Buy Verified Cash App Accounts.pdf
Course Overview and Agenda cloud security
AI_Cyberattack_Solutions AI AI AI AI .pptx
TITLE DEFENSE entitle the impact of social media on education
415456121-Jiwratrwecdtwfdsfwgdwedvwe dbwsdjsadca-EVN.ppt
Ad

Whats really going on inside panels and does it matter

  • 1. What’s really going on inside panels? And does it matter? ________ NORSTAT SUMMER LOUNGE June 2014 Dan Kvistbo
  • 2. SORT OF AN AGENDA  HOW FAR WE ARE WHERE WE ARE GOING 
  • 4. BARRIERS ALL BUT GONE 1999-2001 _____ • Toluna • GMI • Ciao! • Lightspeed • Zapera • Research Now • etc. _____
  • 5. LOW COST | HIGH SPEED LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.
  • 6. FOCUS ON BENEFITS | DATA ASSUMED REP2 RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE
  • 7. CASE IN POINT | P&G P&G | 1999: 15%  2000: 40%  2001: 50% ”To save money and expedite market research surveys”... _____ ”[...] because surveys are completed 75% faster over the Web-and at half the cost of conventional methods [...]”
  • 8. TROUBLED WATERS | 2003‘ish - 2006 ()
  • 9. CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT “Two surveys a week apart yielded different recommendations… _____ I never thought I was trading data quality for cost savings” A WOMAN WITH A BUDGET Kim Dedeker, Procter & Gamble's VP-Global Consumer & Market Knowledge, 2006 (Now Chairman, the Americas at Kantar)
  • 10. WHAT HAD GONE WRONG?
  • 11. PART (I) OF THE TALE “...clients, have put many pressures on the research suppliers […]” _____ ”We’ve focused on cost and speed and an unintended consequence was erosion in quality. Now, we’re starting to feel the effects”… Kim Dedeker
  • 12. PART (II) OF THE TALE • Heavy survey taking • Panel conditioning • Declining cooperation • Increase in demand • Persistent mode differences • Long and complex questionnaires • Multi panel membership • The 0.25% doing 30+% • ‘Yes we can’ • Etc. _____ (Source: Reg Baker, Market Strategies International)
  • 13. MR INDUSTRY RESPONSE? ”It’s like the whole in the ozone layer. Everyone knows it’s a growing problem.” _____ ”But they just ignore it and go on to the next project.”… (S. Morwood, VP Worldwide Market Research for IBM. AdAge oct. 2006)
  • 14. TRUTH BE TOLD ESOMAR 25 2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 – The Grand Mean Project ARF ORQC ”Million dollar” FoQ project ISO 26362 AAPOR Report on Online Panels FoQ 2 (Reduce the Risk) Report on the AAPOR task force on non-probability sampling ESOMAR/GRBN (draft) Guideline For Online Sample Quality *(Consultation draft) < 26 28  HALFWAY THROUGH! 
  • 15. BUT IN ESSENCE “75 years of basing quantitative market research on a paradigm of projecting the results of samples onto populations via assumptions of random probability sampling, has been replaced by a willingness to use online access panels, which do not approximate to this model.” _____ - Ray Poynter (Handbook of Online and Social Media Research, 2010)
  • 16. AND DESPITE THE EFFORTS Lemon market | Conditions: _____ • Asymmetry of information (buyer vs. seller) • Incentive for the seller to pass off a low-quality product as a higher-quality one • Sellers with a bettter product have no way of demonstrating this credibly to buyers • Deficiency of effective quality assurances/guarantees _____ http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased) Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism - George A. Akerlof The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500
  • 17. BLACK BOX White-box testing _____ a method of testing [software] that tests internal structures or workings [of an application], as opposed to [its] functionality. (http://en.wikipedia.org/wiki/White-box_testing)
  • 18. SO... WHAT IS GOING ON INSIDE?  CLOSER...
  • 19. DIRTY LITTLE SECRETS OF ONLINE PANELS
  • 20. ISSUE #1 CONDUCT _____ N = 1000, whatever it takes
  • 22. ISSUE #3 RESPONSE RATES Human Interaction? Trust that the survey is legitimate? Time involvement with each respondent? Attention given to each respondent? Respondent control over access? Respondent control over whether to respond?      
  • 23. CONSEQUENCE The concept of TSE gets a whole new meaning… • “Total” sampling error • “Total” coverage error • “Total” non-response error • Etc.
  • 24. IS IT EITHER OR? A Data Q sample A convenience sample of a convenience sample Random probability sample Convenience sample A quota sample of a random sample Yes, you are ”...trading data quality for cost savings...”
  • 25. TO CONCLUDE “If you pay peanuts for your sample, chances are you're getting monkeys” (Gordon Morris Global Insights Manager at Sony Ericsson) ”I don’t want it done scientifically, I just want it done right!” (Client, Ron Sellers) “Sometimes close counts in more than horseshoes” (Reg Baker)
  • 26. LOOKING FORWARD A few words on the future of online research 1. It’s (increasingly) mobile 2. It moves beyond self-reported attitudes, emotions and behaviours to merge survey data and behavioral data 3. Success depends as much on buyers acknowledging their responsibility (ask better questions and adapt to the realm of mobile!) as on sample providers to deliver the appropriate sample & platform
  • 27. THANK YOU Or google ...there’s only one of me _____ DAN KVISTBO dan.kvistbo@norstat.dk +45 27772016 @Kvistbo