Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

Is the Crowd wise enough to predict the Election
Results for the German “Bundestag”?
Thomas Donath, NORDLIGHT research GmbH
Christoph Irmer, ODC Services GmbH

Agenda

01 Check In

02 Research Question

03 Methods & Data

04 Results

05 Summary / Outlook

Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 2

What does it take to produce smarter
outcomes with a crowds “collective
intelligence”?

01 Diversity of opinion

02 Independence of opinion

03 Decentralization

04 Aggregation


The Research Question

Could a „Wisdom of Crowds“ approach to
election prognosis potentially compete
with or even outperform
traditional means of prognosis


Sample

2,323 valid respondents (X,XXX invitations -> XX% valid response rate)
 18 years and older
 widely spread demographics
 no other screening or weighting
 recruited through the ODC Services GmbH
online access panel “opinion-people”

Data filtering: respondents were only excluded if
 sum of % >100.0
 sum of % <100.0
 % for a single party = 100.0
Incentive
 for the smallest summed absolute deviation in %
 1. price: 300 €, 2. and 3. price 100 € each


Procedure for Data Collection

 Instruction (translated from German)
 1. We welcome each individual
participation. We only ask you to
submit a serious estimate.
 2. Please submit your personal
prognosis. To do so, please think
about how the percents will be
distributed across the parties.
 3. Please submit a prognosis as
precise as possible and make use of visual Feedback:
the position after the decimal point. sum is displayed green if equal
to 100.0
 How many percent of votes will the
following parties receive in the
Bundestagswahl 2009 on next
Sunday? Please do not enter the result
you wish for, but your estimate what
the official final result will be.


Wisdom of Crowd Result
CDU / CSU
 Distribution of estimate
CDU / CSU
400
 survey estimate, mean = 35.0%
reading help: approx. 300  election result: 33.8%
individuals estimated the CDU
/ CDU to obtain approx. 34%   +1.2
(34.0% - 34.9%)
300  CDU / CSU overestimated
 standard deviation = 5.5
 95% confidence interval: 34.7 –
200 35.2
 kurtosis = 6.4  leptokurtic,
„supergaußförmig“
100  skewness = -0.1  very low
skewness, symmetrical distribution
 n=2,323

10% 20% 30% 40% 50% 60% 70% 80% 90%

CDU / CSU


SPD
SPD
400
 election result: 23.0%
  +4.4
300  SPD strongly overestimated
200 27.6
 kurtosis = 2.0  slightly
leptokurtic, „supergaußförmig“
100  skewness = 0.3  low skewness,
fairly symmetrical distribution
 n=2,323

10% 20% 30% 40% 50% 60% 70% 80% 90%

SPD


FDP
FDP
400
  -2.3
300  FDP strongly underestimated
200 12.5
100  skewness = 1.3  slightly right-
skewed
 n=2,323

10% 20% 30% 40% 50% 60% 70% 80% 90%

FDP


Die Linke
Die Linke
400
  -1.4
300  Die Linke underestimated
200 10.7
100  skewness = 1.9  slightly right-
skewed
 n=2,323

10% 20% 30% 40% 50% 60% 70% 80% 90%

Die Linke


B90 – Die Grünen
Bündnis 90 - Die Grünen
400
  -0.7
300  B90 / Die Grünen
underestimated
200  95% confidence interval: 9.9 – 10.2
100  skewness = 0.3  very low
skewness, fairly symmetrical
distribution
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%

Bündnis 90 - Die Grünen


Andere
Andere (other parties)
400
  -1.2
300  “Andere” underestimated
 95% confidence interval: 4.6 – 4.9
200  kurtosis = 33.6  strongly
leptokurtic, „supergaußförmig“
 skewness = 4.1  right-skewed
100  n=2,323

10% 20% 30% 40% 50% 60% 70% 80% 90%

Andere (other parties)


Comparison with Polls
35,0
36,0
33,0
CDU / CSU 35,0
35,0
35,0
33,8

27,4
25,0
25,0
SPD 24,0
25,0 larger deviation from
26,0
23,0 final result and polls
12,3
13,0
14,0 larger deviation from
FDP 13,5
13,0
14,0 final result and polls
14,6
10,5
sum of absolute
11,0
12,0
%-points difference
Die Linke 11,5
12,0
from final result
11,0
11,9 18.09.09 - 23.09.09 Wisdom of Crowds 11.1%-points
10,0 20.09.09 Forschungsgruppe Wahlen (FGW) 8.4%-points
10,0
Bündnis 90 – 10,0
11,0 25.09.09 Forsa 4.2%-points
Die Grünen 11,0
10,0
10,7 22.09.09 IfD Allensbach 5.0%-points
4,8
5,0 18.09.09 Emnid 7.2%-points
6,0
Andere 4,0
5,0
17.09.09 Infratest dimap 8.4%-points
4,0
6,0 27.09.09 Final Result for Election

Moderator Variables I
sum %-point
deviation (group estimate n
vs. final result)
sex Mann-Whitney-U-Test
male 9,7% (1237) Z -11,9
female 12,7% (1073) p 0,000

age Kruskal-Wallis-Test
18 - 29 years 9,4% (441) Chi-Quadrat 72,0
30 - 39 years 11,6% (418) p 0,000
40 - 49 years 12,3% (486)
50 - 59 years 11,1% (412)
60 - 69 years 11,1% (371)
70 years and older 11,6% (142)

graduation Kruskal-Wallis-Test
Hauptschulabschluss (Volksschulabschluss) 12,2% (499) Chi-Quadrat 91,5
Realschule (Mittlere Reife) 11,6% (676) p 0,000
Abschluss der Polytechnischen Oberschule 10,8% (127)
Fachhochschulreife 10,5% (228)
Abitur (allgemeine oder fachgebundene Hochschulreife) 10,3% (686)

How often do you inform yourself about the political
Kruskal-Wallis-Test
occurances?
to be honest, not at all 14,2% (171) Chi-Quadrat 235,1
less than once per month 12,1% (189) p 0,000
once per month 16,0% (162)
once per week 11,7% (465)
several times per week 9,6% (1212)

 Virtually all socio- and psychographic moderator variables influence the precision of Wisdom
of Crowds estimates.

Moderator Variables II
sum %-point
deviation (group estimate n
vs. final result)
How interested are you in politics? Kruskal-Wallis-Test
to be honest, I am not interested in politics at all 14,4% (202) Chi-Quadrat 247,7
I am interested a tiny bit in politics. 12,9% (405) p 0,000
I am interested somewhat in politics. 11,6% (915)
I am strongly interested in politics. 9,5% (595)
I am very strongly interested in politics. 8,8% (191)

Which statement fits you personally the most? Kruskal-Wallis-Test
I have already voted via letter vote. 10,3% (381) Chi-Quadrat 75,1
I am going to vote and know exactly which party I will vote
10,8% (1167) p 0,000
for with my Zweitstimme (Landesliste).
I am going to vote, but I do not yet know exactly which
11,2% (405)
party I will vote for with my Zweitstimme (Landesliste).
I do not know yet whether I will vote, but I would know
exactly which party I would vote for with my Zweitstimme 13,2% (94)
(Landesliste).
I do not know yet whether I will vote and also I would not
know exactly which party I would vote for with my 12,3% (115)
Zweitstimme (Landesliste).
I am definitely not going to vote. 13,0% (105)

Have you seen prognoses for the results of the
Kruskal-Wallis-Test
Bundestagswahl 2009 within the last weeks?
yes, and I can remember (roughly) the results 10,1% (597) Chi-Quadrat 221,9
yes, but I cannot remember exactly the results 11,0% (961) p 0,000
no, I have not seen any prognoses 12,8% (700)

other sign. moderator variable: state (Bundesland), household size, marital status, employment status, type of work, income


Correlation with own Choice
personal estimate for final result
Korrelati onen

Bündnis 90 -
CDU / CSU SPD FDP Die Linke Die Grünen Andere
Kendall-Tau-b Sonntagsf rage: CDU / Korrelationskoef f izient ,221** -,072** ,094** -,087** -,058** -,051**
CSU Sig. (2-seitig) ,000 ,000 ,000 ,000 ,001 ,003
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: SPD Korrelationskoef f izient -,066** ,166** -,107** -,023 ,043* -,035*
own choice for vote

Sig. (2-seitig) ,000 ,000 ,000 ,181 ,013 ,044
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: FDP Korrelationskoef f izient ,018 -,081** ,164** -,028 ,008 -,020
Sig. (2-seitig) ,297 ,000 ,000 ,099 ,645 ,251
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: DIE Korrelationskoef f izient -,125** -,075** -,067** ,236** -,012 ,003
LINKE Sig. (2-seitig) ,000 ,000 ,000 ,000 ,488 ,867
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: BÜNDNIS Korrelationskoef f izient -,048** ,036* -,044** -,027 ,145** -,029
90/DIE GRÜNEN Sig. (2-seitig) ,005 ,034 ,010 ,107 ,000 ,092
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: Andere Korrelationskoef f izient -,085** -,019 -,033 ,027 -,036* ,127**
Sig. (2-seitig) ,000 ,270 ,053 ,120 ,037 ,000
N 2323 2323 2323 2323 2323 2323
**. Die Korrelation ist auf dem 0,01 Niv eau signif ikant (zweiseitig).
*. Die Korrelation ist auf dem 0,05 Niv eau signif ikant (zweiseitig).

 There is an opinion bias: Respondents tend to estimate the final result higher for those parties they
would also vote for.
 The effect is on average 2,7%-points.
 I.e. people estimate the final result for parties they would vote for 2,7%-points higher compared
to people who would vote for other parties.


Wisdom of Crowds vs. Unweighted, non-
represantative Poll
28,1
CDU / CSU 35,0
33,8
22,1
SPD 27,4
23,0
13,5
FDP 12,3
14,6
14,6
DIE LINKE 10,5
11,9 sum of absolute
BÜNDNIS 90/DIE 10,0 %-points deviation
10,0 from final result
GRÜNEN 10,7
11,7
Sonntagsfrage (Votes for the Panel) 16.8%-points
Andere 4,8 Wisdom of Crowds (Estimate of the Panel) 11.1%-points
6,0 Final Result for Election
 The Wisdom of Crowds method yields a better estimate for the outcome of the election than a non-
weighted, non-representative poll (Sonntagsfrage) from the same panel would have yielded.
 The “subjective” opinion bias – overestimating the preferred parties outcome – is clearly weaker
than the “objective” reasoning (see esp. CDU/CSU, SPD, Andere).


Effect of seen Prognoses on Precision
 People with (self-claimed) awareness or knowledge of recent prognoses yield better estimates.
 However, it can be shown that Wisdom of Crowds also works in absence of such knowledge.
 Political interest in itself – even without such knowledge – increases the precision.

 Result: People who claim to have seen recent prognoses submit better estimates.
 Hypothesis: Without representative polls in media, even people interested in politics would be unable to estimate the outcome precisely.
 However, there is a confoundation of seeing prognoses with general interest, the latter also being a strong moderator variable for estimate precision. (Kendall Tau-
b=0.390, p=0.000, n=2,244, fairly linear function for the used scales)

 Partial Correlation (functions of inter-correlation for used scales are all fairly linear)
 having seen prognoses with estimate precision (controlled for interest): r=0.172, p=0.000, n=2,241, r2=2.9%
 interest with estimate precision (controlled for having seen prognoses): r=0.220, p=0.000, n=2,241, r 2=4.9%

 Group 1: no prognoses seen, slight to high interest in politics difference = 11.3 %-points
 Group 2: no prognoses seen, low interest difference = 14.7 %-points
 (prognoses seen and roughly remembered difference = 10.1 %-points)
 n=389; n=307
 Mann-Whitney-U-Test: Z=-6.2, p=0.000

 Hypothesis can be rejected, based on self-report of participants: even without seeing recent prognoses, participants who are interested in politics provide good
estimates.
 Of course there is no reasonable way to truly eliminate seeing or benefiting from past prognoses seen by participants. However, this is not a requirement of the
Wisdom of Crowds approach. It actually requires to some extend informed participants.


Summary of Results
 Wisdom of Crowds outperforms unweighted, non-representative poll result of the same
sample.

 In this single study however, Wisdom of Crowds is inferior compared to representative
polls.

 The self-reported knowledge of recent prognoses drastically increases the quality of the
prediction. But also people interested in politics without that knowledge yield fairly
accurate predictions drastically outperforming participants without interest.

 Generally prediction quality is influenced massively by moderator variables.
 This fact is somewhat contrary to the claim by Surowiecki that the individual knowledge of
people hardly matters for the precision.


Outlook
 Wisdom of Crowds might not serve to compete with existing polling methods (yet!)

 but…

 imagine it would not have been election polls but e.g. Product-Tests, Ad-Tests, … :

35
CDU/CSU
Product C 33,8

27,4
SPD
Product D 23

FDP
Product A 12,3
14,6 Wisdom Of Crowds
Linke Final Result
Product F 10,5
11,9

B90 / Grüne
Product E 10
10,7

Andere
Product B 4,8
6


THANK YOU VERY MUCH!

Thomas Donath Christoph Irmer
NORDLIGHT research GmbH ODC Services GmbH
t.donath@nordlight-research.com ci@odc-services.com


Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

More Related Content

Viewers also liked (17)

Recently uploaded (20)

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?