Margins of Error: public understanding of statistics in an era of big data

MarginsMargins
f Eof Error
John PullingerJohn Pullinger,
President of the Royal Statistical Society
© Ipsos MORI / King’s College London

Public trust andPublic trust and
understandingg
Bobby DuffyBobby Duffy
Director, Ipsos MORI Social Research Institute,
Visiting Senior Fellow, King’s College London

Focus on
understanding
d l b tand value – but
firstly on trustfirstly on trust…

Scientists and academics win...
How much trust do you have in information provided by the following types of
people?
28 46 3
A great deal A fair amount None at all
Scientists
18
13
45
42
6
10
Academics
Accountants
12
8
37
31
12
15
Statisticians
Economists 8
9
31
28
15
8
Economists
Actuaries
2
1
21
7
23
54
Pollsters
Politicians
Base: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013

Trust in scientists vs trust in clergy
– a new age of reason?
...would you generally trust them to tell the truth, or not?
90
Clergymen/Priests Scientists
75
80
85
Clergymen/Priests Scientists
% Yes
65
70
75
50
55
60
40
45
50
30
35
98 99 00 01 02 03 04 05 06 07 08 09 10 11 12* 13**
Base: c.1,000-2,000
Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013
IM telephone

Trust in civil servants vs politicians – views have
diverged...
...would you generally trust them to tell the truth, or not?
Civil Servants Government Ministers Politicians Generally Journalists
60
70
% Yes
40
50
30
40
10
20
0
Base: c.1,000-2,000
Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013
IM telephone

But government less trusted with our data
than online retailers?
5A great
Companies such as
supermarkets and online5
38
2
A great
deal
A fair
supermarkets and online
retailers collect a lot of data
on their customers (for
example through loyalty38
40
30
A fair
amount
Not very
cards). To what extent, if at
all, do you trust companies
to use the data they collect
about you appropriately40
12
41
Not very
much
about you appropriately
The government collects a
lot of data on citizens (for
l th h t12
6
20
Not at all example through tax
returns). To what extent, if at
all do you trust the
government to use the data6
6
Don't know
g
they collect about you
appropriately?
Base: c. 500 British adults aged 16-75 Source: RSS/Ipsos MORI 2013

Big, technicalg,
issues for
people to come
to view on…

...not least, debt vs deficit...
As you may know there is currently a lot of discussion about our national “debt” and “deficit”.
Can you tell me what these words mean when talking about government finances?
The difference between
3
Debt means Deficit means
The difference between
what government spends
and the income it
receives each year3
7
8
4
7
8
receives each year
The total amount of money
that the government owes
4
Both mean the same
Don’t know
82 8282 82

...it is a tricky one...

...but public also not so clear when “use it
in a sentence”...
And can you tell me whether the following statement is true or false?
“The national debt will always go down if the deficit is decreasing”
20
Those who got definitions right
The national debt will always go down if the deficit is decreasing
28
20
TRUE
Those who got definitions right
no more likely to get this right
FALSE
Don't know
Public think 40% of planned cuts
already been made
52
already been made...

Basic
understandingunderstanding
of numbers isof numbers is
key to statisticaly
literacy – and it
i i dis mixed…

Most get very simple questions correct...
What is 50 expressed as a percentage of 200?
2010 89% t
92
10%
25%
2010: 89% correct
92
3
25%
50%
175%
1Other
Don't
3
Don t
know

...and slightly trickier...
What is the average of the following three numbers – 5, 10 and 15?
2010: 71% correct
16
70
5
10
2010: 71% correct
70
1
10
12
515
3Other
5Don't know

...but real difficulties with probabilities...
If you spin a coin twice what is the probability of getting two heads?
1 % 2010 30% t1
26
15%
25%
2010: 30% correct
240%
58
1
50%
75% 1
2
75%
Other
Strong relationship with education (A-level+),
10Don't know
Strong relationship with education (A level ),
but also big differences by age, younger groups
more likely to get right...

There are alsoThere are also
known biases in
how we consider
t ti tistatistics…

A personal optimism bias...
What do you think the chance or probability is of the following being injured or killed
in a road accident this year (whether as a road user or a pedestrian)?
S i G t B it i Y
2
3
1About 1 in 2
About 1 in 5
Someone in Great Britain You
8
6
2
1
About 1 in 5
About 1 in 10
Ab t 1 i 20
Mean probability:
Someone = 4.1%
Y 1 6%
6
7
7
2
3
About 1 in 20
About 1 in 50
You = 1.6%
Actual probability = c1.2%?
7
20
24
5
19
About 1 in 100
About 1 in 1000
24
23
40
27
About 1 in 10,000
Don't know

...but focus on negative information
Imagine you have a life-threatening illness and your doctor has told you that you need
an operation to treat it. How likely, if at all, are you to have this operation if your
doctor tells you that...y
90% of people who have the operation are alive for at least 5 years following the operation
10% of people who have the operation die within 5 years of the operation
56
33
39
Very likely
33
3
38
6
Quite likely
Not very likely
1
6
2
y y
Not at all likely Avoid targets on “negatives”,
even if hit them? Waiting
7
16
Don't know
even if hit them? Waiting
times, immigration...

But does it
matter? Do
l idpeople consider
evidence – orevidence or
think their leaders
do?

Principle-based policy-making...
Politicians will take decisions partly based on what they think is right, and partly on evidence of
what works. Do you think they base their decisions more on what they think is right than on
evidence, more on evidence than on what they think is right, or do you think they consider them
b th i l ?both in equal measure?
More on what they think is right
than on evidence
18
than on evidence
More on evidence than what
they think is right
16
y g
On evidence and what they
think is right about the same
t
52
16 amount
Don't know
13

...but mirrors people’s own use of
evidence
People have different attitudes towards statistics. Which of the following do you agree
with most?
My own experiences or those of my
family and friends are more
important than statistics in helping
26
Statistics are more important than
me keep track of how the
government is doing
46
my own experiences or those of my
family and friends in helping me
keep track of how the government is
doing
18
Both equally
Neither/Don’t know
9
Neither/Don t know

More broadlyMore broadly,
understandingg
numbers is
undervalued?

We’re not embarrassed about lack of
understanding of numbers...
Which of the following things would you feel most embarrassed about
admitting to friends and family?
6I'm not very good with numbers
15I'm not very good at reading and writing
75
y g g g
Neither 75Neither
5Don't know
Base: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013

...and there’s little pride in doing it well
Thinking about your child/if you had a child, which of the following would
make you most proud?
13If they were very good with numbers
55
If they were very good at reading and
iti
55
16
writing
N ith 16Neither
15Don't know
Base: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013

We’ve got a long way to go...
I keep saying that the sexy job in the next 10 years will
be statisticians. And I’m not kidding.
Hal Varian, chief economist at Google
Statistical thinking will one day be as necessary for
efficient citizenship as the ability to read or write
HG Wells
Value of statisticsNumber of people reachedQuantity of statistical infoMedia affectRelevanceTrustNumeracy
VAS = N * [(QSA * MF) * RS * TS * NL]
Enrico Giovannini, Former Chief Statistician, OECD

Thank youy
bobby duffy@ipsos combobby.duffy@ipsos.com
@BobbyIpsosMORI

Understanding andUnderstanding and
Trust in StatisticsTrust in Statistics
Andrew Dilnot CBE,Andrew Dilnot CBE,
Chair of the UK Statistics Authority

GDP 1948-2012 (Index 2009=100)
120
100
)
60
80
009=100)
40
60
ndex(20
20
I
0

GDP 2000-2013
120
100
60
80
009=100)
40
60
Index(20
20
0

GDP 2000-2013
110
100
105
90
95
009=100)
85
90
Index(20
75
80
70

GDP Revisions
Available data
( )
Preliminary Estimate Second Estimate Quarterly National Accounts
(output measure)
25 days approx 55 days approx 3 months

Former minister slams
'national catastrophe' of'national catastrophe' of
teenage mothers
addicted to benefits UK has highest teen
pregnancy rate in p g y
EuropeTEENAGE PREGNANCY
SOARS
NO SET OF VALUES
SOARS
FOR GYM-SLIP
MUMS
MUMS

Under 18 conception rate for
England and Wales

Norovirus

Norovirus lab reports
Source: Health Protection Agency

Norovirus confidence intervals
• 1:1500 (1 lab case = 1500 in community).
• 2000 lab cases = 3million in community
• But maybe 1:140But maybe 1:140
• (=280,000 cases)
• Or maybe 1:17,000
• (=34 million cases)
• Community study lab cases…
• =1

The 2011 Census and uncertainty
1 4
Relative Confidence Interval width
1.2
1.4t)
0 8
1
(percent
0.6
0.8
alwidth(
0 2
0.4
Interva
0
0.2
England South East Kent Canterbury
England South East Kent Canterbury

Trends in police recorded crime
and CSEW

Lies, damn lies
Crime
and crime
statistics
statistics
were
statistics
POLICE FAIL were
distorted by
POLICE FAIL
TO RECORD y
politics
TO RECORD
CRIME
PROPERLY

Crime falls to new low despite
recession and unemploymentrecession and unemployment
...The 6% fall in crime reported in the latest quarterly p q y
figures by both the Crime Survey for England and Wales
and the separate police recorded crime figures means that
crime has now dropped by more than 50 % since it peaked
in the mid‐1990s...
The Guardian, 19 October 2013

Public Understanding of
t ti ti i f bistatistics in an era of big
datadata
Denise Lievesley,Denise Lievesley,
Head of School of Social Science and Public Policy,
King’s College London

Challenges facing statisticians
H ilit C fid
R l
Humility
A t
Confidencevs
Relevance
T t
Autonomy
S i i
vs
Trust Scepticismvs
Measurement Quality
vs
Pragmatism Purismvs

Humility vs. Confidence
Being a statistician means
never having to say
you’re certain

Humility – being aware of our limitations
“Good science should not turn a blind eye to known imperfections –
nor should these be concealed from users”
Sir Roger Jowell 2007
“The absence of excellent evidence does not make evidence-based
decision making impossible: what is required is the best evidence
available not the best evidence possible”
Si M i G 199Sir Muir Gray 1997

ISI declaration on professional ethics 1985
• One of the most important but difficult responsibilities of the
t ti ti i i th t f l ti t ti l f th i d t t thstatistician is that of alerting potential users of their data to the
limits of their reliability and applicability. The twin dangers of
either overstating or understating the validity or generalisabilityeither overstating or understating the validity or generalisability
of data are nearly always present.
• Confidence in statistical findings depends critically on their
faithful representation. Attempts by statisticians to cover up
i i i i l b derrors, or to invite over- interpretation, may not only rebound
on the statisticians concerned but also on the reputation of
statistics in generalstatistics in general.

Confidence –
using the data to make a difference
•We need to provide information of high quality,We need to provide information of high quality,
integrity and robustness which can be relied on.
•We should be confident about our findings and
prepared to account for them.p p

Communication
We need to improve our communication skills
and think about impact.
We should learn how to tell a story with datay
and remember that communication is not what
is delivered but what is receivedis delivered but what is received.
e.g.
• Bill Gates has a personal fortune greater than the combined
wealth of the 106 million poorest Americans.
• The cost of putting all children into school is less than is spent
on icecream in Europe each year

Sir Gus O’Donnell
(former UK Cabinet Secretary)
“I want [the ONS] to be boring, to put out the plain facts, and
nothing but the facts and on clear predictable deadlines ” henothing but the facts, and on clear, predictable deadlines, he
said. It would then be for politicians and government press
officers to interpret the figures, he added.p g ,

Response of the Royal Statistical Society
• it is clearly the task of statisticians to interpret the figures
in a statistical context, to facilitate understanding and
avoid misunderstanding.
• The Code of Practice of the UK Statistics Authority
explicitly states that Official statistics accompanied byexplicitly states that Official statistics, accompanied by
full and frank commentary, should be readily accessible
to all users and that all UK bodies that are responsibleto all users and that all UK bodies that are responsible
for official statistics should prepare and disseminate
commentary and analysis that aid interpretation andcommentary and analysis that aid interpretation, and
provide factual information about the policy or
operational context of official statistics
operational context of official statistics.

Relevance vs. Autonomy
UN Fundamental Principles of Official Statistics
Principle 1
“Offi i l t ti ti id i di bl l t i th“Official statistics provide an indispensable element in the
information system of a democratic society, serving the
Government the economy and the public To this end officialGovernment, the economy and the public ... To this end, official
statistics that meet the test of practical utility are to be compiled
and made available on an impartial basis by official statistical
agencies..”

Impartiality
• The role of statisticians: to inform political debate and decisions
without taking partt out ta g pa t
• Fear that enhancing statistical utility will compromise impartiality
• There must be no political interference with the data and no
perception that there isperception that there is
But does this mean we are too cautious?
Are statisticians so afraid of being accused of political motivesg
that they dare not make reports useful for the public debate?

The value of statistics to society must not
just be asserted; it must be demonstrated
“Were a balance sheet for official statistics to be prepared the“Were a balance sheet for official statistics to be prepared, the
costs would be clear enough. The benefit, or value, would
however be found to be much more diffuse and harder to treat inhowever be found to be much more diffuse and harder to treat in
traditional accounting terms. Given this, it is possible that the
vital asset that official statistics represent is undervalued in
public sector planning processes. And we observe that little
systematic consideration is given to how the public value could
be maximised”be maximised .
(UK Statistics Commission, The Use Made of Official Statistics,
(UK Statistics Commission, The Use Made of Official Statistics,
2007)

Trust vs. Scepticism
• Pre-requisite for evidence based policy and for
managing for results is that the data must be trustworthy

But it is not enough that the data are
trustworthy they must also be trusted
• Otherwise they won’t be used
• There will be fights about the data rather than about the
issues
• Data need to be the currency of public debates

Evidence sometimes resisted...
“There is nothing a governmentThere is nothing a government
hates more than to be well-
informed: for it makes the process
of arriving at decisions much moreof arriving at decisions much more
complicated and difficult.”p
John Maynard Keynes

Inconvenient truths
• Governments prefer good news stories
• Bad news stories may be delayed or buried
• They are often too focussed on populism• They are often too focussed on populism
• The government’s horizons can be shorter than those of
i i i !statisticians!
• They can prefer their own spin to that of the statisticiany p p

Important aspects of building trust
• Autonomy of statisticians
St ti ti l l i l ti• Statistical legislation
• Existence of an independent statistical board
D l t f d f d t• Development of codes of conduct
• Breaches of the code identified, investigated and publicised
• Appointment of senior statisticians removed from the political
process
U h ld b i l d i tti th d ( ki th• Users should be involved in setting the agenda (asking the
awkward questions)
• External audits of the statistical processes should be employed• External audits of the statistical processes should be employed
• Audit body should report to Parliament

Measurement vs. Quality
• Statisticians need to guard against “what can’t be measured isn’t
real”
• The danger with a measurement culture is that excessive attention
i i t h t b il d t th f h t iis given to what can be easily measured, at the expense of what is
difficult or impossible to measure quantitatively even though this
may be fundamentalmay be fundamental.

Challenges to integrity –
the rise of performance monitoring
• Performance data can be used in establishing 'what
orks' among polic initiati es to identif ell performingworks' among policy initiatives; to identify well-performing
or under-performing institutions and public servants; and,
equally important to hold Ministers to account for theirequally important, to hold Ministers to account for their
stewardship of the public services
H t i b th it i th bli i• Hence, government is both monitoring the public services,
and being monitored, by performance indicators.
• Because of government's dual role, performance
monitoring must be done with integrity and shielded from
undue political influence

Hitting the target but missing the point
htt // k/PDF/P f M it i df
http://www.rss.org.uk/PDF/PerformanceMonitoring.pdf

Audit Commission report
“What makes a target ‘good’ is not just the way a target isWhat makes a target good is not just the way a target is
expressed—it’s about the way it was derived, the extent
to which service users were involved in its developmentto which service users were involved in its development,
the extent to which it helps to achieve policy objectives,
the extent to which it has the support of the staff whosethe extent to which it has the support of the staff whose
efforts will achieve it, the quality of the data used to
measure its achievement and the clarity andmeasure its achievement, and the clarity and
transparency of its definition”

Pragmatism vs. Purism
• To what extent should we exploit data from a widerTo what extent should we exploit data from a wider
range of sources?
• May allow us to produce more timely data at lower cost
• Opportunities provided by BIG data

Fundamental changes to data sources might need
to involve review as to the nature of evidence
• Use of ‘free form’ data raises questions about how to
i h li d i i d i h hcommunicate the quality and uncertainty associated with the
evidence
• In the context of some moves towards greater formalisation of
evidence (such as randomised control trials)
It does not remo e the need for SCIENCE• It does not remove the need for SCIENCE

The use of big data brings challenges?
• Need programmes of work on the technical and analytic
challenges especially relating to data qualitychallenges especially relating to data quality
• But also on
• Communication and dissemination of statistics
• Culture of statistical agencies• Culture of statistical agencies
• Trust of the public
• Changing relationships with users and providers
• The responsibilities of official statisticians
• The meaning of privacy in this new world
• etc
• etc.

Develop statisticians for the future
• Foster adaptability
• Transferable skills
• Build research and innovation skills
• Create a cadre of people who challenge pre-Create a cadre of people who challenge pre-
conceptions
• Not to mould them in our own image
• Nor to create homogeneous communities• Nor to create homogeneous communities
• Education is about opening minds not closing them

Margins of Error: public understanding of statistics in an era of big data

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Margins of Error: public understanding of statistics in an era of big data (20)

More from Ipsos UK (20)

Recently uploaded (20)

Margins of Error: public understanding of statistics in an era of big data