Quality Approaches to Big Data
0 likes486 views
This document discusses quality approaches for big data in statistics. It outlines limitations of established quality frameworks for big data, including population not being known, unbalanced data coverage, and unclear relevance of data sources. Options presented to address these limitations include deriving background information, using modeling approaches, and calibration or correlation studies. The document advocates that statistical organizations validate information from other big data producers, get to know big data sources, use big data for efficiency and early indicators, and create an environment conducive to innovative big data approaches.
Quality Approaches to Big Data
- 1. Q2014 – Special Session Big Data Vienna, 4 June 2014 Quality Approaches to Big Data Peter Struijs and Piet Daas
- 2. 2 Limitations of the established quality frameworks and methodology Options What to do in the changing context of making statistics
- 3. Approaches and data sources Surveys / questionnaires e.g. sampling theory Administrative data sources Where does Big Data fit in? 3
- 4. Two levels of quality Quality as related to methodology General quality criteria as defined in Code of Practice: ‐ Relevance ‐ Accuracy and reliability ‐ Timeliness and punctuality ‐ Coherence and comparability ‐ Accessibility and clarity 4
- 5. 5 Limitations of the established quality frameworks and methodology
- 6. Small, medium-sized & large vehicles 22
- 7. 7 Figure 1. Development of daily, weekly and monthly aggregates of social media sentiment from June 2010 until November 2013, in green, red and black, respectively. In the insert the development of consumer confidence is shown for the identical period.
- 8. Daytime population based on mobile phone data
- 9. The top three issues 9 Population not known Unbalanced coverage Relevance of data not clear
- 10. 10 Options
- 11. Population not known 11 Derive background information Relate population at meso- or macro-level to other information
- 12. Unbalanced coverage 12 Use modeling approaches
- 13. Relevance of data not clear 13 Calibration / fitting Study correlations Use Big Data for “stand alone” information
- 14. 14 What to do in the changing context of making statistics
- 15. 15
- 16. Strategic aspects Others start producing statistics • there may be quality issues • but they are extremely rapid • and there is obviously demand Need for good, impartial information will remain • without a monopoly for NSIs NSIs must validate information produced by others 16
- 17. The way forward Get to know Big Data Use Big Data for efficiency and response burden reduction Use Big Data for early indicators Start with Big Data, not with the desired outcome Create the right environment 17
- 18. 18
- 19. 19