Big Data Visualization
2 likes1,231 views
This document discusses using visualization techniques to summarize and gain insights from large datasets, or "big data". It provides three examples: 1. A "tableplot" was used to visualize census data from 17 million records sorted by age to check data quality. 2. A "heat map" showed age versus income from a social security register containing 20 million monthly records to obtain insights. 3. Sentiment analysis of social media data was visualized at different granularities from daily to monthly and found to correlate highly (0.88) with consumer confidence surveys. The document concludes that big data is an interesting source for official statistics and visualization is an effective way to explore and validate such large datasets.
Big Data Visualization
- 1. Edwin de Jonge, December 3, 2013 Big Data Visualization “Turning Statistics into Knowledge”, Aguascalientes With thanks to Piet Daas, MartijnTennekes and Alex Priem
- 2. Overview 2 • Big Data • Research ‘theme’ at Stat. Netherlands • Data driven approach •Visualization as a tool •Why? •Examples in our office •Census •Social Security •Social Media •Not shown:Traffic loops, Mobile phone data
- 3. Why Visualization? October 1st 2013, Statistics Netherlands
- 4. Effective Display! (seeTor Norretranders, “Band width of our senses”)
- 5. Anscombes quartet… 5 DS1 x y DS2 x y DS3 x y DS4 x y 10 8.04 10 9.14 10 7.46 8 6.58 8 6.95 8 8.14 8 6.77 8 5.76 13 7.58 13 8.74 13 12.74 8 7.71 9 8.81 9 8.77 9 7.11 8 8.84 11 8.33 11 9.26 11 7.81 8 8.47 14 9.96 14 8.1 14 8.84 8 7.04 6 7.24 6 6.13 6 6.08 8 5.25 4 4.26 4 3.1 4 5.39 19 12.5 12 10.84 12 9.13 12 8.15 8 5.56 7 4.82 7 7.26 7 6.42 8 7.91 5 5.68 5 4.74 5 5.73 8 6.89
- 6. Anscombe’s quartet Property Value Mean of x1, x2, x3, x4 All equal: 9 Variance of x1, x2, x3, x4 All equal: 11 Mean of y1, y2, y3, y4 All equal: 7.50 Variance of y1, y2, y3, y4 All equal: 4.1 Correlation for ds1, ds2, ds3, ds4 All equal 0.816 Linear regression for ds1, ds2, ds3, ds4 All equal: y = 3.00 + 0.500x Looks the same, right?
- 7. Lets plot!
- 8. Visualization For Big Data: Use appropriate: - Summarization - Granularity - Noise filtering Research:What works for big data?
- 9. 9 Scatter plot with 100 data points
- 10. 10 Scatter plot with 100 000 data points
- 12. Example Virtual Census ‐ Every 10 years a Census needs to be conducted ‐ No longer with surveys in the Netherlands • Last traditional census was in 1971 ‐ Now by (re-)using existing information • Linking administrative sources and available sample survey data at a large scale • Check result • How? • With a visualisation method: the Tableplot 11
- 13. Making the Tableplot 1. Load file 17 million records 2. Sort record according to 17 million records key variable • Age in this example 3. Combine records 100 groups (170,000 records each) • Numeric variables • Calculate average (avg. age) • Categorical variables • Ratio between categories present (male vs. female) 4. Plot figure of select number of variables • Colours used are important up to 12 12
- 15. October 1st 2013, Statistics Netherlands tableplot of the census test file
- 16. Tableplot: Monitor data quality 16 – All data in Office passes stages: ‐ Raw data (collected) ‐ Preproccesed (technically correct) ‐ Edited (completed data) ‐ Final (removal of outliers etc.)
- 17. Processing of data Raw (unedited) data Edited data Final data
- 18. Example 2 : Social Security Register 15
- 19. Social Security Register – Contains all financial data on jobs, benefits and pensions in the Netherlands ‐ Collected by the DutchTax office ‐ A total of 20 million records each month ‐ How to obtain insight into so much data? • With a visualisation method: a heat map 19
- 20. October 1st 2013, Statistics Netherlands Heat map: Age vs. ‘Income’ 16 Age Income(euro)
- 21. 17 amount mount
- 22. 22 Example 3: Social media
- 23. Daily Sentiment in Dutch Social Media Social media: daily sentiment in Dutch messages 23
- 24. Granilarity: From day to week Social media, daily sentiment in Dutch messagesSocial media: daily & weekly sentiment in Dutch messages 24
- 25. Granularity: From day to month Social media, daily sentiment in Dutch messagesSocial media: daily, weekly & monthly sentiment in Dutch messages 25
- 26. Enter: Consumer confidence! Social media, daily sentiment in Dutch messagesSocial media: monthly sentiment in Dutch messages & Consumer confidence 26 Corr: 0.88
- 27. Conclusions Big data is a very interesting data source for official statistics Visualisation is a great way of getting/creating insight Not only for data exploration, but also for finding errors 27
- 28. The future of statistics?