Heatmaps best practices Strata Hadoop
0 likes2,073 views
The document discusses the use of income tax data in the Netherlands for analyzing patterns and meta patterns. It details the challenges and methods for visualizing complex distributions and highlights the importance of careful visualization practices when interpreting large datasets. The authors, statistical consultants from Statistics Netherlands, emphasize the value of heatmaps as a tool while cautioning against biases and the potential impact of outliers on analysis.
![20
1. Take two numerical variables x and y
2. Determine range 푟푥=[min푥,max(푥)]
3. Chop 푟푥 in 푛푥 equal pieces
4. Repeat for y
5. We now have 푛푥.푛푦 bins
6. Count # records in each bin
7. Assign colors to counts
8. Plot matrix
9. Enjoy!](https://image.slidesharecdn.com/heatmapsstratahadoop-141122044021-conversion-gate02/85/Heatmaps-best-practices-Strata-Hadoop-20-320.jpg)
![23
1. Take two numerical variables x and y
2. Determine range 퐫퐱=[퐦퐢퐧퐱,퐦퐚퐱(퐱)]
3. Chop 푟푥 in 푛푥 equal pieces
4. Repeat for y
5. We now have 푛푥.푛푦 bins
6. Count # records in each bin
7. Assign colors to counts
8. Plot matrix
9. Enjoy!](https://image.slidesharecdn.com/heatmapsstratahadoop-141122044021-conversion-gate02/85/Heatmaps-best-practices-Strata-Hadoop-23-320.jpg)
![28
1. Take two numerical variables x and y
2. Determine range rx=[minx,max(x)]
3. Chop 풓풙 in 풏풙 equal pieces
4. Repeat for y
5. We now have 푛푥.푛푦 bins
6. Count # records in each bin
7. Assign colors to counts
8. Plot matrix
9. Enjoy!](https://image.slidesharecdn.com/heatmapsstratahadoop-141122044021-conversion-gate02/85/Heatmaps-best-practices-Strata-Hadoop-28-320.jpg)
![38
1. Take two numerical variables x and y
2. Determine range rx=[minx,max(x)]
3. Chop 푟푥 in 푛푥 equal pieces
4. Repeat for y
5. We now have 푛푥.푛푦 bins
6. Count # records in each bin
7. Assign colors to counts
8. Plot matrix
9. Enjoy!](https://image.slidesharecdn.com/heatmapsstratahadoop-141122044021-conversion-gate02/85/Heatmaps-best-practices-Strata-Hadoop-38-320.jpg)
![42
1. Take two numerical variables x and y
2. Determine range rx=[minx,max(x)]
3. Chop 푟푥 in 푛푥 equal pieces
4. Repeat for y
5. We now have 푛푥.푛푦 bins
6. Count # records in each bin
7. Assign colors to counts
8. Plot matrix
9. Enjoy!](https://image.slidesharecdn.com/heatmapsstratahadoop-141122044021-conversion-gate02/85/Heatmaps-best-practices-Strata-Hadoop-42-320.jpg)
Heatmaps best practices Strata Hadoop
- 1. Alex Priem (@_alex_priem_) Edwin de Jonge (@edwindjonge) Strata, 21 nov 2014, Barcelona Patterns and meta patterns in Income Tax Data
- 2. Age vs mortgage debt (men)
- 3. Who are we? Statistical consultants / Data scientists working @ R&D department of Statistics Netherlands Statistics Netherlands (SN): -Government agency -Produces all official statistics of The Netherlands 3
- 4. Income statistics based on Tax data 4
- 5. Income Tax data –Contains all income tax records for the Netherlands –Approx 17M records with 550 variables. –Used to produce income statistics! Analysis is not trivial –Income Tax is complex (at least in the Netherlands) ‐stages of progressive tax ‐Complex Tax deductions (mortgage, flex workers) ‐Complex Tax benefits (child care, social benefits) 5
- 6. Tax data (2) -550 variables (for each person in NL): -15 identificators/unique keys -Dwelling, person id, etc. -70 categorical -250 numerical variables from the income tax form ->200 derived variables (useful for analysis) -E.g. expandable income, income of dwelling/household 6
- 7. Income/tax distributions Income (re)distribution hot topic since Piketty So how are income/tax/benefits distributed? -Look at 1D distributions: histograms -Look at 2D distributions: heatmaps -Problem: potentially 0.5 n(n-1) > 100k heatmaps! -even more when categorical included 7
- 8. Let look at Patterns… 8
- 9. Heatmap Patterns –What defines a pattern in heatmap? ‐Peak/Spike? (mode, 0D point) ‐Stripe (1D): •Horizontal Line? •Vertical Line? •Band? •Ridge? ‐Blob (2D) ‐Similarity between distributions (2D) 9
- 10. Meta pattern? Meta patterns constitutes of repeating pattern in: ‐different subpopulations •E.g. Male/female, Social economic status, Works in branch of Industry ‐different pairs of variables •Income x age •Benefits x age •Etc. So patterns that are generic over different heatmaps. 10
- 11. Looking for patterns Subpopulations: – Generate heatmap per category e.g. Age x Gross Income per social economic status –Automatic cluster heatmaps on distribution simularity Pairs of variables: -Generate heatmaps for all pairs -Prune: remove heatmaps with low support 1. Use image classification to cluster them 2. Or Cluster on extracted mode/line (wip) You will still need to look at the result! 11
- 13. Anscombes quartet… 13 DS1 x y DS2 x y DS3 x y DS4 x y 10 8.04 10 9.14 10 7.46 8 6.58 8 6.95 8 8.14 8 6.77 8 5.76 13 7.58 13 8.74 13 12.74 8 7.71 9 8.81 9 8.77 9 7.11 8 8.84 11 8.33 11 9.26 11 7.81 8 8.47 14 9.96 14 8.1 14 8.84 8 7.04 6 7.24 6 6.13 6 6.08 8 5.25 4 4.26 4 3.1 4 5.39 19 12.5 12 10.84 12 9.13 12 8.15 8 5.56 7 4.82 7 7.26 7 6.42 8 7.91 5 5.68 5 4.74 5 5.73 8 6.89
- 14. Anscombe’s quartet Property Value Mean of x1, x2, x3, x4 All equal: 9 Variance of x1, x2, x3, x4 All equal: 11 Mean of y1, y2, y3, y4 All equal: 7.50 Variance of y1, y2, y3, y4 All equal: 4.1 Correlation for ds1, ds2, ds3, ds4 All equal 0.816 Linear regression for ds1, ds2, ds3, ds4 All equal: y = 3.00 + 0.500x Looks the same, right?
- 15. Lets plot!
- 16. Machine learning So clustering (machine learning) different? 16
- 17. 17
- 18. Visualization helps to … –Test your (hidden model) assumptions! – To find structure in data, e.g. “How is my data distributed?” –Visually explore patterns: ‐Are there clusters? ‐Are there outliers? 18
- 20. 20 1. Take two numerical variables x and y 2. Determine range 푟푥=[min푥,max(푥)] 3. Chop 푟푥 in 푛푥 equal pieces 4. Repeat for y 5. We now have 푛푥.푛푦 bins 6. Count # records in each bin 7. Assign colors to counts 8. Plot matrix 9. Enjoy!
- 21. Easy as pie? Best practices and problems with heatmaps: -Resolution -Rescaling -Zooming -Outliers -Color scales 21
- 22. 22
- 23. 23 1. Take two numerical variables x and y 2. Determine range 퐫퐱=[퐦퐢퐧퐱,퐦퐚퐱(퐱)] 3. Chop 푟푥 in 푛푥 equal pieces 4. Repeat for y 5. We now have 푛푥.푛푦 bins 6. Count # records in each bin 7. Assign colors to counts 8. Plot matrix 9. Enjoy!
- 24. Range: Outliers? (1D) 24 +5M€ -1M€ Gross Income
- 25. Range: outliers removed (1% removed) 25 Gross Income +150k€
- 26. Range: outliers… Does your data contain outliers? -If so: most pixels are empty -Most cases: outliers have low mass and are barely visible Truncate range: in x or y direction: e.g. 99% quantile -Interactively: allow for zoom and pan. 26
- 27. Range: data skewed? 27 –Many variables are not normal distributed: ‐Power law: 풙훼 ‐Exponential: 푒푎풙+푏 So rescale x or y or both
- 28. 28 1. Take two numerical variables x and y 2. Determine range rx=[minx,max(x)] 3. Chop 풓풙 in 풏풙 equal pieces 4. Repeat for y 5. We now have 푛푥.푛푦 bins 6. Count # records in each bin 7. Assign colors to counts 8. Plot matrix 9. Enjoy!
- 29. Chop: AKA “Binning” 29
- 30. 30 Chop: resolution Resolution matters
- 31. 31 25 x 25
- 32. 32 50 x 50
- 33. 33 100 x 100
- 34. 34 250 x 250
- 35. 35 500 x 500
- 36. Chop: Too small / Too big If #bins too small: - patterns are hidden If #bins too large: - heatmap is noisy (signal vs noise) Optimal nr bins depends on data. (kernel based approx), but always play with bin size / resolution! 36
- 38. 38 1. Take two numerical variables x and y 2. Determine range rx=[minx,max(x)] 3. Chop 푟푥 in 푛푥 equal pieces 4. Repeat for y 5. We now have 푛푥.푛푦 bins 6. Count # records in each bin 7. Assign colors to counts 8. Plot matrix 9. Enjoy!
- 39. Count: zero counts Not every variable is relevant for each person! 39
- 40. Count: exclude zero values 40
- 42. 42 1. Take two numerical variables x and y 2. Determine range rx=[minx,max(x)] 3. Chop 푟푥 in 푛푥 equal pieces 4. Repeat for y 5. We now have 푛푥.푛푦 bins 6. Count # records in each bin 7. Assign colors to counts 8. Plot matrix 9. Enjoy!
- 43. 43
- 44. Colors: scales –Color ‘intensity’ implies value –Percieved response depends on ‘color’ and ‘color lightness’ (compare #00ff00 with #0000ff) –Different models for color response: ‐RGB (models computer monitor) ‐HSV ‐HCL ‐CIELAB (models human eye) –Gradient generator: http://davidjohnstone.net/pages/lch-lab-colour-gradient-picker 44
- 45. Colors –Color has two functions in heatmap: ‐Show ‘counts’ in your data ‐Show ‘patterns’ At least, use a perceptually uniform gradient -Libs: chroma.js, colorbrewer (R) …but patterns need distinct colors 45
- 46. Color scales –Range of color scale depends on distribution of data. –Often have multiple populations/distributions in data –Severe spikes/stripes drown the smaller distributions: ‐We suggest log scale ‐Sometimes log scale is not enough –In practice, linear scale with low maximum cut-off works well –Effect is best understood in 3D (!). 46
- 47. Peaks are best cut-off 47
- 48. Example: Linear gradient 48
- 49. Log-gradient 49
- 50. Linear gradient with cut-off 50
- 53. Heatmaps side-by-side: gross income, men vs women 53 men
- 54. Meta pattern Meta patterns constitutes of repeating pattern in: ‐different subpopulations ‐different pairs of variables So patterns that are generic over different heatmaps. 54
- 55. Heatmaps decomposed in subpopulations: 55
- 56. Gross income by socioeconomic status 56
- 57. Gross income, men, categorized by socioeconomic status 57
- 58. Patterns –Stripes are real, not outliers: –Corresponds with benefits, tax breaks –Needs paradigm shift: data is not normally distributed (but we knew that). 58
- 59. Meta pattern Meta patterns constitutes of repeating pattern in: ‐different subpopulations ‐different pairs of variables So patterns that are generic over different heatmaps. 59
- 60. Image classification of heatmaps 60
- 61. No Domain knowledge required? 61
- 62. 62
- 63. Salary pay structure 63
- 64. Domain knowledge, take II 64
- 65. Pattern removal: Effect of weighting 65
- 66. Summary Heatmaps: –ideal tool for analyzing big datasets –Be aware of perceptual and data biases! 66
- 67. Questions? Thank you for your attention! More info? ah.priem@cbs.nl / @_alex_priem e.dejonge@cbs.nl / @edwindjonge Heatmapping code available at https://github.com/alexpriem/heatmapr 67