We trained a neural network on satellite imagery to predict wealth from space.
2 likes2,172 views
The document discusses a training model using satellite imagery and census data to predict income levels from space, employing a neural network (ResNet 50) for analysis. It highlights the model's performance confidence in various income quartiles and addresses the need for better understanding of the model's internal workings. Additionally, it raises questions about potential applications and considerations for different cities and seasonal variances.
We trained a neural network on satellite imagery to predict wealth from space.
- 1. What does money ⨠look like from space?⨠(here comes the neighborhood)
- 2. The Training Model Satellite Imagery From GBDX
- 3. The Training Model Census Tracts (2013)
- 4. Quartiles The Training Model Quartile 0 < $34,176 Quartile 1 $34,177 - $49,904 Quartile 2 $49,905 - $71,875 Quartile 3 $71,876+
- 5. Satellite Imagery & Census Data The Training Model
- 7. Centroid Outlines The Training Model
- 8. Satellite Imagery & Census Data The Training Model (Resnet 50)
- 9. Satellite Imagery & Census Data Neural Network The Training Model (Resnet 50)
- 10. Satellite Imagery & Census Data Output = Neural Network The Training Model (Resnet 50) Q0 = 91%⨠Q1 = 5.64%⨠Q2 = 2.55%⨠Q3 = .41%
- 11. Satellite Imagery & Census Data Output = Neural Network The Training Model (Resnet 50) The model carries⨠what it has learned and ⨠repeats the process. Q0 = 91%⨠Q1 = 5.64%⨠Q2 = 2.55%⨠Q3 = .41%
- 12. The Training Model (Resnet 50) Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution Neural Network The data is fed into the model.
- 13. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network A number of ďŹlters are applied to the image.
- 14. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network The resulting new values are normalized to be within learned mean and standard deviations of the dataset.
- 15. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Separates out features from important and non important ones.
- 16. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Repeat.
- 17. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Results are merged with previous iteration.
- 18. The Training Model (Resnet 50) TensorBoard data from CMU
- 19. So now you have a ClassiďŹer
- 22. Satellite Imagery & Census Data
- 23. What is the classiďŹer seeing?
- 24. NYC Areas ClassiďŹed as Q0
- 25. NYC Areas ClassiďŹed as Q1 & Q2
- 26. NYC Areas ClassiďŹed as Q3
- 27. ClassiďŹcation ConďŹdence⨠97% Q0⨠3% Q1 0% Q2⨠0% Q3
- 28. ClassiďŹcation ConďŹdence⨠92% Q0⨠7% Q1 0% Q2⨠0% Q3
- 29. ClassiďŹcation ConďŹdence⨠91% Q0⨠5% Q1 2% Q2⨠.4% Q3
- 30. ClassiďŹcation ConďŹdence⨠2% Q0⨠2% Q1 6% Q2⨠90% Q3
- 31. ClassiďŹcation ConďŹdence⨠0% Q0⨠.1% Q1 0% Q2⨠99% Q3
- 32. âDespite this encouraging process, there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance. From a scientiďŹc standpoint, this is deeply unsatisfactory. Without clear understanding of how and why they work, the development of better models is reduced to trial-and-error.â Visualizing and Understanding Convolutional Networks - Matthew D. Zeiler, Dept. of Computer Science, Courant Institute, New York University - Rob Fergus, Dept. of Computer Science, Courant Institute, New York University
- 33. Baseball Field Experiment ConďŹdence⨠91% Q0⨠5% Q1 2% Q2⨠.4% Q3 Original Image
- 34. Baseball Field Experiment ConďŹdence⨠79% Q0 (-12)⨠12% Q1 (+7) 7% Q2 (+5)⨠.8% Q3 (+.4) Added Trees
- 35. Baseball Field Experiment ConďŹdence⨠63% Q0 (-28)⨠19% Q1 (+14) 13% Q2 (+11)⨠3% Q3 (+2.6) Added More Trees
- 36. Baseball Field Experiment ConďŹdence⨠68% Q0 (-23)⨠17% Q1 (+12) 11% Q2 (+9)⨠2% Q3 (+1.6) And Added More Trees
- 37. Baseball Field Experiment ConďŹdence⨠44% Q0 (-47)⨠12% Q1 (+7) 23% Q2 (+21)⨠18% Q3 (+17.6) Added All The Trees
- 38. Tree Experiment ConďŹdence⨠79% Q0⨠10% Q1 5% Q2⨠5% Q3 Original Image
- 39. Tree Experiment ConďŹdence⨠79% Q0⨠10% Q1 5% Q2⨠5% Q3
- 40. Tree Experiment ConďŹdence⨠77% Q0 (-2)⨠10% Q1 (0) 6% Q2 (+1)⨠5% Q3 (0)
- 41. Tree Experiment 2 ConďŹdence⨠89% Q0⨠10% Q1 0% Q2⨠0% Q3 Original Image
- 42. Tree Experiment 2 ConďŹdence⨠89% Q0⨠10% Q1 0% Q2⨠0% Q3
- 43. Tree Experiment 2 ConďŹdence⨠87% Q0 (-2)⨠11% Q1 (+1) 0% Q2 (0)⨠0% Q3 (0)
- 44. Trump Tower Experiment ConďŹdence⨠0% Q0⨠.1% Q1 0% Q2⨠99% Q3 Original Image
- 45. Trump Tower Experiment ConďŹdence⨠16% Q0 (+16)⨠25% Q1 (+24.9) 11% Q2 (+11)⨠45% Q3 (-54) Added Grass
- 46. Trump Tower Experiment ConďŹdence⨠34% Q0 (+34)⨠47% Q1 (+46.9) 6% Q2 (+6)⨠10% Q3 (-89) Built a Wall
- 47. Conclusions: ⢠It's possible to predict income levels from space ⢠Underlying data can provide valuable assistance to complex neural networks ⢠Human-based empirical inquiry has legs ⢠Teasing out why it knows what it knows is interesting
- 48. Questions: ⢠What does this thing do when you point it at other cities? ⢠What are the similarities and differences between cities from space? ⢠Can we construct a model to account for seasonal variance? ⢠Can we construct a model to account for architectural difference?
- 49. cmu.edu Who stamen.com amantiwari.comgbdx.geobigdata.io Carnegie MellonStamen Design Aman TiwariDigital Globe