The Weather of the Century Part 3: Visualization
1 like1,424 views
The document discusses a presentation by Jesse Jiryu Davis on visualizing temperature data using MongoDB, Python, and various libraries such as NumPy and Matplotlib. It highlights the initial data extraction process, the implementation of the Monary library for faster querying, and future developments in data handling and schema adjustments. The talk emphasizes the efficiency of different methods in processing and visualizing large datasets.
![import numpy!
import pymongo!
!
data = []!
db = pymongo.MongoClient().my_database!
!
for doc in db.collection.find(query):!
data.append((!
doc['position']['coordinates'][0],!
doc['position']['coordinates'][1],!
doc['airTemperature']['value']))!
!
arrays = numpy.array(data)!](https://image.slidesharecdn.com/weatherofthecenturyvisualization-140805155041-phpapp02/85/The-Weather-of-the-Century-Part-3-Visualization-12-320.jpg)
![# NumPy column access syntax.!
lons = arrays[:, 0]!
lats = arrays[:, 1]!
temps = arrays[:, 2]!](https://image.slidesharecdn.com/weatherofthecenturyvisualization-140805155041-phpapp02/85/The-Weather-of-the-Century-Part-3-Visualization-13-320.jpg)
![import numpy!
import pymongo!
!
data = []!
db = pymongo.MongoClient().my_database!
!
for doc in db.collection.find(query):!
data.append((!
doc['position']['coordinates'][0],!
doc['position']['coordinates'][1],!
doc['airTemperature']['value']))!
!
arrays = numpy.array(data)!
Not terrifically fast](https://image.slidesharecdn.com/weatherofthecenturyvisualization-140805155041-phpapp02/85/The-Weather-of-the-Century-Part-3-Visualization-33-320.jpg)
![import monary!
!
data = []!
connection = monary.Monary()!
!
arrays = monary_connection.query(!
db='my_database',!
coll='collection',!
query=query,!
fields=['lon', 'lat', 'temp'],!
types=[!
'float32', 'float32', 'float32'])!](https://image.slidesharecdn.com/weatherofthecenturyvisualization-140805155041-phpapp02/85/The-Weather-of-the-Century-Part-3-Visualization-36-320.jpg)
![{!
ts: ISODate("1991-01-01T00:00:00Z"),!
position: {!
type: "Point",!
coordinates: [!
-94.6,!
39.117!
]!
},!
airTemperature: {!
value: 45,!
quality: "1"!
}!
}!
Original Schema](https://image.slidesharecdn.com/weatherofthecenturyvisualization-140805155041-phpapp02/85/The-Weather-of-the-Century-Part-3-Visualization-39-320.jpg)
The Weather of the Century Part 3: Visualization
- 1. A. Jesse Jiryu Davis #MongoDBWorld The Weather of the Century! Part III:! Visualization Senior Python Engineer, MongoDB
- 5. This Talk
- 6. Where’s the data from?
- 7. Where’s the data from?
- 8. How Much Is There?
- 9. Deployment
- 10. Visualization
- 11. Visualization Pipeline MongoDB PyMongo NumPy Matplotlib Python dicts SciPy
- 12. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
- 13. # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
- 15. from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)! Magic!! Also magic!!
- 17. from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
- 18. Triangulation
- 19. Triangulation
- 21. Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
- 23. Interpolation
- 24. Interpolation
- 25. Contours
- 26. Contours
- 33. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)! Not terrifically fast
- 34. Analyzing large datasets • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter“Monary”
- 35. MongoDB PyMongo NumPy Matplotlib Python dicts MongoDB Monary NumPy Matplotlib Monary by David Beach
- 36. import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=['lon', 'lat', 'temp'],! types=[! 'float32', 'float32', 'float32'])!
- 37. Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
- 38. Visualization
- 39. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }! Original Schema
- 40. {! ts: ISODate("1991-01-01T00:00:00Z"),! lon: 39.117,! lat: -94.6,! temp: 45! }! Target Schema
- 41. Future of Monary • Author: David Beach • Interns: Kyle Suarez Matt Cotter • Mentors: A.Jesse Jiryu Davis Jason Carey
- 42. Future of Monary • Subdocuments: "airTemperature.value" • Aggregation cursor • Packaging • Bugfixes • Python 3
- 43. Thanks • Monary • NumPy • SciPy • Matplotlib
- 44. Thanks
- 45. Thank you #MongoDBWorld A. Jesse Jiryu Davis Senior Python Engineer, MongoDB