nogago distributed bulk rendering
- 1. Distributed bulk rendering on the Amazon cloud Dr. Raphael Volz nogago GmbH, Gilching, Germany State of the map, Girona, July 9, 2010
- 2. Another 6 years into the project, the planet will have 5 TB (compressed). Hypothesis July 2010 10 GB July 2016 5 TB Distributed bulk rendering on the Amazon cloud
- 3. OSM is power law distributed with top 10% covering only 1 ‰ of planet Observation Top 100 6°x6° areas Size (Bytes GZ-compressed) # 1 / 1 GB West 6 South 48 # 2 / 0,6 GB West -84 South 30 6x6 splits by size Distributed bulk rendering on the Amazon cloud
- 4. Lower barriers of entry and increase the number of eyes ! How to achieve the 5 TB planet Distributed bulk rendering on the Amazon cloud
- 5. Ride the smartphone wave ! Smartphones in use (in Mio.) Source: Nogago analysis of Juniper Research, Gartner Spring 2009 forecasts Note: Assumption: Devices sold are in use for two years Distributed bulk rendering on the Amazon cloud
- 6. For example nogago outdoor is a smartphone app that accompanies your outdoor activities Plan Share activities experiences • View maps and • Download guides anytime • View tracks Maps and offline • View photos • Choose from • Record and (geo-referenced) 400.000 tracks follow Tracks • Share with • Plan new family, friends tracks Enjoy the or the world ! Outdoors ! Distributed bulk rendering on the Amazon cloud
- 7. Our customers will download offline maps for regions of choice Raster map POI DB Ways DB Max. 200 MB per 1°x1° Distributed bulk rendering on the Amazon cloud
- 8. Generating offline maps takes significant time and space Using the OSM map rendering tool chain… Get Load into Render XML planet PNG Database Maps osm.bz2 wget osm2pgsql mapnik … takes space and time … Zoom Level 15 1 Billion 5 621 (1:14000 @eq.) Tiles TB Days Zoom Level 18 68 Billion 350 109 (1:1700 @eq.) Tiles TB Years Assumptions: 16 KB per Tile on average, 20 Tiles per Second to render on modern hardware Distributed bulk rendering on the Amazon cloud
- 9. Basic approach to parallelization using Amazon cloud Render Image Nogago.com Ubuntu 10.04 Start Msg. Msg. Shut off Apache Instance started finished instance Postgres 8.4 User Amazon EC2 osm2pgsql Data osmosis mapnik Render Image TimKay‘s aws Get planet Load into Render Store Database Maps mapnik-OSM osm.bz2 on S3 Nogago tools Amazon EC2 runs images Rarely Changing Amazon S3 Content stores files Total: 3 GB Amazon S3 Note: UD = User Data, thanks to Eric Hammond (http://ec2-run-user-data.notlong.com) Distributed bulk rendering on the Amazon cloud
- 10. To make things fit consider corner cases Example: 1 x1 to be rendered at zoom level 10 1,352 x 1,352 to be loaded into database 190% overhead at zoom level 10 Data to load into Postgres Area to be rendered (depends on zoom levels) by mapnik Distributed bulk rendering on the Amazon cloud
- 11. Optimization: Do not always get planet Prepare chunks periodically • Two step approach, first: – Download planet, periodically 1 Day to process – Split into chunks and copy on S3 – Store chunks on EC2 • Then, many rendering images – Get appropriate chunk from S3 Max. – Load (parts of) chunk into Postgres 3 hours per – Render from Postgres 1 x1 Distributed bulk rendering on the Amazon cloud
- 12. What have we learned ? Rendering the world on zoom level 15 in 28 hours instead of 621 days ( 2 ‰ of the time) By distributing bulk rendering on the Amazon cloud Distributed bulk rendering on the Amazon cloud
- 13. EU FP7 Thank project partner you ! sought for mapping-related project proposal Distributed bulk rendering on the Amazon cloud