OpenStreetMap in the age of Spark
- 1. OpenStreetMap in the age of Spark @adrianulbona
- 2. OpenSteetMap - The Wikipedia of Maps - https://www.openstreetmap.org - nodes: geo-localized points on the map - ways: roads, building contours, borders (multiple nodes), … - relations: highways (multiple ways), schools (building contours), ...
- 3. OpenSteetMap - the data The data is available at http://planet.openstreetmap.org and it comes in two formats: - XML ~ around 53 GB - PBF ~ around 34 GB Take a look at http://osmstats.neis-one.org, if interested on how the map evolves on a daily basis.
- 4. OpenSteetMap - story 1 - download the PBF/XML files - wait days to import the data in PostgreSQL - write some SQL - grab a coffee - grab a second coffee - see some query results - … - manage scary scripts that keep your OSM DB updated
- 5. OpenSteetMap - story 2 - download the PBF/XML files - extract various pieces of information in obscure CSVs - write MR jobs full of string parsing bugs - run the jobs - grab a coffee - fix your job - grab a second coffee - … - MR is not cool anymore
- 6. OpenSteetMap - story 3 - one day some weird guy comes and asks: what is the total road network length from OSM? - you have ways as collections node ids - you have nodes with ids and coordinates (latitude, longitude) - all this mixed-up in a huge protobuf file - options?
- 7. what the actual problem is?
- 8. we need parallel data access
- 9. we need the data structured
- 10. Spark will handle the rest
- 11. Parquet will give you more … performance
- 12. Apache Parquet Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language. google paper: Dremel - Interactive Analysis of Web-Scale Datasets twitter blog: Dremel Made Simple with Parquet
- 13. Apache Parquet row-based storage column-based storage data
- 14. Apache Parquet - protobuf-like - primitives, arrays, structs - bonus: nested structs
- 15. Parquet + Spark
- 16. osm-parquetizer - github.com/adrianulbona/osm-parquetizer - input: one OSM PBF file - output: one parquet file for each entity type (nodes, ways, relations) - minutes for countries like Romania - between 2 and 4 hours for the entire planet - parquet files size ~ 3 x original PBF (~ 100 GB the planet)