Q4 2016 GeoTrellis Presentation
5 likes977 views
The document provides an overview of Geotrellis, a Scala library designed for handling geospatial data and operations, which integrates with Apache Spark for scalable processing. It covers key features, such as storage and querying capabilities, as well as its development history, future roadmap, and potential use cases. Additionally, it discusses the integration of raster and vector data and how to process geospatial data at scale.
![on
s3 key layerName/zoom/[SFC Index (Hilbert or Z order)]
s3 value
Avro Encoded Seq[(K, V)] where
K = Key Type (e.g. SpatialKey)
V =Value Type (e.g.Tile)](https://image.slidesharecdn.com/geotrellisslides-161103220506/85/Q4-2016-GeoTrellis-Presentation-115-320.jpg)
Q4 2016 GeoTrellis Presentation
- 2. What we’ll be covering What GeoTrellis is, what it can do Demo of GeoTrellis in action Talk about what the next steps for GeoTrellis are, look into some of the possible use cases for GeoTrellis that we’re excited about, and talk about our roadmap Feel free to ask questions throughout!
- 13. 2011 - 2013
- 14. 2013 - Present
- 16. What is ?
- 17. GeoTrellis a Scala library for geospatial data types and operations. enables Spark with geospatial capabilities storage and query raster data from HDFS, S3, Accumulo, and Cassandra (HBase soon)
- 18. Geo + Rasters +
- 19. Rasters, someVector + v1.0 Q4 2016
- 20. Rasters,Vector, VectorTiles, Point Cloud + ROADMAP v1.1 w/
- 21. Vector Data with GeoTrellis (non-Spark) Wraps JTS GeoJson,WKT,WKB reading/writing Reprojection (Proj4j) Kriging Interpolation
- 22. Rasters with GeoTrellis (non-Spark) Read GeoTiffs Map Algebra (local, focal, zonal) Polygonal Summaries Generally transform and combine raster data Kernel Density, rasterization, vectorization Get histograms Render via color breaks
- 23. GeoTrellis & Spark Ingest data to local file system, HDFS,Accumulo, S3, or Cassandra Distributed computations of Spatial and Spatio- temporal raster data Map algebra on distributed tile sets General ways to transform and combine distributed tile sets
- 24. BACKGROUND
- 25. PROCESSING GEOSPATIAL DATA @ SCALE
- 26. PROCESSING GEOSPATIAL DATA @ SCALE
- 27. Geospatial Data Core of GIS (Geographic information system) Raster (images, weather data) Vector (points of interest, country boundries)
- 28. Geospatial Data Core of GIS (Geographic information system) Raster (images, weather data) Vector (points of interest, country boundries) VectorTiles, Point Cloud
- 29. Raster Data
- 30. Raster Data
- 31. Raster Data
- 32. Raster Data
- 35. Vector Data (Polygons) Source: https://ryouready.wordpress.com/2009/11/16/infomaps-using-r-visualizing-german-unemployment-rates-by-color-on-a-map/
- 36. Vector Data
- 37. PROCESSING GEOSPATIAL DATA @ SCALE
- 38. Contains
- 39. Contains
- 41. Zonal Statistics
- 42. Feature Extraction (Image Segmentation) Source: http://www.professeurs.polymtl.ca/christopher.pal/
- 43. Map Algebra
- 44. Local Operation
- 45. Focal Operation
- 46. Map Algebra in GeoTrellis
- 53. PROCESSING GEOSPATIAL DATA @ SCALE
- 54. NED 1/3 arc second
- 55. NED 1/3 arc second
- 56. NED 1/3 arc second
- 57. NED 1/3 arc second
- 58. NED 1/3 arc second
- 59. • 170 X 180 km • 2gb each. • 11 bands • 700 scenes per day • 1.4 TB / day • 255,500 scenes / year • 0.25 PB / year Landsat 8
- 60. Landsat 8 on • All Landsat 8 scenes from 2015 and beyond. • Selection of cloud-free scenes from 2013 and 2014.
- 61. Landsat 8 on 645,763 scenes
- 62. Landsat 8 on ≈1 Petabyte
- 66. 64 GB
- 67. 32 Landsat 8 Scenes
- 69. This many people’s phones could hold all the Landsat 8 AWS is holding.
- 70. PROCESSING GEOSPATIAL DATA @ SCALE
- 72. Project to build a better search engine, back in the early 2000’s. Worked for small datasets, but was not scalable.
- 74. After reading the papers, Nutch developers added a distributed file system and MapReduce model to Nutch. In 2006, those portions were spun out of Nutch to form…
- 76. Apache Hadoop Heavily supported byYahoo, which moved it’s large data processing to Hadoop. by 2007,Twitter, Facebook, LinkedIn and many others were doing serious work with Hadoop 2008 Hadoop graduated to a top level Apache project
- 78. Matei Zaharia Worked with Hadoop at UC Berklee Noticed Hadoop was not a good fit for Machine Learning algorithms and other iterative models. So in 2009, he created…
- 81. Open sourced in 2010 under BSD license Maintained by UC Berkeley’s AMPLab Donated to the Apache Software Foundation in 2013 and relicensed as Apache 2.0 Graduated to a top level Apache project in 2014 Apache Spark
- 82. Apache Spark a distributed computation engine. An API that lets you work with distributed data as a collection. Written in Scala, with language bindings for use with Java, Python, and R.
- 85. Hey Flyers Fans, what is the total count of Landsat 8 Scenes on your phones?
- 87. Data Node Data Node Data Node Name Node Master Tablet Server Tablet Server Tablet Server Accumulo BigTable clone (columnar database) Records stored on HDFS Lexicographically sorted table index
- 89. Apache Accumulo Created by the NSA in 2008 Donated to the Apache Foundation in 2011 Graduated to a top level project in 2012
- 90. 2006
- 92. (Sec. 929) Prohibits any DOD component from utilizing the cloud computing database developed by the National Security Agency (NSA) and known as "Accumulo" after the end of FY2013, unless the DOD CIO certifies that: (1) there are no viable commercial open source databases that have such security features, or (2) Accumulo itself has become a successful open source database project. Requires DOD and intelligence community officials to coordinate the use by DOD components of cloud computing infrastructure and services offered by the intelligence community for purposes other than intelligence analysis.
- 93. (Sec. 929) Prohibits any DOD component from utilizing the cloud computing database developed by the National Security Agency (NSA) and known as "Accumulo" after the end of FY2013, unless the DOD CIO certifies that: (1) there are no viable commercial open source databases that have such security features, or (2) Accumulo itself has become a successful open source database project. Requires DOD and intelligence community officials to coordinate the use by DOD components of cloud computing infrastructure and services offered by the intelligence community for purposes other than intelligence analysis.
- 94. Hey Flyers Fans, what is the total count of Landsat 8 Scenes on your phones per month?
- 95. PROCESSING GEOSPATIAL DATA @ SCALE
- 96. Hey Flyers Fans, can you take the average pixel value of each scene’s band and derive a EPSG:3857 tile set of PNGs to be served on web maps?
- 97. Hey Flyers Fans, can you take the average pixel value of each scene’s band and derive a EPSG:3857 tile set of PNGs to be served on web maps?
- 99. How does work?
- 102. Polygonal Summaries
- 103. Polygonal Summaries
- 104. SPACE FILLING CURVES
- 105. Hey Flyers Fans, what is the total count of Landsat 8 Scenes on your phones per month, per country?
- 106. Hey Flyers Fans, what is the total count of Landsat 8 Scenes on your phones per country?
- 107. SPACE FILLING CURVES
- 108. Z curve
- 109. Hilbert Curve
- 110. Space Filling Curves
- 112. Range Decomposition 70 -> 75 92 -> 99 116 -> 121
- 114. on
- 115. on s3 key layerName/zoom/[SFC Index (Hilbert or Z order)] s3 value Avro Encoded Seq[(K, V)] where K = Key Type (e.g. SpatialKey) V =Value Type (e.g.Tile)
- 116. Hey Flyers Fans, what is the total count of Landsat 8 Scenes on your phones A) per month, B) per country, C) per both?
- 117. Why ?
- 118. Sharding raster data across the cluster Caching operation results across cluster HDFS support Advanced fault tolerance Advanced task scheduling
- 119. Source: http://cs.calvin.edu/courses/cs/374/exercises/12/lab/MapReduceWordCount.png
- 120. Say we have a large set of imagery, and would like to apply two filters to each band: First, we want to apply a simple threshold filter: if a value is above 10,000, we want to discard it Second, we would like to apply a 5 x 5 median filter. Example Problem: Filtering
- 124. (1, 1) (2, 1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2)
- 125. (1, 1) (2, 1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 126. (1, 1) (2, 1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 127. (1, 1) (2, 1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 128. (1, 1) (2, 1)(0, 1) Node 1 Node 2 Node 3
- 129. (1, 1) (2, 1)(0, 1) Node 1 Node 2 Node 3
- 130. (c, r)
- 131. Example Problem: Querying We want to retrieve all imagery for the city of Rio de Janeiro taken in March 2016, find the maximum NDVI values for each pixel and save it as a GeoTiff.
- 133. What are uses of ?
- 135. 100 spot instance m3.xlarge workers @ $0.04 / hr = $4.00 / hr 400 CPUs / ≈1.5 TB memory 1 master m3.xlarge on-demand instance @ $0.26 / hr EMR cluster charge, $0.07 / hr $4.37 / hr Rendering elevation with hillshade + NLCD on AWS EMR
- 138. NED 1/3 arc second + NLCD
- 139. NED 1/3 arc second + NLCD
- 140. NED 1/3 arc second + NLCD
- 141. GLOBAL CIRCULATION MODELS Models for predicting world temperature and precipitation.
- 143. NASA NEX Downscaled Climate Projections (NEX-DCP30) • Monthly data over conterminous US • Historical from 1950 - 2006 • 4 RCP scenarios from 2006 - 2099 • 8190 netCDF files on S3 - s3://nasanex/NEX-DCP30 • 15.3 TB in compressed GeoTiff tiles. • RCP 8.5, max for datatype/model combo: 90.92 GB
- 145. Landsat NDVI/NDWI change detection demo
- 146. Static vs Dynamic
- 147. serving static data pre- processed through a batch transformation pipeline vs serving data dynamically processed on-demand from unprocessed source data
- 148. Static vs Dynamic GeoTrellis systems tend to have two major components: A batch pre-processing pipeline, which processes large amounts of data into some static data at rest. A dynamic pipeline which processes data at the time the user requests it.
- 150. “Raw” Data Served Data Completely dynamic Application Data Processing at request time
- 151. “Raw” Data Served Data Completely static Batch data pre-processing Application Data
- 152. “Raw” Data Served Data Application Data Mix of static and dynamic Batch data pre-processing Processing at request time
- 153. “Raw” Data Served Data Application Data Mix of static and dynamic Ingest/ETL Server
- 154. “Raw” Data Served Data Application Data More static Faster to serve, less flexibility
- 155. “Raw” Data Served Data Application Data More dynamic More flexible, slower to serve
- 156. Ingesting Landsat data Landsat images are pulled off of S3 or Google’s public Earth Engine storage. In an Spark job run on EMR, these images are reprojected, tiled, indexed, and saved off to Accumulo or HDFS. The indexed tile set is now ready to be used by the server application.
- 157. Landsat GeoTiffs on S3 PNGs, JSON EPSG:3857 tiled imagery in Accumulo Ingest/ETL Server
- 158. Landsat GeoTiffs on S3 PNGs, JSON EPSG:3857 tiled imagery in Accumulo Ingest/ETL Server
- 159. Landsat GeoTiffs on S3 PNGs, JSON EPSG:3857 tiled imagery in Accumulo Ingest/ETL Server
- 160. DEPLOYMENT
- 161. Example Deployment
- 164. ROAD MAP
- 165. Release Schedule v1.0 Q4 2016 v1.1 Q2 2017 Graduation
- 166. Rasters,Vector, VectorTiles, Point Cloud + ROADMAP v1.1 w/
- 169. DOCUMENTATION!
- 172. Integration work
- 174. POINT CLOUD