Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
0 likes424 views
The document provides an overview of cloud-native formats for optimizing storage, access, and processing of geospatial data in cloud environments, highlighting their scalability and efficient handling of large data volumes. It discusses various data formats, such as Cloud Optimized GeoTIFF, STAC, FlatGeobuf, COPC, Zarr, and GeoParquet, detailing their functionalities, benefits, and real-world applications. Key takeaways include the importance of tailored solutions for specific use cases and the evolving nature of cloud-native technologies in the geospatial field.
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
- 1. Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
- 2. Dean Hintz Technical Support Team Lead, Strategic Solutions Safe Software Kailin Opaleychuk Technical Support Specialist, FME Form Safe Software
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. Agenda 1 Introduction 2 What is Cloud Native? 3 Perspectives from Radiant Earth, Planet 4 STAC & COGs 5 FlatGeoBuf 6 COPC & Zarr 7 GeoParquet 8 Lessons learned 9 Q&A Agenda
- 5. Poll: What’s your leading motive for considering the use of cloud native?
- 7. W ● Cloud native formats = cloud-optimized ● Specifically designed to optimize the storage, access, and processing of geospatial data in cloud computing environments ● Supports data chunking, indexing, tiling and targeted metadata to minimize response footprint ● Optimize access by thin web clients, whether browser or mobile base ● Partial & Parallel reads ● Read just what you need What does Cloud-Native Mean? Introduction
- 8. ● Lazy access and intelligent subsetting ● Integrates well with high level analysis and distributed systems ● Scalable performance - increases the applicability of cloud-scale tools ● Decreases the burden and costs for data providers ● Tailored to leverage the scalability, flexibility, and processing power of cloud infrastructure, enabling efficient handling of large data volumes. Article: Cloud native data formats Benefits of Cloud-Optimized Data Introduction
- 9. Data Inspector COG Example: Canada DEM COG Canada DEM on S3: full width, lowest zoom by default Search envelope & CRS needed to limit request: ● CRS of dataset on server ● Search envelop CRS - same units as extent, can be different than source dataset ● Pyramid level to read. In this case level 1 = 30m resolution
- 10. Data Inspector COG Example: Canada DEM 2 seconds to access DEM for all of Canada from 31GB COG source dataset level 10 = 16 km x 16 km grid cells
- 11. Data Inspector COG Example: Canada DEM 4 seconds to access 30m DEM for all Fraser Valley from 31GB COG source dataset 31GB COG dataset - do not open in browser! https://datacube-prod-data-public.s3.ca-central-1.amazonaws.com/store/elevation/cdem-cdsm/cdem/cdem-canada-dem.tif
- 12. Part 1 Recap Chris Holmes VP of Product, Strategy, Partnerships Planet Michelle Roby Developer Advocate Radiant Earth
- 13. Cape Town, South Africa • March 19, 2017 Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine Cloud Native Geospatial Origins Chris Holmes
- 15. About Radiant Earth About: ● An incubator of data-driven initiatives, services, and 21st century institutions needed to foster shared understanding of our world Initiatives: ● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient approaches to working with geospatial data on the Internet. ● Source Cooperative → Data publishing utility for easy data sharing over the web.
- 16. What makes cloud-optimized challenging? From Task 51 Study: “There is no one-size-fits-all packaging for data, as the optimal packaging is highly use-case dependent.” Authors: Chris Durbin, Patrick Quinn, Dana Shum
- 17. New Cloud-Native Format Support Format Support Version Available Cloud Optimized Geotiff R / W 2023.0 Cloud Optimized Point Cloud R / W 2023.1 / 2023.2 (2024.0) FlatGeoBuf R / W 2023.0 GeoParquet R / W 2023.1 SpatioTemporal Asset Catalog (Metadata + Asset) R 2024.0 (FME Hub)* ZARR R / W 2023.1 (2024.0)
- 19. ● Spatio-Temporal Asset Catalog is a format that stores cloud-based assets that relate to a geographic area or time. ● The assets are templated in a JSON catalog/collection. ● Supports raster and vector assets ○ For example, a STAC Collection can have Assets that store geopackage layers or COG bands as asset items. STAC
- 20. STAC Package (FME Hub) - STAC Package V2.1.1 now available on the FME Hub. - STAC Metadata Reader* - STAC Asset Reader - V2.0.0 requires FME 24.0 minimum build 24134
- 21. STAC Metadata Reader Images demonstrating how to use the STAC Metadata Reader to dig down into a STAC Collection https://spot-canada-ortho.s3.amazonaws.com/catalog.json Catalog > Collection > Item > Asset
- 22. Slide Title Consume a GeoTIFF in STAC and convert to Cloud Optimized GeoTIFF Goal Key Result Working with STAC Asset Reader in FME Form Use the FME platform to refine and translate data from one location to another Output Cloud Optimized Geotiff ready for further analysis on S3
- 23. Demo
- 24. ● Use raster transformers to post-process STAC assets ○ Combining raster bands ○ Setting & removing no data ● FME’s S3Connector can publish COGs to the cloud Demo Summary Removing no data FME Form Workspace
- 25. Demo Results
- 26. FME & STAC Overview ● One set of item assets can be read or accessed by a single reader feature type ● STAC Metadata Reader can be used to filter and select assets of interest ● Coming Soon ○ The ability to access authentication required assets ○ Pre-defined popular STAC API options to improve usability
- 28. ● Supports raster data ● Built off the GeoTIFF specification, which offers functionality for compression and pyramiding ● Benefits from partial reads, through MinMax extents or clipping. COG
- 29. COG Reader ● Search Envelope ● Pyramid level options COG Writer ● Writer feature type ○ Compression ○ Layout: Cloud Optimized Tiles ○ Pyramid level options
- 30. COG Reader in FME Form https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif
- 31. COG Reader - Search Envelope Reading entire dataset Reading with Search Envelope constraint
- 32. Slide Title Create an insightful report on recent fires West of Kelowna Goal Key Result Current Fire Mapping for West Kelowna Use transformers to extract, combine & reformat data An interactive HTML report with embedded images and links
- 33. Demo
- 34. Demo Results
- 35. ● FlatGeoBuf and COG readers support spatial filter operations ● Use polygon mask to refine points on Nodata areas ● XMLTemplater can be used to help format HTML elements, such as tables Key Demo Takeaways
- 36. 4 FlatGeoBuf
- 37. ● Vector format built on Google’s Flatbuffers library ● A buffer is considered a file and everything within it. ● Although it is not required, FlatGeobuf uses indexing to help reduce the amount of data that would need to be transferred over a potentially slow network. FlatGeoBuf
- 38. FlatGeoBuf Reader ● Verify file buffers ● Search envelope FlatGeoBuf Writer ● Create spatial index
- 39. Slide Title Create a service that automatically uploads a range of vector data to S3 as FlatGeoBuf Goal Key Result FlatGeoBuf S3 Uploader App Generic Reader paired with user parameters Uploaded buffers and an upload html upload report
- 40. Demo
- 41. ● User parameters help make workspaces more dynamic ● PROJReprojector with online grids enabled Summary of FlatGeoBuf S3 Uploader App
- 43. ● Point cloud storage optimized for the web ● Based on the LAS standard ● Only read what you need. This is especially powerful for point clouds given 3d data data volumes can be huge ● Query XY min/max ● Essentially uses the LAS reader / writer but with the COPC structure COPC
- 44. ● Point cloud generated from drone imagery using dense point matching: ODM ● 1.1 million points ● Converted from LAS to COPC and loaded to S3 COPC - White Rock Pier Post Storm from Drone Survey
- 45. ● Uses the LAS reader / writer but with the COPC structure ● LAZ compression ● Select Write as Cloud Optimized Point Cloud ● Set CRS COPC Writing
- 46. ● Use S3Loader to upload COPC to S3 bucket ● Compressed vs uncompressed tradeoff COPC S3 Loader
- 47. COPC Reading: Convert PointCloud to Polygon Features 1. Read COPC from S3 2. Filter out ground points 3. Convert to raster 4. Classify raster 5. Convert raster to polygons 6. Filter out donuts & small polygons 7. Write result to geojson
- 48. Read only points close to pier COPC Reading - Extents filter
- 49. Convert pier point cloud to areas, calculate distance of collapse (47m) COPC Reading: Pier Polygons Written to GeoJSON
- 50. ● Point cloud storage optimized for the web ● 2024.0 fixes and enhancements (round trip) ● Extents query supported and optimized ● Coming: FeatureReader clipping by extents COPC Summary
- 51. 6 ZARR Format
- 52. ● Multidimensional raster array / time series storage optimized for the web ● Based on NetCDF / HDF data cube formats ● Only read what you need ● Particularly powerful for raster time series, as multidimensional arrays often mean huge volumes ● Query XY& band* extents ● Zarr reads cube with each time step as a separate band with properties - easy to work with ZARR
- 54. ● Time series raster storage optimized for the web ● Based on NetCDF data cube ● NetCDF reads cube as multigrid with 1 band for each time step (hundreds of bands) and properties in attribute lists ● Zarr reads cube with each time step as a separate band with properties - easier to work with ● Default translation from NetCDF to Zarr just works* NetCDF to ZARR
- 55. ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
- 56. ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
- 57. ZARR Climate Model Band Range Request: Python Parameters
- 58. ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
- 59. OGC Climate Resilience Pilot 2023 Pilot Goals: ● Build climate resilience ● Expand audience for climate services ● Demonstrate the value of OGC standards and SDI’s (FAIR) ● Show how OGC can support international climate change goals ● Build a community of stakeholders better understand the range of possible impacts - allows us to better prepare and compensate for them https://www.ogc.org/initiatives/crp/
- 60. How to provide the data needed for climate impact and disaster indicators to a wider audience? ● Goal: Connect Climate and Disaster Pilots ● Data: Current situational awareness ○ Base map: physical, land use, infrastructure, pop ○ EO data: hazards and impacts ○ Drought & hydrologic monitoring ● Data: Future change awareness - risk scenarios due to climate change ○ Climate model outputs - time series data cubes ○ Temperature, precipitation and moisture projections ○ Analysis Ready Data (ARD) model results summary ○ Climate services known in climate community but not well known or utilized across affected impact domains NetCDF from Environment Canada Disaster Pilot 2023: Disaster and Climate Data Sources to ARD & Impacts
- 61. MB Drought Risk: Combined Precip Temp Query OGC API Features Query Parameters: Start Year: 2020 End Year: 2060 BBox: -100.0,49.0,-96.0,50.5 Limit: 2,000,000 MinPeriodValue: 0 (PrecipDelta) MaxPeriodValue: 0.75 (PrecipDelta) MinTemp: 23C (Min Mean Monthly Temp) Find all time step points over the next 40 years for southern Manitoba where projections indicate: ● > 25% dryer than historical mean AND ● mean monthly temperature > 23C
- 62. MB Precipitation: Future Delta PrecipDelta = PrecipFuture / PrecipHistoricalMean / = Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
- 63. MB Drought Risk: Combined Precip Temp Output
- 64. ● Multidimensional raster array / time series storage optimized for the web ● Based on NetCDF / HDF data cube formats ● Only read what you need ● Zarr reads cube with each time step as a separate band ● Query XYextents ● Band range - supports max not min ● Be careful with feature cache ● Data Inspector refresh needed with stacked rasters: select range and then select cell again ZARR Summary
- 65. 7 GeoParquet
- 66. GeoParquet ● Cloud native / cloud friendly vector data storage ● Built on & follows Parquet standards ● Column oriented ● Highly optimized for accessing very large data volumes where you need access to a few columns and geometry, such as for analysis ● Benefits from a mature set of applications, libraries & tools available for Parquet ● Supports a range of geometries ● Not spatially indexed yet (use partitioning, duckDB)
- 67. Slide Title Optimize reading and analysis of published large vector dataset Goal Block Key GeoParquet reader performance demo Result Internet bandwidth and local processing limitations Structure data so you only read what you need Test case: Geoparquet is 2 - 3 X faster than other alternatives
- 68. Demo
- 69. Performance: Geoparquet vs OSM, Geopackage 1 millions records, select and spatially analyze 107k water areas
- 70. GeoParquet Partitioning Nested structure with folder by feature type and separate files for each value for selected attribute
- 71. GeoParquet Partitioning Only read the features with the feature type and values you want Nested structure with folders by feature type and separate files for each value for selected attribute
- 72. Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted OSM reporter* 23.2 60.4 38.1 Geopackage reporter* 1.2 102.8 14 GeoParquet reporter* 1.3 37.5 7.2 GeoParquet partitioned* 0.3 15.2 4.9 Performance: Geoparquet vs OSM, Geopackage *1 millions records, select and spatially analyze 100k water areas. Process time in seconds
- 73. ● Column oriented vector format ● Geoparquet test: 2-3X faster than others ● Cloud native for vector not as easy as for raster, point cloud ● Adds requirement for appropriate cataloging ● Additional speed improvements with more attribute level partitioning ● This addresses some of the debate around geoparquet as cloud native ● DuckDB with Geoparquet to improve cloud native performance Key Lessons GeoParquet
- 74. Other Cloud Data Stores: Cloud Databases … to name a few
- 76. ● Start publishing now! ● Keep the processing close to the data ● Minimize traffic footprint - select just what you need ● Leverage data side filtering, microservices, lazy evaluation ● Metadata: enrich and update ● Optimization strategy: transactions volume vs data volume, response time requirements ● Test! Especially your core usage scenarios Integration Strategies Key limitations & Integration Strategies
- 77. Considerations ● Heavier preprocessing, larger size required to structure and store data for optimized read ● Updates are a challenge - automation helps ● FME’s implementation based on third party libraries - collaboration for fixes, enhancements ● Newer cloud native formats: less data publicly available so far: COPC, ZARR ● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf ● Supporting infrastructure: duckDB for Geoparquet etc Key limitations & Integration Strategies
- 79. 9 Conclusions
- 81. Lessons Learned ● Cloud native is all about making it easy to publish data without a server, optimizing responses to web data requests: read just what you need! ● No one size fits all: each format has its strengths and limitations ● STAC: steeper learning curve, collections within catalogs and vice versa, implementations vary; security ● COG, COPC: perhaps the most intuitive - 2d and 3d arrays are just easier to manage. STAC/COG most data ● Vector - still evolving: FlatGeoBuf more effective in its cloud native support but newer, less widely adopted. Geoparquet has more tooling but design and support needed to make effectively cloud native ● ZARR - powerful but complex - as a very new and niche format, support is still growing
- 82. Summary ● Cloud native is all about making it easy to publish data without a server, optimizing responses to web data requests: read just what you need! ● Safe’s strategy is to track and support emerging standards across a range of data types so FME users can stay ahead of evolving web technologies ● FME allows you to integrate between hybrid environments as needed ● Keep the processing close to the data ● Minimize traffic footprint - reader filtering ● Open standards enable community-wide adoption and access ● No one size fits all - know your key requirements & test!
- 83. One platform, two technologies FME Form FME Flow Build and run data workflows Automate data workflows FME Flow Hosted Safe Software managed instance fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 84. 10 Resources
- 85. Resources ● Geospatial Cloud Native Overview ● Webinar: Cloud Revolution: Exploring the New Wave of Serverless Spatial Data ● Webinar: Leveraging FME for Cloud Native Databases ● Cloud Native Databases - Blog ● Radiant Earth Blog: Cloud Native Geospatial Solutions ● Cloud Native Geospatial Foundation & Slack channel ● OGC - Cloud Native Geospatial ● Source Cooperative ● Chris Holmes: FOSS4G NA 2023 | Towards a Cloud Native Spatial Data Infrastructure ● guide.cloudnativegeo.org ● Cloud Native Data Formats ● Safe’s Participation in OGC Pilots
- 87. Data Sources STAC / COG: ● catalogue.dataspace.copernicus.eu/stac/ ● cmr.earthdata.nasa.gov/stac/ ● planetarycomputer.microsoft.com/catalog ● usgs.gov/landsat-missions/landsat-collection-2 ● planetarycomputer.microsoft.com/api/stac/v1/colle ctions/sentinel-2-l2a ● https://planetarycomputer.microsoft.com/api/stac/ v1/collections/nrcan-landcover ZARR: ● https://console.cloud.google.com/marketplace/pro duct/noaa-public/cmip6 COPC: ● github.com/PDAL/data/tree/master/autzen ● copc.io/#example-data
- 88. 11 Next Steps
- 89. Next Steps ● Coming: ○ Blogs ○ Tutorials ● Community involvement: Cloud Native Geospatial Foundation, OGC ● Events: ○ cncf.io/events/ ○ safe.com/company-updates/2024-upcomi ng-events/ ○ ogcmeet.org/ ● New functionality: what are your priorities? ○ DuckDB? ○ ZARR band range? ○ Other STAC media types?
- 90. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 91. ClaimYour Community Badge ● Get community badges for watching webinars! ● fme.ly/WebinarBadge ● Today’s code: SLMWB Join the Community today!
- 92. 12 Q&A
- 93. ThankYou Recap of Next Steps 1 Contact us: info@safe.com 2 Check out the new landing page to get started! Please fill out our webinar survey