Flood and Landslide Impact Components for the OGC 2021 Disaster Pilot using FME
0 likes197 views
The document outlines the OGC 2021 Disaster Pilot using FME, focusing on flood and landslide impacts while demonstrating integration capabilities and lessons learned from various scenarios, including the Red River flood and British Columbia floods. It addresses open standards, metadata, and FME components used to develop recipes for disaster management, emphasizing the need for high-quality data and near real-time applications. Future steps include improving cloud services, integrating more data sources, and enhancing modeling components to support climate change implications.
Flood and Landslide Impact Components for the OGC 2021 Disaster Pilot using FME
- 1. Flood and Landslide Impact Components for the OGC 2021 Disaster Pilot using FME
- 2. FME User Conference 20 22 Kailin Opaleychuk Support Specialist Dean Hintz Application Expert Jenna Kim Support Specialist Presenters
- 3. 20 22 FME User Conference Agenda Today we’ll be covering… ● Open Standards & OGC ● OGC Disaster Pilot Project Overview ● Red River Flood Scenario ● Other sites: (BC Flood Scenario, Assiniboine River) ● Other hazards: Landslides ● FME Components ● Trends ● Lessons learned, climate change implications, next steps
- 5. 20 22 FME User Conference How Has FME Embraced Open Standards? • Improvements to existing formats: XSD XML, Cesium, GML 3.3 • Continual functionality and performance improvements to open standards infrastructure: HTTPCaller, XML / GML / JSON core, security updates • New wrapper formats coming for XML, GML, JSON (which should we prioritize? InfraGML, GeoSciML?) • FME OGC Certification for GML, Geopackage, GeoTIFF, OGC KML • New open standards formats: OGC CSW, CityJSON, WMTS, OGC API Features, CityGML3, COG
- 6. FME User Conference 20 22 Standards: Metadata ● ISO 19115*, 19139 ● OGC CSW ● STAC ● Dublin core ● Esri shp.xml ● FGDC / NIEM ● EU INSPIRE data management, quality and automation *recent XSD XML reader / writer supports metadata standards see metadata webinar for more information
- 8. 20 22 FME User Conference OGC Disaster Pilot Project Overview
- 9. 20 22 FME User Conference Pilot Goals OGC Pilot - an opportunity to test end-to-end information flow related to all phases of disaster management. Safe Software participation goals: ● demonstrate our integration capabilities ● collaborate with other participants ● derive in depth product feedback from real world applications and datasets The pilot focussed on 3 principle application scenarios: ● Flooding in the Red River Valley, ● Pandemic in Louisiana, and ● Landslides in Peru
- 10. 20 22 FME User Conference Red River Flood Scenario
- 11. FME User Conference 20 22 Red River ● The Red River flows north from South Dakota and Minnesota into Manitoba Canada & Hudson Bay (287,500 km2, 890 km) ● The flat slope of the Red River Basin means flow is slow, allowing runoff to backfill ● Ice jams and ground frost in spring impede flow and infiltration ● Highest water levels and flows in modern history. ● Flooding estimated to be once in 330 - 2000 yrs ● Epic snow melts ● Flooding affected three million hectares of farmland. States of emergency declared in Province & 70 Manitoba communities. ● Floodwater forced the closure of 850 roads & Trans-Canada Highway. Red River Flood 2011: https://www.gov.mb.ca/mit/wms/rrf/historical_2011.html
- 12. 20 22 FME User Conference Red River Flood Routing Recipe: ARD to DRI Pilot Recipe: ● Recipe workflow transforms ARD into IRD to drive DRI’s for specific critical management decisions ● Recipe goal: generate indicator for best transportation routes to avoid flooded areas ● Flood impacts can be rapidly evaluated based on evolving disaster effects ● Recipes are designed to be reusable in different contexts: area (MB, BC, Peru) or time (past, present, future). Recipe Components: RSS Hydro, DFO (Dart. Flood Obs): Flood Model + Extents ● DEM + flood gauge + Modis extents ⇒ flood time series Safe Software: Grid to Vector DB FME Transform Model ● Flood time series grids ⇒ Flood contours vector Geopackage Skymantics: Routing Web App ● Flood Contours Vector Geopackage + Road Network + Tolerance rules ⇒ Flood mitigated routing
- 13. FME User Conference 20 22 Disaster Pilot ARD> IRD > DRI Value Chain - FME Components
- 14. 20 22 FME User Conference RSS Hydro Flood Modeling - MATLAB 2011 Flood Model Input source datasets: ● High resolution LIDAR DEM 5m xy, 10cm z ● Stream gauge data MATLAB flood model ARD Outputs: ● 90 day raster time series with very accurate flood depths (cm) ● Numeric Grids as GeoTIFF ARD
- 15. 20 22 FME User Conference RSS Hydro Flood Modeling - MATLAB 2011 Flood Model Input source datasets: ● High resolution LIDAR DEM 5m xy, 10cm z ● Stream gauge data MATLAB flood model ARD Outputs: ● 90 day raster time series with very accurate flood depths (cm) ● Numeric Grids as GeoTIFF ARD
- 16. FME User Conference 20 22 Source ARD GeoTIFF from RSS Hydro Flood Model - Time Step 34
- 17. 20 22 FME User Conference Flood Routing Recipe: Analysis Ready Data Process to Integration Ready Data FME Conversion of Flood Time Series Grids to Flood Contours Vector Geopackage ● Raster classify, vectorize to polygons, simplify geometry, calibrate, reproject & enrich ● Generate IRD: integration ready outputs: Geopackage and GeoJSON ● Complete model run for input 2011 flood model 70 timesteps ⇒ output 200K records ● Post to GeoNode (GeoServer)
- 18. 20 22 FME User Conference Flood Routing Recipe: Analysis Ready Data Process to Integration Ready Data FME Conversion of Flood Time Series Grids to Flood Contours Vector Geopackage ● Raster classify, vectorize to polygons, simplify geometry, calibrate, reproject & enrich ● Generate IRD: integration ready outputs: Geopackage and GeoJSON ● Complete model run for input 2011 flood model 70 timesteps ⇒ output 200K records ● Post to GeoNode (GeoServer)
- 19. FME User Conference 20 22 Flood Contours Calibrated Vector Geopackage Output - Time Step 34 Flood Contours ARD to IRD calibrated to support DRI routing rules: Depth (m) Status < 0.2 Open 0.2 - 0.4 Restricted > 0.4 Closed
- 20. 20 22 FME User Conference Automation Workflow to process 70+ timesteps - WorkspaceRunner ● Calls RasterToVector_FloodContours workspace for every input time series grid GeoTIFF and appends results to single destination flood contours Geopackage.
- 21. FME User Conference 20 22 Vectorized Flood Contours Geopackage => Skymantics Routing DRIs Model run for all input RSSHydro 2011 flood model timesteps from Mar 20 to May 30 (198K records)
- 22. FME User Conference 20 22 Flood Routing Recipe: Flood Contours Published to Pilot GeoNode/GeoServer DP21 Pilot GeoNode ● Layer search ● Metadata ● Styling ● Interactive web map interface ● Time Series ● Data download ● OGC web services: WMS / WFS
- 23. FME User Conference 20 22 Flood Routing Recipe: Flood Contours Published to Pilot GeoNode/GeoServer DP21 Pilot GeoNode ● Layer search ● Metadata ● Styling ● Interactive web map interface ● Time Series ● Data download ● OGC web services: WMS / WFS VIDEO
- 24. 20 22 FME User Conference Routing Client for Exploring Decision Ready Indicators Optimal route Select flood levels to road blocks New optimal route
- 25. FME User Conference 20 22 Deploying FME Workspace as a JSON REST Service
- 26. FME User Conference 20 22 Flood Extents REST Service: GeoJSON streaming
- 27. FME User Conference 20 22 Red River Flood Impact Web App ● Based on Red River 2011 flood impacts recipe (RSS Hydro, DFO, Safe Software and Skymantics) ● Source datasets - Geopackage: ○ Flood Contours for 70 timesteps (300K records) ○ OSM Infrastructure for roads, buildings and amenities ● FME workspace accepts these inputs, user choices, and generates a map of affected infrastructure ● Workflow published on FME Cloud, hosted by Amazon AWS (no-code publication of desktop authored workflow, scalable architecture)
- 28. 20 22 FME User Conference Red River Flood Impact Web App Flood contours + infrastructure = Impact by severity
- 29. FME User Conference 20 22 Red River Flood Impact Web App
- 30. 20 22 FME User Conference Other sites: (BC Flood Scenario, Assiniboine River)
- 31. 20 22 FME User Conference Disaster Pilot Pivot: Present and Future Current Flooding in BC, Canada • One of the worst natural disasters in BC history (time, spatial extent, cost) • BC cut off from the rest of Canada - most transportation routes were still closed one month on • Multiple hazard impacts at multiple locations over time: landslides, pandemic, loss of infrastructure in addition to flooding (cascading) • Much of the burden for disaster response on often under resourced local municipalities Sumas Prairie Flooding - CBC photo
- 32. FME User Conference 20 22 NRCAN + Flooding ● Natural Resources Canada = National department focusing on natural resources. ● Performs satellite-based monitoring of flood events in Canada and internationally. ● Pilot goal: achieving near real-time disaster information flows through standards. ● Outcomes provide a path forward to achieve this vision. November 2021 flooding near Abbotsford, British Columbia Images Courtesy of B.C. Ministry of Transportation
- 33. FME User Conference 20 22 Overlap of < 4m elevation with Nov 19 BC flood extents (processed by FME component used for Red River, MB) Nov 19 Fraser Valley flood extents from Natural Resources Canada: active floods (RADARSAT) Classified, vectorized Fraser Valley LIDAR DEM (NRCan) Flood risk recipe -current time ● Red River flood impact recipe reused in BC context ● A central pilot goal was to provide components that are able to rapidly rerun recipes in near real time ● The Red River vectorization FME process was rerun using HRDEM from NRCan and current flooding in the lower Fraser Valley ● When overlaid with NRCan active flood extents from RADARSAT a close correlation can be observed for local areas < 4m in Sumas Prairie Pivot to Present: Flooding in BC, Canada ARD Process to IRD
- 34. FME User Conference 20 22 Flooding in BC, Canada - NRCan HRDEM - Fraser Valley East Mosaic (x,y 2m; z cm)
- 35. FME User Conference 20 22 Flooding in BC, Canada: NRCan Active Floods - Fraser Valley East Flooding Nov 19
- 36. 20 22 FME User Conference FME Conversion of DEM Grids to Flood Hazard Contours Vector Geopackage ● Raster classify, vectorize, simplify, calibrate, reproject & enrich ● Generate integration ready outputs: Geopackage and GeoJSON ● Post to GeoNode (GeoServer)
- 38. FME User Conference 20 22 Assiniboine River Flood Risk: Winnipeg, MB - HAND Approach
- 39. FME User Conference 20 22 Demo - Assiniboine River Flood Risk: Winnipeg, MB
- 41. 20 22 FME User Conference Landslides Key factors ● Slope ● Rainfall ● Vegetation ● Geology ● Land use ● Changes to any of the above (recent land use or fires) Challenges ● Lack of high resolution DEM availability for Peru ● Domain expert support (input to recipes for how to combine above factors) ● Highly localized nature of the hazard (unlike floods which can affect large areas) Image credit: Globe and Mail
- 42. 20 22 FME User Conference Slope Grid for Fraser Valley, BC ➔ Derived from NRCan high resolution DEM Workflow: 1. Generate slope grid from DEM 2. Classify slope grid and convert to slope areas geopackage 3. Analyze proximity of infrastructure to high slope areas
- 43. 20 22 FME User Conference Raster to Vector - Slope Contour Generation Classify slope grid and convert to slope areas geopackage
- 44. 20 22 FME User Conference Raster to Vector Output: Data input for impact analysis
- 45. 20 22 FME User Conference Slope Contours to Assess At Risk Highways At Risk Highways Trial Recipe: • Road type = highway or motorway • Slopes > 40 degrees • Proximity to highways < 120 m Result: Highways at possible landslide risk due to steep slopes (help focus areas for further investigation)
- 46. FME User Conference 20 22 At Risk Highways vs Hazardous Slopes & November Landslides
- 47. 20 22 FME User Conference FME Components & Contributions
- 48. FME User Conference 20 22 OGC Disaster Pilot: FME Component Context
- 49. FME User Conference 20 22 Disaster Pilot ARD - DRI Value Chain: FME Components Context
- 50. 20 22 FME User Conference ARD Workflows: OSM PBF to Geopackage, GeoJSON ● Spatial, Geometry, FeatureType & Value Filters, Simplification, Schema Mapping – supports database limitations ● Generate Geopackage and GeoJSON outputs
- 51. 20 22 FME User Conference ARD Workflows: SPOT STAC / COG to GeoTIFF Custom reader workspace: reads STAC* service JSON and calls STAC API for each item and item asset Final step: FeatureReader reads GeoTIFF from COG *STAC Reader coming soon in FME 2022.2
- 52. FME User Conference 20 22 ARD Workflows: SPOT STAC / COG to GeoTIFF
- 54. 20 22 FME User Conference Cloud Native ● Complete, cloud based workflows ● Serverless architecture (S3 or equivalent) ● Supports scalable processing, automation and publication ● Cloud optimized data formats for storage, analysis and display ● Open standards cloud native formats: ○ COG - Cloud Optimized GeoTIFF - 2022.0 ○ STAC - Spatial Temporal Asset Catalog* ○ Publication: Cesium, I3S, JSON, GeoJSON, HTML ○ MRF - meta raster format from NASA JPL ○ Coming soon: COPC, ZARR *STAC in FME 2022.2
- 55. 20 22 FME User Conference Cloud Optimized GeoTIFF Reader / Writer This feature can be found in the GeoTIFF Reader/Writer Often referenced via STAC Zoom levels: 0 = original resolution 6 = lowest resolution / overview for this dataset (varies) MODIS example Cogeo.org https://www.earthdata.nasa.gov/engage/cloud-optimized-geotiffs
- 56. 20 22 FME User Conference Disaster Pilot - Lessons Learned / Summary
- 57. 20 22 FME User Conference Lessons Learned ● Pilot Components are agile but not fully service connected ● Only a few recipes developed for impacts ARD and DRI ● Difficult to find high quality, high resolution source data (DEM) for regions like Peru ● Some hazards are difficult to observe / predict at scale: landslides ● Most scenarios used historical data - need more integration with real time flooding services and forecasts ● Lack of standardized data model and metadata for hazards and impacts
- 58. FME User Conference 20 22 Lessons Learned - BC+ Flood Impact Recipe - Near real-time reuse: ● Able to rapidly reuse & rerun recipes in near real time for new disaster areas ● The Red River vectorization FME process was rerun using HRDEM from NRCan and observed flooding in the Lower Fraser Valley with minimal modification (< 1 hr) ● Similar workflow rerun for Peru (limited resolution) Explore physical and spatial characteristics to serve as proxies for hazard risk: e.g. HAND (height above nearest drainage), steep slopes near roads. ● Comparing these characteristics with historical hazard data may reveal patterns that can be used to help estimate future risk & prioritize areas for monitoring
- 59. 20 22 FME User Conference Next Steps - DP22+ ● Build out architecture to use more services / APIs ● Add other rules and analysis: population impacts & evacuation ● Add other EO & sensor sources for near real-time flooding ● Integrate with more modelling components to support future projection scenarios (local models + climate change flood model forecasts, downsamples) ● Deploy process chain to cloud to ease implementation and improve performance
- 60. 20 22 FME User Conference Disaster response and climate change impact mitigation implications
- 61. FME User Conference 20 22 Summary ● Recipe pattern to support decision indicators (DRI) helped drive development of data flows ● Model based approach supported minimal wrangling & rapid prototyping: no code / low code dev by domain experts with limited dev experience ● Transform models were easy to integrate between different datasets & types, automate and integrate with cloud ● Decision rules constrained by data availability and resolution (Peru) ● Careful review of indicator requirements allowed design of higher value integration ready data model to better support multiple impacts and rules ● Analysis trend - hi res low info to low vol hi info ● Negotiating common interface standards, formats, and locations is key to collaboration across mixed systems & readiness support ● Web and cloud services with automation key for supporting availability & scalability ● FME -> great potential for integrating across related models & results
- 62. 20 22 FME User Conference Resources OGC Disaster Pilot - Safe Contribution: community.safe.com/s/article/OGC-Disaster-Pilot-2021 Towards a Cloud Native OGC: ogc.org/blog/4607 FME and Open Standards Blog: safe.com/blog/2021/03/importance-ogc-open-standards/ Metadata Matters Webinar: safe.com/webinars/metadata-matters-what-it-is-and-how-to-manage-it/ XMLathon Webinar: safe.com/webinars/xml-athon-with-don-and-dean/ Using Data Integration to Deliver Intelligence to Anyone, Anywhere (Disaster Focus) safe.com/webinars/using-data-integration-to-deliver-intelligence-to-anyone-anywhere/