FME Driven Metadata & Data Governance
0 likes81 views
The presentation by Peter Veenstra at the FME User Conference emphasizes the importance of excellent data driven by FME and thorough metadata tracking. It outlines the processes used by Pivvot to manage data, including data ingestion and validation practices, as well as the significance of tracking metadata at the row level for data provenance and quality. The goal is to ensure the most up-to-date and reliable geospatial data is consistently available for decision-making.
FME Driven Metadata & Data Governance
- 1. FME Driven Metadata and Data Governance
- 2. FME User Conference 20 22 Peter Veenstra - Director of Data Architecture 31 years in geo-spatial industry - analyst, programmer, architect, consultant, director Love my family, hockey, mapping, GIS, data, fly-fishing, drawing, FME & the world we live in … Pivvot Location Intelligence, Software as a Service Company (4 years, 20 people, remote) Terracon Geotechnical, Environmental, Facilities, Materials Engineering Company (50+ years, 5000+ people, 175 offices/remote)
- 3. 20 22 FME User Conference Making informed decisions requires excellent data aPPropriate PRECISE Pedigree Provenance
- 4. 20 22 FME User Conference Informal Poll: How many people/organizations here actively track metadata for your data layers? If yes, is it done at the table level or the row level or something else? If yes, how do you document it? If no, why not?
- 5. 20 22 FME User Conference 1. FME is the data tool (for us) 2. Row level metadata is requirement for excellent data … The two things that I would hope you would take away from this presentation are …
- 6. 20 22 FME User Conference Proposition … Excellent data is driven by FME and Metadata … ● Our data environment and how we use FME to manage data. ● How we create, track, and use Metadata.
- 7. 20 22 FME User Conference Part 1: Our data environment
- 8. 20 22 FME User Conference Pivvot Data in our platform …
- 10. 20 22 FME User Conference FME Cloud The Stack AWS RDS AWS S3 AWS WS AWS SMS AWS Quicksite FME WS
- 12. 20 22 FME User Conference Rules around how/why/when/where we add data … ● We do not edit source data ○ we flag completely fubar features and store them someplace else ● We merge datasets ○ From separate source keeping the best by numbers, attributes, timeliness ● We deprecate datasets, agencies, sources ● Data is fit-for-purpose from trusted agencies who publish on a regular basis ● Rules, procedures, how-too’s all documented in Confluence ● All ingestion performed by FME ● All validation performed by FME The Rules
- 13. 20 22 FME User Conference Data Source Data Source Data Source Pivvot Dataset SCRATCH WHARF VERA DEV STAGE PROD •Zero Records •Data Source exists •Data Load does NOT already exist •Subtype Validity •Polygon/Line Crossing 180° longitude •Invalid Geometry •Proper SRID assigned •Missing Metadata •Invalid geometry format •POLYGON characteristics •DEM s
- 14. 20 22 FME User Conference Why FME? What other tool can handle the types of data we consume? ● 3218+ File Geodatabases in a Directory ● ArcGIS Online Web Services ● Shape Files ● Streaming Data in GeoJSON Format ● Blobs stored in Microsoft Azure ● GeoTiffs and COGTiffs in S3 ● Raster Elevation Models ● Lidar Point Clouds ● CSV Files ● KMZ Files ● GeoPackage Files Or the data we write too … and how? ● GeoTiffs and COGeoTiffs in S3 (support for rasters in Geoserver as layers) ● PostGIS Vector Data (Simplified for the Web) - 5m simplification ● 3 Band PostGIS Rasters (elevation, aspect, slope) from USGS DEM ● Composite_Parcels (Fuzzy Matcher) from Parcels using proximity and owner name ● All in the Cloud
- 15. 20 22 FME User Conference A set of workbenches that solve the problems ● Each data source has it’s own ingestion workbench (947 + ) ● Each workbench once proven is scheduled to run in FME Cloud TODO Show fuzzy matcher workbench (for location and name) - highlight metadata ingestion Show interconnect queue workbench (highlight metadata ingestion. Show Data DIctionary publication Workbench. Show USGS Stream Gauge Livestream Workbench (hightlight metadata ingestion)
- 16. 20 22 FME User Conference Part 2: Metadata and Excellent Data
- 17. 20 22 FME User Conference Metadata: What is it? Why is it important? ● Metadata describes (for Pivvot) … ○ The Provenance (origin, processing, agency) and, ○ The Pedigree (completeness, accuracy, timeliness) of the Data ● This is important at the ROW level (not the table) ○ Pivvot combines data from different ‘sources’ ■ Who did we get the data from, are they reliable, trusted, regular, consistent? ○ Some Data are processed differently at different times to be merged into a single dataset ■ There are FEDERAL Datasets, and STATE Datasets (some show more data, and better attributed data) ○ Understanding at a minimum the Source Agency survey and publication data, the Pivvot last checked and last downloaded date ■ Now we can work the 30/90 day re-check, if found, download, process (with FME), validate (with FME), assign a metadata ID to each record, rip-replace the old agency data with the new data
- 18. 20 22 FME User Conference Each record in each table is assigned a metadata_pivvot_id (GUID) ● This ID is common to all PIVVOT METADATA tables and idenifies: ○ What agency the data came from ○ The File/URL source of the data (one agency can produce many) ○ The table the data will be loaded in ○ The LOAD that the data was placed into the table with including the: ■ The date the data was last checked ■ The date the data was last loaded metadata_pivvot_id AGENCY SOURCE LOAD TABLE
- 19. 20 22 FME User Conference Metadata Tracking Show Example of Metadata Pivvot ID and Standard Attribtues Show Transformer Show FME Cloud Interface for utilizing
- 20. 20 22 FME User Conference Operationally ● Roll-backs ● Data Source Tracing ● Querying the Database faster … metadata_pivvot_id in (select pivvot_id from md.vw_data_load where table_physical_nm = ' gssurgo_chorizon ' and data_source_nm like ' NC%' order by agency_publish_dt desc limit 1) metadata_pivvot_id
- 21. 20 22 FME User Conference How does metadata lead to excellent data? Assume that the data from the source is good and fit-for-purpose (curation, research) Assume that the data has been QA/QC’ed for use in the platform Track each load, from each source, for each agency including processing/transformation/projection as performed and logged by FME We have the knowledge of provenance and pedigree of each tuple of data in our database and can extract, delete, roll-back any push to production based on that metadata = Defensibility
- 22. 20 22 FME User Conference Summary No data is perfect Geospatial data is inherently an abstraction of the real-world Our goal is to provide the ‘best’ and most ‘up-to-date’ from source to platform on a consistent and regular basis. The only way for us to achieve this at scale, in a growth-mindset, for geospatial is FME and row-level Metadata aPPropriate PRECISE Pedigree Provenance