1Spatial: Leeds FME World Tour: Improving the flow of asset information with FME
0 likes46 views
The document describes how FME was used to consolidate a client's large national drainage asset database. Overlapping asset schemes caused duplication that needed resolving. FME processed the data through stages including duplicate removal, subject matter expert review, quality assurance checks, and submission to the client's system. It automated tasks like updating upstream/downstream references and support connections. FME also determined asset elevations from LiDAR point clouds. The resulting single consolidated drainage network provides the client an improved understanding of their extended asset inventory.
1Spatial: Leeds FME World Tour: Improving the flow of asset information with FME
- 1. Improving the flow of asset information with FME Ian Walker and Rogelio Jante Atkins
- 2. Hello! Ian Walker Rogelio Jante 16th March 2017
- 3. Drainage Data Consolidation Provide Client with better understanding of their own assets
- 4. Some Background • Client has large national database • Represents inventory of drainage assets • Originally captured as-built records to create a baseline dataset • Subsequently added to over number of years with new schemes • Focus has been purely on compound capture • Desktop study to manage risks from unnecessary site visits
- 5. The Data • Solution around since 2006 • Holds drainage assets as points, lines, regions and associated schedule information
- 6. The Problem • Drainage information uploaded to database is contained within Scheme boundaries. • Multiple schemes overlap with no management of how they work together. • Overlapping schemes also means possible duplication of assets • Database holds Valuable information that can not be easily accessed or read outside of systems
- 8. Technical Team • Geospatial Team • Manage spatial data and data analysis • Visualisation Team • Validate existence of key assets from route videos • Drainage Subject Matter Experts (SME) • Review and verify drainage network • Quality Assurance • Create and enforce validation rules
- 9. The Process 1. Setup 2. Initial Spatial Review 3. Drainage SME Review 4. Pre QA 5. QA 6. Submission
- 10. Stage 1 • First step was to find method to handle aprrox. 100,000 asset records • Grouped any assets with a connection into manageable packages of work • Initial assessment identified 674 individual sections • Uniquely reference as Logically Connected Assets (LCA)
- 11. Stage 2 • Automated processing to identify duplicates • Retain most, recent, complete or accurate asset • 2 phases 1. Process duplicate points by proximity 2. Reject lines and update no overlapping schemes • Features ‘failing’ the above were tagged for review by the Drainage SMEs
- 12. Process duplicate points by proximity
- 13. Reject Lines and update non overlapping schemes
- 14. Stage 3 • SME Review • FME used to upload data into editable webGIS • Data assessed by drainage SME and comments added • FME downloads data and changes/corrections made in desktopGIS
- 15. Stage 4 • PreQA stage designed to asses data processing carried out during Stage 2 • Again a number of workbenches • Validation report and assignment of references for ‘New’ assets • FME generates error table for further review/editing in desktop GIS • Checks basic geometry and attribute integrity
- 16. Stage 4 • Error table Error Description Potential Cause Solution No. of point assets is not equal to No. in LCA spreadsheet Excluding ‘New’ features, does the number of features in this LCA equal the number of features originally recorded in the progress register for this LCA. Assets may have been deleted or duplicated or ‘New’ assets may have been given an incorrect DR_Status. Use archived source data for this LCA to join on asset reference and find any differences. Any duplicates should be deleted, deleted records added or DR_Status updated. DATE_OF_SU must be in string format 'yyyymmdd' Date of survey does not contain / and is a string of length 8. Update or delete date of survey as appropriate. Invalid Geometry type Geometry does not match the type expected (point, line, polygon) or has no geometry. Remove problem asset and re- add from source or delete invalid record. ASSET_REF is not populated Does each feature have an asset reference New data not being given an asset reference or accidental deletion of asset ref. If asset is new add asset reference as next sequentially. If asset is existing use source or archive data to determine the asset reference. Duplicate ASSET_REF Does an asset reference exist in any of the other feature classes Copy and pasting of data where asset ref is not changed, splitting lines, duplicating data. Determine whether the duplicated records are exact copies (same attributes and geometry). If so delete one. Otherwise determine which of the asset references needs to be changed. Use the source or archive data where appropriate. 2 points with identical geometry Matcher on geometry Duplication or point accidently snapped to another. Use source or archive data to determine whether one point should be moved. If originally two points on top of one another this can be ignored. LCAID is not correct? LCAID attribute does not match the base name LCAID entered incorrectly or asset needs to be merged with another LCA. Update LCAID or if LCA needs to be merged, follow procedure outlined in Error! Reference source not found.. Item Type Code is missing or invalid Item type code is not one of the valid HADDMS item types or is missing. Typo or entered incorrectly. Determine the correct item type and update. DR_Status is not permitted DR Status is either missing, Flag, Pending or GIS Action. Asset not updated or action still required. Update DR Status or take appropriate action. CN does not have a SUPP_CONN value No value is found in the SUPP_CONN field of a point where item type code is CN Add SUPP_CONN or leave for workbench in Error! Reference source not found. CN does not have Retained SUPP_CONN value The SUPP_CONN value specified for a CN does not exist as the asset reference of a line in this LCA. Update SUPP_CONN or leave for workbench in Error! Reference source not found. Point contains a SUPP_CONN value which is not retained The SUPP_CONN value specified does not exist as the asset reference of a line in this LCA. Update SUPP_CONN. This will not be resolved as part of the SUPP_CONN workbench in Error! Reference source not found.at this stage. DR_Status = New, but DR_Source is empty Asset has been identified as new but no source has been defined. Determine source and update DR_Status. PIPE_REFER is not populated Does each feature have a pipe reference? New data not being given an asset reference or accidental deletion of pipe ref. If asset is new add asset reference as next sequentially. If asset is existing use source or archive data to determine the pipe reference. Upstream Ref is not valid The attribute in the UPSTR_REF field does not match the asset reference of a point in this LCA This is automatically updated based on geometry in Error! Reference source not found..
- 17. Stage 4 • Workbench used to update US-DS references and SUPP_CONNS • Workbench will automatically update all UPSTR_REF and DOWNSTR_REF values based on the location of the start and end coordinates of each line.
- 18. Stage 4 • Update the SUPP_CONN attribute of Connector Nodes (CN) based on the geometry of the features. The SUPP_CONN is mandatory for any points where the item type code is CN and should contain the asset reference of the line which the node connects to.
- 19. Stage 4 • Workbench 6.1 QA Schema Loader • The schema loader removes rejected assets and formats various attributes to comply with Client Schema before outputting shapefiles • Pass data over to QA experts……….
- 20. QA Checks and Elevation FME was used to ensure the inventory data met all requirements and to determine the elevation of point assets from LiDAR point clouds
- 21. QA and Elevation • We developed a workflow to apply automated checks to the asset networks • This involved making sure that specific attributes are populated and that they have the right format • The way in which assets connect to each other is determined by their asset type • We used FME to ensure the network connects together in the right way
- 22. LiDAR tiles
- 25. Stage 6 • Submission of final Shapefiles to Client Business Collaborator site • Throughout project regular progress reports provided to PM and Client • FME used to track progress of LCAs/assets through the process and populate Client reporting template with current status
- 27. Consolidated Data • Single consolidated drainage network • Data as consolidated as it can be without the prohibitive cost of new site surveys • Extended across network • Improve usability of valuable asset inventory