MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases

Editor's Notes

• #11: Standards were documented in an Excel Spreadsheet by Charlie Jessome, from ADMIE. Required Feature classes were defined (highlighted in yellow), together with their attributes and accepted values.
• #12: FME Desktop 2013 SP3, from Safe Software, was used to build the Self-Validation ETL in May 2014 in order to allow to “copy and paste” all contents of this workspace to an ArcGIS Data Interoperability ETL
• #13: As it may be seen on the left, an output feature class is generated for each input feature class every time errors and warnings are detected for each particular class. Feature classes of the output FGDB have the same name and schema of the input feature classes, but with the addition of the string ‘_QA’ at the end of each input feature class’ name (e.g. ‘P_Building_QA’) and with two new fields: ‘ETL_timestamp’ and ‘Type_Error’. These fields show the date and time when each individual feature was processed and the description of the type of error, warning or non-compliance issue found (over 160 types of errors and warnings have been typified and may be reported here). When this FME workspace is run at MCE, additional feature classes are generated with the string ‘_WH’ at the end (e.g. ‘P_Building_WH’). These features contain the input features and the result of comparing each individual input feature against features of the same feature class already loaded into the RPSDW SQL Server. These output feature classes include two new fields: ‘ETL_timestamp’ and ‘TypeMatchWH’. The last field shows one of the three following values: “Identical Geometry is NOT found in Warehouse”, “Identical Geometry and Different Attribute values are found in Warehouse” or “Identical Geometry and Attribute values are found in Warehouse”. This Self Validation FME workspace is followed at MCE by other workspaces to ensure proper data management of the SQL Server database and communication with DND Bases. These workspaces are not presented here. As it may be see on the right, bookmarks have been built for each feature class, to allow to refocus the main window on any feature after clicking on them and to allow the user to enable/disable all objects included in a bookmark (this can be very handy if the user would like to run the ETL for one or more selected feature classes)
• #14: At he ‘Check Point Charlie’ the first bookmark on top (light green), it is first verified that input data provided in one of the formats accepted does not contain multi-part features (green colored custom transformer) and then it is verified that all input data lies within the corresponding DND Base data reporting area (this area is equal or larger than the respective DND Base legal area). At the ‘P_Building Non-Standard Attribute Validation Area’ bookmark, the defined schema and the accepted values for non-standard attributes (i.e. the attributes and accepted values that are unique to one specific feature class) are validated. At the ‘P_Building Standard Attribute Validation’ bookmark, the accepted values for standard attributes (i.e. the attributes and values that are common to many or all feature classes) are validated . At the ‘P_Building Geometry Validation’ bookmark, selected issues in input features are detected (each input feature is processed individually). These selected issues are those relevant to obtain OGC compliance, such as self-intersections, duplicate points and several others. At the ‘P_Building of input from DND Base with data existing on MCE Warehouse’ bookmark, a comparison is performed between each individual feature of the input P_Building data against the P_Building data already loaded into the MCE warehouse.
• #16: Annotations are provided at different parts of the ETL in order to provide the user with additional information of a particular testing carried out by a transformer
• #17: Since the data loaded to MCE Warehouse may be used to publish mapping services, this geometry validation (meeting Open Geospatial Consortium standards) and subsequent data repairs are required to prevent the discard of failed features at the time of publishing these services.
• #18: This comparison generates an output feature class indicating for each feature whether an object with identical geometry and location is already loaded to MCE warehouse, and whether two matched objects have identical or different attribute values. This comparison is essential to prevent duplicated data and to manage data, changing the status of features updated.
• #21: Errors, warnings and non-compliance issues are sent first to a visual interactive interface (FME Data Inspector) and then, if requested by the user, all errors and issues detected can be written to a destination FGDB. In the screen capture above, results for the ‘P_Building’ feature class are inspected. For all features, the location of the input data is reported, together with a time-stamp and a description of the type or error or warning found, like in the case of the examples indicated in the sub-title. More that 160 individual types of errors and warning have been typified and may be reported here.
• #22: All errors and issues detected can be written to a destination FGDB in a new run of this ETL. Users with some basic knowledge of FME Desktop may request to send the output data to a format of their choice.