Using Dynamic FME Workspaces to Build an ETL Solution For a Spatial Asset Visualization Application
Download as PPTX, PDF1 like721 views
The document discusses the development of a dynamic ETL solution for a spatial asset visualization application by UDC, a system integrator for gas and electric utilities. It outlines the ETL requirements and the specific challenges encountered, such as handling domains, attributes, and network connectivity, along with the proposed solutions using FME and ArcPy. The process emphasizes automation and the effective management of data extraction, transformation, and loading.
Using Dynamic FME Workspaces to Build an ETL Solution For a Spatial Asset Visualization Application
- 1. Using dynamic FME workspaces to build an ETL solution for a spatial asset visualization application Sriram Ganesh
- 2. • Founded in 2005 and privately owned, we are an Associate Partner of Safe Software based in Englewood, CO • We are a System Integrator for Gas & Electric Utilities • We work with our client’s Systems and Team to make their investments work • We have over 200 employees and offer a full range of services to support every aspect of the Utility Project and Program lifecycles • We provide solution frameworks and software tool kits which offer considerable savings to our clients About UDC
- 3. FLIGHT PLAN ● Takeoff - ETL Requirements ● Ascending - Challenges ● Cruising - FME, ArcPy ● Autopilot - Automate ● Touchdown - ‘Safe’ Landing!
- 5. Complete and Incremental Extract From landbase, gas and electric Esri SDE Geodatabases to an Oracle Spatial Object database ● Domains and Attributes ● Feature classes ● Relationship classes ● Object classes ● Representations ● Network Connectivity
- 6. Challenges
- 7. Challenges ● Domains and Attributes ○ Challenge - stored as xml workspaces ○ Solution – Python Scripts within FME to extract all domains and then use change detector ● Representations ○ Challenge - stored as blobs ○ Solution – Write ArcPy scripts to make a copy of the feature class and update cartography to extract representational geometry into the main shape field of the copied feature class ● Network Connectivity ○ Challenge - stored and maintained internally in the SDE ○ Solution – Write ArcObjects code to extract complete network. Then apply FME change detector as applicable
- 8. The Solution – FME, ArcPy
- 9. Domains – Extract -> Transform -> Load
- 10. Domains – Extract -> Transform -> Load
- 11. Representation Extracted to main Geometry
- 12. Feature Classes Complete Load Dynamic Workspace
- 13. Feature Classes Incremental Load Dynamic Workspace
- 14. Feature Classes Incremental Load Dynamic Workspace
- 15. Network Connectivity Complete Load Workspace
- 16. Network Connectivity Incremental Load Workspace
- 17. Automate
- 18. Automate – Batch files and Schedule
- 19. 1Gather detailed Requirements To sum it up… 3Automate 2Identify Challenges & Address them
Editor's Notes
- #4: Flight Plan analogy taken all the way to a ‘Safe’ touchdown!
- #6: These were the ETL requirements for a Utility Client in the Midwest
- #12: Representation (Alternate Geometry) extracted to main geometry field in the copied feature class. This is done through an ArcPy script. Once extracted and geometry updated the copied feature class is handled the same way as any other feature class by the ETL workspace
- #13: Feature classes to extract are passed to the dynamic reader via User parameter ‘Feature Types to Read’. Feature classes written by dynamic Oracle Spatial Object Database writer. The lines had to be dedensified and the arc stroked to poly lines.
- #14: UDC Change Detection Tool populates the change detection tables Add, Edit and Delete. FME workspace reads the Objectid and the Tablename from the change detection tables and then reads that specific feature and adds, updates or deletes it from the Target Oracle Spatial Objects Database dynamically after any required processing. Schema of the feature class is passed to the writers.
- #15: The entire incremental load dynamic workspace with the Delete workflow. The table structure of the change detection tables is shown. The configuration of the dynamic writer is also shown.
- #16: UDC Network extract tool outputs a csv file. An FME workbench processes that file to write to a working table. The network connectivity workspace reads each elementid (eid) from the working table and then reads the associated edge and junction features attached and then splits the complex linear feature at the junctions (vertex). A spatial filter transformer only selects the linear feature that touches the two vertexes, generalizes the line and writes it to the Oracle Spatial Database with the geometry of the edge. Vertexes (vertices) are extracted into a separate table (FME workspace for vertex extraction is very simple as no geometries are extracted. It is not shown here)
- #17: An FME workspace with a change detector transformer compares the new and old network extracts and writes the added and deleted connectivity rows to the working table. The FME workspace shown here processes the changed records from the working table and either adds or deletes the rows from the target table based on the change detector output. Also, all edited records are reprocessed for any updated geometries.
- #19: The network extract FME workspace is called multiple times to process different batches of edge_eids. This will improve performance and also avoids any memory optimizations that FME would have to otherwise do with large datasets. This dataset has close to 5 million rows.