FME Server Workspace Patterns - Continued
Download as PPTX, PDF0 likes463 views
This document discusses several patterns for organizing FME Server workflows. The patterns include: - Submitter Iterator with Status, which selects parts of a dataset to process independently on different engines while tracking status. - Runner/Controller Workspace, which uses FMEServerJobSubmitter for workflow control and allows running the controller on a desktop. - Dataset Tool Switch, which synchronously processes the same dataset with different modes of operation through successive refinement. - Flatten, Explode, Export, which involves three stages - flattening data, unpacking it, and exporting to different formats - to manage master data.
FME Server Workspace Patterns - Continued
- 1. FME Server Workspace Patterns - Continued Reed Whittington Burns & McDonnell
- 2. About Us FME Certified Pro Services Burns & McDonnell, Dallas, TX Reed Whittington Burns & McDonnell is a full-service engineering, architecture, construction, environmental and consulting solutions firm. Our multidiscipline staff of more than 5,000 employee-owners includes engineers, architects, construction experts, planners, estimators, economists, technicians and scientists, representing virtually all design disciplines.
- 3. Business Technology Services Burns and McDonnell Technology Consulting Services is a virtual team of technical experts around the globe offering a wide array of services and deliverables including: • Web Sites/Information Management Portals/Doc Management solutions • 2d/Web Maps/3D/4D Globe Applications • Field Data Collection/Inspections/Real-Time Monitoring • CAD/GIS/Database Integration • Automated Report Generation and Distribution • Project Metrics/Analytics/KPI Dashboards Integrated, field tested, project proven solutions Many of these services are bundled under OneTouchPM which is a hybrid cloud based, web enabled, managed service allowing project managers to engage their customers and share information, summary progressing with aggregated executive views from all disciplines.
- 4. In software engineering, a software design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. It is a description or template for how to solve a problem that can be used in many different situations. Not a specific implementation; design patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system. https://en.wikipedia.org/wiki/Software_design_pattern
- 5. 4 FME Server Workspace Patterns • Workflow Submitter Iterator (or Work Status) • Strategy for Asynchronous processing of large datasets • Runner / Controller Workspace (or Tool Chain) • General approach using FMEServerJobSubmitter for workflow control • Dataset Tool Switch (or Multi-Pass) • Strategy for Synchronous Sequential processing • Operates on same dataset with different modes of operation • Flatten, Explode, Export (or Import/Merge/Publish) • 3 stages of processing typically used for master data management
- 6. Submitter Iterator with Status • Uses a selection strategy to select mutually exclusive parts of a source data set so that the parts can be operated on independently and run on different fme engines. • Used to process files or db operations so that sql select statements can be used. • Needs a better name • Work Status, Delegator, … • This was the subject of a previous presentation. • http://cdn.safe.com/events/FME2011WorldTour/Austin_Reed_Whittington.pdf
- 7. Example using Path reader to Iterate and track status. can also use url as path or list of SQL queries Submitter Iterator with Status Wait=No
- 8. Runner / Controller Workspace • General strategy for Job Control and Tool Chain • Lets you run the controller on desktop • FME 2015 improves FMEServerJobsubmitter: • In Engine Sequential Run with Wait=Yes • Output datasets to temp files or as specified Excellent Article by DeWet@Safe: https://knowledge.safe.com/articles/1439/fme-server-workflow-management-1.html
- 9. Dataset Tool Switch (aka multi-pass) • The “Multi-pass" pattern presents a common interface to iterate over datasets using successive refinement. • Meant to be run multiple times with different RUN_MODEs • RUN_MODE Choice defines the Switch behavior • TestFilter on RUN_MODE controls Feature RoutingUseful for processing directory/tree of files and/or database exports or publishing. • Interface includes: • SourceDir, StagingDir, Run Modes, DB connection, Publish location
- 10. Use the Path reader to Iterate and Route based on Run Mode
- 11. Flatten, Explode, Export • 3 separate workspaces that act together but run asynchronous • Each may use “multi-pass” • Flatten • Denormalization into one big table • Pack attributes into dictionaries • Think NoSQL and Operational Data Store • Explode • Unpacks big table • Data model mapping and normalization happens here • May use templates for standard data models • Might also use Dynamic Schema
- 12. Flatten, Explode, Export (cont) • Export • Database to files (.kmz, .csv, .json, .gdb, …) • Heavy use of Fanout (ds and ft) • May use Tool Switch RUN_MODE to define export type or staging/publish processing steps
- 13. More possibilities with FME 2016… What do you use? Tool Chain Workspace • FeatureReader – Selective reading/caching – Error Checking • FeatureWriter – Write then read back in same .fmw – In workspace tool chain – Removes the need for temp output files of the FMEServerJobSubmitter Before FME 2016 • Required chaining FMEJobSubmitters together • Next in chain reads output of Previous
- 14. Main point: FME Server Patterns help you develop and organize your data management tools.
- 15. Thank you! Reed Whittington FME Certified Pro Services Burns & McDonnell Dallas, TX rwhittington@burnsmcd.com