FME as an Orchestration Tool - Peak of Data & AI 2025

FME as an
Orchestration
Tool
with priciples from Data Gravity
The Peak of Data
and AI 2025

2025
The
Peak
of
Data
and
AI
Nils
Juell
Department Manager Data Science
GEODATA
Antonio
Armas
Senior Consultant GIS
GEODATA

1. Data Gravity
2. Data orchestration
3. Examples & Application tips
4. Pros & cons
5. Conclusion
Agenda

2025
The
Peak
of
Data
and
AI
Non optimal design
System A
Data
Data input
Data
Data output

2025
The
Peak
of
Data
and
AI
Data gravity principles applied
System A
Data
Data

Data
orchestration
deﬁnition
Section 2

2025
The
Peak
of
Data
and
AI
System A
System B
Data
Data
Tools

2025
The
Peak
of
Data
and
AI
System C
System A
FME Flow speciﬁc System B
Data
Tools
Engine A1 Engine A2
Engine B1
Engine C1 Engine Cn

Data gravity, when to apply
Apply Don’t apply
Big data ﬂow Proof of Concept
Repetitive tasks Too complex data ﬂows
SQL, or other Access System tool,
is supreme

2025
The
Peak
of
Data
and
AI
Example 1 – Where Clause
● Classic FME – simple reading
● Traditional FME use with ﬁltering
● Run time 1 min 56 s
23200
23200 195

2025
The
Peak
of
Data
and
AI
Example 1 – Where Clause
● Data Gravity applied – simple reading
● WhereClause on the Reader
● Run time 1.8 s
195

2025
The
Peak
of
Data
and
AI
Example 2 – File property filtering
● Task: List files between two dates
● Classic FME: Reader -> Tester
● 25000 files in 2.5 minutes…
● How can we solve this faster??

2025
The
Peak
of
Data
and
AI
● FME with Data Gravity principles:
Let’s use PowerShell commands on the OS that contains the ﬁles(!)
● 25000 ﬁles in 1.3 seconds!

2025
The
Peak
of
Data
and
AI
● Hmm, PowerShell scripting is difficult, so how about some some AI help here,
prompt:
I want a powershell code to run in a cmd. I want it to list all the files
updated after a certain time. The command should also work with
path-strings with spaces, consider use of escape characters. Write the
result, file with all properties, to a csv-file
● Suggested command:
powershell -Command "& {Get-ChildItem -Path 'C:Your Path Here' -Recurse
| Where-Object { $_.LastWriteTime -gt '2024-04-01 12:00:00' } |
Select-Object * | Export-Csv -Path 'C:Your Output Pathoutput.csv'
-NoTypeInformation -Encoding UTF8}"
● Modify the paths and it works like a charm in SystemCaller

2025
The
Peak
of
Data
and
AI
Example 3 – XML Wrestling with use of xpath()
● XML handling can lead to complex workspaces
● How can Data Gravity thinking help us here?
● In PostGRES, we can use the xpath() function to extract the desired attributes
from XML

2025
The
Peak
of
Data
and
AI
● Classic FME, extensive use of CustomTransformers
● Read, test, ﬁlter, merge
● Run Time: 5 min 7 s

2025
The
Peak
of
Data
and
AI
● Data Gravity applied
● Two steps:
○ Raw XML to PostGRES table, type cast string as “xml”
○ Orchestrate SQL on table
● Use PostGRES function xpath() in SQL to extract values
● Run Time: 11 s

2025
The
Peak
of
Data
and
AI
Example 4 – GML python hug
● No Data Gravity applied
● FeatureReader through a bunch of GML ﬁles inside a folder
Run Time: 1 min 32 s

2025
The
Peak
of
Data
and
AI
Example 4 – GML python hug
● Two steps:
○ Get all paths
○ Use PythonCaller to go through all ﬁles, get the right feature type with the ﬁlter constraints and return
them as lists
● Run Time: 4 s

2025
The
Peak
of
Data
and
AI
Example 5 – PostGIS patchwork
● No Data Gravity applied
● Read all, SpatialRelator and then write the output into a new table
Run Time: 5 min 25 s

2025
The
Peak
of
Data
and
AI
● Send a SQL command to the database server
Run Time: 1 min 8 s

2025
The
Peak
of
Data
and
AI
4362 features
against
7 618 602 grid cells

2025
The
Peak
of
Data
and
AI

Data gravity – Pros & Cons
Pros Cons
Speed Non-FME thinking
Simplicity* Knowledge of remote system
Load on remote system Less control of load
*Tips: You have to document thoroughly in FME what happens remotely,
as it might be difficult to trace and find otherwise.

Data gravity – best practises
*Tips: You have to document thoroughly in FME what happens remotely,
as it might be difficult to trace and find otherwise.
- Do cloud to cloud instead of cloud to local to cloud
- Do run more functions and procedures than run select * against
databases
- Do “explain analyze” on databases to test performance of
queries
- Do use of all kinds of query possibilities to minimize the amount
of data to be read into the workbench
- Do check if queries are being run periodically, If so, keep them
allocated properly
- Use AI tools to help you create for unknown systems

2025
The
Peak
of
Data
and
AI
ThankYou
nils.juell@geodata.no
antonio.armas.diaz@geodata.no

FME as an Orchestration Tool - Peak of Data & AI 2025

More Related Content

Similar to FME as an Orchestration Tool - Peak of Data & AI 2025 (20)

More from Safe Software (20)

Recently uploaded (20)

FME as an Orchestration Tool - Peak of Data & AI 2025