Peak of Data & AI Encore- AI for Metadata and Smarter Workflows

Copyright of Shell International B.V. 1
AI Agents & Metadata
Peak of Data & AI 2025
Grace Cai
Innovation Lead
(Shell Global Solutions Canada Inc.)
April 2025
Copyright of Shell International B.V.

Motivation
April 2025 5

Metadata is important and yet…
April 2025 6

The Semantics Layer
AI Agents are hungry for context
 To meaningfully deploy AI Agents, we need more, high-quality metadata not less!
 We need a ‘semantics layer’ between AI applications and data/tools
Context windows are finite
 Brute forcing context is expensive and ineffective
 It’s an optimization problem
It’s helpful for humans too!
 Automating metadata generation provides ‘a staircase of value’
 Let’s use AI to automate the boring stuff!
April 2025 7

Completeness Score > 75%
April 2025 8

It’s possible
April 2025 9

Iterations
April 2025 10

Iteration #1
FME 2023.2 + AzureOpenAI + GPT 4o
April 2025 11
Fetch AGOL
Item info
Sample
records
Ugh Lists…
Field
Description
Field Value
Type
AGOL Item
Description
You are a helpful technical writer with extensive
knowledge of oil and gas exploration and development.
Given the following information write a three-sentence
metadata description of a table called @Value(_name).
Focus on what a geoscientist or data scientist might find
useful. Consider the data sources: @Value(Source) and
geometry types: @Value(Geom) Consider the geographic
extent '@Value(_extent)' to indicate if the data set is
suitable for global, regional or local use. Identify the
geographic region this dataset covers if possible. Do not
include the extent coordinates in your summary. Do not
list specific column names or data values in your output.

Iteration #2
April 2025 12

Iteration #3
April 2025 13
Deactivated
Column
Desc. Gen.
Thought
Gen.
Editor &
Acronym
Lookup
Parse results
Get Feature
Count

Iteration #3
Table Description ‘Thought’ Prompt
You are a data detective for Shell, the energy company. your mission is to
uncover the intent of the following data package. Approach the clues with a
healthy dose of skepticism and do not be too confident in your response. Think
about deeper meanings of repeated clues, particularly geoscience key words and
layer names. Pay attention to any data filters or expressions that might alter the
use of the data. Think about the layer feature counts in relation to data
completeness. Look for clues and acronyms indicating organizations or vendors
who may have initially provided the data, say you are unsure if the evidence is
not clear. Remember, it may be generated by Shell staff if source is unclear. Try
to identify the geographic region(s) of the data, do not include coordinates in
your response, say you are unsure if the evidence is not clear. Provide your
answer concisely.
April 2025 14

Iteration #3
Table Description ‘Edit & Format’ Prompt
Read the user generated data description carefully and do the following in English.
- Generate a concise Summary
- Edit the user provided description for publication. Use the provided acronym look up to
correct any factual errors. replace coordinates ranges with named geographies.
-Generate a concise Title, following the format of <where > <what > (<who>). <Where> can
be a country, continent, region, geological basin, or other. Use ISO three letter alpha 3 codes
to indicate location if possible. Use "Global" for worldwide datasets. Use "International" for
datasets excluding the Americas. <Who> is the organization or vendor that sourced the data,
this may be Shell. If you are unsure use "Unknown".
-Generate 3 to 5 key word Tags that may help users find this data
-Select the most applicable Theme: {…}
-Select the most applicable Source Type, leave blank if unknown: {…}
-Select the most applicable Business Focus: {…}
April 2025 15
Format your response in json
like the following example:
{"Summary": "",
"Description": "",
"Title": "",
"Tags": "",
"Business Focus": "",
"Theme": "",
"Source Type": "" }
Acronym look up list:
{…}

Observations &
Learnings
April 2025 16

Observations & Learnings
LLMs have a bias towards action
 It would rather hallucinate than say “I don’t know”
 So, give it permission to not know things!
 “say you are unsure if the evidence is not clear”
Schema-Only Analysis led to overconfidence
 Large schemas make the data appear comprehensive when it may not be…
 “this is a comprehensive dataset” – there were 2 polygons in the dataset
 Total feature count provides valuable context
April 2025 17

Observations & Learnings
Colloquial phrasing and Awkward Phrasing
 ‘gas station’ vs ‘retail station’
 ‘spanning a broad geographic area, covering regions from the western to eastern hemispheres’
 Did not know Shell acronyms
 Would like to implement RAG MCP tool call in the future
Context filtering and Formatted outputs
 ‘Format your response in JSON’ allow you to ask for multiple outputs with a single LLM call
 But doing too much in one go led to sub-optimal results
 Don’t pollute the context window (more isn’t always better)
April 2025 18

LLMs understand WGS 84 coordinates
April 2025 19
45%
49%
65%

If no Evals, then just vibes
 Items with good metadata often include context not available by looking at the data
 Items without metadata are expensive to use as evaluations
 Crowdsource evaluations
 LLM as judge (if there is a high-quality evaluation set)
April 2025 20

Conclusion
April 2025 21

Conclusions
April 2025 22
1. Business use cases value predictability and
explainability – deterministic workflows are still
vital in a world racing towards of AGI
2. We don’t need the ‘agent’ hammer for all the
nails!
3. Agent performance is directly correlated to the
quality of the ‘semantics layer’
4. Everything becomes an optimization problem in
the end (cost/tokens/compute vs. quality of
output)
https://www.anthropic.com/engineering/building-effective-agents

@gracecai on FME Community
Q&A
April 2025 23

Peak of Data & AI Encore- AI for Metadata and Smarter Workflows

FME AI
CHECKER
VEREMES
The Peak of Data
and AI 2025

2025
The
Peak
of
Data
and
AI
Alexandre
Bijaye
FME Expert | Trainer
Veremes

2025
The
Peak
of
Data
and
AI
1. Use of an AI checking tool -Why?
2. Methods
2.1 Structure of .fmw files
2.2 Communicating with AI
2.3 Workflow Design
3. Results obtained
4. Discussion
4.1 A new approach to workflow review?
4.2 Areas of improvement
5. Conclusion
Agenda

2025
The
Peak
of
Data
and
AI
Use of an AI
Checking tool
Introduction
Why?

2025
The
Peak
of
Data
and
AI
“I think my workﬂow is ready
to be published
on our production FME Flow.”
— Alex, New Hire

2025
The
Peak
of
Data
and
AI
The workﬂow:
With a trained eye, we immediately see:
Two misconﬁgured transformers
SpatialFilter missing a connection

2025
The
Peak
of
Data
and
AI
Demo Video

2025
The
Peak
of
Data
and
AI
Analyzed Workﬂow
All issues have been detected; annotations have been attached
to their respective transformers.

2025
The
Peak
of
Data
and
AI
An intelligent & automated
FMW review tool could...
...allow for instant verification of
● Transformers configuration
● Logic and expressions in transformers according to their role
● Global workflow design
...provide instant feedback on inefficiencies and suggest potential
improvements.

2025
The
Peak
of
Data
and
AI
Structure of
.fmw ﬁles
Methods

2025
The
Peak
of
Data
and
AI
All FME Form Workflows are saved as .fmw files.
These files are XML documents.
XML files' structure is a hierarchical, tree-like organization of
tagged elements, allowing for structured data representation.
In a FME Form Workflow, we find multiple tags, for example:
● <DATASETS> Tag contains all Reader/Writer information and parameters
● <TRANSFORMERS> Tag stores all Transformers and their parameters
● <COMMENTS> Tag encapsulates all Annotations and their locations
● etc.

2025
The
Peak
of
Data
and
AI
<DATASETS> Tag stores all Reader/Writer information and parameters
<TRANSFORMERS> Tag stores all Transformers and their parameters
<COMMENTS> Tag stores all Annotations and their locations
We split the workﬂow's XML ﬁle to keep the <TRANSFORMERS> section.
The information stored in the tag will be provided to the AI service for review.

2025
The
Peak
of
Data
and
AI
Communicating
with AI
Methods

2025
The
Peak
of
Data
and
AI
Fragments of our FME Form Workﬂow (<TRANSFORMERS> section)
are being sent to OpenAI ChatGPT using the Connector.
We framed how the answer needs to be returned:
a consistent structure allows smooth data manipulation.

2025
The
Peak
of
Data
and
AI
@Value(fragment) Review the workflow by analyzing transformers parameters.The error is $(error)
YOU MUST WARN about :
- Missing critical parameters not set.
- Unparametered transformers.
- Wrong logic.
YOU MUST Format your response as follows, and do not include verbose:
- WHAT: Describe the issue.
- FIX: Propose the correction.
- IDENTIFIER: Provide the transformer’s identifier from the XML file.
- POSITION: Provide the transformer’s position from the XML file without any /oror"or'
End each set of answers with - EOC|
Main task
Example of a prompt:
Rule

2025
The
Peak
of
Data
and
AI
Example of a response following our rule:
YOU MUST Format your response as follows, and do not include verbose:
- WHAT: Describe the issue.
- FIX: Propose the correction.
- IDENTIFIER: Provide the transformer’s identifier from the XML file.
- POSITION: Provide the transformer’s position from the XML file without any /oror"or'
End each set of answers with - EOC|
Rule

2025
The
Peak
of
Data
and
AI
AI CHECKER
Project Design
Methods

2025
The
Peak
of
Data
and
AI
Project Design: 3 Main Sections
WORKFLOW
MANIPULATION
WORKFLOW
ANALYSIS
FINAL
ASSEMBLY

2025
The
Peak
of
Data
and
AI
Project Design – 1. Workﬂow Manipulation
Extraction of <TRANSFORMERS> tag
Creation of <TRANSFORMER> groups : <25k characters per feature
allows to bypass token limit per call

2025
The
Peak
of
Data
and
AI
Project Design – 2. Workﬂow Analysis
Two execution modes: Error checking or Annotation mode

2025
The
Peak
of
Data
and
AI
Project Design – 3. Final Assembly
Reconstruction of original .fmw
Integration of AI's suggestions as <COMMENTS>
Merged in original .fmw XML document

2025
The
Peak
of
Data
and
AI
Results
Obtained
Results

2025
The
Peak
of
Data
and
AI
Detection of errors
Misconﬁgured AttributeCreator transformer – Wrong logic

2025
The
Peak
of
Data
and
AI
Detection of errors
Misconﬁgured JSONFragmenter transformer – Wrong JSON Query
JSON Query should have been
json["states"][*]

2025
The
Peak
of
Data
and
AI
Detection of errors
Wrong expected geometry in PointOnAreaOverlayer

2025
The
Peak
of
Data
and
AI
Detection of errors
Missing connections between transformers
The Filter port of the SpatialFilter transformer
is not connected to anything

2025
The
Peak
of
Data
and
AI
Detection of errors

2025
The
Peak
of
Data
and
AI
Workﬂow annotation
Global annotation of all transformers of a workﬂow

2025
The
Peak
of
Data
and
AI
Reviewing
answers' quality
Results

2025
The
Peak
of
Data
and
AI
Global results
Key numbers after 100 uses
We ran FME AI CHECKER 100 times:
• Error checking mode 57 times and Annotation mode 43 times
• Overall answer quality: 6.12/10 (sd 3.2)
• We observed over 100 runs:
o Impressive results 27 times (≥ 9/10)
o Disappointing answers 28 times (≤ 3/10)
• Annotation mode showed 35/43 (>80%) acceptable results (≥ 5/10)
• Error checking mode showed 37/57 (65%) acceptable results (≥ 5/10)

2025
The
Peak
of
Data
and
AI
Over 100 workﬂow reviews,
• Incorrect Conﬁguration of Attribute Transformers
• Poorly Implemented Custom Code (Python, R)
• Miscommunication with Web Services
• Improper Transformer implementation
• Spatial-related issues
• Wrong Transformer logic
• Niche transformers (RasterPyramider, VoronoiDiagrammer, etc.) are not well reviewed
• HUB Custom Transformers and Connectors are not understood
These issues were the
easiest to identify
These issues are not
well understood

2025
The
Peak
of
Data
and
AI
A new approach to
workﬂow review?
Discussion

2025
The
Peak
of
Data
and
AI
FME AI CHECKER performance:
● 65% of the time provided instant feedback on errors and
suggested potential improvements.
● 80% of the time annotated a workﬂow in seconds.
So... Can we use a general AI service as a trustworthy review machine?
Can I cancel my "code review sessions?"

2025
The
Peak
of
Data
and
AI
Multiple issues to take into consideration
Over 100 uses, we encountered
● Inconsistencies and Laziness (15% of results)
○ Would fail to adhere properly to the required answering structure
○ Would fail to review all transformers and halt its analysis midway

2025
The
Peak
of
Data
and
AI
Over 100 uses, we encountered
● Inconsistencies and Laziness (15% of results)
○ Would fail to adhere properly to the required answering structure
○ Would fail to review all transformers and halt its analysis midway
● Hallucinations (<5% of results)
○ Could invent parameters that
do not exist in the transformer
● 'TEST_CLAUSE' does not exist
in ListConcatenator transformer

2025
The
Peak
of
Data
and
AI
● Inconsistencies and Laziness
● Hallucinations
● Token limit:
○ A limit of 30,000 tokens per call forced us to dissect the XML ﬁle to only
analyze the <TRANSFORMERS> Tag
▪ Some errors/issues could be located in Readers/Writers, Scripted
Parameters, etc...
● Won't be reviewed by this project.

2025
The
Peak
of
Data
and
AI
● Inconsistencies and Laziness
● Hallucinations
● Token limit
● Untrained on specific areas
○ A general LLM lacks knowledge over very specific information
■ This is especially true for:
● Raster-based workflows
● HUB Transformers/Connectors
● Niche Transformers

2025
The
Peak
of
Data
and
AI
● Increase/remove token limits per call
○ Would allow complete review of the XML file
● Train our own model!
○ Domain-Specific Expertise
■ Custom GPT would better understand FME transformers, workflows, and errors
○ Increased Efficiency and Automation
■ The model could automate better documentation based on learned information, and could now
suggest transformers and propose workflow optimization
○ Customization + Control
■ Tailoring the model will ensure responses are relevant to our industry – more relevance
○ Overall boosted accuracy
Areas of improvement

2025
The
Peak
of
Data
and
AI
How to improve
this project?
Discussion

2025
The
Peak
of
Data
and
AI
“Accuracy in AI is not just about getting the
right answer; it’s about understanding the
conﬁdence and limits of that answer.”
— Pr. Li Fei-Fei, Computer Scientist

2025
The
Peak
of
Data
and
AI
ThankYou
Alexandre Bijaye
VEREMES | NARBO.data
alexandre.bijaye@veremes.com | alex@narbodata.com

Peak of Data & AI Encore- AI for Metadata and Smarter Workflows

More Related Content

Similar to Peak of Data & AI Encore- AI for Metadata and Smarter Workflows (20)

More from Safe Software (20)

Recently uploaded (20)

Peak of Data & AI Encore- AI for Metadata and Smarter Workflows