Scaling the Peak - AWS, FME & Snowflake Spatial

Scaling the Peak
AWS, FME & Snowﬂake Spatial

Presenter
The
Peak
of
Data
Integration
20
23
Tristan Wright
Sr Principal - GIS Development
S&P Global
Commodity Insights

The
Peak
of
Data
Integration
20
23
At S&P Global Commodity Insights, our complete
view of global energy and commodities markets
enables our customers to make decisions with
conviction and create long-term, sustainable value.
North America – FME
○ Spatial data ingestion
○ Building spatial derived data
○ Delivering spatial data to our applications
○ Delivering spatial data to our customers

The
Peak
of
Data
Integration
20
23
Agenda
1. Who are We?
2. What was the goal?
3. AWS Architecture
4. Snowﬂake Spatial
5. FME Workﬂow Example

The
Peak
of
Data
Integration
20
23
Introduction

The
Peak
of
Data
Integration
20
23
Create a cloud-based FME
environment to replace our
on-prem FME Servers

The
Peak
of
Data
Integration
20
23
What were the requirements?
• AWS-based
• Single environment for all FME jobs
• User access for multiple data teams
• Reduce ESRI licensing costs
• Optimize use of FME engines

The
Peak
of
Data
Integration
20
23
Consolidation of FME Servers
Multiple FME servers within each data center.
2-16 Static Engines per server
All ESRI ArcGIS capable
Single FME
environment

The
Peak
of
Data
Integration
20
23
Technical Challenges
• Capacity:
• How many engines?
• How to optimize engines
• Can it be scalable based on demand?
• Reliability
• Our entire spatial data operations depend on one single server environment
• Performance
• Complex & compute intensive processes
• Need lots of engines, but for a short amount of time (once per week)
• Cost
• Reduce ESRI dependency
• Optimize engine infrastructure

The
Peak
of
Data
Integration
20
23
The solution
AWS, FME & Snowﬂake

The
Peak
of
Data
Integration
20
23
AWS FME Architecture
Distributed FME architecture
Built on Windows EC2 instances
How many engines per instance?
Engine queues for purpose
Scalable default queue

The
Peak
of
Data
Integration
20
23
AWS Deployment
● Terraform & Azure Devops Pipeline
● Blue/Green deployment
● Daily FME backups to S3 bucket
● Automated monthly rebuild for patching
● Health checks
● Disaster recovery to different AWS regions

The
Peak
of
Data
Integration
20
23
Spatial Data Storage – ESRI SDE or Snowflake?
○ Source data already in Snowflake
○ Can we use snowflake to store our spatial data?
○ What are the benefits & performance gains?

The
Peak
of
Data
Integration
20
23
What is Snowﬂake?
Cloud-based database platform
“Self-managed” service, no need for DBAs or IT.
Spatial storage options: Geography (WGS84) or Geometry
Powerful geospatial SQL functions.
FME “Snowﬂake Spatial” readers & writers!

The
Peak
of
Data
Integration
20
23
Multiple Geographies
per Table
● Store multiple
geographies & coordinate
systems in the one table
● Simpliﬁes storage of
spatial data
● Previously in ESRI SDE
these would have all been
separate feature classes

The
Peak
of
Data
Integration
20
23
Snowﬂake Geospatial
Functions
Native SQL functions for processing and
manipulating spatial data
Spatial join between:
Well production data: 77 million rows
Area-of-interest polygons
SQL completed in 12 seconds
Reading data from 2 different databases

The
Peak
of
Data
Integration
20
23
How does this help FME?
● Use snowflake SQL to prepare the
data for FME
○ spatial filters and joins
● Best of both tools
○ Powerful and fast snowflake SQL
functions
○ FME to transform and build the data
● FME only reads the data that it
needs.
● Need less memory and CPU
resources
● Eliminates data silos – no longer
need “replication” ETL

The
Peak
of
Data
Integration
20
23
Optimizing Engine Usage
● Switch to snowﬂake spatial
● Parallel processing - Break ETL processes into chunks
○ E.g. North American processes we might break into “Plays”
and process them individually.
○ Data delivery processes use dynamic workbench with
customer database table

The
Peak
of
Data
Integration
20
23
FME Workﬂow – Acreage Grading
Derived data generated by FME.
Input Datasets:
● North American Well & Borestick data
● Production data
● Acreage grid polygons
FME parallel processing by play & reservoir. Generates entire North America dataset

The
Peak
of
Data
Integration
20
23
FME Workﬂow
Acreage Grading
1. Create well bore 3D lines
from directional survey data
Snowflake FME
Retrieving & ordering
survey data
Building line geometries
Deriving 3D profile points
(heel, kick off point, toe)
Creating lateral line
geometries

The
Peak
of
Data
Integration
20
23
FME Workﬂow
Acreage Grading
2. Join to production data
and overlay 1 mile x 1 mile
polygon grid
Snowflake FME
Querying production
data
Polygon grid creation
Spatial filter polygon
grid
Chopping laterals to grid
squares & summary stats

The
Peak
of
Data
Integration
20
23
FME Workﬂow
Acreage Grading
3. Grid calculations & quality
interpolation
Snowflake FME
Grid data storage &
retrieval
Quality interpolation
Spatial point-in-polygon
joins to full well dataset
Aggregation & summary
stats per grid square
(majority operator, total
footage of laterals, etc)

The
Peak
of
Data
Integration
20
23
FME Workﬂow
Acreage Grading
4. 3D nearest neighbor &
well spacing calculations
Snowflake FME
Spatial filtering 3D nearest neigbour
calculation to find best
neighbor candidates
Candidate selection
based on attribute data
Determining orientation &
wells located on Left vs
Right side of well bore

The
Peak
of
Data
Integration
20
23
FME Workﬂow - Acreage Grading
✔ By combining the power of Snowﬂake Spatial functions and FME parallel
processing the entire North American dataset can be rebuilt in 2 hours.
✔ This allows us to generate the data more regularly and improves data currency
for customers.
✔ Using a scalable FME environment we get performance when we need it ,
without the cost of having a large number of engines running when we don’t
need them.

The
Peak
of
Data
Integration
20
23
Conclusion

The
Peak
of
Data
Integration
20
23
Summary
● Consolidation and simpliﬁcation of multiple FME environments
● Scalable environment: ﬂexibility, reliability and availability
● Impacted dependency on 3rd party licensing

The
Peak
of
Data
Integration
20
23
Cloud technology is constantly
changing, how can you bring new
technology into your FME
Workﬂows?

Call to Action
1. Invest in research time – new technologies & platforms
2. Learn about Snowﬂake – Snowﬂake University
3. Challenge yourself to make every FME process more
efficient

The
Peak
of
Data
Integration
20
23
Resources
Deploying FME Server in the cloud
Snowflake University – Free Courses
Snowflake Geospatial Data Types
Snowflake Geospatial Functions
S&P Global – Commodity Insights

ThankYou!
tristan.wright@spglobal.com

© 2023 by S&P Global Inc. All rights reserved. S&P Global, the S&P Global logo, S&P Global Commodity Insights, and Platts are trademarks of S&P Global Inc. Permission for any commercial use of these trademarks must be obtained in
writing from S&P Global Inc. You may view or otherwise use the information, prices, indices, assessments and other related information, graphs, tables and images (“Data”) in this publication only for your personal use or, if you or your company
has a license for the Data from S&P Global Commodity Insights and you are an authorized user, for your company’s internal business use only. You may not publish, reproduce, extract, distribute, retransmit, resell, create any derivative work
from and/or otherwise provide access to the Data or any portion thereof to any person (either within or outside your company, including as part of or via any internal electronic system or intranet), firm or entity, including any subsidiary, parent, or
other entity that is affiliated with your company, without S&P Global Commodity Insights’ prior written consent or as otherwise authorized under license from S&P Global Commodity Insights. Any use or distribution of the Data beyond the
express uses authorized in this paragraph above is subject to the payment of additional fees to S&P Global Commodity Insights. S&P Global Commodity Insights, its affiliates and all of their third-party licensors disclaim any and all warranties,
express or implied, including, but not limited to, any warranties of merchantability or fitness for a particular purpose or use as to the Data, or the results obtained by its use or as to the performance thereof. Data in this publication includes
independent and verifiable data collected from actual market participants. Any user of the Data should not rely on any information and/or assessment contained therein in making any investment, trading, risk management or other decision. S&P
Global Commodity Insights, its affiliates and their third-party licensors do not guarantee the adequacy, accuracy, timeliness and/or completeness of the Data or any component thereof or any communications (whether written, oral, electronic or
in other format), and shall not be subject to any damages or liability, including but not limited to any indirect, special, incidental, punitive or consequential damages (including but not limited to, loss of profits, trading losses and loss of goodwill).
ICE index data and NYMEX futures data used herein are provided under S&P Global Commodity Insights’ commercial licensing agreements with ICE and with NYMEX. You acknowledge that the ICE index data and NYMEX futures data herein
are confidential and are proprietary trade secrets and data of ICE and NYMEX or its licensors/suppliers, and you shall use best efforts to prevent the unauthorized publication, disclosure or copying of the ICE index data and/or NYMEX futures
data. Permission is granted for those registered with the Copyright Clearance Center (CCC) to copy material herein for internal reference or personal use only, provided that appropriate payment is made to the CCC, 222 Rosewood Drive,
Danvers, MA 01923, phone +1-978-750- 8400. Reproduction in any other form, or for any other purpose, is forbidden without the express prior permission of S&P Global Inc. For article reprints contact: The YGS Group, phone +1-717-505-9701
x105 (800-501-9571 from the U.S.). For all other queries or requests pursuant to this notice, please contact S&P Global Inc. via email at ci.support@spglobal.com

Scaling the Peak - AWS, FME & Snowflake Spatial

More Related Content

Similar to Scaling the Peak - AWS, FME & Snowflake Spatial (20)

More from Safe Software (20)

Recently uploaded (20)

Scaling the Peak - AWS, FME & Snowflake Spatial