Optimizing XML & JSON Workflows for Seamless Data Integration
0 likes80 views
The document outlines optimizing XML and JSON workflows for effective data integration using FME tools by addressing common challenges such as validation and automation. It includes details about the upcoming Peak of Data Integration 2025 event, along with an agenda covering the use of XML and JSON, real-world applications, troubleshooting best practices, and resources available for participants. The conclusion emphasizes FME's capability to simplify handling object-oriented web formats and enable efficient data workflows.
![● Test result by
reading back with
JSON reader
● Use json for root
and json[“info”][*]
for info array
● Set geometry
format to
GeoJSON
● Set geometry name
to area.polygon
● Set CRS to LL84
Demo: Converting CAPXML to CAPJSON
using JSONTemplater](https://image.slidesharecdn.com/optimizingxmljsonworkflowsforseamlessdataintegration-oct3020243-241030174419-ffae71c0/85/Optimizing-XML-JSON-Workflows-for-Seamless-Data-Integration-38-320.jpg)
![● Test result by
reading back with
JSON reader
● Use json for root
and json[“info”][*]
for info array
● Set geometry
format to
GeoJSON
● Set geometry name
to area.polygon
● Set CRS to LL84
Demo: Converting CAPXML to
CAPJSON using JSONTemplater](https://image.slidesharecdn.com/optimizingxmljsonworkflowsforseamlessdataintegration-oct3020243-241030174419-ffae71c0/85/Optimizing-XML-JSON-Workflows-for-Seamless-Data-Integration-39-320.jpg)
Optimizing XML & JSON Workflows for Seamless Data Integration
- 1. Optimizing XML & JSON Workflows for Seamless Data Integration
- 2. Tame Your Data Monsters… XML & JSON don’t have to be SCARY!
- 3. Dean Hintz Technical Support Team Lead, Strategic Solutions Safe Software Kailin Opaleychuk Technical Support Specialist, FME Form Safe Software Meet the Team
- 4. Peak of Data Integration 2025 Seattle, WA | May 6–8, 2025 ● Abstracts due Nov 29th, 2024 ○ All levels FME proficiency welcome ○ 15 & 25 min options (incl. Q&A) ○ Special speaker rate for accepted talks ● Registration is open! peakofdataintegration.com
- 5. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 6. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button
- 7. Poll: What do you use more, JSON or XML?
- 8. Agenda 1 Introduction 2 Using XML & JSON in FME 3 XSD-Driven XML 4 Transformer Comparison 5 Real-World Use Cases & Live Demos 6 Troubleshooting & Best Practices 7 Conclusion 8 Resources & Next Steps 9 Q&A Agenda
- 10. One platform, two technologies FME Form FME Flow The authoring environment for all workflows The deployment environment delivering services to your organization FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 11. Work with any API and save time handling, extracting and validating XML and JSON with FME.
- 12. Key Challenges Common frustrations people face with this topic are: ● Object models integrate GIS ● Dealing with nested data <-> relational ● Handling XML quirks like namespaces, code lists, number & data types ● Validation - sometimes difficult to get output to validate and diagnose problems ● Automation - many XML and JSON workflows involve scripts that may be difficult to integrate and automate without FME
- 13. You don’t have to face these frustrations. FME offers streamlined solutions for handling object models, nested data, XML quirks, validation, and automation, making your workflows smoother and more efficient!
- 14. XML Format ● Object model ● Schema specific ● XML elements, namespaces ● Tightly structured, formal rules, names space, code lists & internal references (such as xlinks) ● Data type can be important (e.g. ISO date format) ● Bigger ● Easier to validate JSON Format ● Object model ● Usually more application oriented ● JSON objects, Key value pairs ● JSON can be less formal, more flexible, easier to adopt, use & implement. ● Generally strings or typeless ● More streamlined ● Harder to validate
- 15. Why XML & JSON? ● Structured text is easier to work with than raw or binary formats. ● Similar structures: Making reading, writing, and GIS transformations easier. ● XML: Ideal for standards with built-in schema validation and support for complex geometries (e.g., OSM, INSPIRE, AIXM, CityGML, national mapping formats). ● JSON: Streamlined for REST services, lightweight apps, and mobile clients (e.g., GeoJSON, ESRIJSON, IMDF, STAC). ● Web services and enterprise automation are built primarily on JSON and XML ● Many open standards are encoded in XML, GML or JSON ● Both are foundational technologies. Core concepts haven’t changed, even as uses evolve. Here to stay.
- 16. Flash Demo
- 17. Summary Slide ● National Weather Service (NWS) API was used to return DSWL XML and JSON-LD forecasts ● To change the request Header Accept value ○ Accept application/vnd.noaa.dwml+xml ○ Accept application/ld+json
- 18. XML
- 19. JSON
- 20. 2 Using XML & JSON in FME
- 21. Public Safety / Health ● HL7 / FHIR ● NG911 ● CAP ● DWML General ● ATOM ● XHTML ● GeoRSS ● GML ● GeoJSON Mapping / Surveying ● OSM ● INSPIRE ● JSON ND ● 12D ● GPX ● ESRIJSON* 3D / AEC / BIM / Indoor ● CityGML ● CityJSON ● SOSI ● IMDF Utilities / Transportation ● CIM ● AIXM How FME Reads, Writes, and Validates: XML & JSON Found in virtually every industry.
- 22. New JSON Schema (coming soon) 1. Author JSON in JSONTemplater ○ Schema unaware 2. Produce flat JSON ○ Handles Feature type names ○ JSON by row ○ No nesting, series or data type support 3. Schema-Driven JSON (screenshot from XSD XML) ○ Series ○ Nested ○ Data-types Supports OpenAPI
- 23. 3 XSD-Driven XML
- 24. What is an XML Schema Definition? ● AKA, XSD. ● Defines the dataset schema, including the allowable elements and attributes ● Outlines rules or constraints for XML documents, such as data types & the number and order of child elements ● Default values and code lists ● Very useful for validation ● May be useful for data interpretation
- 25. When ShouldYou Use an XSD Over a Templater? ● XMLTemplater: requires knowledge of exact XML structure you want to produce ○ Manually define templates for individual objects and parent objects ● XSD-XML: More automated. Objects, properties, data types derived from XML schemas ○ Requires to know object types you want to write to, plus underlying XML configuration (namespaces, codelists etc)
- 26. How to Create & Read an XSD-Driven XML File
- 27. Demo
- 28. Summary Slide ● Used a CSV to generate an output XSD schema and XML dataset using the XML Writer ● Re-read the XML with the XSD-Driven XML Reader
- 29. Generated XSD
- 31. Introduction to XML & JSON FME transformers 1. XML/JSONFragmenter 2. XML/JSONTemplater 3. XML/JSONFormatter 4. XML/JSONFlattener 5. XML/JSONValidator (we’ll speak to this later in the webinar)
- 32. Demo
- 33. Demo: Parsing JSON with JSONFragmenter and JSONFlattener
- 34. Summary Slide ● XML and JSON transformers > similar functionality, going from nested to flattened ● Both require the use of an AttributeExposer, import from feature cache to bring schema onto features ● General workflow for reading is fragment then flatten ○ Ability to flatten within the fragmenter ● General workflow for writing is templater followed by validator ● XML can be written using XMLTemplater + Text File Writer or XSD Writer ● JSON is typically written using JSONTemplater + Text File Writer
- 35. Demo: Converting CAPXML to CAPJSON using JSONTemplater ● JSONTemplater requires sample JSON objects for root, children ● Obtain from existing JSON or generate with JSON writer ● We use XML to JSON workflow to create JSON Templates ● Filter out unwanted attributes and use Automatic mode for destination schema *Note: don’t select all for feature path, just select nodes for feature types *
- 36. ● Create XML to Textfile Writer workspace ● Add JSONTemplater and connect alert to it ● Paste json for alert into root template and info into sub-template ● Use AttributeSplitter to break up geometry into vertex pairs ● Create GeoJSON string (AttributeCreator) ● Duplicate info record, modify attributes then feed to SUB ● Validate and format before writing Demo: Converting CAPXML to CAPJSON using JSONTemplater
- 37. ● Paste json for alert into root template and info into sub-template ● Add fme:process-features(“ SUB”) to embed child info array within alert. ● To merge attributes into the template, select the value in the JSON, then double click the corresponding FME Feature Attribute ● fme:get-attribute for strings and fme:get-json-attribute for json fragments Demo: Converting CAPXML to CAPJSON using JSONTemplater alert root template info sub-template JSONTemplater
- 38. ● Test result by reading back with JSON reader ● Use json for root and json[“info”][*] for info array ● Set geometry format to GeoJSON ● Set geometry name to area.polygon ● Set CRS to LL84 Demo: Converting CAPXML to CAPJSON using JSONTemplater
- 39. ● Test result by reading back with JSON reader ● Use json for root and json[“info”][*] for info array ● Set geometry format to GeoJSON ● Set geometry name to area.polygon ● Set CRS to LL84 Demo: Converting CAPXML to CAPJSON using JSONTemplater
- 40. 5 Real-World Use Cases & Live Demos
- 42. Customer Story: Pacific Spatial: CityGML / Takashi user story
- 43. Customer Story: Pacific Spatial: CityGML
- 44. Montreal: CityGML Many cities publish CityGML ● Montreal, NYC, USA ● Across Europe ● Japan Rich data model ADE - extensions for Energy, Planning, Utilities
- 45. Montreal: CityJSON ● Lighter weight - about size 50% of CityGML ● Faster - no schema scan ● Easier for thin clients ● CityJSON v1 based on CityGML2, CityJSON v2 based on CityGML3 ● Not as widely adopted as CityGML ● Harder to validate - json schema? ● More nested - walls, roofs inside buildings
- 46. NG911 Demo
- 47. Customer Story Santa Clara County Project Improve 911 dispatch system with a map of city-sourced address points. Solution Integrated 17 city datasets, covering the entire county, performed QA, and generated multiple output formats. Results ● Regional address maps that cities contribute addresses to on a quarterly basis. ● 50% increase in the number of known addresses. ● Improved emergency response time and location accuracy. “FME allowed me to make fast iterative changes to workflows as each city’s data turned out to be a discovery process where something unexpected always occurred. This flexibility better prepares me for upcoming changes that may be necessary for Next Generation 911.” - Steven Hong, Santa Clara County
- 48. Input Data: Source CSV from data.winnipeg.ca NENA CLDFX XML from Winnipeg Parcel Data
- 49. Demo Workflow Convert from CSV to CLDFX XML: 1. Read Parcel CSV 2. Filter features for valid addresses 3. Map attributes to NENA CLDFX schema with AttributeCreator 4. Create collection container for root 5. XSD XML writer to write CLDFX using civicAddr.xsd Parcel CSV to NENA CLDFX XML
- 50. Result: ● NENA CLDFX XML ● Validate against civicAddr.xsd ● Test by reading back with the XSD XML reader Parcel CSV to NENA CLDFX XML
- 51. Customer Story: Finavia & Spatialworld - Improving Safety & Operations at Finland’s Airports ● FME for automatic PDF map generation ○ Security, ports, police, fire ○ Process topographic data from DGN, PostGIS ● Topographic, cadastral, and basemap databases ○ Retrieve data from National Land Survey ○ Write to PostGIS ● Flight obstacle maps ○ Integrate Finland’s LAS (point cloud) datasets ○ Analyze data, identify high elevations ● Water monitoring automation ○ Check XML file from consultants, perform QA ○ Trigger email alerts, transform/convert data to PostGIS
- 52. AIXM Demo
- 53. Convert YVR AIXM 4.5 RWY, RWY_DIRECTION to AIXM 5.1 Runways 1. Read AIXM 4.5 and perform joins 2. Interpret parameters & build geometries 3. Perform schema mapping & define feature properties 4. Transform geometry to match destination schema (GeometryPropertySetter) 5. Generate ids & parent / child relationships 6. Write parent, child and container elements to AIXM5 AIXM 4.5 to 5.1 Demo
- 54. AIXM 4.5 XML to AIXM 5.1 GML AIXM 4.5 XML RWY to AIXM 5.1 GML Runways
- 56. ● Tips for optimizing performance when handling large XML & JSON datasets (AttributeRemover/Keeper) ● Feature Paths vs XSD Types (R/W)* ● Small test datasets, pretty printing, logs, Notepad++ (compare result with source) ● JSON: syntax, array vs root ● XML: schema, multi-root, duplicate Ids, namespaces, parent / child relations ● XML Validation: ○ Syntax, Syntax and Namespace, Syntax and Schema ○ error message interpretation* Reach out to us! Troubleshooting Best Practices *element 'sender' is not allowed for content model '(identifier,sender,sent,status,msgType,source?,sco pe,restriction?,addresses?,code*,note?,references? ,incidents?,info*)' (from 3rd party library) If the structure or syntax is wrong, you may get an error message, or simply no element written (JSON). Reading back is the ultimate test.
- 57. ● Remove unneeded fragments, lists often (AttributeRemover/Keeper) ● XSD Types more comprehensive than Feature Paths ○ XSD types as features = thoroughfare: reads trail, street or road; with feature paths need a path for each of trail, street road ● Small test datasets, pretty printing, Notepad++ (compare result with source) ● XML Validation: ○ Check Syntax, then Syntax and Namespace, then Syntax and Schema ○ error message interpretation* ● Reading back is the ultimate test. ● Check your logs! ● More than one way to read a dataset (ignore schema) Best Practices
- 58. 7 Conclusion
- 59. Summary ● FME helps you address the challenges of working with object oriented web formats like XML & JSON and integrating them with relational / GIS ● FME’s XML & JSON tools allow you to easily access your data so you can spend more time building application workflows and less time wrangling data ● XML, JSON are key to integrating with web service workflows, APIs, enterprise automation & standards ● Automation is key, whether you need to interpret schemas, transform your data, or publish workflows ● FME supports validation and diagnostic approaches ● We track new developments and data types so you don’t have to: CityGML3, OGC APIs, JSON schema
- 60. 6 Safe Software
- 61. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 30+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
- 62. Safe Software is recognized as Customers’ Choice again in the 2024 Gartner Peer Insights ‘Voice of the Customer’: Data Integration Tools report. We are now recognized as Customers’ Choice in North America and Midsize Enterprise segments. GARTNER is a registered trademark and service mark, and PEER INSIGHTS is a registered trademark, of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved. Gartner Peer Insights content consists of the opinions of individual end users based on their own experiences, and should not be construed as statements of fact, nor do they represent the views of Gartner or its affiliates. Gartner does not endorse any vendor, product or service depicted in this content nor makes any warranties, expressed or implied, with respect to this content, about its accuracy or completeness, including any warranties of merchantability or fitness for a particular purpose. Read Full Report
- 63. 7 Resources
- 64. Resources ● Getting Started with GML, XML & Web Services ● How to Consume and Produce XML using Application Schemas with the XSD-Driven XML Writer ● How to Read XSD-Driven XML ● Getting Started with XML ● Getting Started with JSON ● Getting Started with GML ● Working with Web Services ● Blog: Open Standards ● Blog: CityGML
- 65. Resources: Webinars ● JSON ● XMLathon ● Metadata ● CityGML ● AIXM ● REST APIs Dive into more specifics for your favorite data types or services
- 66. Resources: FMEHub ● XML ● JSON ● Connectors ● CityGML
- 67. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base support.safe.com Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 68. Check out our podcasts on-demand. featuring special guest speakers over at EM360 Resources
- 69. 8 Next Steps
- 70. We’d love to help you get started. Get in touch with us at info@safe.com Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 71. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: WE7JVX Join the Community today! Next Steps
- 72. 9 Q&A
- 73. ThankYou Recap of Next Steps 1 Follow us on LinkedIn! 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey