XML-athon with Don and Dean
0 likes2,493 views
The document outlines the features and capabilities of FME, particularly focusing on its support for XML, GML, and JSON formats. It includes various demonstrations and strategies for reading, writing, and transforming XML data, as well as highlights the importance of using specific formats and tools for optimal data management. Additionally, it emphasizes FME's role as a no-code integration solution for diverse data types and its ongoing enhancements to support evolving standards.
![{"employees":[
{ "firstName":"John", "lastName":"Doe" },
{ "firstName":"Anna", "lastName":"Smith" },
{ "firstName":"Peter", "lastName":"Jones" }
]}](https://image.slidesharecdn.com/xmlwebinar1-200708212008/85/XML-athon-with-Don-and-Dean-18-320.jpg)
![Let us know in the
chat…
[question]](https://image.slidesharecdn.com/xmlwebinar1-200708212008/85/XML-athon-with-Don-and-Dean-92-320.jpg)
XML-athon with Don and Dean
- 1. “XML-athon” With Dean Hintz and Don Murray
- 2. Don Murray Co-Founder Dean Hintz Senior FME Data Expert Product Owner XML Technology Presenters Trent Kading XML and FME Expert on Q&A
- 3. Agenda ● Who we are and what do we do? ● XML & The FME Platform ● XML Highlights and Strategies A dozen demos that just touch on what FME can do. ● CityGML, Simple Reading/Writing, Validation ● Using XSD documents to enhance XML Experience ● Metadata, OSM, Catalogs and Web API’s ● Complex XML & GML, Converting XML to JSON. ● Resources and Q&A
- 4. OUR MISSION To help organizations maximize the value of data.
- 5. FME Desktop FME Server FME Cloud Build & Run Workflows Automate Workflows Hosted FME Server FME® Integration Platform Connect. Transform. Automate. FME is the data integration solution with the best support for spatial data worldwide.
- 6. An extension for ArcGIS Pro and ArcMap developed by Safe Software. FME Desktop functionality. Data Interop in 2.6 is exciting!!
- 7. Let’s Get Started and Have fun!!!
- 8. FME Supported Data Types - Slicing XML for ~ 20 years! CAD GIS FME release by year NumberofsupportedformatsinFME 1995 10500100300 20202015201020052000 TABULAR DATABASE RASTER POINT CLOUD BIM 3D WEB XML/ JSON CLOUD BIG DATA IOT BI AR / VR AI / ML INDOOR MAPPING GAMING XML Representations
- 9. FME support for XML is Wide and Deep!
- 10. Let us know in the chat what type of XML data you are working with.
- 11. XML Formats supported in FME
- 12. All are XML! FME wraps formats to help you. XMLGMLCityGML Adds Value GeometryTextures Adds Value
- 13. USE MOST SPECIFIC FORMAT FME supports many XML variants and there are many ways to work with a specific file.. The best way is almost always using the specific format. Reader Reader Reader
- 14. Demo: General -> Specific Format Support (CityGML) 1.
- 15. Most General Reader : XML Reader Read CityGML by XML reader Feature Path: Building No geometry - must process in workspace -> GreenBuildingXML example
- 16. ● Read CityGML with GML Reader (no schema mode) ● Reads features, geometries ● Ignores textures since these are not part of GML standard ● Auto reader selection More Specific Reader: GML Reader
- 17. Read CityGML with CityGML Reader Note textures and citygml properties read (LOD) Richest support when most specific R/W chosen Most Specific Reader: CityGML Reader
- 18. {"employees":[ { "firstName":"John", "lastName":"Doe" }, { "firstName":"Anna", "lastName":"Smith" }, { "firstName":"Peter", "lastName":"Jones" } ]}
- 20. JSON Formats Note: IMDF, CityJSON, TopoJSON - some of the newer formats added recently
- 21. XML Format Highlights XML GML KML OGCWFS XSD based XML 3D: CityGML, Indoor GML GMLJP2000 INSPIRE GML Geodatabase - xml metadata AIXM Geo RSS UK OS Mastermap AEC/CAD/BIM: 12dXML, IFCXML GML in JP2000
- 22. FME’s XML Power ● XSD based XML Reading and Writing ● Standard XML Reader / Writer ● List Processing ● GML Reader / Writer ● WFS Reader - complex GML support ● Writer validation / XMLValidator ● XSLT, XFMaps ● XML Transformers Most of these will be highlighted within the examples below XML
- 23. Does Zipster Run on XML or Love XML? Zipster runs on machine code called: Xtraordinary Machine Language XML Zipster Loves XML! But....
- 25. XML Transformers - Favorites XMLFragmenter / XMLFlattener XMLValidator XMLFormatter XMLUpdater ListExploder Honorable mention: ● XMLXQueryExploder Historical mention: ● XMLTemplater
- 26. XML Concepts Tag <Course> XML Element <Course>FME</Course> Child element <Course><Id>101</Id>… Element attribute <Course id=“101”> GML Object Object property Object attribute XML Schema - xsd's Namespaces <ps-f:geometry>
- 27. XML Data Structures: Object vs Relational attribute1 attribute2 Active Date_From Active Date_To Coordinates John Vancouver 11-22-99 12-11-09 -123.1,49.25 -122.9,49.15 June Surrey 02-25-05 9-15-10 -122.8,49.12 -122.5,49.0
- 28. XML Strategies XML comes in many flavours so important to have a well thought out approach to leverage the tools you have and quickly get the data in the form you need ● Understand the type, logical structure and meaning of your source data ● Understand your target data model and what transforms are likely to be needed ● Choose the tools best suited (Readers/Writers/Transformers) to do the work
- 29. XML Strategies: Reading Use most specific format (Text or XML or GML or profile?) ● How to identify format type? (GML, XML, HTML, profile) ● Format clues? See namespace prefixes and header, geometry types, xsd ● Use application schema if there is one (ask or search) If XML not specific supported then use our generic XML tools: ● Don’t forget to inspect data first: Data Inspector, Notepad++ ● Feature Paths ● XMLFragmenter / XMLFlattener ● List processing ● GeometryReplacer
- 30. XML Strategies: Writing ● Use most specific writer format (Text or XML or GML or profile?) ● Use XSD if it is available ● If XML not specifically supported then use our generic XML tools: ○ XMLTemplater ○ GeometryExtractor ○ List processing: ListRenamer (now supports manipulation of entire lists) ○ AttributeCreator, Aggregator to build lists ○ XMLFormatter and XMLValidator ○ Other writer parameters: namespace override ○ Destination schema: understand differences between xml data types ■ xml_buffer, xml_geometry, xml_xml ■ node level writing - new in 2019.2+
- 32. Arbitrary XML/No schema: UNESCO World Heritage Sites ● Inspect first to determine type of xml ● Read with Feature Path: /query/row https://whc.unesco.org/en/list/xml/
- 33. ● Nested XML sliced into flattened features at selected node ● Parent.child notation within features for nested attributes UNESCO World Heritage Sites: Read with Feature Path
- 34. 1. Add XML reader: a. Dataset: https://whc.unesco.org/en/list/xml/ b. Elements to match query/row 2. VertexCreator: longitude, latitude 3. CoordSystemSetter:LL84 4. KMLPropertySetter a. Name: site b. Title: Heritage Sites c. Content: short_description 5. Drop xml format fields 6. Fanout: category UNESCO World Heritage Sites: XML to KML Workspace
- 35. Set kml_name, description, folder Geometry and attribute schema Illustrates how FME supports easy extraction from XML to client friendly formats like KML UNESCO World Heritage Sites: KML Result
- 36. Q&A
- 37. Demo: Writing Simple XML (using XML Writer, XMLTemplater) 3.
- 38. Writing Flat XML Database export with generic XML writer Automatic schema definition based on source or feature schema Database export with generic XML writer
- 39. Default values for most parameters Make sure output data and schema not suppressed Writing Flat XML: Workspace
- 40. FME generates namespaces as needed Preview output in XML Editor Writing Flat XML: Result
- 41. Writing Simple Flat XML: Result Before FME 2020, could validate using XSD but had to read XML data with feature paths and ignore schema Now output can be checked with Data Inspector using XSD
- 42. Write XML with XMLTemplater - Metadata Insert attribute values using fme:get-attribute{} functions Write out XML document using Text File writer
- 43. Write XML with XMLTemplater - Metadata Insert attribute values using fme:get-attribute{} functions Write out XML document using Text File writer
- 45. XML Validator
- 46. XML Validator Possible Values: ● None ● Syntax ● Syntax and Namespace ● Syntax and Schema New in 2020: ● Multiple XSD support Also useful to check if data you receive is valid Test to see if data you produce is valid!
- 48. XSD Driven XML Reading - BeerXML Read XML feature types based on application schema and feature paths (different path). Feature type attribute and geometry definitions based on XSD not data. Similar to GML application schema based reading Prefix with namespace to differentiate types
- 49. XSD Driven XML Reading - BeerXML 1. Set source XML dataset and application schema paths 2. Set feature paths (base:beer_xml) 3. Set XSD Types to be features (all except units) 4. Max attributes per nested attribute - becomes xml fragment after that
- 50. XSD Driven XML Reading - BeerXML Reading recipe additions (schemas from beerxml.com )
- 51. Demo: XSD Driven Writing (Emergency Alerts) 6.
- 52. ● Cap 1.2 XML ● Write XML using Feature type definitions based on XSD, path. ● Set attribute values as you would any typical FME format - no templaters required. ● Similar to GML application schema based writing ● Validation option XSD Driven XML Writing: Common Alerting Protocol
- 53. XSD Driven XML Writing: Common Alerting Protocol http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2.html
- 54. XSD Driven XML Writing: CAP - Validation Pt 2 Missing attribute msgType: ERROR |XML Validation: Error in ‘6.EmergencyAlerts_CAP1.2 XMLcapOut.xml' on line 27, column 13: 'element 'source' is not allowed for content model '(identifier,sender,sent,status,msgType,source?,scope,restriction?,addresses?,code*,note?,referen ces?,incidents?,info*) Missing attribute ‘severity’: ERROR |XML Validation: Error in ‘6.EmergencyAlerts_CAP1.2 XMLcapOut.xml' on line 26, column 13: 'element 'certainty' is not allowed for content model '(language?,category+,event,responseType*,urgency,severity,certainty,audience?,eventCode*,effe ctive?,onset?,expires?,senderName?,headline?,description?,instruction?,web?,contact?,parameter *,resource*,area*) Messages from open source Apache Xerces parsing library
- 55. XSD Driven XML Writing: Common Alerting Protocol
- 56. OGC Open API: Features - WFS3 New, Updated Tutorials: GML, XML, JSON CityGML v3 GeoSciML* S121 GML* CIM XML ARML FXXM/WXXM* List Processing Others? What is Next for XML? *Stay tuned!
- 57. Q&A
- 58. Choose your XML Adventure Vote on: ● Arbitrary XML ● Catalogues ● Metadata
- 60. XSD Based XML R/W: ISO 19115 Metadata XSD based reader / writer new in FME 2020 Updates contact{0}
- 61. XSD Based XML R/W: ISO 19115 Metadata Reader settings: Feature Path Writer settings: Feature Path
- 62. XSD Based XML: ISO 19115 Metadata: XMLTemplater
- 63. XSD Based XML R/W: ISO 19115 Metadata Result source updated
- 65. OSM XML: Generating 3D Buildings Workflow OSM comprehensive global coverage for base map data Full access to rich property sets
- 66. OSM XML: 3D Buildings Visualization FME can read all available OSM outputs: XML, PBF, Geopackage and GeoJSON. Each format has its own properties Data from: https://extract.bbbike.org/
- 67. Demo: Catalogs / Web API 9.
- 68. OGC CSW Client Workspace Series of HTTPCallers send requests to CSW service for Entries, Details and then WFS features based on user selection
- 70. CSW Client - Details: URI
- 71. CSW Details URI: WFS Feature Retrieval S-121 GML
- 72. Q&A
- 73. XML Resources Tutorials & Articles XML Tutorial https://knowledge.safe.com/articles/29553/tutorial-getting-started-wit h-xml.html GML Tutorial - INSPIRE https://knowledge.safe.com/articles/1321/eu-inspire-initiative-tutorial. html JSON Tutorial https://knowledge.safe.com/articles/39188/tutorial-getting-started-wit h-json.html Reading XML/GML https://knowledge.safe.com/articles/888/reading-xmlgml.html GML - Writing https://knowledge.safe.com/articles/812/gml-writing-with-application- schemas.html
- 74. Title Join us tomorrow! Keeping Cool, Calm, and Connected with FME 2020.1
- 75. Choose your XML Adventure Vote on: ● XML Web APIs ● OSM ● XML to JSON
- 77. Power of XML Reader: GreenBuilding XML Custom reader available on FME Hub XML reader assembles object attributes and geometries to construct BIM components gbXMLStandard Single Family Residential 2016.xml
- 78. Power of XML Reader: GreenBuilding XML ChapelHillOffice.xml
- 79. Power of XML Reader: GreenBuilding XML Custom format just looks like any other format when reading from Data Inspector or a workspace Data from gbxml.org
- 80. Power of XML Reader: GreenBuilding XML 1.Explode to points elements 2.Generate point geometry 3. Build line, area and surfaces
- 81. Is there an easier way for Green Building XML? hub.safe.com
- 82. XSLT Example: S-121 GML to HTML ● Option to specify XSLT path on GML or XML writer ● Example from the OGC Maritime Boundaries Pilot ● XSLT takes output S-121 GML and transforms it into HTML regulatory text representation ● Net result is a fully automated DFO/CHS geodatabase to HTML via GML to support their regulation submission workflow
- 84. Power of Generic GML R/W: Complex GML These next 3 examples show the power of FME to support reading and writing complex GML, such as that used for scientific applications and complex regulatory models. GeoSciML: Just drag and drop demo dataset into DI - GML reader automatically gets the xsd from the ns header - make sure you set axis order to 1,2. This is Australian data - geoscience australia. WXXM: need to specify the application schema in the folder provided: wxxm.xsd S-121 GML: No need to specify app schema. This is because the xsi:schemaLocation points to the local application schema which in this case is s121.xsd. This needs to be in the same folder as the dataset.
- 85. Power of Generic GML R/W: GeoSciML Open standard for geologic data. Submitted as an OGC standard. Aligned with INSPIRE
- 86. Weather Information Exchange Model Interoperable meteorological information exchange covering the needs of the aviation industry eurocontrol.int faa.gov Power of Generic GML: WXXM
- 87. S-121 GML from OGC Maritime Boundaries Pilot Collaboration between IHO and OGC S-121 generated from DFO/CHS geodatabase using FME ogc.org charts.gc.ca Power of Generic GML: S-121
- 89. XML/GML to JSON conversion / differences GML schema driven directly from XSD. Complex structures / nesting modelled within the feature type definitions and using parent / child ids. Currently JSON writer requires the use of JSONTemplater for complex properties / nesting, such as for lists or multi-part elements such as date ranges.
- 90. JSON / GeoJSON Enhancements New formats: CityJSON, TopoJSON, IMDF update Writing: New Feature Collection mode in FME2020.1+. Choose how to assemble feature types: ● Single feature collection ● Array of feature collection objects ● Object with feature type names as keys Performance improvements To come: nested schema writing. In meantime use JSONTemplater to write to root objects
- 91. Preview GeoJSON on GitHub, geojson.io Geojson is automatically rendered as an interactive map on GitHub Use geojson.io to test your geojson with an external client
- 92. Let us know in the chat… [question]
- 93. Q&A
- 94. Other Topics to Explore ● Metadata -> AI/ML - metadata tags for training ● XML manipulation: XML Extraction & Updating: XMLUpdater, XQueryExtractor ● XFMaps and XSLT - XML reader scripting, dynamic schema, style sheets ● OGC Support ○ Web Services: OGC Service hosting ● Geometry extraction/ composition ● JSON parsing ● Other formats: Indoor, BIM (IFCXML), AEC: 12DXML, CIM XML What do you want to hear / learn more about? Go to Knowledge Base for more info on most of these topics
- 95. Summary - Key Takeaways ● XSD XML - Powerful new reader / writer: e.g. metadata 19115 ● GML - XSD mode - very flexible and continues to evolve ● FME as your no-code XML solution: many XML tools embedded in FME (Validation, Formatting, SampleGenerator, XSLT, XML Update) ● FME constantly evolving: OpenAPI/WFS 3 etc - keep upgrading to get latest functionality and performance ● JSON patterns very similar to XML ones ● FME’s XML and JSON support critical for web integration and open standards support: W3C, OGC, EU INSPIRE, ISO, ITRF Reach out to us for Help
- 96. Q&A And suggestion sharing! Please share your XML ideas in the chat panel.
- 97. XML Resources Tutorials & Articles XML Tutorial https://knowledge.safe.com/articles/29553/tutorial-getting-started-wit h-xml.html GML Tutorial - INSPIRE https://knowledge.safe.com/articles/1321/eu-inspire-initiative-tutorial. html JSON Tutorial https://knowledge.safe.com/articles/39188/tutorial-getting-started-wit h-json.html Reading XML/GML https://knowledge.safe.com/articles/888/reading-xmlgml.html GML - Writing https://knowledge.safe.com/articles/812/gml-writing-with-application- schemas.html Keep the XML conversation going! We want to know what else you want to see in FME to make XML easier. Please reach out on safe.com/live-chat or post on safe.com/community
- 98. Demo Links And supporting articles CityGML https://knowledge.safe.com/articles/904/writing-citygml-example.html XMLUpdater https://knowledge.safe.com/articles/868/using-the-xmlupdater-transfo rmer-to-manipulate-met.html OGC Maritime Boundaries S-121 GML Pilot https://knowledge.safe.com/articles/112578/ogc-maritime-limits-and-b oundaries-s-121-gml-pilot.html Writing to nested properties in AIXM: https://knowledge.safe.com/questions/2949/aixm-annotations.html https://knowledge.safe.com/questions/47251/how-to-transform-aixm- 45-to-aixm-51-xml-geometry-g.html WFS Troubleshooting: https://knowledge.safe.com/articles/19452/troubleshooting-wfs-errors. html Keep the XML conversation going! We want to know what else you want to see in FME to make XML easier. Please reach out on safe.com/live-chat or post on safe.com/community
- 99. Title Join us tomorrow! Keeping Cool, Calm, and Connected with FME 2020.1
- 100. Thank you! Get a free trial of FME or live chat at safe.com