Stetl: Preparing Rich GML Data for deegree - The ETL Challenge
0 likes798 views
The document discusses the challenges of preparing rich GML data for the deegree platform, emphasizing the complexities of managing large datasets and multiple transformation steps. It proposes using a Spatial ETL (Extract, Transform, Load) process involving various tools such as ogr2ogr and Python to streamline data conversion. Additionally, it highlights case examples such as Dutch addresses and the integration of national GML datasets into the deegree environment.
Stetl: Preparing Rich GML Data for deegree - The ETL Challenge
- 1. PreparingYour Rich GML Data for deegree - the ETL Challenge Just van den Broecke deegree Community Space 2012, Bonn November 13, 2012 www.justobjects.nl
- 2. About Me Independent Open Source Geospatial Professional Trailblazer OSGeo Dutch Local Chapter Just van den Broecke just@justobjects.nl www.justobjects.nl
- 5. THE DUTCH KADASTER GETS INSPIRED WITH deegree day - nov 16, 2010
- 6. THE DUTCH KADASTER GETS INSPIRED WITH deegree Community Space - nov 13, 2012
- 9. METADATA + NL INSPIRE ACCESS POINT
- 10. PDOK – Open Source & Open Standards Applications OGC/ISO/INSPIRE Web Services Storage Conversion Services sETLFME?
- 11. OSGeo - Bolsena - 2010
- 12. BOLSENA 2012
- 14. BOLSENA 2012
- 15. PreparingYour Rich GML Data for deegree - the ETL Challenge Just van den Broecke deegree Community Space 2012, Bonn November 13, 2012 www.justobjects.nl
- 18. Rich GML = Complex Mess
- 21. The Streetname!
- 27. A.K.A.
- 29. But what about... ....FOSS ? ... Stetl?
- 30. FOSS ETL - High Level
- 31. FOSS ETL - Lower Level But Each Powerful by Itself ogr2ogr
- 32. FOSS ETL - DIY ? (No!)
- 33. FOSS ETL - How to Combine? =+ + ? ogr2ogr
- 34. FOSS ETL - Add Python to Equation =+ + ?( ) ogr2ogr
- 37. Process Chain Input Filter Output gml Filter Stetl concepts
- 38. Speed: Streaming Input Filter Output gml Stetl concepts
- 39. Speed: Going Native Input Filter Output gml ogr2ogr sETLsETL Native C Libs/Progs Calls Stetl concepts
- 40. Example: GML to PostGIS Reader XML Splitter ogr2ogr gml Stetl concepts
- 41. Example: INSPIRE Model Transform ogr2ogr XSLT Writer gml Stetl concepts
- 42. Example: deegree Store ogr2ogr XSLT deegree Writer Stetl concepts
- 43. Process Chain - How? Input Filters Output Stetl concepts
- 44. Example: XML to Shape The Source
- 45. Example: XML to Shape First: XSLT Transform to GML
- 46. Example: XML to Shape XML Input XSLT ogr2ogr
- 47. Example: XML to Shape The SETL Chain Config File Process Chain Reader XSLT ogr2ogr
- 48. Example Components Input Filters Output Stetl concepts XMLFile XSLT GMLFile ogr2gml GMLSplitter gml2ogr LineStream XMLValidator WFS-T deegree* FeatureExtractor deegree* YourInput YourFilter YourOutput
- 49. Data Structures Stetl concepts ✴Components exchange Packet ✴Packet contains data ✴Data format: xml_line_stream etree_doc etree_feature_array xml_doc_as_string any
- 51. Cases ✴INSPIRE Download Services publish to deegree store (WFS) GML files (for Atom Feed) ✴National GML Datasets GML to PostGIS
- 52. Case: Dutch Addresses Source <GML> sETL sETL deegree WFS sETL INSPIRE <GML> Atom Feed Other INSPIRE Addresses Dutch Addresses+ Buildings Dutch Geocoder deegree blobstore
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n