Geospatial ETL with Stetl - GeoPython 2016
3 likes2,785 views
The document discusses the STETL framework for spatial ETL (Extract, Transform, Load) processes, highlighting its capabilities and implementation in geospatial data processing. It covers concepts, configurations, components, and case studies, emphasizing the integration of Python for improved efficiency. The current status of the project and future plans, including installation options and documentation, are also outlined.
![Running Stetl
stetl -c etl.cfg
stetl -c etl.cfg [-a <properties>]](https://image.slidesharecdn.com/stetl-geopython-2016-160624130756/85/Geospatial-ETL-with-Stetl-GeoPython-2016-31-320.jpg)
![[etl]
chains = input_xml_file|my_filter|output_std
[input_xml_file]
class = inputs.fileinput.XmlFileInput
file_path = input/cities.xml
# My custom component
[my_filter]
class = my.myfilter.MyFilter
[output_std]
class = outputs.standardoutput.StandardXmlOutput
class MyFilter(Filter):
# Constructor
def __init__(self, configdict, section):
Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc,
produces=FORMAT.etree_doc)
def invoke(self, packet):
log.info("CALLING MyFilter OK!!!!")
return packet
Your Own Components
Stetl concepts
Step 1- Define Class
Step 2- Configure Class](https://image.slidesharecdn.com/stetl-geopython-2016-160624130756/85/Geospatial-ETL-with-Stetl-GeoPython-2016-38-320.jpg)
Geospatial ETL with Stetl - GeoPython 2016
- 1. Geospatial Data Processing with Stetl Just van den Broecke GeoPython 2016 Muttenz - Switserland June 24, 2016 www.justobjects.nl www.stetl.org
- 2. About Me Independent Open Source Geospatial Professional + Secretary OSGeo Dutch Local Chapter + Member of the Dutch OpenGeoGroep Just van den Broecke just@justobjects.nl www.justobjects.nl
- 3. Agenda • Spatial ETL • Stetl • Concepts • Cases • Status • Q & A
- 4. Spatial ETL
- 6. ETL - Extract Transform Load
- 8. GML PostGIS Shapefile WFS CSV SQLite XML Spatial ETL Example non-standard source
- 11. Plenty of Tools… Each tool is powerful by itself but cannot do the entire ETL ogr2ogr Spatial ETL
- 12. FOSS ETL - How to Combine Components? = + + ?+ ..
- 13. Example - 2011 INSPIRE-FOSS http://inspire.kademo.nl/doc/design-etl.html Nice ideas but hard to scale, deploy and reuse. Need Framework
- 14. Solution: Add Python to the Equation =+ + ?( )+ ..
- 15. Stetl Solution: Add Python to the Equation =+ +( )+ ..
- 17. Stetl Concepts
- 18. Process Chain Input Filter OutputFilter Stetl concepts Source Target
- 19. Process Chain Input Filter Output gml Filter Stetl concepts
- 20. Example: GML to PostGIS GML Reader PG Output gml Stetl concepts
- 21. Example: Data Model Transform OGR Reader XSLT GML Writer gml Stetl concepts Simple Features Complex Features or Jinja2 !
- 22. Process Chain - How? Input Filters Output Stetl concepts Stetl Config File Instantiate
- 23. Example: XML to Shape XML Input XSLT Filter OGR Output
- 24. Example: XML to Shape The Source File
- 25. Example: XML to Shape XML Input
- 26. Example: XML to Shape XML Input XSLT Filter
- 27. Example: XML to Shape Prepare XSLT Script
- 28. Example: XML to Shape XSLT GML Output
- 29. Example: XML to Shape XML Input XSLT Filter OGR Output OGC Simple Features XML DOM
- 30. Example: XML to Shape The Stetl Config File Process Chain XML InputXSLT Filter ogr2ogr Output
- 31. Running Stetl stetl -c etl.cfg stetl -c etl.cfg [-a <properties>]
- 32. Installing Stetl - PyPi Deps •GDAL+Python bindings •lxml (xml proc) •psycopg2 (Postgres) sudo pip install stetl
- 33. Installing Stetl - new: Docker https://hub.docker.com/r/ justb4/stetl
- 34. Speed: Streaming Input Filter Output gml Stetl concepts
- 35. Speed: Going Native Input Filter Output gml ogr2ogr StetlStetl Native C Libs/Progs Calls Stetl concepts
- 36. Example Components Input Filters Output Stetl concepts XMLFile XSLT GML ogr2ogr XMLAssembler GDAL/OGR LineStream XMLValidator WFS-T Postgres/PostGIS Jinja2 Postgres/PostGIS deegree* FeatureExtractor deegree* YourInput YourFilter YourOutput
- 37. Example: XsltFilter Pythonfrom util import Util, etree from filter import Filter from packet import FORMAT log = Util.get_log("xsltfilter") class XsltFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) self.xslt_file_path = self.cfg.get('script') self.xslt_file = open(self.xslt_file_path, 'r') # Parse XSLT file only once self.xslt_doc = etree.parse(self.xslt_file) self.xslt_obj = etree.XSLT(self.xslt_doc) self.xslt_file.close() def invoke(self, packet): if packet.data is None: return packet return self.transform(packet) def transform(self, packet): packet.data = self.xslt_obj(packet.data) log.info("XSLT Transform OK") return packet
- 38. [etl] chains = input_xml_file|my_filter|output_std [input_xml_file] class = inputs.fileinput.XmlFileInput file_path = input/cities.xml # My custom component [my_filter] class = my.myfilter.MyFilter [output_std] class = outputs.standardoutput.StandardXmlOutput class MyFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) def invoke(self, packet): log.info("CALLING MyFilter OK!!!!") return packet Your Own Components Stetl concepts Step 1- Define Class Step 2- Configure Class
- 39. Data Structures Stetl concepts • Components exchange Packets • Packet contains data and status • Data formats, e.g. : xml_line_stream etree_doc etree_element (feature) etree_element_array string any . .
- 40. Cases
- 42. Cases - GML to PostGIS • National Dutch Open Datasets (GML) http://nlextract.nl ✴ Topography:Top10NL, BGT ✴ Cadastral Parcels (BRK) • Ordnance Survey UK ✴ PoC Topography (OS Mastermap)
- 43. Cases - IoT/SensorWeb • SOSPilot ✴ Dutch Air Quality Data ✴ publish to PostGIS+SOS (SOS-T) ✴ EU Reporting (Jinja2 Filter) ✴ sospilot.geonovum.nl • Smart Emission ✴ AQ Sensors hosted by citizens ✴ Calibration/Aggregation ✴ publication to SOS and OGC SensorThings ✴ data.smartemission.nl
- 44. Project Status - June 24, 2016 • v1.0.9 installable via PyPi or Docker • Documentation on www.stetl.org • Real world transforms done
- 45. Project - Planned & in progress • PyWPS integration • GUI - Flask+Celery+Redis? - Node Red? - Jupiter Notebook? • More on GitHub https://github.com/geopython/stetl