Stetl for INSPIRE Data Transformation
0 likes1,656 views
The document introduces Stetl, a lightweight Python framework designed for geospatial ETL (Extract, Transform, Load) processes, particularly addressing the complexities of rich GML data. It discusses the framework's capabilities, including spatial data streaming, integration with various geospatial tools, and support for custom components. The author also highlights use cases, installation details, and ongoing development efforts while seeking community feedback and contributions.
![[etl]
chains = input_xml_file|my_filter|output_std
[input_xml_file]
class = inputs.fileinput.XmlFileInput
file_path = input/cities.xml
# My custom component
[my_filter]
class = my.myfilter.MyFilter
[output_std]
class = outputs.standardoutput.StandardXmlOutput
class MyFilter(Filter):
# Constructor
def __init__(self, configdict, section):
Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc,
produces=FORMAT.etree_doc)
def invoke(self, packet):
log.info("CALLING MyFilter OK!!!!")
return packet
Your Own Components
Stetl concepts
Step 1- Define Class
Step 2- Config Class](https://image.slidesharecdn.com/2-stetl-inspire-transform-v1-131011040246-phpapp01/85/Stetl-for-INSPIRE-Data-Transformation-50-320.jpg)
![[etl]
chains = input_sql_pre|schema_name_filter|output_postgres,
input_big_gml_files|xml_assembler|transformer_xslt|output_ogr2ogr,
input_sql_post|schema_name_filter|output_postgres
# Pre SQL file inputs to be executed
[input_sql_pre]
class = inputs.fileinput.StringFileInput
file_path = sql/drop-tables.sql,sql/create-schema.sql
# Post SQL file inputs to be executed
[input_sql_post]
class = inputs.fileinput.StringFileInput
file_path = sql/delete-duplicates.sql
# Generic filter to substitute Python-format string values like {schema} in string
[schema_name_filter]
class = filters.stringfilter.StringSubstitutionFilter
# format args {schema} is schema name
format_args = schema:{schema}
[output_postgres]
class = outputs.dboutput.PostgresDbOutput
database = {database}
host = {host}
port = {port}
user = {user}
password = {password}
schema = {schema}
# The source input file(s) from dir and produce gml:featureMember elements
[input_big_gml_files]
class = inputs.fileinput.XmlElementStreamerFileInput
file_path = {gml_files}
element_tags = featureMember
Top10NL Extract
Parameter
Substitution](https://image.slidesharecdn.com/2-stetl-inspire-transform-v1-131011040246-phpapp01/85/Stetl-for-INSPIRE-Data-Transformation-54-320.jpg)
Stetl for INSPIRE Data Transformation
- 1. INSPIRE Transformation with Stetl - A lightweight Python Framework for Geospatial ETL Just van den Broecke EuroGeographics - KEN Workshop Paris, Oct 8, 2013 www.justobjects.nl
- 2. About Me Independent Open Source Geospatial Professional Secretary OSGeo Dutch Local Chapter Member of the Dutch OpenGeoGroep Just van den Broecke just@justobjects.nl www.justobjects.nl
- 5. Rich GML = Complex Mess
- 6. INSPIRE Dutch National Datasets Germany:AFIS-ALKIS-ATKIS UK: OS Mastermap . .
- 7. “Semi GML” e.g. Dutch Addresses & Buildings (BAG) Arbitrary Nesting
- 8. The Street Name! A Street Element in an INSPIRE Annex I Address..
- 16. But How ? (with FOSS)
- 17. FOSS ETL - DIY ? Maybe
- 18. FOSS ETL - High Level
- 19. FOSS ETL - Lower Level Each powerful individually but cannot do the entire ETL ogr2ogr
- 20. FOSS ETL - How to Combine? =+ + ? ogr2ogr
- 21. Example - 2011 Kadaster ESDIN http://inspire.kademo.nl/doc/design-etl.html Good ideas but hard to scale and reuse. Need Framework
- 22. FOSS ETL :Add Python to Equation =+ + ?( ) ogr2ogr
- 25. GML1 GML2 Stetl From Barrels of GML to Maps
- 27. From Local National Data to INSPIRE DL Services Source <GML> NLExtract Stetl deegree WFS INSPIRE <GML> Atom Feed INSPIRE Addresses Dutch Addresses+ Buildings deegree blobstore Stetl
- 28. Stetl Concepts
- 29. Process Chain Input Filter OutputFilter Stetl concepts Source Target
- 30. Process Chain Input Filter Output gml Filter Stetl concepts
- 31. Example: GML to PostGIS Reader ogr2ogr gml Stetl concepts
- 32. Example: INSPIRE Model Transform ogr2ogr XSLT Writer gml Stetl concepts Simple Features Complex Features
- 33. Example: deegree Store ogr2ogr XSLT deegree Writer Stetl concepts Or via WFS-T
- 34. Process Chain - How? Input Filters Output Stetl concepts
- 35. Example: XML to Shape XML Input XSLT Filter ogr2ogr Output
- 36. Example: XML to Shape The Source
- 37. Example: XML to Shape XML Input
- 38. Example: XML to Shape XML Input XSLT Filter
- 39. Example: XML to Shape Prepare XSLT Script
- 40. Example: XML to Shape XSLT GML Output
- 41. Example: XML to Shape XML Input XSLT Filter ogr2ogr Output
- 42. Example: XML to Shape The Stetl Config File Process Chain XML InputXSLT Filter ogr2ogr Output
- 43. Running Stetl stetl -c etl.cfg
- 44. Result Shapefile viewed in QGIS
- 45. Installing Stetl via PyPi Deps •GDAL+Python bindings •lxml (xml proc) •psycopg2 (Postgres) sudo pip install stetl
- 46. Speed: Streaming Input Filter Output gml Stetl concepts
- 47. Speed: Going Native Input Filter Output gml ogr2ogr StetlStetl Native C Libs/Progs Calls Stetl concepts
- 48. Example Components Input Filters Output Stetl concepts XMLFile XSLT GMLFile ogr2ogr XMLAssembler ogr2ogr LineStream XMLValidator WFS-T deegree* FeatureExtractor deegree* YourInput YourFilter YourOutput
- 49. Example: XsltFilter Python from util import Util, etree from filter import Filter from packet import FORMAT log = Util.get_log("xsltfilter") class XsltFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) self.xslt_file_path = self.cfg.get('script') self.xslt_file = open(self.xslt_file_path, 'r') # Parse XSLT file only once self.xslt_doc = etree.parse(self.xslt_file) self.xslt_obj = etree.XSLT(self.xslt_doc) self.xslt_file.close() def invoke(self, packet): if packet.data is None: return packet return self.transform(packet) def transform(self, packet): packet.data = self.xslt_obj(packet.data) log.info("XSLT Transform OK") return packet
- 50. [etl] chains = input_xml_file|my_filter|output_std [input_xml_file] class = inputs.fileinput.XmlFileInput file_path = input/cities.xml # My custom component [my_filter] class = my.myfilter.MyFilter [output_std] class = outputs.standardoutput.StandardXmlOutput class MyFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) def invoke(self, packet): log.info("CALLING MyFilter OK!!!!") return packet Your Own Components Stetl concepts Step 1- Define Class Step 2- Config Class
- 51. Data Structures Stetl concepts • Components exchange Packets • Packet contains data and status • Data formats, e.g. : xml_line_stream etree_doc etree_element (feature) etree_element_array string any . .
- 53. Cases - The Netherlands •INSPIRE Download Services publish to deegree store (WFS) generate GML files (for Atom Feed) •National GML Datasets GML to PostGIS (Top10NL, BGT)
- 54. [etl] chains = input_sql_pre|schema_name_filter|output_postgres, input_big_gml_files|xml_assembler|transformer_xslt|output_ogr2ogr, input_sql_post|schema_name_filter|output_postgres # Pre SQL file inputs to be executed [input_sql_pre] class = inputs.fileinput.StringFileInput file_path = sql/drop-tables.sql,sql/create-schema.sql # Post SQL file inputs to be executed [input_sql_post] class = inputs.fileinput.StringFileInput file_path = sql/delete-duplicates.sql # Generic filter to substitute Python-format string values like {schema} in string [schema_name_filter] class = filters.stringfilter.StringSubstitutionFilter # format args {schema} is schema name format_args = schema:{schema} [output_postgres] class = outputs.dboutput.PostgresDbOutput database = {database} host = {host} port = {port} user = {user} password = {password} schema = {schema} # The source input file(s) from dir and produce gml:featureMember elements [input_big_gml_files] class = inputs.fileinput.XmlElementStreamerFileInput file_path = {gml_files} element_tags = featureMember Top10NL Extract Parameter Substitution
- 55. Top10NL+BAG (Dutch Topo + Buildings)
- 56. BGT - Dutch Large Scale Topo
- 57. Cases - INSPIRE Transforms •Simple: Dutch Admin Borders to AU •Advanced: Dutch Addresses to AD
- 58. INSPIRE - XSLT STRUCTURE Local CP GML to INSPIRE SpatialDataset Local CP GML to INSPIRE GML Generate CP INSPIRE GML Reusable XSLT ScriptsReusable XSLT Scripts Theme CP Local AU GML to INSPIRE SpatialDataset Local AU GML to INSPIRE GML Generate AU INSPIRE GML Theme AU Local GN GML to INSPIRE SpatialDataset Local GN GML to INSPIRE GML Generate GN INSPIRE GML Theme GN Called by All Locally Specific XSL Generic XSL XSLT Template Call
- 59. XSLT - 3 MAIN STEPS/SCRIPTS 1.Generate Spatial Dataset GML Container (specific) 2.Extract data values from local OGR simple feature data (specific) 3. Call XSLT template per Theme Feature type (generic)
- 60. XSLT AU - STEP 1
- 61. XSLT AU - STEP 2
- 62. XSLT AU - STEP 3
- 63. XSLT - REUSE
- 64. STETL CONFIG
- 65. STETL CONFIG AD
- 66. Case: INSPIRE DL Services - Dutch Addresses Source <GML> NLExtract Stetl deegree WFS INSPIRE <GML> Atom Feed INSPIRE Addresses Dutch Addresses+ Buildings deegree blobstore Stetl Other Uses (Geocoder etc)
- 67. Project Status - Sept 21, 2013 • v1.0.4 installable via PyPi • Documentation on www.stetl.org • Real world transforms done • Seeking feedback, support and contributors