Presentation pdi data_vault_framework_meetup2012
0 likes468 views
The document describes the Data Vault modeling technique which involves storing historical data from different sources in a series of normalized tables. It outlines the key components of a Data Vault including hubs, links, and satellites. It then discusses how to implement a Data Vault using the Pentaho Data Integration (Kettle) tool including generating and loading metadata, and executing jobs and transformations in parallel via a framework that standardizes the ETL process.
Presentation pdi data_vault_framework_meetup2012
- 2. Data Vault Definition The Data Vault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3rd normal form (3NF) and star schema. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise. It is a data model that is architected specifically to meet the needs of enterprise data warehouses. Source: Dan Linstedt http://www.tdan.com/view-articles/5054/
- 3. Data Vault Building Blocks different sources/rate of change Source: Dan Linstedt http://www.slideshare.net/dlinstedt/introduction-to-data-vault-dama-oregon-2012
- 4. Data Vault Fundamentals: Hub Source: data-vault-modeling-guide GENESEE ACADEMY, LLC, Hans Hultgren
- 5. Data Vault Fundamentals: Link Source: data-vault-modeling-guide GENESEE ACADEMY, LLC, Hans Hultgren
- 6. Data Vault Fundamentals: Satellite Source: data-vault-modeling-guide GENESEE ACADEMY, LLC, Hans Hultgren
- 7. Data Vault Fundamentals: Model Source: data-vault-modeling-guide GENESEE ACADEMY, LLC, Hans Hultgren
- 8. Data Vault ETL Many objects to load, standardized procedures This screams for a generic solution! I don't want to: throw ETL tool away and code it all myself manage too many ETL objects connect similar columns in mappings by hand I do want to: generate ETL (Kettle) objects? No Take it one step further: there's only 1 parameterised hub load object. Don't need to know xml structure of PDI objects
- 9. Tools Operating System Database Virtualization Data Integration 'Productivity' Sql Development Version Control
- 10. Place of framework in architecture Files MySQL ETL: Kettle MySQL DBMS ETL Data Data ETL Vault Vault CSV Frame Files work ERP Staging Central DWH & Area Data Marts Sources ETL Process Data Warehouse EUL
- 11. What has to be taken care of? Data Vault designed and implemented in database Staging tables and loading procedures in place (can also be generic, we use PDI Metadata Injection step for loading files) Mapping from source to Data Vault specified (now in an Excel sheet)
- 12. Framework components PDI repository (file based), jobs and transformations Configuration files: kettle.properties shared.xml repositories.xml Excel sheet that contains the specifications MySQL database for metadata Virtual machine with Ubuntu 12.04 Server
- 13. Design decisions Updateable views with generic column names (MySQL more lenient than PostgreSQL) Compare satellite attributes via string comparison (concatenate all columns, with | (pipe) as delimiter) 'inject' the metadata using Kettle parameters Generate and use an error table for each Data Vault table
- 14. Metadata tables All have history tables
- 15. Metadata in Excel Data Vault connections source systems source tables
- 16. Metadata in Excel (hub + sat) x 200 (max)
- 17. Metadata in Excel (link) x 10 link attributes
- 18. Metadata in Excel (link satellite) x 10 x5 x 200 (max)
- 19. Last seen date applicable for hubs and links existing hubs and links: update 'last_seen_dts'!
- 20. Link validity satellite Link has 'business key': not all hub id's
- 22. 'design errors' Checks to avoid debugging: (compares design metadata with Data Vault DB information_schema) hubs, links, satellites that don't exist in the DV key columns that do not exist in the DV missing connection data (source db) missing attribute columns
- 23. A complete run
- 24. Metadata needed for a hub name key column business key column source table source table business key column (can be expression, e.g. concatenate for composite key)
- 25. Job for hub
- 27. Metadata needed for a link name key column for each hub (maximum 10, can be a ref-table) hub name column name for the hub key in the link (roles!) column in the source table → business key of hub link 'attributes' (part of key, no hub, maximum 5) link validity satellite needed? last seen date needed? source table
- 28. Job for link
- 29. Transformation for link Last seen? Lookup hubs Remove columns not in link Run table needed for validity sat ?
- 30. Metadata needed for a hub satellite name key column hub name column in the source table → business key of hub for each attribute (maximum 200) source column target column source table
- 31. Job for hub satellite
- 32. Transformation for hub satellite
- 33. Metadata needed for a link satellite name key column link name for each hub of the link: column in the source table → business key of hub for each key attribute: source column for each attribute: source column → target column source table
- 34. Job for link satellite
- 35. Transformation for link satellite
- 36. Executing in a loop ..
- 37. .. and parallel
- 38. Logging Custom logging PDI logging Configuring log tables for concurrent access
- 39. Version Control: PDI objects
- 40. Version Control: database objects
- 41. Some points of interest Easy to make mistake in design sheet Generic → a bit harder to maintain and debug Application/tool to maintain metadata? Data Vault generators (e.g. Quipu)? Spinoff using Informatica and Oracle: Sander Robijns Thanks to: Jos van Dongen Kasper de Graaf
- 42. Sourceforge!