M3D - Metadata Driven Development
0 likes58 views
The document describes metadata driven development (M3D) at Adidas, outlining its framework, components, and advantages such as cloud and platform agnosticism and easy feature development. M3D facilitates the creation of data lake environments and manages data flows through various systems, improving the definition and scheduling of data loads. Future developments include enhanced orchestration with Airflow, the publication of the M3D API as a library, and a PySpark version of the M3D engine.
M3D - Metadata Driven Development
- 2. Content • What is M3D? • M3D Components • Why we need it? • M3D @ Adidas • What’s to come?
- 3. What is it? Framework Cloud and platform agnostic Easy development of new feats Multiple source/target system Creation of data lake envs
- 4. What is it?
- 5. M3D Components M3D Engine M3D API https://github.com/adidas/m3d-engine https://github.com/adidas/m3d-api
- 6. Features in action! Creation of data lake envs Inbound Raw File Raw File Raw File Raw File Raw File Raw File
- 7. Features in action! Creation of data lake envs Inbound Landing Raw File Raw File Raw File Raw File Raw File Raw File
- 8. Features in action! Creation of data lake envs Inbound Landing Lake Parqu et Parqu et Parqu et Parqu et Parqu et Parqu et
- 9. Features in action! Creation of data lake envs Inbound Landing Lake Out View View View View Lake
- 10. Features in action! Inbound Landing Lake Out Lake Multiple source/target system M3D Target Exasol Target Y Target Target Z AWS Cloud and platform agnostic Cloudera
- 11. Features in action! Inbound Landing Lake Parqu et Parqu et Parqu et Parqu et Parqu et Parqu et Easy development of new feats Easily define new readers/writers based on traits Introduce transformations on datasets Add new APIs to M3D Api
- 12. Why We Need It? Simplification of definition of new tables in lake Easy way to schedule loads Clear semantics defined globally Single extraction from source systems Simple definition of views as interface for applications
- 13. M3D @ Adidas We have the so called “Cockpit” for definition of global semantics Orchestration with Jenkins as a Service Loading data to lake (example): Place files in Inbound layer python m3d_main.py -function create_emr_cluster - core_instance_type m4.large - master_instance_type m4.large - core_instance_count 3 - destination_system emr - destination_database emr_database -destination_environment test - config config.json -emr_version emr- 5.23.0 python m3d_main.py -function create_table -config config.json - destination_system emr - destination_database emr_database - destination_environment test - destination_table table_name - emr_cluster_id id-of-started-cluster python m3d_main.py -function load_table -config config.json - destination_system emr -destination_database emr_database - destination_environment test -destination_table table_name - load_type FullLoad -emr_cluster_id id-of-started-cluster
- 14. M3D @ Adidas 9 TB In Memory Lake Size 150 TB Lake Size on Big Data Platform 500+ tables in Data Lake > 900 views consumed by Products
- 16. M3D @ Adidas
- 17. M3D @ Adidas
- 18. What’s to come? Airflow as orchestration for M3D Publish M3D Api as library UI for quick setup of M3D Api/Engine PySpark version for M3D engine