INGKA DIGITAL: Linked Metadata by Design
0 likes65 views
Shirley Bacso, a data architect at Ingka Digital, discusses the integration of linked metadata by design to enhance knowledge graphs, focusing on the importance of human collaboration in metadata creation. The document outlines the iterative process of ontology development and emphasizes addressing key questions regarding data compliance, quality, and accessibility. It also introduces future potentials such as improved search capabilities and the use of large language models in knowledge graph applications.
INGKA DIGITAL: Linked Metadata by Design
- 1. Linked Metadata by Design Presented by: Shirley Bacso Data Architect at Ingka Digital, Netherlands
- 2. A few words about me • Data Architect at Ingka Digital, Netherlands • Specialize in metadata • Love problem-solving & learning • Graph enthusiast since June 2021
- 3. My journey June 2021 Oct 2021 May 2022 Implementing Business Glossary Ingka Graph Event with Neo4j Feb 2023 Knowledge Graphs 2023, Prof. Dr. Harald Sack Oct 2023 DKG MVP NeoDash Feb 2024 Learn graph technology Exam case studies Research metadata management best practices Study organization realities Observe people Hunt for metadata Hunt for metadata Hunt for metadata Observe people Study organization realities Hunt for metadata Observe people Metadata – the knowledge of our data Dec 2023
- 4. Rationale
- 5. What types of questions should be answered by the knowledge graph? Do all the data columns/fields have business names? Are the data compliant to (retention) rules? What are the gaps between the data quality dimension goals and the reality? Do all the databases connected to the software systems? What systems does a certain digital domain have? What type of personal data does a software system have? Find the most relevant dataset (recommendation engine) and request access. Which data products are overlapping? (Think about the Principle of Data as a Product) How was the data transformed throughout the lifetime? (Provenance and operational lineage) …
- 6. Which knowledge areas should be covered ? Collectively capture the broader knowledge about data
- 7. arrows.app What concepts and relationships should be in the knowledge graph?
- 8. The Ontology Development Process Knowledge Graphs 2023, Prof. Dr. Harald Sack, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure & Karlsruhe Institute of Technology Ontology development in practice is an iterative process that repeats continuously and improves the ontology. What types of questions should be answered by the knowledge graph? Which knowledge areas should be covered by the knowledge graph? What concepts and relationships should be in the knowledge graph?
- 9. Pain points of “Metadata as a by-product” and relieves DAMA DMBOK has pointed out: “Metadata is often created as a by-product of application processing rather than as an end product (i.e.. it is not created with consumption in mind). As with other forms of data, there is a lot of work in preparing Metadata before it can be integrated.”
- 10. What is linked metadata by design? • Linked metadata represents the integration of outcomes from human collaboration. It collectively forms the knowledge about data, which is captured in the data knowledge graph. • By design emphasizes that the capture and integration of metadata should start from the conceptual level and continue through the logical level, as human collaboration begins at the design phase of data product development.
- 11. Solution
- 12. Architecture Various metadata sources L0 L1 L2 L3 “Sources of knowledge” “Data knowledge graph”
- 13. What has been done
- 14. Column Table Schema System group Database Digital domain Digital unit System Architectural area The current version of the DKG has close to 1 million of nodes and 1 million of edges and it will continue to grow. This is a snapshot from Neo4j Bloom. Only 10K nodes (1%) are displayed here. Ingka Data Knowledge Graph (DKG)
- 15. NeoDash (UI) Query Visualize Post changes Ingka Data Knowledge Graph (DB) https://github.com/neo4j-labs/neodash NeoDash - Neo4j Dashboard Builder NeoDash is an open source tool for visualizing your Neo4j data. It lets you group visualizations together as dashboards, and allow for interactions between reports. Explicit, formal and shareable knowledge about data/ data products stored in a graph database
- 16. Catalogued Systems Quick Insights Intuitive Visuals Coherent Understanding Ø Systems are categorized • by relations to other asset types: • Architectural Area • System Group • Digital Unit | Digital Domain • by categorical attributes Ø Explore details of the system in interest
- 17. Catalogued RDBs Ø Datasources are categorized by relations to other asset types: System à Digital Unità Digital Domain Ø Explore details of the datasource in interest and its hierarchical assets Column à Table à Schema à Datasource Quick Insights Intuitive Visuals Coherent Understanding
- 18. Consumer - Universal Search Universal and exploratory search for various asset types, e.g. System, Datasource…
- 19. Find | Inspect | Connect | Verify Without the need to build and maintain UIs Link Data Concept to: • Business Term • Data Entities • … Link Data Attribute/Entity to: • Business Term • Code List • Rules • Policies • … Link Data Attribute to: • Columns, Fields... Provider – Link Metadata
- 20. Current development and future
- 21. Column Table Schema System group Database Digital domain Digital unit System Architectural area
- 22. Near future potentials • Knowledge graph completion: Link prediction with Word Embeddings and KG Embeddings
- 23. Near future potentials • Knowledge graph completion: Link prediction with Word Embeddings and KG Embeddings Knowledge Graphs 2023, Prof. Dr. Harald Sack, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure & Karlsruhe Institute of Technology
- 24. Near future potentials • Better search: Use combined capabilities of semantic and structural search • Refinement enables more precise or more complete search results. Query String • Enables complementing search results with additional associated or similar information. Cross Referencing • Enables the determination of nearby results and results related by content. Fuzzy Search • Enables visualization and navigation of the search space. Exploratory Search via Linked Data • Enables complementing search results with implicitly given information. Reasoning Knowledge Graphs 2023, Prof. Dr. Harald Sack, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure & Karlsruhe Institute of Technology
- 25. Near future potentials • Leverage LLM + KG RAG Tomaz Bratanic Learn | Share | Collaborate Ask DKG anything about your data
- 26. Near future potentials • Build fast apps in OpenSource and InnerSource
- 27. Knowledge Building Community o Terminology, ontology and modeling o Data governance o Data engineering o Graph data science o UX design and UI building o … • Iteration of curating and extracting knowledge • Following the architecture of building and consuming the knowledge graph • By the many people, for the many people
- 29. Thank you! Reach out on Shirley Bacso