Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
0 likes24 views
The document discusses the concept of a data mesh, which is an organizational paradigm for data management that decentralizes data ownership to autonomous domains rather than relying on a centralized data team. It highlights common myths surrounding data mesh, particularly the misconception that data virtualization is unnecessary when adopting a data mesh strategy. The presentation emphasizes the benefits of integrating data virtualization with a data mesh approach to enhance data accessibility, governance, and usability across different domains.
Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
- 1. I’m building a data mesh, so I don't need data virtualization WEBINAR SERIES
- 2. SPEAKERS Paul Moxon SVP Data Architectures & Chief Evangelist, Denodo
- 3. 3 MYTH #7: I’m building a data mesh, so I don't need data virtualization
- 4. AGENDA 1. What is a Data Mesh? 2. Origins of the Myth 3. Just the Facts, Ma’am 4. The Proof is in the Pudding 5. Summary and Conclusions 6. Q&A 7. Next Steps
- 5. What is a Data Mesh?
- 6. 6 Traditional Approach to Data Management Illustration: Zhamak Dehghani, Data Mesh Principles and Logical Architecture, December 2020
- 7. 7 Traditional Approach to Data Management
- 8. 8 The ‘Knowledge Gap’ Challenge Domain experts who understand the domain data Domain experts who understand the domain processes Tool experts who understand how to develop data pipelines Illustration: Zhamak Dehghani, How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh, May 2019
- 9. 9 What is a Data Mesh? • The Data Mesh is a new organizational paradigm for data management • It moves from a centralized data infrastructure managed by a single team to a distributed organizational paradigm • Several autonomous units (domains) are in charge of managing and exposing their own “Data Products” to the rest of the organization • Data Products should be easily discoverable, understandable and accessible to the rest of the organization
- 10. 10 Data Mesh – 1. Domain based organization • Organizational units (domains) are responsible for managing and exposing their own data • Domains understand better how the data they own should be processed and used • Gives them autonomy to use the best tools to deal with their data, and to evolve them when needed • Results in shorter and fewer iterations until business needs are met
- 11. 11 Data Mesh – 2. Data as a Product • To ensure that domains do not become isolated data silos, the data exposed by the different domains must be: • Easily discoverable • Understandable • Secured • Usable by other domains • The processes and pipelines to generate the product (e.g. cleansing and deduplication) are internal implementation details and hidden to consumers
- 12. 12 Data Mesh – 3. Self-serve Data Platform • Self-Serve: while operated by a global data infrastructure team, it allows the domains to create and manage the data products themselves • The platform should be able to automate or simplify tasks such as: • Data integration and transformation • Security policies and identity management • Exposure of data APIs • Publish and document in a global catalog
- 13. 13 • Data products created by the different domains need to interoperate with each other and be combined to solve new needs • e.g. to be joined, aggregated, correlated, etc. • This requires agreement about the semantics of common entities (e.g. customer, product), about the formats of field types (e.g. SINs, entity identifiers,...), about addressability of data APIs, etc. • Managed globally and, when possible, automatically enforced • Security must be enforced globally according to the applicable regulations and policies Data Mesh – 4. Federated Computational Governance
- 14. 14 TRADITIONAL ATTRIBUTE DATA MESH Centralized ORGANIZATION Decentralized – Domain Ownership Monolithic & Physical ARCHITECTURE Distributed & Logical Top-Down Design OPERATIONALLY Federated Computational Governance Data is physically collected MINDSET Data Products – Connect and Share Analytical separate from Operational INFRASTRUCTURE Integrated How is it different from existing data delivery models?
- 15. Origins of the Myth
- 16. 16 Nascent Technologies = Confusion! • The fine details are not clearly defined • Lots of vendors jump on to the bandwagon • Original definition is distorted to fit existing product capabilities “I’m perhaps writing this book a bit earlier than I would have liked. We are still in the early years of a fundamentally different approach in sharing and creating data for analytical and machine learning (ML) use cases. But our industry has the tendency to percolate new concepts and buzzwords beyond recognition.” Zhamak Dehghani, Data Mesh – Delivering Data-Driven Value at Scale, O’Reilly
- 17. 17 Nascent Technologies = Confusion (Cont’d) Image: Sven Balnojan, There’s More Than One Kind of Data Mesh — Three Types of Data Meshes, Towards Data Science
- 18. Just the Facts, Ma’am
- 19. 19 TRADITIONAL ATTRIBUTE DATA MESH Centralized ORGANIZATION Decentralized – Domain Ownership Monolithic & Physical ARCHITECTURE Distributed & Logical Top-Down Design OPERATIONALLY Federated Computational Governance Data is physically collected MINDSET Data Products – Connect and Share Analytical separate from Operational INFRASTRUCTURE Integrated How is it different from existing data delivery models?
- 20. 20 Centralized vs. Decentralized Data Mesh DATA MESH ‘Centralized’ Data Mesh ‘Decentralized’ Data Mesh Decentralized – Domain Ownership ✓ ✓ Distributed & Logical ✓ Federated Computational Governance ? ✓ Data Products – Connect and Share ✓ ✓ Integrated ✓ ✓
- 21. 21 Centralized Data Mesh – Still ‘Business As Usual’? Illustration: Zhamak Dehghani, Data Mesh Principles and Logical Architecture, December 2020
- 22. The Proof is in the Pudding
- 23. 23 Data Mesh + Data Virtualization: Decentralized Organization – Centralized Control Domains create virtual models in separate schemas. Execution servers can be scaled independently 1 Domains can choose and autonomously evolve their data sources 2 Domains can share standardized definitions. Products can be used to define other products 3 A central team can set guidelines and enforce global security, quality and governance policies in the virtual models 4
- 24. 24 Data Mesh + Data Virtualization – Distributed Topology
- 25. 25 Other Data Mesh Reference Architectures Image: Data Mesh: Just another buzzword or the next-generation enterprise data platform, PwC Germany, May 2022
- 26. 26 1. Reusability: Data Virtualization platforms include strong capabilities to create and manage rich, layered semantic layers which foster reuse and expose data to each type of consumer in the form most suitable for them 2. Polyglot consumption: Data Virtualization allows data consumers to access data using any technology, not only SQL. For instance, self-describing REST, GraphQL and OData APIs can be created with a single click. Multidimensional access based on MDX is also possible 3. Top-down modelling: you can create ‘interface data views’ which set ‘schema contracts’ which developers of data products need to comply with • This helps to implement the concept of federated computational governance 4. Data marketplace: Ready-to-use data catalog which can act as a data marketplace for the data products created by the different domains Additional Benefits of a Data Virtualization approach to Data Mesh
- 28. 28 Summary and Conclusions • Data Mesh is a new paradigm for data management and analytics • It’s getting a lot of interest for diverse organizations • How to implement a Data Mesh is surrounded by hype and confusion • Data Virtualization offers a solid foundation to implement this new paradigm • Easy to use for business domain personas • Can leverage domain infrastructure or direct them towards a centralize repository • Simple yet advanced graphical modeling tools to define new products • Full governance and security controls Domain Domain Domain Domain Data Platform Domain Domain Domain Domain
- 29. 29 MYTH #7: I’m building a data mesh, so I don't need data virtualization
- 30. Q&A
- 31. 31 Get Started Today Try Denodo with the 30-day free trial in the cloud marketplaces CHOICE Under your cloud account SUPPORT Community forum AND remote sales engineer OPPORTUNITY 30 minutes free consultation with Denodo Cloud specialist denodo.com/free-trials
- 32. Thanks! www.denodo.com info@denodo.com © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.