Denodo DataFest 2017: Multi-zone Data Virtualization for Data Lakes
2 likes311 views
The document outlines a strategy for implementing multi-zone data virtualization in data lakes for the Dutch government to enhance data sharing while preserving privacy and security. It identifies key challenges such as the need for faster data sharing, cost reduction, and improved data accessibility. Next steps include completing a proof-of-concept and developing a minimal viable product by the second half of 2018.
Denodo DataFest 2017: Multi-zone Data Virtualization for Data Lakes
- 1. Multi-zone Data Virtualization for Data Lakes How to share data with other government agencies preserving privacy and security guidelines Paul Grooten October 26th, 2017
- 2. Statistics Netherlands (CBS) Key Characteristics 2Multi-zone Data Virtualization for Data Lakes | Autonomous Public Body with a Legal Entity (“ZBO”) Official Statistics Economic - Social - Census National and Regional 180 mEur 2000+ The Hague Heerlen Bonaire Founded in 1899 (5 fte, 2 rooms), now 3 offices Ambition: to become the Data Hub of the Dutch Government Data Collection Data Processing Publishing Statistical process
- 3. Which problems do we want to solve • Current methods and technologies are not sufficient anymore to share data easily on a bigger scale • We want to share more statistical data (also with external parties) • We want to become faster and need a shorter time to market • We need to reduce costs (storage, infrastructure) • We need to work on secure & privacy preserved data sharing • Data sets should be easy to find 3Multi-zone Data Virtualization for Data Lakes |
- 4. The layered Data Architecture 4 Demand Supply (Legacy) Datasources Data Source Layer (DSL) CSV SQL DB Web Srv ETL tooling XLS App CBDS Vraag Consumer Layer (CL) Web PageS2STooling P V A P V A= Data Prep = Data Visualization = Data Analytics Security DataVirtualization DENODO Data Transformation Layer (DTL) Data Provisioning Layer (DPL) Building Block 1 Building Block 2 Building Block 3 Building Block 4 Web- Service C OData Web- Service B Web- Service A Security User Que. DataGovernance Tech Meta MetadataManagement Import Conceptual Meta Conn. String Existing New CIO office | Versie 1.81 Security&Autorisation Multi-zone Data Virtualization for Data Lakes |
- 5. …towards a multi zone DaaS Architecture 5 Security CL Datasources DSL DataVirtualization DTL DPL DataGovernance Existing New User Que. MetadataManagement Tech Meta Zone CBS DDC=Departemental Data Center | UDC=Urban Data Center | CL=Consumer Layer DPL=Data Provisioning Layer | DTL=DataTransformation Layer | DSL=Data Source Layer Building Block 1 Building Block 2 Web- Service A CBDS DSC P V A P V A= Data Prep = Data Visualization = Data Analytics Security Zone DDC1 Building Block 7 Building Block 8 Web- Service D DDC1 Secured VPN P V A Zone UDC1 Building Block 3 Building Block 4 Web- Service B UDC1 Secured VPN P V A Zone UDC2 Building Block 5 Building Block 6 Web- Service C UDC2 Secured VPN P V A EHB Security&Autorisation
- 6. So what is a Zone? 6 Security CL Datasources DSL DataVirtualization DTL DPL DataGovernance Existing New User Que. MetadataManagement Tech Meta Zone CBS DDC=Departemental Data Center | UDC=Urban Data Center | CL=Consumer Layer DPL=Data Provisioning Layer | DTL=DataTransformation Layer | DSL=Data Source Layer Building Block 1 Building Block 2 Web- Service A CBDS DSC P V A P V A= Data Prep = Data Visualization = Data Analytics Security Zone EZ Building Block 7 Building Block 8 Web- Service D DDC1 Secured VPN P V A Zone UDC1 Building Block 3 Building Block 4 Web- Service B UDC1 Secured VPN P V A Zone UDC2 Building Block 5 Building Block 6 Web- Service C UDC2 Secured VPN P V A EHB Security&Autorisation A Zone : • Is a virtual container in which a specified set of Data Governance rules apply • Has a specific user group • Contains virtual datasets • Has it’s own authorization (which can and will differ from other zones) • Has an owner • Has it’s own Change Advisory Board • Can have it’s own cache database on it’s own hardware
- 7. What do we want to achieve with the Data Lake 7 € M { " stimulateCost data- access Statistical Risc Growth Re-use Time to Market reduce Multi-zone Data Virtualization for Data Lakes |
- 8. What are our next steps • Finish Proof-of-Concept by end 2017 • Develop product (MVP) • Get approval from Board of Management • Implement Minimal Viable Product in 2018/H2 • Enhance MVP with new functionalities, like disclosure control (confidentiality on-the-fly) protection 8Multi-zone Data Virtualization for Data Lakes |
- 9. Recommendations • Check whether your strategy is in line with your plans (v.v.) • Start experimenting with Data Virtualization in an early stage (start with the express version) • Build a culture that embraces change and communicate your plans as often as possible 9Multi-zone Data Virtualization for Data Lakes |