The Extreme Data Cloud (XDC) Project
Download as PPTX, PDF0 likes93 views
The XDC project aims to develop scalable data management technologies for distributed computing environments. It will improve existing federated data management services by adding new functionalities requested by research communities. These include intelligent dataset distribution, policy-driven data orchestration, data preprocessing, smart caching, and enhanced metadata management. The project is co-funded by Horizon2020 and involves 8 partners representing 7 European countries and 7 research communities.
![XDC Consortium
8 partners, 7 countries
7 research communities represented + EGI
XDC Total Budget: 3.07Meuros
XDC started on Nov 1st – will run for 27 months
24/01/2018 D.Cesini - The eXtreme DataCloud Project 5
ID Partner Country Represented Community Tools and system
1 INFN
(Lead)
IT HEP/WLCG INDIGO-Orchestrator, INDIGO-
CDMI(*)
2 DESY DE Research with Photons
(XFEL)
dCache
3 CERN CH HEP/WLCG EOS, DYNAFED, FTS
4 AGH PL ONEDATA
5 ECRIN [ERIC] Medical data
6 UC ES Lifewatch
7 CNRS FR Astro [CTA and LSST]
8 EGI.eu NL EGI communities](https://image.slidesharecdn.com/4-xdceudatconf240118-180209120304/85/The-Extreme-Data-Cloud-XDC-Project-5-320.jpg)
The Extreme Data Cloud (XDC) Project
- 1. eXtreme DataCloud is co-funded by the Horizon2020 Framework Program – Grant Agreement 777367 Copyright © Members of the XDC Collaboration, 2017-2020 Data Management for extreme scale computing The XDC project Daniele Cesini daniele.cesini<at>extreme-datacloud.eu
- 2. XDC Objectives The eXtreme DataCloud is a software development and integration project Develops scalable technologies for federating storage resources and managing data in highly distributed computing environments Focus efficient, policy driven and Quality of Service based DM The targeted platforms are the current and next generation e- Infrastructures deployed in Europe European Open Science Cloud (EOSC) The e-infrastructures used by the represented communities 24/01/2018 D.Cesini - The eXtreme DataCloud Project 2
- 3. XDC Foundations XDC takes the move from the INDIGO Data management activity the experience of the project partners on data-management Improve already existing, production quality, Federated Data Management services By adding missing functionalities requested by research communities Must be coherently harmonized in the European e-Infrastructures 3 INDIGO PaaS Orchestrator INDIGO CDMI Server24/01/2018 D.Cesini - The eXtreme DataCloud Project
- 4. Represented research communities 24/01/2018 D.Cesini - The eXtreme DataCloud Project 4
- 5. XDC Consortium 8 partners, 7 countries 7 research communities represented + EGI XDC Total Budget: 3.07Meuros XDC started on Nov 1st – will run for 27 months 24/01/2018 D.Cesini - The eXtreme DataCloud Project 5 ID Partner Country Represented Community Tools and system 1 INFN (Lead) IT HEP/WLCG INDIGO-Orchestrator, INDIGO- CDMI(*) 2 DESY DE Research with Photons (XFEL) dCache 3 CERN CH HEP/WLCG EOS, DYNAFED, FTS 4 AGH PL ONEDATA 5 ECRIN [ERIC] Medical data 6 UC ES Lifewatch 7 CNRS FR Astro [CTA and LSST] 8 EGI.eu NL EGI communities
- 6. The New Functionalities Intelligent & Automated Dataset Distribution Orchestration to realize a policy-driven data management Data distribution policies based on Quality of Service (i.e. disks vs tape vs SSD) supporting geographical distributed resources (cross-sites) Software lifecycle management Data pre-processing during ingestion Data management based on access patterns Move to ‘glacier-like’ storage unused data, move to fast storage “hot” data at infrastructure level Smart caching Transparent access to remote data without the need of a-priori copy Metadata management Sensitive data handling secure storage and encryption 24/01/2018 D.Cesini - The eXtreme DataCloud Project 6
- 7. Policy driven Data Management Intelligent & Automated Dataset Distribution A typical workflow Initially the data will be stored on low latency devices for fast access To ensure data safety, the data will be replicated to a second storage device and will be migrated to custodial systems, which might be tape or S3 appliances Eligible users will get permission to restore archived data if necessary After a grace period, Access Control will be changed from “private” to “open access” Data management based on access pattern 724/01/2018 D.Cesini - The eXtreme DataCloud Project
- 8. Data pre-processing Data pre-processing during ingestion Automatically run user defined applications and workflows when data are uploaded i.e. for Skimming, indexing, metadata extraction, consistency checks Implement a solution to discover new data at specific locations Create the functions to request the INDIGO PaaS Orchestrator to execute specific applications on the computing resources on the Infrastructure Implement a high-level workflow engine, that will execute applications defined by the users Implement the data mover to store the elaborated data in the final destination 824/01/2018 D.Cesini - The eXtreme DataCloud Project
- 9. Smart caching Develop a global caching infrastructure supporting the following building blocks: dynamic integration of satellite sites by existing data centres creation of standalone caches modelled on existing web solutions federation of the above to create a large scale caching infrastructure 924/01/2018 Smart caching D.Cesini - The eXtreme DataCloud Project
- 10. Onedata developments 1024/01/2018 D.Cesini - The eXtreme DataCloud Project Unified data access platform at a PaaS level at the Exascale Multi-region support in Onedata Advanced metadata management with no pre-defined schema Encryption Services and Secure Storage Sensitive data management and key storage within Onedata
- 11. Metadata handling use cases D.Cesini - The eXtreme DataCloud Project 11 LIFEWATCH CTA ECRIN Metadata management to handle heterogeneous and large datasets Different data types, formats, source and ways to access e.g. Copernicus data: ~16PB per year Used as input for water quality forecasting systems Use of standards like EML (Ecological Metadata Language) and adopting best practices like FAIR+R principles The CTA distributed archive lies on the « Open Archival Information System » (OAIS) ISO standard. Event data are in files (FITS format) containing all metadata. Metadata are extracted from the ingested files, with an automatic filling of the metadata database. Metadata will be used for the further query of archive. The system should be able to manage replicas, tapes, disks, etc, with data from low-level to high-level. Clinical trial data objects available for sharing with others a variety of access mechanisms wide variety of different locations growing number of general and specialised data repositories trial registries Publications the original researchers’ institutions ‘discoverability’ will become much worse in the future as more and more materials is made available for sharing 24/01/2018
- 12. XDC high level architecture 24/01/2018 D.Cesini - The eXtreme DataCloud Project 12
- 13. Project Structure WP1 - Project Management (NA1) WP2: New functionalities definition and Research Communities support (NA2) WP3: Software Lifecycle Management, Pilot Infrastructures & Exploitation (SA1) WP4: Orchestration and policy driven data management (JRA1) WP5: Unified cross federations data management (JRA2) 24/01/2018 D.Cesini - The eXtreme DataCloud Project 13
- 14. Management bodies 24/01/2018 D.Cesini - The eXtreme DataCloud Project 14 ELG is responsible for maintaining active relationships with the infrastructure and technology providers, discussing synergies, strategies, roadmaps and requirements workflow for the software released by the project.
- 15. The plan for the next couple of years Main Milestones Research communities requirements for new functionalities collected Research communities requirements analysis performed Project architecture detailed Development schedule defined Event with User Communities XDC reference releases – 1 XDC reference releases – 2 Functionalities and scalability demonstrated 24/01/2018 D.Cesini - The eXtreme DataCloud Project 15 PM3 PM6 PM12 PM24 PM27 PM9 - Joint with DEEP in Santander
- 16. Conclusion XDC has an ambitious development plan for data management services We want to support very diverse use cases and requirements We need really a modular and flexible approach in building our platform We will support as much as possible standards protocols to make the solutions as general as possible Sustainability of the products: Provide upstream to the original project all the changes developed by XDC Involving the user communities in exploiting the XDC outputs in their production environments Pushing XDC developments in the EOSC Service Catalogue 24/01/2018 D.Cesini - The eXtreme DataCloud Project 16
- 17. XDC Contacts Website: www.extreme-datacloud.eu @XtremeDataCloud on Twitter Mailing list: info<at>extreme-datacloud.eu 24/01/2018 D.Cesini - The eXtreme DataCloud Project 17