The XDC project

eXtreme DataCloud is co-funded by the Horizon2020
Framework Program – Grant Agreement 777367
Copyright © Members of the XDC Collaboration, 2017-2020
Data Management for extreme scale computing
The XDC project
Alessandro Costantini
INFN
alessandro.costantini<at>extreme-datacloud.eu
Daniele Cesini
INFN - Project Coordinator
daniele.cesini<at>extreme-datacloud.eu

Outline
XDC: an introduction
The XDC-1/Pulsar Release: the first year achievements
Next Steps & Conclusion
26/03/2019 A. Costatnini, D.Cesini - XDC Project Overview – EOSC-hub week 2019 2

XDC
The eXtreme DataCloud is a software development and integration project
Develops scalable technologies for federating storage resources and
managing data in highly distributed computing environments
Focus efficient, policy driven and Quality of Service based DM
The targeted platforms are the current and next generation e-
Infrastructures deployed in Europe
European Open Science Cloud (EOSC)
The e-infrastructures used by the represented communities
Addresses the EINFRA-21-2017 (b)-2: “Computing e-infrastructure with
extreme large datasets”
Deal with heterogeneous datasets
Bring to TRL8 and include in a unified service catalogue services and prototype at
least at TRL6
26/03/2019 3A. Costatnini, D.Cesini - XDC Project Overview – EOSC-hub week 2019

XDC Consortium
8 partners, 7 countries
6 research communities represented + EGI
XDC Total Budget: 3.07Meuros
26/03/2019 4
ID Partner Country Represented Community Tools and system
1 INFN
(Lead)
IT HEP/WLCG
INDIGO-Orchestrator
2 DESY DE Research with Photons
(XFEL)
dCache
3 CERN CH HEP/WLCG EOS, DYNAFED, FTS, RUCIO
4 AGH PL ONEDATA
5 ECRIN [ERIC] Medical data
6 UC ES Lifewatch
7 CNRS FR Astro [CTA and LSST]
8 EGI.eu NL EGI communities
A. Costatnini, D.Cesini - XDC Project Overview – EOSC-hub week 2019

The Approach
Improve already existing, production quality Data Management
services
By adding missing functionalities requested by research communities
Based mainly on technologies provided by the partners and by the
INDIGO-Datacloud project
Must be coherently harmonized in the European e-Infrastructures
5
INDIGO PaaS
Orchestrator
INDIGO CDMI
Server
FTS
26/03/2019 A. Costatnini, D.Cesini - XDC Project Overview – EOSC-hub week 2019

XDC Approach
The partners owning/involved in each of the tools are the main
developers for that solution in XDC
We always aim to push back the code in the main
development tree on the original projects
This widely increase the sustainability of the services
26/03/2019 6
Upstream repositories
XDC new features

XDC standards and protocols
We always rely on standard and widely adopted protocols for
services-to-services and for user-to-services interaction
This concretely opens the possibility to interact with external services
At the infrastructure level we provide few reference implementations
others services/initiative are implementing the same functionalities using
same standard/protocols

Participation to standardization bodies
RDA - Research Data Alliance
Participating (DESY) to the Storage Service Definition WG
SNIA - Storage Networking Industry Association
XDC presented at the Storage Development Conference 2018
DESY is leading the standardization work on QoS for the SNIA CDMI
protocol
DOMA – Data Organization Management Access in WLCG
Not a proper standardization body, but affects the computing models
of a huge community
XDC leads the QoS working groups (Paul Millar, DESY)
XDC participates to all other working groups
Thirdparty copy
Caching (Access)

A User Driven Project

The New Functionalities
Intelligent & Automated Dataset Distribution
Orchestration to realize a policy-driven data management
Data distribution policies based on Quality of Service (i.e. disks vs tape vs SSD)
supporting geographical distributed resources (cross-sites)
Data lifecycle management
Data pre-processing during ingestion
Metadata management
Data management based on storage events
Smart caching
Transparent access to remote data without the need of a-priori copy
To support dynamic inclusion of diskless sites
To improve efficiency in multi-site storage systems and storage federations (i.e. Datalakes)
Sensitive data handling
secure storage and encryption

General Architecture Definition
XDC acts at all the e-infrastructure levels
Storage systems at sites
Federations of storage systems
regional and global
High level orchestration
User experience
26/03/2019 11
The XDC Toolbox
The “toolbox” was mapped in those levels
to define the general architecture
Taking into account the user requirements

First XDC Release
Involved tools
CachingOnDemand
dCache
Dynafed
EOS
Key technical highlights
OpenIDConnect support for token based
authentication
New QoS types integration and support
in dCache, FTS, GFAL
Orchestrator integration with other
components
Performance improvements in Onedata
Support for groups and roles in Onedata
EOS-dCache integration
Caching systems instantiation
Storage events notification in dCache
EOS caching with XCache for geographic
deployment
EOS external storage adoption
26/03/2019 12
https://releases.extreme-datacloud.eu/en/latest/releases/pulsar/index.html
FTS,GFAL
Onedata
PaaS Orchestrator plugin
TOSCA types & templates
plugin

First XDC Release
OpenIDConnect support for
token based authentication
26/03/2019 13
Slide by Akos Frohner ~2003
Slide by Andrea Ceccanti CHEP2018

First XDC Release
26/03/2019 14
Deployment of Geo-distributed
caches
Network of unmanaged storage
for hot data
On-demand cache resources
Slide from Diego Ciangottini
Caching systems instantiation

First XDC Release
26/03/2019 15
Storage events notification in
dCache

First XDC Release
26/03/2019 16
Storage events notification in
dCache

First XDC Release
26/03/2019 17
Performance improvements in
Onedata
Support for groups and roles in
Onedata
Onedata Transparent POSIX File System
Processing transparently cached data - 37GBytes/sec
- Data Transfer Mesh
- 3 Oneproviders connected by
20+Gbit/s links
-Transfer data between all them
- Single VM Node per Provider
- Linear scalability
Data Migration at combined
Throughput 56 Gbit/s

Exploitation and sustainability path
The Service Providers Board (SPB) goal is to link XDC with Service Provider within and
outside of the project consortium in order to have a regular dialog among XDC and all the
Service Providers
First meeting @EOSC-hub week 2019
Making XDC products available through EGI distribution channels
UMD release already delivers dCache, FTS, GFAL, XRootd
Identification and interaction with the EOSC-HUB Service Providers
To be included in the XDC SPB (if not already present)
Pushing developments in the upstream repositories of all the services to ensure
sustainability beyond the project
Double path
XDC repositories
Upstream repository
XDC solutions included in other projects or actions
Dealing with BigData infrastructures
Dealing with the DataManagement of extreme scale experiments

Next Steps
Start working on the second release (codenamed Quasar)
Will complete the implementation of the XDC architecture
Message bus, full orchestration, finalize integration of RUCIO, secure storage in
Onedata, finalize the ECRIN use case developments, complete caching
reference workflows
Continue the dissemination activities in events and
standardization bodies
Finalize the XDC Service Catalogue dissemination material
Organize a training event towards the EOSC
Closer interaction with Service Providers and EOSC-HUB
Scalability testing and verification

XDC Contacts
Website: www.extreme-datacloud.eu
@XtremeDataCloud on Twitter
Mailing list: info<at>extreme-datacloud.eu

Internal Events

The XDC project

More Related Content

What's hot (6)

Similar to The XDC project (20)

More from EOSC-hub project (20)

Recently uploaded (20)

The XDC project