The Archiver project

ARCHIVER Project
EOSC-hub Week
11th of April 2019 – Prague, Czech Republic
João Fernandes
Project Coordinator
CERN

Pre-Commercial Procurement
08.04.2019 http://www.archiver-project.eu 2
Focus: Archiving and Data Preservation Services using commercial cloud services
Procurement Budget: 3.4M euro
Starting Date: 1st of January 2019
Duration: 36 Months
Coordinating Partner: CERN

Consortium
Includes Buyers and Experts in the preparation, execution and promotion of the Procurement of R&D
Procurers - Public organisations committing funds to contribute to a joint-procurement
Experts – Partner organisations bringing expertise in requirement assessment and promotion activities, not part of the Buyers Group

Problem
Data volumes growing; Communities Growing; Needs of advanced functionality increase
Objective: perform R&D to demonstrate long-term preservation and archiving services for scientific data in the PB range under
F.A.I.R. principles, using commons solutions, while research groups keep total stewardship of their data sets
Standardize resulting services under the OAIS reference model
Hybrid Deployments
PB data volume; High sustained ingest rates (1-10 Gbps)
Access through GÉANT network; Federated AAI
Cost-effective and transparent business models
Reduce fragmentation expanding existing solutions to several scientific domains
Basic Scientific Data
Archiving & Bit
Preservation

OAIS Reference Model
Relevant Standards
Preservation: ISO 14721/16393, 26324 and related
Infrastructure Services: ISO 27000, ISO 27040, 19086

Cost-effective
Business Model
Taking into account:
- Scale
- Ingest rates
- Archive lifetime
- # of copies
- Exit strategies
- Portability
- SLAs
Regulation &
Legislation
- Auditing
- Self-assessment
- Data Retention
- GDPR
R&D Challenge
08.042019 http://www.archiver-project.eu 6
Data integrity/security; cloud/hybrid deployment; data
volume in the PB range; high, sustained ingest data rates;
ISO certification: 27000, 27040, 19086 and related
Archives connected to the GEANT network.
OAIS conformant services: data readability formats, normalization,
obsolesce monitoring, files fixity, authenticity checks, etc.;
ISO 14721/16393, 26324 and related standards
User services: search, discover, share, indexing, data removal, etc.;
Access under Federated IAM
Layer 1
Storage/Basic Archiving/Secure
backup
Layer 2
Preservation
Layer 3
Baseline user services
Layer 4
Advanced services
High level services: visual representation of data (domain specific),
reproducibility of scientific analyses, etc.;
Minimum
R&D
Objectives
Added
value
R&D
Fundamental aspect
of the challenge

Initial List of Deployment Scenarios
Field Scenario Name
High Energy Physics
[4]
BaBar Archive Stage 1
DPHEP EOSC Science Demonstrator
CERN Open Data Cloud Archive Services / CODCAS
CERN E-Ternity
Life Sciences
[2]
EMBL/FIRE
EMBL Cloud-caching for Data Analysis
Astronomy and Cosmology [3] Second copy of data for Disaster Recovery / DISASTER
Analysis dataset server for gamma-ray astronomy / GAMMADAT
Open Data Publisher / OPENPUB
Photon Science
[3]
Photon-Science/Scientist
Photon-Science/Working Group
Photon Science/Collaboration
11/04/2019 http://www.archiver-project.eu 7

Data Volumes
Type Deployment Scenario Name Data Volumes
Low Range Scenarios
[3]
Analysis dataset server for gamma-ray astronomy /
GAMMADAT
0.01 PB
Open Data Publisher / OPENPUB 0.01 PB
DPHEP EOSC Science Demonstrator 0.1+ PB
Medium Range Scenarios
[3]
Photon-Science/Scientist 0.5 PB
EMBL Cloud-caching for Data Analysis 0.5 PB
CERN E-Ternity 0.7 PB
High Range Scenarios
[6]
Second copy of data for Disaster Recovery / DISASTER 0.3 PB / year
Photon-Science/Working Group 1 PB
BaBar Archive Stage 1 2 PB
CERN Open Data Cloud Archive Services / CODCAS 2+ PB
EMBL on Fire 20+ PB
Photon Science/Collaboration 100 PB

Retention Period
Type Deployment Scenario Name Retention Period
Short Retention Period [2] Second copy of data for Disaster Recovery / DISASTER <5 years
EMBL Cloud-caching for Data Analysis <5 years
Medium Retention Period [8] Photon Science/Collaboration 10+ years
Photon-Science/Working Group 10+ years
Photon-Science/Scientist 10+ years
BaBar Archive Stage 1 10 years
DPHEP EOSC Science Demonstrator 10 years
GAMMADAT
10+ years
CERN Open Data Cloud Archive Services / CODCAS 5 - 10 years
CERN E-Ternity 10+ years
Long Retention Period [2] Open Data Publisher / OPENPUB 25+ years
EMBL on Fire 25+ years

Data Ingest Rates
Type Deployment Scenario Name Data Ingest Rates
Low Rates [1] CERN E-Ternity 0.01 GB/s
Medium Rates
[3]
CERN Open Data Cloud Archive Services / CODCAS 1 GB/s
Photon-Science/Scientist 1-2 GB/s
EMBL on Fire 1 – 2 GB/s
High Rates
[7]
Second copy of data for Disaster Recovery / DISASTER 1 – 10 GB/s
Photon-Science/Working Group 1-10 GB/s
GAMMADAT
1 – 10 GB/s
BaBar Archive Stage 1 1 – 10 GB/s
EMBL Cloud-caching for Data Analysis 1 – 10 GB/s
DPHEP EOSC Science Demonstrator 1 – 10 GB/s
Open Data Publisher / OPENPUB 1 – 10 GB/s
Very High Rates [1] Photon Science/Collaboration 8-20 GB/s

Data Volume vs Retention Period vs Data Ingest

ARCHIVER & European Open Science Cloud (EOSC)
ARCHIVER Services can be procured
2019 2020 2021 2022 2023
State-of-the-art analysis in the EOSC context
National repositories via OpenAIRE project
ESFRI cluster projects: ENVRI-FAIR, EOSC-LIFE, ESCAPE, PANOSC and SSHOC
A number of organisations have expressed interest in becoming early adopters
Consulting Experts: DPC (Data Preservation Coalition)

Research Infrastructures
ESFRI Landmarks
RIs already implemented or in an advanced Implementation Phase.
ESFRI Projects
RIs in a preparation phase selected for scientific case excellence and maturity.
http://roadmap2018.esfri.eu/media/1049/roadmap18-part3.pdf

ARCHIVER timeline
We are here
1 PB per Contractor - Prototype and Pilot
Suppliers to sign a framework agreement in order to participate in the call-offs
Goal: Services to be commercialized after the end of the PCP

ARCHIVER Public Consultation Process
Goals:
Allow Suppliers to comment on the Functional Specifications
Open dialogue sharing as much information as possible before Tender publication
Series of Open Market Consultation events
dedicated to companies; OAIS training and
ARCHIVER webinars for the research community
Need to be ready to bid 

Open Market Consultation Kick-Off event – 8th of April
70 participants from 26 companies physically
present
Digital Preservation Experts, Cloud Providers, Software
integrators
42 participants connected remotely (via webcast)
Planning poker to assess Value vs. Risk in a direct
dialogue with companies

Open Market Consultation Events
Registration open: https://www.archiver-project.eu/all-events

Next Steps
ARCHIVER open market consultation April-June 2019
Open dialogue with suppliers for tender drafting from April-June 2019
Tender publication in October 2019
Information about ARCHIVER at: http://archiver-project.eu

Summary
The objective of ARCHIVER is to perform R&D to demonstrate functionality and
performance of services for long-term preservation and archiving for scientific data in the
PB range under F.A.I.R. guiding principles, following best practices (OAIS reference model
and related standards) while ensuring that research groups will retain stewardship of their
data sets
ARCHIVER Pre-Commercial Procurement will run an open tender and the objective is to
integrate the resulting services on the EOSC catalogue for broad availability
Feedback on the drafts of the “Functional Specifications” document from the suppliers
The Buyers group will co-design and co-develop with you a test plan - based on the
outcome of the Design Phase, the Functional Specifications and the Deployment Scenarios
The test assessment will be a deciding factor to qualify solutions to the subsequent phases
The tests will focus on basic functionality capabilities during the prototype phase and
performance, efficiency, and scalability during the pilot phase

The Archiver project

More Related Content

What's hot (9)

Similar to The Archiver project (20)

More from EOSC-hub project (20)

Recently uploaded (20)

The Archiver project

Editor's Notes