3 archiver omc deployment_scenarios
Download as PPTX, PDF1 like559 views
The document discusses deployment scenarios for the ARCHIVER project, which aims to demonstrate long-term preservation and archiving services for scientific data. It outlines scenarios for high energy physics, life sciences, astronomy, and photon science. These scenarios involve data volumes ranging from terabytes to petabytes and retention periods ranging from less than 5 years to over 25 years. The document also provides information on data ingest rates, which range from gigabytes per second to tens of gigabytes per second. It concludes with a summary of next steps for the project.
![Initial List of Deployment Scenarios
Field Scenario Name
High Energy Physics
[4]
BaBar Archive Stage 1
DPHEP EOSC Science Demonstrator
CERN Open Data Cloud Archive Services / CODCAS
CERN E-Ternity
Life Sciences
[2]
EMBL/FIRE
EMBL Cloud-caching for Data Analysis
Astronomy and Cosmology [3] Second copy of data for Disaster Recovery / DISASTER
Analysis dataset server for gamma-ray astronomy / GAMMADAT
Open Data Publisher / OPENPUB
Photon Science
[3]
Photon-Science/Scientist
Photon-Science/Working Group
Photon Science/Collaboration
10/04/2019 http://www.archiver-project.eu 7](https://image.slidesharecdn.com/3archiveromcdeploymentscenarios-190410124737/85/3-archiver-omc-deployment_scenarios-7-320.jpg)
![10/04/2019 http://www.archiver-project.eu 8
High Energy Physics Scenario Goals
In 2020 the BaBar Experiment infrastructure at SLAC will be decommissioned. As a result, BaBar
data [2 PBs] can no longer be stored at the host laboratory and alternative solutions need to be
found. Currently a copy of the data is being held by CERN IT. We want to ensure that a complete
copy of Babar data will be retained for possible comparisons with data from other experiments
and sharing through the CERN Open Data Portal.
The CERN Open Data portal disseminates close to 2 PBs of open particle physics data released by
LHC experiments and is being used for both education and research purposes. We want to
establish a “passive” data archive for disaster-recovery purposes as well as an additional “active”,
exposed via protocols such as S3 and XRootD, which will allow users to run open data analysis
examples.
We want to archive the ~1 PB of CERN Digital Memory, containing analog documents produced by
the institution in the 20th century as well as digital production of the 21st century, including new
types like web sites, social medias, emails, etc.](https://image.slidesharecdn.com/3archiveromcdeploymentscenarios-190410124737/85/3-archiver-omc-deployment_scenarios-8-320.jpg)
![Data Volumes
Type Deployment Scenario Name Data Volumes
Low Range Scenarios
[3]
Analysis dataset server for gamma-ray astronomy /
GAMMADAT
0.01 PB
Open Data Publisher / OPENPUB 0.01 PB
DPHEP EOSC Science Demonstrator 0.1+ PB
Medium Range Scenarios
[3]
Photon-Science/Scientist 0.5 PB
EMBL Cloud-caching for Data Analysis 0.5 PB
CERN E-Ternity 0.7 PB
High Range Scenarios
[6]
Second copy of data for Disaster Recovery / DISASTER 0.3 PB / year
Photon-Science/Working Group 1 PB
BaBar Archive Stage 1 2 PB
CERN Open Data Cloud Archive Services / CODCAS 2+ PB
EMBL on Fire 20+ PB
Photon Science/Collaboration 100 PB
10/04/2019 http://www.archiver-project.eu 13](https://image.slidesharecdn.com/3archiveromcdeploymentscenarios-190410124737/85/3-archiver-omc-deployment_scenarios-13-320.jpg)
![Retention Period
10/04/2019 http://www.archiver-project.eu 14
Type Deployment Scenario Name Retention Period
Short Retention Period [2] Second copy of data for Disaster Recovery / DISASTER <5 years
EMBL Cloud-caching for Data Analysis <5 years
Medium Retention Period [8] Photon Science/Collaboration 10+ years
Photon-Science/Working Group 10+ years
Photon-Science/Scientist 10+ years
BaBar Archive Stage 1 10 years
DPHEP EOSC Science Demonstrator 10 years
Analysis dataset server for gamma-ray astronomy /
GAMMADAT
10+ years
CERN Open Data Cloud Archive Services / CODCAS 5 - 10 years
CERN E-Ternity 10+ years
Long Retention Period [2] Open Data Publisher / OPENPUB 25+ years
EMBL on Fire 25+ years](https://image.slidesharecdn.com/3archiveromcdeploymentscenarios-190410124737/85/3-archiver-omc-deployment_scenarios-14-320.jpg)
![Data Ingest Rates
10/04/2019 http://www.archiver-project.eu 15
Type Deployment Scenario Name Data Ingest Rates
Low Rates [1] CERN E-Ternity 0.01 GB/s
Medium Rates
[3]
CERN Open Data Cloud Archive Services / CODCAS 1 GB/s
Photon-Science/Scientist 1-2 GB/s
EMBL on Fire 1 – 2 GB/s
High Rates
[7]
Second copy of data for Disaster Recovery / DISASTER 1 – 10 GB/s
Photon-Science/Working Group 1-10 GB/s
Analysis dataset server for gamma-ray astronomy /
GAMMADAT
1 – 10 GB/s
BaBar Archive Stage 1 1 – 10 GB/s
EMBL Cloud-caching for Data Analysis 1 – 10 GB/s
DPHEP EOSC Science Demonstrator 1 – 10 GB/s
Open Data Publisher / OPENPUB 1 – 10 GB/s
Very High Rates [1] Photon Science/Collaboration 8-20 GB/s](https://image.slidesharecdn.com/3archiveromcdeploymentscenarios-190410124737/85/3-archiver-omc-deployment_scenarios-15-320.jpg)
3 archiver omc deployment_scenarios
- 1. Buyers Group Deployments Scenarios Evangelos Motesnitsalis Technical Coordinator OMC Kick-off Event 8 April 2019
- 2. 10/04/2019 http://www.archiver-project.eu 2 Contents OAIS Reference Model FAIR Principles Deployment Scenarios Buyers Group Goals High Energy Phyics Goals Life Science Goals Astronomy Goals Photon Science Goals Data Volumes Data Ingest Rates Retention Period Summary
- 4. 10/04/2019 http://www.archiver-project.eu 4 OAIS Reference Model Relevant Standards Preservation: ISO 14721/16393, 26324 and related standards Storage/Basic Archiving/Secure backup: ISO 27000, 27040, 19086
- 5. 10/04/2019 http://www.archiver-project.eu 5 FAIR Principles Findable AccessibleInteroperable Re-Usable • Accurate and relevant description • Data usage license and detailed provenance • Retrievable with free protocols • Accessible metadata even after deletion • Global, unique identifiers • Rich Metadata, indexes, search capabilities • Qualified reference to other data • Formal, shared and broadly applicable knowledge representation standards https://www.go-fair.org/
- 7. Initial List of Deployment Scenarios Field Scenario Name High Energy Physics [4] BaBar Archive Stage 1 DPHEP EOSC Science Demonstrator CERN Open Data Cloud Archive Services / CODCAS CERN E-Ternity Life Sciences [2] EMBL/FIRE EMBL Cloud-caching for Data Analysis Astronomy and Cosmology [3] Second copy of data for Disaster Recovery / DISASTER Analysis dataset server for gamma-ray astronomy / GAMMADAT Open Data Publisher / OPENPUB Photon Science [3] Photon-Science/Scientist Photon-Science/Working Group Photon Science/Collaboration 10/04/2019 http://www.archiver-project.eu 7
- 8. 10/04/2019 http://www.archiver-project.eu 8 High Energy Physics Scenario Goals In 2020 the BaBar Experiment infrastructure at SLAC will be decommissioned. As a result, BaBar data [2 PBs] can no longer be stored at the host laboratory and alternative solutions need to be found. Currently a copy of the data is being held by CERN IT. We want to ensure that a complete copy of Babar data will be retained for possible comparisons with data from other experiments and sharing through the CERN Open Data Portal. The CERN Open Data portal disseminates close to 2 PBs of open particle physics data released by LHC experiments and is being used for both education and research purposes. We want to establish a “passive” data archive for disaster-recovery purposes as well as an additional “active”, exposed via protocols such as S3 and XRootD, which will allow users to run open data analysis examples. We want to archive the ~1 PB of CERN Digital Memory, containing analog documents produced by the institution in the 20th century as well as digital production of the 21st century, including new types like web sites, social medias, emails, etc.
- 9. 10/04/2019 http://www.archiver-project.eu 9 Life Sciences Scenario Goals EMBL-EBI provides data archiving services to the global molecular biology community. These data archives are currently based on an internal service (FIRE: FIle REplication) that stores the files in two different systems: a distributed object store and tape. FIRE currently holds 20PB of data and is growing at 40% per year. We want to ensure that: FIRE can achieve cost-effective scaling via cloud-based storage solutions Data can effectively be distributed on cloud infrastructure, covering the increasing needs for cloud-hosted analysis As research communities access more and more of internal data from cloud services for their data analysis, it makes sense to progressively cache/store data in the cloud, with the on- premises data being replicated and discarded as required. Which data should be cached/stored, how much and for how long, will be a tradeoff between the cost of cloud storage and of having the network capacity/latency to download the data multiple times.
- 10. 10/04/2019 http://www.archiver-project.eu 10 The MAGIC Cherenkov gamma-ray telescopes and the PAUcam camera for the William Herschel Telescope are located in the Observatorio del Roque de los Muchachos, in Canary Islands, Spain. The first Large Scale Telescope of the next-generation Cherenkov Telescope Array (CTA) is also there. They produce about 0.3 PB of raw data per year which is automatically sent to PIC in Barcelona. Data are rarely recalled –less than once per year – but whenever required, they must be accessible within 3 weeks. Our goal is: to ensure that a second copy of data is retained for disaster recovery purposes. to replace the current data distribution service at PIC by a commercial service with better functionality, easier maintenance and lower cost. to acquire a method to publish certain datasets as Open Data according to Digital Library standards and link them to publications. Astronomy Scenario Goals
- 11. 10/04/2019 http://www.archiver-project.eu 11 Photon Science Scenario Goals Individual scientists at DESY need a service to create archives for their experiment data as well as their publications with specific capabilities such as continuous data ingestion via browser or third-party copies Working groups want to be able to create/manage/delete archives based on accepted data policies supporting a wide range of options for cloud and on-prem storage, while being able to utilize existing user credentials, authentication techniques and identification mechanisms. Long-lived collaborations present a growing need to plan and execute archiving operations in a fully automated and policy-based, certified, documented way via API and a close to 100% automated procedures.
- 13. Data Volumes Type Deployment Scenario Name Data Volumes Low Range Scenarios [3] Analysis dataset server for gamma-ray astronomy / GAMMADAT 0.01 PB Open Data Publisher / OPENPUB 0.01 PB DPHEP EOSC Science Demonstrator 0.1+ PB Medium Range Scenarios [3] Photon-Science/Scientist 0.5 PB EMBL Cloud-caching for Data Analysis 0.5 PB CERN E-Ternity 0.7 PB High Range Scenarios [6] Second copy of data for Disaster Recovery / DISASTER 0.3 PB / year Photon-Science/Working Group 1 PB BaBar Archive Stage 1 2 PB CERN Open Data Cloud Archive Services / CODCAS 2+ PB EMBL on Fire 20+ PB Photon Science/Collaboration 100 PB 10/04/2019 http://www.archiver-project.eu 13
- 14. Retention Period 10/04/2019 http://www.archiver-project.eu 14 Type Deployment Scenario Name Retention Period Short Retention Period [2] Second copy of data for Disaster Recovery / DISASTER <5 years EMBL Cloud-caching for Data Analysis <5 years Medium Retention Period [8] Photon Science/Collaboration 10+ years Photon-Science/Working Group 10+ years Photon-Science/Scientist 10+ years BaBar Archive Stage 1 10 years DPHEP EOSC Science Demonstrator 10 years Analysis dataset server for gamma-ray astronomy / GAMMADAT 10+ years CERN Open Data Cloud Archive Services / CODCAS 5 - 10 years CERN E-Ternity 10+ years Long Retention Period [2] Open Data Publisher / OPENPUB 25+ years EMBL on Fire 25+ years
- 15. Data Ingest Rates 10/04/2019 http://www.archiver-project.eu 15 Type Deployment Scenario Name Data Ingest Rates Low Rates [1] CERN E-Ternity 0.01 GB/s Medium Rates [3] CERN Open Data Cloud Archive Services / CODCAS 1 GB/s Photon-Science/Scientist 1-2 GB/s EMBL on Fire 1 – 2 GB/s High Rates [7] Second copy of data for Disaster Recovery / DISASTER 1 – 10 GB/s Photon-Science/Working Group 1-10 GB/s Analysis dataset server for gamma-ray astronomy / GAMMADAT 1 – 10 GB/s BaBar Archive Stage 1 1 – 10 GB/s EMBL Cloud-caching for Data Analysis 1 – 10 GB/s DPHEP EOSC Science Demonstrator 1 – 10 GB/s Open Data Publisher / OPENPUB 1 – 10 GB/s Very High Rates [1] Photon Science/Collaboration 8-20 GB/s
- 17. Summary and Next Steps
- 18. 10/04/2019 http://www.archiver-project.eu 18 Summary and Next Steps The objective of ARCHIVER is to perform R&D to demonstrate functionality and performance of services for long-term preservation and archiving for scientific data in the PB range under F.A.I.R. principles, while ensuring that research groups will retain stewardship of their data sets ARCHIVER Pre-Commercial Procurement will run an open tender and the resulting services will be integrated on the EOSC catalogue and made broadly accessible to various organizations We welcome your feedback on the draft of the “Functional Specifications” document which will be released shortly after this event The Buyers group will co-design and co-develop with you a test plan - based on the outcome of the Design Phase, the Functional Specifications and the Deployment Scenarios The test assessment will be a deciding factor to qualify solutions to the subsequent phases The tests will focus on basic functionality capabilities during the prototype phase and performance, efficiency, and scalability during the pilot phase
Editor's Notes
- #4: So enough with who I am let’s move on to the next important question. What is CERN? Do you guys know what CERN is? Do you know what the LHC is? No worries, if you don’t know, I am going to explain everything in the next slide.
- #7: So enough with who I am let’s move on to the next important question. What is CERN? Do you guys know what CERN is? Do you know what the LHC is? No worries, if you don’t know, I am going to explain everything in the next slide.
- #13: So enough with who I am let’s move on to the next important question. What is CERN? Do you guys know what CERN is? Do you know what the LHC is? No worries, if you don’t know, I am going to explain everything in the next slide.
- #18: So enough with who I am let’s move on to the next important question. What is CERN? Do you guys know what CERN is? Do you know what the LHC is? No worries, if you don’t know, I am going to explain everything in the next slide.