XDC demo: CTA
- 1. eXtreme DataCloud is co-funded by the Horizon2020 Framework Program – Grant Agreement 777367 Copyright © Members of the XDC Collaboration, 2017-2020 Data Management for extreme scale computing CTA USE CASE Review Gillardo Frédéric gillardo@lapp.in2p3.fr 11/04/2019 1
- 2. CTA Data Workflow Reconst Calib DL0/Bulk Archive x Scientist Interface DL3 / High level data Canary Island , Online Process Chile , Online Process
- 3. eXtreme DataCloud 3 Use Case Goals • Objectives: Archive PBs of data using an ingest method. Index files using metadata contained in the header files Query files using metadata parameters • XDC Services Requirements: • OneData: • Onedata • Metadata management • QoS (policy definition) 3 Ingest DL0 Preprocessing Metadata database Query (Metadata) List of filenames files Data management Archive Producer Consumer Retrieve (fileName)
- 4. User Stories As an “Archive Manager”, I can configure the system to extract metadata during the preprocessing operation. Related with : As an “Archive Manager”, I can define rules based on metadata to archive files on tape only or on disc & on tape, at ingest time. As an “Archive Manager”, I can define rules based on metadata to duplicate archive files on tape on low latency storage to be soon quickly retrieved. As an “Archive Manager”, I can define rules based on metadata to create replicas on a specific data center, at ingest time. As an “Archive Manager”, I can define rules based on metadata to prevent deletion of files As an “Archive Manager”, I can delete files based on metadata values As a “Archive User”, I can make a query based on metadata parameters to get a list of logical files names 4
- 5. OneData installation at LAPP and CCIN2P3 POSIX 40TB LAPP- PROVIDER XDC-DEMO1 S3 4TB CC-PROVIDER-02 POSIX 6TB 5
- 6. Use Case Architecture eXtreme DataCloud 6 Query by Metadata Restquery Query by LFMFile System preprocessing CTA HDF5 Extraction Metada ingestFile Generator
- 7. Conclusion What has been demoed: • Metadata index creation using REST API • Deploy the « preprocessing’s service » using docker compose • Ingest a HDF5 files • Query File ID using HDF5 header • Retrieve file using CDMI interface and File ID eXtreme DataCloud 7