Introducing the HACC Simulation Data Portal
1 like412 views
The document discusses the HACC simulation data portal introduced at the Globus World 2019 conference, focusing on its role in advancing cosmological research by providing access to large-scale simulations needed to tackle fundamental questions about the universe. It outlines the required infrastructure, including storage and analysis capabilities, facilitated through platforms like Globus and HPC allocations. Additionally, it emphasizes the importance of public access to simulation data to improve the outcomes of cosmological research collaborations.
Introducing the HACC Simulation Data Portal
- 1. Introduction to the HACC Simulation Data Portal Globus World 2019; Chicago, May 1, 2019 Katrin Heitmann (Argonne National Laboratory) Based on: arXiv:1904.11966
- 2. Introduction ! In cosmology we study the origin, evolution, and make-up of the Universe ! Many unsolved questions: ○ What is the nature of dark energy and dark matter, making up 95% of the energy-matter budget of our Universe? ○ What is the mass of the lightest particle in the Universe, the neutrino? ○ How can we learn more about the very first moments of the Universe? ! Upcoming cosmological surveys try to answer these questions and rely on detailed, complex simulations ○ Simulations are carried out and analyzed on the largest supercomputers available world-wide ○ Cosmological simulations generate large amounts of data (PBs) to capture the evolution of the Universe faithfully ○ Given the resources required for these simulations, it is crucial to share them with the community to enable the best possible science outcome HACC/Galacticus/GalSim Hubble Ultra Deep Field NASA
- 3. What is needed ... A large-scale effort that provides easy access to a range of simulation products to the world’s cosmologists as well as analysis capabilities to established survey collaborations
- 4. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis User community via web and community-specific clients simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer Datasets Collaboration-installed Web/ Data Interfaces • LSST DM Butler • Jupyter • PDACS (Galaxy) • DESCQA • Visualization • Databases • Globus • Workflows Globus Online Petrel O(1 PB, 100TB to start) • Portal • Globus ALCF-hosted Collaboration-controlled Resources Physical/Virtual Machine(s) Phoenix In collaboration with Tom Uram, Mike Papka, Ian Foster
- 5. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer
- 6. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer Temporary storage, expires with allocation, only collaborators on the project have direct access
- 7. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer Globus Online Petrel O(1 PB, 100TB to start) Datasets
- 8. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer Globus Online Petrel O(1 PB, 100TB to start) Datasets • Portal • Globus User community via web and community-specific clients
- 9. Storage O( 50PB total) Simulation (HPC allocations, e.g., INCITE, ALCC) Analysis User community via web and community-specific clients simulation job description analysis job description Public access to cosmological data and computational support for collaborations CooleyTheta (10PF) job submission/adaptation layer Datasets Collaboration-installed Web/ Data Interfaces • LSST DM Butler • Jupyter • PDACS (Galaxy) • DESCQA • Visualization • Databases • Globus • Workflows Globus Online Petrel O(1 PB, 100TB to start) • Portal • Globus ALCF-hosted Collaboration-controlled Resources Physical/Virtual Machine(s) Phoenix In collaboration with Tom Uram, Mike Papka, Ian Foster
- 10. What exists ... • Petrel and Phoenix • Simulations • First version of web portal using Globus
- 11. ! Petrel: Data Management and Sharing Pilot, hosted at Argonne ! 1.7PB parallel filesystem ! Embedded in Argonne’s 100+Gbps network fabric to allow high-speed data transfers ! Web and API access via Globus ! Federated login ! Self-managed by PIs ! https://press3.mcs.anl.gov/petrel/
- 12. ! Webportal for easy access to simulations ! Currently: ~ 82.5 TB in our project covering three simulation projects ! Step 0: Register with Globus ! Step 1: Select simulation project ! Step 2: Select data products, information about data size available ! Step 3: Transfer with Globus to endpoint of your choice
- 13. ! Webportal for easy access to simulations ! Currently: ~ 82.5 TB in our project covering three simulation projects ! Step 0: Register with Globus ! Step 1: Select simulation project ! Step 2: Select data products, information about data size available ! Step 3: Transfer with Globus to endpoint of your choice
- 14. ! Webportal for easy access to simulations ! Currently: ~ 82.5 TB in our project covering three simulation projects ! Step 0: Register with Globus ! Step 1: Select simulation project ! Step 2: Select data products, information about data size available ! Step 3: Transfer with Globus to endpoint of your choice
- 15. “The purpose of computing is insight not numbers” - Richard Hamming