Pathways for EOSC-hub and MaX collaboration
- 1. Pathways for EOSC-hub and MaX collaboration A platform for reproducible science with full provenance Giovanni Pizzi giovanni.pizzi@epfl.ch Theory and Simulation of Materials, EPFL Lausanne
- 2. The MaX European Centre of Excellence Materials design at the Exascale Partners 14 partners with unique expertise in: • Materials Science (CEA, CNR, EPFL, ETHZ, FZ Jülich, ICN2, SISSA) • Software development and code validation (AiiDA, BigDFT, CP2K, Fleur, Quantum ESPRESSO, SIESTA, Yambo) • HPC (5 TIER-0 HPC Centres: BSC, CEA, CINECA, CSCS, JSC) • Technology (ARM, E4) • Communication & Outreach (ICTP, Psi-K, CECAM, UGent, TRUST-IT) Key-Actions • Restructure MaX codes towards exascale and extreme scaling performance • Co-design activities for HPC architectures • Develop broader ecosystem enabling the convergence of HPC, HTC and HPDA (WP5) • Widen the access to codes and foster transfer of know-how to user communities Domains of interest • High-Performance Computing (HPC) • High-Throughput Computing (HTC) • High-Performance Data Analytics (HPDA) http://www.max-centre.eu
- 3. Leverage supercomputers to compute and predict materials’properties In our context, high-throughput means “10’000+ HPC simulations per day” Scientific aim: Compute properties for all of them (and even new, invented ones) and discover novel functional materials
- 4. How to manage data, simulations and their provenance? IS THERE A REPRODUCIBILITY CRISIS? Nature 533 452–454 (2016)
- 5. How to manage data, simulations and their provenance? IS THERE A REPRODUCIBILITY CRISIS? Nature 533 452–454 (2016) We need a tool to help us automate research, organise it, store provenance guaranteeing reproducibility, then analyze results, and finally share them and collaborate
- 6. MaX Centre of Excellence: AiiDA and Materials Cloud The challenges we address • Allow high-throughput research (10’000+ simulations/day); automate simulations, automatically track provenance • Share simulations according to the FAIR principles and beyond, guaranteeing reproducibility • Encode scientists’knowledge in automated workflows (scientific know-how, numerical parameters, choice of data to preserve and share) • Provide advanced data analytics tools • Provide HPC resources as services in an easy, accessible way to anybody, via simple interfaces
- 7. MaX Centre of Excellence: AiiDA and Materials Cloud Data provenance: Directed Acyclic Graphs G. Pizzi et al., Comp. Mat. Sci. 111, 218-230 (2016) http://www.aiida.net MIT license (open source) Developed since 2013 Used in production from many scientific research projects
- 8. “Simple”graphs of workflows for a single material KpointsData (216283) (372 kpts) MatdynCalculation (216285) FINISHED kpoints BandsData (216385) 'Phonon bands' output_phonon_bands FolderData (216383) retrieved RemoteData (216320) remote_folder ParameterData (216384) output_parameters Code (209961) 'matdyn-5.1.2-module' code ForceconstantsData (216273) parent_calc_folder ParameterData (216284) parameters ParameterData (216282) settings Q2rCalculation (215983) FINISHED force_constants FolderData (215981) parent_calc_folder Code (209960) 'q2r-5.1.2-module' code ParameterData (215982) parameters InlineCalculation (215980) recollect_qpoints_inline() retrieved FolderData (215920) retrieved_6 FolderData (215976) retrieved_7 FolderData (215978) retrieved_4 FolderData (215797) retrieved_5 FolderData (215848) retrieved_2 FolderData (215924) retrieved_3 FolderData (215831) retrieved_0 FolderData (215912) retrieved_1 FolderData (215895) retrieved_8 FolderData (215829) retrieved_9 FolderData (215252) initial_folder InlineCalculation (215309) distribute_qpoints_inline() retrieved PhCalculation (215430) FINISHED retrieved PhCalculation (215433) FINISHED retrieved PhCalculation (215526) FINISHED retrieved PhCalculation (215427) FINISHED retrieved PhCalculation (215520) FINISHED retrieved PhCalculation (215523) FINISHED retrieved PhCalculation (215517) FINISHED retrieved PhCalculation (215514) FINISHED retrieved PhCalculation (215436) FINISHED retrieved PhCalculation (215529) FINISHED retrieved PhCalculation (215224) FINISHED retrieved ParameterData (215428) settings RemoteData (214841) parent_calc_folder parent_calc_folderparent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder parent_calc_folder Code (209959) 'ph-5.1.2-module' code codecode code code code code code code code code ParameterData (215429) parameters KpointsData (215312) (1 kpts) qpoints ParameterData (215431) settings ParameterData (215432) parameters KpointsData (215313) (1 kpts) qpoints ParameterData (215524) settings ParameterData (215525) parameters KpointsData (215310) (1 kpts) qpoints KpointsData (215311) (1 kpts) qpoints ParameterData (215426) parameters ParameterData (215425) settings ParameterData (215518) settings ParameterData (215519) parameters KpointsData (215316) (1 kpts) qpoints ParameterData (215521) settings ParameterData (215522) parameters KpointsData (215317) (1 kpts) qpoints ParameterData (215516) parameters ParameterData (215515) settings KpointsData (215314) (1 kpts) qpoints ParameterData (215512) settings ParameterData (215513) parameters KpointsData (215315) (1 kpts) qpoints KpointsData (215318) (1 kpts) qpoints ParameterData (215435) parameters ParameterData (215434) settings ParameterData (215527) settings ParameterData (215528) parameters KpointsData (215319) (1 kpts) qpoints ParameterData (215223) parameters ParameterData (215222) settings KpointsData (214808) 4x4x4 (+0.0,0.0,0.0) qpoints PwCalculation (214830) scf FINISHED remote_folder qpoint_6 qpoint_7qpoint_4 qpoint_5 qpoint_2 qpoint_3 qpoint_0 qpoint_1 qpoint_8 qpoint_9 Code (139993) 'pw-5.1.2-module' code UpfData (1658) pseudo_O UpfData (1660) pseudo_Ti ParameterData (214828) parameters ParameterData (214829) settings KpointsData (214807) 6x6x6 (+0.0,0.0,0.0) kpoints UpfData (1905) pseudo_Ba StructureData (214814) BaO3Ti structure structure Phonon dispersion (atom oscillations around equilibrium positions: thermal transport, electronic mobility, …) Molecular dynamics of Lithium in a solid electrolyte (Discover novel, safe and efficient electrolytes for Li- batteries) Elastic constants (response of materials to stresses and deformations)
- 9. Graphical representation of an AiiDA database of calculations and workflows of DFT band structure and Wannier functions
- 10. MaX Centre of Excellence: AiiDA and Materials Cloud An ecosystem of plugins https://aiidateam.github.io/aiida-registry/ 34 plugin entries for Materials Science, supporting 65 codes, 62 workflows, …
- 11. MaX Centre of Excellence: AiiDA and Materials Cloud Open Science Platform: AiiDA + Materials Cloud https://www.materialscloud.org Online since February 2018 Cloud dissemination platform for FAIR data sharing and more (cloud simulation and data generation platform) + + …
- 12. MaX Centre of Excellence: AiiDA and Materials Cloud Open and FAIR data sharing: Archive, Discover, Explore Direct links to Discover & Explore DOIs assigned FAIRsharing.org re3data.org + Recommended data repository by Nature’s journal Scientific Data
- 13. MaX Centre of Excellence: AiiDA and Materials Cloud DISCOVER (CURATED DATA) & EXPLORE (RAW DATA) UUID links to jump to the provenance graph in the EXPLORE section DISCOVER EXPLORE Browse the full AiiDA provenance graph (inputs, outputs, …) at any level
- 14. WORK: AiiDA Lab (submission) • Our cloud data generation platform and data analysis platform • Based on AiiDA + Jupyter + App Mode
- 15. WORK: AiiDA Lab (submission) • Our cloud data generation platform and data analysis platform • Based on AiiDA + Jupyter + App Mode Graph generated by the previous run
- 16. MaX Centre of Excellence: AiiDA and Materials Cloud WORK: AiiDA Lab
- 17. MaX Centre of Excellence: AiiDA and Materials Cloud Possible integration/collaboration points • Integration plans • AiiDA Lab • Deployment with kubernetes for autoscaling using EOSC services • Integration fo Authentication and Authorization with B2ACCESS and/or EGI Check-In • Registration of AiiDA Lab as a service on EOSC • Development and deployment of“turn-key”workflows for the materials science community as the“services” • Archive • Migration of the Materials Cloud Archive to Invenio v3 or EUDAT’s B2SHARE, still deployed on our premises • Integration in EUDAT’s B2FIND
- 18. MaX Centre of Excellence: AiiDA and Materials Cloud Possible integration/collaboration points • Integration plans • AiiDA Lab • Deployment with kubernetes for autoscaling using EOSC services • Integration fo Authentication and Authorization with B2ACCESS and/or EGI Check-In • Registration of AiiDA Lab as a service on EOSC • Development and deployment of“turn-key”workflows for the materials science community as the“services” • Archive • Migration of the Materials Cloud Archive to Invenio v3 or EUDAT’s B2SHARE, still deployed on our premises • Integration in EUDAT’s B2FIND EGICheck-in: scheduled W orkin progress Scheduled W orkin progress Scheduled B2ACCESS: done
- 19. MaX Centre of Excellence: AiiDA and Materials Cloud Acknowledgements: The AiiDA and Materials Cloud teams Giovanni Pizzi (EPFL) Boris Kozinsky (BOSCH) Martin Uhrin (EPFL) Spyros Zoupanos (EPFL) Nicola Marzari (EPFL) Snehal P. Kumbhar (EPFL) Leonid Kahle (EPFL) Sebastiaan P. Huber (EPFL) Marco Borelli (EPFL) Elsa Passaro (EPFL) Thomas Schulthess (ETHZ,CSCS) Leopold Talirz (EPFL) Joost VandeVondele (ETHZ,CSCS) Aliaksandr Yakutovich (EPFL) Contributors for the 25+ plugins Contributors to aiida_core and former team members — Valentin Bersier, Jocelyn Boullier, Jens Broeder, Andrea Cepellotti, Fernando Gargiulo, Dominik Gresch, Rico Häuselmann, Eric Hontz, Christoph Koch, Espen Flage-Larsen, Andrius Merkys, Nicolas Mounet, Tiziano Müller, Riccardo Sabatini, Ole Schütt, Phillippe Schwaller Berend Smit (EPFL) Casper W. Andersen (EPFL)
- 20. MaX Centre of Excellence: AiiDA and Materials Cloud Acknowledgements and funding H2020 Centre of Excellence“MaX” SNSF NCCR“MARVEL” Swissuniversities P-5“Materials Cloud” Discovery of new materials via simulations and dissemination of curated data Scaling towards exascale machines and high-throughput efficiency Scaling the web platform, extending to more disciplines Moreover: H2020 Marketplace Providing data and simulation services in a EU Marketplace platform for industry H2020 Intersect Develop AiiDA workflows to compute transport properties of materials
- 21. MaX Centre of Excellence: AiiDA and Materials Cloud Conclusions • Open Science Platform for computational materials science: the MaX tool to converge HPC, HTC and HPDA • Guarantee reproducibility + FAIR sharing of data with their provenance • Define and run scientific workflows for materials and provide data analytics tools • Provide HPC workflows as a service + MaX can bring services to EOSC to streamline access to HPC
- 22. Contacts Materials Cloud: http://www.materialscloud.org - AiiDA Lab: http://aiidalab.materialscloud.org - Archive: http://archive.materialscloud.org @aiidateamhttps://www.facebook.com/aiidateam Website: http://www.aiida.net Docs: http://aiida-core.readthedocs.io Git repo: https://github.com/aiidateam/aiida_core/ Plugin registry: http://aiidateam.github.io/aiida-registry Quantum Mobile: http://www.materialscloud.org/work/ quantum-mobile Website: http://www.max-centre.eu @max_center2https://www.linkedin.com/company/max-center