Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities
2 likes1,335 views
The document discusses the EGI Federated Cloud, which supports various user communities through a federated system of cloud resources across multiple countries. It highlights the growth in both resource providers and users, emphasizing open standards and technology independence to enhance collaboration and service integration. The infrastructure aims to meet expanding research needs while preventing vendor lock-in and promoting a global cloud marketplace.
Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities
- 1. EGI-InSPIRE RI-261323 www.egi.eu EGI-InSPIRE Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities David Wallom University of Oxford Chair, EGI Federated Cloud
- 2. EGI-InSPIRE RI-261323 www.egi.eu Overview • Why • What • Who
- 3. EGI-InSPIRE RI-261323 www.egi.eu Growth of Providers Grid Limited academic resource providers Cloud Diversity and scale of new resource providers
- 4. EGI-InSPIRE RI-261323 www.egi.eu Growth of Userbase 20k researchers 2M researchers! Few related use cases Many diverse use cases
- 5. EGI-InSPIRE RI-261323 www.egi.eu Hardware Hardware Hardware Hardware Hardware Cloud Management Cloud Management Cloud Management Cloud Management Cloud Management User Communities User Communities User Communities Federated interfaces Federated services • Open Standards •Technology independant •Provider agnosticism •Production integration and reuse Principles of Federation
- 6. EGI-InSPIRE RI-261323 www.egi.eu EGI Cloud Infrastructure 6 EGI Core Platform Federated AAI Service Registry Monitoring Accounting EGI Cloud Infrastructure Platform Instance Mgmt Information Discovery Storage Management Cloud Management Stacks (OpenStack, OpenNebula, Synnefo, …) Help and Support Security Co- ordination Training and Outreach EGI Collaboration Tools EGI Application DB Image Repository EGI Cloud Service Marketplace Sustainable Business Models User Community GSI GLUE2 OCCI CDMI SAM UR OVF Cloudinit
- 7. EGI-InSPIRE RI-261323 www.egi.eu Compatibility for technology providers Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI OpenStack Yes Yes* Yes Yes Yes Yes OpenNebula Yes Yes* Yes Yes Yes Yes Synnefo Yes Yes Yes - Yes - Cloudstack - - - - Yes - Emotive Yes Yes* - N/A N/A Yes SlipStream Yes N/A N/A Yes Yes Yes* Perun Yes Yes N/A N/A N/A N/A 7 Example documentation for a CMF:- https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack
- 8. EGI-InSPIRE RI-261323 www.egi.eu Geographical dispersion • 12 countries provide 19 certified resources – Czech Republic, Germany, Greece, Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom • 2 countries currently integrating – France, Finland, Portugal, Croatia • 5 countries interested – Bulgaria, Israel*, The Netherlands, Switzerland • Worldwide interest – South Africa* (SAGrid) – South Korea* (KISTI) – United States* (NIST, NSF Centres) * Not shown on map
- 9. EGI-InSPIRE RI-261323 www.egi.eu Utilisation 9 217352 VMs launched across 25 clouds
- 10. EGI-InSPIRE RI-261323 www.egi.eu Federated Cloud Services Federated IaaS and STaaS Cloud EYR TC monthly meeting 10 Tier 1: Reliable Infrastructure Cloud Tier 4: Zero ICT Infrastructures Tier 3: Platform as a Service Tier 2: General-purpose platform services PaaS PaaS DB aaS Hadoop aaS VRE Secure storage Key Mgmt Encryption ACL mgmt Virtual eLaboratory 17 Feb 2014
- 11. EGI-InSPIRE RI-261323 www.egi.eu AppDB – Virtual Appliance catalogue 11
- 12. EGI-InSPIRE RI-261323 www.egi.eu EGI FedCloud Communities 9/2014 • Ecology – BioVeL: Biodiversity Virtual e-Laboratory • Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology • Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb) • Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for volcano and earthquakes monitoring (Collaboration with Helix Nebula) • Software Engineering – SCI-BUS: simulated environments for portal testing • Software Engineering – DIRAC: deploying ready-to-use distributed computing systems • Software Engineering – Catania Science Gateway Framework • Musicology – Peachnote: dynamic analysis of musical scores • Earth Observation – ENVRI: Common Operations of Environmental Research infrastructures (collaboration with EISCAT3D) • Geology – VERCE: Virtual Earthquake and seismology Research • Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem Research • High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users 12
- 13. EGI-InSPIRE RI-261323 www.egi.eu New EGI FedCloud Communities since launch • Education – Cranfield University distributed systems course • Cultural Heritage – DCH-RP management of preservation services in the cloud • Hydrological Modelling – Running Hydrological models to support real time analysis • Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud environment • Systems implementations – deployment of FTK developed tools and services and data preservation • Internet of Things – Smart Grid systems investigation • Software Development – deployment of research PaaS • RNA Sequencing – deployment of analysis engines in the cloud • Physiological Modelling – Calibration, scenario mapping and development More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users 13
- 14. EGI-InSPIRE RI-261323 www.egi.eu WeNMR e-Infrastructure (VirtualCing) EGI Federated Cloud Infrastructure … NMR Protein DB ToPoS Virtual CING Virtual CING Virtual Cing Virtual CING Virtual CING Virtual Cing Virtual CING Virtual CING Virtual Cing OCCI client Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR, has been deployed via the EGI Federated Cloud 14
- 15. EGI-InSPIRE RI-261323 www.egi.eu ESA HN Flagship on the EGI Federated Cloud The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federated Cloud: 1.Use SlipStream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET) 2.Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring 15 Automatic Cluster Deployment EGI Federated Cloud Computing Cluster Cluster Head Working Nodes OCCI Globus Job submission BLUE BOX
- 16. EGI-InSPIRE RI-261323 www.egi.eu 16 EISCAT 3D PoC EISCAT_3D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space. The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data.
- 17. EGI-InSPIRE RI-261323 www.egi.eu Peachnote 10/9/2014 17 Peachnote is a music score search engine and analysis platform. Hundreds of thousands of music scores are being digitized by libraries all over the world. In contrast to books, they generally remain inaccessible for content-based retrieval and algorithmic analysis. There is no analogue to Google Books for music scores, and no large corpora exists that can empower advanced analysis on music scores. Peachnote want to help change that providing visitors and researchers access to a massive amount of symbolic music data. EGI Federated Cloud OMR Worker PDF Splitter OMR Feeder OMR Worker …
- 18. EGI-InSPIRE RI-261323 www.egi.eu OpenModeller on the Biovel Portal 10/9/2014 18 The Ecological Niche Modeling (ENM) Workflow takes as input a file containing species occurrence points to create a model with the openModeller Web Service. • The EUBrazilOpenBio ENM service is exposed through an extended openModeller Web Service interface • Multi-staging and multi-parametric oM experiments are implemented through COMPSs that dynamically creates the virtual resources to execute the operations. • An OCCI connector is used for the VMs management while data management supports CDMI endpoints. ENM Service (OMWS2) VENUS-C Cloud Middleware COMPSs Workflow Orchestrator OCCI CDMI EGI Federated Cloud Service available at https://portal.biovel.eu/
- 19. EGI-InSPIRE RI-261323 www.egi.eu 10/9/2014 20 Running GNU Octave and R on the CHAIN-REDS Science Gateway 20 CHAIN-REDS Science Gateway - CHAIN-REDS RI-306819
- 20. EGI-InSPIRE RI-261323 www.egi.eu Strengthening the underpinning platform Technology Evolution • Broader support for open standards in Cloud management frameworks – Utilisation of rOCCI for interfaces to commercial cloud frameworks • New feature additions to foundational tools depending on requests – Accounting, monitoring, service discovery, Application DB • Innovative deployments of opensource PaaS • Large improvements on documentation for all 3 stakeholder groups 22 Value added services for User Communities
- 21. EGI-InSPIRE RI-261323 www.egi.eu Conclusions The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud Services to researchers in Europe and worldwide • A single cloud system able to – Scale to user needs – Integrate multiple different providers to give resilience – Prevent vendor lock-in through open standards – Enable resource provision targeted towards the research community • Paving the way for a global federated cloud marketplace – Established best practice – Illustrating leadership – Open standards, open technology – Open membership, open processes – Open competition Current capability – ~6,000 cores, 300 TB storage Promised through EGI-Inspire – 18,000 cores, 6000 TB storage 2020 Target – 1M cores, 1EB storage 23
- 22. Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh Nguyen, Björn Hagemeier, Boris Parak, Boro Jakimovski, Cal Loomis, Christos Loverdos, Daniele Cesini, Daniele Lezzi, David Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos, Ivan Diaz, Jan Meizner, John Gordon, Kostas Koumantaros, Malgorzata Krakowian, Marios Chatziangelou, Marco Verlato, Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge, Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi, Soonwook Hwang, Steven Newhouse, Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah and all the other members of the EGI federated Cloud Task Force, activity, workpackage
- 23. EGI-InSPIRE RI-261323 www.egi.eu Thank you! Questions? 25 Task Force resources Mailing List: fedcloud-tf@mailman.egi.eu Wiki site: http://go.egi.eu/tf-fedclouds GitHub: https://github.com/EGI-FCTF Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112