EGI Cloud Compute service for EOSC-hub
- 1. eosc-hub.eu @EOSC_eu EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536. Run virtual machines on demand with complete control over computing resources EGI Cloud Compute
- 2. 2 Motivation and driving consideration about the service Service architecture and interfaces: overview - How the user can access the service E.g.: REST, GUI, CLIs, etc. - Service options and attributes Acceptable Usage Policy (AUP) Access policy and business model Use cases Documentation/tutorial/information 11/27/2018 Content
- 3. 3 EGI Cloud Compute provides users with a distributed computing service to deploy and scale virtual machines on-demand. It offers access to API- controlled computational resources in a secure and isolated environment without the overhead of managing physical servers. Cloud Compute offers the possibility to select pre-configured virtual appliances (e.g. CPU, memory, disk, operating system or software) from a catalogue replicated across all EGI cloud providers. With Cloud Compute you can: - Execute compute- and data-intensive workloads (both batch and interactive) - Host long-running services (e.g. web servers, databases or applications servers) - Create disposable testing and development environments on virtual machines and scale your infrastructure needs - Select virtual machine configurations (CPU, memory, disk) and application environments to fit your requirements - Manage your resources in a flexible way with integrated monitoring and accounting capabilities 11/27/2018 Motivation
- 4. Cloud Management Framework IaaS API Cloud Management Framework IaaS API Direct API Access 411/27/2018 Service architecture and interfaces EGI Federation Services: Accounting, Monitoring, Conf. DB, Info Discovery, AppDB EGI Resource Center A AppDB VMOpsGUI Access IaaS Federated Access Tools Federated Access EGI Resource Center B
- 5. 5 GUI access: - AppDB VMOps https://dashboard.appdb.egi.eu/vmops API/CLI access: - Discovery: AppDB IS API (REST and GraphQL) https://wiki.egi.eu/wiki/Federated_Cloud_Discovery#AppDB - IaaS Federated Access Tools: https://wiki.egi.eu/wiki/Federated_Cloud_IaaS_Orchestration - Direct IaaS access, several APIs depending on the provider: https://wiki.egi.eu/wiki/Federated_Cloud_APIs_and_SDKs 11/27/2018 Service access
- 6. 6 Service Options: - General purpose instance: base performance instance type with up to 8 cores and up to 4 GB RAM per core - Compute-intensive instance: optimised instance for cpu-bound tasks. Up to 64 cores (non-overcommitted) and possible low- latency network - High-memory instance: optimised instances for tasks with big memory requirements. Up 120GB RAM per core. Service attributes: - Associated storage to each VM (GB) - Access type: opportunistic (non guaranteed resources) or reserved - Start of service (date) and duration (days) - API of the provider (OpenStack, OCCI) 11/27/2018 Service options and attributes
- 7. 7 EGI AuP: - https://documents.egi.eu/public/ShowDocument?docid=2623 Further VO-level AuPs may also apply 11/27/2018 Acceptable Usage Policy
- 8. 8 Access policies: - Policy-based - Wide-access - Market-drive Payment model: - Sponsored (free for a certain quota) - Wide-access - Market-driven 11/27/2018 Access policies and Funding models
- 9. Access to the EGI Cloud Virtual Organisations = Resource pools VO 1 (cloud a, b, c) VO 2 (cloud b, c, d, e, f) 1. Community-specific VOs – e.g. CHIPSTER, Highthroughtputseq, EISCAT, etc. (SLA, OLAs) 2. Training VO = training.egi.eu To be used today 3. Generic VOs – e.g. fedcloud.egi.eu Incubator for new users (recommended for follow up) Browse VOs at http://operations-portal.egi.eu/vo/search (both grid and cloud) c e f b a d
- 10. 10 Compute and data intensive workloads - Batch and interactive (e.g. Jupyter notebooks) with scalable and customized environments Service Hosting - Long-running services (e.g. web server, database, application server) Datasets repository - Store and manage large datasets (in a storage volume) Disposable and testing environments - Host training environments, test applications 11/27/2018 Use cases - Scenarios
- 11. Appliances Marketplace (AppDB) Clouds of the Chipster Virtual Organisation Virtual Appliances of the Chipster VO Chipster VM VM VM VM VM Cloud calls (CMD/API) VM Block storage Chipster portal Push VM image Prepare, register VM image Chipster developer (CSC, Finland) Bioinformaticians Analysis software contains over 300 analysis tools for NGS, microarray, proteomics and sequence data. VM Block storage Read more... Cache Cache Cache Cache Use case – Sharing community applications: Chipster
- 12. 11/27/2018 Use case - Scalable bioinformatics web- servers powered by cloud computing: NBIS A distributed Swedish research infrastructure providing bioinformatics support to life science researchers. Query Frontend via web-page or API Clouds of the NBIS Virtual Organisation VM VM VM shared storage Read more… Scalable handling of jobs using VMs from EGI Cloud Compute NBIS Servers powered by EGI: Pcons.net TOPCONS SCAMPI2 ProQ3 PconsC2/PconsC3 BOCTOPUS2
- 13. Use case – EMSO Data Management platform Data access DMP tools DATA MANAGEMENT PLATFORM Asynchreal-timeingestion. Data ingestion ENVRI Reference Model v2.0 Collection and Analysis of Ocean Big Data: EGI Cloud hosts EMSO Data Management Platform services providing acquisition, processing, publishing and curation capabilities of all data collected by deep sea observatories Read more… EMSO DMP is powered by Apache Spark and Apache Zeppelin running on EGI Cloud.
- 14. Use case – GeoHazards TEP Read more… Cloud platform developed and operated by Terradue. It provides a rich set of ready to use EO Data processing services for geohazards analysis and monitoring. EGI Cloud Compute contributes resources to run near real-time processing of satellite data acquisitions. www.terradue.com User Communities 4 Image taken from slides at https://indico.egi.eu/indico/event/3455/session/13/contribution/84
- 15. 15 User documentation: - https://wiki.egi.eu/wiki/Federated_Cloud_user_support Providers documentation: - http://egi-federated-cloud-integration.readthedocs.io/ 11/27/2018 Documentation
Editor's Notes
- #14: Apache Spark: fast and general engine for large-scale data processing . It is able to execute both batch and streaming data processing. Apache Zeppelin: data analytics tool