EGI Cloud Container Compute Service

www.egi.eu
@EGI_eInfra
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme
EGI: Advanced Computing for Research
In support of biomedical research
EGI Cloud Container Compute
EGI Foundation
Enol Fernándes, Björn Backeberg & Yin Chen

@EGI_eInfrawww.egi.eu 29 March 2019 2
• About EGI and the EGI Federation
• EGI Services
• The EGI Federated Cloud
§ Cloud Compute
§ Cloud Container Compute
• Examples of community use cases
EGI Cloud Container Compute Service
Outline

About EGI and the EGI Federation

Mission
Create and deliver open solutions for science and research infrastructures by
federating digital capabilities, resources and expertise between communities and
across national boundaries.
Vision
Researchers from all disciplines have easy, integrated and open access to the
advanced scientific computing capabilities, resources and expertise needed to
collaborate and to carry out data/compute intensive science and innovation.
Our objective
Mission statement and vision

23 Countries
1 EIRO: CERN
www.egi.eu/about/egi-foundation/
The EGI Federation
EGI resources are provided by 100s of
computing and data centres (mostly in
Europe) offering computing and storage
resources to researchers.

A wide offer of services
Supporting Research and Innovation

• High Throughput Data Analysis
• Federated Cloud: Easy access to EU IaaS and PaaS (support for Docker, Kubernetes,
Jupyter Notebook Service)
• Federated Open Data Processing: Integrate a generic easy-to-use platform that allows
users to share, discover, and process open data. The platform also provides interfaces to
existing storage solutions
• Federated operations
• Community-driven Innovation and Support: Co-create new solutions, and support
researchers
• Policy advice: Research on policies and prepare input and advice that help policy makers
shape future policies for more effective digital and open science
(Some of) our offerings

Our external Service Portfolio

Our internal Service Portfolio
Delivered to the EGI Federation members facilitating collaboration

Commitment to excellence
CERTIFICATE
The Certification Body
of TÜV SÜD Management Service GmbH
certifies that
EGI Foundation / EGI.eu
Science Park 140
1098 XG Amsterdam
Netherlands
has established and applies
a Quality Management System for
A management system to deliver all
services covered by the service catalogue.
An audit was performed, Report No. 707067543.
Proof has been furnished that the requirements
according to
ISO 9001:2015
are fulfilled.
The certificate is valid from 2018-02-19 until 2020-03-06.
Certificate Registration No.: 12 100 53643 TMS.
Product Compliance Management
Munich, 2018-02-19
CERTIFICATE
The Certification Body
of TÜV SÜD Management Service GmbH
certifies that
EGI Foundation / EGI.eu
Science Park 140
1098 XG Amsterdam
Netherlands
has established and applies
a Management System for IT-Services.
A management system to deliver all
IT services covered by the service catalogue.
An audit was performed, Report No. 707067543.
Proof has been furnished that the requirements
according to
ISO/IEC 20000-1:2011
are fulfilled.
The certificate is valid from 2018-02-19 until 2020-03-06.
Certificate Registration No.: 12 410 53644 TMS.
Product Compliance Management
Munich, 2018-02-19
ISO 9001:2015
outlines the
requirements an
organization
must maintain in
their quality
system.
ISO/IEC 20000-
1:2011 outlines
requirements for
design, transition,
delivery and
improvement of IT
services for both
the customer and
the service
provider.

Services offered to SMEs and industry
Access
• Test workflows
• Pilot applications &
models
• Develop advanced ICT
product / service
Enhance visibility /
networking in EGI’s ecosystem
• EGI Events
• EGI publications
• EGI international network
Market
Re-use
Capitalise on open
• Research data
• Tools / softwares
• Applications
Co-design
Team up with experts to
• Deploy your application on
the distributed
infrastructure
• Co-develop new products
and services.

The EGI Federated Cloud
The Cloud Compute and
Cloud Container Compute services

Technology agnostic:
• OpenStack,
• OpenNebula
• Synnefo
Multi-cloud IaaS with Single Sign-On via Check-in
Additional features
• Single-Sign On via Check-in
• Appliance Library
• Resource discovery
• Single GUI dashboard
• Computation near data
Cloud compute Cloud container
compute
Online
storage
AoD Training
infrastructure
Run Virtual Machines on demand similar to AWS EC2/EBS or GCP Compute Engine

The EGI Federated Cloud is a IaaS-type cloud, comprising
academic private cloud and virtualised resources built
around open standards.21 cloud providers:

Architecture
EGI Federation services:
Accounting, Monitoring, Configuration Database, Information Discovery, VM
Marketplace
EGI
Check-in
IaaS Federated Access Tools
Community PlatformsAppDB VMOps
Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
The EGI Federated Cloud is not just AppDB
Providers have their APIs
that can be used with
EGI Check-in accounts,
opening the door to
automation of cloud-
native applications.
layer helps users of the cloud to
deal with the heterogeneity in
the IaaS API and EGI Federation
services
EGI Federated Cloud no longer mandates a
single API for every provider. OCCI still widely
supported but sites are moving to native APIs
(mainly OpenStack!)

Manage VMs via AppDB VMOps
Single
dashboard for
all providers
Wizard-like
creation of
VMs
Complete
Check-in
integration

Manage VMs via AppDB VMOps
Individual
management
of VMs
Topologies
are a set
of related
VMs
Global
management
of VMs
GGUS
integration

VO based access
VO 1
(cloud a, b, c)
VO 2
(cloud b, c, d, e, f)
1. Community-specific VOs – e.g. CHIPSTER,
Highthroughtputseq, EISCAT, etc. (SLA, OLAs)
2. Training VO = training.egi.eu
3. Generic VOs – e.g. fedcloud.egi.eu à Incubator for new users
Browse VOs at http://operations-portal.egi.eu/vo/search
c
e
f
b
a
d

Main difference between VMs and Containers
VMs virtualize hardware to
run multiple OS instances
Containers virtualize OSes to
run multiple workloads

Docker
Open platform for building, shipping and running
distributed applications
Docker commoditizes containers
• Hides and automates container management process
• One-command-line deployment of applications
• Easy to move from development to production
• Provides ecosystem to create and share images

Container orchestration
Container
Orchestrator
App A
Container
Container
Container
Container
App B
Container
Container
Container
Container
Infrastructure
• Container applications tend to be comprised
of several containers running in a
coordinated way
• Running few containers can be easily
managed on a single node (e.g. with docker-
compose)
• Manages a cluster of nodes where to run the
containers
• Automates the deployment, management,
scaling, networking, and availability of
container-based applications running on that
cluster
• Main products: Docker Swarm, Kubernetes &
Mesos
• Schedule containers to physical or virtual
machines
• Restart containers if they stop
• Provide private container network
• Scale up and down
• Service discovery

• k8s is a portable, extensible open-source platform for managing containerized
workloads and services, that facilitates both declarative configuration and
automation.
• k8s operates at the container level (not the hardware level), providing a
container-centric management environment.
• It orchestrates computing, networking, and storage infrastructure on behalf of
user workloads.
• k8s
§ gives you the ability to deploy and scale Docker containers on-demand
§ without the overhead of installing, managing and operating the nodes
§ manages the nodes where to run the containers
§ automates the deployment, management, scaling, networking, and availability of
container-based applications running on that cluster
Kubernetes (k8s)

1. Single node: start the EGI Docker VM and run containers directly (or with
docker compose)
2. EGI Cloud Container Compute:
§ Start a cluster of VMs and create a Kubernetes cluster to run your containers
3. udocker: run containers as jobs in the EGI HTC service
Running containers on the EGI Federated Cloud
Two (+ 1) options to run containers on the EGI

Service architecture and interfaces
Cloud Management
Framework
IaaS API
EGI Cloud Container
Compute
1. Provision cluster VMs
3. Use native Kubernetes tooling
with EGI Check-in authentication
2. Deploy Kubernetes cluster on provisioned VMs
EGI Cloud Compute Service

Comparison
EGI Cloud Compute, Cloud Container Compute and HTC
EGI Cloud Compute EGI Cloud Container
Compute
EGI High Throughput
Compute
What is it? Multi-cloud IaaS Kubernetes on top of EGI
Cloud Compute
The grid, a scalable batch
system
What you run? VMs (Docker) Containers Jobs
Typical
workloads
Lift and shift existing
applications
Specific OS (kernel)
requirements
Cloud-native containerised
applications.
Execution of parallel
computing tasks to analyse
large datasets.
Pros / Cons [+] Complete control on
resources, run (almost)
anything you’d like
[-] Complex operation
[+] Industry standard
[+] Hides complexity of
Kubernetes setup
[-] Kubernetes steep learning
curve
[+] No management of
resources, just submit jobs
[-] Legacy interfaces
[-] Porting of applications
Configurability Abstraction

Examples of community use cases
EGI Notebooks
ELIXIR’s Global Alliance for Genomics and
Health
Group on Earth Observations (GEO) effort to
build a Global Earth Observation System of
Systems (GEOSS)

EGI Notebooks
Jupyter Notebooks as a Service
A one-click solution, just log in and start using
Kubernetes Cluster (EGI Cloud Container Compute)
User sessions
EGI Check-in
SSL Certificate
Notebooks
persistent
storage
Monitoring Accounting
Kubernetes
Ingress
EGI ARGO EGI Accounting

Large amounts of Earth Observation
data are vastly underexploited for
change monitoring and management
implementation actions supporting
the UN Sustainable Development
Goals.
Global Earth Observation System of Systems
(GEOSS)
• Discover the service through the
EOSC Portal
• Identify available related workflows
related to the issue of interest
• Explore available data through
GEOSS
• Execute the workflow taking
advantage of the EOSC infrastructure

GEOSS
Architecture overview
VLab APIs
VLab
Portal and apps
Long-term
preservation
archives
Knowledge bases
Remote processing
services
Source code
Workflows

• The Global Alliance for Genomics and Health (GA4GH) is an international
consortium of academic and industry partners that try to establish standards to
promote and facilitate collaboration and data exchange in the life sciences.
• As part of the 'Cloud Workstream' of this effort 4 standards have been proposed
to facilitate running scientific workflows in a cloud environment:
1. the Data Object Service (DOS),
2. Tool Registration Service (TRS),
3. Workflow Execution Service (WES) and the
4. Task Execution Service (TES).
• TESK is an implementation of the TES standard using the Kubernetes container
orchestration platform.
§ TES orchestrates staging in/out of data and execution of tasks
ELIXIR’s GA4GH

ELIXIR’s GA4GH
TESK Architecture

• Request access to the EGI services via our Marketplace:
https://marketplace.egi.eu/
• Cloud Container Compute:
https://marketplace.egi.eu/33-cloud-container-compute-beta
• Deploying Docker and Kubernetes in EGI FedCloud
https://wiki.egi.eu/wiki/Federated_Cloud_Containers
Getting started
EGI’s Cloud Container Compute

Opportunity for research communities, IT-service
providers, technology providers, members and partners of
EGI to get together, present and co-design services for
advanced computing in Europe.
Including co-located events:
1. Design your e-Infrastructure workshop (9 May)
2. FitSM Foundation Training (9 May)
More information at:
https://indico.egi.eu/indico/event/4431/
Upcoming event
EGI Conference 2019

This work by the EGI Foundation
is licensed under a Creative Commons
Attribution 4.0 International License.
Questions?
Thank you
for your attention.
www.egi.eu
@EGI_eInfra
EGI: Advanced Computing for Research
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme

EGI Cloud Container Compute Service

More Related Content

What's hot (9)

Similar to EGI Cloud Container Compute Service (20)

Recently uploaded (20)

EGI Cloud Container Compute Service