Distributed scientific computing for open science, eResearch Africa 2019

www.egi.eu
@EGI_eInfra
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme
EGI: Advanced Computing for Research
eResearch Africa 2019
Distributed scientific
computing for open science
Technical Director, EGI Foundation
Tiziana Ferrari

@EGI_eInfrawww.egi.eu 17/04/2019 2
•EGI Federation
•Federation in action: Cloud, AAI and Data Management
•Towards Open Science
•Trends and Outlooks
Outline

About EGI
Advanced Computing for Research

EGI Federation (April 2019)
• ZA-UCT-ICTS University of Cape Town - ICTS HPC site
• ZA-UFS (University of the Free State Computing
Centre)
HPC site of the University of Bloemfontein
• ZA-WITS-CORE (University of the Witwatersrand
CORE)
Core research cluster of the University of the
Witwatersrand
4.4 Billion
CPU core
wall time
(2018)
> 1 Million
computing
cores in 2019
> 740 PB disk
& tape
2,915 service
end-points

A global system of e-Infrastructures worldwide

European Participants

EGI Service Portfolio

EGI Internal Service Portfolio
For the members of the federation

• Leverages national e-Infrastructure investments
• Opens access to part of the nationally funded capacity
• Supports international user groups
• Integrates community, private and/or public infrastructures into a scalable
data/computing platform for research
• Uses federated identities, authentication and authorization
• Ensures interoperability of scientific applications and data across multiple
providers bringing distributed computing to data
Benefits of Federation

Federated Operations

Federation in action
Cloud, AAI and Data Management

• Multi-cloud IaaS infrastructure with Single Sign-On
• Federation features:
 Common VM image catalogue
 Discovery, accounting, monitoring
 Unified GUI dashboard
EGI Federated Cloud
Cloud
Compute
Cloud Container
Compute BETA
Training
Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA

Infrastructure

Architecture and Interfaces
• Tools to deal with
heterogeneity:
 IaaS orchestration tools
with support for multiple
APIs:
o Infrastructure Manager,
Terraform, OCCOPUS, …
o https://wiki.egi.eu/wiki/Fe
derated_Cloud_IaaS_Orch
estration
 IaaS libraries with support
for multiple APIs:
o libcloud, jclouds,…

Architecture (Cloud provider view)

• Identity and Access Management solution
 Single sign-on to services through eduGAIN, social media and other institutional or
community-managed identity providers
 Only one account needed for federated access to multiple heterogeneous (web and
non-web) service providers using different technologies (SAML, OpenID Connect,
OAuth 2.0, X509)
 Identity linking enables access to resources using different login credentials
(institutional/social)
 Assurance information associated to each authenticated identity
 Aggregation and harmonisation of authorisation information (VOs/groups, roles,
assurance) from multiple sources
Federated AAI: Check-in

• Conforms to AARC blueprint architecture
• Registered in eduGAIN as an SP complying
with REFEDS Research & Scholarship and
Sirtfi
• All community SPs can have one statically
configured IdP
• No need to run an IdP Discovery Service
on each SP
• Connected SPs get consistent/harmonised
user identifiers and accompanying
attribute sets from different IdPs/AAs that
can be interpreted in a uniform way for
authorisation purposes
Check-in: Identity Provider and Service
Provider Proxy

Check-in: IdP Discovery

Federated accounting

Federated Monitoring

Application Database 1/3
Cloud Marketplace

Cloud Marketplace

VM distribution and management

• Heterogeneous
backend storage
• Common interfaces
(Web, REST, POSIX,
CDMI)
• Common AAI with
Check-in
• Discovery of Datasets
in the EGI DataHub
Federation of Data Repositories

• Clients uses one
ore more providers
to access data
• Data can be
accessed over
multiple protocols
Transparent Data Access

Data Caching
• Cloud provider A hosts data
& computing resources
• Provider B only hosts data
Provider X can use data from A
and B
• Without pre-staging
• Via pre-staging using APIs
• Local data access “à la”
POSIX with FUSE

Multi-cloud Analytics with Jupyter

Towards Open Science
Sharing of data, applications and research
outputs for scientific reproducibility

Chipster: sharing community applications
Read more...

NBIS: Scalable bioinformatics web-servers
powered by cloud computing
Read more…

ESA Geohazards Thematic Exploitation
Platform Read more…

Cloud Containers on the EGI Federated Cloud

EGI Notebooks

• The computational tools to solve a problem
 Python, R, Julia, and wide ecosystem of libraries and tools for science
• An interface to facilitate coding / creating  Jupyter
• A way to communicate work  Notebooks
• A way to share work  GitHub other similar repositories
• A way to pack it all for replication  Docker
• A way to persistently identify it  DOIs (Digital Object Identifiers)
Reproducible Open Science with
EGI Notebooks, Binder, Zenodo
https://documents.egi.eu/document/3442

Open Science in action

Usage Trends and Outlooks
Towards increasing integration of e-infrastructures

Installed Compute Capacity 2011-2019

CPU wall time consumed 2009-2018

Africa’s contribution to the federation
https://accounting.egi.eu/

Scientific Disciplines/HTC
65.6 Million hours for CTA/LOFAR/SKA

Scientific Disciplines/Cloud

Today’s scenario
• Difficult cross-border access due to different funding models, access and
provisioning policies
 Data and service provisioning to international user communities possible only when supported by
sound business models or existing collaboration agreements. Today only a few structured int.
research groups have achieved this.
• Needs of large investments for the creation, processing, preservation, access
and reuse of research data  will the funding match the anticipated needs of
future data-intensive science?
 Opportunities for economies of scale and aggregation of demand can arise with joint provisioning
of infrastructure common components
• Major separation between data preservation and data exploitation
infrastructures in many disciplines
 Ris and e-Infrastructures should collaborate to support the entire research workflow of an
experiment

Tomorrow’s scenario
The International Data Commons
A federation of research data, computing, applications
and other open science resources, responding to the
problem of scalable access to research data through a
new data provisioning service approach that is
complementary to the traditional data download
model.

The Data Commons Should…
Allow to discover, access and analyze major research
datasets and information for third-party exploitation
Provide access to the data & data products close to
processing facilities while avoiding duplication of
local data storage & compute infrastructures across
research performing organizations in Europe

The Data Commons Should…
• Offer a hybrid distributed compute platform (HTC, HPC,
cloud) and integrated rich portfolio of scientific application
tools supporting self-service provisioning
• Offer tools for scalable data movement across data
preservation infrastructures and distributed interconnected
network of “data hubs”
• Provide integrated capabilities for publishing and sharing
scientific outputs from experiments to support open science
• Support federated authentication and authorization for use
of existing personal credentials and easy to use access
channels

The federated infrastructure and supporting initiative
providing
all researchers, innovators, companies and citizens
with seamless access to an open-by-default, efficient and
cross-disciplinary environment
for storing, accessing, reusing data, tools, publications and
other scientific outputs for research, innovation and
educational purposes
About the European Open Science Cloud

This work by the EGI Foundation
is licensed under a Creative Commons
Attribution 4.0 International License.
Questions?
Thank you
for your attention.
www.egi.eu
@EGI_eInfra
EGI: Advanced Computing for Research
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme

Distributed scientific computing for open science, eResearch Africa 2019

More Related Content

What's hot (20)

Similar to Distributed scientific computing for open science, eResearch Africa 2019 (20)

More from EGI Federation (6)

Recently uploaded (20)

Distributed scientific computing for open science, eResearch Africa 2019