Containers for Science and High-Performance Computing

Containers for Science and
High-Performance Computing
Dmitry Spodarets
Root Linux GlobalLogic Conference 2017

Who am I
Dmitry Spodarets
• Founder and CEO at FlyElephant
• PhD candidate at Odessa National University
• Lecturer at Odessa Polytechnic University
• Organizer of technical conferences about
Data Science, HPC, JS, Web Technologies …

Data ScienceRendering
Engineering
Simulation
Academia

FlyElephant
We automate Data Science and Engineering Simulation
and help teams to work efficiently.
Computing
resources
Ready-computing
infrastructure
Collaboration
& Sharing
Fast
Deployment
Expert
Community

Agenda
•Scientific tasks
•Computing resources
•High-Performance Computing
•Containers
•Singularity
•Conferences

Healthcare Financial RetailLife Sciences Manufacturing

Computing resources
NVIDIA DGX-1 Deep Learning Supercomputer
GPU Xeon Phi
FPGA

1080 vs Titan X vs K80 vs P100
0,25
8,8
0,3
10,1
2,9
8,7
5,3
10,6
0
2
4
6
8
10
12
FP32 (Single precision) FP64 (Double precision)
1080 Titan X K80 P100
http://www.nvidia.com/
TFLOPS

High-Performance Computing
and Supercomputer

HPC
Sunway TaihuLight / China
10,649,600 Cores
125,436 TFlop/s

What about elastic environment?

Timeline of LXC Tools
http://kiwenlau.blogspot.com/2015/01/linux-container-technology-overview.html

Containers vs Virtual Machines

Docker (Dockerfile)
FROM registry.flyelephant.net/leap
MAINTAINER Dmitry Spodarets <d.spodarets@flyelephant.net>
RUN zypper ref && zypper -n in blender && zypper clean

Docker (build.sh)
#!/bin/bash
function docker_build {
docker build -t $1 ./$1;
docker tag $1 registry.flyelephant.net/$1
docker push registry.flyelephant.net/$1
docker rmi $1 registry.flyelephant.net/$1
}case $1 in
all)
for i in `cat build.list`; do
docker_build $i;
done
;;
*)
docker_build $1;;
esac

Docker
1. docker images
2. docker run --memory 512m --cpus="2" --name mycont registry.flyelephant.net/tensorflow
3. docker exec -i -t mycont bash
4. docker ps
5. docker stats
6. docker stop CONTAINER ID
7. docker start CONTAINER ID
8. docker rm CONTAINER ID

Docker Machine
• Amazon Web Services
• Digital Ocean
• Exoscale
• Generic
• Google Compute Engine
• IBM Softlayer
• Microsoft Azure
• Microsoft Hyper-V
• OpenStack
• Oracle
• VirtualBox
• Rackspace
• VMware Fusion
• VMware v
• Cloud Air
• VMware vSphere
docker-machine create --driver azure --azure-subscription-id subscription-id --azure-
resource-group resourcename --azure-ssh-user user --azure-size machine-name
docker-machine ssh machine-name

Generally containers technologies
have been designed to solve a
single primary use case for the
enterprise:
micro-service virtualization

This is *NOT* the scientific or
HPC use case!
So… what is our use case?

Scientists are like Pirates, pillaging
for resources instead of booty!
We want to run our jobs. We want
to get results.

Containers for Science and High-Performance Computing

And when we find available resources, we need to
ensure application and environment compatibility.
This is where containers can be a perfect fit…
Our use-case and needs are very different from
enterprise!

For example, reproducibility is
of the utmost importance in
science.

Science that isn’t easily
reproducible is challenged
and sometimes
embarrassingly erased.

Software environments can
often be very difficult to
reproduce… Sometimes
impossible.

And as scientists, we need
reproducibility!

So why not just keep using Docker?
The good news:
You can! It works great for local and private resources. You can use
it to develop and share your work with others using Docker-hub.
The bad news:
If you ever need to scale beyond your local resources, it maybe a
dead end path! Docker, and other enterprise focused containers,
are not designed for, efficient or even compatible with traditional
HPC.
No HPC centers allow it!

Shifter vs Charlie Cloud vs Docker vs Singularity

Singularity
• Project lead: Gregory M. Kurtzer
• Sponsored by: HPC @ LBNL
• URL: http://singularity.lbl.gov

Singularity - Containers for Science
• First public release in April 2016, followed by a massive uptake
• HPC Wire Editor’s choice: Top Technologies to Watch for 2017
• Simple integration with resource managers, InfiniBand, GPUs, MPI, file
systems, and supports multiple architectures (x86_64, PPC, ARM, etc..)
• Limits user’s privileges (inside user == outside user)
• No root owned container daemon
• Network images are supported via URIs and all require local caching:
○ docker:// - This will pull a container from Docker Hub
○ http://, https:// - This will pull an image or tarball from the URL, cache and run it
○ shub:// - Pull an image from the Singularity Hub

Singularity: access and privilege
User contexts are always maintained when the container is
launched.
When launched by a particular user, the programs inside
will be running as that user. Any escalation pathways inside
the container are blocked. Thus…
If you want to be root inside the container,
you must first be root outside of the container!

Singularity: access and privilege
[m31@centos7-x64 ~]$ whoami
m31
[m31@centos7-x64 ~]$ singularity shell /tmp/debian.img
Singularity: Invoking an interactive shell within container…
Singularity.debian.img> whoami
m31
Singularity.debian.img> sudo whoami
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file
system without root privileges?
Singularity.debian.img> ls -l /usr/bin/sudo
-rwsr-xr-x. 1 root root 136808 Aug 17 13:20 /usr/bin/sudo
Singularity.debian.img> exit
[m31@centos7-x64 ~]$ sudo singularity shell /tmp/debian.img
Singularity: Invoking an interactive shell within container...
Singularity.debian.img> whoami
root
Singularity.debian.img> exit

Singularity: Creating the container
$ sudo singularity create --size 2048 /tmp/Centos-7.img

Singularity: Bootstrapping the container
$ sudo singularity bootstrap /tmp/Centos-7.img centos.def

The container is ready
$ singularity shell /tmp/Centos-7.img
Singularity: Invoking an interactive shell within container...
Singularity.Centos-7.img> cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Singularity: Bootstrap definition - RedHat/YUM
$ cat examples/centos.def
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
%runscript
echo "This is what happens when you run the container..."
%post
echo "Hello from inside the container"
yum -y install vim-minimal
● The header defines the core operating system
● %runscript: What should this container do when “run” (singularity run, or ./container.img)
● %post: After the core operating system has been built, this gets executed inside the new container

Singularity: Bootstrap definition - Debian/Ubuntu
$ cat examples/debian.def
BootStrap: debootstrap
OSVersion: stable
MirrorURL: http://ftp.us.debian.org/debian/
%runscript
%post
apt-get update
apt-get -y install vim

Singularity: Bootstrap definition - OpenSuse
$ cat examples/opensuse.def
BootStrap: zypper
OSVersion: 42.2
MirrorURL: http://download.opensuse.org/distribution/leap/%{OSVERSION}/repo/oss/
%runscript
%post

Usage Examples
$ python ./hello.py
Hello World: The Python version is 2.7.5
$ sudo singularity exec --writable /tmp/debian.img apt-get install python
…
$ singularity exec /tmp/debian.img python ./hello.py
$ cat hello.py | singularity exec /tmp/debian.img python

Usage Examples
$ mpirun singularity exec /tmp/Centos7-ompi.img /usr/bin/mpi_ring
Process 0 sending 10 to 1, tag 201 (4 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 exiting
Process 1 exiting
Process 2 exiting
Process 3 exiting

Container Image types supported
● Local images supported:
○ Singularity 2.x default image format
○ SquashFS
○ Tarballs (requires caching)
○ Flat directories (chroots)
● Network images are supported via URIs and all require local caching:
○ docker:// - This will pull a container from Docker Hub
○ http://, https:// - This will pull an image or tarball from the URL, cache and run it
○ shub:// - Pull an image from the Singularity Hub … Wait, what?! Patience, all in due time!

Usage Examples
$ singularity exec docker://python:latest /usr/local/bin/python hello.py
library/python:latest
Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Downloading layer: sha256:e41da2f0bac3da1769ecdac8b0f5df53c1db38603e39b9e261cafd10caf904de
Downloading layer: sha256:75ef15b2048b4cfb06c02f2180f4d89033d02c63f698672d2909b8c9878c4270
Downloading layer: sha256:45b2a7e03e44b5ea7fad081537134c9cc725bddf94f9093b00e1fa8d8ebbcda1
Downloading layer: sha256:52f3db4b5710849a53bc2eea0b6f0895c494d751c38c597404d805da82b3f37c
Downloading layer: sha256:76610ec20bf5892e24cebd4153c7668284aa1d1151b7c3b0c7d50c579aa5ce75
Downloading layer: sha256:fce5728aad85a763fe3c419db16885eb6f7a670a42824ea618414b8fb309ccde
Downloading layer: sha256:5040bd2983909aa8896b9932438c3f1479d25ae837a5f6220242a264d0221f2d

Usage Examples
$ singularity exec docker://tensorflow/tensorflow python -m tensorflow.models.image.mnist.convolutional
tensorflow/tensorflow:latest
...
Initialized!
Step 0 (epoch 0.00), 5.1 ms
Minibatch loss: 8.334, learning rate: 0.010000
Minibatch error: 85.9%
Validation error: 84.6%
Step 100 (epoch 0.12), 140.0 ms
…
Step 8500 (epoch 9.89), 134.2 ms
Validation error: 0.9%
Test error: 0.8%

Singularity Hub: Container Registry

Singularity Hub: Build/Import workflow
● Singularity bootstrap definition is
committed and pushed to a GitHub
repository (named ‘Singularity’)
● GitHub communicates with Singularity
Hub via a web hook, and it gets queued to
be built via continuous integration
● Once built the resulting product is stored
in Google Cloud and available to be
accessed
● The container will be archived and
can/should be cited in your scientific
publications so others can replicate and
leverage your work with:
$ singularity shell shub://$UNIQUE_ID
$ singularity run shub://$USER/$CONTAINER:$TAG
Sochat V., Prybol C, Kurtzer GM. Singularity Hub: Registry for Reproducible Containers (manuscript in progress)

singularity.lbl.gov
Twitter: @SingularityApp
GitHub: http://www.github.com/singularityware/singularity
Slack: http://singularity-container.slack.com (email me for invite)

DS and HPC Conferences
• Data Science Lab
• May 13, Odessa, Ukraine
• http://www.datascience.in.ua/
• ISC HIGH PERFORMANCE
• June 18-22, Frankfurt, Germany
• http://www.isc-hpc.com/
• SC Conference
• November 12-17, Denver, Colorado, USA
• http://sc17.supercomputing.org/

Dmitry Spodarets
d.spodarets@flyelephant.net
www.flyelephant.net

Containers for Science and High-Performance Computing

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Containers for Science and High-Performance Computing (20)

Recently uploaded (20)

Containers for Science and High-Performance Computing