ISC HPCW talks

Copyright©2019 NTT Corp. All Rights Reserved.
Akihiro Suda ( @_AkihiroSuda_ )
NTT Software Innovation Center
My ISC HPCW Talks
1. Current state of rootless dockerd
2. Rootless build with BuildKit
3. OCI Image Spec & Distribution
5th Annual High Performance Container Workshop, ISC (June 20, 2019)

Current state of rootless dockerd

3 
What is rootless dockerd?
• Run Docker daemon (and also containers of course) as a
non-root user
• Don’t confuse with:
• sudo
• usermod -aG docker penguin
• docker run --user
• dockerd --userns-remap
• Experimentally supported since Docker v19.03
https://get.docker.com/rootless
Image: https://xkcd.com/149/

4 
Why?
• For Cloud-Native envs:
• To mitigate potential vulnerability of container runtimes and
orchestrator
• For HPC envs:
• To run containers without the risk of breaking other users
environments

5 
How it works: User Namespaces
• User namespaces allow non-root users to pretend to be
the root
• Root-in-UserNS can have “fake” UID 0 and also create
other namespaces (MountNS, NetNS..)
• Unlike Singularity, NetNS can be unshared
• By using either usermode TCP/IP stack (VPNKit, slirp4netns) or
SETUID binary (lxc-user-nic)

6 
System requirements: /etc/{subuid,subgid}
• If /etc/subuid contains “1001:100000:65536”
• Having 65536 sub-users should be enough for most
containers
0 1001 100000 165535 232Host
UserNS
primary user
sub-users
start
sub-users
length
0 1 65536

7 
Unresolved issues (Contribution wanted!)
• Hard to maintain subuid & subgid in LDAP/AD envs
• NSS module is being under discussion
https://github.com/shadow-maint/shadow/issues/154
• Single-mapping mode w/o subuid & subgid is also under
discussion
• uses ptrace and xattrs
(slow!)
• seccomp could be
used for acceleration
https://github.com/rootless-containers/runrootless

8 
Unresolved issues (Contribution wanted!)
• Lacks cgroup
• cgroup2 (uniﬁed-mode) supports unprivileged mode but
migration may take a few years… or even more
• For cgroup1, pam_cgfs could be used instead, but not available
in Fedora / RHEL due to a security concern
• Kernel / VM / HW may have vulns
• Not suitable for real multi-tenancy
• gVisor might able to mitigate some of them

Rootless build with BuildKit

10 
What is BuildKit?
• Next-generation docker build with focus on performance
and security
• Accurate dependency analysis
• Concurrent execution of independent instructions
• Support injecting secret ﬁles...
• Integrated to Docker since v18.06
(export DOCKER_BUILDKIT=1)
• Non-Docker standalone BuildKit is also available
• Works with Podman and CRI-O as well :P

11 
Rootless mode
• Rootless mode allows building images as a non-root user
• Dockerﬁle RUN instructions are executed as a “fake root” in
UserNS (So apt-get/yum works!)
• Produces Docker image / OCI image / raw tarball
• Compatible with Rootless Docker / Rootless Podman / …
whatever
• Even works inside a container
• Good for distributed CI/CD on Kubernetes
• Works with default securityContext conﬁguration
(but seccomp and AppArmor needs to be disabled for nesting containers)

12 
Rootless BuildKit vs kaniko
• https://github.com/GoogleContainerTools/kaniko
• Kaniko runs as the root but “unprivileged”
• No need to disable seccomp and AppArmor because kaniko
doesn’t nest containers on the kaniko container itself
• Kaniko might be able to mitigate some vuln that Rootless
BuildKit cannot mitigate - and vice versa
• Rootless BuildKit might be weak against kernel vulns
• Kaniko might be weak against runc vulns

OCI Image Spec & Distribution

14 
Open Containers Initiative Speciﬁcations
• OCI Runtime Spec
• How to create container from conﬁg JSON and rootfs dir
• Based on Docker libcontainer (now runc)
• OCI Image Spec
• How to represent image layers for OCI runtimes
• Based on Docker Image Manifest V2, Schema 2
• OCI Distribution Spec
• How to distribute OCI images
• Based on Docker Registry HTTP API

15 
Image layout
/blobs/sha256/e692418e...
/blobs/sha256/b5b2b2c5...
/blobs/sha256/61be55a8...
/blobs/sha256/3c3a4604...
JSON
JSON
tar.gz
tar.gz
tar.gz
Manifest
• Merkle DAG structure ensures reproducibility of
docker pull foo@sha256:e692418e…
Container Conﬁg
AUFS layer archives
(for each Dockerﬁle
FROM and RUN)
v1.0Manifest list latest

16 
Image layout
latest
amd64
JSON
JSON
tar.gz
tar.gz
tar.gz
JSON
Manifest list
Manifest
• Supports multi-arch (use BuildKit to build)
Container Conﬁg
latest
arm64
AUFS layer archives
FROM and RUN)

17 
Image layout
latest
Ice Lake
JSON
JSON
tar.gz
tar.gz
tar.gz
JSON
Manifest list
Manifest
• And even multi-microarchitectures via qnib/metahub
• https://metahub.qnib.org
Container Conﬁg
latest
Broadwell
Tesla M60
AUFS layer archives
FROM and RUN)

18 
Post-OCI image format?
• Issues of current OCI v1
• Too coarse deduplication granularity
• Containers cannot be started until the entire image is pulled
• An alternative: CernVM-FS
• Supports ﬁle-level deduplication rather than layer-level
• Files are lazy-pulled on demand using FUSE
• Integrating CernVM-FS to containerd is under discussion
https://github.com/containerd/containerd/issues/2943

19 
Post-OCI image format?
• ”OCI v2” https://github.com/openSUSE/umoci/issues/256
• Much ﬁner deduplication granularity
• No implementation yet
• Container Registry Filesystem https://github.com/google/crfs
• Focus on lazy-pulling CI images
• IPCS https://github.com/hinshun/ipcs
• IPFS integration for containerd

ISC HPCW talks

More Related Content

What's hot (20)

Similar to ISC HPCW talks (20)

More from Akihiro Suda (20)

Recently uploaded (20)

ISC HPCW talks