Usernetes: Kubernetes as a non-root user
- 1. Copyright©2018 NTT Corp. All Rights Reserved. Akihiro Suda ( @_AkihiroSuda_ ) NTT Software Innovation Center Usernetes: Kubernetes as a non-root user Kubernetes & Cloud Native Meetup (Oct 25, 2018) https://slideshare.net/AkihiroSuda
- 2. 2 Copyright©2018 NTT Corp. All Rights Reserved. • Software engineer at NTT (The largest telco in Japan) • A maintainer of Moby (former Docker Engine), BuildKit, containerd, and etc... • An organizer of Docker Tokyo meetup • github: @AkihiroSuda • Twitter: @_AkihiroSuda_ Who am I
- 3. 3 Copyright©2018 NTT Corp. All Rights Reserved. • Run kubelet, dockerd (or crio), and everything as a non-root user • Don't confuse with securityContext.runAsUser and "Node- level UserNS" (milestone: v1.13) • Binary releases are available: https://github.com/rootless- containers/usernetes • Just unpack the archive under your $HOME, and you're all set Usernetes: Kubernetes as a non-root user $ tar xjvf usernetes-x86_64.tbz $ cd usernetes $ ./run.sh
- 4. 4 Copyright©2018 NTT Corp. All Rights Reserved. Motivation User application Kubernetes CRI runtimes (Docker/containerd/CRI-O) OCI runtimes (runc) Kernel VM Hardware Out of scope of container security, basically (e.g. Meltdown) Can be hardened with containers
- 5. 5 Copyright©2018 NTT Corp. All Rights Reserved. Motivation User application Kubernetes CRI runtimes (Docker/containerd/CRI-O) OCI runtimes (runc) Kernel VM Hardware Can we really rely on them...?
- 6. 6 Copyright©2018 NTT Corp. All Rights Reserved. • Kubernetes CVE-2017-1002101 • A malicious container can access the host filesystems due to a volumeMounts.subPath symlink issue • Kubernetes CVE-2017-1002102 • A malicious containers ca remove files on the host filesystems due to a secret/configMap/downwardAPI volume issue Kubernetes CVEs https://kubernetes.io/blog/2018/04/04/fixing-subpath-volume-vulnerability/ https://github.com/kubernetes/kubernetes/issues/60814
- 7. 7 Copyright©2018 NTT Corp. All Rights Reserved. • Docker CVE-2014-9357 • A malicious container can execute arbitrary binaries on the host due to an LZMA archive issue • containerd #2001 • A malicious container image can remove /tmp on the host when pulled (not when launched!) due to an image layer issue • CRI-O CVE-2018-1000400 • A malicious user in a container can gain the root in the container due to a capability issue • runc CVE-2016-9962 • A malicious container can gain the FDs of additional `runc exec` processes before it is fully placed in the container due to a process initialization issue • runc CVE-2016-3697 • A malicious container can gain unexpected UID due to a UID strconv issue • And more! Runtime CVEs
- 8. 8 Copyright©2018 NTT Corp. All Rights Reserved. • Execute container runtimes and orchestrators as well as containers as an unprivileged user, by using User Namespaces • Can mitigate the security risk effectively, although not panacea • Kernel might still have security bugs that can't be mitigated... but we have more "eyeballs" for the kernel • Officially adopted by runc, img, BuildKit, Buildah, Podman, and CRI-O • Patch available for Docker/Moby, containerd, and Kubernetes Our approach: Rootless containers
- 9. 9 Copyright©2018 NTT Corp. All Rights Reserved. • An unprivileged user can create Network Namespaces along with a User Namespace, but cannot set up the VETH pair across the parent and the child namespaces • i.e. No internet connection Challenge: Networking The Internet Host (“parent”) UserNS + NetNS (“child”) NetNS NetNS
- 10. 10 Copyright©2018 NTT Corp. All Rights Reserved. • Prior work: LXC uses SETUID binary (lxc-user-nic) for setting up the VETH pair across the parent and the child namespaces • Problem: SETUID binary can be dangerous! • CVE-2017-5985: netns privilege escalation • CVE-2018-6556: arbitrary file open(2) Challenge: Networking
- 11. 11 Copyright©2018 NTT Corp. All Rights Reserved. Challenge: Networking • Our approach: use usermode network (“Slirp”) with a TAP device (https://github.com/rootless-containers/slirp4netns) • Similar to `qemu –netdev user` • Completely unprivileged • iperf3 benchmark on Travis: 9.21 Gbps The Internet Host UserNS + NetNS NetNS NetNS TAP “Slirp” TAPFD “sendfd” (SCM_RIGHTS cmsg over socketpair)
- 12. 12 Copyright©2018 NTT Corp. All Rights Reserved. • OverlayFS • Kernel-mode overlayfs cannot be used in User Namespaces (except on Ubuntu kernel) • But FUSE implementation of overlayfs can be used (Kernel >= 4.18) • Even without overlayfs, copy_file_range(2) can be still used for file deduplication on XFS • Cgroups • pam_cgfs.so can be used for delegating Cgroups configuration to an unprivileged user • But not recommended by systemd maintainers, due to potential DOS • Cgroups v2 supports delegation properly (nsdelegate), but not adopted in OCI/Kubernetes ecosystem yet • Usernetes can be executed without access to Cgroups, except some features like `kubectl top` Other challenges
- 13. 13 Copyright©2018 NTT Corp. All Rights Reserved. • Based on Kubernetes 1.12 and Docker 18.09 • Supports both Docker and CRI-O • Support for containerd is also on plan • Multi-node cluster will be supported soon • Binary available: https://github.com/rootless-containers/usernetes Current status of Usernetes $ tar xjvf usernetes-x86_64.tbz $ cd usernetes $ ./run.sh
- 14. 14 Copyright©2018 NTT Corp. All Rights Reserved. Demo
Editor's Notes
- #2: https://wantedly.connpass.com/event/105371/ 10 minutes