![Mockerfile
23
● apt-get in highly declarative YAML
# syntax = r2d4/mocker
apiVersion: v1alpha1
images:
- name: demo
from: ubuntu:16.04
package:
repo:
- deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8
gpg:
- https://bazel.build/bazel-release.pub.gpg
install:
- bazel](https://image.slidesharecdn.com/buildingimagesefficientlyandsecurelyonkuberneteswithbuildkit-190521101727/85/KubeConEU-Building-images-efficiently-and-securely-on-Kubernetes-with-BuildKit-23-320.jpg)
[KubeConEU] Building images efficiently and securely on Kubernetes with BuildKit
- 1. Building images efficiently and securely on Kubernetes with BuildKit Akihiro Suda, NTT 1
- 2. Raise your hand if you have heard of BuildKit? 2
- 3. Raise your hand if you are already using BuildKit? 3
- 4. Raise your hand if you are already running BuildKit on Kubernetes? 4
- 5. Part 1 Introduction to BuildKit 5
- 6. BuildKit: next-generation docker build 6 ● Concurrent multi-stage build ● Efficient caching ● Secure access to private assets ● Flexible syntax for build definition ● Does not require root privileges
- 7. BuildKit: next-generation docker build 7 ● BuildKit is included in Docker since v18.06 ● But this talk will focus on the standalone version of BuildKit (buildkitd & buildctl) ○ No dependency on Docker $ export DOCKER_BUILDKIT=1 $ docker build ...
- 8. LLB DAG 8 ● LLB is to Dockerfile what LLVM IR is to C ● Typically compiled from Dockerfile ● Accurate dependency expression with DAG structure ○ Efficient caching ○ Concurrent execution docker-image://alpine Image git://foo/bar docker-image://gcc Run("apk add ..")Run("make")
- 9. LLB DAG 9 FROM golang AS stage0 ... RUN go build –o /foo ... FROM clang AS stage1 ... RUN clang –o /bar ... FROM debian AS stage2 COPY --from=stage0 /foo /usr/local/bin/foo COPY --from=stage1 /bar /usr/local/bin/bar 0 2 1
- 10. •DAGはマルチステージDockerfileを用いて記述できる BuildKit: 次世代 `docker build` FROM golang AS stage0 ... RUN go build –o /foo ... FROM clang AS stage1 ... RUN clang –o /bar ... FROM debian AS stage2 COPY --from=stage0 /foo /usr/local/bin/foo COPY --from=stage1 /bar /usr/local/bin/bar 0 2 1 https://t.co/aUKqQCVmXa
- 14. RUN --mount=type=cache 14 ● Allows preserving caches of compilers and package managers # syntax = docker/dockerfile:1.1-experimental ... RUN --mount=type=cache,target=/root/.cache go build ...
- 15. RUN --mount=type=cache 15 ● Allows preserving caches of compilers and package managers # syntax = docker/dockerfile:1.1-experimental FROM ubuntu RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt apt-get update && apt-get install -y gcc
- 17. RUN --mount=type=secret 17 ● Allows accessing private assets without leaking credential in the image # syntax = docker/dockerfile:1.1-experimental ... RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ... $ buildctl build –-secret id=aws,src=~/.aws/credentials ...
- 18. RUN --mount=type=secret 18 ● Note: DON’T do this! ... COPY my_aws_credentials /root/.aws/credentials RUN aws s3 cp s3://... … RUN rm -f /root/.aws/credentials ...
- 19. RUN --mount=type=secret 19 ● Note: DON’T do this either! $ docker build --build-arg MY_AWS_CREDENTIALS=$(cat ~/.aws/credentials)
- 20. RUN --mount=type=ssh 20 ● Akin to --mount=type=secret but specific to SSH ● Supports passphrase # syntax = docker/dockerfile:1.1-experimental ... RUN --mount=type=ssh git clone ssh://github.com/... $ eval $(ssh-agent) $ ssh-add ~/.ssh/id_rsa (Enter your passphrase) $ buildctl build –-ssh default=$SSH_AUTH_SOCK ...
- 21. Non-Dockerfiles 21 ● LLB can be also compiled from non-Dockerfiles ● Several languages are being proposed ○ Buildpacks ○ Mockerfile ○ Gockerfile ● You can also create your own language
- 22. Buildpacks 22 ● Ported from Heroku/CloudFoundry Buildpacks ● No support for Cloud Native Buildpacks yet # syntax = tonistiigi/pack --- applications: - name: myapp memory: 128MB disk_quota: 256MB random-route: true buildpack: python_buildpack command: python hello.py
- 23. Mockerfile 23 ● apt-get in highly declarative YAML # syntax = r2d4/mocker apiVersion: v1alpha1 images: - name: demo from: ubuntu:16.04 package: repo: - deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8 gpg: - https://bazel.build/bazel-release.pub.gpg install: - bazel
- 24. Gockerfile 24 ● Really simple ● Specific to Golang # syntax = po3rin/gocker repo: github.com/foo/bar path: ./cmd/baz version: v0.0.1
- 25. Part 2 Deploying BuildKit on Kubernetes 25
- 26. Why build images on Kube? 26 Two kinds of motivation: 1. CI/CD 2. Developer Experience
- 27. Why build images on Kube? 27 BK Pod BK Pod BK Pod Some Pod Some webhook 1 1. CI/CD
- 28. 28poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1 1. CI/CD 2 Why build images on Kube?
- 29. Issue with docker build on Kube 29 ● The common pattern was to run docker Pod with /var/run/docker.sock hostPath ● Or run docker:dind Pod with securityContext.privileged ● Both are insecure
- 31. Rootless mode 31 ● BuildKit can be executed as a non-root user ● No extra securityContext configuration needed ● Protect the host from potential BuildKit vulns
- 32. Demo 32
- 33. ● Not true since BuildKit v0.4.0 ● But you need to disable “Process Sandbox”: launch buildkitd with --oci-worker-no-process-sandbox ○ Disable unsharing PIDNS and mounting /proc for RUN instructions 33 myth 1: requires securityContext.privileged
- 34. BuildKit daemon worker container (e.g. RUN gcc …) Host Process sandbox 34 myth 1: requires securityContext.privileged Process sandbox (needs to be disabled)
- 35. BuildKit daemon --oci-worker-no-process-sandbox worker container (e.g. RUN gcc …) Host worker container can kill(2) the daemon Host is still protected Process sandbox 35 myth 1: requires securityContext.privileged
- 36. ● To enable Process Sandbox, securityContext.procMount needs to be set to Unmasked ○ Requires Kubernetes v1.12+ with Docker v18.06+ / containerd v1.2+ / CRI-O v1.12 36 myth 1: requires securityContext.privileged
- 37. 37 myth 2: seccomp and AppArmor need to be disabled
- 38. ● Not a myth :P ● seccomp (and AppArmor) is typically disabled by default on Kubernetes anyway ○ In Kubernetes world, seccomp is still in alpha status and AppArmor is in beta 38 myth 2: seccomp and AppArmor need to be disabled
- 39. 39 myth 2: seccomp and AppArmor need to be disabled BuildKit daemon worker container (e.g. RUN gcc …) Host seccomp seccomp (needs to be disabled)
- 40. 40 myth 2: seccomp and AppArmor need to be disabled BuildKit daemon worker container (e.g. RUN gcc …) worker containers are still protected with seccomp Host seccomp
- 41. Future work: gVisor integration? ● gVisor: Yet another Linux kernel implementation in userspace ● взор (vzor): gVisor-based sandbox for runc containers https://github.com/tonistiigi/vzor 41 Host runc взор BuildKit daemon
- 42. Future work: gVisor integration? ● No need to disable seccomp/AppArmor for runc ● Can also mitigate kernel vulns 42 Host runc взор BuildKit daemon
- 43. Future work: gVisor integration? 43 ● Currently BuildKit fails with EINVAL due to syscall incompatibility ● Or User-Mode Linux? ○ Full Linux compatibility ○ 20 yo, still alive :)
- 44. Rootless BuildKit vs Kaniko 44 ● Kaniko runs as the root but “unprivileged” ○ No need to disable seccomp and AppArmor ● Kaniko might be able to mitigate some vuln that Rootless BuildKit cannot mitigate - and vice versa ○ Rootless BuildKit might be weak against kernel vulns ○ Kaniko might be weak against runc vulns
- 47. Deployment strategy 47 ● Deployment ○ Most typical deployment ● DaemonSet ○ Better Pod placement ○ But unlikely to hit daemon-local cache if you have a bunch of replicas ○ So might not be always optimal for large clusters w/ complex Dockerfiles
- 48. Deployment strategy 48 ● StatefulSet ○ Consistent Pod names ○ Good for Consistent Hashing (discussed later)
- 49. Deployment strategy 49 ● Job (“Daemonless”) ○ buildctl and ephemeral buildkitd in a single container in an ephemeral Pod ○ No need to manage the life cycles of the daemons ○ Needs PR: moby/buildkit#1005 ■ or github.com/genuinetools/img (lacks some upstream features)
- 50. How to connect to BuildKit? 50 ● BuildKit daemon can listen on TCP (with TLS) ● The entire operation (build & push) just needs a single gRPC connection ● So you can create Kubernetes Service for connecting to BuildKit Deployment / DaemonSet / StatefulSet
- 51. How to connect to BuildKit? 51 BK Pod BK Pod BK Pod ServiceClient gRPC request Load-balancing component (Can be just headless svc with DNSRR)
- 52. How to connect to BuildKit? 52 ● But you don’t need to necessarily create Service ● buildctl CLI can directly connect to a daemon in a Pod without Service ○ Internally invokes kubectl exec
- 53. How to connect to BuildKit? 53 $ kubectl run --generator=run-pod/v1 --image=moby/buildkit:master-rootless bk -- --oci-worker-no-process-sandbox $ export BUILDKIT_HOST=kube-pod://bk $ buildctl build ...
- 54. Coming soon: docker buildx for Kube 54 ● docker buildx is the next generation CLI for integrating BuildKit to Docker ○ Supports building multi-arch image with remote ARM machines ○ “Bake”: compose-like build ● docker buildx will support connecting to BuildKit on Kubernetes in the same UX
- 56. Remote cache 56 ● Cache can be shared via either registry or shared FS ● Similar to classic docker build --cache-from but more chance of hitting cache ● For building non-container artifacts (it’s a valid use-case), FS cache might be useful
- 57. Remote cache 57 BK pod BK pod BK pod Service Load-balancing component (Can be just headless svc with DNSRR) RegistryClient gRPC request Image Cache
- 58. Remote cache 58 ●Remote cache might be slow compared to the daemon-local cache ●Example from Part 1 slides: ○ No cache: 2m50s ○ Remote cache: : 36s ○ Daemon-local cache: 0.5s
- 59. Consistent hashing 59 ●Consistent hashing allows sticking a build request to a specific Pod in StatefulSet ●So the build request can always hit the daemon-local cache in the Pod
- 64. Consistent hashing 64 ●Caveats: ○ High I/O overhead on specific set of nodes ○ Some nodes might not be used at all ●See examples/kube-consistent-hashing in the moby/buildkit repo
- 65. Remote cache vs Consistent hashing? 65 ● If your cache registry is fast enough for your Dockerfiles, remote cache w/ load-balancing might be better ● If you don’t like transferring cache, consistent hashing might be better
- 66. Part 2.4 CRD 66
- 68. CRD 68 Container Builder Interface (CBI) ● The first common build CRD ● Supports Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, Azure Container Registry Build, and OpenShift S2I ● Complex design with a bunch of microservices ● Now being deprecated
- 69. CRD 69 Container Builder Interface (CBI) ● Simpler than CBI and easily extensible ● The build component (not entire Knative) might be going to be replaced by Tekton? ● Spun out from Knative ● Much more simple and extensible
- 70. CRD 70 Container Builder Interface (CBI) ● Simpler than CBI and easily extensible ● The build component (not entire Knative) might be going to be replaced by Tekton? ● Spun out from Knative ● Much more simple and extensible
- 71. Tekton 71 apiVersion: tekton.dev/v1alpha1 kind: TaskRun metadata: name: foobar spec: taskRef: name: buildkit ... The interface is same as other image builders (Buildah, Kaniko, and Makisu)
- 72. Tekton 72 inputs: resources: - name: source resourceSpec: type: git params: - name: url value: git@github.com:foo/bar.git SSH credential is loaded from the Secret associated with the ServiceAccount
- 73. Tekton 73 outputs: resources: - name: image resourceSpec: type: image params: - name: url value: registry.example.com/foo/bar Registry credential is loaded from the Secret associated with the ServiceAccount
- 74. Wrap-up ● BuildKit is fast and secure ● Several deployment plans, w/ and w/o daemon ● Example: BuildKit BuildKit BuildKit Headless Service Tekton Controller BuildKit Client DaemonSet Registry 74 YAML