SlideShare a Scribd company logo

[DockerCon 2023] Reproducible builds with BuildKit for software supply chain security

A
Akihiro Suda

The document discusses the importance of reproducible builds in Docker for software supply chain security, emphasizing that non-reproducible builds cannot be validated as buildable from harmless sources. It outlines the need for reproducibility to ensure transparency and verify that Docker images are indeed created from their source code. The text also highlights challenges such as timestamps and base image versions that affect the reproducibility of Docker images.

Software
1 of 45
Download to read offline
1
2
3
Most read
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
Reproducible builds with BuildKit for software supply chain security Akihiro Suda Software Engineer | NTT
2 Background • Security assessment has been hard for Docker images, due to lack of verifiability in the software supply chain • Even when the source code (Dockerfile) is public, and the source code appears to be harmless, it is hard to prove that the image is actually buildable from the source code • Reproducible builds help proving it (But whether the source code is harmless is another topic)
3 What are Reproducible Builds? • Same source, same binary • Attestable by anybody • Attestable at anytime Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67 sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67 sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67
4 Why do we need reproducible builds? Because non-reproducible builds cannot be proved to be buildable from harmless sources
5 Why do we need reproducible builds? sha256:AAAAA… Upstream build docker pull some-image Pull Because non-reproducible builds cannot be proved to be buildable from harmless sources
6 Because non-reproducible builds cannot be proved to be buildable from harmless sources Why do we need reproducible builds? FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… Upstream build docker pull some-image Pull Find the source repo
7 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:BBBBB… Upstream build docker pull some-image Pull Find the source repo Because non-reproducible builds cannot be proved to be buildable from harmless sources Your own build (Non-reproducible)
8 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:BBBBB… Upstream build docker pull some-image Pull Find the source repo Your own build (Non-reproducible) Is this image really buildable from the source repo? Because non-reproducible builds cannot be proved to be buildable from harmless sources
9 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:AAAAA… Upstream build docker pull some-image Pull Find the source repo Proved to be buildable from the source Because non-reproducible builds cannot be proved to be buildable from harmless sources Your own build (Reproducible)
10 Why do we need reproducible builds? • Reproducibility per se doesn’t prove any harmlessness • Non-reproducibility doesn’t prove any harmfulness, either
11 Why do we need reproducible builds? • Reproducibility proves that the image is actually buildable from its source • The source still has to be reviewed • The source may still be malicious • But at least the image contains no secret code that you can never review
12 Docker Hub images are actually reproducible?
13 Docker Hub images are actually reproducible? • No, mostly
14 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang
15 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang DOCKER_BUILDKIT=0 with Docker 20.10.23 corresponds to the current Docker Hub image (Will change in the future)
16 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang DiffOCI: diff for Open Container Initiative (OCI) images https://github.com/reproducible-containers/diffoci
17 Docker Hub images are actually reproducible? $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Desc application/vnd.docker.distribution.manifest.v2+json b25862... 3c4eca0... ... File etc/ssl/certs/3e45d192.0 2023-08-09 03:36:47 +0000 UTC 2023-09-21 08:35:31 +0000 UTC ... (More than 14,000 lines) ... File go/ 2023-09-06 18:31:40 +0000 UTC 2023-09-21 08:35:45 +0000 UTC
18 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
19 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
20 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference lib/apk/db/scripts.tar differ due to the timestamp information inside scripts.tar (the “--semantic” flag isn’t still clever enough to ignore this “boring” difference”) The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
21 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference lib/apk/db/scripts.tar differ due to the timestamp information inside scripts.tar (the “--semantic” flag isn’t still clever enough to ignore this “boring” difference”) This image is not fully reproducible, but its non-reproducibility is explainable (So, this image appears to be actually buildable from the public Dockerfile) The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
22 Why are images not reproducible? • Timestamps • Version of the base image (“FROM” images in Dockerfiles) • Versions of the packages (apt-get, pip, etc.) • Others: - Filesystem characteristics (e.g., OverlayFS) - Ordering of files - Randomized mktemp, etc.
23 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
24 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
25 Timestamps • BuildKit (since v0.11) supports rewriting the timestamps for OCI Image Config and OCI Index • Support was incomplete in v0.11 and v0.12; using v0.13 [beta] is recommended (see the next couple of slides) OCI = Open Container Initiative buildctl build --opt build-arg:SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) docker buildx build --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) Unix epoch (int64, seconds from 1970-01-01 00:00:00 UTC)
26 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
27 Timestamps • BuildKit v0.13 [beta] supports rewriting the timestamps in the OCI image layers too • Docs: https://github.com/moby/buildkit/blob/master/docs/build-repro.md OCI = Open Container Initiative buildctl build --opt build-arg:SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) --output type=image,name=example.com/image,push=true,rewrite-timestamp=true docker buildx build --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) --output type=image,name=example.com/image,push=true,rewrite-timestamp=true BuildKit v0.13 is still beta and its CLI is still subject to change until its GA
28 Timestamps • The SOURCE_DATE_EPOCH arg is also propagated to ”RUN” containers as an environment variable • The SOURCE_DATE_EPOCH env var is recognized by gcc, clang, cmake, and a bunch of other tools to make application binaries reproducible: https://reproducible-builds.org/docs/source-date-epoch/
29 Pinning the base image FROM debian
30 Pinning the base image FROM debian:bookworm FROM debian
31 Pinning the base image FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian
32 Pinning the base image FROM debian:bookworm-20230904@sha256:b4042f895d5d1f8df415caebe7c416f9dbcf0dc8867abb225955006de50b21f3 FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian
33 Pinning the base image FROM debian:bookworm-20230904@sha256:b4042f895d5d1f8df415caebe7c416f9dbcf0dc8867abb225955006de50b21f3 FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian apt-get on bookworm-20230904 still installs the latest packages, not the past packages (So, not reproducible)
34 Pinning packages: Debian and Ubuntu FROM debian:bookworm-20230904-slim RUN rm -rf /etc/apt/sources.list* && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20230904T000000Z bookworm main' >/etc/apt/sources.list && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20230904T000000Z bookworm-security main' >>/etc/apt/sources.list && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20230904T000000Z bookworm-updates main' >>/etc/apt/sources.list && apt-get update && apt-get install -y gcc snapshot.debian.org and snapshot.ubuntu.com keep old packages
35 Pinning packages: Debian and Ubuntu FROM debian:bookworm-20230904-slim ADD --chmod=0755 https://raw.githubusercontent.com/reproducible-containers/repro-sources-list.sh/v0.1.0/repro-sources-list.sh /usr/local/bin/repro-sources-list.sh RUN --mount=type=cache,target=/var/cache/apt repro-sources-list.sh && apt-get update && apt-get install -y gcc More examples at: https://github.com/reproducible-containers/repro-sources-list.sh repro-sources-list.sh simplifies the Dockerfile, and enables caching dpkg files Caching is practically necessary, as snapshot servers are slow
36 Pinning packages: Debian and Ubuntu • RUN --mount=type=cache,target=/var/cache/apt can be saved on GitHub Actions using: https://github.com/reproducible-containers/buildkit-cache-dance steps: - uses: actions/cache@v3 with: path: var-cache-apt key: var-cache-apt-${{ hashFiles('Dockerfile') }} - uses: reproducible-containers/buildkit-cache-dance@v2.1.2 with: cache-source: var-cache-apt cache-target: /var/cache/apt
37 Pinning packages: Debian and Ubuntu • The (checksums of the) packages on snapshot.debian.org are signed by Debian, just like regular apt-get repositories • The signatures are fetched and verified against the package metadata checksums on running apt-get update (Not on apt-get install) • If /var/lib/apt (metadata) is compromised, apt-get update will fail • If /var/cache/apt (dpkg files) is compromised, apt-get install will fail • The situation is same for snapshot.ubuntu.com (signed by Canonical)
38 Pinning packages: Debian and Ubuntu • If you don’t trust the latest package signatures, you can reproduce the most of the packages by yourself: https://wiki.debian.org/ReproducibleBuilds/Howto
39 https://tests.reproducible-builds.org/debian/bookworm/index_suite_amd64_stats.html (Captured on 2023-09-24) Reproducibility rate: 95.3%
40 Pinning packages: NixOS • Repro build is much easier with NixOS (although NixOS per se is often considered to be hard to learn) • The flake.lock file contains the checksums of the sources • If the binary is present on cache.nixos.org, the cached binary is used; otherwise the package is built from the source, with very good reproducibility (99.77% for nixos.iso_minimal.x86_64-linux installation, according to https://r13y.com/)
41 Pinning packages: Alpine, Rocky, Alma, etc. • These distros do not provide snapshot servers like snapshot.debian.org • You have to preserve /etc/apk/cache , /var/cache/dnf, etc. by yourself • Examples can be found at: https://github.com/reproducible-containers/repro-pkg-cache • In the long term, BuildKit frontends may have features to help pinning packages: https://github.com/moby/buildkit/issues/4259
42 Future work (Help wanted) • Proposal to make well-known images reproducible (at least for Debian-based ones) • ”Single-click” platform for attesting reproducibility and sharing the result
43 Recap • Repro builds prove that an image is actually buildable from its source • Whether the source is harmless or not is another topic • Ideally, every image should be bit-for-bit reproducible • Practically, subtle differences can be allowed, when they are explainable (e.g., timestamps)
44 Recap Tools and examples: https://github.com/reproducible-containers • diffoci: diff for OCI images, to analyze non-reproducible builds • repro-sources-list.sh: reproducibility helper for Debian, Ubuntu, etc. • repro-pkg-cache: reproducibility helper for Alpine, Alma, Rocky, etc. • buildkit-cache-dance: apt-get cache for GitHub Actions BuildKit docs: https://github.com/moby/buildkit/blob/master/docs/build-repro.md OCI = Open Container Initiative
45 Recap • Slides will be uploaded to https://github.com/AkihiroSuda (README → “Presentation slides”)

More Related Content

PDF
Transparent Data Encryption in PostgreSQL and Integration with Key Management...
Masahiko Sawada
 
PDF
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...
confluent
 
PDF
Secrets of Performance Tuning Java on Kubernetes
Bruno Borges
 
PPTX
Envoy and Kafka
Adam Kotwasinski
 
PDF
PostgreSQLをKubernetes上で活用するためのOperator紹介！（Cloud Native Database Meetup #3 発表資料）
NTT DATA Technology & Innovation
 
PPTX
え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
 
PDF
KubeCon + CloudNativeCon Europe 2022 Recap - Batch/HPCの潮流とScheduler拡張事例 / Kub...
Preferred Networks
 
PPTX
@Indeedeng: RAD - How We Replicate Terabytes of Data Around the World Every Day
indeedeng
 
Transparent Data Encryption in PostgreSQL and Integration with Key Management...
Masahiko Sawada
 
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...
confluent
 
Secrets of Performance Tuning Java on Kubernetes
Bruno Borges
 
Envoy and Kafka
Adam Kotwasinski
 
PostgreSQLをKubernetes上で活用するためのOperator紹介！（Cloud Native Database Meetup #3 発表資料）
NTT DATA Technology & Innovation
 
え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
 
KubeCon + CloudNativeCon Europe 2022 Recap - Batch/HPCの潮流とScheduler拡張事例 / Kub...
Preferred Networks
 
@Indeedeng: RAD - How We Replicate Terabytes of Data Around the World Every Day
indeedeng
 

What's hot (20)

PPTX
nftables: the Next Generation Firewall in Linux
Tomofumi Hayashi
 
PDF
Advanced backup methods (Postgres@CERN)
Anastasia Lubennikova
 
PDF
Azure DevOpsとセキュリティ
Kazushi Kamegawa
 
PDF
Kafka Security 101 and Real-World Tips
confluent
 
PDF
Ingress on Azure Kubernetes Service
Toru Makabe
 
PDF
Learn O11y from Grafana ecosystem.
HungWei Chiu
 
PPTX
iostat await svctm の 見かた、考え方
歩 柴田
 
PPTX
Oracleからamazon auroraへの移行にむけて
Yoichi Sai
 
PDF
[오픈소스컨설팅]엔터프라이즈 오픈소스 도입전략
Ji-Woong Choi
 
PPTX
Kafka Tutorial: Kafka Security
Jean-Paul Azar
 
PDF
30分でRHEL6 High Availability Add-Onを超絶的に理解しよう!
Etsuji Nakai
 
PPTX
SQL Server のロック概要
Oda Shinsuke
 
PPTX
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
Grant McAlister
 
PDF
Cluster management with Kubernetes
Satnam Singh
 
PPTX
Kafka replication apachecon_2013
Jun Rao
 
PDF
Kubernetes Architecture - beyond a black box - Part 1
Hao H. Zhang
 
PDF
グラフデータベース Neptune 使ってみた
Yoshiyasu SAEKI
 
PDF
An Introduction to Apache Kafka
Amir Sedighi
 
PDF
MySQL 5.7の罠があなたを狙っている
yoku0825
 
PDF
PostgreSQL初心者がパッチを提案してからコミットされるまで（第20回PostgreSQLアンカンファレンス@オンライン 発表資料）
NTT DATA Technology & Innovation
 
nftables: the Next Generation Firewall in Linux
Tomofumi Hayashi
 
Advanced backup methods (Postgres@CERN)
Anastasia Lubennikova
 
Azure DevOpsとセキュリティ
Kazushi Kamegawa
 
Kafka Security 101 and Real-World Tips
confluent
 
Ingress on Azure Kubernetes Service
Toru Makabe
 
Learn O11y from Grafana ecosystem.
HungWei Chiu
 
iostat await svctm の 見かた、考え方
歩 柴田
 
Oracleからamazon auroraへの移行にむけて
Yoichi Sai
 
[오픈소스컨설팅]엔터프라이즈 오픈소스 도입전략
Ji-Woong Choi
 
Kafka Tutorial: Kafka Security
Jean-Paul Azar
 
30分でRHEL6 High Availability Add-Onを超絶的に理解しよう!
Etsuji Nakai
 
SQL Server のロック概要
Oda Shinsuke
 
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
Grant McAlister
 
Cluster management with Kubernetes
Satnam Singh
 
Kafka replication apachecon_2013
Jun Rao
 
Kubernetes Architecture - beyond a black box - Part 1
Hao H. Zhang
 
グラフデータベース Neptune 使ってみた
Yoshiyasu SAEKI
 
An Introduction to Apache Kafka
Amir Sedighi
 
MySQL 5.7の罠があなたを狙っている
yoku0825
 
PostgreSQL初心者がパッチを提案してからコミットされるまで（第20回PostgreSQLアンカンファレンス@オンライン 発表資料）
NTT DATA Technology & Innovation
 
Ad

Similar to [DockerCon 2023] Reproducible builds with BuildKit for software supply chain security (20)

PDF
[FOSDEM2023] Bit-for-bit reproducible builds with Dockerfile
Akihiro Suda
 
PPTX
London Node.js User Group - Cloud-native Node.js
Bethany Nicolle Griggs
 
PPTX
Start tracking your ruby infrastructure
Sergiy Kukunin
 
PDF
classdockerimages.pdf
WaiYipLiew
 
PPTX
Effective images remix
🎥 Brent Langston
 
PDF
Silicon Valley Code Camp 2019 - Reaching the Cloud Native World
Chris Bailey
 
PDF
Node Summit 2018: Cloud Native Node.js
Chris Bailey
 
PDF
Comparing Next-Generation Container Image Building Tools
Akihiro Suda
 
PDF
The Challenges of Container Configuration
Gareth Rushgrove
 
PDF
Debugging Complex Systems - Erlang Factory SF 2015
lpgauth
 
PPTX
Docker for Development
allingeek
 
PPTX
Cloud native buildpacks_collabnix
Suman Chakraborty
 
PDF
FullStack London - Cloud Native Node.js
Bethany Nicolle Griggs
 
PDF
Using Docker Containers to Improve Reproducibility in Software and Web Engine...
Vincenzo Ferme
 
PDF
Using Docker For Development
Laura Frank Tacho
 
PDF
Docker by Example - Quiz
CodeOps Technologies LLP
 
PDF
Docker by Example - Quiz
Ganesh Samarthyam
 
PDF
Docker & FieldAware
Jakub Jarosz
 
PDF
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
apidays
 
PDF
Optimizing Docker Images
Brian DeHamer
 
[FOSDEM2023] Bit-for-bit reproducible builds with Dockerfile
Akihiro Suda
 
London Node.js User Group - Cloud-native Node.js
Bethany Nicolle Griggs
 
Start tracking your ruby infrastructure
Sergiy Kukunin
 
classdockerimages.pdf
WaiYipLiew
 
Effective images remix
🎥 Brent Langston
 
Silicon Valley Code Camp 2019 - Reaching the Cloud Native World
Chris Bailey
 
Node Summit 2018: Cloud Native Node.js
Chris Bailey
 
Comparing Next-Generation Container Image Building Tools
Akihiro Suda
 
The Challenges of Container Configuration
Gareth Rushgrove
 
Debugging Complex Systems - Erlang Factory SF 2015
lpgauth
 
Docker for Development
allingeek
 
Cloud native buildpacks_collabnix
Suman Chakraborty
 
FullStack London - Cloud Native Node.js
Bethany Nicolle Griggs
 
Using Docker Containers to Improve Reproducibility in Software and Web Engine...
Vincenzo Ferme
 
Using Docker For Development
Laura Frank Tacho
 
Docker by Example - Quiz
CodeOps Technologies LLP
 
Docker by Example - Quiz
Ganesh Samarthyam
 
Docker & FieldAware
Jakub Jarosz
 
Apidays Paris 2023 - Forget TypeScript, Choose Rust to build Robust, Fast and...
apidays
 
Optimizing Docker Images
Brian DeHamer
 
Ad

More from Akihiro Suda (20)

PDF
20250617 [KubeCon JP 2025] containerd - Project Update and Deep Dive.pdf
Akihiro Suda
 
PDF
20250616 [KubeCon JP 2025] VexLLM - Silence Negligible CVE Alerts Using LLM.pdf
Akihiro Suda
 
PDF
20250403 [KubeCon EU] containerd - Project Update and Deep Dive.pdf
Akihiro Suda
 
PDF
20250403 [KubeCon EU Pavilion] containerd.pdf
Akihiro Suda
 
PDF
20250402 [KubeCon EU Pavilion] Lima.pdf_
Akihiro Suda
 
PDF
20241115 [KubeCon NA Pavilion] Lima.pdf_
Akihiro Suda
 
PDF
20241113 [KubeCon NA Pavilion] containerd.pdf
Akihiro Suda
 
PDF
【情報科学若手の会 (2024/09/14】なぜオープンソースソフトウェアにコントリビュートすべきなのか
Akihiro Suda
 
PDF
【Vuls祭り#10 (2024/08/20)】 VexLLM: LLMを用いたVEX自動生成ツール
Akihiro Suda
 
PDF
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
Akihiro Suda
 
PDF
20240321 [KubeCon EU Pavilion] Lima.pdf_
Akihiro Suda
 
PDF
20240320 [KubeCon EU Pavilion] containerd.pdf
Akihiro Suda
 
PDF
20240201 [HPC Containers] Rootless Containers.pdf
Akihiro Suda
 
PDF
[Podman Special Event] Kubernetes in Rootless Podman
Akihiro Suda
 
PDF
[KubeConNA2023] Lima pavilion
Akihiro Suda
 
PDF
[KubeConNA2023] containerd pavilion
Akihiro Suda
 
PDF
[DockerConハイライト] OpenPubKeyによるイメージの署名と検証.pdf
Akihiro Suda
 
PDF
[CNCF TAG-Runtime] Usernetes Gen2
Akihiro Suda
 
PDF
The internals and the latest trends of container runtimes
Akihiro Suda
 
PDF
[KubeConEU2023] Lima pavilion
Akihiro Suda
 
20250617 [KubeCon JP 2025] containerd - Project Update and Deep Dive.pdf
Akihiro Suda
 
20250616 [KubeCon JP 2025] VexLLM - Silence Negligible CVE Alerts Using LLM.pdf
Akihiro Suda
 
20250403 [KubeCon EU] containerd - Project Update and Deep Dive.pdf
Akihiro Suda
 
20250403 [KubeCon EU Pavilion] containerd.pdf
Akihiro Suda
 
20250402 [KubeCon EU Pavilion] Lima.pdf_
Akihiro Suda
 
20241115 [KubeCon NA Pavilion] Lima.pdf_
Akihiro Suda
 
20241113 [KubeCon NA Pavilion] containerd.pdf
Akihiro Suda
 
【情報科学若手の会 (2024/09/14】なぜオープンソースソフトウェアにコントリビュートすべきなのか
Akihiro Suda
 
【Vuls祭り#10 (2024/08/20)】 VexLLM: LLMを用いたVEX自動生成ツール
Akihiro Suda
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
Akihiro Suda
 
20240321 [KubeCon EU Pavilion] Lima.pdf_
Akihiro Suda
 
20240320 [KubeCon EU Pavilion] containerd.pdf
Akihiro Suda
 
20240201 [HPC Containers] Rootless Containers.pdf
Akihiro Suda
 
[Podman Special Event] Kubernetes in Rootless Podman
Akihiro Suda
 
[KubeConNA2023] Lima pavilion
Akihiro Suda
 
[KubeConNA2023] containerd pavilion
Akihiro Suda
 
[DockerConハイライト] OpenPubKeyによるイメージの署名と検証.pdf
Akihiro Suda
 
[CNCF TAG-Runtime] Usernetes Gen2
Akihiro Suda
 
The internals and the latest trends of container runtimes
Akihiro Suda
 
[KubeConEU2023] Lima pavilion
Akihiro Suda
 

Recently uploaded (20)

PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PPTX
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
Transform Your Business with a Software ERP System
Canada Software Company
 
AI in Product Development-omnex systems
omnex systems
 
Introduction Database Management System for Course Database
ReinertYosua
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 

[DockerCon 2023] Reproducible builds with BuildKit for software supply chain security

  • 1. Reproducible builds with BuildKit for software supply chain security Akihiro Suda Software Engineer | NTT
  • 2. 2 Background • Security assessment has been hard for Docker images, due to lack of verifiability in the software supply chain • Even when the source code (Dockerfile) is public, and the source code appears to be harmless, it is hard to prove that the image is actually buildable from the source code • Reproducible builds help proving it (But whether the source code is harmless is another topic)
  • 3. 3 What are Reproducible Builds? • Same source, same binary • Attestable by anybody • Attestable at anytime Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67 sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67 sha256:6ea7098583cb6c9470570df28c154 cfec58e122188382cd4a7ceab8a9a79cb67
  • 4. 4 Why do we need reproducible builds? Because non-reproducible builds cannot be proved to be buildable from harmless sources
  • 5. 5 Why do we need reproducible builds? sha256:AAAAA… Upstream build docker pull some-image Pull Because non-reproducible builds cannot be proved to be buildable from harmless sources
  • 6. 6 Because non-reproducible builds cannot be proved to be buildable from harmless sources Why do we need reproducible builds? FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… Upstream build docker pull some-image Pull Find the source repo
  • 7. 7 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:BBBBB… Upstream build docker pull some-image Pull Find the source repo Because non-reproducible builds cannot be proved to be buildable from harmless sources Your own build (Non-reproducible)
  • 8. 8 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:BBBBB… Upstream build docker pull some-image Pull Find the source repo Your own build (Non-reproducible) Is this image really buildable from the source repo? Because non-reproducible builds cannot be proved to be buildable from harmless sources
  • 9. 9 Why do we need reproducible builds? Build FROM debian RUN apt-get install -y gcc make ... COPY . . RUN make sha256:AAAAA… sha256:AAAAA… Upstream build docker pull some-image Pull Find the source repo Proved to be buildable from the source Because non-reproducible builds cannot be proved to be buildable from harmless sources Your own build (Reproducible)
  • 10. 10 Why do we need reproducible builds? • Reproducibility per se doesn’t prove any harmlessness • Non-reproducibility doesn’t prove any harmfulness, either
  • 11. 11 Why do we need reproducible builds? • Reproducibility proves that the image is actually buildable from its source • The source still has to be reviewed • The source may still be malicious • But at least the image contains no secret code that you can never review
  • 12. 12 Docker Hub images are actually reproducible?
  • 13. 13 Docker Hub images are actually reproducible? • No, mostly
  • 14. 14 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang
  • 15. 15 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang DOCKER_BUILDKIT=0 with Docker 20.10.23 corresponds to the current Docker Hub image (Will change in the future)
  • 16. 16 Docker Hub images are actually reproducible? $ docker pull golang:1.21.1-alpine@ sha256:96634e55b363cb93d39f78fb18aa64abc7f96d372c176660d7b8b6118939d97b $ DOCKER_BUILDKIT=0 docker build -t my-golang "https://github.com/docker-library/golang.git# 585c8c1e705a7a458455f0629922a4f90628ce08:1.21/alpine3.18” $ go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang DiffOCI: diff for Open Container Initiative (OCI) images https://github.com/reproducible-containers/diffoci
  • 17. 17 Docker Hub images are actually reproducible? $ diffoci diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Desc application/vnd.docker.distribution.manifest.v2+json b25862... 3c4eca0... ... File etc/ssl/certs/3e45d192.0 2023-08-09 03:36:47 +0000 UTC 2023-09-21 08:35:31 +0000 UTC ... (More than 14,000 lines) ... File go/ 2023-09-06 18:31:40 +0000 UTC 2023-09-21 08:35:45 +0000 UTC
  • 18. 18 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
  • 19. 19 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
  • 20. 20 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference lib/apk/db/scripts.tar differ due to the timestamp information inside scripts.tar (the “--semantic” flag isn’t still clever enough to ignore this “boring” difference”) The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
  • 21. 21 Docker Hub images are actually reproducible? $ diffoci --semantic diff docker://golang:1.21.1-alpine docker://my-golang TYPE NAME INPUT-0 INPUT-1 Layer ctx:/layers-1/layer length mismatch (457 vs 454) Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0 Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar eef110e... e9bfe18... Layer ctx:/layers-2/layer length mismatch (13939 vs 13938) Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0 File lib/apk/db/scripts.tar 60e22bb... 67f2648... Layer ctx:/layers-3/layer length mismatch (4 vs 3) Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0 “.wh..wh..opq” (AUFS whiteouts) are missing due to the filesystem difference lib/apk/db/scripts.tar differ due to the timestamp information inside scripts.tar (the “--semantic” flag isn’t still clever enough to ignore this “boring” difference”) This image is not fully reproducible, but its non-reproducibility is explainable (So, this image appears to be actually buildable from the public Dockerfile) The “--semantic” flag ignores ”boring” differences (timestamps, file ordering, etc.)
  • 22. 22 Why are images not reproducible? • Timestamps • Version of the base image (“FROM” images in Dockerfiles) • Versions of the packages (apt-get, pip, etc.) • Others: - Filesystem characteristics (e.g., OverlayFS) - Ordering of files - Randomized mktemp, etc.
  • 23. 23 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
  • 24. 24 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
  • 25. 25 Timestamps • BuildKit (since v0.11) supports rewriting the timestamps for OCI Image Config and OCI Index • Support was incomplete in v0.11 and v0.12; using v0.13 [beta] is recommended (see the next couple of slides) OCI = Open Container Initiative buildctl build --opt build-arg:SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) docker buildx build --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) Unix epoch (int64, seconds from 1970-01-01 00:00:00 UTC)
  • 26. 26 Timestamps • The images have timestamps in: - the “created” property in the OCI Image Config - the “history” property in the OCI Image Config - the “org.opencontainers.image.created” annotation in the OCI Index - the timestamps of the files in the image layers OCI = Open Container Initiative
  • 27. 27 Timestamps • BuildKit v0.13 [beta] supports rewriting the timestamps in the OCI image layers too • Docs: https://github.com/moby/buildkit/blob/master/docs/build-repro.md OCI = Open Container Initiative buildctl build --opt build-arg:SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) --output type=image,name=example.com/image,push=true,rewrite-timestamp=true docker buildx build --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) --output type=image,name=example.com/image,push=true,rewrite-timestamp=true BuildKit v0.13 is still beta and its CLI is still subject to change until its GA
  • 28. 28 Timestamps • The SOURCE_DATE_EPOCH arg is also propagated to ”RUN” containers as an environment variable • The SOURCE_DATE_EPOCH env var is recognized by gcc, clang, cmake, and a bunch of other tools to make application binaries reproducible: https://reproducible-builds.org/docs/source-date-epoch/
  • 29. 29 Pinning the base image FROM debian
  • 30. 30 Pinning the base image FROM debian:bookworm FROM debian
  • 31. 31 Pinning the base image FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian
  • 32. 32 Pinning the base image FROM debian:bookworm-20230904@sha256:b4042f895d5d1f8df415caebe7c416f9dbcf0dc8867abb225955006de50b21f3 FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian
  • 33. 33 Pinning the base image FROM debian:bookworm-20230904@sha256:b4042f895d5d1f8df415caebe7c416f9dbcf0dc8867abb225955006de50b21f3 FROM debian:bookworm-20230904 FROM debian:bookworm FROM debian apt-get on bookworm-20230904 still installs the latest packages, not the past packages (So, not reproducible)
  • 34. 34 Pinning packages: Debian and Ubuntu FROM debian:bookworm-20230904-slim RUN rm -rf /etc/apt/sources.list* && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20230904T000000Z bookworm main' >/etc/apt/sources.list && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20230904T000000Z bookworm-security main' >>/etc/apt/sources.list && echo 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20230904T000000Z bookworm-updates main' >>/etc/apt/sources.list && apt-get update && apt-get install -y gcc snapshot.debian.org and snapshot.ubuntu.com keep old packages
  • 35. 35 Pinning packages: Debian and Ubuntu FROM debian:bookworm-20230904-slim ADD --chmod=0755 https://raw.githubusercontent.com/reproducible-containers/repro-sources-list.sh/v0.1.0/repro-sources-list.sh /usr/local/bin/repro-sources-list.sh RUN --mount=type=cache,target=/var/cache/apt repro-sources-list.sh && apt-get update && apt-get install -y gcc More examples at: https://github.com/reproducible-containers/repro-sources-list.sh repro-sources-list.sh simplifies the Dockerfile, and enables caching dpkg files Caching is practically necessary, as snapshot servers are slow
  • 36. 36 Pinning packages: Debian and Ubuntu • RUN --mount=type=cache,target=/var/cache/apt can be saved on GitHub Actions using: https://github.com/reproducible-containers/buildkit-cache-dance steps: - uses: actions/cache@v3 with: path: var-cache-apt key: var-cache-apt-${{ hashFiles('Dockerfile') }} - uses: reproducible-containers/buildkit-cache-dance@v2.1.2 with: cache-source: var-cache-apt cache-target: /var/cache/apt
  • 37. 37 Pinning packages: Debian and Ubuntu • The (checksums of the) packages on snapshot.debian.org are signed by Debian, just like regular apt-get repositories • The signatures are fetched and verified against the package metadata checksums on running apt-get update (Not on apt-get install) • If /var/lib/apt (metadata) is compromised, apt-get update will fail • If /var/cache/apt (dpkg files) is compromised, apt-get install will fail • The situation is same for snapshot.ubuntu.com (signed by Canonical)
  • 38. 38 Pinning packages: Debian and Ubuntu • If you don’t trust the latest package signatures, you can reproduce the most of the packages by yourself: https://wiki.debian.org/ReproducibleBuilds/Howto
  • 40. 40 Pinning packages: NixOS • Repro build is much easier with NixOS (although NixOS per se is often considered to be hard to learn) • The flake.lock file contains the checksums of the sources • If the binary is present on cache.nixos.org, the cached binary is used; otherwise the package is built from the source, with very good reproducibility (99.77% for nixos.iso_minimal.x86_64-linux installation, according to https://r13y.com/)
  • 41. 41 Pinning packages: Alpine, Rocky, Alma, etc. • These distros do not provide snapshot servers like snapshot.debian.org • You have to preserve /etc/apk/cache , /var/cache/dnf, etc. by yourself • Examples can be found at: https://github.com/reproducible-containers/repro-pkg-cache • In the long term, BuildKit frontends may have features to help pinning packages: https://github.com/moby/buildkit/issues/4259
  • 42. 42 Future work (Help wanted) • Proposal to make well-known images reproducible (at least for Debian-based ones) • ”Single-click” platform for attesting reproducibility and sharing the result
  • 43. 43 Recap • Repro builds prove that an image is actually buildable from its source • Whether the source is harmless or not is another topic • Ideally, every image should be bit-for-bit reproducible • Practically, subtle differences can be allowed, when they are explainable (e.g., timestamps)
  • 44. 44 Recap Tools and examples: https://github.com/reproducible-containers • diffoci: diff for OCI images, to analyze non-reproducible builds • repro-sources-list.sh: reproducibility helper for Debian, Ubuntu, etc. • repro-pkg-cache: reproducibility helper for Alpine, Alma, Rocky, etc. • buildkit-cache-dance: apt-get cache for GitHub Actions BuildKit docs: https://github.com/moby/buildkit/blob/master/docs/build-repro.md OCI = Open Container Initiative
  • 45. 45 Recap • Slides will be uploaded to https://github.com/AkihiroSuda (README → “Presentation slides”)