P2P Container Image Distribution on IPFS With containerd and nerdctl
1 like1,432 views
The document discusses the experimental support of nerdctl for peer-to-peer (p2p) container image distribution on IPFS, highlighting its user-friendly interface and integration with tools like Kubernetes and buildkit. It addresses challenges in traditional image distribution methods, such as slow pull times and registry dependency, offering a solution that allows for faster, p2p image sharing from bandwidth-limited seeders. The document also mentions compatibility with existing OCI image distribution techniques, including lazy pulling and image encryption.
![Copyright(c)2022 NTT Corp. All Rights Reserved.
Problems in image distribu1on
3
Registry
push pull
pusher node receiver node
l Pulling is time-consuming
• Pulling packages accounts for 76% of container start
time [Harter et al. 2016]
• Can be slower under limited bandwidth between
registry and node
l Images can’t be shared if no access to the registry (e.g.,
registry outage, rate limited, no access to the internet, …)](https://image.slidesharecdn.com/slides-220207015141/85/P2P-Container-Image-Distribution-on-IPFS-With-containerd-and-nerdctl-3-320.jpg)
![Copyright(c)2022 NTT Corp. All Rights Reserved.
Configura)on of OCI image for IPFS
7
{
“schemaVersion”: 2,
“manifests”: [
{
“mediaType”: “application/vnd.oci.image.manifest.v1+json”,
“digest”: “sha256:f6eed19a2880f1000be1d46fb5d114d094a59e350f9d025580f7297c8d9527d5”,
“size”: 506,
“urls”: [
“ipfs://bafkreihw53izukea6eaaxyoun625cfgqsssz4niptubflahxff6i3fjh2u”
],
・・・
application/vnd.oci.image.
manifest.v1+json
application/vnd.oci.
image.config.v1+json
application/vnd.oci.image.
layer.v1.tar+gzip
CID
CID
{
"mediaType": "application/vnd.oci.image.index.v1+json",
"digest": "sha256:28bfa1fc6d491d3bee91bab451cab29c747e72917efacb0adc4e73faffe1f51c",
"size": 313,
"urls": [
"ipfs://bafkreibix6q7y3kjdu565en2wri4vmu4or7hfel67lfqvxcoop5p7ypvdq"
]
}
ipfs:// bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
application/vnd.oci.image.
index.v1+json
CID
Each item in OCI image supports arbitrary URLs as the data source
→ we store CID (formed as IPFS URL)
l Constructing DAG by CIDs
l Image is referenced by CID of the topmost “OCI descriptor” JSON
OCI descriptor JSON](https://image.slidesharecdn.com/slides-220207015141/85/P2P-Container-Image-Distribution-on-IPFS-With-containerd-and-nerdctl-7-320.jpg)
![Copyright(c)2022 NTT Corp. All Rights Reserved.
Lazy pulling: eStargz
16
l Lazy pulling: Starting up containers without waiting for the pull completion
• Each chunk/file in the image is downloaded on-demand
l eStargz: OCI-compatible image format for lazy pulling with prefetch support
• Can be lazily pulled from standard registries
l Stargz Snapshotter: Plugin of containerd for enabling lazy pulling
https://github.com/containerd/stargz-snapshotter
0 5 10 15 20 25 30 35 40 45
estargz
estargz-noopt
legacy
Start up time of python:3.7 (print “hello”)
pull create run [sec]
Figure from “Faster Container Image Distribution on a Variety of Tools with Lazy Pulling - Kohei Tokunaga & Tao Peng.
KubeCon+CloudNativeCon North America 2021. https://sched.co/lV2a “](https://image.slidesharecdn.com/slides-220207015141/85/P2P-Container-Image-Distribution-on-IPFS-With-containerd-and-nerdctl-16-320.jpg)
![Copyright(c)2022 NTT Corp. All Rights Reserved.
Related projects about image distribution on IPFS
21
l ipcs: https://github.com/hinshun/ipcs
• Proposed by Edgar Lee (Netflix)
• containerd content store plugin backed by IPFS
• Focuses on content deduplication but incompatible to OCI image
l ipdr: https://github.com/ipdr/ipdr
• Proposed by Miguel Mota
• Docker registry backed by IPFS
• No native integration with runtime (requires a dedicated CLI)
• Lazy pulling unsupported
l EdgePier[1]
• Proposed by Soeren Becker, et al.
• Integrated ipdr (mentioned in the above) with Kubernetes
• Fast image distribution under bandwidth-restricted environment
• No OSS implementation
[1] Soeren Becker, et al."EdgePier: P2P-based Container Image Distribution in Edge Computing Environments". IEEE International Performance
Computing and Communications Conference 2021](https://image.slidesharecdn.com/slides-220207015141/85/P2P-Container-Image-Distribution-on-IPFS-With-containerd-and-nerdctl-21-320.jpg)
P2P Container Image Distribution on IPFS With containerd and nerdctl
- 1. Copyright(c)2022 NTT Corp. All Rights Reserved. P2P Container Image Distribution on IPFS With containerd and nerdctl Kohei Tokunaga, NTT Corporation FOSDEM 2022 (February 6)
- 2. Copyright(c)2022 NTT Corp. All Rights Reserved. Summary 2 l nerdctl experimentally supports P2P image distribution on IPFS • simple UI/UX for P2P • allows IPFS-agnostic tools to get images from IPFS (e.g. BuildKit, Kubernetes) • fast image distribution from bandwidth-limited seeder l Combination with existing OCI image distribution techniques • lazy pulling of eStargz • distributing encrypted image by OCIcrypt Registry nerdctl push nerdctl pull pusher node receiver node peer peer nerdctl push nerdctl pull Registry-based image distribution IPFS-based image distribution IPFS
- 3. Copyright(c)2022 NTT Corp. All Rights Reserved. Problems in image distribu1on 3 Registry push pull pusher node receiver node l Pulling is time-consuming • Pulling packages accounts for 76% of container start time [Harter et al. 2016] • Can be slower under limited bandwidth between registry and node l Images can’t be shared if no access to the registry (e.g., registry outage, rate limited, no access to the internet, …)
- 4. Copyright(c)2022 NTT Corp. All Rights Reserved. IPFS-based P2P image distribution with nerdctl 4 l nerdctl CLI (>= v0.14) of containerd experimentally supports image distribution on IPFS • Images are shared in P2P manner without relying on the centralized registry • Simple UI/UX for P2P image distribution • Fast image distribution from bandwidth-limited seeder (discussed later) Registry nerdctl push nerdctl pull pusher node receiver node peer peer nerdctl push nerdctl pull Registry-based image distribution IPFS-based image distribution IPFS
- 5. Copyright(c)2022 NTT Corp. All Rights Reserved. nerdctl: Docker-compatible CLI of containerd 5 l Has same UI/UX as Docker https://github.com/containerd/nerdctl l Supports cutting-edge features • rootless • lazy-pulling (eStargz) • encrypted images (OCIcrypt) • P2P image distribution (IPFS) • container image signing and verifying (cosign) l Adopted by lima and Rancher Desktop • container management tool for desktop • https://medium.com/nttlabs/containerd-and-lima-39e0b64d2a59 nerdctl CLI nerdctl run -it --rm alpine nerdctl build -t foo /dockerfile-dir nerdctl push ghcr.io/ktock/myalpine:latest containerd API
- 6. Copyright(c)2022 NTT Corp. All Rights Reserved. IPFS 6 l P2P and content addressable data sharing protocol l No central server is needed l Content addressable by CID h@ps://ipfs.io ipfs add myfile.txt ipfs get QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN Content Addressable by CID (identifier based on the content’s hash) peer peer IPFS
- 7. Copyright(c)2022 NTT Corp. All Rights Reserved. Configura)on of OCI image for IPFS 7 { “schemaVersion”: 2, “manifests”: [ { “mediaType”: “application/vnd.oci.image.manifest.v1+json”, “digest”: “sha256:f6eed19a2880f1000be1d46fb5d114d094a59e350f9d025580f7297c8d9527d5”, “size”: 506, “urls”: [ “ipfs://bafkreihw53izukea6eaaxyoun625cfgqsssz4niptubflahxff6i3fjh2u” ], ・・・ application/vnd.oci.image. manifest.v1+json application/vnd.oci. image.config.v1+json application/vnd.oci.image. layer.v1.tar+gzip CID CID { "mediaType": "application/vnd.oci.image.index.v1+json", "digest": "sha256:28bfa1fc6d491d3bee91bab451cab29c747e72917efacb0adc4e73faffe1f51c", "size": 313, "urls": [ "ipfs://bafkreibix6q7y3kjdu565en2wri4vmu4or7hfel67lfqvxcoop5p7ypvdq" ] } ipfs:// bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze application/vnd.oci.image. index.v1+json CID Each item in OCI image supports arbitrary URLs as the data source → we store CID (formed as IPFS URL) l Constructing DAG by CIDs l Image is referenced by CID of the topmost “OCI descriptor” JSON OCI descriptor JSON
- 8. Copyright(c)2022 NTT Corp. All Rights Reserved. Adding an image to IPFS 8 nerdctl push ipfs://ubuntu:20.04 l nerdctl supports ipfs:// prefix for an arbitrary image name l nerdctl pushes the image to IPFS instead of registry • Automatically configures the OCI image for IPFS (see previous slide) l The image is distributed on IPFS in a p2p manner without registry peer peer IPFS
- 9. Copyright(c)2022 NTT Corp. All Rights Reserved. Pulling an image from IPFS 9 nerdctl pull ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze l ipfs://CID references an image on IPFS l nerdctl gets the image from IPFS instead of the registry l The image needs to be configured for IPFS • “nerdctl push ipfs://” automatically does this nerdctl run ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze peer peer IPFS
- 10. Copyright(c)2022 NTT Corp. All Rights Reserved. Building image based on images on IPFS 10 FROM localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze RUN echo hello > /hello Dockerfile l localhost:5050/ipfs/CID references an image on IPFS • Dockerfile should support “ipfs://CID” image reference in the future l Base image is acquired from IPFS l The result image can also be pushed to IPFS using “nerdctl push ipfs://” peer peer IPFS
- 11. Copyright(c)2022 NTT Corp. All Rights Reserved. IPFS-based image distribution for IPFS-agnostic tools 11 l Provides a read-only localhost registry backed by IPFS • image is accessible via localhost:5050/ipfs/CID l IPFS-agnostic tools (e.g. Kubernetes) can pull images from IPFS • nerdctl build (backed by BuildKit) uses this functionality nerdctl ipfs registry ・・・ IPFS-agnostic tools peer IPFS node RO Registry API (via localhost) nerdctl ipfs registry subcommand localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze Kubernetes CRI BuildKit
- 12. Copyright(c)2022 NTT Corp. All Rights Reserved. Example: node-to-node image sharing on Kubernetes 12 nerdctl ipfs registry DaemonSet Pull from IPFS nerdctl ipfs registry DaemonSet Pull from IPFS node node l “nerdctl ipfs registry” can be used for node-to-node image sharing l In the future, Kubernetes should support “ipfs://CID” image reference l Example configuration: running ipfs daemon as DaemonSet on each node https://github.com/containerd/nerdctl/pull/678 Kubernetes cluster IPFS IPFS node (seeder) l nerdctl build l nerdctl push ipfs://… l ・・・ Build image, push it to IPFS, … share images among nodes IPFS daemon IPFS daemon IPFS IPFS ・・・ Distribute images via IPFS (WIP)
- 13. Copyright(c)2022 NTT Corp. All Rights Reserved. Image distribution latency 13 l GKE v1.21.5-gke.1302 (20 nodes) • instance: e2-standard-8 (asia-northeast1-a) • OS: ubuntu_containerd (upgraded containerd to v1.5.8 manually) l private seeder/registry (1 node) • instance: e2-standard-8 (asia-northeast1-a) • OS: Ubuntu 20.04 l image: ghcr.io/stargz-containers/jenkins:2.60.3-org (726.4 MiB) l Measured the worst time to take for pull with configuring bandwidth using linux tc l commit: https://github.com/containerd/nerdctl/commit/3b5ed0df186d05d986b9cdb7c47773f29febed29 • ipfs v0.11.0 (k8s nodes), ipfs v0.10.0 (seeder), nerdctl bb682bc l benchmarking script: https://github.com/ktock/stargz-snapshotter/tree/nerdctl-ipfs-registry-kubernetes- benchmark/script/nerdctl-ipfs-registry-kubernetes-benchmark Measured Zme to take to distribute images under several bandwidth situaZons IPFS node (seeder) 20 nodes Kubernetes cluster IPFS nodes share images via IPFS container registry registry API 20 nodes Kubernetes cluster nodes pull images from registry several bandwidth situations several bandwidth situations
- 14. Copyright(c)2022 NTT Corp. All Rights Reserved. 0 10000 20000 30000 40000 50000 60000 70000 1 5 10 15 20 time to distribute image (msec) number of images to pull 955 Mbits/sec registry ipfs 0 20000 40000 60000 80000 100000 120000 1 5 10 15 20 Jme to distribute image (msec) number of images to pull 478 Mbits/sec registry ipfs 0 5000 10000 15000 20000 25000 1 5 10 15 20 time to distribute image (msec) number of images to pull 3.82 Gbits/sec registry ipfs Image distribution latency 14 l On lower bandwidth with many images, IPFS distributes images faster than registry l On higher bandwidth or with small number of images, IPFS can be slower than registry • will work on further invesJgaJon and miJgaJon lower is better lower is better lower is better
- 15. Copyright(c)2022 NTT Corp. All Rights Reserved. Combination with existing image distribution techniques 15
- 16. Copyright(c)2022 NTT Corp. All Rights Reserved. Lazy pulling: eStargz 16 l Lazy pulling: Starting up containers without waiting for the pull completion • Each chunk/file in the image is downloaded on-demand l eStargz: OCI-compatible image format for lazy pulling with prefetch support • Can be lazily pulled from standard registries l Stargz Snapshotter: Plugin of containerd for enabling lazy pulling https://github.com/containerd/stargz-snapshotter 0 5 10 15 20 25 30 35 40 45 estargz estargz-noopt legacy Start up time of python:3.7 (print “hello”) pull create run [sec] Figure from “Faster Container Image Distribution on a Variety of Tools with Lazy Pulling - Kohei Tokunaga & Tao Peng. KubeCon+CloudNativeCon North America 2021. https://sched.co/lV2a “
- 17. Copyright(c)2022 NTT Corp. All Rights Reserved. Lazy pulling (eStargz) on IPFS 17 l eStargz can be stored to IPFS l Stargz Snapshotter supports lazy pulling of eStargz from IPFS • mounts eStargz image from IPFS to container’s rootfs l Chunks are fetched lazily thus hopefully faster cold-start https://github.com/containerd/stargz-snapshotter proc container Node Stargz Snapshotter Fetching files/chunks on demand Moun@ng rooAs as FUSE Lazy pulling eStargz image IPFS Peer nerdctl
- 18. Copyright(c)2022 NTT Corp. All Rights Reserved. Image encryption: OCIcrypt 18 l nerdctl supports encryp<on/decryp<on of image layers with key pair l OCIcrypt (imgcrypt plugin for containerd) is used Registry nerdctl image encrypt nerdctl image decrypt image image https://github.com/containerd/imgcrypt Public key Private key encrypts an image decrypts an image
- 19. Copyright(c)2022 NTT Corp. All Rights Reserved. Image encryption (OCICrypt) on IPFS 19 l Encrypted image can be pushed to IPFS l Configuration of the image for IPFS (urls field) is done by “nerdctl push ipfs://” nerdctl image encrypt nerdctl image decrypt image image IPFS Public key Private key
- 20. Copyright(c)2022 NTT Corp. All Rights Reserved. Future works 20 l Performance improvements • Especially on high bandwidth environment with small number of images • Pulling hangs when the searching image isn’t found on IPFS l Enabling “ipfs://CID” reference on a various tools (e.g. BuildKit, Kubernetes, …) l CID reproducibility • “nerdctl push ipfs://” will produce different CIDs for different configurations of IPFS • e.g. different chunk size l Higher availability of data • “pinning services” can be used maybe l …
- 21. Copyright(c)2022 NTT Corp. All Rights Reserved. Related projects about image distribution on IPFS 21 l ipcs: https://github.com/hinshun/ipcs • Proposed by Edgar Lee (Netflix) • containerd content store plugin backed by IPFS • Focuses on content deduplication but incompatible to OCI image l ipdr: https://github.com/ipdr/ipdr • Proposed by Miguel Mota • Docker registry backed by IPFS • No native integration with runtime (requires a dedicated CLI) • Lazy pulling unsupported l EdgePier[1] • Proposed by Soeren Becker, et al. • Integrated ipdr (mentioned in the above) with Kubernetes • Fast image distribution under bandwidth-restricted environment • No OSS implementation [1] Soeren Becker, et al."EdgePier: P2P-based Container Image Distribution in Edge Computing Environments". IEEE International Performance Computing and Communications Conference 2021
- 22. Copyright(c)2022 NTT Corp. All Rights Reserved. Summary 22 l nerdctl experimentally supports P2P image distribution on IPFS • simple UI/UX for P2P • allows IPFS-agnostic tools to get images from IPFS (e.g. BuildKit, Kubernetes) • fast image distribution from bandwidth-limited seeder l Combination with existing OCI image distribution techniques • lazy pulling of eStargz, image encryption by OCIcrypt Registry nerdctl push nerdctl pull pusher node receiver node peer peer nerdctl push nerdctl pull Registry-based image distribution IPFS-based image distribution IPFS Thanks to Akihiro Suda (NTT) for the discussion!