Navigating container technology for enhanced security by Niklas Saari
0 likes37 views
The document discusses the evolution and optimization of container technology for enhanced security, emphasizing the importance of managing dependencies and minimizing image sizes to reduce vulnerabilities in software supply chains. It outlines techniques such as multi-stage builds and the use of distroless images to improve the security and efficiency of container deployments. The speaker also emphasizes the need for best practices like principle of least privilege and the management of software bills of materials (SBOM) to enhance security when using containers.
Navigating container technology for enhanced security by Niklas Saari
- 1. Navigating Container Technology for Enhanced Security Niklas Saari 21.11.2023 1 / 34
- 2. About speaker ██ Niklas Saari • Doctoral Researcher in the University of Oulu ◦ Oulu University Secure Programming Group (OUSPG) ◦ The topic is about the impact of open-source software for software security of the satellite systems ▪ Software supply chain security ▪ Software quality analysis with fuzzing and static methods ██ About containers In the past, maintained and developed around three years many containerized information security tools as part of CinCan project. Also, container hobbyist... 2 / 34
- 3. What is this all about • What are containers and how do we optimise them? • How about the role of dependencies for software supply chain security? • Focus on how to optimise your image build process and what it means • Adapt your thinking of security when you operate with containers! The logo of Docker Inc. became well-known symbol of containers 3 / 34
- 4. Containers? • Usually, we use them for isolation • We like to control resource usages in context basis • Lightweight; they are actually just processes • Make application build process and deployment portable, shareable and maybe even reproducible • With higher usability than traditional solutions • Usually we mean Linux-based on containers, when we talk about them 4 / 34
- 5. You can even make Linux container from Windows games Magic The Gathering: Arena works well when played from the Linux container and many others. • Managing different runtime dependencies more convenient • For example, you can have different GPU driver for every game! 5 / 34
- 6. Evolution of modern containers on Linux ——————————————————————————————————————————————————————————————————————————————————————————————————————————— • 1979 - chroot was introduced • 2002 - First version of Linux Namespaces • 2006 - "Process Containers" by Google, merged to Linux Kernel as cgroups, (Control Groups) • 2008 - LXC (LinuX Containers), first, complete implementation • 2013-2015 - Let Me Contain That For You (LMCTFY) Google Container Stack - influenced the modern libcontainer • 2013 - Docker, initially based on LXC, later created own libcontainer • NOW - Docker emerged to so-called Open Container Inititiative ——————————————————————————————————————————————————————————————————————————————————————————————————————————— 6 / 34
- 7. Chroot • In Linux, everything is a file • Change the root of filesystem to new / for the process • Isolates on filesystem level but not otherwise • Usually used for testing and compatibility reasons 7 / 34
- 8. Namespaces "A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource." - Linux Manual As since the Kernel version of 5.6, 8 kinds of namespaces: Namespace │ Flag │ Isolates ──────────┼─────────────────┼───────────────────────────────────── Cgroup │ CLONE_NEWCGROUP │ Cgroup root directory IPC │ CLONE_NEWIPC │ System V IPC, POSIX message queues Network │ CLONE_NEWNET │ Network devices, stacks, ports, etc. Mount │ CLONE_NEWNS │ Mount points PID │ CLONE_NEWPID │ Process IDs Time │ CLONE_NEWTIME │ Boot and monotonic clocks User │ CLONE_NEWUSER │ User and group IDs UTS │ CLONE_NEWUTS │ Hostname and NIS domain name 8 / 34
- 9. Control Groups cgroups • Limits, accounts for, and isolates resource usage (CPU, memory, disk I/O, etc. ) ◦ Set process-specific limits for resource usage ◦ Prioritization ◦ Reporting and monitoring of the resources ◦ Control • Hierarchial management 9 / 34
- 10. LinuX Containers (LXC) • Adapts namespaces, control groups and chroot to provide isolation and resource limits between containers • Adapts overlay filesystems to provide a union of multiple filesystems • AppArmor and SELinux profiles ——————————————————————————————————————————————————————————————————————————————————————————————————————————— 10 / 34
- 11. Docker • Became the de facto container solution to pack your application as single unit and operate it • Until version 0.9, used LXC, systemd-nspawn and libvirt as execution driver • After that, used their own libcontainer to operate on Linux kernel, later used just abstraction for different implementations • Good user experience (UX) as the major reason for success 11 / 34
- 12. Basic architecture +------------------------+ | Client | | (docker CLI, K8s, etc.)| +-----------+------------+ | Docker Engine (HTTP API) | +-----------+------------+ | containerd | +-----------+------------+ | runC (contains libcontainer) | +-----------+------------+ | Linux Kernel | | (namespaces, cgroups, | | UnionFS, etc.) | +------------------------+ 12 / 34
- 13. Open Container Initiative (OCI) "The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes." • https://opencontainers.org/ • Founded in 2015 by Docker, CoreOS and other leaders in the container industry. • Standardisation of the following container specifications: ◦ The OCI Distribution Specification ◦ The OCI Image Format Specification ◦ The OCI Runtime Specification ◦ https://github.com/opencontainers/distribution-spec 13 / 34
- 14. The OCI Distribution Specification • To distribute, push, pull, update and manage all the container images ◦ Multi-arch support ◦ Metadata ◦ Digests and their meaning • Often known as Docker Hub, GitHub Container Registry, Quay or self-hosted registries and so on.. Each providers follows the same API and image storage format to describe images, but they usually have some own additional endpoints for managing them on scale. 14 / 34
- 15. Understanding the image format ██ Demo • The digest you see in the container registry, is the digest of the manifest • Manifest describes the chain of cryptographic hashes of the image layers and the configuration file ◦ As JSON file ◦ Each layer is TAR file, described as SHA256 digest • Layers re-usable and cacheable. E.g. base image for some other image. 15 / 34
- 16. Demo commands to get single layer from container image with cURL ██ Authentication token curl -sSL "https://ghcr.io/token?service=registry.ghcr.io&scope=repository:nicceboy/testimage:pull" > auth.json ██ Manifest curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept: application/vnd.oci.image.manifest.v1+json" "https://ghcr.io/v2/nicceboy/testimage/manifests/latest" | jq ██ Configuration curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept: application/vnd.oci.image.layer.v1.tar+gzip" "https://ghcr.io/v2/nicceboy/testimage/blobs/sha256:6dee449fabe300f361391e87da6c6204caa7c5cb9569f2c8759679e d96d08e0f" | jq ██ Binary blob with .txt file curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept: application/vnd.oci.image.layer.v1.tar+gzip" "https://ghcr.io/v2/nicceboy/testimage/blobs/sha256:5897666521003f1bd481840975b2f7368359a8ad382124b4ff0c583 cd3b59f7a" > out.tar ██ Unpack the layer content tar xvf out.tar 16 / 34
- 17. Inspecting the image Using curl, jq, tar and less will get you started. Or just podman inspect. However, there are better tools... ◦ https://github.com/wagoodman/dive ◦ View all changes of the container image from all layers ◦ Can help you to optimise the image -> We can conclude that there are a lot dependencies and additional data which might be possible to be reduced. -> Also we were leaking some data on purpose, which might happen by accident for someone. 17 / 34
- 18. Software Bill of Materials (SBOM) • A detailed catalog how the software is built • You must known your software dependencies - they can be vulnerable too • Dependencies must be controllable, listable and trackable • Software supply chain provides additional attack vector • 2023 was record breaking again with software supply chain attacks: increase by 742% between 2019 and 2022, according to Sonatype DALL-E view of SBOM 18 / 34
- 19. Dependencies and why they do matter ██ The OCI image itself ◦ Do you trust the base image? ◦ If compromised, your build process and new deployment is compromised ██ Application dependencies ◦ How many are not actually needed? ▪ Can increase the attack surface ▪ Compromised package by malicious actor or developer ▪ Vulnerability ▪ Additional noise when identifying outdated or vulnerable software ██ Make things more secure and efficient -> Build the images with as little dependencies as possible, and by using trusted sources -> Also reduced disk usage and bandwidth cost 19 / 34
- 20. Basic optimisation: multi-stage builds with traditional base images ██ Demo • Build your application in separate stage • Then copy into the runtime image with runtime dependencies only ◦ Select runtime for you needs (Alpine Linux, Debian slim-based, something else?) • Avoids leaking build-time secrets to runtime image • Reduces the image size and dependencies are closer to what are actually required 20 / 34
- 21. Hardcore: from scratch ██ Demo • Sometimes your application can be statically build and has only a little of dependencies • Container is just a process, image can contain only one executable! • However, not always practical and there are some issues 21 / 34
- 22. What you usually need in the container • User management (/etc/passwd and /etc/group files) for namespace isolation • Some important folders that applications' expect to exist (/var, /tmp, /home) • libc or musl as C Standard Library • Timezone database • CA certificates for TLS to work 22 / 34
- 23. Entering "distroless" • Google provides so-called "distroless" base images which attempts solve the previous • Development in https://github.com/GoogleContainerTools/distroless • gcr.io/distroless/* • Different dependency options, depending on the needs • All images are cryptographically signed 23 / 34
- 24. Basic relationships Image credits: Ivan Velichko - https://iximiuz.com/ 24 / 34 ▓▓▓ Demo
- 25. Going further Distroless images are often enough for your needs. What if your application is very complex? • You will need an advanced image builder ◦ Bazel: https://bazel.build • Declarative package managers are especially useful ◦ Nix package manager: https://nixos.org ◦ Guix: https://guix.gnu.org ◦ Fully reproducible images, with explicit dependencies and nothing else 25 / 34
- 26. Tracking your dependencies Once you have the optimised image - you need to track the lifecycle of your dependencies; are there known vulnerabilities? You can integrate different static analysis tools for your container registry or CI build pipelines. Depending on your application, different ways. • RedHat's clair: https://github.com/quay/clair • Trivy: https://github.com/aquasecurity/trivy • Multi-layered - e.g. use GitHub's dependency bot to check package.json for Node-based applications • There are many, depending on the context 26 / 34
- 27. How about using containers more securely? Know defaults when running containers. What do you need? Applications can be very different with different requirements. Find out what you need. • Kernel capabilities • Mount points and ports • User privileges • Container resources (CPU && RAM?) • Principle of least privilege (or anything) 27 / 34
- 28. Kernel capabilities • Never use --privileged mode • By default, Docker runs with many capabilities. • UX over security? 🤨 docker run --cap-drop=all my_container Select explicitly what you need! 28 / 34
- 29. Docker default kernel capabilities • CAP_CHOWN: Allows changing file ownership. • CAP_DAC_OVERRIDE: Bypasses file read, write, and execute permission checks. • CAP_FSETID: Don't clear set-user-ID and set-group-ID permission bits when a file is modified. • CAP_FOWNER: Bypasses permission checks on operations that require the filesystem UID of the process to match the UID of the file. • CAP_MKNOD: Allows creation of special files using mknod(). • CAP_NET_RAW: Allows use of RAW and PACKET sockets. • CAP_SETGID: Allows setting the GID and group membership. • CAP_SETUID: Allows setting the UID. • CAP_SETFCAP: Allows setting file capabilities. • CAP_SETPCAP: Allows transferring or dropping any capability in the calling thread's permitted capability set. • CAP_NET_BIND_SERVICE: Allows binding to Internet domain sockets. Ports less than 1024 • CAP_SYS_CHROOT: Allows use of chroot(). • CAP_KILL: Allows sending signals to processes other than those belonging to the current user. • CAP_AUDIT_WRITE: Allows writing to the Linux kernel's audit log. 29 / 34
- 30. Mount points Mount only when there is no other option. • Only directories which are specifically for that purpose • Prefer copying instead of mounting if possible • Usually the only way to break the isolation, excluding runner or kernel vulnerabilities • Mount explicitly as read-only, if no write access needed 30 / 34
- 31. Principle of least privilege • Make an user and make sure that UID/GIDs are unique across all containers • Run with --security-opt=no-new-privileges to prevent privilege escalation by using setuid or setgid binaries. • Disable inter-container communication --icc=false • Use AppArmor, SELinux or seccomp when possible • Excellent summary with more details by OWASP: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html#rule-9-use-stat ic-analysis-tools Give only the resources required for containers in order to prevent exhausting whole system by single container. 31 / 34
- 32. Container runners Architecture can have significant impact for security. Run on rootless mode when possible. • Vulnerability on the runner does not affect every container or whole system • Container cannot be run as "real" root by accident ██ Docker vs. Podman Feature │ Docker │ Podman ────────────────────┼───────────────────────────────┼────────────────────────────────────────────────────── Daemon │ Runs as a central daemon │ Daemonless, each container runs in a separate process Root Privileges │ Root by default │ Rootless by default User Namespaces │ Supported, but not by default │ Supports rootless mode by default SELinux Integration │ Supported │ Better default integration with SELinux Seccomp Profiles │ Enabled by default │ Enabled by default, with more restrictive profiles API Compatibility │ Docker API │ Compatible with Docker CLI, can run docker commands Default Network │ Bridge │ Host network, with bridge available 32 / 34
- 33. There are many more • colima, Rancher, OrbStack (commercial) • Variations of kubernetes (usually containerd under the hood) 33 / 34
- 34. Conclusion • Manage your SBOM from the beginning when creating and operating with containers ◦ Minimal dependencies ◦ Trusted image sources • Follow your dependencies lifecycle • Principle of least anything when running and using containers • Know what happens on default and what your container images contain 34 / 34