Let's Do Bad Things to Unsecured Containers
0 likes36 views
There is plenty of advice about what to do when building and deploying containers to make sure we are secure. But why do we need to do them? How important are some of these “best” practices? Can someone take over my entire system because I missed one step? What is the worst that could happen, really? Join Gene as he guides you through exploiting unsecured containers. We’ll abuse some commonly missed security recommendations to demonstrate the impact of not properly securing containers. We’ll exploit these lapses and discover how to detect them. Nothing reinforces good practices more than seeing what not to do and why. If you’ve ever wondered why those container recommendations are essential, this is where you can find out.
![Layers
Containers are layers of filesystems
Files overlap, but do not replace RUN set -xe && echo '#&/bin/sh' > … 811B
ADD file:435d9776fdd3a183… 72.9MB
RUN [ -z "$(apt-get indextargets)" ]… 0B
RUN mkdir -p /run/systemd && echo ‘d… 7B
CMD ["/bin/bash"] 0B
instance data container layer
(read-write)
immutable
image
layers
(read-only)
ubuntu:20.04 container
@OtherDevOpsGene #KCDTexas 6](https://image.slidesharecdn.com/doingbadthingstounsecuredcontainers-lab-kcdtexas-250515195335-ebe1a598/85/Let-s-Do-Bad-Things-to-Unsecured-Containers-6-320.jpg)
![Better: Run as non-root
• Create a user in the image, if necessary.
• Set the user in the Dockerfile.
• Restricts the blast radius if the container is exploited.
FROM ubuntu:latest
RUN useradd -&create-home -&shell /bin/bash newuser
USER newuser
CMD ["runs", "as", "newuser"]
@OtherDevOpsGene #KCDTexas 26](https://image.slidesharecdn.com/doingbadthingstounsecuredcontainers-lab-kcdtexas-250515195335-ebe1a598/85/Let-s-Do-Bad-Things-to-Unsecured-Containers-25-320.jpg)
Let's Do Bad Things to Unsecured Containers
- 1. 1 Let's Do Bad Things to Unsecured Containers Gene Gotimer DevOps Engineer at Praeses, LLC @OtherDevOpsGene May 15, 2025
- 2. “ ” What’s the worst that could happen? @OtherDevOpsGene #KCDTexas 2
- 3. “ ” Plan for the worst, hope for the best. Otherwise known as threat modeling. @OtherDevOpsGene #KCDTexas 3
- 4. Security • Confidentiality • Integrity • Availability @OtherDevOpsGene #KCDTexas 4 ` Availability
- 6. Layers Containers are layers of filesystems Files overlap, but do not replace RUN set -xe && echo '#&/bin/sh' > … 811B ADD file:435d9776fdd3a183… 72.9MB RUN [ -z "$(apt-get indextargets)" ]… 0B RUN mkdir -p /run/systemd && echo ‘d… 7B CMD ["/bin/bash"] 0B instance data container layer (read-write) immutable image layers (read-only) ubuntu:20.04 container @OtherDevOpsGene #KCDTexas 6
- 7. Containerization Containers are not miniature VMs They are apps that run on the host @OtherDevOpsGene #KCDTexas 7
- 9. Bad: Big base images • We used to run the app on Ubuntu, so just install it on an Ubuntu container image. • We can build and deploy with the same image. • We need these tools for troubleshooting. • I’m not sure what we need, so just use a standard distro. @OtherDevOpsGene #KCDTexas 10
- 10. Problem: Large attack surface Any • security issue, • vulnerable version of software, • misconfiguration, • default open setting on an unused application, • unnecessary open port could be a way to gain elevated or unauthorized access to the system. @OtherDevOpsGene #KCDTexas 11
- 11. Better: Minimal images In order of preference 1. scratch (nothing but the application) 2. Distroless (e.g., Google) 3. Small image (e.g., Alpine or BusyBox) 4. Minimal Linux (e.g., Slim or Red Hat UBI Micro) 5. Anything else (you have made a mistake) and much, much lower @OtherDevOpsGene #KCDTexas 12
- 12. Lab: Size matters @OtherDevOpsGene #KCDTexas 13 # Look at the relative sizes $ docker image ls ubuntu:latest $ docker image ls alpine:latest $ docker image ls gcr.io/distroless/static-debian12 $ docker image ls hello-world:latest # See what packages are installed $ syft ubuntu:latest $ syft alpine:latest $ syft gcr.io/distroless/static-debian12 $ syft hello-world:latest
- 13. Bad: Old base images • We aren’t adding new functions, so we don’t need to rebuild the app. • We scanned it a few weeks ago, so we know it is secure. • We know that version works, so don’t touch it. @OtherDevOpsGene #KCDTexas 14
- 14. Problem: Known vulnerabilities Unpatched vulnerabilities in • base image, • supporting files, • application dependencies could be a way to gain elevated or unauthorized access to the system. And an upgrade could be available that fixes the vuln. @OtherDevOpsGene #KCDTexas 15
- 15. Better: Rebuild regularly • Have an automated pipeline • Repeatable, reliable builds • Continually scan for vulnerabilities (Trivy, Grype, Checkov) • Scanners improve • New vulnerabilities are identified • Updated base image • Updated supporting software • Updated application dependencies @OtherDevOpsGene #KCDTexas 16
- 16. Lab: Latest and greatest @OtherDevOpsGene #KCDTexas 17 # Count the known vulnerabilities $ grype ubuntu:focal-20231003 $ grype ubuntu:focal-20240216 $ grype ubuntu:focal-20240530 $ grype ubuntu:focal-20250404 # You can also use trivy image <image name> # See what images are available for the latest Ubuntu LTS (noble) $ skopeo list-tags docker:&&ubuntu # Compare the known vulnerabilities for an early and recent base image
- 17. Bad: Secrets in “hidden” layers FROM ubuntu:latest # Even with the rm, the files are still in the image COPY keypair.pem password.txt / RUN rm /keypair.pem /password.txt # Hard-coded RUN curl -&user guest:guest https:&&jigsaw.w3.org/HTTP/Basic/ @OtherDevOpsGene #KCDTexas 18
- 18. Problem: They aren’t hidden The manifest contains each command. A tool like dive can show you the contents of each layer. @OtherDevOpsGene #KCDTexas 19
- 19. Lab: Hunting for treasure @OtherDevOpsGene #KCDTexas 20 # Build an image with “hidden” secrets $ cd ~/environment/secrets-abound $ cat Dockerfile $ docker build -t secrets:demo . # do not forget the period, i.e., this dir # See what was left behind $ docker run -it -&rm secrets:demo /bin/bash # ls -l # exit $ docker history secrets:demo $ docker history -&no-trunc secrets:demo $ dive secrets:demo
- 20. Lab (part 2): Dig deeper @OtherDevOpsGene #KCDTexas 21 # Collect then extract the layers $ docker save -o layers.tar secrets:demo # Extract the layer contents $ tar xvf layers.tar $ cd blobs/sha256/ $ mkdir extracted $ tar xvf layerID -&directory extracted/ # layerID from skopeo $ cd extracted/ $ ls $ cat password.txt $ cat keypair.pem
- 21. Better: Keep secrets off image • Use multi-stage builds • Copy resulting files onto fresh, blank image • Scan builds for secrets • Consider GitGuardian or TruffleHog • GitHub and GitLab also detect secrets in source code @OtherDevOpsGene #KCDTexas 22
- 22. Bad: Containers that run as root • What does it matter? It is just a container. • I just run as the default user. • It is easier if it is root, because then everything works. @OtherDevOpsGene #KCDTexas 23
- 23. Problem: Elevated privileges If there are any vulnerabilities in the application or container, the attacker has root on the Docker host. Remember, containers are processes running on the host. @OtherDevOpsGene #KCDTexas 24
- 24. Lab: Superpowers @OtherDevOpsGene #KCDTexas 25 $ cat /sooper-secret.txt # fails $ lsblk -io KNAME,TYPE,SIZE,MODEL # to find device name # Build an image that runs as root $ cd ~/environment/privileged $ docker build -t privileged:root . # do not forget the period # Look at the local disk, not the image $ docker run -&rm -it -&privileged privileged:root /bin/bash # mount /dev/nvme0n1p1 /mnt/gotcha # cd /mnt/gotcha # cat sooper-secret.txt # exit
- 25. Better: Run as non-root • Create a user in the image, if necessary. • Set the user in the Dockerfile. • Restricts the blast radius if the container is exploited. FROM ubuntu:latest RUN useradd -&create-home -&shell /bin/bash newuser USER newuser CMD ["runs", "as", "newuser"] @OtherDevOpsGene #KCDTexas 26
- 26. Lab (part 2): Kryptonite @OtherDevOpsGene #KCDTexas 27 # Build an image that runs as non-root # Uncomment the lines in Dockerfile $ docker build -t privileged:nonroot . # do not forget the period # Test our access, even with privilege $ docker run -&rm -it -&privileged privileged:nonroot /bin/bash # mount /dev/nvme0n1p1 /mnt/gotcha # fails # sudo mount /dev/nvme0n1p1 /mnt/gotcha # also fails # whoami # exit
- 27. Bad: Using latest image • This way, we always have the newest version. • Otherwise, we have to keep changing the Dockerfile. @OtherDevOpsGene #KCDTexas 28
- 28. Problem: Supply chain attacks If someone uploads • a broken build, • an untested version, • a vulnerable version, • a trojan horse now you are using it. Non-repeatable builds- poor configuration management. @OtherDevOpsGene #KCDTexas 29
- 29. Lab: Typical day using Docker @OtherDevOpsGene #KCDTexas 30 # Note the digest $ docker pull otherdevopsgene/helloworld:latest # Completely benign image $ docker run -&rm -it otherdevopsgene/helloworld:latest
- 30. Lab (part 2): Later that day @OtherDevOpsGene #KCDTexas 31 $ docker pull otherdevopsgene/helloworld:latest # Try it again $ docker run -&rm -it otherdevopsgene/helloworld:latest
- 31. Better: Use digests • Completely repeatable builds. • Excellent configuration management. • No chance of an image being replaced with the same tag. FROM ubuntu@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e @OtherDevOpsGene #KCDTexas 32
- 32. Lab (part 3): Be specific @OtherDevOpsGene #KCDTexas 33 # Try it one more time, all on one line $ docker run -&rm -it otherdevopsgene/helloworld@sha256:imageID # Can even keep the label, but it is ignored $ docker run -&rm -it otherdevopsgene/helloworld:latest@sha256:imageID # Bad one is still there $ docker run -&rm -it otherdevopsgene/helloworld:latest
- 34. Key take aways • There are real reasons behind security recommendations. These aren’t just theoretical attacks. • Do some threat modeling, even informal back-of-the-napkin. • Remember YAGNI- you ain’t gonna need it, aka the principle of least privilege. • Restrict the blast radius when practical. @OtherDevOpsGene #KCDTexas 35
- 35. Recommended practices Center for Internet Security (CIS) Benchmarks • Docker Benchmark • https:&&w&&.cisecurity.org/benchmark/docker/ • Hosts, Docker daemon, container images, container runtime • Kubernetes Benchmark • https:&&w&&.cisecurity.org/benchmark/kubernetes/ • Control plane, etcd, worker nodes, policies • Amazon EKS, Azure AKS, Google GKE, Red Hat OpenShift @OtherDevOpsGene #KCDTexas 36
- 36. Suggestions: Early warning SAST @OtherDevOpsGene #KCDTexas 37 # Check an IaC file $ checkov -f Dockerfile # Check all the IaC in a directory $ checkov -d . # Check a Docker manifest $ trivy config Dockerfile $ grype file:Dockerfile
- 37. Tools mentioned • Google Distroless: https:&&github.com/GoogleContainerTools/distroless • Syft: https:&&github.com/anchore/syft • Grype: https:&&github.com/anchore/grype • Trivy: https:&&github.com/aquasecurity/trivy • Skopeo: https:&&github.com/containers/skopeo • Dive: https:&&github.com/wagoodman/dive • GitGuardian: https:&&w&&.gitguardian.com/ • TruffleHog: https:&&trufflesecurity.com/trufflehog • Checkov: https:&&w&&.checkov.io/ @OtherDevOpsGene #KCDTexas 38
- 38. Repositories • https:&&github.com/OtherDevOpsGene/secrets-abound • https:&&github.com/OtherDevOpsGene/privileged • https:&&github.com/OtherDevOpsGene/hello-world-sh @OtherDevOpsGene #KCDTexas 39
- 39. Questions? Gene Gotimer DevOps Engineer at Praeses, LLC @OtherDevOpsGene @OtherDevOpsGene #KCDTexas 40