Debugging Distroless Containers: When Your Container Has No Shell

Introduction

In this article we will explore how to debug distroless containers in Kubernetes when your application container has no shell, no package manager, and basically no debugging tools whatsoever. If you’ve ever tried to kubectl exec into a distroless container only to be greeted with "executable file not found in $PATH: /bin/sh", you know the pain.

Distroless containers are amazing for security and size, they contain only your application and its runtime dependencies, nothing else. No shell, no package managers, no debugging tools. They’re perfect for production… until something goes wrong and you need to poke around inside.

But fear not! Kubernetes has a solution: ephemeral containers via kubectl debug. We'll not only see how to use this feature, but also how to manually set up a user environment and access the main container's filesystem through the /proc/1/root trick.

What are Distroless Containers?

Before we dive into debugging, let’s quickly understand what we’re dealing with. Distroless containers, popularized by Google, contain only:

• Your application binary

• Runtime dependencies (libraries, certificates, timezone data)

• A minimal user setup (usually just root or a dedicated user)

What they DON’T contain:

• Package managers (apt, yum, apk)

• Shells (bash, sh, zsh)

• Debugging tools (ps, netstat, curl, wget)

• Text editors (vi, nano)

• Pretty much anything that makes debugging easy

This is fantastic for security (smaller attack surface) and performance (smaller images), but terrible when you need to debug a running container.

The Problem

Let’s say you have a Go application running in a distroless container and it’s behaving strangely. Your natural instinct is:

But you’re greeted with:

Even trying different shells won’t help:

Enter kubectl debug

Kubernetes 1.18+ introduced ephemeral containers, and kubectl debug makes them easy to use. Think of it as attaching a debugging sidecar to your running pod temporarily.

Here’s the basic syntax:

This command:

• Creates an ephemeral container using the ubuntu image

• Attaches to it interactively (-it)

• Shares the process namespace with the target container

But there’s a catch, even with this setup, you still can’t directly access your application’s filesystem. That’s where the /proc/1/root magic comes in.

The /proc/1/root Trick

In Linux, /proc/1/root is a symbolic link to the root filesystem of process ID 1. When containers share a process namespace (which kubectl debug does by default), you can access the main container's filesystem through this path.

Here’s the full debugging workflow:

Step 1: Create the Debug Container

You’ll be dropped into a shell in the Ubuntu container. The --share-processes flag ensures you can see all processes from both containers.

Step 2: Verify Process Sharing

You should see both your application process (PID 1) and the shell processes from the debug container. If your app is running as PID 1, you’re good to go.

Step 3: Access the Main Container’s Filesystem

This shows you the filesystem of your distroless container! You can now navigate and inspect files:

Step 4: Creating a Proper User Environment

Sometimes you might want to work more comfortably. Here’s how to set up a proper user environment in your debug container:

Now you have a full debugging environment with all the tools you need, while still being able to access your distroless container’s filesystem.

Advanced Debugging Techniques

Once you have access, here are some powerful debugging techniques:

Network Debugging:

File System Investigation:

Process Analysis:

Real-World Example

Let me show you a complete example. Let’s say you have a Go application in a distroless container that’s failing health checks:

The issue was a misconfigured health check endpoint, the service was configured to check port 8080, but the app was listening on 3000.

When kubectl debug Isn’t Available

If you’re running an older Kubernetes version (< 1.18) or your cluster doesn’t support ephemeral containers, you have a few alternatives:

Option 1: Add a debug sidecar to your pod spec

Option 2: Use kubectl cp to get files out

Troubleshooting Common Issues

Debug container can’t see main container processes: Make sure you’re using--share-processes or that shareProcessNamespace: true is set in your pod spec.

Permission denied accessing /proc/1/root: This can happen if your debug container doesn’t have sufficient privileges. Try:

To determine the UID/GID and create an user with these values to be able to read/write.

Main container isn’t PID 1: If your app isn’t running as PID 1, find the correct process:

Security Considerations

While kubectl debug is incredibly useful, keep these security considerations in mind:

• Debug containers can access sensitive information from the main container

• They run with the same service account permissions

• Logs from debug containers might contain sensitive data

• Always clean up debug containers when done (they’re ephemeral by default)

Best Practices

Here are some best practices I’ve learned over the years:

• Use minimal debug images: Start with alpine or ubuntu, add tools as needed

• Document your debugging process: Save useful commands for your team

• Create debugging runbooks: Common issues and their investigation steps

• Use labels: Tag your debug containers for easy identification

• Set resource limits: Debug containers can consume cluster resources too

Creating a Debug Container Template

You might want to create a pre-configured debug image for your team:

Build and push this to your registry, then use it for debugging:

Conclusion

Debugging distroless containers doesn’t have to be a nightmare. With kubectl debug and the /proc/1/root technique, you can investigate issues in even the most minimal containers. The key is understanding that you're not trying to add tools to the distroless container - you're bringing your own toolbox and accessing the container's filesystem from the outside.

This approach gives you the security benefits of distroless containers in production while maintaining the ability to debug when things go wrong. It’s the best of both worlds , secure, minimal containers with full debugging capabilities when you need them.

Remember, the goal isn’t to avoid issues entirely (though that would be nice), but to be able to quickly identify and resolve them when they inevitably occur. With these techniques in your toolkit, distroless containers become much less scary to debug.

Hope you found this useful and enjoyed reading it, until next time!