K8s Auth in OpenBao: What I Got Wrong, How I Fixed It.

Kubernetes auth in OpenBao (Vault) isn’t hard—but it is precise. A few missing fields, a bad path, or a default token can quietly block the whole workflow.

I recently ran through the full debug cycle: a pod trying to authenticate to OpenBao via the Kubernetes auth method. Here's what went wrong, how I debugged it, and the final working configuration.

✅ What I Set Out to Do

I needed a pod (openbao-dev-full) in the openbao namespace to authenticate to OpenBao using the Kubernetes auth method. The pod uses the openbao-openbao service account. Once wired correctly, it should send a token to /v1/auth/kubernetes/login and receive a valid Vault token tied to specific policies.

❌ What Went Wrong

1. Wrong Service Account

The pod was using the default service account, not openbao-openbao. That mismatch led to a silent login failure.

Fix: Updated the pod spec:

serviceAccountName: openbao-openbao        

2. JWT Not Found or Not Valid

Testing outside the cluster (or in misconfigured pods) failed because /var/run/secrets/kubernetes.io/serviceaccount/token was missing or invalid.

Fix: Fetched the JWT directly from the Kubernetes secret:

export SA_SECRET_NAME=$(kubectl get serviceaccount openbao-openbao -n openbao -o jsonpath="{.secrets[0].name}") export SA_TOKEN=$(kubectl get secret $SA_SECRET_NAME -n openbao -o jsonpath="{.data.token}" | base64 -d)        

3. Audience Validation Failure

The JWT’s aud field was https://kubernetes.default.svc.cluster.local, but the Vault role had no bound_audiences set.

Fix: Explicitly defined the correct audience:

vault write auth/kubernetes/role/aj-test-role \ bound_service_account_names=openbao-openbao \ bound_service_account_namespaces=openbao \ audience="https://kubernetes.default.svc.cluster.local" \ 
policies=default \ 
ttl=24h        

4. Incorrect Role Path

I tried to create the role at kubernetes/role/aj-test-role, which failed.

Fix: Used the correct mount path:

vault write auth/kubernetes/role/aj-test-role ...        

5. Vault CLI Confusion

Inside the pod, vault auth enable kubernetes had been done—but the CLI still threw Unknown auth method: kubernetes.

Fix: Verified with:

vault auth list        

When the CLI was unhelpful, I used curl directly.

📡 Request / Reply: What Success Looks Like

Here’s the raw request I sent from inside the pod:

curl --request POST \ --header "Content-Type: application/json" \ 
--data "{\"role\": \"aj-test-role\", \"jwt\": \
"$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\"}" \ http://openbao-openbao:8200/v1/auth/kubernetes/login        

And the successful response:

{ "auth": { "client_token": "s.DQRKY0uru69rEvb1wovyiyuL", "policies": ["default"], "metadata": { "role": "aj-test-role", "service_account_name": "openbao-openbao", "service_account_namespace": "openbao" }, "lease_duration": 86400, "renewable": true, "token_type": "service" } }        

This confirmed everything was wired up: the token was valid, the role matched, the policies applied, and the lease was renewable.

✅ What I Got Right

• Reset from scratch I disabled and re-enabled the Kubernetes auth backend to avoid legacy config drift.
• Configured everything explicitly Including:
• Verified JWT contents Decoded the JWT to confirm aud, sub, and iss aligned with my Vault role.
• Used curl to test the flow end-to-end The CLI can mislead. The raw API is always honest.

🔍 Lessons for Platform Teams

• Exactness matters Vault won’t guess or fill in missing config. You have to be precise—especially with JWT fields.
• Self-service requires stable wiring If auth fails silently, developers will never adopt the platform. This config now lives in code, not one-off YAML.
• Always test the edge If I’d started with a raw curl and JWT decode, I could’ve avoided a lot of back-and-forth.

📌 Next Steps

• Replace the default policy with a purpose-built k8s-login policy.
• Use projected service account tokens for better TTL and rotation.
• Add a GitHub Action that validates the full auth flow as part of platform CI.
• Build a diagram that shows the full lifecycle: Pod → JWT → Vault → Policy → Token.

If you're integrating Vault or OpenBao into Kubernetes, validate each step independently—and never trust defaults to carry you over the finish line.

#Kubernetes #OpenBao #HashiCorpVault #Vault #DevOps #PlatformEngineering #Security #Authentication #CloudNative #Debugging #GitOps #SecretsManagement