💥 Title: “Rebuilt Vault in One Night: A Real-World Journey with OpenBao, Raft, and Secrets That Work”

THE TRIGGER

HashiCorp Vault works, but the licensing doesn’t scale. The cost was outpacing its actual use. The goal was to build a replacement that kept the features that mattered — HA storage, transit encryption, secrets engines — without carrying the enterprise price tag.

Started work at 4:45 PM. Final test script passed at 2:49 AM.

STEP 1: BUILDING A PRODUCTION-READY CHART

The OpenBao Helm chart worked for local demos, but not much more. Rebuilt the chart with these priorities:

• Enabled the UI by exposing it directly from the container
• Added Kustomize support for managing multi-env overlays (dev, test, prod)
• Set up persistent volumes with one PVC per pod for Raft storage
• Fixed readiness probes — the default checks restarted pods before unseal was complete

Skipped injector and webhook for this iteration to stay focused on the control plane.

STEP 2: STABILIZING RAFT BACKEND FOR HA

Used Raft (Integrated Storage) for stateful HA.

Config included:

• raft storage backend with 3 replicas
• Ingress TLS via cert-manager
• Bootstrapped manually with unseal keys saved to a PVC for development use

Problems hit:

• Pods restarted on loop without timely unseal
• Split-brain state until Raft join peer entries were manually defined
• Readiness had to hook into actual vault status output, not just port checks

Wrote a short init script to:

• Generate root and recovery keys
• Store them on the PVC
• Print them once and exit

STEP 3: CONFIGURED AND TESTED ENGINES

Used five engines, each tested with real integration.

kv-v2

• Configured with versioning and TTL
• Policies scoped to environment and role
• Confirmed data versioning and TTL expiration logic

transit

• Created encryption-only keys
• Used transit-check.py to validate encryption, decryption, and signature behavior
• Measured round-trip latency inside the pod (~40ms)

pki

• Created a root CA and role to issue wildcard certs
• Integrated with cert-manager using the Vault Issuer type
• Rotated test certs every 90 days

database

• Spun up a Postgres container
• Registered it with Vault using the database engine
• Created dynamic roles and used db-dynamic-check.py to test logins and query runtime

ssh

• Set up a CA to sign short-lived SSH certs
• Wrote and tested ssh-gen-sign.py and ssh-sign-check.py
• Validated login flow against a test Ubuntu box using signed certs

WHAT NEEDED FIXING

• The Helm chart doesn’t cover unseal/init flows. Handled it with scripting and PVCs.
• Raft join needed explicit peer list or the nodes stayed isolated
• The UI isn’t productionized in the chart. Exposed it manually and locked it behind basic auth.
• Every engine required its own policies, roles, and mount paths. No shortcuts.

COST ANALYSIS

HashiCorp Vault with Raft, DR, UI, and enterprise auth integration is expensive — and billing grows with usage and teams. This cost goes away with OpenBao. Invested time instead of money. Got:

• HA secrets store
• Fully functioning encryption-as-a-service
• Short-lived credential issuance for DBs and SSH
• A working PKI pipeline for app certs

No license. No audit tier upsells. No enterprise sales cycle.

WHAT COMES NEXT

• Move init/unseal and bootstrap into GitHub Actions
• Add secrets injector back in for runtime mounts
• Extend policy automation via scriptable roles and bindings
• Integrate OIDC for GitHub runners
• Test replication and DR on Raft clusters

No team. No vendor calls. No purchase orders. Just time and a sharp goal.

🛠️ Every part is real, tested, and running.

Want to see the scripts or overlays? Happy to share.