HashiCorp Vault on Kubernetes: Production Secret Management Done Right

The Problem with Kubernetes Secrets
What Vault Actually Does
Architecture: How Injection Works
Setup: Dedicated Cluster for Vault
Configure Vault for Kubernetes Auth
Inject Secrets into a Pod
Proving the Security Model
The Init Container Pattern
Vault vs Kubernetes Secrets — The Real Comparison
Key Takeaways

The Problem with Kubernetes Secrets

Before building anything, understand what you're replacing.

# Create a "secret"
kubectl create secret generic my-app-secrets \
--from-literal=DB_PASSWORD=supersecret
# Anyone with kubectl access can read it instantly
kubectl get secret my-app-secrets \
-o jsonpath='{.data.DB_PASSWORD}' | base64 -d
# prints: supersecret

Kubernetes Secrets are base64 encoded, not encrypted. Base64 is an encoding format — reversible in one command, with no key required. Any developer with kubectl get secret permissions in a namespace can read every secret in that namespace.

The problems compound:

No audit trail — no record of who read which secret when
No rotation — changing a secret means updating YAML and redeploying
No access control granularity — you can read the secret or you can't
etcd storage — secrets stored in plaintext in etcd unless you configure encryption at rest separately

For a side project this is acceptable. For a company handling user data, payment information, or regulated workloads — it's not.

HashiCorp Vault solves all of this.

What Vault Actually Does

Vault is a secrets management system with four core capabilities:

Storage — secrets are encrypted at rest using AES-256-GCM. The encryption key itself is protected by a master key split across multiple unseal keys. Even with direct database access, secrets are unreadable without the keys.

Access control — fine-grained policies define exactly which paths each identity can access and what operations they can perform. A backend service can read secret/data/my-api but not secret/data/payment-processor.

Authentication — Vault supports multiple auth methods. For Kubernetes, pods authenticate using their service account token. Vault verifies the token with the Kubernetes API and returns a scoped Vault token. No passwords, no API keys, no secrets needed to get secrets.

Audit logging — every operation is logged: who authenticated, which secret was accessed, at what time, from which IP. Required for SOC2, PCI-DSS, and HIPAA compliance. Kubernetes Secrets have no equivalent.

Architecture: How Injection Works

The pattern used in production is Vault Agent sidecar injection. The flow:

Pod is created
↓
Mutating webhook intercepts the request
↓
Vault Agent injector adds init container + sidecar to the pod spec
↓
Init container authenticates to Vault using the pod's K8s service account
↓
Vault verifies the service account token with the K8s API
↓
Vault checks the role — is this service account allowed?
↓
Vault returns a scoped token valid for 1 hour
↓
Init container fetches secrets, renders them to /vault/secrets/
↓
App container starts — secrets already available on disk
↓
Sidecar runs continuously, renewing the token and refreshing secrets

Your application never talks to Vault. It just reads files or environment variables. The authentication complexity is handled entirely by the agent.

Setup: Dedicated Cluster for Vault

One practical lesson from this build: run Vault on a dedicated cluster or node group with guaranteed resources. Vault runs internal health checks on tight intervals. When those checks time out due to resource contention, Vault seals itself as a safety measure — by design, to protect against infrastructure attacks.

# Dedicated minikube profile — isolated from other workloads
minikube start -p vault-lab \
--driver=docker \
--memory=3072 \
--cpus=3
kubectl config use-context vault-lab

Install Vault via the official Helm chart:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
kubectl create namespace vault
helm install vault hashicorp/vault \
--namespace vault \
--set "server.dev.enabled=true" \
--set "server.dev.devRootToken=root" \
--set "injector.enabled=true"

injector.enabled=true deploys the Vault Agent Injector — a mutating webhook that intercepts pod creation and adds the agent sidecar automatically when it sees specific annotations.

Verify Vault is unsealed:

kubectl exec -n vault vault-0 -- vault status
# Sealed: false ← dev mode auto-unseals

Configure Vault for Kubernetes Auth

All configuration happens inside the Vault pod:

kubectl exec -it -n vault vault-0 -- sh
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

Enable Kubernetes auth method:

vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

This tells Vault: "trust Kubernetes as an identity provider. When a pod presents a service account token, verify it against the Kubernetes API."

Store secrets:

# KV v2 is pre-enabled in dev mode
vault kv put secret/my-api \
DB_PASSWORD="supersecret-from-vault" \
API_KEY="vault-managed-api-key-xyz" \
DB_HOST="prod-db.example.com"
vault kv get secret/my-api

Write an access policy:

vault policy write my-api-policy - <<EOF
path "secret/data/my-api" {
capabilities = ["read"]
}
EOF

This policy grants read access to exactly one path. The my-api service can read its own secrets and nothing else. Least privilege enforced at the secret engine level.

Create a Kubernetes role:

vault write auth/kubernetes/role/my-api-role \
bound_service_account_names=default \
bound_service_account_namespaces=default \
policies=my-api-policy \
ttl=1h

This role says: "when a pod running as the default service account in the default namespace authenticates, give it a token valid for 1 hour with the my-api-policy attached."

The TTL matters. Vault tokens expire. The agent handles renewal automatically, but if the agent can't reach Vault for more than the TTL, the token expires and secrets become unreadable. This is a feature — a leaked token has a bounded lifespan.

Inject Secrets into a Pod

The injection is entirely annotation-driven. Add four annotations to your pod template and the injector handles everything:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-test
  template:
    metadata:
      labels:
        app: vault-test
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-api-role"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/my-api"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/my-api" -}}
          export DB_PASSWORD="{{ .Data.data.DB_PASSWORD }}"
          export API_KEY="{{ .Data.data.API_KEY }}"
          export DB_HOST="{{ .Data.data.DB_HOST }}"
          {{- end }}
    spec:
      serviceAccountName: default
      containers:
        - name: app
          image: alpine:3.18
          command: ["sh", "-c", "while true; do sleep 30; done"]

Apply it and watch the pod start:

kubectl apply -f vault-test.yaml
kubectl get pods --watch

0/2 Init:0/1 ← Vault Agent init container fetching secrets
0/2 PodInitializing ← secrets written to /vault/secrets/
2/2 Running ← app starts with secrets already available

That 2/2 is the key difference from a standard pod. Two containers: your app and the Vault Agent sidecar.

Verify the secrets landed:

POD=$(kubectl get pod -l app=vault-test \
--field-selector=status.phase=Running \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec $POD -c app -- cat /vault/secrets/config

export DB_PASSWORD="supersecret-from-vault"
export API_KEY="vault-managed-api-key-xyz"
export DB_HOST="prod-db.example.com"

These values came from Vault. They are not stored in Kubernetes anywhere.

Proving the Security Model

No Kubernetes Secret exists:

kubectl get secrets
# default-token only — no my-api-secrets, nothing readable

Compare this to the old world where kubectl get secret my-api-secrets -o yaml would hand anyone with cluster access your production database password.

Rotate without redeploying:

# Update in Vault
kubectl exec -it -n vault vault-0 -- sh -c "
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
vault kv put secret/my-api \
DB_PASSWORD='rotated-password-v2' \
API_KEY='rotated-api-key-v2' \
DB_HOST='prod-db.example.com'
"
# Restart pods to pick up new version
kubectl rollout restart deployment/vault-test

No image rebuild. No YAML change. No Git commit. The new secret is live in under a minute. With automatic rotation enabled, you can do this on a schedule without any human involvement.

The Init Container Pattern

One architectural detail worth understanding: Vault Agent runs as an init container first, then continues as a sidecar.

The init container runs to completion before any app container starts. This guarantees your application has secrets available from its very first line of code — no startup race condition, no "secret not yet available" errors on cold start.

The sidecar continues running alongside your app, renewing the Vault token before it expires and updating the secrets file if values change. Your app can optionally watch the file for changes to pick up rotated secrets without restarting.

Vault vs Kubernetes Secrets — The Real Comparison

Concern	Kubernetes Secrets	HashiCorp Vault
Encryption at rest	Only with extra etcd config	Always, AES-256-GCM
Access control	Namespace RBAC only	Fine-grained path policies
Audit logging	None	Every operation logged
Secret rotation	Manual redeploy	Automated, no redeploy
Dynamic secrets	Not possible	DB creds generated per-pod
Multi-cluster	Duplicate secrets everywhere	Single source of truth
Compliance	Difficult	SOC2, PCI-DSS, HIPAA ready

Dynamic secrets deserve a callout — it's Vault's killer feature for databases. Instead of storing a static DB_PASSWORD, Vault generates a unique username and password for each pod at startup, with a TTL matching the pod's lifetime. When the pod dies, the credentials expire automatically. A compromised pod's database credentials are useless the moment the pod is gone.

Key Takeaways

Kubernetes Secrets are base64 encoded, not encrypted — treat them as readable by anyone with cluster access
Vault authenticates pods via Kubernetes service account tokens — no credentials needed to get credentials
The injector webhook pattern means zero application code changes — just annotations
Init container guarantee means secrets are always available before your app starts
Secret rotation without redeployment is the operational win that justifies the setup overhead
Audit logging is not optional for compliance workloads — Vault provides it, Kubernetes Secrets don't

Part of a hands-on DevOps learning series. Code at github.com/kaungmyathan22/golang-k8s-portfolio.

Quick navigation