Short-lived secrets in Kubernetes: practical Vault patterns for rotation, auth, and delivery

Secrets that never change are the easiest attack surface to exploit. HashiCorp Vault gives engineering teams a way to move away from static credentials and toward short-lived, auditable secrets that rotate automatically — but adopting Vault safely inside Kubernetes requires deliberate choices: how secrets are issued, how workloads authenticate, and how secrets are delivered and renewed at runtime. This article lays out a practical pattern that balances security, availability, and operational simplicity, and calls out the key tradeoffs you’ll face.

Why short-lived (dynamic) secrets matter

• Static credentials stored in code or cluster etcd are high-value and long-lived targets. Dynamic secrets remove that single point of failure by issuing credentials on demand with expiry and optional automatic revocation.
• Vault’s database secrets engine can generate credentials dynamically, bound to a role and a lease, allowing automatic rotation and revocation when a lease expires or is revoked. This reduces blast radius if a pod is compromised and lets teams automate credential lifecycle instead of relying on human rotation. (hashicorp.com)

Three integration patterns for Kubernetes workloads Vault supports multiple ways to deliver secrets into Kubernetes workloads. Choosing the right integration depends on whether you need live renewal, whether you want secrets persisted in Kubernetes objects, and how much application change you’ll accept.

• Vault Agent Injector (sidecar): injects secrets into pod volumes or environment variables and runs a local agent to renew leases and refresh secrets while the pod is alive. This is the right pattern when you need automatic renewal without modifying application code. (developer.hashicorp.com)

• Secrets Store CSI provider (CSI): mounts secrets as files into a pod via the Kubernetes CSI Secrets Store driver (optionally syncing to Kubernetes Secrets). The CSI provider is excellent when you want no long-lived Kubernetes Secret objects in etcd and prefer file mounts; however, it does not provide the same auto-renewal behavior as a sidecar by default — choose it when secret contents are fine to be re-read on pod restart or when your workflow can handle periodic refreshes. (developer.hashicorp.com)

• Vault Secrets Operator / sync-to-secret CRD: synchronizes Vault entries into Kubernetes Secret objects under operator control. This works well for GitOps pipelines and tooling that expect Kubernetes Secrets, but be aware that storing secrets in etcd requires careful RBAC and etcd encryption planning. (developer.hashicorp.com)

Key guidance when selecting:

• Need continuous renewal (e.g., DB connection pooled across pod life): use Agent sidecar.
• Want minimal Kubernetes Secret exposure and file-based access: favor CSI provider (with attention to refresh semantics).
• Need GitOps-friendly persisted secrets: use the Secrets Operator but protect etcd and access tightly.

How workloads should authenticate Avoid baking Vault tokens into images or config. Kubernetes-native authentication using service account tokens validated by Vault’s JWT/OIDC authentication method gives a low-friction, tokenless bootstrap: Vault validates the pod’s service account token and maps it to a Vault role that has tightly-scoped policies. That makes pod identity manageable through Kubernetes RBAC and service account lifecycle instead of static Vault credentials. (docs.hashicorp.com)

Design principles for auth and policies:

• Map Kubernetes service accounts (or namespaces) to narrow Vault roles instead of broad, cluster-wide roles.
• Limit each role’s allowed paths and functions (read vs create) with least-privilege policies.
• Use short TTLs for tokens issued to agents so stolen tokens quickly expire.

Production hardening essentials Vault’s security and availability configuration matter as much as how you consume secrets. Production guidance includes strong audit logging, secure auto-unseal, HA storage, and operational controls.

• Unseal and key management: don’t leave unseal keys or root tokens lying around. Use cloud KMS providers or an HSM-backed auto-unseal mechanism so Vault nodes can start without manual key sharing. This reduces human error and speeds recovery. (developer.hashicorp.com)

• Storage and HA: Integrated storage (Raft) is a supported HA backend that provides intra-cluster replication; design for enough nodes and proper snapshot/backup procedures so data-plane operations remain available during maintenance or failures. (developer.hashicorp.com)

• Audit and telemetry: enable Vault audit devices and integrate with centralized logging so secret access is traceable. Treat audit outputs as part of incident response and compliance reporting.

• Least privilege and programmatic hygiene: follow programmatic best practices — limit who can read or write terraform state, apply least privilege policies for automation tools, and avoid baking sensitive information into upstream artifacts. Use short leases and avoid issuing long-lived tokens unless absolutely necessary. (developer.hashicorp.com)

A practical architecture pattern (example) Here’s a compact architecture that combines good practices for the common case: applications in Kubernetes needing short-lived DB credentials and runtime renewal.

• Vault cluster: deployed in HA mode with integrated storage (Raft) and auto-unseal via cloud KMS.
• Authn: Vault’s Kubernetes auth (OIDC/JWT) validates pod service account tokens. Create Vault roles per application namespace or service account.
• Secret engine: Database secrets engine configured for your DB (Postgres/MySQL). Roles define the SQL template for credential creation and TTL/renewal rules.
• Delivery:
  • If the app can handle file-mounted secrets and does not require live renewal: use the CSI provider to mount credentials as files (and avoid syncing to Kubernetes Secrets if you want to keep values out of etcd).
  • If the app needs pooled connections or continuous renewals: use the Vault Agent injector sidecar to fetch and renew DB creds and place them on a shared emptyDir mount for the app to consume.
• Observability: Vault audit logs forwarded to central log store; Prometheus metrics from Vault and the Agent; alerts for lease failures or renewal errors.

Minimal Vault policy example A Vault policy should be as focused as possible. The example below grants a workload the ability to request dynamic DB creds from a single role only:

# policy.hcl
path "database/creds/app-role" {
  capabilities = ["read"]
}

When bound to a Vault role that is in turn mapped to a specific Kubernetes service account, this policy enforces that the workload only gets the credentials it needs and nothing more.

Operational tradeoffs and traps to avoid

• Don’t mix long-lived human tokens with workload identities. Human access should be separate, with higher scrutiny and different TTLs.
• If you sync Vault secrets into Kubernetes Secrets for convenience, add encryption at rest for etcd and tighten RBAC so only the smallest set of controllers and namespaces can read those Secrets.
• Choose the delivery mechanism consciously: sidecars add pod resources but give renewals; CSI keeps fewer secrets in etcd but requires refresh handling; operators fit GitOps flows but intentionally persist secrets inside the cluster.
• Keep the Vault admin/root token lifecycle strict — generate it only when needed, rotate it when used, and prefer limited bootstrap tokens for automation.

Summary: make secret rotation operational Short-lived secrets are a powerful control for reducing risk, but they only help if your authentication, delivery, and operational controls align. Use Vault’s dynamic secrets engines for credential rotation, authenticate pods via Kubernetes OIDC/JWT rather than embedding tokens, pick the integration pattern (Agent, CSI, or Operator) that matches your renewal and persistence needs, and harden the Vault cluster with auto-unseal, HA storage, and audit logging. These choices turn Vault from a static secret store into an operational credential system that enforces rotation and least privilege at scale. (hashicorp.com)