Written by Nelson Koning
on 

Short-lived Secrets, Long-term Confidence: Best Practices for Dynamic Secrets with HashiCorp Vault

Secrets management used to feel a lot like maintaining a vinyl collection: carefully labeled, cherished, and impossible to rotate without breaking the groove. In modern cloud-native environments we need the flexibility of streaming playlists — ephemeral, replaceable, and orchestrated so the music never stops. HashiCorp Vault’s dynamic secrets model is exactly that: short-lived credentials, automatic rotation, and programmatic revocation that reduce blast radius and operational toil.

This article walks through practical design patterns and operational best practices for using dynamic secrets in Vault — focusing on databases, cloud providers, and Kubernetes workloads — with concrete examples and references to official guidance.

Why dynamic secrets matter

Dynamic secrets are credentials Vault creates on-demand (for example, a database user or a cloud IAM keypair) and hands to a client with a limited lease. When the lease expires Vault can revoke the credential, removing access without any manual procedure. That model turns long-lived static secrets — the ones that linger in config files and container images — into transient capabilities that are far less useful to an attacker. Vault supports dynamic secrets for databases, AWS/GCP/Azure, Kubernetes tokens, and more. (developer.hashicorp.com)

Think of it as issuing concert tickets that expire at the end of the show instead of handing out permanent passes.

Core Vault features to leverage

Practical patterns and tradeoffs for Kubernetes workloads

Kubernetes is where Vault and dynamic secrets meet real-world demands. Vault offers three mainstream integrations: the Agent sidecar injector, the Secrets Store CSI provider, and the newer Vault Secrets Operator (VSO). Each has different tradeoffs:

Choose the integration that matches your constraints (performance, platform compatibility, secret exposure rules). The HashiCorp comparison docs offer a helpful matrix for these tradeoffs. (developer.hashicorp.com)

Example: a minimal SecretProviderClass snippet (Secrets Store CSI) mounts Vault secrets into /mnt/secrets-store as ephemeral files:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.mycompany.internal"
    roleName: "k8s-role"
    objects: |
      - objectName: "db-creds"
        secretPath: "database/creds/app-role"
        fileName: "creds.json"

(Refer to the Secrets Store CSI driver and HashiCorp’s Vault CSI provider docs for exact fields and auth details.) (secrets-store-csi-driver.sigs.k8s.io)

Best practice checklist

Below are recommendations based on Vault documentation and operational guidance. Each item is framed as a practice your organization can adopt in design and operations.

Short examples — policies and database role

A compact read-only policy for a service that can fetch DB creds might look like:

# policies/db-read.hcl
path "database/creds/app-readonly" {
  capabilities = ["read"]
}

path "sys/leases/lookup" {
  capabilities = ["read"]
}

Create a Vault database role that issues ephemeral PostgreSQL users (illustrative CLI):

vault write database/config/my-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-readonly" \
  connection_url="postgresql://:@db.example.internal:5432/mydb?sslmode=disable"

vault write database/roles/app-readonly \
  db_name=my-postgres \
  creation_statements="CREATE ROLE \"\" WITH LOGIN PASSWORD '' VALID UNTIL ''; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"\";" \
  default_ttl="1h" \
  max_ttl="24h"

Refer to Vault’s database secrets engine documentation for exact plugin fields and supported databases. (developer.hashicorp.com)

Monitoring, ops, and human factors

Operational discipline is the difference between a secure proof-of-concept and a reliable platform:

Final note — balancing convenience and security

Dynamic secrets reduce the time a leaked credential is valuable, but they don’t eliminate human error or misconfiguration. Treat Vault as part of a layered approach: identity-first auth, least privilege policies, automated rotation, and robust audit and monitoring together make secrets ephemeral and manageable.

If static secrets are unavoidable for particular use cases, minimize their footprint, rotate them frequently, and ensure Vault mediates access where possible so the long-lived values are the exception, not the rule. The goal is to move from a dusty vinyl shelf to a streaming service: the music keeps playing, but no single record can spoil the whole show. (hashicorp.com)

References (selected):