HashiCorp Vault Dynamic Secrets for PostgreSQL & Kubernetes

Finally, we'll create a Kubernetes authentication role in Vault that maps a Kubernetes ServiceAccount to this policy. This role specifies which Kubernetes service accounts and namespaces are allowed to authenticate and what Vault policies they receive.

vault write auth/kubernetes/role/my-app-role \
    bound_service_account_names="myapp-sa" \
    bound_service_account_namespaces="default" \
    policies="postgresql-app-policy" \
    ttl="1h"

This configuration means any pod in the `default` namespace using the `myapp-sa` ServiceAccount can authenticate with Vault and obtain tokens with the `postgresql-app-policy` attached.

Configuring the PostgreSQL Secrets Engine

Enable the `database` secrets engine:

vault secrets enable database

Configure the PostgreSQL connection string. Replace `postgres` and `my_pg_password` with your actual administrative credentials and `localhost:5432` with your PostgreSQL server's address.

vault write database/config/postgresql-db \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="my-app-role" \
    connection_url="postgresql://{{username}}:{{password}}@localhost:5432/mydb?sslmode=disable" \
    username="postgres" \
    password="my_pg_password"

Here, `postgresql-db` is the name Vault will use to refer to this specific PostgreSQL connection configuration. `allowed_roles="my-app-role"` specifies which Vault roles are allowed to use this connection to generate credentials. `connection_url` uses `{{username}}` and `{{password}}` as placeholders for the administrative user Vault will use to connect.

Next, define a Vault role that specifies how dynamic users are created and revoked in PostgreSQL. This is where the magic happens. Vault will execute these SQL statements when a secret is requested and revoked.

vault write database/roles/my-app-role \
    db_name="postgresql-db" \
    creation_statements='CREATE ROLE "{{name}}" WITH LOGIN PASSWORD ''{{password}}'' VALID UNTIL ''{{expiration}}''; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";' \
    revocation_statements='DROP ROLE IF EXISTS "{{name}}";' \
    default_ttl="1h" \
    max_ttl="24h"

In this `my-app-role` configuration:

• `db_name="postgresql-db"` links this role to the PostgreSQL connection we configured.
• `creation_statements` are the SQL commands Vault executes to create a new user. `{{name}}` and `{{password}}` are placeholders Vault replaces with a unique username and a strong, randomly generated password. `{{expiration}}` is the timestamp when the user should expire. We're granting `SELECT` privileges on all tables in the `public` schema. You should tailor these grants to the principle of least privilege for your application.
• `revocation_statements` are the SQL commands Vault executes to drop the user when its lease expires or is explicitly revoked.
• `default_ttl` sets the default lease duration for secrets generated by this role (1 hour).
• `max_ttl` sets the maximum possible lease duration (24 hours).

To test if our PostgreSQL secrets engine is working, let's request a dynamic secret directly from Vault CLI:

vault read database/creds/my-app-role

You should see output similar to this:

Key             Value
---             -----
lease_id        database/creds/my-app-role/xxxxxxxxxxxxxxxxxxxxxxxx
lease_duration  1h
lease_renewable true
password        yyyyyyyyyyyyyyyyyyyyyyyy
username        v-token-my-app-role-zzzzzzzzzzzzzzzzzz

This confirms Vault can successfully create dynamic PostgreSQL users. You can verify this by logging into your PostgreSQL database as the `postgres` user and listing roles: `\du`.

Integrating with Kubernetes Workloads using Vault Agent Injector

First, ensure the Vault Agent Injector is installed in your Kubernetes cluster. This is typically done via Helm. We'll assume it's already installed and configured. If not, you can install it using:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault --set "server.enabled=false" --set "injector.enabled=true" --set "injector.externalVaultAddr=http://127.0.0.1:8200" # Replace with your Vault address

Make sure `injector.externalVaultAddr` points to your Vault server. If Vault is running inside the cluster, you'd use its internal service name (e.g., `http://vault.vault.svc.cluster.local:8200`).

Consuming Dynamic Secrets in a Kubernetes Pod

First, create a ServiceAccount that our pod will use. This ServiceAccount will be bound to the Vault Kubernetes authentication role we created earlier (`my-app-role`).

# myapp-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sa-token-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-sa-token-reader
  namespace: default
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: default
roleRef:
  kind: Role
  name: sa-token-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f myapp-sa.yaml

Next, create a Kubernetes Deployment that uses this ServiceAccount and includes annotations for the Vault Agent Injector.

# postgres-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-app
  labels:
    app: postgres-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-app
  template:
    metadata:
      labels:
        app: postgres-app
      annotations:
        # Enable Vault Agent Injector for this pod
        vault.hashicorp.com/agent-inject: "true"
        # Specify the Vault Kubernetes authentication role
        vault.hashicorp.com/role: "my-app-role"
        # Define the secret to inject and its path within Vault
        vault.hashicorp.com/agent-inject-secret-database-creds.txt: "database/creds/my-app-role"
        # Specify the template for the injected secret file (HCL or JSON)
        # This will create a file named database-creds.txt in /vault/secrets
        vault.hashicorp.com/agent-inject-template-database-creds.txt: |
          {{- with secret "database/creds/my-app-role" -}}
          username="{{ .Data.username }}"
          password="{{ .Data.password }}"
          db_host="localhost" # Or your actual DB host
          db_port="5432"
          db_name="mydb"
          {{- end -}}
    spec:
      serviceAccountName: myapp-sa
      containers:
      - name: my-app-container
        image: busybox:latest # Replace with your actual application image
        command: ["sh", "-c"]
        args:
        - |
          echo "Application started. Waiting for secrets..."
          # Vault Agent will mount secrets in /vault/secrets
          # Wait for the secrets to be ready (optional, but good for robust apps)
          while [ ! -f /vault/secrets/database-creds.txt ]; do
            echo "Waiting for /vault/secrets/database-creds.txt to appear..."
            sleep 2
          done
          echo "Secrets found!"
          cat /vault/secrets/database-creds.txt
          # In a real application, you would parse these credentials
          # and use them to connect to PostgreSQL.
          # Example:
          # PG_USERNAME=$(grep username /vault/secrets/database-creds.txt | cut -d'=' -f2 | tr -d ' ')
          # PG_PASSWORD=$(grep password /vault/secrets/database-creds.txt | cut -d'=' -f2 | tr -d ' ')
          # psql -h $db_host -p $db_port -U $PG_USERNAME -d $db_name -W $PG_PASSWORD
          sleep 3600 # Keep the container running for demonstration

kubectl apply -f postgres-app-deployment.yaml

Let's break down the key annotations:

• `vault.hashicorp.com/agent-inject: "true"`: This annotation tells the Vault Agent Injector to process this pod.
• `vault.hashicorp.com/role: "my-app-role"`: This specifies the Vault Kubernetes authentication role that the pod's ServiceAccount will use to authenticate with Vault.
• `vault.hashicorp.com/agent-inject-secret-database-creds.txt: "database/creds/my-app-role"`: This tells the Vault Agent to fetch a secret from `database/creds/my-app-role` and make it available at `/vault/secrets/database-creds.txt` within the pod.
• `vault.hashicorp.com/agent-inject-template-database-creds.txt`: This is a multi-line annotation defining a Go template that the Vault Agent will use to render the fetched secret data into the `database-creds.txt` file. We're extracting `username` and `password` and adding static `db_host`, `db_port`, and `db_name`.

After applying the deployment, the Vault Agent Injector will modify the pod definition, adding a `vault-agent` sidecar container and a shared `vault-secrets` volume. The `vault-agent` container will:

1. Authenticate with Vault using the pod's ServiceAccount token.
2. Fetch the dynamic `database/creds/my-app-role` secret.
3. Render the secret data into `/vault/secrets/database-creds.txt` based on the provided template.
4. Automatically renew the secret lease before it expires.
5. Revoke the secret when the pod is terminated.

You can inspect the logs of your `my-app-container` to see the secrets being printed:

kubectl logs deployment/postgres-app -c my-app-container

You will see the `username` and `password` of the dynamically generated PostgreSQL user. This user will only exist for the lease duration defined in Vault, and Vault will automatically revoke it upon lease expiry or pod termination, significantly reducing the window of exposure for credentials.

Security Considerations

• Vault Security: The security of your entire system hinges on the security of your Vault instance. Ensure Vault is deployed securely with:
  • TLS/SSL: All communication to and from Vault must be encrypted.
  • Secure Storage Backend: Use a robust, highly available, and encrypted storage backend (e.g., cloud object storage with encryption, Consul with TLS).
  • Unseal Process: Implement a robust unseal process, ideally using Shamir's Secret Sharing or an Auto Unseal mechanism (e.g., KMS integration).
  • Network Segmentation: Restrict network access to Vault's API to only necessary clients.
  • Audit Logging: Enable and monitor Vault's audit logs to track all secret access and administrative actions.
• Least Privilege: Always apply the principle of least privilege:
  • Vault Policies: Grant only the necessary `capabilities` on specific `paths` in Vault policies. For example, `read` access for applications, `create`/`update`/`delete` for administrators.
  • Database Privileges: Ensure the `creation_statements` in the database role grant only the minimum required privileges to the dynamically created database users. Avoid granting `ALL PRIVILEGES` unless absolutely necessary.
  • Kubernetes RBAC: The ServiceAccount used by your pods should only have the necessary Kubernetes permissions.
• Template Security: Be cautious with Go templates used by Vault Agent. Ensure they don't accidentally expose sensitive information or allow injection of malicious code.
• Secret Rotation: While dynamic secrets are inherently rotated, ensure your applications are designed to handle secret rotation gracefully without requiring restarts. Vault Agent handles lease renewal transparently, but applications need to re-read the secret files if they cache them.
• Disaster Recovery: Have a clear disaster recovery plan for your Vault instance, including regular backups of its storage backend and the ability to restore and unseal.
• Monitoring: Monitor Vault's health, performance, and audit logs. Set up alerts for failed authentications, secret access, or lease expirations.

Best Practices

• Automate Vault Deployment and Configuration: Use Infrastructure as Code (IaC) tools like Terraform to deploy and configure Vault, its secrets engines, policies, and roles. This ensures consistency, reproducibility, and version control.
• Separate Concerns with Vault Namespaces: For larger organizations or multi-tenancy, leverage Vault namespaces to provide isolation between different teams, applications, or environments.
• Use Specific Database Roles: Instead of a generic `my-app-role`, create specific Vault database roles for each application or microservice, tailoring the `creation_statements` to their precise needs. This reinforces least privilege.
• Secure Vault Agent Injector Deployment: Ensure the Vault Agent Injector itself is deployed securely, with appropriate resource limits, network policies, and RBAC permissions.
• Application Design for Ephemeral Secrets: Design applications to be "secret-aware" and resilient to secret changes. They should read secrets from the mounted files (`/vault/secrets/`) at startup and be able to gracefully handle re-reading them if the underlying content changes (e.g., upon lease renewal leading to a new password).
• Avoid Caching Secrets Indefinitely: Applications should avoid caching secrets indefinitely in memory. Instead, re-read them from the mounted volume when needed, or at least periodically, to pick up renewals.
• Monitor Lease Durations: Keep an eye on the `default_ttl` and `max_ttl` values for your dynamic secret roles. Ensure they are appropriate for your application's needs and security requirements. Shorter leases generally mean better security.
• Integrate with CI/CD: Integrate Vault into your CI/CD pipelines to automatically provision and manage secrets required during build and deployment processes, further reducing manual intervention and static secret exposure.
• Regular Policy Review: Periodically review your Vault policies and authentication roles to ensure they still adhere to the principle of least privilege and reflect the current state of your infrastructure and applications.

FAQ

Q1: How does Vault handle database connection pooling with dynamic secrets?

This is a common concern. When an application uses a connection pool, it typically fetches credentials once and reuses them for multiple connections. With dynamic secrets, the credentials have a limited lease. If the application holds onto expired credentials, it will fail to connect. The Vault Agent sidecar handles lease renewal transparently, refreshing the secret file before the lease expires. Applications using connection pools should be designed to detect changes in the mounted secret file (e.g., by monitoring `inotify` events or periodically checking the file's modification timestamp) and then gracefully refresh their connection pool with the new credentials. Some database drivers or ORMs might have built-in mechanisms for this, or you might need to implement custom logic.

Q2: What happens if Vault goes down or becomes unreachable?

If Vault goes down, existing applications that have already obtained dynamic secrets will continue to function until their current secret lease expires. However, they won't be able to renew their leases, and new pods won't be able to authenticate with Vault or obtain secrets. This underscores the importance of deploying Vault in a highly available configuration with a robust disaster recovery plan. For critical applications, ensure a sufficiently long `default_ttl` to provide a buffer for Vault recovery, but balance this with security requirements. Vault Agent can also cache tokens for a short period to tolerate brief Vault outages.

Q3: Can I use dynamic secrets for other databases or services?

Absolutely! The HashiCorp Vault database secrets engine supports various popular databases beyond PostgreSQL, including MySQL, MongoDB, Microsoft SQL Server, Oracle, Cassandra, and more. The principle remains the same: configure the database connection, define creation and revocation statements for roles, and then applications can request dynamic credentials. Furthermore, Vault supports dynamic credentials for many other services, such as AWS IAM, Azure Service Principals, GCP Service Accounts, SSH keys, and RabbitMQ, making it a versatile tool for managing ephemeral access across your entire infrastructure.

Conclusion