HashiCorp Vault Dynamic Secrets for PostgreSQL & K8s Workloads

2.4 Create a Kubernetes Auth Role in Vault

This role maps a Kubernetes ServiceAccount (and namespace) to a Vault policy. This is how Vault knows which Kubernetes identity can request which secrets.

vault write auth/kubernetes/role/webapp \
    bound_service_account_names=webapp-sa \
    bound_service_account_namespaces=default \
    policies=webapp-policy \
    ttl=24h

This configures Vault so that any pod in the default namespace using the webapp-sa ServiceAccount can authenticate and receive the webapp-policy, which allows it to fetch credentials from database/creds/webapp-readonly and database/creds/webapp-readwrite.

2.5 Deploy Vault Agent Injector to Kubernetes (if not already present)

If you don't have the Vault Agent Injector installed, you can deploy it using Helm. First, add the HashiCorp Helm repo:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Then, install the Vault chart with the injector enabled. This is a minimal configuration for the injector.

helm install vault hashicorp/vault \
    --set "server.enabled=false" \
    --set "injector.enabled=true" \
    --set "injector.externalVaultAddr=http://host.docker.internal:8200" \
    --set "injector.logLevel=debug" \
    --set "global.enabled=true" \
    --set "global.tlsDisable=true"

Note: injector.externalVaultAddr points to our local Vault running on the Docker host. For a production Vault cluster, this would be your Vault's load balancer or service URL. global.tlsDisable=true is for development convenience; always use TLS in production.

2.6 Deploy a Sample Application in Kubernetes

Now, let's create a Kubernetes Deployment that uses a ServiceAccount and Vault Agent Injector annotations to fetch dynamic PostgreSQL credentials.

First, create the ServiceAccount:

# webapp-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: webapp-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: webapp-pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: webapp-read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: webapp-sa
  namespace: default
roleRef:
  kind: Role
  name: webapp-pod-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f webapp-sa.yaml

Next, the application Deployment. Pay close attention to the annotations under template.metadata.

# webapp-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamic-secret-webapp
  labels:
    app: dynamic-secret-webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamic-secret-webapp
  template:
    metadata:
      labels:
        app: dynamic-secret-webapp
      annotations:
        # Enable Vault Agent Injector for this pod
        vault.hashicorp.com/agent-inject: "true"
        # The Vault authentication role for Kubernetes
        vault.hashicorp.com/role: "webapp"
        # The path to the secret engine and role to retrieve
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/webapp-readwrite"
        # The template to render the secret into a file
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/webapp-readwrite" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          export DB_HOST="host.docker.internal" # Or postgres_db if in same network
          export DB_PORT="5432"
          export DB_NAME="postgres"
          {{- end -}}
    spec:
      serviceAccountName: webapp-sa # Use the ServiceAccount we created
      containers:
      - name: webapp
        image: alpine/git:latest # Using a simple image to demonstrate secret access
        command: ["/bin/sh", "-c"]
        args:
        - |
          # Source the secret file to get environment variables
          . /vault/secrets/db-creds
          echo "Database Username: $DB_USERNAME"
          echo "Database Password: $DB_PASSWORD"
          echo "Connecting to PostgreSQL with dynamic credentials..."
          # In a real application, you would use these env vars to connect to the DB
          # For demonstration, we'll just sleep to keep the pod running
          sleep 3600
        env:
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

kubectl apply -f webapp-deployment.yaml

After the pod starts, inspect its logs:

kubectl get pods -l app=dynamic-secret-webapp
# (Wait for the pod to be Running)
kubectl logs -f $(kubectl get pods -l app=dynamic-secret-webapp -o jsonpath='{.items[0].metadata.name}') -c webapp

You should see output similar to:

Database Username: v-token-webapp-readwrite-xxxxxxxx
Database Password: YzXyFpGz1234567890
Connecting to PostgreSQL with dynamic credentials...

This confirms that the Vault Agent sidecar successfully authenticated with Vault, fetched the dynamic credentials, rendered them into a file (/vault/secrets/db-creds), and the application container sourced that file to access the secrets as environment variables.

The Vault Agent Injector handles the entire lifecycle: authentication, secret fetching, renewal, and rotation. When a secret's lease is about to expire, the agent automatically renews it or fetches a new one. If the lease cannot be renewed, the agent can be configured to notify the application or even restart the pod, ensuring applications always operate with valid, rotated credentials.

Security Considerations

Implementing dynamic secrets with Vault significantly improves security, but certain considerations must be kept in mind:

• Vault Server Hardening: Your Vault server is the heart of your secret management. It must be highly available, run on hardened hosts, use a robust storage backend (e.g., Consul, Integrated Storage with Raft), and all communication must be over TLS. Access to Vault should be strictly controlled and audited.
• Least Privilege:
  • Vault Database Role: The PostgreSQL user Vault uses to create dynamic roles (postgres in our example) should have only the necessary permissions to create/revoke users and grant privileges. Avoid giving it blanket superuser access if possible.
  • Vault Policies: Grant only the necessary read capabilities for specific secret paths to your Kubernetes roles.
  • Kubernetes RBAC: The ServiceAccount used by your applications should have minimal permissions. The ServiceAccount used by Vault for Kubernetes authentication should also be restricted (e.g., system:auth-delegator).
• Network Segmentation: Isolate your Vault server and database servers in private networks. Access should only be allowed from trusted sources (e.g., Kubernetes API server for auth, Vault Agent for secret retrieval).
• Auditing and Logging: Enable Vault's audit devices to log all requests and responses. Integrate these logs with your SIEM solution for monitoring and alerting on suspicious activities.
• Lease Management: Configure appropriate default_ttl and max_ttl for your dynamic secrets. Shorter TTLs reduce the blast radius of a compromised secret. Ensure applications are designed to handle secret rotation gracefully.
• Secret Revocation: Vault automatically revokes dynamic secrets upon lease expiration. In case of a suspected compromise, you can manually revoke leases immediately using vault lease revoke <lease_id>.
• TLS Everywhere: All communication between Vault and its clients (Vault CLI, Vault Agent, applications) and between Vault and its backend storage/databases should be encrypted using TLS.

Best Practices

• Automate Vault Deployment and Configuration: Use Infrastructure as Code (IaC) tools like Terraform or Ansible to deploy and configure your Vault server, secret engines, policies, and roles. This ensures consistency, repeatability, and version control.
• Use Distinct Roles: Create specific Vault roles for different applications or teams (e.g., webapp-readonly, admin-tool-readwrite). Avoid generic roles that grant too much access.
• Regularly Review Policies: Periodically review your Vault policies and Kubernetes RBAC configurations to ensure they adhere to the principle of least privilege and haven't become overly permissive over time.
• Monitor Vault Health and Performance: Set up monitoring and alerting for your Vault cluster's health, performance metrics, and audit logs. This helps in detecting issues and potential security breaches early.
• Implement Robust Backup and Recovery: Have a clear and tested backup and recovery strategy for your Vault data, especially the unseal keys and root token.
• Educate Developers: Train your development teams on how to interact with Vault, the benefits of dynamic secrets, and best practices for secret consumption within their applications.
• Leverage Vault Agent/Agent Injector: For Kubernetes, always prefer the Vault Agent Injector. It simplifies secret management for developers, enhances security by not exposing Vault tokens directly to application code, and handles secret renewal/rotation automatically.
• Secret Zero Problem: Understand that even with dynamic secrets, there's always a "secret zero" – the initial credential or token used by a component (like the Vault Agent) to authenticate with Vault. Ensure these initial tokens are securely managed (e.g., Kubernetes ServiceAccount tokens, cloud IAM roles).

FAQ

Q: What happens if the Vault server goes down?

A: If a Vault server in a production HA setup goes down, another node should take over as the active leader, ensuring continuous service. If the entire cluster is unavailable, applications will lose the ability to fetch new dynamic secrets or renew existing leases. This means that once current leases expire, applications will no longer be able to connect to the database. This highlights the critical importance of a highly available and resilient Vault deployment, as well as designing applications to gracefully handle temporary secret unavailability or restart upon secret rotation.

Q: How often do dynamic secrets rotate? Can I control it?

A: Dynamic secrets rotate based on their lease duration (TTL). When a lease is about to expire, the Vault Agent attempts to renew it or fetch a new one. You control the rotation frequency by setting the default_ttl and max_ttl parameters when defining your database roles in Vault. default_ttl is the initial lease duration, and max_ttl is the maximum duration a secret can exist before a new one must be generated. Shorter TTLs provide higher security but might increase the load on Vault and the database.

Q: Can I use dynamic secrets for other databases or services?

A: Absolutely! HashiCorp Vault supports dynamic secrets for a wide array of databases (MySQL, MSSQL, Oracle, MongoDB, Cassandra, etc.) and other services (AWS IAM, Azure AD, GCP IAM, SSH, RabbitMQ, etc.) through various secret engines. The principles remain the same: configure the respective secret engine, provide Vault with administrative credentials to the target service, define roles, and then allow applications to request credentials based on those roles. This extensibility is one of Vault's core strengths, allowing you to centralize secret management across your entire infrastructure.

Conclusion

The journey from static, perilous secrets to ephemeral, dynamic credentials with HashiCorp Vault is a transformative one for any organization serious about security and operational efficiency. By implementing Vault's dynamic secrets for PostgreSQL and integrating seamlessly with Kubernetes workloads, we've demonstrated a robust pattern for eliminating hardcoded database credentials, reducing the blast radius of potential breaches, and automating the cumbersome process of secret rotation.

This approach empowers developers to focus on building applications without worrying about the intricacies of secret management, while providing security teams with granular control, comprehensive auditing, and a significantly hardened posture against credential-based attacks. The combination of Vault's powerful secret engines and the Kubernetes Agent Injector creates an elegant, secure, and scalable solution for modern cloud-native environments. Embracing dynamic secrets isn't just a best practice; it's an essential component of a resilient and secure DevOps ecosystem.