Secure Azure AKS Secrets: Workload Identity & Key Vault CSI Driver

Master Azure AKS Workload Identity & Key Vault CSI driver. Securely access secrets from Key Vault in your Kubernetes pods.

Navigating Secure Secrets Management in Azure AKS: A Deep Dive into Workload Identity and Key Vault CSI Driver

As organizations increasingly adopt Kubernetes for container orchestration, the secure management of application secrets becomes paramount. Traditional methods involving storing secrets directly in container images, environment variables, or even Kubernetes native secrets (which are base64 encoded, not truly encrypted at rest by default in all scenarios) pose significant security risks. In the dynamic world of cloud-native development, a robust and secure mechanism for applications to access sensitive information like database connection strings, API keys, and certificates is non-negotiable.

Azure Kubernetes Service (AKS) offers a powerful combination of features to address this challenge: Azure AD Workload Identity and the Azure Key Vault Provider for Secrets Store CSI Driver. Together, these technologies provide a seamless, secure, and Kubernetes-native way for your applications running in AKS to authenticate with Azure Active Directory (Azure AD) and retrieve secrets directly from Azure Key Vault without ever exposing them to your application code or Kubernetes manifests.

The Problem with Traditional Secret Management

Before diving into the solution, let's briefly touch upon the common pitfalls:

Kubernetes Secrets: While better than hardcoding, Kubernetes Secrets are base64 encoded by default. Without additional encryption at rest (e.g., using AAD-backed encryption for etcd), they can be easily decoded by anyone with access to the cluster's etcd or API server.
Environment Variables: Exposing secrets as environment variables makes them visible to anyone with access to the pod's shell (e.g., `kubectl exec`). They also persist in pod definitions and logs.
Container Images: Baking secrets into container images makes them immutable, difficult to rotate, and easily discoverable if the image is compromised.
Application-Managed Credentials: Storing credentials directly within application code or configuration files is a serious security anti-pattern, making rotation difficult and increasing the risk of exposure.

The goal is to eliminate the need for developers to manage credentials directly within their applications, shifting the responsibility to a secure, managed identity system.

Introducing the Solution: Workload Identity and Key Vault CSI Driver

This article will guide you through setting up a secure secrets management pipeline in AKS using Azure AD Workload Identity and the Key Vault CSI driver. This approach offers:

Zero Trust Principle: Applications authenticate using a managed identity tied to a Kubernetes Service Account, removing the need for long-lived credentials.
Least Privilege: Granting only the necessary permissions to specific identities for specific secrets.
Centralized Secret Management: Azure Key Vault provides a secure, auditable, and highly available store for all your application secrets.
Kubernetes Native Experience: Secrets are mounted as files or environment variables directly into the pod's filesystem, making them accessible to applications without code changes in many cases.
Automatic Rotation: Secrets stored in Key Vault can be rotated automatically or manually, and the CSI driver can refresh them in the pod without restarting the application (depending on application design).

Let's get started.

Prerequisites

Before we begin, ensure you have the following tools and resources configured:

Azure Subscription: An active Azure subscription.
Azure CLI: Installed and configured. Ensure you're logged in with an account that has permissions to create resource groups, AKS clusters, Key Vaults, and Managed Identities.
kubectl: Installed and configured to connect to your AKS cluster.
Helm (Optional but Recommended): For easier installation of certain components, though we'll primarily use `kubectl` for clarity.
Existing AKS Cluster: An AKS cluster running Kubernetes version 1.22 or higher is recommended, as Workload Identity relies on Kubernetes ServiceAccountTokenVolumeProjection. Workload Identity should be enabled on your cluster.
Existing Azure Key Vault: A Key Vault instance where your secrets are stored.

For this walkthrough, we will assume you have an AKS cluster and a Key Vault already provisioned. If not, don't worry, we'll cover the basic creation commands.

Step-by-Step Implementation

Let's walk through the process of setting up Workload Identity and the Key Vault CSI driver to securely retrieve secrets.

Step 1: Create or Update AKS Cluster with Workload Identity Enabled

If you don't have an AKS cluster, create one. Ensure Workload Identity is enabled. If your existing cluster doesn't have it enabled, you can update it.

1.1. Create Resource Group


RESOURCE_GROUP="TechNewsVentureRG"
LOCATION="eastus"

az group create --name $RESOURCE_GROUP --location $LOCATION

1.2. Create AKS Cluster with Workload Identity

When creating the cluster, enable the oidcIssuer and workloadIdentity features. These are crucial for Workload Identity to function.


CLUSTER_NAME="tnv-aks-cluster"

az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --node-count 1 \
    --enable-oidc-issuer \
    --enable-workload-identity \
    --generate-ssh-keys \
    --kubernetes-version 1.27.7 # Or your preferred version >= 1.22

1.3. Get AKS Cluster Credentials


az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --overwrite-existing

1.4. Update Existing AKS Cluster (if Workload Identity is not enabled)

If you have an existing cluster, you can enable these features:


az aks update \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --enable-oidc-issuer \
    --enable-workload-identity

Verify the OIDC issuer URL for your cluster:


AKS_OIDC_ISSUER="$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo $AKS_OIDC_ISSUER

This URL will be used later when creating the federated identity credential.

Step 2: Create Azure Key Vault and Store a Secret

If you don't have a Key Vault, create one and add a secret to it.

2.1. Create Key Vault


KEYVAULT_NAME="tnv-kv-secrets-001" # Must be globally unique
TENANT_ID="$(az account show --query tenantId -o tsv)"

az keyvault create \
    --name $KEYVAULT_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --enabled-for-rbac false # We'll use access policies for simplicity in this demo, but RBAC is recommended for production.

Note on Access Policies vs. RBAC: While we use --enabled-for-rbac false and will manage access via Key Vault access policies for this demonstration's simplicity, Azure RBAC is the recommended and more granular approach for managing Key Vault permissions in production environments. If using RBAC, you would assign roles like "Key Vault Secret User" to your managed identity.

2.2. Store a Secret in Key Vault


SECRET_NAME="my-app-secret"
SECRET_VALUE="SuperSecretValue123!"

az keyvault secret set \
    --vault-name $KEYVAULT_NAME \
    --name $SECRET_NAME \
    --value $SECRET_VALUE

Step 3: Create an Azure Managed Identity

This identity will be used by your application pods to authenticate with Azure AD.


USER_ASSIGNED_IDENTITY_NAME="tnv-aks-workload-id"

az identity create \
    --name $USER_ASSIGNED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION

Retrieve the Client ID of the newly created managed identity:


USER_ASSIGNED_IDENTITY_CLIENT_ID="$(az identity show --name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --query "clientId" -o tsv)"
echo "Managed Identity Client ID: $USER_ASSIGNED_IDENTITY_CLIENT_ID"

Step 4: Grant Permissions to the Managed Identity on Key Vault

The managed identity needs permissions to retrieve secrets from your Key Vault. We'll grant it the 'Get' permission for secrets.


az keyvault set-policy \
    --name $KEYVAULT_NAME \
    --secret-permissions get \
    --object-id $USER_ASSIGNED_IDENTITY_CLIENT_ID

Step 5: Establish Federated Identity Credential

This is the core of Workload Identity. It creates a trust relationship between your Azure Managed Identity and the OIDC issuer of your AKS cluster. This allows the managed identity to exchange a Kubernetes Service Account token for an Azure AD token.


FEDERATED_IDENTITY_CREDENTIAL_NAME="tnv-aks-fid"
AKS_OIDC_ISSUER="$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -o tsv)"

az identity federated-credential create \
    --name $FEDERATED_IDENTITY_CREDENTIAL_NAME \
    --identity-name $USER_ASSIGNED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --issuer "$AKS_OIDC_ISSUER" \
    --subject "system:serviceaccount:default:tnv-app-sa" \
    --audience "api://AzureADTokenExchange"

Explanation:

`--issuer`: The OIDC issuer URL of your AKS cluster.
`--subject`: This specifies the Kubernetes Service Account that will be allowed to use this federated credential. The format is `system:serviceaccount::`. Here, we're assuming a service account named `tnv-app-sa` in the `default` namespace. We will create this service account in the next step.
`--audience`: This is a standard value for Azure AD token exchange.

Step 6: Create a Kubernetes Service Account

Your application pods will use this Kubernetes Service Account. We'll annotate it with the Client ID of the Azure Managed Identity.

Create a file named `service-account.yaml`:


apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: "" # Replace with your Managed Identity Client ID
  name: tnv-app-sa
  namespace: default

Note: Replace `` with the actual value of `$USER_ASSIGNED_IDENTITY_CLIENT_ID` obtained in Step 3.


# Replace the placeholder in the YAML file
sed -i "s||$USER_ASSIGNED_IDENTITY_CLIENT_ID|g" service-account.yaml

kubectl apply -f service-account.yaml

Step 7: Install and Configure Key Vault CSI Driver

The Key Vault CSI driver is an AKS add-on. If you created your AKS cluster with Workload Identity enabled, the CSI driver should already be enabled. You can verify:


az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query "addonProfiles.azureKeyvaultSecretsProvider.enabled" -o tsv

If it returns `true`, it's enabled. If `false`, enable it:


az aks enable-addons --addons azure-keyvault-secrets-provider --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP

Step 8: Create a SecretProviderClass

This Kubernetes custom resource tells the CSI driver *which* secrets to retrieve from Key Vault and *how* to mount them.

Create a file named `secret-provider-class.yaml`:


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-tnv
spec:
  provider: azure
  parameters:
    usePodIdentity: "false" # Set to false for Workload Identity
    useVMManagedIdentity: "false" # Set to false for Workload Identity
    userAssignedIdentityID: "" # Client ID of the managed identity
    keyvaultName: "" # Name of your Key Vault
    cloud: "AzurePublicCloud" # Or "AzureUSGovernment", "AzureChinaCloud"
    objects: |
      array:
        - |
          objectName: my-app-secret # Name of the secret in Key Vault
          objectType: secret
          objectVersion: "" # Optional: specify a version, or leave empty for latest
    tenantId: "" # Your Azure Tenant ID

Note: Replace the placeholders with your actual values.


# Replace placeholders in the YAML file
sed -i "s||$USER_ASSIGNED_IDENTITY_CLIENT_ID|g" secret-provider-class.yaml
sed -i "s||$KEYVAULT_NAME|g" secret-provider-class.yaml
sed -i "s||$TENANT_ID|g" secret-provider-class.yaml

kubectl apply -f secret-provider-class.yaml

Step 9: Deploy an Application Using the Mounted Secrets

Finally, deploy an application that consumes the secrets. We'll use a simple Nginx deployment that mounts the secret as a file.

Create a file named `deployment.yaml`:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: tnv-app-deployment
  labels:
    app: tnv-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tnv-app
  template:
    metadata:
      labels:
        app: tnv-app
        aadpodidbinding: "false" # Important: Set to false for Workload Identity
      annotations:
        azure.workload.identity/use: "true" # Enable Workload Identity for this pod
    spec:
      serviceAccountName: tnv-app-sa # Link to the service account created earlier
      containers:
        - name: tnv-app-container
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          # Example of how to use the secret (e.g., in a startup script)
          # command: ["/bin/sh", "-c"]
          # args: ["echo 'Secret content:' && cat /mnt/secrets-store/my-app-secret && sleep 3600"]
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-keyvault-tnv"

Key points in the Deployment YAML:

`serviceAccountName: tnv-app-sa`: Links the pod to the Kubernetes Service Account configured with Workload Identity.
`azure.workload.identity/use: "true"`: This annotation on the pod template enables Workload Identity injection for the pod.
`volumeMounts` and `volumes`: These sections define how the Key Vault secrets are mounted into the container's filesystem. The `secretProviderClass: "azure-keyvault-tnv"` links to the `SecretProviderClass` we defined.


kubectl apply -f deployment.yaml

Step 10: Verify Secret Access

Wait for the pod to be running, then exec into it and check the mounted secret.


kubectl get pods -l app=tnv-app

# Once the pod is running, get its name
POD_NAME=$(kubectl get pods -l app=tnv-app -o jsonpath='{.items[0].metadata.name}')

# Exec into the pod and list the mounted secrets
kubectl exec -it $POD_NAME -- ls /mnt/secrets-store

# View the secret content
kubectl exec -it $POD_NAME -- cat /mnt/secrets-store/my-app-secret

You should see `my-app-secret` listed and its content displayed as `SuperSecretValue123!`. This confirms your application successfully retrieved the secret from Key Vault using Workload Identity and the CSI driver.

Security Considerations

Implementing secure secret management requires careful thought beyond just the technical setup:

Least Privilege: Always grant the Managed Identity only the minimum necessary permissions (e.g., `secrets get`, not `secrets list` or `secrets set`).
Azure RBAC for Key Vault: While we used access policies for simplicity, Azure RBAC offers more granular control and is recommended for production Key Vaults.
Network Security: Restrict Key Vault access to specific virtual networks or IP ranges using Key Vault firewall rules. Ensure your AKS cluster nodes can reach Key Vault. Consider Private Endpoints for Key Vault for enhanced security.
Audit Logging: Enable diagnostic settings for Azure Key Vault to send logs to Azure Monitor, Log Analytics, or a SIEM solution. Monitor for unauthorized access attempts or unusual secret retrieval patterns.
Secret Rotation: Implement a strategy for regular secret rotation. Key Vault supports automatic rotation for certain secret types (e.g., storage account keys). The CSI driver will automatically refresh mounted secrets in pods when they are rotated in Key Vault (with a configurable polling interval).
Workload Identity Best Practices: Use distinct Managed Identities and Kubernetes Service Accounts for different applications or microservices to maintain strict isolation and least privilege.
Admission Controllers: Utilize Kubernetes admission controllers like Gatekeeper or Azure Policy for Kubernetes to enforce policies, such as ensuring all deployments use a `SecretProviderClass` or specific annotations.

Best Practices

Dedicated Managed Identities: Create a unique user-assigned managed identity for each application or microservice that requires access to Azure resources. This limits the blast radius if an identity is compromised.
Namespace Isolation: Deploy applications into dedicated namespaces and create Service Accounts within those namespaces.
Environment-Specific Key Vaults: Maintain separate Key Vault instances for different environments (Development, Staging, Production) to prevent accidental cross-environment access and enforce environment-specific security policies.
Automate Deployment: Integrate the creation and configuration of Managed Identities, Federated Credentials, Key Vaults, and Kubernetes resources into your CI/CD pipelines. This ensures consistency and reduces manual errors.
Monitor and Alert: Set up monitoring and alerting for Key Vault access logs, especially for failed attempts or access from unusual locations. Also monitor the health of your AKS cluster and CSI driver pods.
Consider External Secrets Operator: For more complex scenarios or when you need to create native Kubernetes Secrets from Key Vault secrets, consider using tools like the External Secrets Operator alongside the CSI driver. This can be useful for applications that expect Kubernetes Secrets specifically.
Application Design for Secret Refresh: While the CSI driver can refresh mounted secrets, applications might need to be designed to re-read these files or restart to pick up new values immediately. For critical secrets, plan for graceful restarts or dynamic configuration reloading.

FAQ

Q1: What's the main difference between Azure AD Workload Identity and the deprecated AAD Pod Identity?

A1: Azure AD Workload Identity is the successor and recommended approach. AAD Pod Identity used a mutating webhook and Node Managed Identity (NMI) components to intercept token requests, which could sometimes lead to conflicts or complex debugging. Workload Identity leverages Kubernetes native features like `ServiceAccountTokenVolumeProjection` and OIDC federation. It binds an Azure Managed Identity directly to a Kubernetes Service Account, simplifying the architecture, improving security posture by not requiring a mutating webhook, and being more aligned with upstream Kubernetes best practices.

Q2: Can I use the Key Vault CSI driver to mount secrets as environment variables instead of files?

A2: Yes, the Key Vault CSI driver allows you to project secrets from the mounted file system into environment variables using Kubernetes' `envFrom` or `env` definitions in your pod specification. You would typically mount the secrets to a path (e.g., `/mnt/secrets-store`) and then use a `configMapKeyRef` or `secretKeyRef` (referencing a Kubernetes Secret that itself was created by the CSI driver, if configured) or a custom script to read the files and set them as environment variables during container startup.


# Example using envFrom to read from a Kubernetes Secret created by CSI driver
# This requires a 'Secret Sync' feature of the CSI driver, enabled in SecretProviderClass
# or using External Secrets Operator
# containers:
#   - name: my-app
#     image: myapp:latest
#     envFrom:
#       - secretRef:
#           name: my-synced-secret-name

Alternatively, the application can directly read the secret from the mounted file path ` /mnt/secrets-store/my-app-secret`.

Q3: How does secret rotation in Key Vault affect applications using the CSI driver?

A3: When a secret is rotated in Azure Key Vault, the Key Vault CSI driver, by default, polls Key Vault every minute (configurable via `syncInterval` in `SecretProviderClass`) to check for updated versions. If a new version is detected, the CSI driver automatically updates the mounted secret file(s) within the pod's filesystem. However, it's crucial that your application is designed to gracefully handle this. Applications should periodically re-read the secret files rather than reading them once at startup and caching them indefinitely. If an application caches the secret and doesn't re-read, it won't pick up the rotated value until the pod restarts.

Conclusion

The combination of Azure AD Workload Identity and the Azure Key Vault Provider for Secrets Store CSI Driver represents a significant leap forward in secure secrets management for applications running on Azure Kubernetes Service. By eliminating the need to embed credentials in application code or Kubernetes manifests, this architecture enhances security, simplifies operational overhead, and aligns perfectly with the principles of zero trust and least privilege.

Adopting this pattern ensures that your sensitive application data is managed centrally, securely, and with a robust audit trail, providing peace of mind for developers and security teams alike. As you continue your cloud-native journey on Azure, embracing these powerful capabilities will be instrumental in building resilient and secure applications.

Secure Azure AKS Secrets: Workload Identity & Key Vault CSI Driver