Azure AKS: Secure Secrets Management with Workload Identity & Key Vaul

Securely access Azure Key Vault secrets in AKS using Workload Identity & CSI driver. Learn setup, benefits, and best practices for your Kubernetes apps.

Unlocking Secure Secret Management in AKS: A Deep Dive into Workload Identity and Key Vault CSI Driver

In the dynamic world of cloud-native applications, managing secrets securely is paramount. Kubernetes, while providing a powerful orchestration platform, presents its own set of challenges when it comes to handling sensitive information like API keys, database credentials, and certificates. Traditional methods often involve storing secrets directly in Kubernetes, which, despite base64 encoding, isn't encryption at rest and requires careful RBAC management. External secret stores like Azure Key Vault offer a more robust solution, but bridging the gap between Kubernetes pods and these external stores securely and efficiently has been an evolving journey.

Enter Azure Workload Identity and the Azure Key Vault Container Storage Interface (CSI) driver. This powerful combination represents the modern, secure, and idiomatic way to inject secrets from Azure Key Vault directly into your Azure Kubernetes Service (AKS) pods. It eliminates the need for manual secret rotation, reduces the attack surface by removing hardcoded credentials, and simplifies compliance by centralizing secret management in a dedicated, highly secure service.

As a senior technology writer at TechNews Venture, I've witnessed firsthand the transformation this architecture brings to enterprises grappling with secret sprawl and compliance mandates. This article will provide a comprehensive, step-by-step guide to integrating Azure Workload Identity with the Key Vault CSI driver in your AKS clusters, alongside crucial security considerations, best practices, and a detailed FAQ.

Overview: The Power Duo for Secure Secrets

Before we dive into the implementation, let's understand the core components and why they are so effective together:

Azure Key Vault: A cloud service for securely storing and accessing secrets, keys, and certificates. It provides hardware security module (HSM)-backed storage, granular access policies, and comprehensive auditing. It's the central repository for our sensitive data.
Azure Key Vault CSI Driver: This Kubernetes driver allows pods to mount secrets, keys, and certificates stored in Azure Key Vault as a volume. Instead of the application directly calling Key Vault APIs, the secrets are projected into the pod's filesystem, making them accessible just like any other file. This eliminates the need for applications to include Key Vault SDKs or manage authentication to Key Vault themselves.
Azure Workload Identity: This feature in AKS enables Kubernetes applications to access Azure resources securely using Azure Active Directory (Azure AD) identities. It's built on top of Kubernetes' native Service Accounts and OpenID Connect (OIDC). Instead of using pod identities (which are deprecated) or client secrets, Workload Identity allows a Kubernetes Service Account to act as an Azure AD identity. This means your pods can authenticate to Azure services like Key Vault, Storage Accounts, or databases without needing any secrets stored within Kubernetes itself. The authentication flow is managed by Azure AD, leveraging federated credentials.

Together, the Key Vault CSI driver uses Azure Workload Identity to authenticate to Azure Key Vault. The pod requests a secret via the CSI driver, the driver authenticates to Key Vault using the pod's associated Workload Identity, fetches the secret, and mounts it into the pod's filesystem. This entire process is seamless, secure, and adheres to the principle of least privilege.

Prerequisites

Before we begin, ensure you have the following in place:

Azure Subscription: An active Azure subscription.
Azure CLI: Version 2.40.0 or higher.
```
az --version
        
```
kubectl: Configured to connect to your AKS cluster.
```
kubectl version --client
        
```
Permissions: An Azure account with permissions to create resource groups, AKS clusters, managed identities, and Key Vaults, and assign roles.
jq: A lightweight and flexible command-line JSON processor.
```
jq --version
        
```

Step-by-step Implementation

Let's walk through the process of setting up AKS with Workload Identity and integrating it with Azure Key Vault using the CSI driver.

1. Create an Azure Resource Group

First, we'll create a resource group to hold all our resources.


RESOURCE_GROUP="technevs-aks-wi-kv-rg"
LOCATION="eastus"

az group create --name $RESOURCE_GROUP --location $LOCATION

2. Create an AKS Cluster with Workload Identity and Key Vault CSI Driver Enabled

Now, create an AKS cluster. Crucially, we need to enable Workload Identity (`--enable-workload-identity`), OIDC Issuer (`--enable-oidc-issuer`), and the Key Vault CSI driver (`--enable-secret-store-csi-driver`) at cluster creation time.


AKS_CLUSTER_NAME="technevs-aks-wi-kv"
NODE_COUNT=3 # Adjust as needed

az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME \
    --node-count $NODE_COUNT \
    --enable-oidc-issuer \
    --enable-workload-identity \
    --enable-secret-store-csi-driver \
    --generate-ssh-keys \
    --kubernetes-version 1.28.3 # Specify a supported version

This command will take a few minutes to complete. Once done, configure `kubectl` to connect to your new cluster.


az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --overwrite-existing

Verify the OIDC issuer URL for your cluster, which is essential for Workload Identity.


AKS_OIDC_ISSUER="$(az aks show --name $AKS_CLUSTER_NAME --resource-group $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo "AKS OIDC Issuer URL: $AKS_OIDC_ISSUER"

3. Create an Azure AD User-Assigned Managed Identity

This managed identity will be used by our Kubernetes pods to authenticate to Azure Key Vault.


MANAGED_IDENTITY_NAME="technevs-kv-mi"

az identity create \
    --resource-group $RESOURCE_GROUP \
    --name $MANAGED_IDENTITY_NAME

MANAGED_IDENTITY_CLIENT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $MANAGED_IDENTITY_NAME --query "clientId" -o tsv)"
MANAGED_IDENTITY_RESOURCE_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $MANAGED_IDENTITY_NAME --query "id" -o tsv)"
MANAGED_IDENTITY_TENANT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $MANAGED_IDENTITY_NAME --query "tenantId" -o tsv)"

echo "Managed Identity Client ID: $MANAGED_IDENTITY_CLIENT_ID"
echo "Managed Identity Resource ID: $MANAGED_IDENTITY_RESOURCE_ID"
echo "Managed Identity Tenant ID: $MANAGED_IDENTITY_TENANT_ID"

4. Create an Azure Key Vault and Store a Secret

Now, create an Azure Key Vault and add a sample secret that our application will consume.


KEY_VAULT_NAME="technevs-kv-sujay-$(head /dev/urandom | tr -dc a-z0-9 | head -c 8)" # Ensure globally unique name

az keyvault create \
    --name $KEY_VAULT_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --sku standard

# Add a sample secret
az keyvault secret set \
    --vault-name $KEY_VAULT_NAME \
    --name "my-db-password" \
    --value "SuperSecurePassword@123!"

echo "Key Vault Name: $KEY_VAULT_NAME"

5. Grant Permissions to the Managed Identity on Azure Key Vault

The managed identity needs permission to retrieve secrets from the Key Vault. We'll grant it the "Key Vault Secrets User" role.


KEY_VAULT_ID="$(az keyvault show --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP --query id -o tsv)"

az role assignment create \
    --role "Key Vault Secrets User" \
    --assignee $MANAGED_IDENTITY_CLIENT_ID \
    --scope $KEY_VAULT_ID

This grants the necessary data plane access to read secrets.

6. Configure Kubernetes Service Account for Workload Identity

In your Kubernetes cluster, you need a Service Account that your pods will use. This Service Account will be annotated to link it to the Azure AD Managed Identity.


cat < service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: "$MANAGED_IDENTITY_CLIENT_ID"
    azure.workload.identity/tenant-id: "$MANAGED_IDENTITY_TENANT_ID" # Optional, but good practice
  name: technevs-sa
  namespace: default
EOF

kubectl apply -f service-account.yaml

Next, we need to establish a federated credential between the Kubernetes Service Account and the Azure AD Managed Identity. This tells Azure AD that the specified Service Account, when authenticated via OIDC from our AKS cluster's issuer, can be treated as our Managed Identity.


FEDERATED_CREDENTIAL_NAME="technevs-fed-cred"
SERVICE_ACCOUNT_NAME="technevs-sa"
SERVICE_ACCOUNT_NAMESPACE="default"

az identity federated-credential create \
    --name $FEDERATED_CREDENTIAL_NAME \
    --identity-name $MANAGED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --issuer "$AKS_OIDC_ISSUER" \
    --subject "system:serviceaccount:$SERVICE_ACCOUNT_NAMESPACE:$SERVICE_ACCOUNT_NAME"

7. Create a SecretProviderClass

The `SecretProviderClass` custom resource defines which secrets from Key Vault should be made available to your pods and how. It acts as a bridge between the Kubernetes deployment and the Key Vault CSI driver.


cat < secretproviderclass.yaml
apiVersion: secrets-store.csi.k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-technevs-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: "false" # Important: Set to false for Workload Identity
    useEmulator: "false"
    keyvaultName: "$KEY_VAULT_NAME" # Replace with your Key Vault name
    tenantId: "$MANAGED_IDENTITY_TENANT_ID" # Replace with your tenant ID
  secretObjects: # Optional: If you want to create native K8s secrets from Key Vault
  # - secretName: my-db-password-k8s
  #   type: Opaque
  #   data:
  #     - objectName: my-db-password
  #       key: dbPassword
  objects:
    - objectName: my-db-password
      objectType: secret
      objectVersion: "" # Use latest version
EOF

kubectl apply -f secretproviderclass.yaml

Note the `usePodIdentity: "false"` parameter. This is crucial for Workload Identity, as it tells the CSI driver not to use the older pod identity mechanism.

8. Deploy a Sample Application

Finally, deploy an application that uses the `technevs-sa` Service Account and mounts the secrets defined in the `azure-kv-technevs-secrets` SecretProviderClass. The application will then be able to read the secret from the mounted volume.


cat < deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-kv-app
  labels:
    app: nginx-kv-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-kv-app
  template:
    metadata:
      labels:
        app: nginx-kv-app
    spec:
      serviceAccountName: technevs-sa # Link to our Workload Identity enabled Service Account
      containers:
        - name: nginx
          image: nginx:latest
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          env:
            - name: MY_DB_PASSWORD_PATH
              value: "/mnt/secrets-store/my-db-password"
          command: ["/bin/bash", "-c"]
          args:
            - |
              while true; do
                echo "Attempting to read secret..."
                if [ -f "$MY_DB_PASSWORD_PATH" ]; then
                  DB_PASSWORD=$(cat "$MY_DB_PASSWORD_PATH")
                  echo "Successfully read DB Password (first 5 chars): ${DB_PASSWORD:0:5}*****"
                else
                  echo "Secret file not found at $MY_DB_PASSWORD_PATH"
                fi
                sleep 10
              done
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kv-technevs-secrets" # Reference our SecretProviderClass
EOF

kubectl apply -f deployment.yaml

9. Verify Secret Injection

Check the logs of your deployed application to see if it successfully read the secret.


kubectl logs -f deployment/nginx-kv-app

You should see output similar to:

Attempting to read secret...
Successfully read DB Password (first 5 chars): Super*****

This confirms that the Key Vault CSI driver, leveraging Workload Identity, successfully fetched the secret from Azure Key Vault and mounted it into your pod's filesystem.

Security Considerations

While Workload Identity and the Key Vault CSI driver significantly enhance security, it's crucial to understand and implement additional security measures:

Principle of Least Privilege: Always grant the absolute minimum permissions required. For the managed identity, only assign the "Key Vault Secrets User" role (or specific Get/List permissions) to the Key Vault. Avoid granting broader roles like Contributor.
Network Security: Restrict Key Vault access to specific virtual networks or IP ranges. If your AKS cluster uses Azure CNI and is in a VNet, configure Key Vault firewalls to only allow access from your AKS subnet.
Secret Rotation: Leverage Key Vault's built-in secret rotation capabilities. While the CSI driver automatically refreshes mounted secrets, ensure your applications are designed to handle secret rotation gracefully without requiring a pod restart (e.g., by re-reading the file periodically).
Auditing and Logging: Enable diagnostic logging for Azure Key Vault and send logs to an Azure Log Analytics workspace or Azure Storage. Monitor access patterns and alert on suspicious activity. Similarly, monitor AKS audit logs for Service Account usage.
Kubernetes RBAC: Implement strong Kubernetes RBAC to control which users or Service Accounts can create/modify `SecretProviderClass` resources and which pods can use specific Service Accounts.
Container Image Security: Ensure your container images are free of vulnerabilities. A compromised container could potentially access mounted secrets.
Pod Security Standards: Enforce Pod Security Standards (PSS) in your cluster to restrict capabilities, prevent privilege escalation, and ensure containers run with minimal privileges.

Best Practices

To maximize the benefits and maintain a secure, scalable, and manageable environment, consider these best practices:

Environment Separation: Use separate Key Vaults and Managed Identities for different environments (e.g., development, staging, production). This prevents secrets from one environment from accidentally or maliciously being accessed by another.
GitOps Approach: Manage your Kubernetes manifests (Service Accounts, SecretProviderClasses, Deployments) using a GitOps workflow with tools like Flux or Argo CD. This ensures version control, auditability, and automated deployment.
Automated Provisioning: Automate the creation of Key Vaults, Managed Identities, and role assignments using Infrastructure as Code (IaC) tools like Terraform or Bicep.
Non-Root User: Configure your application containers to run as a non-root user. This is a fundamental security practice that limits the impact of a container escape.
Monitoring and Alerting: Implement comprehensive monitoring for both AKS (pod health, Workload Identity errors) and Key Vault (access logs, throttling). Set up alerts for failed secret retrievals or unauthorized access attempts.
Application Design: Design applications to read secrets from the filesystem path where the CSI driver mounts them. Avoid caching secrets indefinitely if rotation is frequent.
Secret Refresh Interval: The CSI driver has a default refresh interval (typically 2 minutes). Be aware of this delay if your application requires immediate secret updates. You can configure this in the `SecretProviderClass`.

FAQ

Q1: What is the main difference between Azure AD Pod Identity (aad-pod-identity) and Azure Workload Identity?

A1: Azure AD Pod Identity (aad-pod-identity) is an older mechanism that uses a controller (MIC - Managed Identity Controller) and a Node Managed Identity (NMI) component. It intercepts IMDS (Instance Metadata Service) calls from pods to redirect them to Azure AD for token acquisition. While effective, it introduces additional components and can have performance implications. Azure Workload Identity, on the other hand, is built on Kubernetes' native Service Accounts and OpenID Connect (OIDC). It leverages federated credentials directly with Azure AD, eliminating the need for IMDS interception or additional components within the cluster, making it more performant, scalable, and a more cloud-native approach. Workload Identity is the recommended and future-proof solution.

Q2: Can I use the Key Vault CSI driver to inject secrets as environment variables instead of files?

A2: Directly, no. The Azure Key Vault CSI driver mounts secrets as files into a volume within your pod. However, you can use an init container or a simple entrypoint script in your main application container to read these files and then export them as environment variables before your main application process starts. For example, a bash script could read /mnt/secrets-store/my-db-password and then set export DB_PASSWORD=$(cat /mnt/secrets-store/my-db-password). While possible, reading from files is generally preferred for security, as environment variables can sometimes be inadvertently exposed (e.g., in logs or by other processes on the same node).

Q3: How does the Key Vault CSI driver handle secret rotation in Key Vault?

A3: The Key Vault CSI driver periodically checks Azure Key Vault for updated versions of the secrets specified in the SecretProviderClass. By default, this refresh interval is 2 minutes. When a secret is updated in Key Vault, the CSI driver automatically fetches the new version and updates the mounted file(s) in the pod's volume. Your application can then be designed to re-read these files periodically or react to file system changes (e.g., using inotify) to consume the new secret without requiring a pod restart. This automated refresh is a significant advantage over manual secret management.

Conclusion

The combination of Azure Workload Identity and the Azure Key Vault CSI driver provides a robust, secure, and developer-friendly solution for managing sensitive application secrets in Azure Kubernetes Service. By leveraging Azure AD identities, it eliminates the need for hardcoded credentials, simplifies compliance, and centralizes secret management in a highly secure service. This architecture not only enhances your security posture but also streamlines the operational overhead associated with secret lifecycle management in a cloud-native environment.

As organizations continue their journey into the cloud-native landscape, adopting mature and secure patterns for identity and secret management becomes non-negotiable. Workload Identity and the Key Vault CSI driver are critical components in building such resilient and secure applications on AKS. Embrace this powerful duo, and you'll unlock a new level of confidence in your Kubernetes deployments.

Azure AKS: Secure Secrets Management with Workload Identity & Key Vault CSI