Secure AKS: Workload Identity & Key Vault CSI Driver Integration

How to use Azure AKS with Workload Identity & Key Vault CSI driver. Get secure, seamless secret access for your Kubernetes applications.

Navigating Secure Secrets Management in Azure AKS: A Deep Dive into Workload Identity and Key Vault CSI Driver

In the dynamic landscape of cloud-native applications, securing sensitive information like API keys, database credentials, and certificates is paramount. Kubernetes, while offering robust orchestration capabilities, doesn't inherently provide a secure, native solution for managing secrets at scale across diverse cloud environments. This gap often leads to developers grappling with less-than-ideal practices, such as embedding secrets directly into container images or relying on Kubernetes native secrets, which are merely base64 encoded and not encrypted at rest by default.

Azure Kubernetes Service (AKS) addresses this challenge by integrating seamlessly with Azure's comprehensive security ecosystem. Two pivotal technologies stand out for enhancing secret management and access control within AKS: Azure AD Workload Identity and the Azure Key Vault Container Storage Interface (CSI) driver. Together, they form a powerful, secure, and idiomatic solution for applications running on AKS to consume secrets from Azure Key Vault without ever exposing them to the application code or Kubernetes manifests.

Azure AD Workload Identity is the successor to AAD Pod Identity, offering a more robust and native Kubernetes-centric approach to granting Azure Active Directory (AAD) identities to Kubernetes pods. It leverages Kubernetes service accounts and OpenID Connect (OIDC) federation, allowing pods to authenticate with Azure AD using a federated identity credential. This eliminates the need for any managed identity components to run within the pod itself, simplifying the deployment and reducing the attack surface.

Complementing Workload Identity is the Azure Key Vault CSI driver. This driver enables Kubernetes pods to mount secrets, keys, and certificates stored in an Azure Key Vault as a volume. Applications can then access these secrets directly from the filesystem, just like any other file, abstracting away the complexities of interacting with the Key Vault API. When combined with Workload Identity, the CSI driver can authenticate to Key Vault using the pod's assigned AAD identity, creating a secure, end-to-end secrets delivery mechanism.

This article will guide you through a detailed, step-by-step implementation of setting up Azure AKS with Workload Identity and the Key Vault CSI driver. We'll explore the underlying concepts, provide real-world CLI commands and configuration examples, and discuss essential security considerations and best practices to ensure your applications handle secrets with the highest level of integrity and confidentiality.

Prerequisites

Before we embark on our journey, ensure you have the following prerequisites in place:

An Azure subscription.
Azure CLI installed and configured. Make sure you are logged in to your Azure account (az login).
kubectl installed and configured to connect to your AKS cluster.
Helm CLI installed (version 3+).
Basic understanding of Kubernetes concepts (pods, deployments, service accounts) and Azure resources (resource groups, Key Vault, managed identities).

Step-by-step Implementation

1. Set up Azure Resources

First, we need to create the foundational Azure resources: a resource group, an AKS cluster with OIDC issuer enabled, an Azure Key Vault, and a User-Assigned Managed Identity. The OIDC issuer is crucial for Workload Identity to function.

Create a Resource Group:


az group create --name tech-news-rg --location eastus

Create an AKS Cluster with OIDC Issuer Enabled:

The --enable-oidc-issuer flag is essential for Workload Identity. We also enable the Key Vault CSI driver addon here, simplifying its deployment.


az aks create \
  --resource-group tech-news-rg \
  --name aks-prod-eastus \
  --node-count 1 \
  --generate-ssh-keys \
  --enable-oidc-issuer \
  --enable-addons azurekeyvaultsecretsdriver \
  --network-plugin azure \
  --kubernetes-version 1.28.5

It might take a few minutes for the AKS cluster to provision. Once completed, get the credentials:


az aks get-credentials --resource-group tech-news-rg --name aks-prod-eastus

Create an Azure Key Vault:

We'll create a Key Vault and store a sample secret. Replace my-unique-kv-12345 with a globally unique name.


az keyvault create \
  --name my-unique-kv-12345 \
  --resource-group tech-news-rg \
  --location eastus \
  --sku standard \
  --enabled-for-rbac false # We'll use access policies for simplicity in this example, but RBAC is recommended for production

Add a Sample Secret to Key Vault:


az keyvault secret set \
  --vault-name my-unique-kv-12345 \
  --name MyApplicationSecret \
  --value "SuperSecretValueFromTechNews"

2. Enable Workload Identity in AKS

While the AKS cluster creation command doesn't explicitly "enable" Workload Identity as a separate add-on, enabling the OIDC issuer is the foundational step. The Workload Identity webhook and related components are typically managed by AKS. You can verify its presence:


kubectl get deployments -n kube-system | grep workload-identity-webhook

You should see a deployment like workload-identity-webhook.

3. Configure Key Vault CSI Driver

Since we enabled the Key Vault CSI driver as an AKS add-on during cluster creation (--enable-addons azurekeyvaultsecretsdriver), it's already installed. Now, we need to create a SecretProviderClass resource. This Kubernetes custom resource defines how the CSI driver should connect to Azure Key Vault and what secrets it should retrieve.

Create a file named secretproviderclass.yaml:


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secret-provider
spec:
  provider: azure
  parameters:
    usePodIdentity: "false" # Important: Set to false for Workload Identity
    useEmulator: "false"
    keyvaultName: "my-unique-kv-12345" # Replace with your Key Vault name
    objects: |
      array:
        - |
          objectName: MyApplicationSecret
          objectType: secret
          objectVersion: "" # Latest version
    tenantId: "" # Replace with your Azure Tenant ID

To get your Azure Tenant ID:


az account show --query tenantId -o tsv

Apply the SecretProviderClass:


kubectl apply -f secretproviderclass.yaml

4. Create Azure AD Application and Kubernetes Service Account for Workload Identity

Workload Identity uses an Azure AD application registration (service principal) and federated identity credentials to link a Kubernetes service account to an Azure AD identity. The pod running with that service account will then inherit the permissions of the Azure AD application.

Create an Azure AD Application Registration:


APP_NAME="my-aks-app-identity"
APP_ID=$(az ad app create --display-name $APP_NAME --query appId -o tsv)
echo "Azure AD App ID: $APP_ID"

Create a Kubernetes Service Account:

This service account will be used by our application pod. We need to annotate it with the Azure AD client ID.


SERVICE_ACCOUNT_NAME="my-app-service-account"
NAMESPACE="default" # Or your desired namespace

kubectl create serviceaccount $SERVICE_ACCOUNT_NAME --namespace $NAMESPACE

# Get the OIDC issuer URL from your AKS cluster
AKS_OIDC_ISSUER=$(az aks show -g tech-news-rg -n aks-prod-eastus --query "oidcIssuerProfile.issuerUrl" -o tsv)
echo "AKS OIDC Issuer: $AKS_OIDC_ISSUER"

# Get the service account ID
SERVICE_ACCOUNT_ID=$(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath='{.metadata.uid}')
echo "Service Account ID: $SERVICE_ACCOUNT_ID"

# Annotate the Service Account
kubectl annotate serviceaccount $SERVICE_ACCOUNT_NAME \
    azure.workload.identity/client-id="${APP_ID}" \
    --overwrite \
    --namespace $NAMESPACE

Create a Federated Identity Credential:

This step links the Azure AD application to the Kubernetes service account, allowing the service account to exchange its OIDC token for an Azure AD token.


az ad app federated-credential create \
  --id $APP_ID \
  --name "aks-federated-credential" \
  --issuer "${AKS_OIDC_ISSUER}" \
  --subject "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
  --description "Kubernetes federated credential" \
  --audiences "api://AzureADTokenExchange"

5. Grant Permissions

Now, we need to grant the Azure AD application (which our pod will assume via Workload Identity) permissions to access the Key Vault secrets.

Grant Key Vault Secret Permissions to the Azure AD Application:

We'll grant the "Get" and "List" secret permissions. Replace my-unique-kv-12345 with your Key Vault name and APP_ID with the ID obtained earlier.


az keyvault set-policy \
  --name my-unique-kv-12345 \
  --secret-permissions get list \
  --spn $APP_ID

Note on RBAC vs. Access Policies: For production environments, Azure Key Vault RBAC (Role-Based Access Control) is generally recommended over traditional access policies. If using RBAC, you would assign roles like "Key Vault Secrets User" to the Managed Identity (or the Service Principal for the AAD App) on the Key Vault scope.

6. Deploy Sample Application

Finally, we'll deploy a sample application that uses the configured service account and mounts the secret from Key Vault via the CSI driver.

Create a file named deployment.yaml:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-reader-app
  labels:
    app: secret-reader-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret-reader-app
  template:
    metadata:
      labels:
        app: secret-reader-app
        azure.workload.identity/use: "true" # Crucial annotation for Workload Identity
    spec:
      serviceAccountName: my-app-service-account # Use the service account we created
      containers:
        - name: busybox
          image: busybox:1.36
          command: ["sh", "-c", "sleep 3600"] # Keep container running for demonstration
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kv-secret-provider" # Reference our SecretProviderClass

Apply the deployment:


kubectl apply -f deployment.yaml

Verify Secret Access:

After the pod is running, exec into it and verify that the secret is mounted and accessible.


kubectl get pods -l app=secret-reader-app
# Note the pod name, e.g., secret-reader-app-xxxxxxxxx-yyyyy

POD_NAME=$(kubectl get pods -l app=secret-reader-app -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it $POD_NAME -- cat /mnt/secrets-store/MyApplicationSecret

You should see the output: SuperSecretValueFromTechNews. This confirms that your application successfully retrieved the secret from Azure Key Vault using Workload Identity and the CSI driver!

Security Considerations

Least Privilege: Always apply the principle of least privilege. Grant only the necessary Key Vault permissions (e.g., get, list) to the Azure AD application. Avoid granting broader permissions like set or delete unless absolutely required.
Secret Rotation: Implement a strategy for regular secret rotation in Key Vault. While the CSI driver can automatically update mounted secrets (by default, every 2 minutes), rotating the underlying secrets in Key Vault is a critical security practice.
Network Security: Configure Key Vault network access restrictions (firewalls, private endpoints) to only allow connections from your AKS cluster's virtual network. This prevents unauthorized access attempts from outside your trusted network perimeter.
Auditing and Logging: Enable logging for Azure Key Vault to monitor access patterns and detect suspicious activities. Integrate these logs with Azure Monitor or a SIEM solution. Similarly, monitor AKS audit logs for service account usage.
Kubernetes RBAC: Use Kubernetes Role-Based Access Control (RBAC) to restrict which users or service accounts can create or modify SecretProviderClass resources and deployments that use specific service accounts.
Workload Identity Lifecycle: Regularly review and prune unused Azure AD application registrations, federated identity credentials, and Kubernetes service accounts to minimize potential attack vectors.
OIDC Issuer Trust: Ensure your AKS cluster's OIDC issuer URL is correctly configured and trusted. Malicious actors could try to impersonate an OIDC issuer to gain unauthorized access.

Best Practices

Automate Everything: Use Infrastructure as Code (IaC) tools like Terraform, Bicep, or ARM templates to provision and configure all Azure resources (AKS, Key Vault, Azure AD apps, federated credentials) and Kubernetes resources (SecretProviderClass, deployments). This ensures consistency, repeatability, and reduces human error.
Environment-Specific Configuration: Maintain separate Key Vaults and Azure AD applications for different environments (dev, staging, prod). This prevents accidental exposure of production secrets and allows for fine-grained access control.
Version Control Your Kubernetes Manifests: Store your SecretProviderClass and deployment YAML files in a version control system (e.g., Git) to track changes, facilitate collaboration, and enable rollbacks.
Health Monitoring: Monitor the health and logs of the Key Vault CSI driver pods (usually in the kube-system namespace) to ensure they are functioning correctly and secrets are being refreshed.
Application Design: Design your applications to gracefully handle scenarios where secrets might temporarily be unavailable or refreshed. Applications should read secrets from the mounted volume at startup and potentially re-read them if they detect changes (though the CSI driver handles the refresh, the application simply reads the updated file).
Avoid Direct Key Vault API Calls: For applications running in AKS, prefer using the Key Vault CSI driver instead of making direct API calls to Key Vault. The CSI driver handles authentication, caching, and rotation, simplifying application code and enhancing security.
Transition from AAD Pod Identity: If you are currently using AAD Pod Identity, plan a migration strategy to Workload Identity, as it is the recommended and more secure approach going forward.

FAQ

Q1: What are the main advantages of Workload Identity over AAD Pod Identity?

Workload Identity offers several key advantages: it's more Kubernetes-native, leveraging standard service accounts and OIDC federation; it eliminates the need for mutating webhooks to inject identity sidecars, reducing complexity and potential conflicts; and it offers better performance and scalability by removing the dependency on the NMI (Node Managed Identity) pods, which were a bottleneck in AAD Pod Identity. It's also the officially recommended path forward by Microsoft.

Q2: How often do secrets mounted by the Key Vault CSI driver get refreshed?

By default, the Azure Key Vault CSI driver polls Key Vault for secret updates every two minutes. If a secret, key, or certificate in Key Vault is updated, the CSI driver will automatically refresh the mounted volume in the pod, making the new version available to the application. This interval can be configured via the syncInterval parameter in the SecretProviderClass, though it's typically best to stick to the default unless specific requirements dictate otherwise.

Q3: Can I use Workload Identity and Key Vault CSI driver with multiple Key Vaults?

Yes, absolutely. You can create multiple SecretProviderClass resources, each pointing to a different Azure Key Vault. Each SecretProviderClass can then be associated with a specific Kubernetes deployment or pod. This allows for fine-grained control, enabling different applications or components within the same application to access secrets from distinct Key Vaults, adhering to the principle of least privilege.

Conclusion

The combination of Azure AD Workload Identity and the Azure Key Vault CSI driver represents a significant leap forward in secure secrets management for applications deployed on Azure Kubernetes Service. By leveraging these powerful integrations, organizations can eliminate the risks associated with hardcoding credentials, storing secrets in plain text, or relying on less secure methods.

Workload Identity provides a robust, Kubernetes-native mechanism for pods to securely authenticate with Azure AD, while the Key Vault CSI driver seamlessly injects secrets into the application filesystem. This synergy empowers developers to build more secure, compliant, and scalable cloud-native applications, knowing that their sensitive data is protected by Azure's enterprise-grade security features. Adopting these technologies is not just a best practice; it's a fundamental requirement for modern, secure cloud deployments.

As the cloud-native ecosystem continues to evolve, solutions like Workload Identity and the Key Vault CSI driver underscore Azure's commitment to providing developers with secure, efficient, and integrated tools to manage their infrastructure and applications. Embracing these patterns will undoubtedly lead to more resilient and trustworthy systems.

Secure AKS: Workload Identity & Key Vault CSI Driver Integration