Azure AKS: Secure Secrets with Workload Identity & Key Vault CSI Drive

Secure AKS secrets! Implement Azure Key Vault with Workload Identity & CSI driver for seamless, secure access from your Kubernetes pods.

Azure AKS with Workload Identity and Key Vault CSI Driver: A Deep Dive into Secure Credential Management

In the evolving landscape of cloud-native applications, securing sensitive information like API keys, database credentials, and certificates is paramount. Kubernetes, while offering robust orchestration capabilities, requires a thoughtful approach to secret management. Traditionally, secrets might be stored directly in Kubernetes secrets (base64 encoded, but not encrypted at rest by default without additional configuration) or injected via environment variables, both of which carry inherent risks. Azure Kubernetes Service (AKS) offers powerful integrations with Azure services, and for secret management, Azure Key Vault stands out as the industry-standard solution.

However, the bridge between AKS pods and Azure Key Vault needs to be secure and seamless. Enter Azure Workload Identity and the Key Vault CSI (Container Storage Interface) driver. This powerful combination allows your Kubernetes workloads to securely access secrets, keys, and certificates stored in Azure Key Vault without embedding credentials directly into your pod definitions or relying on less granular authentication mechanisms. Azure Workload Identity, built on OpenID Connect (OIDC), provides a more secure and flexible way for applications running in AKS to authenticate with Azure AD-protected resources, replacing the older Pod-Managed Identity (AAD Pod Identity) with a more native Kubernetes approach.

This article will guide you through the process of setting up Azure AKS with Workload Identity and integrating it with the Key Vault CSI driver. We'll cover everything from prerequisites to a step-by-step implementation, crucial security considerations, and best practices to ensure your applications handle secrets with the utmost integrity.

Prerequisites

Before we embark on this journey, ensure you have the following tools and resources in place:

Azure Subscription: An active Azure subscription.
Azure CLI: Version 2.47.0 or later installed and configured.
```
az login
```
Kubectl: The Kubernetes command-line tool installed.
```
az aks install-cli
```

Helm: The Kubernetes package manager installed (version 3+).

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

An Existing AKS Cluster (or create a new one): The AKS cluster must have the OIDC issuer enabled. If you're creating a new cluster, this is enabled by default with recent CLI versions. If you have an older cluster, you might need to enable it explicitly.
Permissions: An Azure AD user or service principal with permissions to create resource groups, AKS clusters, Key Vaults, Managed Identities, and assign roles.

For the purpose of this guide, we'll create a new resource group and AKS cluster to ensure a clean slate.

# Define variables for resource naming
RESOURCE_GROUP="rg-techventure-aks-kv"
LOCATION="eastus"
AKS_CLUSTER_NAME="aks-techventure-cluster"
KEY_VAULT_NAME="kv-techventure-secrets-001" # Must be globally unique
MANAGED_IDENTITY_NAME="mi-aks-workload"
KUBERNETES_NAMESPACE="default" # Using default for simplicity, but a dedicated namespace is recommended

# Create an Azure Resource Group
echo "Creating resource group: $RESOURCE_GROUP in $LOCATION..."
az group create --name $RESOURCE_GROUP --location $LOCATION

# Create an AKS cluster with OIDC issuer enabled (default for recent CLI)
echo "Creating AKS cluster: $AKS_CLUSTER_NAME..."
az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME \
    --node-count 1 \
    --enable-oidc-issuer \
    --enable-workload-identity \
    --generate-ssh-keys \
    --node-vm-size Standard_DS2_v2

# Get AKS credentials
echo "Getting AKS credentials..."
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --overwrite-existing

Step-by-step Implementation

Now that our environment is ready, let's walk through the detailed steps to integrate Workload Identity with the Key Vault CSI driver.

1. Create an Azure Key Vault and Store a Secret

First, we need a Key Vault to store our secrets and a sample secret to retrieve.

# Create an Azure Key Vault
echo "Creating Key Vault: $KEY_VAULT_NAME..."
az keyvault create \
    --name $KEY_VAULT_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --sku standard \
    --enable-rbac-authorization false # Using access policies for simplicity in this example

# Store a sample secret
echo "Storing a sample secret in Key Vault..."
az keyvault secret set \
    --vault-name $KEY_VAULT_NAME \
    --name "MyWebAppSecret" \
    --value "SuperSecureValue123!"

2. Create an Azure User-Assigned Managed Identity

This Managed Identity will be used by our AKS pods to authenticate with Azure Key Vault.

# Create a User-Assigned Managed Identity
echo "Creating User-Assigned Managed Identity: $MANAGED_IDENTITY_NAME..."
az identity create \
    --name $MANAGED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION

Retrieve the Client ID of the Managed Identity, which we'll need later.

# Get the Client ID of the Managed Identity
MANAGED_IDENTITY_CLIENT_ID=$(az identity show \
    --name $MANAGED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --query clientId \
    --output tsv)
echo "Managed Identity Client ID: $MANAGED_IDENTITY_CLIENT_ID"

3. Grant Key Vault Permissions to the Managed Identity

The Managed Identity needs permissions to read secrets from our Key Vault.

# Grant the Managed Identity 'Get' and 'List' permissions on secrets in Key Vault
echo "Granting Key Vault permissions to Managed Identity..."
az keyvault set-policy \
    --name $KEY_VAULT_NAME \
    --secret-permissions get list \
    --object-id $MANAGED_IDENTITY_CLIENT_ID

4. Retrieve AKS OIDC Issuer URL

The AKS OIDC issuer URL is crucial for establishing the federated identity credential.

# Get the OIDC issuer URL for the AKS cluster
AKS_OIDC_ISSUER_URL=$(az aks show \
    --name $AKS_CLUSTER_NAME \
    --resource-group $RESOURCE_GROUP \
    --query oidcIssuerProfile.issuerUrl \
    --output tsv)
echo "AKS OIDC Issuer URL: $AKS_OIDC_ISSUER_URL"

5. Create a Kubernetes Service Account and Establish Federated Identity Credential

We'll create a Kubernetes Service Account that our pods will use. This Service Account will then be federated with the Azure Managed Identity.

# Define Kubernetes Service Account name
K8S_SERVICE_ACCOUNT_NAME="my-app-sa"

# Create a Kubernetes Service Account YAML
cat < service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $K8S_SERVICE_ACCOUNT_NAME
  namespace: $KUBERNETES_NAMESPACE
  annotations:
    azure.workload.identity/client-id: "$MANAGED_IDENTITY_CLIENT_ID"
EOF

# Apply the Service Account
echo "Creating Kubernetes Service Account: $K8S_SERVICE_ACCOUNT_NAME..."
kubectl apply -f service-account.yaml

# Establish the Federated Identity Credential
echo "Establishing Federated Identity Credential..."
az identity federated-credential create \
    --name "my-app-federated-credential" \
    --identity-name $MANAGED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --issuer $AKS_OIDC_ISSUER_URL \
    --subject "system:serviceaccount:$KUBERNETES_NAMESPACE:$K8S_SERVICE_ACCOUNT_NAME"

This command links the Azure Managed Identity to the Kubernetes Service Account. When a pod uses this Service Account, AKS will exchange the OIDC token issued by the cluster for an Azure AD token using the federated credential, allowing the pod to assume the identity of the Managed Identity.

6. Deploy the Key Vault CSI Driver

The Key Vault CSI driver allows Key Vault secrets to be mounted as files into your pods.

# Add the secrets-store-csi-driver Helm repo
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

# Update Helm repos
helm repo update

# Install the Key Vault CSI driver
echo "Installing Key Vault CSI driver..."
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

# Install the Azure Key Vault Provider for Secrets Store CSI Driver
echo "Installing Azure Key Vault Provider..."
helm install csi-secrets-store-provider-azure secrets-store-csi-driver/secrets-store-csi-driver-provider-azure --namespace kube-system

7. Create a SecretProviderClass

The SecretProviderClass custom resource defines which secrets from Key Vault should be retrieved and how they should be mounted.

# Create SecretProviderClass YAML
cat < secret-provider-class.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-secret-provider
  namespace: $KUBERNETES_NAMESPACE
spec:
  provider: azure
  parameters:
    usePodIdentity: "false" # Important: Set to false for Workload Identity
    useVMManagedIdentity: "true" # Important: Set to true for Workload Identity
    userAssignedIdentityID: "$MANAGED_IDENTITY_CLIENT_ID" # Client ID of the Managed Identity
    keyvaultName: "$KEY_VAULT_NAME"
    objects: |
      array:
        - |
          objectName: MyWebAppSecret
          objectType: secret
          objectVersion: "" # Latest version
    tenantId: "$(az account show --query tenantId -o tsv)" # Your Azure Tenant ID
EOF

# Apply the SecretProviderClass
echo "Creating SecretProviderClass..."
kubectl apply -f secret-provider-class.yaml

8. Deploy a Sample Application

Finally, we'll deploy a simple Nginx application that uses the Service Account and mounts the secret from Key Vault via the CSI driver.

# Create Deployment YAML
cat < nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-workload-identity
  namespace: $KUBERNETES_NAMESPACE
  labels:
    app: nginx-workload-identity
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-workload-identity
  template:
    metadata:
      labels:
        app: nginx-workload-identity
      annotations:
        kubernetes.azure.com/sgx_feature_required: "false"
    spec:
      serviceAccountName: $K8S_SERVICE_ACCOUNT_NAME # Use the created Service Account
      containers:
        - name: nginx
          image: nginx:latest
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          command: ["/bin/bash", "-c", "cat /mnt/secrets-store/MyWebAppSecret && sleep 3600"] # Command to read the secret
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-keyvault-secret-provider"
EOF

# Apply the Deployment
echo "Deploying sample Nginx application..."
kubectl apply -f nginx-deployment.yaml

9. Verify the Setup

Check if the pod is running and if the secret is successfully mounted.

# Check pod status
echo "Waiting for pod to be ready..."
kubectl get pods --namespace $KUBERNETES_NAMESPACE -l app=nginx-workload-identity -w

# Once the pod is running, check logs to see if the secret was read
echo "Checking pod logs for secret content..."
POD_NAME=$(kubectl get pods -n $KUBERNETES_NAMESPACE -l app=nginx-workload-identity -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD_NAME -n $KUBERNETES_NAMESPACE

# You should see "SuperSecureValue123!" in the logs.
# You can also exec into the pod and verify the mounted file
echo "Executing into the pod to verify mounted secret..."
kubectl exec -it $POD_NAME -n $KUBERNETES_NAMESPACE -- ls /mnt/secrets-store
kubectl exec -it $POD_NAME -n $KUBERNETES_NAMESPACE -- cat /mnt/secrets-store/MyWebAppSecret

If you see "MyWebAppSecret" listed and its content printed, your setup is successful!

Security Considerations

While Workload Identity and Key Vault CSI driver significantly enhance security, it's crucial to be aware of the following considerations:

Least Privilege: Always grant the Managed Identity the absolute minimum permissions required on Key Vault. In our example, we granted `get` and `list` on secrets. Avoid broader permissions like `all`.
Key Vault Access Policies vs. RBAC: While we used access policies for simplicity, Azure Key Vault RBAC (Role-Based Access Control) offers a more granular and centralized way to manage permissions, aligning with Azure's recommended security posture. Consider migrating to RBAC for production environments.
Network Security: For enhanced security, configure Azure Key Vault with private endpoints. This ensures that traffic between your AKS cluster and Key Vault traverses the Azure backbone network privately, never exposing the Key Vault to the public internet.
Secret Rotation: Implement a strategy for secret rotation in Key Vault. While the CSI driver can automatically refresh mounted secrets when they change in Key Vault, your application might need to be designed to detect and react to these changes (e.g., by restarting or reloading configuration).
Auditing and Logging: Enable diagnostic settings for Azure Key Vault to send logs to Azure Monitor, Log Analytics, or a SIEM. This allows you to track who accessed which secrets, when, and from where.
Kubernetes RBAC: Ensure your Kubernetes RBAC policies are correctly configured to restrict who can create, modify, or delete `ServiceAccount` and `SecretProviderClass` resources, especially those linked to sensitive Managed Identities.
OIDC Issuer Security: The OIDC issuer URL is a public endpoint, but the security relies on the cryptographic signing of the OIDC tokens. Ensure your AKS cluster's OIDC issuer is always correctly configured and monitored.

Best Practices

To maximize the benefits and security of this integration, consider these best practices:

Dedicated Managed Identities: Create separate User-Assigned Managed Identities for different workloads or applications. Avoid sharing a single Managed Identity across multiple, unrelated applications, as this violates the principle of least privilege.
Dedicated Namespaces: Deploy your applications and their associated Kubernetes Service Accounts and SecretProviderClasses into dedicated Kubernetes namespaces. This provides better isolation and easier management of resources.
Automate Deployment: Use Infrastructure-as-Code (IaC) tools like Terraform or Bicep to define and deploy your AKS cluster, Key Vault, Managed Identities, and all Kubernetes resources. This ensures consistency, repeatability, and version control.
Monitor Workload Identity: Keep an eye on the health and performance of your Workload Identity setup. Azure Monitor logs can help identify authentication failures.
Application Design for Secret Updates: Design your applications to gracefully handle secret rotation. The Key Vault CSI driver can automatically update the mounted files, but your application might need a mechanism to detect these file changes and reload the secrets without requiring a pod restart.
Version Control for Kubernetes Manifests: Store all your Kubernetes YAML files (Service Accounts, SecretProviderClasses, Deployments) in a version control system like Git.
Secure Your CI/CD Pipelines: Ensure that your CI/CD pipelines, which deploy these configurations, are themselves secure and follow best practices for credential management.

FAQ

Q1: Why choose Workload Identity over the older AAD Pod Identity (Pod-Managed Identity)?

Azure Workload Identity is the recommended and future-proof solution for AKS. It offers several advantages over AAD Pod Identity:

Native Kubernetes Integration: Workload Identity leverages native Kubernetes OIDC tokens, making it more aligned with Kubernetes standards and practices.
Improved Security: It eliminates the need for mutating webhooks (like NMI in AAD Pod Identity) that intercept API calls, reducing potential attack surfaces and simplifying the trust boundary.
Enhanced Performance: By directly using OIDC tokens, it can offer better performance and reduced latency compared to the proxy-based approach of AAD Pod Identity.
Simpler Troubleshooting: Being more native, it often leads to simpler debugging and fewer unexpected interactions.

Q2: What happens if the federated credential expires or is revoked?

Federated credentials in Azure AD do not have an explicit expiry date in the same way service principals or user credentials do. Their validity is tied to the existence of the Managed Identity and the Kubernetes Service Account, and the OIDC issuer's ability to sign valid tokens. If the federated credential is deleted or revoked, the Managed Identity will no longer be able to exchange the Kubernetes Service Account's OIDC token for an Azure AD access token. This will lead to authentication failures for your pods trying to access Azure resources, including Key Vault, resulting in errors within the CSI driver and your application.

Q3: Can I use multiple Key Vaults with one CSI driver setup?

Yes, absolutely. You can configure multiple `SecretProviderClass` resources, each pointing to a different Azure Key Vault or even different sets of secrets within the same Key Vault. Each `SecretProviderClass` can specify its own `keyvaultName` and `objects` array. Your pods can then reference the appropriate `SecretProviderClass` in their `volumeAttributes` to mount secrets from various Key Vaults as needed, potentially using different Managed Identities for each.

Conclusion

The combination of Azure Kubernetes Service, Workload Identity, and the Key Vault CSI driver provides a robust, secure, and cloud-native solution for managing secrets in your containerized applications. By moving away from embedded credentials and leveraging Azure AD's identity capabilities, you significantly reduce your application's attack surface and simplify credential management. This architecture empowers developers to focus on building features while ensuring that sensitive data is handled with the highest security standards, making it an indispensable pattern for modern cloud deployments on Azure.

Azure AKS: Secure Secrets with Workload Identity & Key Vault CSI Driver