Azure AKS: Secure Secrets with Workload Identity & Key Vault CSI Driver [q7px]

Unlocking Secure Secret Management: Azure AKS with Workload Identity and Key Vault CSI Driver

In the dynamic landscape of cloud-native applications, managing secrets securely is not just a best practice—it's a critical imperative. As organizations increasingly adopt Kubernetes on Azure Kubernetes Service (AKS), the challenge of providing applications with secure access to sensitive information like database credentials, API keys, and certificates becomes paramount. Traditional methods, such such as embedding secrets directly into images or relying on less secure environment variables, are fraught with risk.

This article delves into the modern, secure, and recommended approach for secret management in AKS: leveraging Azure Workload Identity in conjunction with the Azure Key Vault Container Storage Interface (CSI) driver. This powerful combination eliminates the need for manual secret rotation, reduces the attack surface, and ensures that your Kubernetes workloads can access secrets stored in Azure Key Vault with minimal operational overhead and maximum security.

Overview of Workload Identity and Key Vault CSI Driver

At its core, secure secret management in Kubernetes revolves around allowing pods to authenticate to an external secret store without hardcoding credentials. Azure Workload Identity and the Key Vault CSI driver address this challenge elegantly:

• Azure Workload Identity: This feature enables Kubernetes applications to access Azure cloud resources securely using an Azure Active Directory (AAD) managed identity. It achieves this by federating a Kubernetes Service Account with an Azure AD user-assigned managed identity. When a pod is configured to use this Service Account, it can then automatically obtain an Azure AD token, which can be used to authenticate to Azure resources. This mechanism replaces the older AAD Pod Identity, offering a more native and streamlined experience by leveraging Kubernetes' OpenID Connect (OIDC) issuer capabilities.
• Azure Key Vault CSI Driver: The CSI (Container Storage Interface) driver for Azure Key Vault allows Kubernetes to mount Azure Key Vault secrets, keys, and certificates into pods as a volume. Instead of the application directly calling Key Vault APIs, the secrets are presented as files within the pod's filesystem. This means applications can simply read these files as they would any other configuration file, simplifying application code and ensuring secrets are only available to the pod for as long as it exists. The CSI driver handles the authentication to Key Vault, using the underlying Workload Identity configured for the pod.

The synergy between Workload Identity and the Key Vault CSI driver provides a robust solution: Workload Identity authenticates the pod to Azure, granting it permission to access Key Vault, and the CSI driver then fetches and mounts the secrets into the pod's filesystem. This architecture ensures that secrets never leave Azure Key Vault unless explicitly requested by an authorized workload, and they are never stored persistently within Kubernetes itself.

Prerequisites

Before we embark on the implementation, ensure you have the following prerequisites in place:

• Azure Subscription: An active Azure subscription.
• Azure CLI: Installed and logged in (`az login`). Ensure you have the latest version.
• `kubectl`: Installed and configured to connect to your AKS cluster.
• `helm`: Installed (version 3+ recommended) for deploying the CSI driver.
• AKS Cluster: An existing AKS cluster. For Workload Identity, the cluster must have the OIDC Issuer and Workload Identity features enabled. If you have an older cluster, you might need to enable these features.
• Azure Resource Group: A resource group to host your AKS cluster and related resources.
• Permissions: Your Azure user account needs permissions to create and manage AKS clusters, Key Vaults, Managed Identities, and assign roles. Typically, "Contributor" role on the resource group and "Key Vault Administrator" on the Key Vault will suffice.

Step-by-Step Implementation

Let's walk through the process of setting up Azure AKS with Workload Identity and the Key Vault CSI driver.

1. Enable Workload Identity on Your AKS Cluster

If your AKS cluster was created recently, Workload Identity might already be enabled. You can check its status:

az aks show -g tech-news-venture-rg -n tnv-aks-cluster --query "oidcIssuerProfile.enabled"
az aks show -g tech-news-venture-rg -n tnv-aks-cluster --query "workloadIdentityProfile.enabled"

If these return `false`, or if you are creating a new cluster, enable them:

# For an existing cluster
az aks update \
  --name tnv-aks-cluster \
  --resource-group tech-news-venture-rg \
  --enable-oidc-issuer \
  --enable-workload-identity

# Or, when creating a new cluster
az aks create \
  --resource-group tech-news-venture-rg \
  --name tnv-aks-cluster \
  --node-count 2 \
  --generate-ssh-keys \
  --enable-oidc-issuer \
  --enable-workload-identity

Once enabled, retrieve the OIDC Issuer URL for your cluster. This URL is crucial for federating the managed identity with the Kubernetes Service Account.

export AKS_OIDC_ISSUER=$(az aks show -g tech-news-venture-rg -n tnv-aks-cluster --query "oidcIssuerProfile.issuerUrl" -o tsv)
echo $AKS_OIDC_ISSUER

The output will be something like `https://westus2.oic.prod-aks.azure.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/`. Store this value.

2. Create an Azure Key Vault and Store a Secret

Next, create an Azure Key Vault and add a sample secret that your application will access.

export KEYVAULT_NAME="tnv-kv-secrets-007"
export RESOURCE_GROUP="tech-news-venture-rg"
export LOCATION="westus2"

az keyvault create \
  --name $KEYVAULT_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --sku standard

az keyvault secret set \
  --vault-name $KEYVAULT_NAME \
  --name "MyApplicationSecret" \
  --value "SuperSecretValue123!"

3. Create an Azure AD User-Assigned Managed Identity

This managed identity will be assigned to your Kubernetes pods via Workload Identity.

export MANAGED_IDENTITY_NAME="tnv-mi-webapp"

az identity create \
  --name $MANAGED_IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION

export MANAGED_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query "clientId" -o tsv)
export MANAGED_IDENTITY_ID=$(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query "id" -o tsv)
echo "Managed Identity Client ID: $MANAGED_IDENTITY_CLIENT_ID"
echo "Managed Identity Resource ID: $MANAGED_IDENTITY_ID"

4. Grant Managed Identity Access to Key Vault

The managed identity needs permissions to retrieve secrets from the Key Vault.

az keyvault set-policy \
  --name $KEYVAULT_NAME \
  --secret-permissions get list \
  --spn $MANAGED_IDENTITY_CLIENT_ID

This command grants the managed identity `get` and `list` permissions on secrets within the specified Key Vault. For production, apply the principle of least privilege, granting only the necessary permissions (e.g., just `get` for specific secrets).

5. Create a Kubernetes Service Account and Establish Federation

Create a Kubernetes Service Account that your application pod will use. Then, establish a federated credential between this Service Account and the Azure AD managed identity.

# Create a Kubernetes Service Account
kubectl create serviceaccount webapp-sa --namespace default

# Get the Service Account's object ID (not strictly needed for federation, but good to know)
# export SERVICE_ACCOUNT_OBJECT_ID=$(kubectl get sa webapp-sa -o jsonpath='{.metadata.uid}')

# Establish federated credential
az identity federated-credential create \
  --name webapp-federated-credential \
  --identity-name $MANAGED_IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP \
  --issuer $AKS_OIDC_ISSUER \
  --subject system:serviceaccount:default:webapp-sa

The `subject` field specifies the Kubernetes Service Account in the format `system:serviceaccount::`. This federation allows the Kubernetes Service Account to exchange its OIDC token for an Azure AD token representing the managed identity.

6. Install the Azure Key Vault CSI Driver

Deploy the Key Vault CSI driver to your AKS cluster using Helm. We'll enable `syncSecrets` to demonstrate both direct file mounting and syncing to a native Kubernetes secret.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system \
  --set syncSecrets.enabled=true \
  --set enableSecretRotation=true \
  --set rotationPollInterval=60s

Verify the driver pods are running:

kubectl get pods -n kube-system -l app=secrets-store-csi-driver

7. Deploy a Sample Application

Now, let's deploy a sample application that uses the Key Vault CSI driver and Workload Identity to access the secret. This involves two main Kubernetes resources: `SecretProviderClass` and a `Deployment` (or `Pod`).

a. Create a SecretProviderClass

The `SecretProviderClass` defines which secrets from Key Vault should be mounted and how. Save the following YAML as `secretproviderclass.yaml`:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: tnv-keyvault-secrets
  namespace: default
spec:
  provider: azure
  parameters:
    usePodIdentity: "false" # Important: Set to "false" for Workload Identity
    useVMManagedIdentity: "true" # Use a user-assigned managed identity
    userAssignedIdentityID: "$MANAGED_IDENTITY_CLIENT_ID" # Client ID of the Managed Identity
    keyvaultName: "$KEYVAULT_NAME" # Your Key Vault name
    objects: |
      array:
        - |
          objectName: MyApplicationSecret
          objectType: secret
          objectVersion: "" # Latest version
    tenantId: "$TENANT_ID" # Your Azure Tenant ID (can get with `az account show --query tenantId -o tsv`)
  secretObjects: # Optional: To sync secrets to native K8s secrets
  - secretName: my-app-k8s-secret
    type: Opaque
    data:
    - key: my-secret-data
      objectName: MyApplicationSecret

Before applying, replace `$MANAGED_IDENTITY_CLIENT_ID`, `$KEYVAULT_NAME`, and `$TENANT_ID` with your actual values. You can get your Tenant ID using `az account show --query tenantId -o tsv`.

# Example of setting environment variables for replacement
export AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
envsubst < secretproviderclass.yaml | kubectl apply -f -

Note: You might need to install `gettext` for `envsubst` if not already present (`sudo apt-get install gettext` on Debian/Ubuntu, `brew install gettext` on macOS, or similar).

b. Deploy the Application Pod

Create a `deployment.yaml` for a simple Nginx pod that mounts the secret. This pod will use the `webapp-sa` Service Account, which is federated with our managed identity.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tnv-webapp-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tnv-webapp
  template:
    metadata:
      labels:
        app: tnv-webapp
    spec:
      serviceAccountName: webapp-sa # Reference the federated Service Account
      containers:
        - name: tnv-webapp
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          command: ["/bin/sh", "-c"]
          args:
            - while true; do
                echo "My Secret: $(cat /mnt/secrets-store/MyApplicationSecret)";
                echo "Native K8s Secret: $(cat /etc/nginx/conf.d/my-native-secret.conf)";
                sleep 5;
              done
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "tnv-keyvault-secrets"

Apply the deployment:

kubectl apply -f deployment.yaml

c. Verify Secret Access

After the pod starts, check its logs to see if it successfully read the secret from the mounted volume.

kubectl logs -f $(kubectl get pod -l app=tnv-webapp -o jsonpath='{.items[0].metadata.name}')

You should see output similar to:

My Secret: SuperSecretValue123!
Native K8s Secret: SuperSecretValue123!

Additionally, since we enabled `syncSecrets`, verify that a native Kubernetes secret has been created:

kubectl get secret my-app-k8s-secret -o jsonpath='{.data.my-secret-data}' | base64 --decode

This should also output `SuperSecretValue123!`. This demonstrates that the CSI driver successfully fetched the secret from Key Vault using the Workload Identity and mounted it into the pod's filesystem, and also synced it to a native Kubernetes secret.

Security Considerations

Implementing this solution significantly enhances your security posture, but a few considerations remain crucial:

• Principle of Least Privilege: Always grant the managed identity the minimum necessary permissions to Key Vault (e.g., `get` for specific secrets, not `all` secrets).
• Network Security for Key Vault: Restrict access to Key Vault using Azure Private Endpoints or Virtual Network service endpoints. This ensures that secret access requests never traverse the public internet.
• AKS Cluster Hardening: Implement Azure Policy for AKS, enable Defender for Cloud, use Network Policies to control pod-to-pod communication, and regularly update your AKS cluster to the latest versions.
• Auditing and Logging: Enable diagnostic settings for Azure Key Vault to send logs to Azure Monitor (Log Analytics Workspace) for auditing access to secrets. Monitor the logs of the Key Vault CSI driver pods for any errors or unauthorized access attempts.
• Secret Rotation: While Key Vault CSI driver can automatically rotate secrets when `enableSecretRotation` is set, applications consuming secrets from mounted files must be designed to re-read these files periodically or react to file changes. For secrets synced to native Kubernetes secrets, applications might need to restart or reload configurations to pick up changes.
• Kubernetes RBAC: Ensure that only authorized users and service accounts can create or modify `SecretProviderClass` resources or deployments that reference them.

Best Practices

To maximize the benefits and security of this setup, consider the following best practices:

• Dedicated Managed Identities: Create separate user-assigned managed identities for each application or microservice. This isolates permissions and limits the blast radius if an identity is compromised.
• Version Your Secrets: Utilize Key Vault's secret versioning. While the CSI driver can fetch the latest version, specifying a version in `SecretProviderClass` can be useful for controlled rollouts or debugging.
• Automate with GitOps: Manage your Kubernetes manifests (Service Accounts, SecretProviderClasses, Deployments) using GitOps principles. This ensures version control, auditability, and automated deployments.
• Monitoring and Alerting: Set up alerts for failed Key Vault access attempts, CSI driver errors, or unexpected changes in secret values.
• Regular Updates: Keep your AKS cluster, the Key Vault CSI driver, and any underlying node images updated to benefit from the latest security patches and features.
• Avoid Direct Key Vault Access from Pods: Wherever possible, prefer the CSI driver for secret consumption. If an application absolutely needs to perform operations like listing or creating secrets, ensure it uses a separate managed identity with highly restricted permissions.
• Consider `SecretProviderClass` for Native Kubernetes Secrets: While direct file mounting is often preferred for simplicity, if your application expects secrets as native Kubernetes secret objects, the `secretObjects` field in `SecretProviderClass` is invaluable.

FAQ

Q1: Can I use Azure Workload Identity with AAD Pod Identity?

No, Workload Identity is the successor to AAD Pod Identity. They are mutually exclusive. Workload Identity offers a more native Kubernetes experience by leveraging the OIDC capabilities of the cluster, reducing the need for mutating webhooks and simplifying the overall architecture. It is the recommended approach for new deployments and migrations.

Q2: How does the Key Vault CSI driver compare to external-secrets.io?

Both are excellent solutions but serve slightly different paradigms. The Key Vault CSI driver directly mounts secrets from Key Vault into the pod's filesystem as a volume. This means secrets are never stored persistently in Kubernetes and are refreshed directly from Key Vault. `external-secrets.io`, on the other hand, acts as a controller that syncs secrets from an external store (like Key Vault) into native Kubernetes `Secret` objects. Choose the CSI driver if you prefer secrets to be ephemeral, filesystem-mounted, and directly managed by the driver. Choose `external-secrets.io` if your applications strictly expect native Kubernetes `Secret` objects and you're comfortable with them being stored (even if synced) within the Kubernetes API server.

Q3: What if my application needs to perform write operations (e.g., create, update, delete secrets) in Key Vault?

The Azure Key Vault CSI driver is primarily designed for read-only access to secrets, keys, and certificates, mounting them into the pod's filesystem. If your application requires write operations to Key Vault, it should use the Azure SDK for its respective language, authenticated via the same Workload Identity. The Workload Identity provides the necessary Azure AD token, and the SDK handles the API calls to Key Vault. Ensure the managed identity has the appropriate `set`, `delete`, or other write permissions on the Key Vault.

Conclusion

The combination of Azure Workload Identity and the Azure Key Vault CSI driver represents a significant leap forward in secure and efficient secret management for AKS. By federating Kubernetes Service Accounts with Azure AD managed identities, we eliminate the need for hardcoded credentials, providing a robust authentication mechanism. The Key Vault CSI driver then seamlessly integrates these identities to mount secrets directly into application pods, streamlining development and bolstering security.

This architecture not only adheres to the principle of least privilege but also simplifies compliance, reduces operational overhead, and ensures that your sensitive application data remains protected within Azure's trusted boundaries. As organizations continue their journey into cloud-native development, adopting patterns like this will be crucial for building resilient, scalable, and inherently secure applications on Azure Kubernetes Service.