Azure AKS Workload Identity & Key Vault CSI Driver for Secure Secrets

Navigating the Cloud Security Labyrinth: Azure AKS with Workload Identity and Key Vault CSI Driver

In the dynamic landscape of cloud-native development, managing sensitive information like database connection strings, API keys, and certificates within Kubernetes clusters has always presented a significant security challenge. Traditional methods often involved storing secrets directly in Kubernetes Secrets objects, which, while encrypted at rest within etcd, still required careful handling of access control and lifecycle management. The risk of exposing these secrets through misconfigurations or unauthorized access loomed large, prompting the need for more robust, secure, and auditable solutions.

Enter Azure Kubernetes Service (AKS) combined with Azure AD Workload Identity and the Azure Key Vault Container Storage Interface (CSI) driver. This powerful triumvirate offers a modern, secure, and idiomatic way for Kubernetes workloads to access secrets, keys, and certificates stored in Azure Key Vault without ever exposing them directly to the pod's environment variables or file system in an unencrypted state. It eliminates the need for manual secret rotation, reduces the attack surface, and integrates seamlessly with Azure's identity and access management capabilities.

Azure AD Workload Identity extends the capabilities of Azure AD Pod Identity (which is now deprecated for new deployments) by leveraging Kubernetes service accounts as identities within Azure AD. This allows your pods to authenticate directly with Azure AD and obtain tokens for accessing Azure resources, adhering to the principle of least privilege. When combined with the Key Vault CSI driver, this means a Kubernetes pod, through its associated service account, can authenticate with Azure AD using Workload Identity, obtain a token, and then use that token to access specific secrets in Azure Key Vault. The CSI driver then mounts these secrets as files into the pod's filesystem, making them accessible to the application as if they were local files, without the application needing to know anything about Azure AD authentication or Key Vault APIs.

This article will guide you through the process of setting up AKS with Workload Identity and integrating it with the Key Vault CSI driver to securely access secrets. We will cover the prerequisites, a detailed step-by-step implementation, crucial security considerations, and best practices to ensure your cloud-native applications handle sensitive data with the highest level of integrity and confidentiality.

Prerequisites

Before we embark on our journey, ensure you have the following tools and Azure resources configured:

• Azure Subscription: An active Azure subscription.
• Azure CLI: Version 2.40.0 or later installed and configured. You can verify your version with az --version.
• kubectl: Kubernetes command-line tool installed and configured to interact with your AKS cluster.
• Helm: Version 3 or later installed. Helm is used to deploy the Key Vault CSI driver and its provider.
• Permissions: An Azure account with permissions to create resource groups, AKS clusters, managed identities, and Key Vaults, as well as assign roles.

Let's begin by logging into Azure CLI:

az login

Step-by-Step Implementation

1. Create an Azure Resource Group

First, we'll create a dedicated resource group to house all our resources.

RESOURCE_GROUP="aks-wi-kv-rg"
LOCATION="eastus"

az group create --name $RESOURCE_GROUP --location $LOCATION

2. Create an AKS Cluster with Workload Identity Enabled

Next, we provision an AKS cluster and explicitly enable Workload Identity. It's crucial to specify --enable-workload-identity during cluster creation.

AKS_CLUSTER_NAME="techventure-aks-cluster"
NODE_COUNT=3

az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME \
    --node-count $NODE_COUNT \
    --enable-managed-identity \
    --enable-workload-identity \
    --generate-ssh-keys \
    --node-vm-size Standard_DS2_v2

# Get AKS credentials
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME

Verify the cluster is running and kubectl is configured:

kubectl get nodes

3. Create an Azure AD User-Assigned Managed Identity

This managed identity will be used by your Kubernetes workloads to authenticate with Azure Key Vault. It acts as the identity for your application.

MANAGED_IDENTITY_NAME="techventure-app-mi"

az identity create \
    --resource-group $RESOURCE_GROUP \
    --name $MANAGED_IDENTITY_NAME

# Store the principal ID and client ID of the managed identity
MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $MANAGED_IDENTITY_NAME --query principalId -o tsv)
MANAGED_IDENTITY_CLIENT_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $MANAGED_IDENTITY_NAME --query clientId -o tsv)

echo "Managed Identity Principal ID: $MANAGED_IDENTITY_PRINCIPAL_ID"
echo "Managed Identity Client ID: $MANAGED_IDENTITY_CLIENT_ID"

4. Create an Azure Key Vault and Store a Secret

We'll create a Key Vault and populate it with a sample secret that our application will later consume.

KEY_VAULT_NAME="techventuresujaykv$(openssl rand -hex 4)" # Ensure unique name

az keyvault create \
    --name $KEY_VAULT_NAME \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --sku Standard \
    --enable-rbac-authorization false # Disable RBAC for simplicity in this example, use AAD Auth for policy.

# Add a sample secret
SECRET_NAME="my-db-password"
SECRET_VALUE="SuperSecureDBPassword123!"

az keyvault secret set \
    --vault-name $KEY_VAULT_NAME \
    --name $SECRET_NAME \
    --value $SECRET_VALUE

5. Grant Managed Identity Access to Key Vault

Now, we grant the user-assigned managed identity permissions to retrieve secrets from the Key Vault. We'll use an access policy here, which is simpler for demonstration. For production, consider using Azure RBAC for Key Vault.

az keyvault set-policy \
    --name $KEY_VAULT_NAME \
    --object-id $MANAGED_IDENTITY_PRINCIPAL_ID \
    --secret-permissions get list

This command grants the managed identity get and list permissions on secrets within the specified Key Vault.

6. Install the Key Vault CSI Driver and Azure Key Vault Provider

The CSI driver allows Kubernetes to mount volumes that represent secrets from external secret stores. The Azure Key Vault provider is specifically for Azure Key Vault.

# Add Helm repositories
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts

# Update Helm repositories
helm repo update

# Install the Secrets Store CSI driver
helm install csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set syncSecret.enabled=true

# Install the Azure Key Vault Provider for the CSI driver
helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system

Verify the pods are running in the kube-system namespace:

kubectl get pods -n kube-system -l app=secrets-store-csi-driver
kubectl get pods -n kube-system -l app=csi-secrets-store-provider-azure

7. Associate Managed Identity with Kubernetes Service Account (Workload Identity)

This is the core of Workload Identity. We will create a Kubernetes service account and link it to our Azure AD user-assigned managed identity using a federated credential. This allows the service account to assume the identity of the managed identity in Azure AD.

Get the OIDC Issuer URL for your AKS cluster:

AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo "AKS OIDC Issuer URL: $AKS_OIDC_ISSUER"

Create a Kubernetes Service Account:

We'll define a service account in a new namespace.

NAMESPACE="techventure-app"
SERVICE_ACCOUNT_NAME="techventure-sa"

kubectl create namespace $NAMESPACE

cat <

FEDERATED_CREDENTIAL_NAME="techventure-fed-cred"

az identity federated-credential create \
    --name $FEDERATED_CREDENTIAL_NAME \
    --identity-name $MANAGED_IDENTITY_NAME \
    --resource-group $RESOURCE_GROUP \
    --issuer "$AKS_OIDC_ISSUER" \
    --subject "system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT_NAME"

cat <

cat <

# Wait for the pod to be running
kubectl get pods -n $NAMESPACE -w

# Get the pod name
POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=nginx -o jsonpath='{.items[0].metadata.name}')

echo "Pod name: $POD_NAME"

# List files in the mounted directory
kubectl exec -it $POD_NAME -n $NAMESPACE -- ls /mnt/secrets-store/

# Display the content of the secret file
kubectl exec -it $POD_NAME -n $NAMESPACE -- cat /mnt/secrets-store/$SECRET_NAME

kubectl get secret my-app-secret-k8s -n $NAMESPACE -o yaml