Azure AKS: Secure Secrets with Workload Identity & Key Vault CSI Driver

Key elements in this deployment manifest:

• `serviceAccountName: tech-news-sa`: Links the pod to the Kubernetes Service Account configured with Workload Identity.
• `azure.workload.identity/use: "true"`: This annotation instructs the Workload Identity webhook to inject the necessary environment variables and projected service account token into the pod for Workload Identity to function.
• `volumeMounts` and `volumes`: These define how the secrets-store CSI driver mounts the secrets into the `/mnt/secrets-store` path within the Nginx container, referencing the `azure-keyvault-tech-news` SecretProviderClass.

14. Verify Secret Mounting

After the pod is running, executive into it and verify the secrets are mounted as files.

kubectl get pods -l app=nginx-secrets

# Wait for the pod to be running, then exec into it
POD_NAME=$(kubectl get pods -l app=nginx-secrets -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- ls /mnt/secrets-store/

You should see `demosecret1` and `dbPassword` listed as files. You can then view their content:

kubectl exec -it $POD_NAME -- cat /mnt/secrets-store/demosecret1
kubectl exec -it $POD_NAME -- cat /mnt/secrets-store/dbPassword

The output should be the secret values you set in Azure Key Vault. This confirms that your application successfully retrieved secrets from Key Vault via the CSI driver, authenticated using Azure Workload Identity.

Security Considerations

• Least Privilege: Always adhere to the principle of least privilege. Grant the Managed Identity only the necessary permissions (e.g., `get`, `list` on secrets) to the specific Key Vault. Avoid granting broader permissions like `all`.
• Key Vault RBAC vs. Access Policies: While access policies were used in this demo for simplicity, Azure Key Vault RBAC offers more granular control and is generally recommended for production environments. It allows assigning specific data plane roles to Azure AD identities.
• Network Security: Implement network restrictions on your Key Vault using private endpoints or virtual network service endpoints. This ensures that only authorized networks (like your AKS VNet) can access the Key Vault, adding another layer of security.
• Kubernetes RBAC: Ensure that only authorized users or CI/CD pipelines can create or modify `ServiceAccount` and `SecretProviderClass` resources, as these control access to your secrets.
• Auditing and Logging: Enable comprehensive auditing and logging for Azure Key Vault and AKS. Monitor access attempts, secret retrievals, and any configuration changes to detect suspicious activity.
• Secret Rotation: While the CSI driver fetches secrets on pod startup and periodically refreshes them, implementing automated secret rotation within Key Vault itself is crucial for long-term security. The CSI driver will automatically pick up rotated secrets during its refresh interval.
• Workload Identity Webhook: Be aware that the Workload Identity webhook modifies your pod's manifest. Ensure its integrity and monitor its behavior.

Best Practices

• One Managed Identity per Workload/Service Account: Create a unique user-assigned managed identity for each application or microservice. This ensures strict isolation and allows for granular permission management. If one workload is compromised, it doesn't automatically grant access to secrets meant for others.
• Environment-Specific Key Vaults: Use separate Key Vaults for different environments (e.g., Dev, Staging, Production). This prevents accidental exposure of production secrets and allows for different access policies per environment.
• CI/CD Integration: Integrate the creation and configuration of Managed Identities, Federated Credentials, and Key Vault permissions into your CI/CD pipelines. Automate the deployment of `SecretProviderClass` and application manifests.
• Monitor Refresh Intervals: Understand the refresh interval of the Key Vault CSI driver (default is 2 minutes). If a secret is rotated in Key Vault, it might take a few minutes for the updated value to appear in the mounted volume within the pod. For highly sensitive, rapidly changing secrets, consider direct API calls via Azure SDKs (also using Workload Identity) if real-time updates are critical, though this adds complexity.
• Avoid Hardcoding IDs: Use environment variables or Azure CLI queries to dynamically retrieve resource IDs (e.g., Key Vault name, Managed Identity Client ID, Tenant ID) in your deployment scripts, rather than hardcoding them into YAML files. This improves portability and reduces errors.
• Regular Security Audits: Periodically review your Key Vault access policies/RBAC assignments and Workload Identity configurations to ensure they align with your security posture and compliance requirements.

FAQ

1. What is the difference between Workload Identity and AAD Pod Identity (legacy)?

AAD Pod Identity (legacy) used Node Managed Identity and a component called MIC (Managed Identity Controller) to intercept token requests and assign identities to pods. While it worked, it had several limitations, including performance overhead, scalability issues (especially with MIC), and a security concern where a compromised node could potentially impersonate other identities.

Azure Workload Identity is a more secure and performant solution. It leverages OIDC and projected service account tokens to allow pods to authenticate directly with Azure AD. There's no intermediary MIC component, making it more robust and scalable. It aligns with the upstream Kubernetes standard for federated identity, offering a more native and future-proof approach.

2. Can I use Workload Identity with multiple Key Vaults?

Yes, absolutely. You can create multiple `SecretProviderClass` resources, each configured to retrieve secrets from a different Azure Key Vault. Each `SecretProviderClass` would specify the `keyvaultName` and the `clientID` of the managed identity that has access to that specific Key Vault. Your application pods can then mount multiple volumes, each corresponding to a different `SecretProviderClass`, effectively accessing secrets from various Key Vaults simultaneously.

3. How does secret rotation work with Key Vault CSI driver?

When a secret in Azure Key Vault is rotated (i.e., a new version is created), the Key Vault CSI driver will automatically detect and update the mounted secret files within the pod. The driver periodically polls Key Vault for updates to the secrets specified in the `SecretProviderClass`. By default, this refresh interval is 2 minutes. Once a new version is detected, the CSI driver updates the corresponding file in the mounted volume. Your application, if designed to re-read the secret file periodically or on change, will then pick up the new value.

Conclusion