Mastering AWS EKS: Karpenter Autoscaling & IRSA Pod Identity

Navigating Cloud-Native Agility: AWS EKS with Karpenter Autoscaler and IRSA Pod Identity

The journey into cloud-native architectures often leads organizations to embrace Kubernetes, and AWS Elastic Kubernetes Service (EKS) stands as a robust foundation for running containerized workloads at scale. However, managing the underlying infrastructure – specifically, ensuring optimal node provisioning and secure access to AWS resources – presents unique challenges. This article delves into a powerful triumvirate: AWS EKS, the next-generation Karpenter autoscaler, and IAM Roles for Service Accounts (IRSA), demonstrating how their synergy delivers unparalleled efficiency, cost optimization, and security for your Kubernetes deployments.

Overview: EKS, Karpenter, and IRSA – A Synergistic Approach

AWS EKS simplifies the deployment, management, and scaling of Kubernetes applications in the AWS cloud. It handles the Kubernetes control plane's availability and scalability, allowing developers to focus on their applications.

Historically, scaling worker nodes in EKS relied heavily on the Kubernetes Cluster Autoscaler, which is a fantastic tool but operates by observing pending pods and interacting with AWS Auto Scaling Groups (ASGs). While effective, this approach can sometimes be slow to react, limited by ASG configurations, and less efficient in selecting the most cost-effective instance types. Enter Karpenter – an open-source, high-performance Kubernetes node provisioner built by AWS. Karpenter directly monitors the Kubernetes API for unschedulable pods and makes intelligent decisions to launch just-in-time, right-sized EC2 instances. It bypasses ASGs, allowing for faster scaling, greater instance type flexibility (including Spot instances), and significant cost savings by optimizing for the exact workload requirements.

Complementing this dynamic scaling, IAM Roles for Service Accounts (IRSA) addresses a critical security and operational challenge: how do applications running inside Kubernetes pods securely interact with AWS services? Traditionally, this involved distributing AWS credentials as environment variables or Kubernetes secrets, which is inherently less secure and harder to manage. IRSA allows you to associate an AWS IAM role with a Kubernetes Service Account. Pods configured to use that Service Account automatically inherit the permissions of the associated IAM role, enabling fine-grained, secure access to AWS APIs (e.g., S3, DynamoDB, SQS) without ever exposing long-lived AWS credentials to the pods.

Together, EKS provides the managed Kubernetes control plane, Karpenter ensures efficient and cost-effective compute capacity, and IRSA secures pod-level access to AWS resources. This integrated solution forms the backbone of a modern, secure, and highly optimized cloud-native platform.

Prerequisites

Before we embark on the implementation, ensure you have the following tools and configurations in place:

• AWS Account: With administrative privileges to create EKS clusters, IAM roles, and EC2 instances.
• AWS CLI: Installed and configured with appropriate credentials. Version 2 is recommended.

aws --version

• kubectl: The Kubernetes command-line tool, installed and configured to interact with your EKS cluster.

kubectl version --client

• eksctl: The official CLI for Amazon EKS, simplifying cluster creation and management, including OIDC provider and IRSA setup.

eksctl version

• helm: The package manager for Kubernetes, used to install Karpenter.

helm version

• jq: A lightweight and flexible command-line JSON processor, useful for parsing AWS CLI output.

jq --version

• Basic understanding: Familiarity with Kubernetes concepts (Pods, Deployments, Service Accounts), AWS IAM, VPC, and EC2 will be beneficial.

Step-by-step Implementation

Step 1: Create an EKS Cluster with eksctl

First, we'll create an EKS cluster. We'll specify a VPC, subnets, and an EKS version. Note that Karpenter requires specific tagging on subnets to discover them. We'll ensure our `eksctl` configuration includes these. For this guide, we'll use `us-east-1` as our region.

Create a file named `eks-cluster.yaml`:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: technews-karpenter-eks
  region: us-east-1
  version: "1.28"

vpc:
  id: "vpc-0abc123def4567890" # Replace with an existing VPC ID or omit to create a new one
  cidr: "10.0.0.0/16"
  subnets:
    private:
      us-east-1a:
        id: "subnet-0a1b2c3d4e5f6g7h8" # Replace with your private subnet ID
        cidr: "10.0.1.0/24"
        tags:
          kubernetes.io/cluster/technews-karpenter-eks: owned
          karpenter.sh/discovery: technews-karpenter-eks
      us-east-1b:
        id: "subnet-0h9g8f7e6d5c4b3a2" # Replace with your private subnet ID
        cidr: "10.0.2.0/24"
        tags:
          kubernetes.io/cluster/technews-karpenter-eks: owned
          karpenter.sh/discovery: technews-karpenter-eks
      us-east-1c:
        id: "subnet-0x1y2z3a4b5c6d7e8" # Replace with your private subnet ID
        cidr: "10.0.3.0/24"
        tags:
          kubernetes.io/cluster/technews-karpenter-eks: owned
          karpenter.sh/discovery: technews-karpenter-eks
    public:
      us-east-1a:
        id: "subnet-0p1q2r3s4t5u6v7w8" # Replace with your public subnet ID
        cidr: "10.0.101.0/24"
      us-east-1b:
        id: "subnet-0l1m2n3o4p5q6r7s8" # Replace with your public subnet ID
        cidr: "10.0.102.0/24"
      us-east-1c:
        id: "subnet-0a1s2d3f4g5h6j7k8" # Replace with your public subnet ID
        cidr: "10.0.103.0/24"

managedNodeGroups:
  - name: initial-ng
    instanceType: t3.medium
    minSize: 1
    maxSize: 2
    desiredCapacity: 1
    labels: { karpenter.sh/discovery: technews-karpenter-eks }
    tags:
      karpenter.sh/discovery: technews-karpenter-eks

Execute the `eksctl create cluster` command:

eksctl create cluster -f eks-cluster.yaml

This command will take 15-20 minutes to complete. It sets up the EKS control plane, an OIDC provider (crucial for IRSA), and an initial managed node group. The `karpenter.sh/discovery` tag on the subnets and node group allows Karpenter to find the cluster's network resources.

Once the cluster is ready, update your `kubectl` context:

aws eks update-kubeconfig --name technews-karpenter-eks --region us-east-1

Verify your cluster nodes:

kubectl get nodes

Step 2: Configure AWS IAM for Karpenter

Karpenter needs specific IAM permissions to launch and terminate EC2 instances, manage launch templates, and interact with other AWS services. We'll create an IAM role for Karpenter and associate it with a Kubernetes Service Account using IRSA.

First, get your cluster's OIDC provider URL:

OIDC_ID=$(aws eks describe-cluster --name technews-karpenter-eks --region us-east-1 --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
echo "OIDC_ID: $OIDC_ID"

Create an IAM policy for Karpenter. Save this JSON as `karpenter-policy.json`:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateLaunchTemplate",
                "ec2:CreateFleet",
                "ec2:RunInstances",
                "ec2:CreateTags",
                "ec2:TerminateInstances",
                "ec2:DeleteLaunchTemplate",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeImages",
                "ec2:DescribeSpotPriceHistory",
                "ssm:GetParameter"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/KarpenterNodeRole-technews-karpenter-eks",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "ec2.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": "arn:aws:eks:us-east-1:*:cluster/technews-karpenter-eks"
        }
    ]
}

Note: Replace `*` with your AWS Account ID in `arn:aws:iam::*:role/KarpenterNodeRole-technews-karpenter-eks` for production environments.

Create the IAM policy:

aws iam create-policy \
    --policy-name KarpenterControllerPolicy-technews-karpenter-eks \
    --policy-document file://karpenter-policy.json

Retrieve the ARN of the created policy:

KARPENTER_POLICY_ARN=$(aws iam list-policies --scope Local --query "Policies[?PolicyName=='KarpenterControllerPolicy-technews-karpenter-eks'].Arn" --output text)
echo "Karpenter Policy ARN: $KARPENTER_POLICY_ARN"

Now, create the Karpenter controller IAM role and associate it with a Kubernetes Service Account using `eksctl`. This command automatically sets up the trust policy for IRSA:

eksctl create iamserviceaccount \
  --cluster technews-karpenter-eks \
  --name karpenter \
  --namespace karpenter \
  --attach-policy-arn ${KARPENTER_POLICY_ARN} \
  --approve \
  --override-existing-serviceaccounts

Additionally, Karpenter needs an EC2 instance profile to launch nodes. This role will be assumed by the EC2 instances provisioned by Karpenter. We'll use `eksctl` to create a default one for us, or you can create one manually with permissions such as `AmazonSSMManagedInstanceCore` and potentially `AmazonEKSWorkerNodePolicy`, `AmazonEC2ContainerRegistryReadOnly`.

Note: If you already have a node instance role from your initial `eksctl` cluster creation, you can reuse it or create a new one. For simplicity, we'll assume a role named `KarpenterNodeRole-technews-karpenter-eks` will be used.

aws iam create-role --role-name KarpenterNodeRole-technews-karpenter-eks --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "ec2.amazonaws.com"},
        "Action": "sts:AssumeRole"
    }]
}'

aws iam attach-role-policy --role-name KarpenterNodeRole-technews-karpenter-eks --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name KarpenterNodeRole-technews-karpenter-eks --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam attach-role-policy --role-name KarpenterNodeRole-technews-karpenter-eks --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# Create the instance profile (needed for EC2 instances to assume the role)
aws iam create-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-technews-karpenter-eks
aws iam add-role-to-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-technews-karpenter-eks --role-name KarpenterNodeRole-technews-karpenter-eks

Step 3: Install Karpenter

Now we install Karpenter using Helm. We'll need the cluster endpoint and the name of the instance profile we just created.

Get cluster endpoint and AWS account ID:

CLUSTER_ENDPOINT=$(aws eks describe-cluster --name technews-karpenter-eks --region us-east-1 --query "cluster.endpoint" --output text)
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
echo "Cluster Endpoint: $CLUSTER_ENDPOINT"
echo "AWS Account ID: $AWS_ACCOUNT_ID"

Add the Karpenter Helm repository:

helm repo add karpenter https://charts.karpenter.sh/
helm repo update

Install Karpenter into the `karpenter` namespace:

helm install karpenter karpenter/karpenter --namespace karpenter --create-namespace \
  --set serviceAccount.create=false \
  --set serviceAccount.name=karpenter \
  --set settings.clusterName=technews-karpenter-eks \
  --set settings.clusterEndpoint="${CLUSTER_ENDPOINT}" \
  --set settings.defaultInstanceProfile=KarpenterNodeInstanceProfile-technews-karpenter-eks \
  --version 0.35.0 # Use the latest stable version

Verify Karpenter controller deployment:

kubectl get pods -n karpenter

Next, we define a Karpenter `NodePool` (in Karpenter v0.32+). This resource tells Karpenter what kind of nodes it can provision. Save this as `karpenter-nodepool.yaml`:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/provisioner-name # This tag is implicit for NodePools, but good to understand
          operator: Exists
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"] # Specify your cluster's zones
      nodeClassRef:
        name: default
  limits:
    resources:
      cpu: "1000"
      memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h # Nodes will be terminated after 30 days
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: AWSNodeClass
metadata:
  name: default
spec:
  amiFamily: AL2 # Can be AL2, Bottlerocket, Ubuntu, etc.
  role: KarpenterNodeRole-technews-karpenter-eks # The IAM role for EC2 instances
  securityGroupSelector:
    kubernetes.io/cluster/technews-karpenter-eks: owned # Selects security groups tagged by eksctl
  subnetSelector:
    karpenter.sh/discovery: technews-karpenter-eks # Selects subnets tagged by eksctl
  tags:
    karpenter.sh/nodepool: default
    karpenter.sh/discovery: technews-karpenter-eks
    Environment: Development

Apply the `NodePool` and `AWSNodeClass`:

kubectl apply -f karpenter-nodepool.yaml

Step 4: Verify Karpenter Operation

To see Karpenter in action, we'll deploy a sample application that requests more resources than currently available on our initial managed node group. This should trigger Karpenter to provision new nodes.

Create a `heavy-app.yaml` file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: heavy-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: heavy-app
  template:
    metadata:
      labels:
        app: heavy-app
    spec:
      containers:
      - name: busybox
        image: public.ecr.aws/karpenter/busybox:1.36
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "1Gi"
      terminationGracePeriodSeconds: 30

Deploy the application:

kubectl apply -f heavy-app.yaml

Watch for pending pods and new nodes being provisioned by Karpenter:

kubectl get pods -w
kubectl get nodes -w

You should observe Karpenter provisioning new EC2 instances, registering them with the EKS cluster, and then scheduling the `heavy-app` pods onto these new nodes. This process typically takes 1-2 minutes per node, significantly faster than traditional ASG scaling.

Once you are done, scale down the application to observe Karpenter's consolidation (node termination):

kubectl scale deployment/heavy-app --replicas=0
kubectl get nodes -w # Observe nodes terminating

Step 5: Implement IRSA (IAM Roles for Service Accounts)

Now, let's demonstrate IRSA by creating an IAM role that allows a pod to list S3 buckets, and then associate it with a Kubernetes Service Account.

First, ensure your EKS cluster has an OIDC provider associated. `eksctl` handles this automatically during cluster creation, but you can verify:

aws eks describe-cluster --name technews-karpenter-eks --region us-east-1 --query "cluster.identity.oidc.issuer" --output text

The output should be an URL like `https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED9C1F884B30F00000000000000000000`.

Create an IAM policy that grants S3 list access. Save this as `s3-list-policy.json`:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        }
    ]
}

Create the IAM policy:

aws iam create-policy \
    --policy-name S3ListAccessPolicy-technews-karpenter-eks \
    --policy-document file://s3-list-policy.json

Retrieve the ARN of the created policy:

S3_POLICY_ARN=$(aws iam list-policies --scope Local --query "Policies[?PolicyName=='S3ListAccessPolicy-technews-karpenter-eks'].Arn" --output text)
echo "S3 List Policy ARN: $S3_POLICY_ARN"

Create a Kubernetes Service Account and associate it with the IAM role using `eksctl`:

eksctl create iamserviceaccount \
  --cluster technews-karpenter-eks \
  --name s3-reader-sa \
  --namespace default \
  --attach-policy-arn ${S3_POLICY_ARN} \
  --approve \
  --override-existing-serviceaccounts

This command creates the Kubernetes Service Account `s3-reader-sa` in the `default` namespace and configures the trust policy on the newly created IAM role to allow the OIDC provider to assume it.

Now, deploy a pod that uses this Service Account and attempts to list S3 buckets. Create `s3-reader-pod.yaml`:

apiVersion: v1
kind: Pod
metadata:
  name: s3-reader-pod
spec:
  serviceAccountName: s3-reader-sa
  containers:
  - name: aws-cli-container
    image: amazon/aws-cli:latest
    command: ["/bin/sh", "-c", "aws s3 ls && sleep 3600"]
  restartPolicy: Never

Deploy the pod:

kubectl apply -f s3-reader-pod.yaml

Check the pod's logs. You should see a list of your S3 buckets, confirming that the pod successfully assumed the IAM role and accessed S3:

kubectl logs s3-reader-pod

If you were to deploy a similar pod *without* the `serviceAccountName: s3-reader-sa` or with a Service Account not linked to the IAM role, it would fail to list S3 buckets due to lack of permissions.

Step 6: Integrate Karpenter with IRSA (Implicitly)

It's important to understand that Karpenter itself leverages IRSA. The Karpenter controller pod, which we installed in Step 3, runs using the `karpenter` Service Account we created in Step 2. This Service Account is associated with the `KarpenterControllerPolicy-technews-karpenter-eks` IAM role, granting Karpenter the necessary permissions to interact with EC2 and EKS APIs. This is a prime example of IRSA in action for system-level components.

For your application pods, like the `s3-reader-pod`, they independently use their own Service Accounts and associated IAM roles (via IRSA) to access AWS services. Karpenter's role is to ensure there's enough compute capacity (EC2 instances) for these pods to run, while IRSA ensures these pods have secure, least-privilege access to the AWS services they need, regardless of which Karpenter-provisioned node they land on.

Security Considerations

Implementing EKS with Karpenter and IRSA significantly enhances security, but it's crucial to follow best practices:

• Least Privilege: Always adhere to the principle of least privilege for both Karpenter's IAM role and any IRSA-associated roles. Grant only the minimum permissions required for each component or application. Regularly review these policies.
• IAM Policy Scoping: For Karpenter's `iam:PassRole` permission, explicitly scope the resource to the node instance profile role (e.g., `arn:aws:iam::ACCOUNT_ID:role/KarpenterNodeRole-*`) rather than `*` in production.
• Network Security: Ensure your EKS cluster's security groups and network ACLs are configured to restrict traffic to only what is necessary. Karpenter nodes should be in private subnets with appropriate NAT Gateway access for outbound internet.
• Karpenter Node IAM Role: The IAM role assumed by Karpenter-provisioned nodes should also follow least privilege. Standard policies like `AmazonEKSWorkerNodePolicy`, `AmazonEC2ContainerRegistryReadOnly`, and `AmazonSSMManagedInstanceCore` are common starting points, but tailor them to your specific needs.
• EKS Control Plane Logging: Enable EKS control plane logging to CloudWatch Logs. This provides crucial audit trails for API calls made to your Kubernetes cluster, aiding in security investigations.
• Node Hardening: Use hardened AMIs (e.g., Bottlerocket or optimized AMIs) for your Karpenter-provisioned nodes. Regularly update node AMIs to patch security vulnerabilities.
• Karpenter and Kubernetes Updates: Keep Karpenter and your EKS cluster (Kubernetes version) updated to benefit from the latest security patches and features.
• Secrets Management: While IRSA eliminates the need for AWS credentials inside pods, other secrets (e.g., database passwords) should be managed securely using AWS Secrets Manager or Kubernetes secrets encrypted by KMS.

Best Practices

• Multiple NodePools: Create different `NodePool` configurations for distinct workload types. For example, a `NodePool` for CPU-intensive tasks, another for memory-intensive, and one for Spot instances