Efficiently scale AWS EKS clusters with Karpenter autoscaler & secure pod identity using IRSA. Optimize your Kubernetes setup.
Overview: Revolutionizing EKS Autoscaling and Pod Identity with Karpenter and IRSA
Welcome to TechNews Venture, where we delve deep into the technologies shaping the cloud-native landscape. Today, we're exploring a powerful combination for AWS EKS users: Karpenter for intelligent, event-driven autoscaling and IAM Roles for Service Accounts (IRSA) for fine-grained pod identity. This duo represents a significant leap forward in managing Kubernetes clusters on AWS, offering unparalleled efficiency, cost optimization, and robust security.
AWS Elastic Kubernetes Service (EKS) provides a managed control plane for Kubernetes, abstracting away much of the operational overhead. However, managing the underlying worker nodes has traditionally presented challenges. The native Kubernetes Cluster Autoscaler, while effective, operates by managing EC2 Auto Scaling Groups (ASGs). This approach can lead to slower scaling times, less optimal instance type selection, and difficulty in consolidating nodes efficiently.
Enter Karpenter, an open-source, high-performance Kubernetes autoscaler built by AWS. Unlike the Cluster Autoscaler, Karpenter directly provisions and deprovisions EC2 instances based on pending pods, bypassing ASGs entirely. This allows it to make more intelligent decisions about instance types, sizes, and even Spot instance usage, leading to significant cost savings and faster scaling. Karpenter monitors the cluster for unschedulable pods and quickly launches the most appropriate EC2 instances, then gracefully terminates nodes when they are no longer needed, optimizing resource utilization.
Complementing this operational efficiency is IAM Roles for Service Accounts (IRSA). In traditional Kubernetes deployments on AWS, pods requiring AWS API access often relied on the IAM role attached to the underlying EC2 instance. This "node instance profile" approach grants all pods on a node the same permissions, violating the principle of least privilege. IRSA solves this by allowing you to associate an AWS IAM role directly with a Kubernetes Service Account. Pods configured to use that service account can then assume the specified IAM role, granting them only the necessary AWS permissions, and enhancing the security posture of your applications.
By integrating Karpenter with an EKS cluster and leveraging IRSA for pod identity, we achieve a highly elastic, cost-efficient, and secure Kubernetes environment. This article will guide you through the process of setting up an EKS cluster, deploying Karpenter, and demonstrating IRSA for your application pods, providing detailed commands and configurations every step of the way.
Prerequisites
Before we begin, ensure you have the following tools and configurations in place:
- An active AWS Account with administrative privileges.
- AWS CLI installed and configured with appropriate credentials. Ensure your default region is set (e.g.,
us-east-1).
kubectl: The Kubernetes command-line tool, configured to interact with your EKS cluster.eksctl: The official CLI for Amazon EKS, used for creating and managing EKS clusters.helm: The package manager for Kubernetes.jq: A lightweight and flexible command-line JSON processor, useful for parsing AWS CLI output.- Basic understanding of Kubernetes concepts (pods, deployments, service accounts) and AWS IAM.
- A VPC with private subnets is highly recommended for EKS worker nodes to enhance security.
eksctl can create this for you.
You can verify the installations with:
aws --version
kubectl version --client
eksctl version
helm version
jq --version
Step-by-step Implementation
Let's dive into the implementation. We will start by creating an EKS cluster, then install Karpenter, and finally demonstrate IRSA.
3.1. Create an EKS Cluster with eksctl
We'll create a new EKS cluster without any managed node groups, as Karpenter will be responsible for provisioning all worker nodes. We'll ensure the cluster is configured with OIDC provider for IRSA.
First, create a cluster configuration file named `cluster.yaml`:
# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: techenews-karpenter-eks
region: us-east-1
version: "1.28"
tags:
KarpenterProvisioned: "true" # Tag for Karpenter to discover the cluster
Environment: "Production"
iam:
withOIDC: true # Enable OIDC provider for IRSA
privateNetworking: true # Ensure worker nodes are in private subnets
# No managed node groups, Karpenter will manage nodes
managedNodeGroups: []
Now, create the EKS cluster using `eksctl`:
eksctl create cluster -f cluster.yaml
This command will take 15-20 minutes to complete. It will create a new VPC (if one isn't specified), subnets, security groups, the EKS control plane, and the OIDC identity provider.
Once the cluster is created, verify your `kubectl` context and cluster status:
kubectl get svc
kubectl get nodes # Should show no nodes initially
3.2. Install Karpenter
Installing Karpenter involves several steps: setting up IAM roles, creating a Kubernetes service account, deploying the Karpenter controller via Helm, and defining Karpenter `NodePool` and `EC2NodeClass` resources.
3.2.1. Retrieve Cluster Information
We need the cluster's OIDC provider URL, VPC ID, and private subnet IDs for Karpenter configuration.
# Get OIDC Issuer URL
OIDC_ISSUER=$(aws eks describe-cluster --name techenews-karpenter-eks --region us-east-1 --query "cluster.identity.oidc.issuer" --output text)
echo "OIDC Issuer: $OIDC_ISSUER"
# Get OIDC Provider ID
OIDC_ID=$(echo $OIDC_ISSUER | sed -e "s/^https:\/\///")
echo "OIDC ID: $OIDC_ID"
# Get VPC ID
VPC_ID=$(aws eks describe-cluster --name techenews-karpenter-eks --region us-east-1 --query "cluster.resourcesVpcConfig.vpcId" --output text)
echo "VPC ID: $VPC_ID"
# Get Private Subnet IDs (assuming eksctl created them with default tags)
SUBNET_IDS=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:eksctl.cluster.k8s.io/v1alpha1/cluster-name,Values=techenews-karpenter-eks" "Name=tag:eksctl.cluster.k8s.io/v1alpha1/subnet-type,Values=private" --query "Subnets[*].SubnetId" --output text | tr '\t' ',')
echo "Private Subnet IDs: $SUBNET_IDS"
# Get Security Group IDs (for the EKS cluster)
# This assumes eksctl created default security groups for the cluster.
# We'll use the default cluster security group for nodes.
CLUSTER_SG_ID=$(aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:eksctl.cluster.k8s.io/v1alpha1/cluster-name,Values=techenews-karpenter-eks" "Name=group-name,Values=*ClusterSharedNodeSecurityGroup*" --query "SecurityGroups[0].GroupId" --output text)
echo "Cluster Security Group ID: $CLUSTER_SG_ID"
3.2.2. Create Karpenter Node IAM Role and Instance Profile
This role will be assumed by the EC2 instances provisioned by Karpenter. It grants the necessary permissions for worker nodes to join the EKS cluster and interact with AWS services like ECR.
# Create IAM Policy for Karpenter Node Role
aws iam create-policy \
--policy-name KarpenterNodePolicy-techenews-karpenter-eks \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AssociateAddress",
"ec2:AssociateIamInstanceProfile",
"ec2:AttachVolume",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CopySnapshot",
"ec2:CreateFleet",
"ec2:CreateLaunchTemplate",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:DeleteLaunchTemplate",
"ec2:DeleteSnapshot",
"ec2:DeregisterImage",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkAcls",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribePlacementGroups",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSnapshots",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:DescribeVpcs",
"ec2:DisassociateAddress",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyLaunchTemplate",
"ec2:ModifyVolume",
"ec2:RebootInstances",
"ec2:RunInstances",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:UnmonitorInstances"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/KarpenterNodeRole-techenews-karpenter-eks"
},
{
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": "arn:aws:ssm:*:*:parameter/aws/service/ami-amazon-linux-2-hvm-x86_64/recommended/image_id"
}
]
}'
# Create IAM Role
aws iam create-role \
--role-name KarpenterNodeRole-techenews-karpenter-eks \
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
# Attach required policies
aws iam attach-role-policy \
--role-name KarpenterNodeRole-techenews-karpenter-eks \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy \
--role-name KarpenterNodeRole-techenews-karpenter-eks \
--policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy \
--role-name KarpenterNodeRole-techenews-karpenter-eks \
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
# Create Instance Profile
aws iam create-instance-profile \
--instance-profile-name KarpenterNodeInstanceProfile-techenews-karpenter-eks
aws iam add-role-to-instance-profile \
--instance-profile-name KarpenterNodeInstanceProfile-techenews-karpenter-eks \
--role-name KarpenterNodeRole-techenews-karpenter-eks
KARPENTER_NODE_INSTANCE_PROFILE="KarpenterNodeInstanceProfile-techenews-karpenter-eks"
echo "Karpenter Node Instance Profile: $KARPENTER_NODE_INSTANCE_PROFILE"
3.2.3. Create Karpenter Controller IAM Role (for IRSA)
This role will be assumed by the Karpenter controller pod via IRSA. It needs permissions to call AWS EC2 and EKS APIs to provision and manage instances.
# Create Trust Policy for Karpenter Controller Role
cat > karpenter-trust-policy.json < karpenter-controller-policy.json <
3.2.4. Deploy Karpenter with Helm
First, add the Karpenter Helm repository and create the `karpenter` namespace.
helm repo add karpenter https://charts.karpenter.sh/
helm repo update
kubectl create namespace karpenter
Now, install Karpenter using Helm, associating it with the IAM role created for the controller. We'll use Karpenter v0.32.0 or newer which uses `NodePool` and `EC2NodeClass`.
helm upgrade --install karpenter karpenter/karpenter --namespace karpenter \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${KARPENTER_CONTROLLER_ROLE_ARN}" \
--set settings.clusterName=techenews-karpenter-eks \
--set settings.clusterEndpoint="${OIDC_ISSUER}" \
--set settings.defaultInstanceProfile="${KARPENTER_NODE_INSTANCE_PROFILE}" \
--set defaultRepository="public.ecr.aws/karpenter/karpenter" \
--wait
Verify Karpenter controller is running:
kubectl get pods -n karpenter
3.2.5. Create Karpenter NodePool and EC2NodeClass
Karpenter uses `NodePool` and `EC2NodeClass` custom resources to define how it should provision nodes.
Create `ec2nodeclass.yaml`:
# ec2nodeclass.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: default
spec:
amiFamily: AL2 # Amazon Linux 2
role: KarpenterNodeRole-techenews-karpenter-eks # The IAM role for EC2 instances
subnetSelectorTerms:
- tags:
eksctl.cluster.k8s.io/v1alpha1/cluster-name: techenews-karpenter-eks
eksctl.cluster.k8s.io/v1alpha1/subnet-type: private
securityGroupSelectorTerms:
- tags:
eksctl.cluster.k8s.io/v1alpha1/cluster-name: techenews-karpenter-eks
tags:
karpenter.sh/discovery: techenews-karpenter-eks # Tag for Karpenter to discover the cluster
Environment: Production
Owner: SujaySingh
# You can specify detailed launch template properties here if needed
# launchTemplate: default-karpenter-lt # Optional, Karpenter creates one by default
Apply the `EC2NodeClass`:
kubectl apply -f ec2nodeclass.yaml
Now, create `nodepool.yaml`:
# nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
name: default # Reference the EC2NodeClass created above
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"] # Use on-demand instances
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: kubernetes.io/os
operator: In
values: ["linux"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"] # Allow compute, memory, and general purpose instances
- key: karpenter.k8s.aws/instance-family
operator: NotIn
values: ["t2", "t3"] # Exclude burstable instances for general workloads
# Taints can be added here if you want to dedicate nodes to specific workloads
# taints:
# - key: dedicated-workload
# value: "true"
# effect: NoSchedule
# labels:
# app.kubernetes.io/name: my-app
limits:
cpu: "100" # Limit total CPU in the node pool to 100 cores
memory: "200Gi" # Limit total memory in the node pool to 200Gi
disruption:
consolidationPolicy: WhenEmpty # Consolidate nodes when they are empty
expireAfter: 720h # Nodes will be terminated after 30 days
Apply the `NodePool`:
kubectl apply -f nodepool.yaml
3.3. Deploy a Sample
