Nginx Ingress, cert-manager, external-dns on EKS: Automated TLS

Nginx Ingress Controller with cert-manager and external-dns on EKS: A Comprehensive Guide

Welcome to TechNews Venture, where we dissect complex cloud-native architectures to provide you with actionable insights. Today, we're diving deep into a fundamental yet powerful pattern for managing application ingress on Amazon Elastic Kubernetes Service (EKS): combining the Nginx Ingress Controller with cert-manager for automated SSL/TLS certificate provisioning and external-dns for dynamic DNS record management. This trifecta creates a robust, automated, and scalable solution for exposing your Kubernetes applications securely and reliably to the internet.

Overview

In the world of Kubernetes, exposing applications to external traffic requires an Ingress Controller. While AWS offers its own ALB Ingress Controller, many organizations prefer the battle-tested features, flexibility, and performance of the Nginx Ingress Controller. However, simply deploying an Ingress Controller isn't enough for production readiness. You need automated SSL/TLS certificate management to secure traffic and dynamic DNS record updates to ensure your services are discoverable.

This is where cert-manager and external-dns come into play:

• Nginx Ingress Controller: This acts as the Layer 7 load balancer and reverse proxy for your EKS cluster. It interprets Ingress resources, routing external HTTP/HTTPS traffic to the correct backend services within your cluster. It leverages AWS Elastic Load Balancers (typically NLB for performance and cost efficiency) to expose itself to the public internet.
• cert-manager: This Kubernetes add-on automates the management, issuance, and renewal of SSL/TLS certificates from various issuing sources like Let's Encrypt. It ensures your applications always have valid certificates, eliminating manual certificate handling and preventing expiration-related outages. We'll configure it to use the DNS01 challenge type with AWS Route 53.
• external-dns: This tool synchronizes exposed Kubernetes services and Ingresses with DNS providers like AWS Route 53. It automatically creates and updates DNS records based on your Ingress resources' hostnames, ensuring that when you deploy or update an application, its DNS record is managed seamlessly.

By integrating these three components, we achieve a fully automated system where:

1. You define an Ingress resource for your application.
2. external-dns creates the necessary DNS A record pointing to your Nginx Ingress Controller's Load Balancer.
3. cert-manager automatically requests and provisions an SSL certificate for your specified hostname, using the DNS01 challenge to prove domain ownership via Route 53.
4. The Nginx Ingress Controller uses this certificate to serve HTTPS traffic, routing it to your application.

This setup significantly reduces operational overhead, enhances security by ensuring valid certificates, and provides a consistent way to manage external access to your EKS workloads.

Prerequisites

Before we begin, ensure you have the following tools installed and configured, along with the necessary AWS resources and permissions:

Tools:

• kubectl: The Kubernetes command-line tool.

  aws eks update-kubeconfig --region us-east-1 --name techventure-eks-cluster

• Helm v3+: The package manager for Kubernetes.

  curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

• AWS CLI v2+: The command-line interface for AWS.

  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  unzip awscliv2.zip
  sudo ./aws/install

• eksctl: A simple CLI tool for creating and managing EKS clusters.

  curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
  sudo mv /tmp/eksctl /usr/local/bin

• jq: A lightweight and flexible command-line JSON processor.

  sudo apt-get install jq -y # For Debian/Ubuntu
  sudo yum install jq -y # For RHEL/CentOS

AWS Resources and Permissions:

• An EKS Cluster: Ensure you have an existing EKS cluster. For this guide, we'll assume a cluster named techventure-eks-cluster in the us-east-1 region.
• IAM OIDC Provider: Your EKS cluster must have an IAM OIDC provider enabled. This is crucial for Kubernetes Service Accounts to assume AWS IAM roles. If not enabled, you can do so with:

  eksctl utils associate-iam-oidc-provider --region us-east-1 --cluster techventure-eks-cluster --approve

• AWS Route 53 Hosted Zone: You need an existing public Route 53 Hosted Zone for your domain (e.g., techventure.cloud). This is where external-dns will create records and cert-manager will perform DNS01 challenges.
• IAM User/Role Permissions: The AWS credentials you're using must have permissions to create and manage IAM roles, policies, and Route 53 records.

Step-by-step Implementation

Let's walk through the setup process systematically.

Step 1: EKS Cluster and OIDC Provider Verification

As mentioned in the prerequisites, ensure your EKS cluster is ready and has an IAM OIDC provider associated. This is fundamental for enabling IAM Roles for Service Accounts (IRSA), which allows Kubernetes service accounts to assume IAM roles, granting fine-grained AWS permissions to your pods without managing AWS credentials directly.

# Verify OIDC provider association
aws eks describe-cluster --name techventure-eks-cluster --region us-east-1 --query "cluster.identity.oidc.issuer" --output text

The output should be a URL like https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED990499C0FA7A082C56272F3E.

Step 2: Install Nginx Ingress Controller

We'll use Helm to deploy the Nginx Ingress Controller. We'll configure it to provision an AWS Network Load Balancer (NLB) for optimal performance and cost.

# 1. Create a namespace for the Nginx Ingress Controller
kubectl create namespace ingress-nginx

# 2. Add the official Nginx Ingress Helm repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# 3. Install the Nginx Ingress Controller
# We specify service.type=LoadBalancer and service.annotations for AWS NLB.
# The `nginx.ingress.kubernetes.io/ssl-redirect: "false"` is for initial testing;
# cert-manager will handle TLS, so we might enable it later or let ingress rules handle it.
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"="internet-facing" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-nlb-target-type"="ip" \
  --set controller.service.externalTrafficPolicy=Local \
  --set controller.metrics.enabled=true \
  --set controller.podLabels.app.kubernetes.io/component=controller \
  --set controller.podLabels.app.kubernetes.io/instance=ingress-nginx \
  --set controller.podLabels.app.kubernetes.io/name=ingress-nginx \
  --set controller.podLabels.app.kubernetes.io/version="$(helm search repo ingress-nginx/ingress-nginx -o json | jq -r '.[0].app_version')"

# 4. Verify the Nginx Ingress Controller deployment
kubectl get pods --namespace ingress-nginx -l app.kubernetes.io/name=ingress-nginx
kubectl get svc --namespace ingress-nginx -l app.kubernetes.io/name=ingress-nginx

You should see a LoadBalancer service with an AWS NLB hostname. Note this hostname, as it's where your DNS records will eventually point.

NAME                                TYPE           CLUSTER-IP     EXTERNAL-IP                                                                    PORT(S)                      AGE
ingress-nginx-controller            LoadBalancer   10.100.1.200   k8s-ingressn-ingressn-xxxxxxxx-yyyyyyyy.elb.us-east-1.amazonaws.com   80:31111/TCP,443:32222/TCP   5m

Step 3: Install cert-manager

cert-manager will automate the provisioning and renewal of SSL/TLS certificates using Let's Encrypt and the DNS01 challenge type with Route 53.

# 1. Create a namespace for cert-manager
kubectl create namespace cert-manager

# 2. Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# 3. Install cert-manager
# We need to install CRDs as part of the installation.
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.12.0 \
  --set installCRDs=true \
  --set extraArgs[0]=--enable-certificate-owner-ref=true \
  --set podLabels.app.kubernetes.io/component=controller \
  --set podLabels.app.kubernetes.io/instance=cert-manager \
  --set podLabels.app.kubernetes.io/name=cert-manager \
  --set podLabels.app.kubernetes.io/version="v1.12.0"

# 4. Verify cert-manager deployment
kubectl get pods --namespace cert-manager -l app.kubernetes.io/name=cert-manager

You should see cert-manager, cert-manager-webhook, and cert-manager-cainjector pods running.

Configure cert-manager with IAM Role for Route 53

cert-manager needs permissions to interact with AWS Route 53 to complete DNS01 challenges. We'll use IRSA for this.

# 1. Define the IAM Policy for cert-manager Route 53 access
# Save this content to cert-manager-route53-policy.json
cat < cert-manager-route53-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "arn:aws:aws:route53:::hostedzone/YOUR_HOSTED_ZONE_ID"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*"
        }
    ]
}
EOF

# Replace YOUR_HOSTED_ZONE_ID with your actual Route 53 Hosted Zone ID.
# You can get it using: aws route53 list-hosted-zones-by-name --dns-name techventure.cloud. --query "HostedZones[0].Id" --output text | cut -d '/' -f 3
HOSTED_ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name techventure.cloud. --query "HostedZones[0].Id" --output text | cut -d '/' -f 3)
sed -i "s|YOUR_HOSTED_ZONE_ID|$HOSTED_ZONE_ID|" cert-manager-route53-policy.json

# 2. Create the IAM Policy
aws iam create-policy \
    --policy-name CertManagerRoute53Policy \
    --policy-document file://cert-manager-route53-policy.json

# 3. Create an IAM Service Account for cert-manager
# eksctl will create the IAM role and attach the policy, and also annotate the K8s service account.
eksctl create iamserviceaccount \
    --cluster techventure-eks-cluster \
    --namespace cert-manager \
    --name cert-manager \
    --attach-policy-arn arn:aws:iam::123456789012:policy/CertManagerRoute53Policy \
    --approve \
    --override-existing-serviceaccounts # Use this if you're re-running or upgrading

# 4. Define a ClusterIssuer for Let's Encrypt (DNS01 challenge)
# Save this content to clusterissuer.yaml
cat < clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: admin@techventure.cloud
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    solvers:
    - dns01:
        route53:
          region: us-east-1
          # The SecretAccessKey and AccessKeyID are not needed here because IRSA is used.
          # The IAM role attached to the cert-manager service account provides the necessary permissions.
          # hostedZoneID: "YOUR_HOSTED_ZONE_ID" # Optional, if you only have one hosted zone or prefer to specify
EOF

# Apply the ClusterIssuer
kubectl apply -f clusterissuer.yaml

# 5. Verify ClusterIssuer status
kubectl get clusterissuer letsencrypt-prod -o yaml

Look for status.conditions with type Ready and status True.

Step 4: Install external-dns

external-dns will automatically create and update DNS records in Route 53 based on your Ingress resources.

# 1. Create a namespace for external-dns
kubectl create namespace external-dns

# 2. Add the external-dns Helm repository
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update

# 3. Define the IAM Policy for external-dns Route 53 access
# Save this content to external-dns-route53-policy.json
cat < external-dns-route53-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "*"
    }
  ]
}
EOF

# 4. Create the IAM Policy
aws iam create-policy \
    --policy-name ExternalDNSUpdaterPolicy \
    --policy-document file://external-dns-route53-policy.json

# 5. Create an IAM Service Account for external-dns
eksctl create iamserviceaccount \
    --cluster techventure-eks-cluster \
    --namespace external-dns \
    --name external-dns \
    --attach-policy-arn arn:aws:iam::123456789012:policy/ExternalDNSUpdaterPolicy \
    --approve \
    --override-existing-serviceaccounts

# 6. Install external-dns
# Replace YOUR_DOMAIN with your actual domain (e.g., techventure.cloud)
helm install external-dns external-dns/external-dns \
  --namespace external-dns \
  --version 1.12.0 \
  --set provider=aws \
  --set serviceAccount.create=false \
  --set serviceAccount.name=external-dns \
  --set domainFilters={techventure.cloud} \
  --set policy=upsert-only \
  --set aws.region=us-east-1 \
  --set rbac.create=true \
  --set podLabels.app.kubernetes.io/component=controller \
  --set podLabels.app.kubernetes.io/instance=external-dns \
  --set podLabels.app.kubernetes.io/name=external-dns \
  --set podLabels.app.kubernetes.io/version="1.12.0"

# 7. Verify external-dns deployment
kubectl get pods --namespace external-dns -l app.kubernetes.io/name=external-dns

You should see the external-dns pod running.

Step 5: Deploy a Sample Application and Ingress

Now, let's deploy a simple Nginx application and an Ingress resource to test our setup.

# 1. Create a namespace for the sample application
kubectl create namespace sample-app

# 2. Deploy a sample Nginx application
# Save this content to sample-app.yaml
cat < sample-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: sample-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.3
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: sample-app
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

kubectl apply -f sample-app.yaml

# 3. Create an Ingress resource
# Replace YOUR_DOMAIN with your actual domain (e.g., techventure.cloud)
# Save this content to sample-ingress.yaml
cat < sample-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: sample-app
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    external-dns.alpha.kubernetes.io/hostname: myapp.techventure.cloud
    # Optional: Force HTTPS redirection at the Nginx Ingress Controller level
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # Optional: Configure HSTS
    nginx.ingress.kubernetes.io/hsts: "true"
    nginx.ingress.kubernetes.io/hsts-max-age: "31536000"
    nginx.ingress.kubernetes.io/hsts-include-subdomains: "true"
spec:
  tls:
  - hosts:
    - myapp.techventure.cloud
    secretName: myapp-tls-secret # cert-manager will create this secret
  rules:
  - host: myapp.techventure.cloud
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
EOF

kubectl apply -f sample-ingress.yaml

Step 6: Verification

Now, let's verify that everything is working as expected:

# 1. Verify Ingress status
kubectl get ingress -n sample-app nginx-ingress

You should see the Nginx Ingress Controller's Load Balancer hostname listed under ADDRESS.

# 2. Verify DNS record in Route 53
# This might take a minute or two for external-dns to create the record.
aws route53 list-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --query "ResourceRecordSets[?Name == 'myapp.techventure.cloud.']"

You should see an A record pointing to your Nginx Ingress Controller's Load Balancer.

# 3. Verify Certificate status
# This might take a few minutes for cert-manager to obtain the certificate.
kubectl get certificate -n sample-app myapp-tls-secret -o yaml

Look for status.conditions with type Ready and status True.

# 4. Access your application
# Open a browser and navigate to https://myapp.techventure.cloud
# You should see the Nginx welcome page with a valid SSL certificate.
curl -v https://myapp.techventure.cloud

Ensure the certificate chain is valid and signed by Let's Encrypt.

Security Considerations

While the setup provides significant automation and security benefits, it's crucial to consider the following:

• Least Privilege IAM Policies: The IAM policies for cert-manager and external-dns should be as restrictive as possible, granting only the necessary Route 53 permissions. Avoid using '*' for resources or actions where specific ARN or actions can be defined.
• Network Policies: Implement Kubernetes Network Policies to control ingress and egress traffic between namespaces and pods. This prevents unauthorized access to your applications and limits potential lateral movement within the cluster.
• TLS Versions and Ciphers: Configure Nginx Ingress Controller to use strong TLS versions (e.g., TLSv1.2, TLSv1.3) and modern cipher suites. This can be done via annotations on the Ingress resource or global configurations in the Nginx Ingress Controller Helm chart values.
• Regular Updates: Keep Nginx Ingress Controller, cert-manager, external-dns, and your EKS cluster up-to-date with the latest security patches and versions.
• AWS WAF/Shield: For enhanced protection against DDoS attacks and common web exploits, consider placing an AWS WAF (Web Application Firewall) or AWS Shield Advanced in front of your Nginx Ingress Controller's NLB. This can be configured by adding specific annotations to the Nginx Ingress Controller service.
• Private Subnets for Worker Nodes: For increased security, deploy your EKS worker nodes into private subnets, allowing the NLB to route traffic to them internally.

Best Practices

• Use NLB for Nginx Ingress: As demonstrated, using AWS NLB for the Nginx Ingress Controller is generally preferred over ALB for performance-sensitive applications due to its lower latency and higher throughput. It also simplifies the architecture as the Nginx Ingress handles Layer 7 routing.
• Separate Namespaces: Deploy infrastructure components (Nginx Ingress, cert-manager, external-dns) into their dedicated namespaces (e.g., ingress-nginx, cert-manager, external-dns) for better isolation and management.
• GitOps Approach: Manage all your Kubernetes manifests (Deployments, Services, Ingresses, ClusterIssuers, etc.) in a Git repository. Tools like Argo CD or Flux CD can automate the deployment and synchronization of these configurations to your EKS cluster, ensuring consistency and traceability.
• Monitoring and Alerting: Set up comprehensive monitoring for all components. Use Prometheus and Grafana to collect metrics from Nginx Ingress Controller, cert-manager, and external-dns. Configure alerts for certificate expiration, Ingress controller errors, or DNS update failures.
• Rate Limiting and WAF: Implement rate limiting at the Ingress Controller level (using Nginx annotations) to protect against abuse. For more advanced threat protection, integrate with AWS WAF.
• Custom Error Pages: Configure custom error pages for your Nginx Ingress Controller to provide a better user experience and hide sensitive information when errors occur.
• Staging Environments: Always test changes in a staging environment before deploying to production. Use Let's Encrypt's staging ACME server (https://acme-staging-v02.api.letsencrypt.org/directory) for testing certificate issuance to avoid hitting rate limits on the production server.

FAQ

Here are some frequently asked questions about this setup:

Q1: Why choose Nginx Ingress Controller over AWS ALB Ingress Controller?

Both have their merits. The AWS ALB Ingress Controller integrates natively with AWS ALBs, offering features like WAF integration and Cognito authentication directly at the ALB. However, the Nginx Ingress Controller offers greater flexibility, advanced routing capabilities (e.g., URL rewriting, header manipulation, custom Lua scripting), better performance for certain workloads, and a consistent experience across different cloud providers or on-premises environments. It's a mature, battle-tested solution that gives you more control over the ingress layer within Kubernetes, often simplifying the overall architecture by having one central Ingress controller handle all Layer 7 concerns.

Q2: How do I handle multiple domains or wildcard certificates?

For multiple domains, simply list all hostnames under the hosts section in your Ingress resource's tls block and ensure each domain is covered by an external-dns.alpha.