Automate TLS on EKS: Nginx Ingress, cert-manager & external-dns

Navigating the Cloud Frontier: Nginx Ingress Controller with cert-manager and external-dns on EKS

As applications increasingly shift to containerized microservices architectures, managing network traffic, SSL/TLS certificates, and DNS records becomes a critical aspect of cloud operations. Kubernetes, coupled with Amazon EKS, provides a robust platform, but the ingress layer often requires careful orchestration. This article delves into a powerful and automated solution: integrating Nginx Ingress Controller with cert-manager for automated SSL/TLS certificate provisioning and external-dns for dynamic DNS record management, all running seamlessly on Amazon Elastic Kubernetes Service (EKS).

This trifecta offers a production-ready setup that automates the most common operational headaches related to exposing applications securely and reliably to the internet. We'll walk through the entire process, from setting up IAM roles to deploying a sample application, ensuring you have a clear, actionable guide to implement this critical architecture.

Overview

At the heart of any modern cloud-native application stack is the ability to securely and efficiently expose services to end-users. On Kubernetes, this is primarily handled by an Ingress Controller. The Nginx Ingress Controller, a battle-tested and highly performant solution, acts as the entry point for external traffic, routing it to the correct services within your EKS cluster.

However, simply routing traffic isn't enough. Modern web applications demand secure communication via HTTPS, which necessitates SSL/TLS certificates. Manually managing these certificates – generating, renewing, and deploying them – is a tedious and error-prone task. Enter cert-manager, a native Kubernetes certificate management controller that automates the entire lifecycle of certificates from various issuing sources like Let's Encrypt. It ensures your applications always have valid, up-to-date certificates without human intervention.

Finally, to make your applications discoverable, you need DNS records. When services are dynamically provisioned in Kubernetes, their external IP addresses or hostnames (like those provided by an AWS Load Balancer) can change. external-dns bridges this gap by automatically creating and updating DNS records in your DNS provider (e.g., AWS Route 53) based on Kubernetes Ingresses and Services. This means no more manual DNS updates when your application's external endpoint changes.

Together, these three components form a robust, self-healing, and highly automated ingress solution for your EKS workloads, significantly reducing operational overhead and improving security posture.

Prerequisites

Before we embark on this technical journey, ensure you have the following prerequisites in place:

• AWS Account: An active AWS account with administrative privileges to create IAM roles, policies, and manage EKS.
• Amazon EKS Cluster: A running EKS cluster. For this guide, we'll assume a cluster named tech-news-eks-cluster in the us-east-1 region.
• kubectl: The Kubernetes command-line tool, installed and configured to connect to your EKS cluster.

  aws eks update-kubeconfig --name tech-news-eks-cluster --region us-east-1

• Helm: The Kubernetes package manager, version 3.x installed.

  helm version

• AWS CLI: The Amazon Web Services command-line interface, configured with appropriate credentials.

  aws configure

• Domain Name: A registered domain name (e.g., devops-demo.online) whose DNS records are managed by AWS Route 53. This is crucial for external-dns to function.
• IAM OIDC Provider for EKS: Your EKS cluster must have an IAM OIDC (OpenID Connect) provider enabled. This allows Kubernetes Service Accounts to assume AWS IAM roles, a feature known as IAM Roles for Service Accounts (IRSA), which is a security best practice. If you don't have one, you can create it with:

  eksctl utils associate-iam-oidc-provider --cluster=tech-news-eks-cluster --approve --region=us-east-1

Step-by-step Implementation

1. Configure IAM Policies and Roles for external-dns and cert-manager

Both external-dns and cert-manager (when using the DNS-01 challenge with Route 53) require permissions to interact with AWS Route 53 to manage DNS records. We'll create separate, least-privilege IAM policies and roles for each.

1.1. IAM Policy for external-dns

This policy grants external-dns the necessary permissions to list hosted zones and modify record sets in Route 53.

cat <<EOF > external-dns-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Resource": [
        "arn:aws:route53:::hostedzone/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

aws iam create-policy \
    --policy-name ExternalDNS_Route53_Policy \
    --policy-document file://external-dns-policy.json

1.2. IAM Policy for cert-manager (DNS-01 Challenge)

cert-manager needs to create specific TXT records in Route 53 to complete the DNS-01 challenge from Let's Encrypt. This policy grants those permissions.

cat <<EOF > cert-manager-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
    --policy-name CertManager_Route53_Policy \
    --policy-document file://cert-manager-policy.json

1.3. Create IAM Roles and Service Accounts for external-dns and cert-manager

Now, we'll create the Kubernetes Service Accounts and associate them with the IAM policies using IRSA. This is done with eksctl for simplicity.

For external-dns:

eksctl create iamserviceaccount \
    --cluster=tech-news-eks-cluster \
    --namespace=external-dns \
    --name=external-dns-sa \
    --attach-policy-arn=$(aws iam list-policies --query "Policies[?PolicyName=='ExternalDNS_Route53_Policy'].Arn" --output text) \
    --override-existing-serviceaccounts \
    --approve \
    --region=us-east-1

For cert-manager:

eksctl create iamserviceaccount \
    --cluster=tech-news-eks-cluster \
    --namespace=cert-manager \
    --name=cert-manager-sa \
    --attach-policy-arn=$(aws iam list-policies --query "Policies[?PolicyName=='CertManager_Route53_Policy'].Arn" --output text) \
    --override-existing-serviceaccounts \
    --approve \
    --region=us-east-1

Note: The --override-existing-serviceaccounts flag is useful if you run the command multiple times or need to update the role attached to an existing SA. Ensure the namespaces external-dns and cert-manager exist or are created by the eksctl command.

2. Install Nginx Ingress Controller

The Nginx Ingress Controller will be deployed as a Kubernetes Deployment and a Service of type LoadBalancer, which will provision an AWS Network Load Balancer (NLB) by default on EKS.

# Add the official Nginx Ingress Controller Helm repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Create a dedicated namespace for the Ingress Controller
kubectl create namespace ingress-nginx

# Install the Nginx Ingress Controller
helm install ingress-nginx ingress-nginx/ingress-nginx \
    --namespace ingress-nginx \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"="internet-facing" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-healthcheck-protocol"="TCP" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-target-group-attributes"="preserve_client_ip.enabled=true" \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.enabled=true \
    --version 4.10.1 # Use a recent stable version

Verify the installation and get the Load Balancer hostname:

kubectl get svc -n ingress-nginx ingress-nginx-controller

# Example Output (look for EXTERNAL-IP, which will be an AWS NLB hostname):
# NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                      AGE
# ingress-nginx-controller      LoadBalancer   10.100.123.45    k8s-ingressn-ingressn-xxxxxxxxxx-xxxxxxxxxx.elb.us-east-1.amazonaws.com   80:30080/TCP,443:30443/TCP   2m

Make a note of the EXTERNAL-IP hostname. This is the entry point for all your ingress traffic.

3. Install cert-manager and Configure ClusterIssuers

cert-manager will handle the automated provisioning and renewal of SSL/TLS certificates using Let's Encrypt.

# Add the cert-manager Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Create a dedicated namespace for cert-manager
kubectl create namespace cert-manager

# Install cert-manager
helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --version v1.14.4 \
    --set installCRDs=true \
    --set serviceAccount.name=cert-manager-sa # Link to the SA created earlier

Verify cert-manager pods are running:

kubectl get pods -n cert-manager

3.1. Create ClusterIssuers for Let's Encrypt

We'll configure two ClusterIssuers: one for staging (for testing) and one for production. Both will use the DNS-01 challenge with Route 53.

cat <<EOF > cluster-issuers.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: your-email@devops-demo.online # Replace with your actual email
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-private-key
    solvers:
    - dns01:
        route53:
          region: us-east-1 # Your EKS cluster region
          # If you created the IAM role using eksctl, you don't need to specify AccessKeyID/SecretAccessKey
          # The service account 'cert-manager-sa' will automatically assume the IAM role.
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: your-email@devops-demo.online # Replace with your actual email
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    solvers:
    - dns01:
        route53:
          region: us-east-1 # Your EKS cluster region
EOF

kubectl apply -f cluster-issuers.yaml

Check the status of the ClusterIssuers:

kubectl get clusterissuers

Ensure both show Ready: True.

4. Install external-dns

external-dns will automatically create and manage DNS records in Route 53 for your Ingresses.

# Add the external-dns Helm repository (if not already added, though it's often stable)
# helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
# helm repo update

# Create a dedicated namespace for external-dns
kubectl create namespace external-dns

# Install external-dns
helm install external-dns external-dns/external-dns \
    --namespace external-dns \
    --version 1.14.0 \
    --set serviceAccount.name=external-dns-sa \
    --set provider=aws \
    --set domainFilters={devops-demo.online} \
    --set aws.region=us-east-1 \
    --set aws.zoneType=public \
    --set policy=upsert-only \
    --set rbac.create=true \
    --set txtOwnerId=devops-demo-online-eks # Unique identifier for external-dns records

Verify external-dns is running:

kubectl get pods -n external-dns

5. Deploy a Sample Application and Ingress

Now, let's deploy a simple Nginx application and an Ingress resource to test our setup. This Ingress will tell Nginx Ingress Controller how to route traffic, external-dns to create a DNS record, and cert-manager to provision an SSL certificate.

cat <<EOF > sample-app-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod # Use the production issuer
    external-dns.alpha.kubernetes.io/hostname: demo.devops-demo.online # Explicitly set hostname if desired
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true" # Redirect HTTP to HTTPS
spec:
  tls:
  - hosts:
    - demo.devops-demo.online # Replace with your subdomain
    secretName: nginx-tls-secret # cert-manager will store the certificate here
  rules:
  - host: demo.devops-demo.online # Replace with your subdomain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
EOF

kubectl apply -f sample-app-ingress.yaml

Verification Steps:

1. Verify DNS Record: external-dns should create an A record for demo.devops-demo.online pointing to your Nginx Ingress Controller's Load Balancer hostname. This might take a few moments.

   dig demo.devops-demo.online +short

   You should see the NLB hostname as an alias or CNAME.
2. Verify Certificate: cert-manager should request and provision the certificate.

   kubectl get cert -w

   Wait until the STATUS becomes Ready.

   kubectl get secret nginx-tls-secret -o yaml

   You should see the TLS certificate and key encoded here.
3. Access the Application: Open your web browser and navigate to https://demo.devops-demo.online. You should see the Nginx welcome page with a valid SSL certificate.

Security Considerations

Implementing an ingress solution on EKS involves several critical security considerations:

• IAM Least Privilege: Ensure that the IAM policies attached to external-dns and cert-manager Service Accounts grant only the minimum necessary permissions. Avoid giving broad Route 53 access if specific hosted zones can be targeted.
• Network Security Groups (NSGs): The AWS Load Balancer provisioned by the Nginx Ingress Controller will have associated NSGs. Configure these NSGs to restrict inbound traffic to only necessary ports (80, 443) and trusted IP ranges if applicable.
• TLS/SSL Best Practices:
  • Always enforce HTTPS redirection (e.g., using nginx.ingress.kubernetes.io/force-ssl-redirect: "true").
  • Implement HTTP Strict Transport Security (HSTS) via Ingress annotations to prevent downgrade attacks.
  • Keep Nginx Ingress Controller and cert-manager up-to-date to benefit from security patches.
• Vulnerability Scanning: Regularly scan your container images (Nginx, cert-manager, external-dns) for vulnerabilities.
• Logging and Monitoring: Integrate Nginx Ingress Controller logs with a centralized logging solution (e.g., CloudWatch Logs, Splunk) and set up monitoring and alerting for certificate expiry, ingress errors, and application health.
• Private Keys Security: cert-manager stores private keys in Kubernetes secrets. Ensure your Kubernetes cluster itself is secure, with strong RBAC policies and encrypted etcd.

Best Practices

To ensure a robust and maintainable ingress setup, consider these best practices:

• Separate Namespaces: Deploy Nginx Ingress Controller, cert-manager, and external-dns into their own dedicated namespaces (e.g., ingress-nginx, cert-manager, external-dns). This improves organization, resource isolation, and simplifies RBAC management.
• Automate with CI/CD: Integrate the deployment of your Ingress Controller, cert-manager, external-dns, and application Ingresses into your CI/CD pipelines. This ensures consistent deployments and faster recovery from failures.
• Monitoring and Alerting: Implement comprehensive monitoring for all components. For cert-manager, monitor certificate expiry and issuance failures. For Nginx Ingress, monitor traffic, error rates, and resource utilization. For external-dns, monitor its logs for any DNS record update failures.
• Staging Environment for cert-manager: Always test new Ingress configurations and cert-manager settings with the Let's Encrypt Staging API (letsencrypt-staging ClusterIssuer) before switching to production. This avoids hitting rate limits on the production API.
• Resource Limits and Requests: Define appropriate CPU and memory limits and requests for all your deployments (Ingress Controller, cert-manager, external-dns, and applications) to prevent resource contention and ensure stable performance.
• Backup cert-manager Configuration: While certificates are ephemeral, ensure you have a backup strategy for your cert-manager configuration, especially the ClusterIssuer definitions and any custom resource definitions (CRDs).
• Use Custom Annotations for Fine-tuning: Leverage Nginx Ingress annotations (e.g., for rewrite rules, client-max-body-size, proxy buffers) to fine-tune traffic management as per your application's needs.

FAQ

Q1: Why choose Nginx Ingress Controller over AWS ALB Ingress Controller on EKS?

Both are excellent choices, but they cater to slightly different needs. Nginx Ingress Controller offers a highly performant, feature-rich, and portable ingress solution. It provides advanced traffic management features (e.g., sophisticated rewrite rules, custom Lua scripting, WebSockets proxying, fine-grained load balancing algorithms) that might not be available with the ALB Ingress Controller. It's also cloud-agnostic, making migrations easier. The ALB Ingress Controller, on the other hand, is deeply integrated with AWS services, leveraging native ALBs, WAF, and Shield, which can simplify operations for AWS-centric teams and potentially optimize costs for certain traffic patterns.

Q2: What should I do if my certificate doesn't provision or renew?

First, check the events of your Certificate resource: kubectl describe cert <your-certificate-name>. Look for any errors or warnings. Next, check the logs of the cert-manager pods in the cert-manager namespace: kubectl logs -n cert-manager -l app=cert-manager. Common issues include incorrect email in the ClusterIssuer, firewall rules blocking ACME challenge verification, or incorrect IAM permissions for Route 53 (if using DNS-01 challenge). For DNS-01, verify the TXT records are being created in Route 53 during the challenge phase.

Q3: Can I use this setup for multiple domains or wildcard certificates?

Absolutely. For multiple domains, simply add more hosts to the tls and rules sections of your Ingress resource. cert-manager will handle requesting certificates for all specified domains. For wildcard certificates (e.g., *.devops-demo.online), you must use the DNS-01 challenge type, which our setup already utilizes. Update your Ingress resource's host fields to use the wildcard (e.g., *.devops-demo.online), and cert-manager will request a wildcard certificate from Let's Encrypt. external-dns will also correctly handle the wildcard DNS record.

Conclusion

The combination of Nginx Ingress Controller, cert-manager, and external-dns on Amazon EKS provides a robust, scalable, and highly automated solution for managing ingress traffic, SSL/TLS certificates, and DNS records. By following this detailed guide, you can establish a secure and efficient entry point for your Kubernetes applications, freeing your team from manual operational tasks and allowing them to focus on delivering value.

This architecture embodies the spirit of cloud-native automation, where infrastructure intelligently responds to application deployments. As your EKS clusters grow and your application portfolio expands, this integrated ingress solution will prove to be an indispensable component, ensuring your services are always discoverable, secure, and performant.