ArgoCD ApplicationSets: Multi-Cluster GitOps Deployment Simplified

Simplify multi-cluster GitOps using ArgoCD ApplicationSets. Deploy & manage apps consistently across all environments with ease.

Overview

In the rapidly evolving landscape of cloud-native development, managing Kubernetes applications across multiple clusters has become a paramount challenge for organizations. Whether it's for geographical redundancy, environment separation (dev, staging, production), multi-tenancy, or edge deployments, the need for a robust, scalable, and automated solution is undeniable. This is precisely where ArgoCD, the declarative, GitOps continuous delivery tool for Kubernetes, and its powerful extension, ApplicationSets, step in.

ArgoCD champions the GitOps paradigm, where the desired state of your applications and infrastructure is defined declaratively in Git. It continuously monitors your Git repositories, detecting any divergence between the declared state and the actual state in your Kubernetes clusters, and automatically reconciles them. While ArgoCD excels at managing applications within a single cluster or a handful of predefined clusters, scaling this model to dozens or even hundreds of clusters can quickly become unwieldy. Manually creating and managing an Application resource for each application on each target cluster is prone to error, time-consuming, and difficult to maintain.

ArgoCD ApplicationSets address this exact pain point. They provide a mechanism to dynamically generate ArgoCD Application resources based on various sources, allowing for the deployment of a single application definition across multiple clusters or multiple namespaces within a cluster, with cluster-specific customizations. This significantly reduces the operational overhead, enhances consistency, and enables true GitOps at scale. ApplicationSets act as a higher-level abstraction, enabling you to define a pattern for application deployment, and letting the ApplicationSet controller instantiate individual Application resources for each target based on that pattern and dynamic inputs.

The core power of ApplicationSets lies in its diverse set of "generators." These generators are responsible for producing parameters that are then used to template the actual Application resources. Key generators include:

List Generator: Provides a static list of parameters, useful for a small, fixed set of targets.
Cluster Generator: Discovers clusters registered with ArgoCD and uses their names or labels as parameters. Ideal for dynamic multi-cluster deployments.
Git Generator: Reads parameters from files within a Git repository, allowing for Git-driven parameterization.
Matrix Generator: Combines the output of two other generators, creating a Cartesian product of parameters.
Merge Generator: Merges the output of multiple generators.
SCM Provider Generator: Discovers repositories within a Git SCM provider (GitHub, GitLab, Bitbucket) and uses their names as parameters.
Pull Request Generator: Creates applications for each open pull request in a Git repository.

By leveraging these generators, organizations can implement sophisticated deployment strategies, such as deploying a common microservice across all production clusters, rolling out new features to specific regions, or managing tenant-specific applications in a multi-tenant environment, all from a single source of truth in Git.

Prerequisites

Before we dive into the practical implementation of ArgoCD ApplicationSets, ensure you have the following prerequisites in place:

Kubectl: The Kubernetes command-line tool, installed and configured to interact with your Kubernetes clusters.
```
kubectl version --client
```
Helm: The Kubernetes package manager, used for installing ArgoCD and the ApplicationSet controller.
```
helm version
```
Management Kubernetes Cluster: A Kubernetes cluster where ArgoCD and the ApplicationSet controller will be installed. This can be a local cluster (e.g., Kind, Minikube) or a cloud-managed cluster (e.g., AWS EKS, Google GKE, Azure AKS). For this article, we'll assume an EKS cluster named argocd-mgmt-cluster is used, and its kubeconfig context is set.
```
aws eks update-kubeconfig --name argocd-mgmt-cluster --region us-east-1
```
Target Kubernetes Clusters (Minimum Two): At least two additional Kubernetes clusters where your applications will be deployed by ArgoCD. These can also be Kind, EKS, GKE, or AKS clusters. For our examples, we'll use two EKS clusters: dev-cluster-01 and prod-cluster-01, both in us-east-1. Ensure their kubeconfig files are accessible.
```
aws eks update-kubeconfig --name dev-cluster-01 --region us-east-1
aws eks update-kubeconfig --name prod-cluster-01 --region us-east-1
```
Git Repository: A Git repository (GitHub, GitLab, Bitbucket, etc.) to store your application manifests and ApplicationSet definitions. We'll use a public GitHub repository for demonstration purposes. Example: https://github.com/tech-news-venture/argocd-appset-demo.git.
ArgoCD Installation: ArgoCD must be installed on your management cluster. If not, follow the official documentation or use the commands below.
ArgoCD CLI: The ArgoCD command-line interface tool, installed and authenticated against your ArgoCD instance.
```
argocd version --client
```

Step-by-step Implementation

Step 1: Install ArgoCD and the ApplicationSet Controller

First, let's ensure ArgoCD and its ApplicationSet controller are running on our management cluster. We'll install them into a dedicated namespace, argocd.

1.1. Create the ArgoCD namespace:

kubectl create namespace argocd

1.2. Install ArgoCD using Helm:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd -n argocd --version 5.51.6 \
  --set server.service.type=LoadBalancer \
  --set server.ingress.enabled=false \
  --set controller.replicas=1 \
  --set redis.enabled=true \
  --set applicationSet.enabled=true

Note the --set applicationSet.enabled=true flag, which ensures the ApplicationSet controller is installed as part of the ArgoCD deployment. If you already have ArgoCD installed, you might just need to upgrade your Helm chart or manually install the ApplicationSet controller.

1.3. Access ArgoCD UI:

Once installed, get the ArgoCD server LoadBalancer IP or hostname:

kubectl get svc argocd-server -n argocd
# Example Output:
# NAME            TYPE           CLUSTER-IP     EXTERNAL-IP                                                                    PORT(S)                      AGE
# argocd-server   LoadBalancer   10.100.20.30   a1234567890abcdef1234567890abcdef-123456789.us-east-1.elb.amazonaws.com   80:30080/TCP,443:30443/TCP   5m

The initial password for the admin user is the name of the ArgoCD server pod:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
# Copy the outputted password

argocd login <ARGOCD_SERVER_EXTERNAL_IP> --username admin --password <YOUR_PASSWORD> --insecure

Step 2: Register Target Clusters with ArgoCD

ArgoCD needs to know about the clusters where it will deploy applications. We'll register our dev-cluster-01 and prod-cluster-01. Ensure your kubeconfig has contexts for both.

2.1. Add dev-cluster-01:

argocd cluster add dev-cluster-01 --kubeconfig ~/.kube/config --context arn:aws:eks:us-east-1:123456789012:cluster/dev-cluster-01 --label env=dev --label region=us-east-1
# Output:
# INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system" 
# INFO[0000] ClusterRole "argocd-manager-role" created    
# INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created 
# Cluster 'https://<dev-cluster-01-api-server-endpoint>' added

2.2. Add prod-cluster-01:

argocd cluster add prod-cluster-01 --kubeconfig ~/.kube/config --context arn:aws:eks:us-east-1:123456789012:cluster/prod-cluster-01 --label env=prod --label region=us-east-1
# Output:
# INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system" 
# INFO[0000] ClusterRole "argocd-manager-role" created    
# INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created 
# Cluster 'https://<prod-cluster-01-api-server-endpoint>' added

Notice the --label flags. These labels are crucial for the Cluster Generator to dynamically select target clusters based on criteria like environment or region.

2.3. Verify registered clusters:

argocd cluster list
# Output (truncated for brevity):
# SERVER                                      NAME             VERSION  STATUS      MESSAGE  REFRESH  LABELS
# https://<argocd-mgmt-cluster-endpoint>      in-cluster       1.27.1   Successful           5m
# https://<dev-cluster-01-api-server-endpoint>  dev-cluster-01   1.27.1   Successful           5m       env=dev,region=us-east-1
# https://<prod-cluster-01-api-server-endpoint> prod-cluster-01  1.27.1   Successful           5m       env=prod,region=us-east-1

Step 3: Prepare Application Manifests in Git

We'll create a simple Nginx deployment and service. For demonstration, let's put them in a Git repository. Imagine a repository structure like this:

argocd-appset-demo/
├── applications/
│   └── nginx-app/
│       ├── base/
│       │   ├── deployment.yaml
│       │   └── service.yaml
│       └── kustomization.yaml
└── appsets/
    ├── list-generator-nginx.yaml
    └── cluster-generator-nginx.yaml

3.1. applications/nginx-app/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

3.2. applications/nginx-app/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

3.3. applications/nginx-app/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

Push these files to your Git repository (e.g., https://github.com/tech-news-venture/argocd-appset-demo.git).

Step 4: Create an ApplicationSet using the List Generator

The List Generator is straightforward for a fixed set of targets where you might need to specify parameters explicitly for each. We'll deploy our Nginx application to both dev-cluster-01 and prod-cluster-01, but with different replica counts and namespaces.

4.1. appsets/list-generator-nginx.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: nginx-list-appset
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: dev-cluster-01
        url: https://<dev-cluster-01-api-server-endpoint> # Replace with actual API server URL
        namespace: dev-apps
        replicas: "1"
      - cluster: prod-cluster-01
        url: https://<prod-cluster-01-api-server-endpoint> # Replace with actual API server URL
        namespace: production-apps
        replicas: "3"
  template:
    metadata:
      name: '{{cluster}}-nginx' # Application name will be like dev-cluster-01-nginx
      labels:
        app.kubernetes.io/part-of: nginx-list-appset
    spec:
      project: default # ArgoCD project
      source:
        repoURL: https://github.com/tech-news-venture/argocd-appset-demo.git
        targetRevision: HEAD
        path: applications/nginx-app/base # Path to our Kustomize base
      destination:
        server: '{{url}}' # Target cluster API server URL
        namespace: '{{namespace}}' # Target namespace
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true # Ensure the namespace is created if it doesn't exist
        # Kustomize specific overrides can be done here for more complex scenarios
        # For this simple example, we'll use a patch for replicas directly
      # Kustomize patch to override replicas based on the 'replicas' parameter
      # Note: For more complex Kustomize overlays, you'd typically have different paths
      # or use `kustomize` block in source with `commonAnnotations` etc.
      # This example uses a simple patch for demonstration.
      # A better approach for Kustomize would be to have different Kustomize overlays
      # in Git and point to them based on the cluster.
      # For simplicity and demonstrating templating, let's inject a patch.
      # In a real-world scenario, you might point to different kustomize overlay paths.
      # As a workaround for simple templating, we can use a Kustomize patch.
      # This is not ideal for complex changes, but works for simple parameter injection.
      # For a realistic Kustomize use case, you'd likely have:
      # path: applications/nginx-app/overlays/{{cluster}}
      # And each overlay would have its own kustomization.yaml with patches.
      # For direct templating as requested, we need to adjust the deployment directly.
      # Let's refine the template to use the replicas parameter directly in the manifest source
      # if we were not using Kustomize. Since we are using Kustomize, we'd typically have
      # different overlay paths. Let's simplify to demonstrate templating directly on the app.
      # The `path` points to base, and we need to patch it.
      # A more robust approach for Kustomize would be:
      # path: applications/nginx-app/overlays/{{cluster}}
      # where overlays/dev-cluster-01/kustomization.yaml would patch replicas: 1
      # and overlays/prod-cluster-01/kustomization.yaml would patch replicas: 3

      # Let's adjust the example to use a more common Kustomize overlay pattern
      # where the 'path' itself changes based on the cluster.
      # This requires modifying the Git repository structure slightly.
      # For this example, let's assume `path` is fixed to `base` and we are applying
      # a Kustomize patch from the ApplicationSet itself, which is less common.
      # A better way is to have `applications/nginx-app/overlays/dev` and `applications/nginx-app/overlays/prod`.
      # Let's modify the Git structure slightly to support this, as it's a more realistic Kustomize usage.

      # NEW GIT STRUCTURE:
      # argocd-appset-demo/
      # ├── applications/
      # │   └── nginx-app/
      # │       ├── base/
      # │       │   ├── deployment.yaml
      # │       │   └── service.yaml
      # │       ├── overlays/
      # │       │   ├── dev/
      # │       │   │   └── kustomization.yaml
      # │       │   └── prod/
      # │       │       └── kustomization.yaml
      # └── appsets/
      #     ├── list-generator-nginx.yaml
      #     └── cluster-generator-nginx.yaml

      # applications/nginx-app/overlays/dev/kustomization.yaml
      # apiVersion: kustomize.config.k8s.io/v1beta1
      # kind: Kustomization
      # resources:
      #   - ../../base
      # patches:
      #   - target:
      #       kind: Deployment
      #       name: nginx-deployment
      #     patch: |-
      #       - op: replace
      #         path: /spec/replicas
      #         value: 1

      # applications/nginx-app/overlays/prod/kustomization.yaml
      # apiVersion: kustomize.config.k8s.io/v1beta1
      # kind: Kustomization
      # resources:
      #   - ../../base
      # patches:
      #   - target:
      #       kind: Deployment
      #       name: nginx-deployment
      #     patch: |-
      #       - op: replace
      #         path: /spec/replicas
      #         value: 3

      # With this structure, the ApplicationSet template would look like this:
      source:
        repoURL: https://github.com/tech-news-venture/argocd-appset-demo.git
        targetRevision: HEAD
        path: applications/nginx-app/overlays/{{namespace | replace "dev-apps" "dev" | replace "production-apps" "prod"}} # Dynamically select overlay path
      # The above path uses templating to map 'dev-apps' to 'dev' and 'production-apps' to 'prod'
      # which corresponds to the overlay directories. This is a common and robust pattern.
      # Let's stick to this more realistic Kustomize overlay approach.
      # Ensure your Git repo has these overlay directories.
      # For the sake of this article, let's assume the Git repo reflects this structure.

Correction/Refinement: The initial thought process for handling replicas with Kustomize was a bit convoluted. A best practice for Kustomize-based multi-cluster deployments is to have distinct overlay directories (e.g., overlays/dev, overlays/prod) in your Git repository. The ApplicationSet then dynamically points to the correct overlay path based on the target cluster. The YAML above has been updated to reflect this more realistic and robust Kustomize integration pattern, using Jinja2-like templating within the path field to select the appropriate overlay based on the namespace parameter from the generator. Make sure your Git repository tech-news-venture/argocd-appset-demo is updated with the overlays/dev and overlays/prod directories as described.

4.2. Apply the ApplicationSet:

kubectl apply -f appsets/list-generator-nginx.yaml -n argocd

4.3. Verify Applications:

argocd app list -o wide -n argocd | grep nginx-list-appset
# Output will show two applications:
# dev-cluster-01-nginx   default  nginx-list-appset  https://github.com/tech-news-venture/argocd-appset-demo.git  applications/nginx-app/overlays/dev  https://<dev-cluster-01-api-server-endpoint>  dev-apps  Synced    Healthy   ...
# prod-cluster-01-nginx  default  nginx-list-appset  https://github.com/tech-news-venture/argocd-appset-demo.git  applications/nginx-app/overlays/prod  https://<prod-cluster-01-api-server-endpoint> production-apps Synced    Healthy   ...

You should see two new applications created in ArgoCD, each targeting a different cluster and namespace, and configured with the correct replica count based on their respective Kustomize overlays.

Step 5: Create an ApplicationSet using the Cluster Generator

The Cluster Generator is powerful for dynamically discovering clusters registered with ArgoCD and using their properties (name, labels) to template applications. We'll use it to deploy an application to all clusters labeled with env=dev.

5.1. appsets/cluster-generator-nginx.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: nginx-cluster-appset
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          env: dev # Target only clusters labeled with env: dev
  template:
    metadata:
      name: '{{name}}-nginx-dynamic' # App name like dev-cluster-01-nginx-dynamic
      labels:
        app.kubernetes.io/part-of: nginx-cluster-appset
    spec:
      project: default
      source:
        repoURL: https://github.com/tech-news-venture/argocd-appset-demo.git
        targetRevision: HEAD
        path: applications/nginx-app/overlays/dev # Always use the 'dev' overlay for dev clusters
      destination:
        server: '{{server}}' # Templated from cluster.server
        namespace: dynamic-dev-apps
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

In this example, {{name}} and {{server}} are templated directly from the properties of the discovered clusters. We are specifically targeting clusters with the label env: dev, which means only dev-cluster-01 will be targeted in our current setup.

5.2. Apply the ApplicationSet:

kubectl apply -f appsets/cluster-generator-nginx.yaml -n argocd

5.3. Verify Applications:

argocd app list -o wide -n argocd | grep nginx-cluster-appset
# Output will show one application:
# dev-cluster-01-nginx-dynamic default  nginx-cluster-appset  https://github.com/tech-news-venture/argocd-appset-demo.git  applications/nginx-app/overlays/dev  https://<dev-cluster-01-api-server-endpoint>  dynamic-dev-apps  Synced    Healthy   ...

Now, if you were to add another cluster with the label env=dev to ArgoCD, the ApplicationSet controller would automatically create a new ArgoCD Application for it, deploying the Nginx application to that new cluster without any manual intervention.

Security Considerations

Deploying applications across multiple clusters using ApplicationSets introduces several critical security considerations that must be addressed:

ArgoCD RBAC: ArgoCD itself has a robust RBAC system. Ensure that users and service accounts interacting with ArgoCD and ApplicationSets have the principle of least privilege applied. For instance, restrict who can create or modify ApplicationSet resources, as these can control deployments across your entire fleet of clusters.
Cluster Credentials: The credentials used by ArgoCD to access target clusters are highly sensitive. These are stored as Kubernetes Secrets within the ArgoCD namespace. Implement strong access controls for these secrets, and ensure they are encrypted at rest and in transit. Consider using external secret management solutions (e.g., HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) integrated with ArgoCD for enhanced security.
Git Repository Security: Your Git repository is the single source of truth. Protect it with strong access controls, multi-factor authentication, and enforce signed commits to ensure integrity and authenticity of changes. Any compromise of your Git repository could lead to widespread unauthorized deployments.
Network Access Control: Ensure that the ArgoCD management cluster has appropriate network access to the API servers of all target clusters. This typically involves configuring security groups, network ACLs, or VPC peering. Restrict this access to only the necessary ports and protocols.
ApplicationSet Permissions: The ApplicationSet controller requires specific RBAC permissions to create, update, and delete Application resources within the ArgoCD namespace. Review and restrict these permissions to only what is necessary for its operation.
Supply Chain Security: Be mindful of the images and manifests you're deploying. Use trusted image registries, scan images for vulnerabilities, and verify the integrity of your application manifests before they are committed to Git. Integrating tools like Sigstore for signing and verifying images and attestations can further enhance security.
Secrets Management for Applications: ApplicationSets themselves do not directly manage application-level secrets. For secrets required by your deployed applications (e.g., database credentials, API keys), use dedicated Kubernetes secret management solutions like Sealed Secrets, External Secrets Operator, or cloud-provider specific secret injection mechanisms. Do not commit sensitive data directly into your Git repository.

Best Practices

To maximize the benefits of ArgoCD ApplicationSets and maintain a robust GitOps workflow, consider

ArgoCD ApplicationSets: Multi-Cluster GitOps Deployment Simplified