ArgoCD ApplicationSets: Multi-Cluster GitOps Deployment Simplified [z36z]

Overview

In the rapidly evolving landscape of cloud-native development, GitOps has emerged as a paramount methodology for managing Kubernetes clusters and applications. By using Git as the single source of truth for declarative infrastructure and applications, GitOps automates deployments, enhances traceability, and fosters consistency. ArgoCD stands as a leading GitOps continuous delivery tool for Kubernetes, providing robust capabilities for declaring, synchronating, and managing applications.

However, as organizations scale their operations, they often encounter the challenge of managing applications across multiple Kubernetes clusters. Whether it's for disaster recovery, regional deployments, multi-tenancy, or environment separation (dev, staging, production), operating a fleet of clusters introduces significant complexity. Manually creating and updating ArgoCD Application resources for each cluster becomes a tedious, error-prone, and unsustainable task. This is where ArgoCD ApplicationSets enter the picture, transforming multi-cluster GitOps from a labor-intensive chore into an automated, scalable, and declarative process.

ArgoCD ApplicationSets extend ArgoCD's capabilities by enabling the programmatic creation and management of ArgoCD Application resources. Instead of defining each application instance individually, you define an ApplicationSet that acts as a factory, generating multiple Application resources based on various criteria. This abstraction is a game-changer for multi-cluster and multi-tenant environments, allowing platform teams to define application deployment patterns once and have them automatically applied across a dynamic set of clusters, or even different paths within a single Git repository. It significantly reduces operational overhead, ensures consistency across deployments, and empowers developers to self-serve application deployments within predefined guardrails.

This article will dive deep into the world of ArgoCD ApplicationSets, exploring their architecture, demonstrating practical implementation steps with real-world examples, and discussing critical security considerations and best practices for leveraging this powerful GitOps tool effectively.

Prerequisites

Before we embark on our journey to master ArgoCD ApplicationSets, ensure you have the following prerequisites in place:

• Kubernetes Clusters: At least two Kubernetes clusters. For demonstration purposes, we'll refer to them as a 'control plane' cluster (where ArgoCD runs) and one or more 'target' clusters (where applications will be deployed). You can use local clusters like Kind or Minikube, or managed services like AWS EKS, GCP GKE, or Azure AKS.
• kubectl: Configured to interact with all your Kubernetes clusters. Ensure you have appropriate context names for each, e.g., kind-control-plane, kind-target-cluster-a, kind-target-cluster-b.
• ArgoCD Installation: ArgoCD must be installed on your 'control plane' Kubernetes cluster. You can follow the official ArgoCD documentation for installation. A common method is via Helm or direct manifest application.
• ArgoCD ApplicationSet Controller: The ApplicationSet controller is typically installed alongside ArgoCD, especially in recent versions. If not, ensure it's installed in the same namespace as ArgoCD. You can verify its presence by checking for the argocd-applicationset-controller deployment.
• Git Repository: A Git repository (e.g., GitHub, GitLab, Bitbucket) to store your Kubernetes manifests and ApplicationSet definitions. This repository will serve as the source of truth for your GitOps deployments.
• Basic ArgoCD Knowledge: Familiarity with basic ArgoCD concepts like Applications, Projects, and Sync policies.
• Git CLI: Installed and configured on your local machine.
• ArgoCD CLI: Installed and configured. You'll use it to manage ArgoCD instances and add clusters.

To install ArgoCD (if you haven't already), you can use the following commands:

# Create the namespace
kubectl create namespace argocd

# Apply the ArgoCD installation manifest (using a stable version)
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.7/manifests/install.yaml

# Wait for ArgoCD pods to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=300s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-repo-server -n argocd --timeout=300s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-applicationset-controller -n argocd --timeout=300s

# Get the initial admin password (replace with your ArgoCD server pod name)
ARGOCD_SERVER_POD=$(kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o jsonpath='{.items[0].metadata.name}')
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

# Port-forward to access the UI (optional, for browser access)
# kubectl port-forward service/argocd-server -n argocd 8080:443

Once ArgoCD is running, log in with the CLI:

argocd login localhost:8080 # If port-forwarding
# Or use the external IP/hostname if exposed differently
argocd login <your-argocd-hostname> --username admin --password <initial-admin-password>

Step-by-step implementation

A. Setting up ArgoCD and Adding Target Clusters

Assuming ArgoCD is installed on your control plane cluster, the next crucial step for multi-cluster deployments is registering your target Kubernetes clusters with ArgoCD. ArgoCD needs credentials to interact with these clusters and deploy applications.

First, ensure your `kubectl` contexts are correctly set up for your target clusters. For instance, if you have two Kind clusters named `cluster-a` and `cluster-b`:

kubectl config get-contexts

You might see output like:

CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
*         kind-control-plane        kind-control-plane        kind-control-plane
          kind-cluster-a            kind-cluster-a            kind-cluster-a
          kind-cluster-b            kind-cluster-b            kind-cluster-b

Now, add your target clusters to ArgoCD. The `argocd cluster add` command simplifies this by using your current `kubectl` context credentials.

# Add cluster-a
kubectl config use-context kind-cluster-a
argocd cluster add kind-cluster-a --label environment=dev --label region=us-east-1

# Add cluster-b
kubectl config use-context kind-cluster-b
argocd cluster add kind-cluster-b --label environment=prod --label region=us-west-2

# Switch back to the control plane context
kubectl config use-context kind-control-plane

The `--label` flag is important here. It allows you to tag your registered clusters, which is invaluable when using ApplicationSet's `ClusterGenerator` to target specific clusters based on these labels.

Verify that your clusters have been added successfully:

argocd cluster list

Expected output:

SERVER                          NAME               VERSION  STATUS      MESSAGE                                        LABELS
https://kubernetes.default.svc  in-cluster         v1.27.3  Successful  Cluster is running                             <none>
https://<ip-a>:<port-a>          kind-cluster-a     v1.27.3  Successful  Cluster is running                             environment=dev,region=us-east-1
https://<ip-b>:<port-b>          kind-cluster-b     v1.27.3  Successful  Cluster is running                             environment=prod,region=us-west-2

B. Git Repository Structure

A well-organized Git repository is fundamental for effective GitOps. For ApplicationSets, a common and recommended structure separates ApplicationSet definitions from the actual application manifests.

├── applicationsets/
│   ├── list-nginx-app.yaml
│   ├── cluster-guestbook-app.yaml
│   └── git-tenant-apps.yaml
├── apps/
│   ├── nginx-app/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   ├── guestbook/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   └── service.yaml
│   │   ├── overlays/
│   │   │   ├── dev/
│   │   │   │   └── kustomization.yaml
│   │   │   └── prod/
│   │   │       └── kustomization.yaml
│   ├── tenant-a-app/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── tenant-b-app/
│       ├── deployment.yaml
│       └── service.yaml

In this structure:

• `applicationsets/`: Contains all your ApplicationSet definitions.
• `apps/`: Contains the actual Kubernetes manifests for your applications. These can be raw YAML, Helm charts, or Kustomize bases/overlays.

Let's create a sample `nginx-app` for our first example:

apps/nginx-app/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.3-alpine
        ports:
        - containerPort: 80

apps/nginx-app/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

C. Example 1: List Generator (Simple Multi-Cluster Deployment)

The `ListGenerator` is the simplest ApplicationSet generator. It allows you to explicitly define a static list of parameters for generating applications. This is useful when you have a fixed, small number of target clusters or environments, and you want fine-grained control over each instance.

Scenario: Deploy the `nginx-app` to `kind-cluster-a` (dev) and `kind-cluster-b` (prod), but with a different replica count for each.

applicationsets/list-nginx-app.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: nginx-multi-cluster
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: kind-cluster-a
        url: https://<ip-a>:<port-a> # Replace with actual cluster-a API server URL
        name: dev-nginx
        replicas: "1"
        namespace: nginx-dev
      - cluster: kind-cluster-b
        url: https://<ip-b>:<port-b> # Replace with actual cluster-b API server URL
        name: prod-nginx
        replicas: "3"
        namespace: nginx-prod
  template:
    metadata:
      name: '{{.name}}' # Uses the 'name' parameter from the list element
      labels:
        app.kubernetes.io/part-of: nginx-fleet
        app.kubernetes.io/environment: '{{index .cluster.labels "environment"}}' # Access cluster labels
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/your-gitops-repo.git # Replace with your Git repo URL
        targetRevision: HEAD
        path: apps/nginx-app
        kustomize:
          patches:
            - target:
                group: apps
                version: v1
                kind: Deployment
                name: nginx-deployment
              patch: |
                - op: replace
                  path: /spec/replicas
                  value: {{.replicas | int}} # Use the 'replicas' parameter
      destination:
        server: '{{.url}}' # Uses the 'url' parameter
        namespace: '{{.namespace}}' # Uses the 'namespace' parameter
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Important: Replace `https://<ip-a>:<port-a>` and `https://<ip-b>:<port-b>` with the actual API server URLs of your `kind-cluster-a` and `kind-cluster-b` respectively. You can find these by running `kubectl config view -o jsonpath='{.clusters[?(@.name=="kind-cluster-a")].cluster.server}'`.

In this example:

• The `generators` section uses a `list` generator.
• Each `element` in the list defines parameters like `cluster`, `url`, `name`, `replicas`, and `namespace`.
• The `template` section defines how the ArgoCD Application resource will be structured.
• We use Go templating (`{{.name}}`, `{{.url}}`, `{{.replicas}}`, `{{.namespace}}`) to inject values from the list elements into the Application definition.
• The `kustomize.patches` field demonstrates how to dynamically modify application manifests (e.g., setting replica count) using parameters from the generator.
• {{index .cluster.labels "environment"}} shows how to access labels associated with the target cluster.

To deploy this:

git add .
git commit -m "Add list generator for nginx app"
git push origin main

# ArgoCD will automatically detect the ApplicationSet in the Git repo.
# You can also apply it directly if not using GitOps for the ApplicationSet itself:
# kubectl apply -n argocd -f applicationsets/list-nginx-app.yaml

After a short while, ArgoCD will create two `Application` resources: `dev-nginx` targeting `kind-cluster-a` and `prod-nginx` targeting `kind-cluster-b`. You can verify this:

argocd app list -A | grep nginx
# Or view in the ArgoCD UI

And check the deployments in the target clusters:

kubectl get deploy -n nginx-dev --context kind-cluster-a
kubectl get deploy -n nginx-prod --context kind-cluster-b

D. Example 2: Cluster Generator (Dynamic Multi-Cluster Deployment)

The `ClusterGenerator` is ideal for deploying applications to a dynamic set of clusters registered with ArgoCD. It can select clusters based on labels, making it powerful for targeting all clusters in a specific environment or region without explicitly listing them.

Scenario: Deploy a `guestbook` application to all clusters labeled `environment=dev`. We'll use a simple Kustomize base for the guestbook application.

First, let's create a simple guestbook app with Kustomize:

apps/guestbook/base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: guestbook-ui
  labels:
    app: guestbook-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: guestbook-ui
  template:
    metadata:
      labels:
        app: guestbook-ui
    spec:
      containers:
      - name: guestbook-ui
        image: gcr.io/google-samples/gb-frontend:v4
        ports:
        - containerPort: 80

apps/guestbook/base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: guestbook-ui
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: guestbook-ui
  type: ClusterIP

apps/guestbook/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

Now, the ApplicationSet definition:

applicationsets/cluster-guestbook-app.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-dev-clusters
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: dev # Target clusters with this label
  template:
    metadata:
      name: 'guestbook-{{.name | replace "kind-" "" | replace "-" ""}}' # Generate unique names, e.g., guestbook-clustera
      labels:
        app.kubernetes.io/part-of: guestbook-fleet
        app.kubernetes.io/environment: '{{.metadata.labels.environment}}' # Access cluster labels
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/your-gitops-repo.git # Replace with your Git repo URL
        targetRevision: HEAD
        path: apps/guestbook/base # Path to the Kustomize base
      destination:
        server: '{{.server}}' # The URL of the target cluster
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

In this example:

• The `generators` section uses a `clusters` generator.
• The `selector.matchLabels` filters for clusters with the label `environment: dev`. (Recall we added `environment=dev` to `kind-cluster-a`).
• The `template` uses parameters like `{{.name}}` (the cluster name) and `{{.server}}` (the cluster API server URL) provided by the generator.
• We're using a simple Go template function `replace` to create more readable application names.

Deploy this ApplicationSet:

git add .
git commit -m "Add cluster generator for guestbook app"
git push origin main

ArgoCD will now create an `Application` for `kind-cluster-a` (because it has the `environment=dev` label). If you were to add another cluster with the same label, a new `Application` would be automatically generated for it.

argocd app list -A | grep guestbook
kubectl get deploy -n guestbook --context kind-cluster-a

E. Example 3: Git Generator (Directory-based Multi-Tenant/Multi-Environment)

The `GitGenerator` is incredibly powerful for scenarios where your Git repository structure dictates different application deployments. It can iterate over files or directories within a Git repository, generating parameters for each matched path. This is often used for multi-tenancy, environment-specific configurations, or deploying multiple independent applications from a single repository.

Scenario: Deploy different applications (or different configurations of the same app) based on subdirectories in the Git repository. Let's imagine `tenant-a-app` and `tenant-b-app` are distinct applications.

First, create some sample application manifests:

apps/tenant-a-app/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-a-app
  labels:
    app: tenant-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tenant-a
  template:
    metadata:
      labels:
        app: tenant-a
    spec:
      containers:
      - name: tenant-a-container
        image: busybox:1.36
        command: ["sh", "-c", "echo 'Hello from Tenant A' && sleep 3600"]

apps/tenant-b-app/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-b-app
  labels:
    app: tenant-b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tenant-b
  template:
    metadata:
      labels:
        app: tenant-b
    spec:
      containers:
      - name: tenant-b-container
        image: busybox:1.36
        command: ["sh", "-c", "echo 'Hello from Tenant B' && sleep 3600"]

Now, the ApplicationSet definition:

applicationsets/git-tenant-apps.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenant-based-apps
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/your-org/your-gitops-repo.git # Replace with your Git repo URL
      revision: HEAD
      directories:
      - path: apps/tenant-*-app # Match directories like 'tenant-a-app', 'tenant-b-app'
  template:
    metadata:
      name: '{{.path.basename}}-{{index .cluster.labels "environment"}}' # e.g., tenant-a-app-dev
      labels:
        app.kubernetes.io/part-of: multi-tenant-fleet
        app.kubernetes.io/tenant: '{{.path.basename | replace "tenant-" "" | replace "-app" ""}}' # Extract tenant name
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/your-gitops-repo.git
        targetRevision: HEAD
        path: '{{.path}}' # The path matched by the generator
      destination:
        server: '{{.server}}' # Assuming we want to deploy this to a specific cluster
        namespace: '{{.path.basename}}' # Namespace derived from the directory name
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

In this example:

• The `generators` section uses a `git` generator.
• `repoURL` and `revision` point to your Git repository.
• `directories` specifies a pattern `apps/tenant-*-app`. For each directory matching this pattern, the generator produces a set of parameters, including `path` (the full directory path) and `path.basename` (just the directory name).
• The `template` then uses these parameters to define `Application` resources. For instance, `{{.path}}` directly refers to the matched directory, allowing ArgoCD to sync manifests from that specific location.
• Note: For simplicity, this example assumes a single target cluster (e.g., `kind-cluster-a`) and uses its server URL. In a real-world scenario, you'd likely combine `GitGenerator` with `ClusterGenerator` using a `MatrixGenerator` to deploy each tenant app to multiple clusters.

Deploy this ApplicationSet:

git add .
git commit -m "Add git generator for tenant apps"
git push origin main

ArgoCD will create `Application` resources for `tenant-a-app` and `tenant-b-app`, each syncing from its respective directory in the Git repository to the target cluster(s).

argocd app list -A | grep tenant
kubectl get deploy -n tenant-a-app --context kind-cluster-a
kubectl get deploy -n tenant-b-app --context kind-cluster-a

F. Combining Generators (Matrix/Merge)

While we've explored individual generators, the true power of ApplicationSets often comes from combining them using `MatrixGenerator` or `MergeGenerator`.

• MatrixGenerator: Takes two or more generators and produces a Cartesian product of their outputs. For instance, combine a `ClusterGenerator` (listing clusters) with a `GitGenerator` (listing application paths) to deploy every application to every matching cluster.
• MergeGenerator: Merges the output of multiple generators. If generators produce parameters with the same key, the later generator's value overwrites the earlier one. This is useful for defining defaults that can be overridden.

For example, a `MatrixGenerator` could look like this (conceptual):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: matrix-example
  namespace: argocd
spec:
  generators:
  - matrix:
      generators:
      - clusters:
          selector:
            matchLabels:
              environment: dev