Platform Engineering: Building Internal Developer Platforms with Backs

Design and implement an Internal Developer Platform using Backstage for developer portal, Crossplane for infrastructure abstraction, and ArgoCD for delivery.

Overview: The Imperative of Platform Engineering

In today's fast-paced digital landscape, engineering teams face increasing pressure to deliver high-quality software rapidly and reliably. This demand often clashes with the inherent complexities of modern cloud-native architectures, fragmented toolchains, and the cognitive load placed on individual developers. Enter Platform Engineering – a discipline focused on building and maintaining the foundational infrastructure, tools, and processes that empower product development teams to operate with greater autonomy and velocity. Platform Engineering aims to create an "Internal Developer Platform" (IDP) – a curated collection of tools and services that abstract away underlying infrastructure complexities, offering a seamless, self-service experience for developers. An effective IDP standardizes best practices, automates repetitive tasks, and ensures consistency across the organization, freeing developers to focus on writing application code and delivering business value. This article delves into how two powerful open-source projects, Backstage and Crossplane, can be synergistically combined to construct a robust and highly effective IDP. Backstage, originally developed by Spotify, serves as the developer portal – a single pane of glass for all developer needs, encompassing a service catalog, documentation, and scaffolding tools. Crossplane, a Cloud Native Computing Foundation (CNCF) project, extends the Kubernetes API to manage and provision infrastructure resources across various cloud providers, on-premises environments, and SaaS offerings. Together, Backstage provides the intuitive user experience and self-service capabilities, while Crossplane acts as the declarative, Kubernetes-native control plane for infrastructure provisioning, making the IDP truly self-sufficient and infrastructure-agnostic.

Prerequisites: Laying the Foundation

Before embarking on building our IDP, ensure you have the following components and knowledge in place. A solid understanding of Kubernetes concepts, Git, and YAML is crucial.

General Requirements:

Kubernetes Cluster: A running Kubernetes cluster (e.g., AWS EKS, Google GKE, Azure AKS, or a local setup like Minikube/k3s for development).
kubectl: Configured to interact with your Kubernetes cluster.
Helm: The Kubernetes package manager, installed on your local machine.
Git: Version control system installed.
Docker Desktop: For local containerization needs, especially if running Backstage locally.

Backstage Specifics:

Node.js: LTS version (e.g., 18.x or 20.x) installed.
Yarn: Package manager for Node.js, installed globally (npm install -g yarn).

Crossplane Specifics (for AWS Example):

AWS Account: With programmatic access (IAM user or role).
AWS CLI: Installed and configured on your machine, primarily for creating IAM roles/users and verifying resources.
IAM Role/User for Crossplane: An IAM entity with sufficient permissions to manage the desired AWS resources. For demonstration purposes, AdministratorAccess can be used, but for production, adhere strictly to the principle of least privilege.

Detailed Steps: Building Your IDP with Backstage and Crossplane

This section guides you through the process of setting up Backstage, deploying Crossplane, configuring its AWS provider, defining custom resource definitions, and finally, integrating them for a seamless developer experience.

Step 1: Setting up Backstage

Backstage provides an excellent command-line tool to bootstrap a new application.

Initialize Backstage App:

Open your terminal and run the following command. Replace `my-backstage-app` with your desired project name.

npx @backstage/create-app

This command will prompt you for a few details. Accept the defaults or provide your own. Once complete, navigate into your new application directory:

cd my-backstage-app

Install the project dependencies:

yarn install

And then start the Backstage development server:

yarn dev

Backstage will now be running locally, typically accessible at `http://localhost:3000`. Take a moment to explore the default setup, including the Service Catalog, TechDocs, and Scaffolder.

Basic Service Catalog Entry:

To demonstrate Backstage's core functionality, let's add a simple service to its catalog. Create a new file `packages/backend/catalog-info.yaml` (you might place it in a `catalog` folder within `packages/backend` or `catalog-info.yaml` in the root of your app for simplicity, then adjust the path in `app-config.yaml`).

# packages/backend/catalog/my-first-service.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: my-first-service
  description: A sample service managed by the Backstage catalog.
  annotations:
    github.com/project-slug: tech-news-venture/my-first-service
spec:
  type: service
  lifecycle: experimental
  owner: sujay.singh@technewsventure.com
  system: my-platform
  providesApis:
    - my-first-service-api

Now, instruct Backstage to ingest this catalog file. Open `app-config.yaml` (usually in the root of your Backstage app) and add or modify the `catalog.locations` section:

# app-config.yaml
catalog:
  locations:
    - type: file
      target: packages/backend/catalog/my-first-service.yaml # Adjust path if different
    # Add other locations for GitHub, GitLab, etc.
    # - type: url
    #   target: https://github.com/tech-news-venture/some-repo/blob/main/catalog-info.yaml
    #   rules:
    #     - allow: [Component, API, System, Resource]

Restart `yarn dev` if it was running, or simply refresh Backstage in your browser. You should now see "my-first-service" listed in the Service Catalog. This demonstrates how Backstage acts as the central registry for all your software components and resources.

Step 2: Deploying Crossplane to Kubernetes

Crossplane runs as a set of controllers within your Kubernetes cluster. We'll use Helm for a straightforward installation.

Add Crossplane Helm Repository:

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

Install Crossplane:

Install Crossplane into its own namespace, `crossplane-system`.

helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace

Verify Installation:

Check if Crossplane pods are running and its Custom Resource Definitions (CRDs) are registered.

kubectl get pods -n crossplane-system
kubectl get crds | grep crossplane

You should see pods like `crossplane-*` in a `Running` state and numerous CRDs related to Crossplane.

Step 3: Configuring Crossplane AWS Provider

Crossplane uses "Providers" to interact with external APIs (like AWS, Azure, GCP, etc.). We'll set up the AWS Provider.

Install AWS Provider:

Install the official AWS Provider for Crossplane. Ensure you pick a stable version.

kubectl crossplane install provider crossplane/provider-aws:v0.40.0

Verify the provider is installed and healthy:

kubectl get providers
# Expected output:
# NAME                   INSTALLED   HEALTHY   PACKAGE
# provider-aws           True        True      crossplane/provider-aws:v0.40.0

Create AWS IAM Credentials Secret:

Crossplane needs AWS credentials to provision resources. For production, consider using IAM Roles for Service Accounts (IRSA) on EKS or similar mechanisms. For a quick start, we'll use an AWS IAM user's access key and secret key. First, create an IAM user in AWS with programmatic access and attach a policy (e.g., `AdministratorAccess` for demo purposes, but use least privilege in production). Then, create a Kubernetes Secret containing these credentials:

# Replace with your actual AWS credentials
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

kubectl create secret generic aws-creds -n crossplane-system \
  --from-literal=credentials="[default]\naws_access_key_id = $AWS_ACCESS_KEY_ID\naws_secret_access_key = $AWS_SECRET_ACCESS_KEY"

Create ProviderConfig:

Now, create a `ProviderConfig` resource that references this secret. This tells Crossplane how to use the AWS Provider. Create a file named `aws-provider-config.yaml`:

# aws-provider-config.yaml
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: credentials

Apply the `ProviderConfig`:

kubectl apply -f aws-provider-config.yaml

Verify it's active:

kubectl get providerconfigs
# Expected output:
# NAME      AGE
# default   Xs

Step 4: Defining Composite Resources (XRs) and Compositions

This is where Crossplane's power for platform engineers shines. We define "Composite Resources" (XRs) as the abstract, opinionated interfaces developers will interact with. "Compositions" then map these XRs to actual "Managed Resources" (like `S3Bucket` or `RDSInstance`) in the cloud.

Concept:

* **XRD (CompositeResourceDefinition):** Defines the schema for your custom, opinionated infrastructure resource. It's like a Kubernetes CRD but for Crossplane composites. * **Composition:** A blueprint that describes how to compose one or more Crossplane Managed Resources from an XRD.

Example: An Opinionated S3 Bucket

Let's create an `XBucket` (e.g., "eXperience Bucket") that automatically applies specific tags and encryption settings, abstracting these details from the developer. First, define the `XBucket`'s schema using an XRD. Create `s3-bucket-xrd.yaml`:

# s3-bucket-xrd.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xbuckets.platform.technewsventure.com
spec:
  group: platform.technewsventure.com
  names:
    kind: XBucket
    plural: xbuckets
  claimNames:
    kind: Bucket
    plural: buckets
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                parameters:
                  type: object
                  description: "Parameters for the XBucket"
                  properties:
                    name:
                      type: string
                      description: "The name of the S3 bucket."
                    tags:
                      type: object
                      description: "Additional tags to apply to the S3 bucket."
                      x-kubernetes-preserve-unknown-fields: true
                  required:
                    - name
              required:
                - parameters
            status:
              type: object
              properties:
                bucketName:
                  type: string
                  description: "The actual name of the provisioned S3 bucket."

Apply the XRD:

kubectl apply -f s3-bucket-xrd.yaml
kubectl get xrd
# Expected output:
# NAME                                                ESTABLISHED   ACCEPTED   AGE
# xbuckets.platform.technewsventure.com               True          True       Xs

Next, create the `Composition` that defines how an `XBucket` translates into an AWS `S3Bucket`. This Composition will enforce server-side encryption and add default `managedBy` and `platform` tags, while allowing developers to specify additional tags. Create `s3-bucket-composition.yaml`:

# s3-bucket-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xbuckets.platform.technewsventure.com
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: platform.technewsventure.com/v1alpha1
    kind: XBucket
  resources:
    - name: s3bucket
      base:
        apiVersion: s3.aws.crossplane.io/v1beta1
        kind: Bucket
        spec:
          forProvider:
            region: us-east-1 # Enforce a specific region
            acl: Private
            serverSideEncryptionConfiguration:
              rules:
                - applyServerSideEncryptionByDefault:
                    sseAlgorithm: AES256
            tags: # Default platform tags
              managedBy: Crossplane
              platform: TechNewsVentureIDP
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.name
          toFieldPath: metadata.annotations[crossplane.io/external-name] # Use this for actual bucket name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.name
          toFieldPath: spec.forProvider.name # For some AWS resources, name is directly in spec.forProvider.name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.tags
          toFieldPath: spec.forProvider.tags
          policy:
            mergeOptions:
              append: true # Merge developer tags with platform tags
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.name
          toFieldPath: status.bucketName

Apply the Composition:

kubectl apply -f s3-bucket-composition.yaml
kubectl get composition
# Expected output:
# NAME                                  AGE
# xbuckets.platform.technewsventure.com Xs

Now, a developer can create an `XBucket` resource, and Crossplane will provision an S3 bucket in AWS with the defined policies. To test this, create a sample `XBucket` claim:

# my-app-s3-bucket.yaml
apiVersion: platform.technewsventure.com/v1alpha1
kind: XBucket
metadata:
  name: sujay-test-bucket-12345 # Must be globally unique for S3
spec:
  parameters:
    name: sujay-test-bucket-12345
    tags:
      environment: dev
      project: tech-news-venture-idp

Apply the claim:

kubectl apply -f my-app-s3-bucket.yaml

Monitor the status:

kubectl get xbucket sujay-test-bucket-12345
# STATUS field should eventually show "Ready"
kubectl get bucket.s3.aws.crossplane.io # See the underlying AWS S3 Bucket managed resource

You can verify in the AWS console that an S3 bucket named `sujay-test-bucket-12345` has been created with the specified encryption and tags.

Step 5: Integrating Backstage with Crossplane (Self-Service Provisioning)

The real magic happens when developers can request these `XBucket`s (or other XRs) directly from Backstage, without needing `kubectl` or deep Kubernetes knowledge. This is achieved using Backstage's Scaffolder.

Enable and Configure Backstage Scaffolder:

Ensure the Scaffolder plugin is enabled in your `app-config.yaml` and `packages/backend/src/plugins/scaffolder.ts`. The default Backstage setup usually includes it.

Create a Custom Scaffolder Template:

We'll create a Scaffolder template that generates an `XBucket` manifest and potentially registers it back into the Backstage catalog. Create a new folder `packages/backend/templates/s3-bucket-template` and inside it, create `template.yaml` and `catalog-info.yaml`. `packages/backend/templates/s3-bucket-template/template.yaml`:

# packages/backend/templates/s3-bucket-template/template.yaml
apiVersion: backstage.io/v1alpha1
kind: Template
metadata:
  name: s3-bucket-provisioner
  title: Provision an S3 Bucket (Crossplane)
  description: Provisions a new S3 bucket using Crossplane with default encryption and tags.
  tags:
    - aws
    - s3
    - crossplane
    - infrastructure
spec:
  owner: sujay.singh@technewsventure.com
  type: service
  parameters:
    - title: S3 Bucket Details
      properties:
        bucketName:
          title: S3 Bucket Name
          type: string
          description: A globally unique name for your S3 bucket.
          ui:autofocus: true
          ui:options:
            rows: 5
        description:
          title: Description
          type: string
          description: A brief description of the bucket's purpose.
        environment:
          title: Environment
          type: string
          description: The environment for this bucket (e.g., dev, staging, prod).
          enum:
            - dev
            - staging
            - prod
          default: dev
  steps:
    - id: createS3Bucket
      name: Create S3 Bucket Request
      action: fs:writeFile
      input:
        path: ./{{ parameters.bucketName }}-s3-bucket.yaml
        content: |
          apiVersion: platform.technewsventure.com/v1alpha1
          kind: XBucket
          metadata:
            name: {{ parameters.bucketName }}
            annotations:
              backstage.io/managed-by-location: '{{ template.metadata.annotations["backstage.io/managed-by-location"] }}'
              backstage.io/managed-by-origin-id: '{{ template.metadata.uid }}'
          spec:
            parameters:
              name: {{ parameters.bucketName }}
              tags:
                environment: {{ parameters.environment }}
                description: {{ parameters.description }}
                requestedBy: {{ user.entity.metadata.name }} # Example of using user context
    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: github.com?owner=tech-news-venture&repo={{ parameters.bucketName }}-s3-bucket-config
        token: ${{ secrets.GITHUB_TOKEN }} # Ensure you have a GitHub token secret configured
        defaultBranch: main
        commitMessage: "feat({{ parameters.bucketName }}): Initial S3 bucket configuration"
    - id: register
      name: Register in Backstage Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: /{{ parameters.bucketName }}-s3-bucket.yaml # Path to the generated XBucket manifest
  output:
    links:
      - title: Open in catalog
        icon: catalog
        entityRef: {{ steps.register.output.entityRef }}
      - title: Open in GitHub
        icon: code
        url: ${{ steps.publish.output.repoUrl }}

`packages/backend/templates/s3-bucket-template/catalog-info.yaml`:

# packages/backend/templates/s3-bucket-template/catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Template
metadata:
  name: s3-bucket-provisioner-catalog
  title: Provision an S3 Bucket (Crossplane)
  description: Provisions a new S3 bucket using Crossplane with default encryption and tags.
  tags:
    - aws
    - s3
    - crossplane
    - infrastructure
spec:
  owner: sujay.singh@technewsventure.com
  type: service
  lifecycle: experimental

Update `app-config.yaml` to include this new template location:

# app-config.yaml
scaffolder:
  # ... other scaffolder config
  locations:
    - type: file
      target: packages/backend/templates/s3-bucket-template/template.yaml

**Important Note on `fs:writeFile` and `publish:github`:** The `fs:writeFile` action writes the `XBucket` manifest to a temporary location. For a real-world GitOps flow, you would typically use `publish:github` (or similar for GitLab/Azure DevOps) to push this manifest to a Git repository monitored by a GitOps operator (like ArgoCD or FluxCD). This operator would then apply the `XBucket` manifest to Kubernetes, and Crossplane would provision the AWS S3 bucket. The `catalog:register` step ensures that once the manifest is in Git, Backstage can track it.

Developer Experience:

1. A developer logs into Backstage. 2. Navigates to the "Create" section (Scaffolder).

Platform Engineering: Building Internal Developer Platforms with Backstage and Crossplane