Boost K8s Monitoring: Prometheus Custom Exporters & Recording Rules

Overview

In the dynamic world of cloud-native computing, Kubernetes has emerged as the undisputed orchestrator for containerized applications. Monitoring these complex, distributed systems is paramount, and Prometheus has become the de facto standard for this task. While Prometheus excels at collecting metrics from a wide array of existing services through its vast ecosystem of exporters (like `node_exporter`, `kube-state-metrics`), there inevitably comes a point where generic metrics fall short. Applications often possess unique internal states, business-critical KPIs, or specific operational details that standard exporters simply cannot expose.

This is where Prometheus custom exporters step in. Custom exporters allow developers to instrument their applications with bespoke metrics, transforming internal application logic into a Prometheus-compatible format. Whether it's the number of active database connections for a specific microservice, the latency of an internal API call, or the count of processed business transactions, custom exporters bridge the gap between application-specific insights and a unified monitoring platform.

However, simply collecting a wealth of metrics isn't enough. As your Kubernetes clusters and applications scale, querying raw, high-cardinality metrics can become resource-intensive and slow. Complex PromQL queries, especially those involving aggregations or rates over large time ranges, can put a significant strain on your Prometheus server. This is where Prometheus recording rules become indispensable. Recording rules allow you to pre-compute and store the results of frequently executed or computationally expensive PromQL expressions as new time series data. This not only dramatically improves query performance for dashboards and alerts but also reduces the load on your Prometheus instance, ensuring a smoother and more responsive monitoring experience.

In this comprehensive guide, we, at TechNews Venture, will delve deep into the practical implementation of Prometheus custom exporters and recording rules for Kubernetes monitoring. We'll walk through developing a custom exporter, deploying it within Kubernetes, and then leveraging recording rules to optimize the monitoring of your custom metrics, providing you with unparalleled visibility and operational efficiency.

Prerequisites

Before we embark on our journey to build and deploy custom Prometheus exporters and recording rules, ensure you have the following components and knowledge in place:

• Kubernetes Cluster:

  A running Kubernetes cluster. This can be a local setup like Kind or Minikube, or a managed cloud service like Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), or Azure Kubernetes Service (AKS). For this guide, we'll assume a generic Kubernetes environment.

• Prometheus Installation:

  Prometheus should be already installed and operational within your Kubernetes cluster. The most common and recommended way to achieve this is by using the kube-prometheus-stack Helm chart, which includes Prometheus, Alertmanager, Grafana, and the Prometheus Operator.

• kubectl:

  The Kubernetes command-line tool (`kubectl`) must be installed and configured to communicate with your Kubernetes cluster.

• Helm:

  Helm, the package manager for Kubernetes, is required for deploying the kube-prometheus-stack.

• Go Language:

  For developing our custom exporter, we will use Go. Ensure Go (version 1.16 or higher) is installed on your development machine.

• Docker:

  Docker is necessary to containerize our custom exporter and push its image to a container registry (e.g., Docker Hub, GCR, ECR).

• Basic Linux/CLI Knowledge:

  Familiarity with command-line operations is assumed.

Step-by-step implementation

1. Setting up Prometheus in Kubernetes (if not already done)

We'll use the kube-prometheus-stack Helm chart, which provides a comprehensive monitoring solution including Prometheus, Alertmanager, Grafana, and the Prometheus Operator. This operator simplifies the management of Prometheus and its components within Kubernetes.

# Add the Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create a namespace for monitoring components
kubectl create namespace monitoring

# Install kube-prometheus-stack
# We'll enable persistence for Prometheus and Grafana for production scenarios.
# Adjust resource requests/limits as per your cluster capacity.
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName="standard" \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage="50Gi" \
  --set grafana.persistence.enabled=true \
  --set grafana.persistence.storageClassName="standard" \
  --set grafana.persistence.size="10Gi" \
  --set grafana.adminPassword="your-strong-password" \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesPrometheusRuleSelector=false \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesPrometheusServiceMonitorSelector=false \
  --wait

# Port-forward the Prometheus UI to access it locally
echo "Prometheus UI will be available at http://localhost:9090"
kubectl --namespace monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 &
echo "Grafana UI will be available at http://localhost:3000 (admin/your-strong-password)"
kubectl --namespace monitoring port-forward svc/prometheus-grafana 3000:80 &

The ruleSelectorNilUsesPrometheusRuleSelector=false and serviceMonitorSelectorNilUsesPrometheusServiceMonitorSelector=false flags are crucial. They ensure that Prometheus Operator only discovers PrometheusRule and ServiceMonitor objects that are explicitly labeled to be picked up by *this specific* Prometheus instance, rather than scraping all of them in the cluster. This provides better isolation and control.

2. Developing a Custom Exporter

For our custom exporter, we will create a simple Go application that exposes a couple of custom metrics. This exporter will simulate an application reporting its active database connections and total processed requests.

2.1. Choosing a Language and Library

Go is an excellent choice for exporters due to its performance, concurrency, and the robust client_golang library. This library provides all the necessary primitives to expose metrics in a Prometheus-compatible format.

2.2. Exporter Code Example (Go)

Let's create a file named main.go:

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// Define custom metrics
var (
	// app_db_connections_active: Gauge metric to track active DB connections.
	// Labels: `db_name` for the specific database.
	dbConnections = promauto.NewGaugeVec(prometheus.GaugeOpts{
		Name: "app_db_connections_active",
		Help: "Current number of active database connections.",
	}, []string{"db_name"})

	// app_http_requests_total: Counter metric to track total HTTP requests.
	// Labels: `path` for the request endpoint, `method` for HTTP method, `status_code` for response status.
	httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "app_http_requests_total",
		Help: "Total number of HTTP requests processed by the application.",
	}, []string{"path", "method", "status_code"})

	// app_processing_duration_seconds: Histogram metric to track request processing duration.
	// Labels: `path` for the request endpoint.
	requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "app_processing_duration_seconds",
		Help:    "Histogram of request processing duration in seconds.",
		Buckets: prometheus.DefBuckets, // Default buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10
	}, []string{"path"})
)

// simulateMetricsUpdate simulates application activity and updates metrics
func simulateMetricsUpdate() {
	ticker := time.NewTicker(2 * time.Second)
	defer ticker.Stop()

	for range ticker.C {
		// Simulate DB connection changes
		dbConnections.WithLabelValues("users_db").Set(float64(time.Now().UnixNano()%10 + 5)) // 5-14 connections
		dbConnections.WithLabelValues("products_db").Set(float64(time.Now().UnixNano()%5 + 2)) // 2-6 connections

		// Simulate HTTP requests
		// Increment total requests
		httpRequestsTotal.WithLabelValues("/api/v1/users", "GET", "200").Inc()
		httpRequestsTotal.WithLabelValues("/api/v1/products", "GET", "200").Inc()
		if time.Now().Second()%3 == 0 {
			httpRequestsTotal.WithLabelValues("/api/v1/users", "POST", "201").Inc()
		}
		if time.Now().Second()%5 == 0 {
			httpRequestsTotal.WithLabelValues("/api/v1/health", "GET", "500").Inc()
		}

		// Simulate request durations
		path := "/api/v1/users"
		duration := float64(time.Now().UnixNano()%500) / 1000.0 // 0 to 0.5 seconds
		requestDuration.WithLabelValues(path).Observe(duration)

		path = "/api/v1/products"
		duration = float64(time.Now().UnixNano()%1000) / 1000.0 // 0 to 1 second
		requestDuration.WithLabelValues(path).Observe(duration)

		log.Println("Metrics updated...")
	}
}

func main() {
	// Initialize go module
	// go mod init my-app-exporter
	// go get github.com/prometheus/client_golang/prometheus

	// Start simulating metric updates in a goroutine
	go simulateMetricsUpdate()

	// Expose the metrics endpoint
	http.Handle("/metrics", promhttp.Handler())
	listenPort := ":8080"
	fmt.Printf("Starting custom exporter on port %s...\n", listenPort)
	log.Fatal(http.ListenAndServe(listenPort, nil))
}

To initialize the Go module and download dependencies:

mkdir my-app-exporter
cd my-app-exporter
go mod init my-app-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp

2.3. Containerizing the Exporter

Next, we need to containerize our Go application using Docker. Create a Dockerfile in the same directory as main.go:

# Use a multi-stage build for a small final image
# Stage 1: Builder
FROM golang:1.18-alpine AS builder

WORKDIR /app

# Copy go mod and sum files
COPY go.mod go.sum ./

# Download all dependencies. Dependencies will be cached if the go.mod and go.sum files are not changed
RUN go mod download

# Copy the source code
COPY . .

# Build the Go application
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o my-app-exporter .

# Stage 2: Final image
FROM alpine:latest

WORKDIR /root/

# Copy the compiled executable from the builder stage
COPY --from=builder /app/my-app-exporter .

# Expose the port our exporter listens on
EXPOSE 8080

# Run the executable
CMD ["./my-app-exporter"]

Now, build the Docker image and push it to a registry. Replace your-dockerhub-username with your Docker Hub username or your private registry path.

docker build -t your-dockerhub-username/my-app-exporter:v1.0.0 .
docker push your-dockerhub-username/my-app-exporter:v1.0.0

2.4. Deploying the Exporter to Kubernetes

We'll deploy our custom exporter as a Kubernetes Deployment and expose it via a Service. Then, we'll use a ServiceMonitor (a Custom Resource Definition provided by Prometheus Operator) to tell Prometheus to scrape metrics from our exporter.

Create a file named my-app-exporter-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-exporter
  namespace: default # Deploy in default namespace for simplicity, but consider dedicated app namespaces
  labels:
    app: my-app-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-exporter
  template:
    metadata:
      labels:
        app: my-app-exporter
    spec:
      containers:
      - name: my-app-exporter
        image: your-dockerhub-username/my-app-exporter:v1.0.0 # Replace with your image
        ports:
        - name: http-metrics
          containerPort: 8080
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "100m"
            memory: "128Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-exporter
  namespace: default
  labels:
    app: my-app-exporter
spec:
  selector:
    app: my-app-exporter
  ports:
    - name: http-metrics
      port: 8080
      targetPort: http-metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-exporter-monitor
  namespace: default # ServiceMonitor should be in the same namespace as the Service it targets
  labels:
    release: prometheus # This label is crucial for kube-prometheus-stack to discover it
spec:
  selector:
    matchLabels:
      app: my-app-exporter # Selects the Service with this label
  endpoints:
  - port: http-metrics # Refers to the name of the port in the Service
    path: /metrics
    interval: 15s # How often Prometheus should scrape this endpoint
  namespaceSelector:
    matchNames:
    - default # Specifies the namespaces where the ServiceMonitor should look for Services

Apply these Kubernetes resources:

kubectl apply -f my-app-exporter-deployment.yaml

Verify that the pod and service are running:

kubectl get pods -l app=my-app-exporter
kubectl get svc -l app=my-app-exporter
kubectl get servicemonitor -l release=prometheus

2.5. Verifying Metrics

After a few moments, Prometheus should discover and start scraping your custom exporter. Access the Prometheus UI (via the port-forwarding command from earlier, http://localhost:9090) and navigate to the "Graph" tab. You should be able to query your custom metrics:

• app_db_connections_active
• app_http_requests_total
• app_processing_duration_seconds_bucket (and `_sum`, `_count`)

Try a query like: app_db_connections_active{db_name="users_db"} or sum by (path) (rate(app_http_requests_total[5m])).

3. Implementing Recording Rules

Now that we have custom metrics flowing into Prometheus, let's create some recording rules to pre-aggregate frequently used queries or compute complex rates, making our dashboards and alerts more efficient.

3.1. Understanding Recording Rules

Recording rules allow you to define new time series based on the results of PromQL queries. When Prometheus evaluates a recording rule, it executes the specified query and stores the result as a new metric with a new name. This is particularly useful for:

• Performance: Complex queries are computed once and stored, reducing the load during dashboard rendering or alert evaluation.
• Consistency: Ensures that everyone uses the same definition for a derived metric.
• Simplification: Provides simpler, more intuitive metrics for users.

3.2. Defining Recording Rules

We'll define a few recording rules in a YAML file. These rules will calculate the 5-minute rate of HTTP requests and the average active database connections.

Create a file named my-app-recording-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-recording-rules
  namespace: default # Deploy in the same namespace where Prometheus Operator is watching
  labels:
    release: prometheus # This label is crucial for kube-prometheus-stack to discover it
spec:
  groups:
  - name: my-app-custom-metrics
    rules:
    - record: app_http_requests_total:rate5m
      expr: |
        sum by (path, method, status_code) (rate(app_http_requests_total[5m]))
      labels:
        description: "5-minute rate of total HTTP requests for the application"

    - record: app_db_connections_active:avg5m
      expr: |
        avg_over_time(app_db_connections_active[5m])
      labels:
        description: "5-minute average of active database connections"

    - record: app_http_errors_total:rate5m
      expr: |
        sum by (path, method) (rate(app_http_requests_total{status_code=~"5.."}[5m]))
      labels:
        description: "5-minute rate of HTTP 5xx errors"

    - record: app_request_latency_p90_5m
      expr: |
        histogram_quantile(0.90, sum by (path, le) (rate(app_processing_duration_seconds_bucket[5m])))
      labels:
        description: "90th percentile of request processing duration over 5 minutes"

A few notes on the rules:

• record: The name of the new metric that will be created. We follow the convention level:metric:operation.
• expr: The PromQL query whose result will be stored.
• labels: Optional labels to add to the new metric.

3.3. Deploying Recording Rules to Kubernetes

With Prometheus Operator (which comes with kube-prometheus-stack), recording rules are managed via the PrometheusRule Custom Resource. The Prometheus instance will automatically pick up any PrometheusRule objects that match its configured label selector (which is release: prometheus in our setup).

kubectl apply -f my-app-recording-rules.yaml

You can validate your rule file before applying it using promtool:

# Assuming you have promtool installed locally
promtool check rules my-app-recording-rules.yaml

3.4. Verifying Recording Rules

Give Prometheus a few minutes to pick up the new rules and start evaluating them. Then, go back to the Prometheus UI (http://localhost:9090) and query the new recorded metrics:

• app_http_requests_total:rate5m
• app_db_connections_active:avg5m
• app_http_errors_total:rate5m
• app_request_latency_p90_5m

You'll notice these metrics appear instantly and are much faster to query than their underlying complex expressions. These pre-aggregated metrics are perfect for populating Grafana dashboards or defining simple, efficient alert rules.

Security Considerations

While custom exporters and recording rules enhance monitoring capabilities, they also introduce potential security vectors that must be addressed.

• Exporter Security:

  • Network Access: Exporters typically expose metrics on an HTTP endpoint. Restrict network access to these endpoints using Kubernetes Network Policies, allowing only Prometheus to scrape them. Do not expose exporter ports directly to the internet.
  • Least Privilege: Run exporter containers with minimal necessary privileges. Avoid running as root. If possible, use a dedicated service account with limited permissions.
  • Metric Content: Never expose sensitive information (e.g., API keys, personally identifiable information, internal system details) via metrics. Sanitize any potential sensitive data before exposing it.
  • Label Injection: Be careful with user-provided input that might end up as metric labels. Malicious input could lead to high cardinality issues or even expose information if not properly sanitized.
  • Authentication/Authorization: For highly sensitive metrics, consider adding basic authentication or mTLS to your exporter endpoint, though this adds complexity to Prometheus scraping configuration. For most internal metrics, network policies are sufficient.

• Prometheus Security:

  • RBAC: Ensure Prometheus itself runs with appropriate Kubernetes Role-Based Access Control (RBAC) permissions. The kube-prometheus-stack generally sets this up correctly, but review it if you're deploying manually.
  • Network Policies: Implement network policies to restrict access to the Prometheus server and its components (e.g., Grafana, Alertmanager) only to trusted sources.
  • TLS: Secure the Prometheus UI and API with TLS/SSL, especially if it's accessible outside your private network.
  • Configuration Access: Restrict access to Prometheus configuration (e.g., PrometheusRule, ServiceMonitor definitions) to authorized personnel to prevent malicious rule injection or alteration.

Best Practices

Adhering to best practices ensures your Prometheus monitoring setup remains robust, scalable, and maintainable.

• Metric Naming Conventions:

  Follow Prometheus's metric naming conventions. Use suffixes like _total for counters, _seconds for durations, and _bytes for sizes. Use clear, concise, and consistent names.

• Labeling Strategy:

  Use labels effectively to add dimensions to your metrics (e.g., instance, job, namespace, pod, service, endpoint). Avoid labels with high cardinality (e.g., user IDs, timestamps, session IDs) as they can explode Prometheus's memory usage and storage requirements.

• Exporter Design:

  • Simplicity: Keep custom exporters lightweight and focused on exposing specific metrics. Avoid embedding complex application logic within the exporter itself.
  • Idempotence: Metric exposition should be idempotent. Multiple scrapes should yield consistent results for a given state.
  • Error Handling: Implement robust error handling within your exporter to prevent crashes and ensure metrics continue to be exposed even if an underlying system fails temporarily.
  • Health Checks: Provide a simple health endpoint (e.g., /healthz) in addition to /metrics for Kubernetes liveness/readiness probes.

• Recording Rule Granularity:

  Balance the granularity of your recording rules. Pre-aggregate frequently used queries, but don't overdo it. Too many recording rules can increase Prometheus's write load and storage. Focus on rules that significantly improve performance or simplify common queries for dashboards and alerts.

• Testing:

  • Exporter Testing: Write unit and integration tests