Predicting Kubernetes Outages: Saturation Metrics and Error Budgets
Intermediate · 11 min read
The Problem
In the world of DevOps, there is a pervasive anxiety known as “alert fatigue.” Engineers are bombarded with alerts from Prometheus, Datadog, or PagerDuty, often waking up at 3 AM for warnings that turn out to be false positives or minor blips. However, the more dangerous scenario is the silent failure: a system that appears healthy according to standard CPU and memory thresholds but is slowly degrading in quality until it collapses under load.
Consider a typical e-commerce platform during a flash sale. Traditional monitoring might show CPU usage at 60%—well below the 80% warning threshold. The monitoring system sends no alert. But behind the scenes, response latency has jumped from 100ms to 2 seconds, and database connection pools are exhausted. Users experience errors, not because the servers are down, but because they are saturated. By the time the team notices the drop in revenue, the outage has already lasted hours.
This tutorial addresses this specific pain point. We will move beyond simple resource utilization (CPU/Memory) and focus on Saturation Metrics and Error Budgets. Saturation metrics measure the pressure on a resource (like queue length or disk I/O wait), which is a leading indicator of failure. Error budgets quantify how much reliability we can afford to lose before impacting business goals. Together, they form a predictive model for outages, allowing teams to intervene before users feel the degradation.
Core Concepts
To understand how we predict outages, we must redefine what “healthy” means in Kubernetes. We rely on two foundational concepts from SRE (Site Reliability Engineering): Google’s Four Golden Signals and the concept of Error Budgets.
1. The Four Golden Signals
Google defines four key metrics that every service should monitor:
- Latency: Time taken to service a request. Note: Do not confuse this with throughput; we care about time for requests that are not failing.
- Traffic: Demand placed on the system. This could be measured as QPS (Queries Per Second), HTTP requests per second, or TCP connections open.
- Errors: Rate of requests that fail. This includes semantic errors (4xx/5xx status codes) and protocol-level failures.
- Saturation: How “full” the service is. This is the most critical metric for prediction. It measures resources that cannot be easily scaled without architectural changes, such as disk space, memory fragmentation, or network bandwidth saturation.
2. Saturation vs. Utilization
Utilization asks, “How much of the resource am I using?” (e.g., 70% CPU). Saturation asks, “How congested is the path through the resource?” (e.g., Disk I/O Wait time or Queue Length). A disk can have low utilization but high saturation if it is waiting for I/O completion. In Kubernetes, standard metrics like `container_cpu_usage_seconds_total` are utilizations. To get saturation, we often need to look at derived metrics like `node_filesystem_avail_bytes` relative to write speed, or application-level metrics like active database connections.
3. Error Budgets
An Error Budget is the amount of unreliability a service can tolerate within a given period while still meeting its Service Level Objective (SLO). If your SLO is 99.9% availability over a month, your error budget is 0.1% downtime. Every bug, deployment, or outage consumes part of this budget. When the budget is exhausted, feature development stops, and the team focuses solely on reliability until the next monthly reset. This shifts the culture from “deploy fast” to “deploy safely based on current health.”
4. SLOs, SLIs, and SLAs
- SLI (Service Level Indicator): A quantitative measure of some aspect of the level of service provided (e.g., percentage of successful HTTP requests).
- SLO (Service Level Objective): A target value or range of values for the SLI (e.g., 99.9% of requests succeed).
- SLA (Service Level Agreement): A commitment to the customer, usually involving financial penalties if breached.
How It Works
The mechanism for predicting outages using these concepts involves a feedback loop between Prometheus (metrics collection), Alertmanager (alerting logic), and Kubernetes (scaling actions).
Step 1: Instrumentation
We instrument our applications to expose custom metrics that reflect saturation (e.g., queue depth) and errors (e.g., 5xx rate). Standard Kubelet metrics provide node-level saturation, but application-level saturation requires code changes or sidecar proxies.
Step 2: Calculation
Prometheus uses PromQL to calculate real-time SLIs. For example, we calculate the error ratio over a 5-minute window. Simultaneously, we track the consumption of the error budget by comparing current errors against the total allowed errors for the month.
Step 3: Prediction & Action
Instead of alerting when CPU > 80%, we alert when:
1. Saturation metrics indicate imminent resource exhaustion (e.g., disk fill rate > 1GB/min).
2. The error budget burn rate is too high (e.g., consuming 2 months of budget in 1 hour).
This triggers horizontal pod autoscaling (HPA) or vertical pod autoscaling (VPA) proactively, rather than reactively after an OOMKill event.
Prerequisites
Before proceeding with this hands-on walkthrough, ensure you have the following environment set up:
- Kubernetes Cluster: A running cluster (v1.24+ recommended). You can use Minikube, Kind, or a cloud provider (EKS/GKE/AKS). This tutorial assumes a Linux-based node environment.
- kubectl Access: Local installation configured to communicate with your cluster.
- Prometheus Stack: Installed via Helm. We will use the official
kube-prometheus-stackchart, which bundles Prometheus, Grafana, and Alertmanager. - Namespace: Create a dedicated namespace for this exercise:
kubectl create ns sre-demo. - Basic Knowledge: Familiarity with YAML manifests, Pods, Deployments, and Services. Understanding of basic HTTP status codes (200, 500, 503).
Hands-On Walkthrough
We will build a simulated microservice that generates traffic and errors, exposing saturation metrics. Then, we will configure Prometheus to monitor error budget burn rates and trigger alerts.
Step 1: Deploy the Sample Application
First, we need a workload that behaves unpredictably. We will deploy a simple Python Flask app that occasionally returns 503 errors and has a configurable “saturation” limit (simulated by thread pool size). Save the following YAML as app-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: unreliable-service
namespace: sre-demo
labels:
app: unreliable-service
spec:
replicas: 2
selector:
matchLabels:
app: unreliable-service
template:
metadata:
labels:
app: unreliable-service
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
spec:
containers:
- name: app
image: python:3.9-slim
command: ["python", "-c",
"import http.server, socketserver, random, time\n"
"class H(http.server.BaseHTTPRequestHandler):\n"
" def do_GET(self):\n"
" if random.random() < 0.1: # 10% error rate\n"
" self.send_response(503)\n"
" else:\n"
" self.send_response(200)\n"
" self.end_headers()\n"
" self.wfile.write(b'OK')\n"
"with socketserver.TCPServer(('0.0.0.0', 9090), H) as s:\n"
" s.serve_forever()"]
ports:
- containerPort: 9090
---
apiVersion: v1
kind: Service
metadata:
name: unreliable-service
namespace: sre-demo
spec:
selector:
app: unreliable-service
ports:
- port: 80
targetPort: 9090
Explanation:
This deployment creates a simple HTTP server. It intentionally introduces a 10% failure rate (503 Service Unavailable) to simulate production instability. The annotations tell Prometheus to scrape metrics from port 9090. While this simple app doesn’t expose rich Prometheus metrics, in a real scenario, you would use a client library (like prometheus-client) to expose counters for requests and errors.
Apply the manifest:
kubectl apply -f app-deployment.yamlStep 2: Install the Prometheus Stack
If you haven’t already installed the Prometheus stack, add the Bitnami repository and install it into your cluster. This provides the monitoring backbone.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kube-prometheus bitnami/kube-prometheus-stack \
--namespace sre-demo \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set grafana.adminPassword=admin123
Note: The flag --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false ensures that PodMonitors defined in Helm charts are picked up even if other selectors don’t match, ensuring broad coverage for our demo.
Step 3: Define the SLO and Error Budget
Let’s define a realistic SLO. Assume our acceptable error rate is 1%. Our current app has a 10% error rate. This means we are burning through our error budget rapidly.
We need to create a PrometheusRule that calculates the error budget burn rate. The burn rate is calculated as:
Burn Rate = (Error Rate / Allowed Error Rate)
Create a file named slo-alerts.yaml:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: sre-budget-alerts
namespace: sre-demo
spec:
groups:
- name: error-budget.rules
rules:
- expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
record:sre_demo:error_rate_5m
- expr: |
sre_demo:error_rate_5m / 0.01
record:sre_demo:budget_burn_rate_5m
- alert: HighBudgetBurnRate
expr: sre_demo:budget_burn_rate_5m > 10
for: 2m
labels:
severity: critical
annotations:
summary: "Error budget burn rate is too high"
description: "The service is consuming its error budget 10x faster than normal. Immediate action required. Current Burn Rate: {{ $value }}"
Explanation:
1. sre_demo:error_rate_5m: Calculates the ratio of 5xx errors to total requests over the last 5 minutes.
2. sre_demo:budget_burn_rate_5m: Divides the observed error rate by our allowed error rate (1% or 0.01). If the error rate is 10%, the burn rate is 10x.
3. HighBudgetBurnRate: Alerts if the burn rate exceeds 10x for 2 minutes. This indicates that at this pace, we will exhaust our monthly error budget in less than 3 days.
Apply the alerts:
kubectl apply -f slo-alerts.yamlStep 4: Configure Horizontal Pod Autoscaler (HPA) based on Custom Metrics
To demonstrate proactive scaling, we assume our app exposes a metric called active_connections (simulated here). In a real app, you’d expose this via the Prometheus endpoint. For this tutorial, we will scale based on the existing error rate metric using the keda operator or native HPA if using Prometheus adapter.
For simplicity in this core tutorial, we will stick to the alerting mechanism. In production, you would integrate this alert with an autoscaler controller that watches the Prometheus API.
Verifying It Works
Now we need to verify that the system detects the instability.
1. Check Pod Status
kubectl get pods -n sre-demo
You should see two pods for unreliable-service running. Ensure they are in the Running state.
2. Generate Traffic
Use a tool like wrk or curl to generate sustained traffic to the service. This ensures Prometheus collects data points.
kubectl run -it --rm load-generator --image=busybox --restart=Never -- sh -c "while true; do wget -q -O- http://unreliable-service.sre-demo.svc.cluster.local; done"
3. Inspect Prometheus Alerts
Access the Prometheus UI. Port-forward the Prometheus service:
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n sre-demo
Open http://localhost:9090/alerts in your browser. You should see the HighBudgetBurnRate alert firing as Firing. The label severity=critical confirms it matches our definition.
4. Verify Calculations in PromQL
Go to the Graph tab in Prometheus. Enter the expression:
sre_demo:budget_burn_rate_5m
You should see a line graph hovering around 10 (since our app has a 10% error rate and our budget allows 1%). This confirms the math is correct.
Common Mistakes
When implementing SLO-based monitoring, teams often fall into these traps:
- Alerting on SLIs instead of SLOs: Alerting when latency > 500ms is an SLI check. Alerting when the percentage of requests > 500ms exceeds 1% is an SLO check. The former causes noise; the latter predicts budget exhaustion.
- Ignoring Multi-dimensional SLOs: A single global error budget hides complexity. If your API gateway has 99.9% success but your payment service has 95%, the overall SLO looks fine. Always break down budgets by service boundary.
- Static Time Windows: Using a fixed 5-minute window for burn rate calculation can miss short-lived spikes or include noise from rolling updates. Use multi-window burn rates (e.g., 1h, 6h, 1d) to distinguish between transient spikes and long-term trends.
- Over-scaling: Automatically scaling based on error budgets can lead to “thundering herds” if the autoscaler doesn’t account for cold starts. Always pair budget alerts with graceful degradation strategies (circuit breakers) before scaling.
Going Further
This tutorial covers the basics of detecting budget burn. To reach production readiness, consider the following advanced topics:
Multi-Window Burn Rates
Implement the classic SRE alerting strategy using three windows:
- Short-term (1h): Detects sudden spikes. Triggers immediate investigation.
- Medium-term (6h): Detects sustained degradation. May trigger on-call paging.
- Long-term (1d): Detects slow leaks. Triggers engineering tickets for backlog planning.
Integration with GitOps
Connect your error budget status to your CI/CD pipeline. If the error budget is < 10% remaining, block deployments automatically. This enforces the “stop deploying if we are broken” principle.
Advanced Saturation Metrics
Move beyond CPU/Memory. Monitor:
- Network Socket Exhaustion: Max file descriptors available.
- Database Connection Pool Saturation: Active connections vs. max pool size.
- GC Pressure: Frequency and duration of Garbage Collection pauses in Java/.NET apps.
Alternative Approaches
Distributed Tracing (Jaeger/Zipkin): While Prometheus gives aggregate metrics, tracing helps identify the specific trace causing latency. Use tracing sampling rates correlated with error budgets to sample more heavily when the budget is low.
Cleanup
To remove all resources created in this tutorial and avoid unexpected charges:
helm uninstall kube-prometheus -n sre-demo
kubectl delete namespace sre-demo
Verify deletion:
kubectl get namespaces
The sre-demo namespace should no longer appear in the list.
References
This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.
