Cutting Your Kubernetes Bill: Identifying and Right-Sizing Idle and Overprovisioned Workloads

Intermediate · 11 min read

The Problem

Cloud infrastructure bills have a notorious habit of spiraling out of control in Kubernetes environments. Unlike traditional virtual machines where you provision a fixed server, Kubernetes allows developers to spin up pods with a few lines of YAML. This ease of deployment often leads to a phenomenon known as request hoarding. Developers, fearing that their application might crash under unexpected load, routinely request far more CPU and memory than the application actually needs.

Consider a common real-world scenario: A growing startup migrates to a managed Kubernetes service. The initial architecture is designed with safety in mind. Every microservice is allocated 2 CPU cores and 4 Gigabytes of memory. Six months later, the engineering team has tripled in size, and dozens of new microservices have been deployed. The monthly cloud bill jumps by forty percent, yet the actual user traffic has barely increased. The cluster is heavily overprovisioned. Nodes are running at ten percent utilization, but the Kubernetes scheduler cannot pack more pods onto them because the declared resource requests are artificially high. Meanwhile, non-production dev and test environments are left running twenty-four seven, consuming expensive compute cycles while sitting completely idle.

Identifying and eliminating these idle and overprovisioned resources is not just a financial necessity; it is a fundamental operational practice. It improves cluster density, reduces scheduling latency, and ensures that your infrastructure costs scale linearly with your actual business value rather than developer anxiety.

Core Concepts

To effectively cut your Kubernetes bill, you must first understand the mechanics of how Kubernetes handles resources. The foundation lies in the difference between resource requests and resource limits.

Requests are guarantees. When you set a CPU request of 500 millicores, you are telling the Kubernetes scheduler to find a node with at least 500 millicores of unallocated CPU capacity. Requests dictate scheduling. If your requests exceed the available capacity of your nodes, new pods will remain in a Pending state.

Limits are hard caps. If a pod exceeds its CPU limit, the kernel throttles the process, slowing it down. If a pod exceeds its memory limit, the kernel terminates the process with an Out Of Memory error. Limits dictate enforcement.

It is also crucial to understand the difference between compressible and incompressible resources. CPU is compressible. If a pod requests 1000 millicores but only uses 10, the unused 990 millicores can be seamlessly shared with other pods on the same node. Memory is incompressible. If a pod requests 1 Gigabyte of memory, that memory is reserved and cannot be used by other pods, even if the application inside the pod is only using 50 Megabytes. Overprovisioning memory is therefore far more costly than overprovisioning CPU, as it directly reduces the packing density of your nodes.

Overprovisioning occurs when the resource requests are set significantly higher than the actual observed usage. Idle resources refer to workloads that are running but performing zero useful work, such as development environments left active over the weekend or deprecated services that were never deleted.

How It Works

Finding overprovisioned resources requires visibility into actual workload behavior. Kubernetes achieves this through a pipeline of metrics collection.

On every node, the kubelet runs a component called cAdvisor. cAdvisor monitors the containers running on that specific node, tracking their CPU usage, memory consumption, network traffic, and disk I/O. However, cAdvisor only stores this data locally on the node.

To make this data available cluster-wide, Kubernetes uses the Metrics Server. The Metrics Server periodically scrapes the cAdvisor data from every kubelet via the Kubelet API. It aggregates this data and exposes it through the Kubernetes Metrics API.

Once the Metrics Server is running, tools like the command-line interface or the Vertical Pod Autoscaler can query the Metrics API. By comparing the real-time usage data provided by the Metrics API against the declared resource requests in your pod specifications, you can calculate utilization percentages. A pod utilizing five percent of its requested memory is a prime candidate for right-sizing. By adjusting the requests to match the actual usage plus a small safety buffer, you free up node capacity, allowing the cluster autoscaler to scale down the underlying nodes and reduce your cloud bill.

Prerequisites

Before proceeding with the hands-on walkthrough, ensure you have the following in place:

• A running Kubernetes cluster, assuming version 1.25 or later for compatibility with standard metrics APIs.
• The kubectl command-line tool configured and authenticated to communicate with your cluster.
• The Metrics Server installed and running in your cluster. Without it, usage metrics will not be available.
• A basic understanding of YAML manifest structures and Kubernetes deployments.

Hands-On Walkthrough

In this walkthrough, we will deploy an intentionally overprovisioned application, measure its actual resource consumption, and then right-size it to reclaim wasted capacity.

First, create a dedicated namespace for this exercise to keep resources isolated.

kubectl create namespace cost-optimization

Next, we will deploy a simple Nginx web server. Nginx is highly efficient and typically requires very few resources to run. However, we will configure the deployment with exaggerated resource requests to simulate a common overprovisioning scenario. Create a file named overprovisioned-nginx.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioned-nginx
  namespace: cost-optimization
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        resources:
          requests:
            cpu: "1000m"
            memory: "1Gi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        ports:
        - containerPort: 80

Apply this manifest to your cluster:

kubectl apply -f overprovisioned-nginx.yaml

This deployment creates three replicas of Nginx. Each replica requests 1 CPU core and 1 Gigabyte of memory. In total, this deployment has requested 3 CPU cores and 3 Gigabytes of memory from the cluster scheduler, reserving these resources exclusively for these pods.

Wait for the pods to enter the Running state:

kubectl wait --for=condition=Ready pod -l app=nginx-demo -n cost-optimization --timeout=60s

Now, we need to observe the actual resource consumption. Ensure your Metrics Server is functioning by running the top command for pods in the namespace:

kubectl top pods -n cost-optimization

You will see output similar to this:

NAME                                   CPU(cores)   MEMORY(bytes)
overprovisioned-nginx-7d8f9b6c4-abcde  2m           15Mi
overprovisioned-nginx-7d8f9b6c4-fghij  1m           14Mi
overprovisioned-nginx-7d8f9b6c4-klmno  2m           16Mi

This output reveals the massive discrepancy. The pods are requesting 1000 millicores of CPU each, but are only using 1 to 2 millicores. They are requesting 1024 Megabytes of memory each, but are only using around 15 Megabytes. The utilization is less than two percent for CPU and less than two percent for memory.

To right-size this deployment, we must adjust the requests to reflect actual usage while leaving a reasonable buffer for startup spikes and minor traffic increases. A common practice is to set requests to the 95th percentile of observed usage, or simply double the observed baseline for small applications. Let us update the requests to 50 millicores of CPU and 64 Megabytes of memory. Create a new file named right-sized-nginx.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioned-nginx
  namespace: cost-optimization
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "200m"
            memory: "128Mi"
        ports:
        - containerPort: 80

Apply the updated manifest:

kubectl apply -f right-sized-nginx.yaml

By reducing the requests, we have freed up nearly 3 CPU cores and 3 Gigabytes of memory back to the cluster scheduler. This reclaimed capacity can now be used to schedule other workloads, potentially allowing the cluster autoscaler to terminate unused underlying nodes.

Verifying It Works

To verify that the right-sizing was successful and the application remains healthy, we will check the new resource allocation and monitor the pod status.

First, describe one of the pods to confirm the new resource requests and limits are applied:

kubectl describe pod -l app=nginx-demo -n cost-optimization | grep -A 5 "Requests:"

Expected output should show the updated values:

Requests:
  cpu:     50m
  memory:  64Mi

Next, verify that the pods are still running and not experiencing any restarts due to insufficient resources:

kubectl get pods -n cost-optimization

Expected output should show all three pods in the Running state with zero restarts:

NAME                                   READY   STATUS    RESTARTS   AGE
overprovisioned-nginx-5c9b8d7f6-xyz12  1/1     Running   0          2m
overprovisioned-nginx-5c9b8d7f6-abc34  1/1     Running   0          2m
overprovisioned-nginx-5c9b8d7f6-def56  1/1     Running   0          2m

Finally, run the top command again to see the new utilization percentages. The CPU and memory usage should now represent a much higher percentage of the requested resources, indicating efficient packing and accurate right-sizing.

Common Mistakes

When optimizing Kubernetes resources, several pitfalls can lead to application instability or missed savings. Understanding why these mistakes happen is crucial for avoiding them.

Mistake 1: Setting CPU limits equal to requests. Many engineers assume that setting limits equal to requests guarantees performance. However, for CPU, this prevents the pod from bursting. If your application experiences a brief spike in traffic, it will be heavily throttled by the kernel’s Completely Fair Scheduler, leading to high latency and timeouts. CPU limits should generally be set higher than requests, or omitted entirely if the cluster is properly sized and you rely on requests for scheduling.

Mistake 2: Right-sizing based on average usage. If you look at a one-hour average of CPU usage, you might see a flat line. But applications have spikes. If you right-size based on the average, you will truncate the spikes, leading to CPU throttling or memory exhaustion. Always analyze the 95th or 99th percentile of resource usage over a period of at least a week to capture true peak behavior.

Mistake 3: Ignoring JVM and application-specific memory overhead. When right-sizing Java applications, looking only at the operating system memory metrics can be dangerous. The Java Virtual Machine allocates heap memory, but it also uses off-heap memory for metaspace, thread stacks, and direct buffers. If you right-size the memory request too close to the observed OS usage, a minor spike in off-heap usage will trigger an Out Of Memory kill. Always leave a generous buffer for JVM-based workloads or configure the JVM to explicitly limit its heap size relative to the container limits.

Mistake 4: Aggressively right-sizing stateful databases. Databases like PostgreSQL or Redis rely heavily on memory for caching data. If you right-size their memory requests based on current usage, you artificially cap their cache size, forcing them to read from disk and destroying performance. Stateful workloads require a different right-sizing strategy that accounts for their intended cache capacity, not just their current footprint.

Going Further

Manual right-sizing using kubectl top is an excellent starting point, but production environments require automation and deeper visibility.

Vertical Pod Autoscaler (VPA): The VPA can automatically analyze historical metrics and recommend or apply new resource requests. In recommendation mode, it simply suggests values without modifying the pods, allowing you to review and apply them via your CI/CD pipeline. In auto mode, it will evict and recreate pods with new requests. While powerful, VPA auto mode can cause brief disruptions, so it is often used in recommendation mode for critical production workloads.

Node-Level Optimization with Karpenter: Right-sizing pods is only half the battle. Once your pods are efficiently packed, you need a node provisioner that matches the exact compute requirements of those pods. Tools like Karpenter evaluate the pending pods and provision the cheapest, most appropriately sized underlying cloud instances, avoiding the waste inherent in traditional node groups.

Financial Visibility with OpenCost: To truly manage your bill, you need to map Kubernetes resource consumption back to actual cloud costs. OpenCost is an open-source project that provides real-time cost monitoring and allocation for Kubernetes environments. It allows you to see exactly how much a specific namespace, deployment, or label costs per hour, enabling chargebacks and informed architectural decisions.

Managing Idle Environments: For non-production workloads, manual deletion is often forgotten. Implement automated scheduling tools or cluster policies that automatically scale down development and staging namespaces to zero replicas during nights and weekends, resuming them automatically before the workday begins.

Cleanup

To remove the resources created during this walkthrough and restore your cluster to its previous state, delete the namespace. This will automatically remove all deployments, pods, and associated resources within it.

kubectl delete namespace cost-optimization

If you installed the Metrics Server specifically for this exercise and wish to remove it, you can delete its components using the manifest you applied, though it is generally recommended to keep the Metrics Server running for ongoing cluster observability and autoscaling capabilities.

Architecture

References

• OpenCost
• Managing Resources for Containers

---

This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.