Right-Sizing Kubernetes Resource Requests and Limits Without Guessing

Intermediate · 5 min read

The Problem

One of the most common challenges in managing Kubernetes clusters is right-sizing resource requests and limits for your applications. Incorrectly configured resources can lead to performance issues, outages, and unnecessary costs. For example, imagine a DevOps team at a growing e-commerce platform. They have noticed that their application is frequently crashing during peak traffic times, and they are unsure if it’s due to insufficient resources or misconfiguration. This scenario highlights the importance of accurately setting resource requests and limits to ensure reliable and cost-effective operations.

Core Concepts

Before diving into the mechanics, let’s define some key terms:

• Resource Requests: The minimum amount of resources (CPU and memory) that a container needs to start. If the node doesn’t have enough free resources to meet the request, the pod will not be scheduled on that node.
• Resource Limits: The maximum amount of resources (CPU and memory) that a container can use. If a container exceeds its limit, it may be throttled or terminated.
• Pod: The smallest deployable unit in Kubernetes, which can contain one or more containers.
• Node: A worker machine in a Kubernetes cluster that runs pods.

Understanding these concepts is crucial for effectively managing your Kubernetes environment.

How It Works

Kubernetes uses resource requests and limits to make scheduling decisions and to manage the quality of service (QoS) for pods. When a pod is created, Kubernetes checks the available resources on each node to determine where the pod can be placed. If a node has enough free resources to meet the pod’s requests, the pod is scheduled on that node. If a pod exceeds its resource limits, Kubernetes may throttle the pod’s resource usage or even terminate it to maintain the stability of the cluster.

Prerequisites

Before you begin, you should have the following:

• A running Kubernetes cluster (e.g., Minikube, GKE, EKS, AKS)
• kubectl command-line tool installed and configured to interact with your cluster
• Basic understanding of Kubernetes concepts (pods, nodes, deployments, etc.)

Hands-On Walkthrough

Let’s walk through an example of how to set and adjust resource requests and limits for a simple web application.

Step 1: Create a Basic Deployment

First, create a basic deployment for a web application. Save the following YAML as nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Apply the deployment using kubectl:

kubectl apply -f nginx-deployment.yaml

Step 2: Set Initial Resource Requests and Limits

Now, let’s set initial resource requests and limits for the Nginx container. Update the nginx-deployment.yaml file with the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

Apply the updated deployment:

kubectl apply -f nginx-deployment.yaml

Step 3: Monitor and Adjust Resources

Monitor the resource usage of the Nginx pod using the following command:

kubectl top pod <pod-name>

If the pod consistently uses more resources than the initial settings, you can adjust the requests and limits. For example, increase the CPU and memory limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 150m
            memory: 192Mi
          limits:
            cpu: 300m
            memory: 512Mi

Apply the updated deployment:

kubectl apply -f nginx-deployment.yaml

Verifying It Works

To verify that the resource requests and limits are correctly set, you can use the following commands:

kubectl get pod <pod-name> -o yaml | grep -i 'resources:' -A 5

This command will show the resource requests and limits for the pod. Additionally, you can check the current resource usage:

kubectl top pod <pod-name>

Expected output should show the CPU and memory usage within the set limits.

Common Mistakes

Here are some common mistakes to avoid when setting resource requests and limits:

• Over-provisioning: Setting excessively high resource requests and limits can lead to underutilized nodes and increased costs. Always start with conservative estimates and adjust based on actual usage.
• Under-provisioning: Setting too low resource requests and limits can cause performance issues and frequent pod restarts. Monitor resource usage and adjust as needed.
• Ignoring QoS Classes: Kubernetes assigns a Quality of Service (QoS) class to each pod based on its resource requests and limits. Pods with no requests or limits are assigned the BestEffort QoS class, which can lead to unpredictable behavior. Always set both requests and limits.

Going Further

For production environments, consider the following advanced topics:

• Scaling: Use Horizontal Pod Autoscaler (HPA) to automatically scale the number of pods based on resource usage. This can help handle varying loads without manual intervention.
• Security: Implement pod security policies and network policies to restrict access and ensure that pods only have the necessary permissions and network connectivity.
• Performance Tuning: Use tools like Prometheus and Grafana for monitoring and alerting. Regularly review and optimize resource configurations based on historical data and performance metrics.
• Alternative Approaches: Consider using tools like VPA (Vertical Pod Autoscaler) to automatically adjust resource requests and limits based on real-time usage. This can reduce the need for manual tuning but requires careful configuration and testing.

Cleanup

To remove the deployment and clean up, run the following command:

kubectl delete -f nginx-deployment.yaml

Architecture

References

• Managing Resources for Containers
• OpenCost

---

This article was written with AI assistance.