PriorityClasses: Keeping Critical Pods Alive When the Cluster is Under Pressure

Intermediate · 5 min read

The Problem

Imagine a scenario where a financial institution’s trading platform is running on a Kubernetes cluster. During a high-traffic event, such as a major market announcement, the cluster experiences a sudden surge in resource demand. In this situation, it’s crucial to ensure that the most critical services, like the trade execution engine, remain operational even if other less critical services need to be scaled down or terminated. This is where Kubernetes PriorityClasses come into play, allowing you to prioritize and manage the eviction of pods based on their importance.

Core Concepts

Before diving into the mechanics, let’s define some key concepts:

• Pods: The smallest deployable units in Kubernetes, representing a single instance of an application.
• Node: A worker machine in Kubernetes, which can be either a physical or virtual machine.
• Resource Requests and Limits: These are the minimum and maximum resources (CPU, memory) that a pod can request and use, respectively.
• Eviction: The process of terminating a pod to free up resources when a node is under pressure.
• PriorityClass: A way to assign a priority level to pods, determining their order of eviction during resource shortages.

How It Works

When a node in a Kubernetes cluster is under resource pressure, the kubelet (the primary node agent) will start evicting pods to free up resources. The order in which pods are evicted is determined by their priority. Pods with lower priority are evicted first, while those with higher priority are kept running for as long as possible.

Priorities are defined using PriorityClass objects. Each PriorityClass has a unique name and a priority value, which is a positive integer. Higher values indicate higher priority. Additionally, you can set a globalDefault field to specify that all pods without a specified priority should use this class.

Prerequisites

To follow this tutorial, you will need:

• A Kubernetes cluster (e.g., Minikube, GKE, EKS, AKS).
• kubectl installed and configured to interact with your cluster.
• Basic understanding of Kubernetes concepts (pods, nodes, deployments, etc.).

Hands-On Walkthrough

Let’s create a simple example to demonstrate how to use PriorityClasses.

Step 1: Create a PriorityClass

Create a file named priorityclass.yaml with the following content:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class is used for high-priority pods."

Apply the PriorityClass:

kubectl apply -f priorityclass.yaml

Step 2: Create a Deployment with High-Priority Pods

Create a file named high-priority-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: high-priority-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: high-priority-app
  template:
    metadata:
      labels:
        app: high-priority-app
    spec:
      containers:
      - name: high-priority-container
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 100m
            memory: 128Mi
      priorityClassName: high-priority

Apply the deployment:

kubectl apply -f high-priority-deployment.yaml

Step 3: Create a Deployment with Default-Priority Pods

Create a file named default-priority-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-priority-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: default-priority-app
  template:
    metadata:
      labels:
        app: default-priority-app
    spec:
      containers:
      - name: default-priority-container
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 100m
            memory: 128Mi

Apply the deployment:

kubectl apply -f default-priority-deployment.yaml

Verifying It Works

To verify that the PriorityClasses are working as expected, you can simulate resource pressure on a node. For simplicity, we’ll use the kubectl drain command to cordon and drain a node, which will cause the kubelet to start evicting pods.

First, list the nodes in your cluster:

kubectl get nodes

Select a node and drain it:

kubectl drain <node-name> --ignore-daemonsets --force

Check the status of the pods:

kubectl get pods -o wide

You should see that the high-priority pods are still running, while the default-priority pods have been evicted.

Common Mistakes

Here are some common mistakes to avoid when using PriorityClasses:

• Not setting a globalDefault: If you don’t set a globalDefault, pods without a specified priority will have a default priority of zero, which might not be what you want.
• Overusing high priorities: Assigning too many pods to high-priority classes can lead to situations where no pods can be evicted, causing the node to become unstable.
• Ignoring resource requests and limits: Even with high priority, pods can still be evicted if they exceed their resource limits or if the node doesn’t have enough resources to meet their requests.

Going Further

In production, there are several considerations and advanced topics to explore:

• Scaling: As your cluster grows, you may need to adjust the number of high-priority pods to ensure that they don’t overwhelm the available resources.
• Security: Prioritize security-critical pods, such as those handling authentication and authorization, to ensure they remain operational during resource shortages.
• Performance Tuning: Regularly monitor and tune the resource requests and limits for your pods to ensure they are balanced and efficient.
• Alternative Approaches: Consider using PodDisruptionBudgets to control the disruption of pods during voluntary disruptions, such as rolling updates and maintenance.

Cleanup

To clean up the resources created in this tutorial, run the following commands:

kubectl delete -f priorityclass.yaml
kubectl delete -f high-priority-deployment.yaml
kubectl delete -f default-priority-deployment.yaml

Architecture

References

• Pod Priority and Preemption

---

This article was written with AI assistance.