The Silent Crash: Safely Draining Nodes Before Maintenance to Prevent Pod Eviction Storms

Intermediate · 11 min read

The Problem

It is 2:00 AM. The on-call engineer receives an alert: High Error Rate on Payment Service. Upon logging into the Kubernetes dashboard, the scene is chaotic. Pods are crashing, restarting, and flapping between CrashLoopBackOff and Running states. The root cause? A routine kernel patching cycle was initiated on a worker node, but the node was not drained properly before the reboot.

In this scenario, the node controller detected the node as NotReady because it stopped responding to heartbeats. Without a graceful eviction process, the API server assumed all pods on that node had failed and immediately scheduled replacements on other nodes. This triggered a “thundering herd” effect. The remaining nodes in the cluster were suddenly flooded with dozens of new pod scheduling requests, exhausting their available CPU and memory resources. New pods couldn’t start, existing pods got OOMKilled due to resource contention, and the entire service degraded or went down entirely.

This guide addresses the critical pain point of performing maintenance on Kubernetes worker nodes without causing service disruption. It moves beyond simple commands to explain the mechanics of safe eviction, ensuring that your infrastructure changes do not become outages.

Core Concepts

To understand why draining fails or succeeds, we must define the underlying mechanisms of Kubernetes scheduling and lifecycle management. These concepts form the foundation of safe node maintenance.

Node Conditions and Status

Every node in a Kubernetes cluster reports its status to the API Server via the Kubelet agent. The two most relevant conditions for this topic are:

• Ready: Indicates the node is capable of accepting workloads. If a node stops sending heartbeats (usually within 40 seconds by default), the condition flips to NotReady.
• MemoryPressure/CPUPressure: Indicates the node is running low on resources. While related, these are distinct from the readiness state caused by network partitioning or shutdown.

Pod Disruption Budgets (PDBs)

A PDB is a policy object that specifies the minimum number of replicas of a workload that must remain available during voluntary disruptions. For example, if a deployment has 5 replicas and a PDB requires at least 3 to be available, Kubernetes will refuse to evict more than 2 pods simultaneously. PDBs are the primary safety net preventing total application downtime during node drains.

Eviction vs. Termination

There is a critical distinction between how Kubernetes handles unexpected failures versus planned maintenance:

• Termination (SIGTERM): When a pod is gracefully removed, the kubelet sends a SIGTERM signal to the main process. The container has a grace period (default 30 seconds) to finish processing requests, flush data, and exit cleanly. The API Server then removes the pod object.
• Eviction: This is the formal API call to remove a pod from a node. An eviction can be graceful (respecting PDBs and grace periods) or forced (bypassing them). During a node drain, Kubernetes attempts graceful evictions first.

The Cordon Mechanism

Cordoning a node sets a taint (NoSchedule) on the node. This prevents new pods from being scheduled onto the node. However, cordoning alone does not remove existing pods. You must combine cordoning with draining to achieve a clean slate.

How It Works

The process of draining a node is orchestrated by the kubectl drain command, which interacts with multiple Kubernetes components. Here is the step-by-step mechanical flow:

1. Tainting: The command adds a taint node.kubernetes.io/unschedulable:NoSchedule to the target node. The scheduler sees this taint and rejects any new scheduling decisions for this node.
2. Listing Pods: The tool queries the API Server for all pods owned by the user (excluding daemonsets and system-critical pods like kube-system components, unless specified otherwise).
3. PDB Validation: Before evicting a pod, the tool checks if there is an associated PodDisruptionBudget. If evicting this pod would violate the PDB (i.e., drop active replicas below the minimum), the eviction is skipped, and the tool waits or fails.
4. Graceful Eviction: For each eligible pod, an eviction request is sent to the API Server. The API Server triggers the termination process:
• The Kubelet sends SIGTERM to the pod’s containers.
• The Kubelet waits for the terminationGracePeriodSeconds defined in the pod spec (or the global default of 30s).
• If the pod exits within the grace period, it is removed.
• If the pod does not exit, the Kubelet sends SIGKILL, forcefully terminating it.
5. Force Option: If the --force flag is used, PDB checks are bypassed, and the grace period may be shortened. This should only be used when the node is unresponsive or in emergency scenarios.

Prerequisites

Before attempting to drain nodes in production, ensure you have the following:

• Kubectl Access: Administrative access to the cluster with permissions to manage nodes and pods.
• Cluster Awareness: Knowledge of which nodes require maintenance (e.g., specific hardware issues, OS updates).
• Application Health Checks: Your applications must implement proper shutdown hooks. If your app ignores SIGTERM and crashes instead of exiting cleanly, graceful drains will fail or result in data loss.
• PDB Configuration: Verify that critical services have appropriate PDBs configured. Services without PDBs are vulnerable to complete removal during a drain.
• Network Stability: Ensure stable connectivity between the control plane and the worker node being drained. Network latency can cause timeout errors during eviction.

Hands-On Walkthrough

We will simulate a safe node maintenance procedure using a test cluster. We assume a cluster with three worker nodes: worker-1, worker-2, and worker-3.

Step 1: Inspect Current State

First, identify the node you intend to maintain. Let’s say worker-1 needs a reboot for kernel patches.

kubectl get nodes -o wide

Verify that the node is currently Ready and check the pods running on it:

kubectl get pods --field-selector spec.nodeName=worker-1 -A

Step 2: Check for Pod Disruption Budgets

Before proceeding, check if any pods on this node are protected by PDBs. This helps predict potential bottlenecks.

kubectl get pdb -A | grep -E "(min-available|max-unavailable)"

Step 3: Execute the Drain Command

Run the drain command with best-practice flags. We will use --ignore-daemonsets because DaemonSets (like logging agents) are designed to run on every node and cannot be easily rescheduled elsewhere. We also use --delete-emptydir-data to handle local storage, though caution is advised here.

kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-local-data \
  --force \
  --timeout=120s

Explanation of Flags:

• --ignore-daemonsets: Skips DaemonSet pods. Without this, the drain might hang indefinitely trying to evict essential system daemons.
• --delete-local-data: Deletes data in emptyDir volumes. Use this with extreme caution; emptyDir data is lost permanently upon pod deletion.
• --force: Bypasses PDB checks. Note: In a highly sensitive production environment, omitting this flag is safer, allowing the command to wait until PDB constraints are met. Only use force if you accept the risk of violating availability SLAs.
• --timeout=120s: Sets the maximum time allowed for each pod eviction. If a pod doesn’t terminate in 120 seconds, kubectl moves on. This prevents one stuck pod from halting the entire maintenance window.

Step 4: Handle Failures

If the command returns an error such as cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet, it means some pods are running as bare pods or static pods. These typically reside in the kube-system namespace and should generally be left alone unless you are performing deep node-level maintenance.

Step 5: Perform Maintenance

Once the drain completes successfully, the node will be marked as SchedulingDisabled. You can now SSH into the node and perform your updates:

sudo apt-get update && sudo apt-get upgrade -y
# Or your specific OS patching command

Verifying It Works

After maintenance is complete, you must verify the node is healthy and rejoin it to the cluster.

1. Uncordon the Node

Remove the unschedulable taint:

kubectl uncordon worker-1

2. Verify Readiness

Check the node status:

kubectl get node worker-1

Expected output should show Ready. If it shows NotReady, check the kubelet logs on the node itself:

journalctl -u kubelet -f

3. Confirm Pod Rescheduling

Ensure your application pods have been rescheduled correctly. Since we uncordoned the node, new pods may schedule back onto worker-1 if resources allow. However, usually, the scheduler will distribute them across all available nodes.

kubectl get pods -o wide | grep worker-1

4. Application Health Check

Run your application health checks to ensure no data corruption occurred during the eviction. Check logs for any connection refused or timeout errors during the transition period.

Common Mistakes

Even experienced DevOps engineers fall into these traps. Understanding why they happen is crucial.

Mistake 1: Draining Without PDBs

Why it happens: Teams often focus on infrastructure stability while neglecting application configuration. They drain a node containing a single-replica database or a critical microservice with no redundancy.

Consequence: The service becomes unavailable for the duration of the pod termination and restart. If the restart takes longer than the load balancer’s health check timeout, traffic is routed to dead endpoints, causing user-facing errors.

Solution: Mandate PDBs for all stateful and critical stateless services. Review PDB thresholds quarterly.

Mistake 2: Using Force Too Aggressively

Why it happens: Impatient engineers want to clear nodes quickly and use --force to bypass waiting periods.

Consequence: This kills pods abruptly, potentially corrupting databases or leaving transactions in an inconsistent state. It also violates the principle of graceful degradation.

Solution: Reserve --force for emergency situations where the node is completely unresponsive. Always prefer graceful eviction in standard maintenance windows.

Mistake 3: Ignoring DaemonSets

Why it happens: Assuming all pods can be moved. DaemonSets are tied to specific nodes for tasks like log collection, monitoring agents, or node-level networking plugins.

Consequence: Attempting to evict DaemonSets can lead to errors or, if ignored incorrectly, leave the node in a broken state where core observability tools are missing.

Solution: Always include --ignore-daemonsets. Do not attempt to drain DaemonSet pods unless you are replacing the entire node image and understanding the implications for your monitoring stack.

Mistake 4: Not Checking Resource Availability

Why it happens: Draining a node assumes the remaining nodes have enough capacity to absorb the workload.

Consequence: If worker-1 holds 80% of the cluster’s CPU, and worker-2 and worker-3 are already at 90% utilization, the evicted pods will enter a Pending state. This leads to a cascading failure where services stop scaling out.

Solution: Check cluster resource usage before draining. Consider scaling down non-critical workloads or adding temporary nodes if the cluster is near capacity.

Going Further

For production environments, manual draining is insufficient. You need automation and advanced strategies.

Automating Drains with Cluster Autoscaler and Maintenance Policies

Integrate node maintenance with tools like cluster-api or kured (Kubernetes Reboot Daemon). Kured monitors for reboot announcements and automatically cordons, drains, and reboots nodes one by one, respecting PDBs. This ensures that only one node is drained at a time, maintaining cluster stability.

Advanced Taint Management

You can customize the drain behavior by managing taints manually. For example, you might add a custom taint maintenance=true:NoSchedule to specific nodes to trigger automated workflows in your CI/CD pipeline before running the drain command.

StatefulSet Considerations

Draining nodes hosting StatefulSets requires extra care. StatefulSets rely on stable storage and identity. If a pod is evicted, it must restart with the same volume attached. Ensure your Persistent Volumes (PVs) are accessible from other nodes in the cluster. If using local SSDs, you cannot move the pod to another node; you must wait for the original node to return. In this case, consider using --disable-eviction=false (default) and ensuring your StatefulSet has sufficient replicas spread across different zones/nodes.

Performance Tuning: Grace Periods

If your applications take longer than 30 seconds to shut down, adjust the terminationGracePeriodSeconds in your pod specs. For example, setting it to 120s allows slow databases to flush buffers. However, be aware that longer grace periods increase the total time required to drain a node.

Cleanup

If you created any test resources during this walkthrough, ensure you clean them up to avoid billing charges or cluster clutter.

# Delete the test namespace if created
kubectl delete namespace test-maintenance

# Reset node status if needed (for local dev clusters)
kubectl reset

Always verify that your cluster has returned to a healthy state with all expected pods running and no pending evictions lingering in the API server history.

References

• Safely Drain a Node

---

This article was written with AI assistance.