Karpenter for Right-Sized Nodes on Demand

ByShubham Mishra

Intermediate · 12 min read

The Problem

Imagine it is a busy Friday afternoon. Your e-commerce platform is preparing for a weekend flash sale. In the “old way” of managing Kubernetes clusters, your team relies on the Cluster Autoscaler. You have pre-configured node groups with fixed instance types (e.g., m5.xlarge) and set up scaling policies based on CPU thresholds. The logic is simple: if CPU usage exceeds 70%, add a node.

However, as the traffic spike begins, something unexpected happens. A new microservice, optimized for high-throughput data processing, starts deploying pods that are memory-intensive but CPU-light. The Cluster Autoscaler sees low CPU utilization on existing nodes and decides not to scale out because the threshold isn’t met. Meanwhile, pods are pending because there isn’t enough memory capacity on the current nodes to schedule them.

You wait. Minutes tick by. Users start seeing timeout errors. Eventually, you manually intervene, spinning up more nodes, but they are the wrong size—too small to hold the workload efficiently or too large, wasting money. Once the sale ends, the nodes sit idle for hours, burning cash, before the autoscaler finally decides to terminate them.

This scenario highlights two critical pain points:

  1. Scheduling Latency: Traditional autoscalers are reactive and often slow to provision the right hardware, leading to pod scheduling delays.
  2. Inefficient Resource Allocation: Fixed node sizes force developers to guess resource needs. Over-provisioning wastes money; under-provisioning causes failures. This is known as “bin-packing inefficiency.”

Karpenter solves this by decoupling node provisioning from static node groups. Instead of telling Kubernetes “add an m5.xlarge,” you tell Karpenter “I need resources,” and it dynamically provisions the exact instance type and size required, right when you need it.

Core Concepts

To understand Karpenter, we must first distinguish it from the traditional Kubernetes Autoscaler. Karpenter is an open-source node provisioning project specifically built for Kubernetes. It operates at a different layer of abstraction than standard cluster autoscalers.

Provisioner

The Provisioner is the core configuration object in Karpenter. It acts as a rulebook. It defines the constraints for what kind of nodes can be created. For example, a Provisioner might specify that only EC2 instances in the us-east-1 region are allowed, or that only spot instances should be used for cost savings. If a pod’s requirements cannot be met by any combination of nodes matching the Provisioner’s rules, Karpenter will not create a node, and the pod will remain pending.

Node Classes (NodeClaim)

In modern Karpenter versions, we use EC2NodeClass (or similar classes for other cloud providers). This object encapsulates the infrastructure details: the AMI (Amazon Machine Image), the IAM Role the nodes assume, the subnet IDs where nodes can be launched, and tags. It essentially defines the “shape” of the infrastructure.

Instance Types and Diversity

Unlike traditional autoscalers that might stick to one or two instance types, Karpenter queries the cloud provider’s API for available instance types in real-time. It evaluates dozens of options (CPU, memory, network performance, price) to find the most efficient fit for the pending workloads. This diversity allows Karpenter to launch smaller, cheaper instances for light workloads and larger ones for heavy tasks, rather than forcing all workloads into a single node size.

Just-in-Time (JIT) Provisioning

Karpenter uses a JIT approach. It does not keep a pool of warm nodes waiting for work. Instead, it waits until a pod is scheduled and fails to find a node with sufficient resources. Only then does it trigger the creation of a new node. This ensures that you pay strictly for the compute power you are actively using, minimizing waste.

Consolidation

When pods finish their work and are terminated, Karpenter checks if the node still has value. If the remaining pods on a node could fit onto other already-running nodes (a process called bin-packing), Karpenter will cordon and drain the empty node, terminating it to save costs. This is the opposite of the “scale-out only” mentality of older tools.

How It Works

Understanding the mechanics of Karpenter requires looking at the control loop. Here is the step-by-step lifecycle of a request:

  1. Pod Scheduling Failure: A user deploys a Deployment. Kubernetes’ scheduler attempts to place the pod on a node. It scans available nodes but finds none with enough memory/CPU to satisfy the pod’s requests.
  2. Event Trigger: Karpenter’s controller watches for these unschedulable pods. It identifies that the current cluster state cannot satisfy the demand.
  3. Evaluation: Karpenter looks at the associated Provisioner. It filters out instance types that violate the constraints (e.g., wrong region, no GPU support if requested).
  4. Bid Calculation: Karpenter calculates a “bid” for various instance types. It considers:
    • Price: On-demand vs. Spot pricing.
    • Capacity: Does the instance type have enough vCPUs and Memory?
    • Diversity: Does launching this instance type help diversify the fleet? (Diversity helps prevent correlated failures across availability zones).
  5. Node Creation: Karpenter sends a request to the Cloud Provider API (e.g., AWS EC2 RunInstances) to create the node. It injects the necessary metadata, including the IAM role and user-data scripts to join the cluster.
  6. Node Join: The new instance boots up, runs the kubelet agent, and joins the Kubernetes cluster as a ready node.
  7. Scheduling: The Kubernetes scheduler now sees a node with sufficient resources and places the pending pod onto it.
  8. Consolidation (Later): When the pod finishes, Karpenter detects the node is underutilized. If consolidation is enabled, it terminates the node.

Prerequisites

Before implementing Karpenter, ensure you have the following environment ready:

  • A Running Kubernetes Cluster: This guide assumes an Amazon EKS (Elastic Kubernetes Service) cluster, as Karpenter’s native integration is deepest with AWS. However, concepts apply to other clouds with appropriate NodeClasses.
  • kubectl Access: You must have administrative access to the cluster.
  • Helm Installed: We will use Helm to deploy Karpenter, as it simplifies managing the many components involved.
  • iamserviceaccount Permissions: You need permissions to create IAM roles and service accounts for the nodes. In EKS, this usually involves the eksctl tool or manual IAM role creation.
  • Version Assumption: This tutorial assumes Karpenter version 0.32+ and Kubernetes 1.24+. Behavior regarding CRDs and IAM roles may vary in older versions.

Hands-On Walkthrough

We will now configure Karpenter to manage our cluster. We will create a scenario where a specific workload requires high memory, and Karpenter will automatically provision an instance type suited for that.

Step 1: Create the IAM Role for Nodes

Karpenter nodes need permission to call AWS APIs (like requesting spot instances). We create an IAM role attached to the worker nodes.

# Using eksctl to create the role
eksctl create iamserviceaccount \
    --name karpenter \
    --namespace karpenter \
    --cluster my-cluster \
    --attach-policy-arn arn:aws:iam::policy/AmazonSSMManagedInstanceCore \
    --override-existing-serviceaccounts \
    --approve

Step 2: Install Karpenter via Helm

Add the Karpenter repository and install the chart. Ensure you replace <cluster-name> and <region> with your actual values.

helm repo add karpenter https://charts.karpenter.sh
helm repo update
helm install karpenter karpenter/karpenter \
  --namespace karpenter \
  --create-namespace \
  --set settings.clusterName=my-cluster \
  --set settings.interruptionQueue=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=karpenter \
  --wait

Step 3: Create the EC2NodeClass

This manifest defines the infrastructure properties. We will use a mix of on-demand and spot instances for cost efficiency.

apiVersion: karpenter.sh/v1alpha5
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AmazonLinux2
  role: "karpenter-my-cluster"
  subnetSelectorTerms:
    - tags:
        Name: private-*
  securityGroupSelectorTerms:
    - tags:
        Env: production
  tags:
    Project: ecommerce
    ManagedBy: karpenter
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 20Gi
        volumeType: gp3
  metadataOptions:
    httpEndpoint: enabled
    httpTokens: required

Explanation:

  • amiFamily: Tells Karpenter to use the latest Amazon Linux 2 AMI compatible with your Kubernetes version.
  • role: Links to the IAM role created in Step 1.
  • subnetSelectorTerms: Ensures nodes are launched in private subnets for security.
  • metadataOptions: Enforces IMDSv2 (Instance Metadata Service version 2) for enhanced security against SSRF attacks.

Step 4: Create the Provisioner

The Provisioner tells Karpenter how to behave. We will set limits to prevent runaway costs.

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand", "spot"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m5.large", "m5.xlarge", "c5.xlarge"]
  limits:
    resources:
      cpu: "100"
      memory: 500Gi
  disruption:
    consolidateAfter: 5m
    budgets:
      - duration: 1h
        nodes: "10%"

Explanation:

  • requirements: Restricts Karpenter to choose from a specific list of instance types. This prevents it from picking extremely expensive instances unless necessary.
  • limits: Caps the total CPU and Memory the cluster can provision. This is your safety net against bill shock.
  • disruption.consolidateAfter: After a node has been stable for 5 minutes, Karpenter will attempt to consolidate workloads onto fewer nodes, terminating empty ones.

Step 5: Deploy a Test Workload

Now, let’s deploy a pod that requests resources outside the typical range, forcing Karpenter to act.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: memory-hog-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memory-hog
  template:
    metadata:
      labels:
        app: memory-hog
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: karpenter.sh/provisioner-name
                    operator: In
                    values: ["default"]
      containers:
        - name: nginx
          image: nginx
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "4"
              memory: "16Gi"

Note: The nodeAffinity rule ensures that only nodes managed by our Karpenter Provisioner can host this pod. Without this, standard Kubernetes nodes might try to accept the pod, fail, and cause confusion.

Verifying It Works

Once the deployment is applied, observe the behavior.

  1. Check Pod Status: 
    kubectl get pods -l app=memory-hog

    You should see the pod transition from Pending to Running. If it stays pending, check events with kubectl describe pod <pod-name>.

  2. Observe Node Creation: 
    watch kubectl get nodes -o wide

    Within seconds (usually 30-60 seconds), a new node should appear. Check the INTERNAL-IP and verify it wasn’t there before.

  3. Inspect Karpenter Logs: 
    kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

    Look for lines containing provisioned. You will see which instance type was chosen (e.g., m5.2xlarge) and why.

  4. Verify Consolidation:
    Delete the deployment after a few minutes. Watch the nodes disappear. Karpenter should detect that the node is no longer needed and terminate it, freeing up resources.

Common Mistakes

Even experienced DevOps engineers stumble when adopting Karpenter. Here are the most frequent pitfalls:

1. Ignoring Instance Type Restrictions

Why it happens: By default, Karpenter may select any instance type available in the region, including rare or expensive ones like p4d (GPU instances).
Avoidance: Always define explicit requirements in your Provisioner to limit the scope to general-purpose or compute-optimized families unless you specifically need GPUs. Use operator: NotIn to exclude massive, expensive instances.

2. Misconfiguring Node Affinity

Why it happens: New users often forget to add the nodeAffinity label to their pods.
Avoidance: If you don’t restrict affinity, standard Kubernetes scheduling logic applies. If your existing nodes are full, the pod will stay pending. Karpenter will still spin up a node, but it might be confusing to debug why the old nodes didn’t take the load. Explicit affinity makes the intent clear: “Only Karpenter-managed nodes handle this.”

3. Overlooking Disruption Budgets

Why it happens: Enabling consolidation without setting budgets.
Avoidance: Aggressive consolidation can cause downtime during rolling updates or maintenance windows. Always set budgets in the Provisioner to limit how many nodes can be disrupted simultaneously (e.g., “10%” or “-1” for unlimited but careful management).

4. Assuming Static IP Addresses

Why it happens: Applications relying on hard-coded IP addresses of workers.
Avoidance: Karpenter nodes are ephemeral. They are born and die frequently. Never rely on node IPs. Use Services and Ingresses for external access, and StatefulSets with persistent volumes for stateful data.

Going Further

With the basics working, consider these production-grade enhancements:

Spot Instance Interruption Handling

Spot instances can be reclaimed by AWS with a 2-minute warning. Karpenter handles this gracefully by draining the node before termination. However, your applications must be resilient. Use terminationGracePeriodSeconds in your pods to allow graceful shutdowns. Consider using topologySpreadConstraints to spread replicas across different AZs to mitigate localized spot interruptions.

Cost Optimization with Bid Shading

Karpenter supports bid shading for spot instances. By configuring the Provisioner to use lower bids, you can reduce costs significantly, though at the risk of higher interruption rates. Monitor your spot interruption rate metrics in AWS CloudWatch to find the sweet spot between cost and stability.

Multi-Provisioner Strategies

For complex architectures, use multiple Provisioners. One Provisioner might handle general web traffic (using cheap spot instances), while another handles database workloads (using guaranteed on-demand instances with specific IOPS storage classes). This isolates risks and optimizes costs per workload type.

Alternative Approaches

Cluster Autoscaler: Still relevant for hybrid environments or non-AWS clouds. It is more mature but less flexible than Karpenter. Trade-off: CA is slower to provision but offers finer-grained control over scaling policies tied to HPA (Horizontal Pod Autoscaler).

Kube-bench/Kyverno: While not provisioning tools, integrate policy engines to enforce security standards on Karpenter-created nodes automatically.

Cleanup

To remove everything created in this tutorial and avoid ongoing charges:

# Delete the test workload
kubectl delete deployment memory-hog-app

# Wait for nodes to be consolidated and terminated
sleep 300

# Uninstall Karpenter
helm uninstall karpenter -n karpenter

# Delete the Karpenter namespace
kubectl delete namespace karpenter

# Delete the IAM service account
eksctl delete iamserviceaccount --name karpenter --cluster my-cluster --approve

# Delete the EC2NodeClass and Provisioner manifests if applied directly
kubectl delete ec2nodeclass default
kubectl delete provisioner default

By moving from static node groups to Karpenter’s dynamic provisioning, you transform your Kubernetes infrastructure from a rigid, costly asset into a fluid, efficient engine. You stop guessing about node sizes and start letting the system react to actual demand in real-time.

References

This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *