Karpenter for Right-Sized Nodes on Demand: A Before-and-After Story

Intermediate · 6 min read

The Problem

Imagine you are a DevOps engineer at a rapidly growing e-commerce company. Your team has been struggling with the Kubernetes cluster’s performance, especially during peak traffic times. The current setup involves manually provisioning and managing nodes, which often leads to over-provisioning or under-provisioning. This results in either wasted resources or performance bottlenecks, impacting the user experience and increasing operational costs.

In this scenario, the pain point is clear: the need for a more efficient and dynamic way to manage Kubernetes nodes to ensure optimal resource utilization and performance. This is where Karpenter comes in, offering a solution to right-size your nodes on demand.

Core Concepts

Before diving into Karpenter, let’s cover some fundamental concepts:

• Kubernetes (K8s): An open-source platform for automating deployment, scaling, and management of containerized applications. It provides a way to run and manage containers across multiple hosts.
• Nodes: Physical or virtual machines that run the applications and services. In Kubernetes, nodes are the worker machines that run the pods (containers).
• Pods: The smallest deployable units in Kubernetes. A pod can contain one or more containers that share storage, network, and other resources.
• Provisioning: The process of allocating and configuring resources, such as nodes, to meet the demands of the application.
• Karpenter: An open-source project that automatically provisions and deprovisions EC2 instances in response to the changing needs of your Kubernetes workloads. It ensures that your cluster always has the right number and type of nodes to handle the load efficiently.

How It Works

Karpenter works by continuously monitoring the Kubernetes cluster for unschedulable pods. When it detects pods that cannot be scheduled due to insufficient resources, it automatically provisions new nodes with the appropriate instance types and configurations. Once the pods are scheduled, Karpenter continues to monitor the cluster and deprovisions nodes that are no longer needed, ensuring that the cluster remains cost-effective and efficient.

Here’s a step-by-step overview of how Karpenter operates:

1. Monitoring: Karpenter watches the Kubernetes API for unschedulable pods.
2. Provisioning: When unschedulable pods are detected, Karpenter creates new EC2 instances with the required specifications.
3. Scheduling: The newly provisioned nodes are added to the cluster, and the unschedulable pods are scheduled onto them.
4. Deprovisioning: Karpenter continuously monitors the cluster and deprovisions nodes that are idle or underutilized, maintaining an optimal balance of resources.

Prerequisites

Before you start using Karpenter, you need to have the following in place:

• A running Kubernetes cluster: You should have a Kubernetes cluster up and running. For this tutorial, we will assume you are using Amazon EKS (Elastic Kubernetes Service).
• Amazon EKS: If you are not already using EKS, you will need to set up an EKS cluster. Follow the official AWS documentation to create an EKS cluster.
• AWS CLI and kubectl: Ensure you have the AWS Command Line Interface (CLI) and kubectl installed and configured on your local machine.
• IAM Roles and Policies: Karpenter requires specific IAM roles and policies to interact with AWS services. You will need to create these roles and policies as part of the setup process.

Hands-On Walkthrough

Let’s walk through the steps to set up Karpenter in your EKS cluster.

Step 1: Install Karpenter

To install Karpenter, you will use the Helm chart provided by the Karpenter project. First, add the Karpenter Helm repository:

helm repo add karpenter https://karpenter.sh

Next, update the Helm repositories:

helm repo update

Install Karpenter using the Helm chart. Replace <cluster-name> and <region> with your EKS cluster name and region:

helm install karpenter karpenter/karpenter --namespace karpenter --create-namespace \\
  --set aws.clusterName=<cluster-name> \\
  --set aws.region=<region> \\
  --set serviceAccount.annotations."eks.amazonaws.com/role-arn"="arn:aws:iam::<account-id>:role/KarpenterControllerRole"

Step 2: Configure Karpenter

Create a Karpenter configuration file, karpenter-config.yaml, to define the settings for Karpenter. Here is an example configuration:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
  limits:
    resources:
      cpu: 1000
      memory: 4Gi
  provider:
    instanceProfile: KarpenterNodeInstanceProfile-
    subnetSelector:
      karpenter.sh/discovery: <cluster-name>
    securityGroupSelector:
      karpenter.sh/discovery: <cluster-name>

Apply the configuration to your cluster:

kubectl apply -f karpenter-config.yaml

Step 3: Deploy a Sample Application

To see Karpenter in action, deploy a sample application that will trigger the provisioning of new nodes. Create a file, sample-app.yaml, with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Deploy the sample application:

kubectl apply -f sample-app.yaml

Verifying It Works

To verify that Karpenter is working correctly, you can check the status of the nodes and pods in your cluster. Use the following commands:

kubectl get nodes
kubectl get pods -o wide

You should see new nodes being provisioned and the sample application pods being scheduled onto these nodes. Additionally, you can check the Karpenter logs to see the provisioning and deprovisioning activities:

kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter

Common Mistakes

Here are some common mistakes to avoid when setting up and using Karpenter:

• Incorrect IAM Roles and Policies: Karpenter requires specific IAM roles and policies to function correctly. If these are not set up properly, Karpenter will not be able to provision or deprovision nodes. Make sure to follow the official Karpenter documentation to create the necessary IAM roles and policies.
• Insufficient Resource Limits: If the resource limits in the Karpenter configuration are too restrictive, Karpenter may not be able to provision nodes with the required specifications. Review and adjust the resource limits as needed to ensure that Karpenter can provision nodes with the appropriate resources.
• Mismatched Instance Types: Karpenter may not be able to find suitable instance types if the requirements in the Karpenter configuration do not match the available instance types in your AWS region. Verify that the instance types specified in the Karpenter configuration are available and supported in your region.

Going Further

Once you have Karpenter up and running, there are several production considerations and advanced topics to explore:

• Scaling: Karpenter can be used to scale your cluster dynamically based on the workload. You can configure Karpenter to automatically scale the number of nodes up or down depending on the demand. This can help you optimize resource utilization and reduce costs.
• Security: Ensure that the IAM roles and policies used by Karpenter are properly secured. Limit the permissions to the minimum required for Karpenter to function. Regularly review and audit the IAM roles and policies to prevent unauthorized access.
• Performance Tuning: Fine-tune the Karpenter configuration to optimize performance. Adjust the resource limits, instance types, and other parameters to ensure that Karpenter can provision nodes quickly and efficiently. Monitor the performance of your cluster and make adjustments as needed.
• Alternative Approaches: While Karpenter is a powerful tool for dynamic node provisioning, there are alternative approaches to consider. For example, you can use Cluster Autoscaler, which is another popular tool for autoscaling Kubernetes clusters. Compare the features and trade-offs of different tools to determine the best fit for your use case.

Cleanup

To clean up the resources created in this tutorial, you can delete the Karpenter installation and the sample application. Use the following commands:

# Uninstall Karpenter
helm uninstall karpenter -n karpenter
# Delete the Karpenter namespace
kubectl delete namespace karpenter
# Delete the sample application
kubectl delete -f sample-app.yaml

Architecture

References

• Karpenter Docs

---

This article was written with AI assistance.