TLS Certificate Expiry in Kubernetes: Finding and Rotating

Intermediate · 6 min read

The Problem

Imagine you are a DevOps engineer at a growing e-commerce company. Your team has been using Kubernetes to manage your microservices architecture, and everything has been running smoothly. One day, you receive an urgent call from the support team: customers are reporting that they can’t access the website, and the error message indicates a TLS certificate issue. Upon investigation, you find that the TLS certificate for one of your critical services has expired, causing the service to become inaccessible. This scenario is not only frustrating for your customers but also detrimental to your business’s reputation. In this tutorial, we will explore how to find and rotate TLS certificates in a Kubernetes cluster, ensuring that such issues do not disrupt your operations.

Core Concepts

Before diving into the solution, let’s cover some fundamental concepts:

• TLS (Transport Layer Security): A cryptographic protocol designed to provide secure communication over a computer network. It is widely used to secure web traffic, email, and other forms of data transmission.
• Certificate Authority (CA): An entity that issues digital certificates. These certificates are used to verify the identity of a server or client and establish a secure connection.
• Certificate Expiry: TLS certificates have a validity period, after which they expire. Expired certificates can cause security warnings and prevent clients from establishing a secure connection.
• Kubernetes Secrets: A Kubernetes resource type used to store sensitive information, such as passwords, OAuth tokens, and TLS certificates. Secrets are stored in a way that prevents them from being exposed in the clear.
• kubectl: The command-line tool for interacting with Kubernetes clusters. It allows you to inspect, create, update, and delete resources in your cluster.

How It Works

To find and rotate TLS certificates in a Kubernetes cluster, you need to follow these steps:

1. Identify the services that use TLS certificates.
2. Check the expiry date of the certificates.
3. Obtain new certificates from a Certificate Authority.
4. Update the Kubernetes Secrets with the new certificates.
5. Restart the affected services to apply the new certificates.

This process ensures that your services continue to operate securely and without interruption.

Prerequisites

Before starting, you will need:

• A running Kubernetes cluster (e.g., Minikube, GKE, EKS, AKS).
• kubectl installed and configured to interact with your cluster.
• Access to a Certificate Authority to obtain new TLS certificates.
• Basic understanding of Kubernetes resources and commands.

Hands-On Walkthrough

In this section, we will walk through the process of finding and rotating TLS certificates in a Kubernetes cluster. We will use a sample Nginx deployment as an example.

Step 1: Identify Services Using TLS Certificates

First, list all the secrets in your namespace to identify the ones containing TLS certificates.

kubectl get secrets -n <namespace>

Look for secrets that contain `tls` in their name or description. For example:

kubectl get secret nginx-tls -o yaml -n <namespace>

Step 2: Check the Expiry Date of the Certificates

Use the following command to extract and check the expiry date of the TLS certificate:

kubectl get secret nginx-tls -o jsonpath='{.data.tls\.crt}' -n <namespace> | base64 --decode | openssl x509 -noout -enddate

This command decodes the base64-encoded certificate and uses OpenSSL to check the expiry date.

Step 3: Obtain New Certificates from a Certificate Authority

Contact your Certificate Authority to obtain new TLS certificates. You will typically receive a `.crt` file for the certificate and a `.key` file for the private key.

Step 4: Update the Kubernetes Secrets with the New Certificates

Create a new secret with the updated certificates:

kubectl create secret tls nginx-tls-new --cert=nginx.crt --key=nginx.key -n <namespace>

Alternatively, you can update the existing secret:

kubectl edit secret nginx-tls -n <namespace>

Replace the contents of the `tls.crt` and `tls.key` fields with the new certificate and key.

Step 5: Restart the Affected Services to Apply the New Certificates

Finally, restart the affected services to apply the new certificates. For example, if you are using a Deployment, you can trigger a rolling update by updating the pod template:

kubectl rollout restart deployment/nginx-deployment -n <namespace>

Verifying It Works

To confirm that the new certificates are in place and working correctly, you can:

• Check the secret to ensure it contains the new certificates:

kubectl get secret nginx-tls -o yaml -n <namespace>

• Verify the expiry date of the new certificate:

kubectl get secret nginx-tls -o jsonpath='{.data.tls\.crt}' -n <namespace> | base64 --decode | openssl x509 -noout -enddate

• Test the service to ensure it is accessible and secure:

curl -v https://<service-url>

You should see the new certificate details and no security warnings.

Common Mistakes

Here are some common mistakes to avoid when rotating TLS certificates in a Kubernetes cluster:

• Not checking the expiry date: Failing to regularly check the expiry date of TLS certificates can lead to unexpected outages. Set up monitoring and alerts to notify you well in advance of certificate expiry.
• Using the wrong certificate or key: Ensure that you are using the correct certificate and key files. Mixing up files can cause the service to fail to start or result in security vulnerabilities.
• Not restarting the service: After updating the secret, make sure to restart the affected services. Without a restart, the new certificates will not be applied, and the old, expired certificates will still be in use.

Going Further

For production environments, consider the following advanced topics and best practices:

• Automated Certificate Management: Use tools like Cert-Manager to automate the issuance and renewal of TLS certificates. This reduces the risk of human error and ensures that certificates are always up-to-date.
• Monitoring and Alerts: Set up monitoring and alerting for certificate expiry. Tools like Prometheus and Grafana can be used to track certificate expiry dates and send notifications when certificates are nearing expiration.
• Security Best Practices: Store TLS certificates and keys securely. Use Kubernetes Secrets with encryption at rest, and consider using a Hardware Security Module (HSM) for added security.
• Performance Considerations: Regularly test the performance of your services after certificate rotation. Ensure that the new certificates do not introduce any latency or performance issues.
• Alternative Approaches: Explore alternative approaches such as using self-signed certificates for internal services or leveraging Let’s Encrypt for free, automated certificate issuance. Each approach has its trade-offs in terms of security, cost, and complexity.

Cleanup

To remove the resources created in this tutorial, run the following commands:

kubectl delete secret nginx-tls-new -n <namespace>

If you updated the existing secret, you can revert the changes by editing the secret again:

kubectl edit secret nginx-tls -n <namespace>

Additionally, clean up the Nginx deployment and related resources:

kubectl delete deployment/nginx-deployment -n <namespace>

Architecture

References

• Manage TLS Certificates in a Cluster

---

This article was written with AI assistance.