Auditing who did what: Kubernetes audit logs that matter

Intermediate · 5 min read

The Problem

In a dynamic and complex environment like Kubernetes, it’s crucial to know who did what and when. Imagine a scenario where a critical service in your Kubernetes cluster experiences an unexpected outage. Without proper auditing, you might struggle to identify the root cause, whether it was a misconfiguration, a malicious action, or an accidental change. This can lead to prolonged downtime, loss of trust, and potential security breaches. Effective auditing helps you maintain a clear and detailed record of all actions, making it easier to diagnose issues, enforce compliance, and ensure the integrity of your cluster.

Core Concepts

Kubernetes audit logs are a vital part of the cluster’s security and operational monitoring. They provide a detailed record of all API requests made to the Kubernetes API server, including who performed the action, what action was taken, and when it occurred. Here are the key concepts:

• Audit Logs: These are the records of all API requests made to the Kubernetes API server. Each log entry includes details such as the user, the action, the resource, and the timestamp.
• Audit Policy: This is a configuration file that defines which events should be logged and how they should be recorded. It allows you to specify the level of detail for different types of events.
• Audit Backend: This is the system or service that stores and processes the audit logs. Common backends include local files, cloud storage, and external logging systems like Elasticsearch or Splunk.

How It Works

When an API request is made to the Kubernetes API server, the following steps occur:

1. The API server receives the request and checks the audit policy to determine if the event should be logged.
2. If the event is to be logged, the API server generates an audit log entry, which includes the user, the action, the resource, and the timestamp.
3. The audit log entry is then sent to the configured audit backend for storage and processing.

This process ensures that every relevant action is recorded, providing a comprehensive and detailed history of all interactions with the Kubernetes API.

Prerequisites

Before you start, make sure you have the following:

• A running Kubernetes cluster (version 1.16 or later).
• kubectl installed and configured to interact with your cluster.
• A text editor for creating and editing YAML files.
• Access to the Kubernetes API server (usually through kubectl).

Hands-On Walkthrough

Let’s walk through the process of setting up and using Kubernetes audit logs.

Step 1: Create an Audit Policy

Create a YAML file named `audit-policy.yaml` with the following content:

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
  users: ["system:admin"]
  verbs: ["get", "list", "watch"]
  resources: ["pods"]
- level: Request
  users: ["system:admin"]
  verbs: ["create", "update", "patch", "delete"]
  resources: ["pods"]
- level: RequestResponse
  users: ["system:admin"]
  verbs: ["create", "update", "patch", "delete"]
  resources: ["secrets"]

This policy logs metadata for read-only operations on pods, full request and response details for create, update, patch, and delete operations on pods, and full request and response details for secret operations.

Step 2: Apply the Audit Policy

Apply the audit policy to your Kubernetes cluster by updating the API server configuration. You can do this by adding the following arguments to the API server command line:

--audit-policy-file=/path/to/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxsize=100
--audit-log-format=json

Note: The exact method to update the API server configuration depends on your cluster setup. For example, in a Kubernetes cluster managed by kubeadm, you would edit the `kube-apiserver` manifest in the `/etc/kubernetes/manifests` directory.

Step 3: Verify the Audit Logs

After applying the audit policy, you can verify that the logs are being generated by checking the specified log file:

cat /var/log/kubernetes/audit.log

You should see entries for the API requests that match the rules defined in your audit policy.

Verifying It Works

To confirm that the audit logs are working as expected, perform some actions in your Kubernetes cluster and check the log file. For example, create a pod and a secret:

kubectl run nginx --image=nginx
kubectl create secret generic my-secret --from-literal=key1=value1

Check the audit log file again:

cat /var/log/kubernetes/audit.log

You should see entries for the `create` actions on the pod and the secret, along with the user, timestamp, and other details.

Common Mistakes

Here are some common mistakes to avoid when setting up Kubernetes audit logs:

• Overly Broad Policies: Defining too broad policies can result in large log files that are difficult to manage and analyze. Be specific about the events you want to log.
• Insufficient Storage: Ensure that the storage for the audit logs is adequate. If the log file grows too large, it can fill up the disk, leading to potential issues.
• Incorrect File Permissions: Make sure that the API server has the necessary permissions to write to the log file. Incorrect permissions can prevent the logs from being written.

Going Further

Once you have a basic setup, consider the following production considerations and advanced topics:

• Scaling: For large clusters, consider using a distributed logging solution like Elasticsearch and Kibana to store and visualize the audit logs.
• Security: Secure the audit logs by encrypting them at rest and in transit. Use role-based access control (RBAC) to restrict access to the logs.
• Performance Tuning: Optimize the audit policy to balance between logging enough information and avoiding excessive overhead. Regularly review and adjust the policy as needed.
• Alternative Approaches: Consider using third-party tools like Falco for real-time detection of anomalous behavior. These tools can complement the audit logs by providing additional context and insights.

Cleanup

To remove the audit policy and stop generating audit logs, revert the changes to the API server configuration. For example, if you edited the `kube-apiserver` manifest, restore the original file or remove the audit-related arguments.

Architecture

References

• Auditing

---

This article was written with AI assistance.