Managing Secrets in Kubernetes: Sealed Secrets vs External Secrets Operator

Intermediate · 4 min read

The Problem

Imagine you are a DevOps engineer at a growing e-commerce company. As your team expands and more applications are deployed, managing sensitive information like API keys, database credentials, and encryption keys becomes increasingly complex. Storing these secrets in plain text or in version control systems is a security risk. You need a secure and scalable way to manage these secrets in your Kubernetes clusters. This is where tools like Sealed Secrets and External Secrets Operator come into play.

Core Concepts

Secrets Management: In the context of Kubernetes, secrets management involves securely storing, distributing, and rotating sensitive data such as passwords, tokens, and keys. The goal is to ensure that this data is only accessible to authorized services and users.

Kubernetes Secrets: Kubernetes provides a built-in resource called Secrets for storing sensitive data. However, managing these secrets manually can be cumbersome and error-prone, especially in large-scale environments.

Sealed Secrets: Sealed Secrets is a tool that allows you to encrypt secrets and store them in a Kubernetes manifest. This encrypted secret can then be safely committed to version control. When the sealed secret is applied to the cluster, it is automatically decrypted and stored as a regular Kubernetes Secret.

External Secrets Operator (ESO): ESO is an operator that integrates with external secret stores (e.g., HashiCorp Vault, AWS Secrets Manager) to manage secrets in Kubernetes. It fetches secrets from the external store and creates or updates Kubernetes Secrets accordingly.

How It Works

Sealed Secrets: The process involves creating a Kubernetes Secret, sealing it using the kubeseal command, and then committing the sealed secret to version control. When the sealed secret is applied to the cluster, the Sealed Secrets controller decrypts it and creates the corresponding Kubernetes Secret.

External Secrets Operator (ESO): ESO works by defining a custom resource called ExternalSecret. This resource specifies the external secret store and the secrets to be fetched. ESO then watches for changes to these resources and syncs the secrets from the external store to Kubernetes Secrets.

Prerequisites

• A running Kubernetes cluster (e.g., minikube, GKE, EKS)
• kubectl installed and configured to interact with your cluster
• Access to an external secret store (for ESO, e.g., HashiCorp Vault, AWS Secrets Manager)

Hands-On Walkthrough

Step 1: Install Sealed Secrets Controller

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

helm install sealed-secrets sealed-secrets/sealed-secrets

Step 2: Create and Seal a Secret

Create a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: dXNlcm5hbWU=
  password: cGFzc3dvcmQ=

Save this as my-secret.yaml and seal it:

kubeseal --cert <path-to-cert> --format yaml < my-secret.yaml > sealed-my-secret.yaml

Commit sealed-my-secret.yaml to your version control system.

Step 3: Apply the Sealed Secret

kubectl apply -f sealed-my-secret.yaml

Step 4: Install External Secrets Operator (ESO)

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets external-secrets/external-secrets

Step 5: Configure and Use ESO

Create a ClusterSecretStore to connect to your external secret store (e.g., HashiCorp Vault):

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"

Create an ExternalSecret to fetch secrets from the external store:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example-external-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: example-secret
  data:
  - secretKey: username
    remoteRef:
      key: my-secret
      property: username
  - secretKey: password
    remoteRef:
      key: my-secret
      property: password

Verifying It Works

Verify Sealed Secrets

kubectl get secret my-secret -o yaml

You should see the unsealed secret with the correct data.

Verify ESO

kubectl get secret example-secret -o yaml

You should see the secret created by ESO with the correct data.

Common Mistakes

• Forgetting to commit the sealed secret: Ensure you commit the sealed secret to version control after sealing it.
• Incorrectly configuring the external secret store: Double-check the configuration of your external secret store and the corresponding ClusterSecretStore.
• Not setting the correct permissions: Ensure the service account used by ESO has the necessary permissions to access the external secret store and create Kubernetes Secrets.

Going Further

Scaling: Both Sealed Secrets and ESO can scale with your cluster. For large-scale deployments, consider using multiple instances of the controllers and distributing the load.

Security: Regularly rotate your secrets and use strong encryption. Consider using role-based access control (RBAC) to restrict access to secrets.

Performance Tuning: Monitor the performance of the controllers and adjust the refresh intervals and resource limits as needed.

Alternative Approaches: Other tools like Bitnami’s Sealed Secrets and HashiCorp’s Vault CSI Driver can also be used for secrets management. Each has its own trade-offs in terms of complexity, security, and integration with existing infrastructure.

Cleanup

kubectl delete -f sealed-my-secret.yaml

kubectl delete -f example-external-secret.yaml

helm uninstall sealed-secrets

helm uninstall external-secrets

Architecture

References

• Sealed Secrets
• External Secrets Operator

---

This article was written with AI assistance.