Image Scanning in CI: Blocking Vulnerable Images with Admission Control

Intermediate · 7 min read

The Problem

In today’s fast-paced development environment, DevOps teams are under constant pressure to deliver software quickly and reliably. However, the rapid deployment of containerized applications can inadvertently introduce security vulnerabilities. Imagine a scenario where a developer unknowingly uses a Docker image with a critical security flaw. This image is then deployed to production, potentially exposing the application to attacks. The pain point here is clear: how can we ensure that only secure and vulnerability-free images are deployed?

This is where image scanning and admission control come into play. By integrating these practices into your Continuous Integration (CI) pipeline, you can automatically scan images for known vulnerabilities and block any that don’t meet your security standards. This not only enhances the security of your applications but also reduces the risk of potential breaches.

Core Concepts

Before diving into the mechanics, let’s define some key terms and concepts:

• Container Image: A lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, runtime, libraries, environment variables, and configuration files.
• Vulnerability: A weakness in an application or system that can be exploited by a threat actor to perform unauthorized actions within a computer system.
• Image Scanning: The process of analyzing container images for known vulnerabilities and other security issues. This is typically done using specialized tools that compare the image’s components against a database of known vulnerabilities.
• Admission Control: In Kubernetes, admission control is a feature that allows you to intercept and modify requests to the Kubernetes API. It can be used to enforce policies, such as blocking the deployment of vulnerable images.
• Continuous Integration (CI): A development practice where developers frequently merge their code changes into a central repository, after which automated builds and tests are run. This helps catch bugs and integration issues early in the development cycle.

How It Works

The process of blocking vulnerable images with admission control in a CI pipeline involves several steps:

1. Image Scanning: As part of the CI pipeline, every container image is scanned for vulnerabilities. This is typically done using a tool like Trivy, Clair, or Aqua Security.
2. Policy Enforcement: Based on the results of the scan, a policy is enforced. For example, if the image contains a critical vulnerability, it is flagged and not allowed to proceed.
3. Admission Control: When a deployment request is made to the Kubernetes API, the admission controller intercepts the request. If the image has been flagged as vulnerable, the request is blocked, and the deployment is prevented.

This multi-layered approach ensures that only secure images are deployed, thereby reducing the risk of security breaches.

Prerequisites

Before you start, you will need the following:

• A Kubernetes cluster (e.g., minikube, GKE, EKS, AKS)
• A CI/CD pipeline (e.g., GitHub Actions, GitLab CI, Jenkins)
• A container image scanning tool (e.g., Trivy, Clair, Aqua Security)
• Familiarity with YAML and basic Kubernetes concepts

Hands-On Walkthrough

Let’s walk through a step-by-step implementation of image scanning and admission control in a CI/CD pipeline. We’ll use GitHub Actions for our CI/CD, Trivy for image scanning, and a custom admission controller in Kubernetes.

Step 1: Set Up Your Kubernetes Cluster

First, ensure you have a Kubernetes cluster up and running. You can use a local cluster like minikube or a managed service like GKE, EKS, or AKS.

minikube start

Step 2: Install Trivy in Your CI Pipeline

Create a GitHub Actions workflow to scan your container images. Add the following `trivy-scan.yml` file to your repository:

name: Trivy Scan Workflow
on:
  push:
    branches:
      - main
jobs:
  trivy-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      - name: Run Trivy Scan
        id: trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'your-image:tag'
          format: 'table'
          exit-code: '1'
          ignore-unfixed: true
          severity: 'CRITICAL,HIGH'

This workflow will trigger on every push to the `main` branch, scan the specified image, and fail the build if any critical or high-severity vulnerabilities are found.

Step 3: Create a Custom Admission Controller

Create a custom admission controller to block deployments of vulnerable images. Add the following `admission-controller.yaml` file to your repository:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admission-controller
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: admission-controller
rules:
  - apiGroups: ["","apps","batch"]
    resources: ["deployments","pods","replicasets","cronjobs"]
    verbs: ["get","watch","list"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["mutatingwebhookconfigurations","validatingwebhookconfigurations"]
    verbs: ["get","watch","list","create","update","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admission-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admission-controller
subjects:
  - kind: ServiceAccount
    name: admission-controller
    namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-controller
  template:
    metadata:
      labels:
        app: admission-controller
    spec:
      serviceAccountName: admission-controller
      containers:
        - name: admission-controller
          image: your-admission-controller-image:latest
          ports:
            - containerPort: 8443
          volumeMounts:
            - mountPath: /etc/webhook
              name: webhook-certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            secretName: admission-webhook-certs
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: admission-controller
webhooks:
  - name: admission-controller.default.svc
    rules:
      - operations: ["CREATE","UPDATE"]
        apiGroups: ["","apps","batch"]
        apiVersions: ["v1","v1beta1"]
        resources: ["deployments","pods","replicasets","cronjobs"]
    clientConfig:
      service:
        namespace: default
        name: admission-controller
      caBundle: <ca-bundle>
    admissionReviewVersions: ["v1","v1beta1"]
    sideEffects: None
    timeoutSeconds: 10

This manifest defines a service account, cluster role, cluster role binding, and a deployment for the admission controller. It also sets up a validating webhook configuration to intercept and validate deployment requests.

Step 4: Deploy the Admission Controller

Deploy the admission controller to your Kubernetes cluster:

kubectl apply -f admission-controller.yaml

Verifying It Works

To verify that the setup is working, you can try deploying a vulnerable image and see if the admission controller blocks it.

Step 1: Push a Vulnerable Image

Push a container image with known vulnerabilities to your registry. For example, you can use the following Dockerfile:

FROM alpine:3.14
RUN apk add --no-cache curl
RUN chmod 777 /tmp

Build and push the image:

docker build -t your-registry/your-image:tag .
docker push your-registry/your-image:tag

Step 2: Attempt to Deploy the Image

Create a Kubernetes deployment for the image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vulnerable-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vulnerable-app
  template:
    metadata:
      labels:
        app: vulnerable-app
    spec:
      containers:
        - name: vulnerable-app
          image: your-registry/your-image:tag
          ports:
            - containerPort: 80

Apply the deployment:

kubectl apply -f deployment.yaml

If the image is flagged as vulnerable by the CI pipeline, the admission controller should block the deployment. You can check the status of the deployment with:

kubectl get deployments

You should see an error message indicating that the deployment was blocked due to the presence of vulnerabilities.

Common Mistakes

Here are some common mistakes to avoid when implementing image scanning and admission control:

• Ignoring Unfixed Vulnerabilities: Some teams may choose to ignore unfixed vulnerabilities, thinking they are not a risk. However, this can lead to a false sense of security. Always aim to fix or mitigate all vulnerabilities, even if they are not yet fixed upstream.
• Not Updating the Vulnerability Database: The effectiveness of image scanning depends on having an up-to-date vulnerability database. Failing to update the database regularly can result in missing new vulnerabilities. Make sure to schedule regular updates.
• Overly Strict Policies: While it’s important to have strict security policies, overly restrictive policies can hinder development. Find a balance between security and flexibility. For example, allow low-severity vulnerabilities in non-critical environments while enforcing stricter policies in production.
• Not Testing the Admission Controller: Before deploying the admission controller to production, thoroughly test it in a staging environment. This helps identify and fix any issues before they impact your production workflows.

Going Further

Once you have the basics in place, consider the following advanced topics and production considerations:

• Scaling: As your cluster grows, ensure that the admission controller can handle the increased load. Consider using a more robust and scalable solution, such as Open Policy Agent (OPA) with Gatekeeper.
• Security: Secure the communication between the admission controller and the Kubernetes API. Use TLS certificates and mutual authentication to prevent man-in-the-middle attacks.
• Performance Tuning: Optimize the performance of the admission controller by tuning its configuration. For example, adjust the timeout settings and resource limits to ensure it can handle the expected load.
• Alternative Approaches: Explore alternative approaches to admission control, such as using third-party solutions like Aqua Security or Sysdig. These solutions often provide additional features and integrations, but may come with higher costs and complexity.

Cleanup

To clean up the resources created in this tutorial, run the following commands:

kubectl delete -f admission-controller.yaml
kubectl delete -f deployment.yaml

Architecture

References

• Admission Controllers
• Kyverno

---

This article was written with AI assistance.