Writing NetworkPolicies that default-deny without breaking your app

ByShubham Mishra

Intermediate · 12 min read

The Problem

In the early days of Kubernetes, security was often an afterthought. Developers focused on getting services to communicate, and networking was treated as a flat, open network where every pod could talk to every other pod. This “trust everything” model worked fine in small, controlled environments, but it became a critical vulnerability as applications grew in complexity and compliance requirements tightened.

Consider a typical microservices architecture: a frontend web server, a backend API, a database, and a caching layer like Redis. By default, if you deploy these pods in the same namespace, they can all see each other. If an attacker compromises the frontend web server due to a vulnerability, they have unrestricted lateral movement. They can scan the internal network, find the database, and exfiltrate sensitive data because no one put up any walls.

This is where NetworkPolicy comes in. It acts as a firewall for your pods. However, there is a significant pain point: implementing a strict “default-deny” policy (blocking all traffic unless explicitly allowed) is notoriously difficult. Developers often spend hours debugging why their application is broken, only to realize they missed a single DNS query or a health check connection. Breaking production while trying to secure it is a career-limiting move. The challenge is not just writing the policies, but writing them in a way that secures the environment without causing outages during implementation.

Core Concepts

To understand how to write effective policies, we must first define the fundamental building blocks of Kubernetes networking security.

What is a NetworkPolicy?

A NetworkPolicy is a Kubernetes resource that specifies how groups of pods are allowed to communicate with each other and other network endpoints. It operates at Layer 4 (Transport Layer) of the OSI model, meaning it controls traffic based on IP addresses, ports, and protocols (TCP/UDP), not Layer 7 (Application Layer) content like HTTP headers.

The Default Behavior: Allow All

By default, Kubernetes has no networking restrictions between pods. If you create three pods in a namespace, Pod A can send traffic to Pod B and Pod C without any configuration. This is known as an “allow-all” state. While convenient, it is insecure.

The Default-Deny Strategy

The core concept of our tutorial is the Default-Deny approach. Instead of listing every allowed path (which is hard to maintain), you start by blocking everything. Then, you selectively whitelist only the traffic your application actually needs.

  • Default Deny Ingress: Blocks all incoming traffic to pods in a namespace unless a specific rule allows it.
  • Default Deny Egress: Blocks all outgoing traffic from pods unless a specific rule allows it.

Key Components of a Policy

  • PodSelector: Identifies which pods the policy applies to. An empty selector ({}) targets all pods in the namespace.
  • policyTypes: Specifies whether the policy affects Ingress (incoming) or Egress (outgoing) traffic.
  • ingress/egress rules: Define the allow-lists. Each rule contains from/ to selectors (who) and ports (what).

CNI Dependency

Crucially, NetworkPolicy resources are only effective if your cluster uses a Container Network Interface (CNI) plugin that supports them. Most major CNIs (Calico, Cilium, Romana) support NetworkPolicies. Standard plugins like Flannel do not enforce them by default. You must verify your cluster’s CNI capability before proceeding.

How It Works

Understanding the mechanics requires looking at how Kubernetes processes network packets relative to NetworkPolicies.

Evaluation Order

Kubernetes evaluates NetworkPolicies in a specific hierarchy. When a packet arrives at a pod:

  1. If there are any NetworkPolicies applied to the destination pod, the default behavior changes from “Allow All” to “Deny All” for the relevant direction (Ingress or Egress).
  2. The system then iterates through all applicable policies.
  3. If at least one policy matches the traffic (source, port, protocol), the traffic is allowed.
  4. If no policy matches the traffic, it is dropped.

This means you don’t need a single monolithic policy file. You can have multiple policies. As long as one of them permits the traffic, it gets through. This modularity is key to avoiding the “breaking your app” problem.

Namespace vs. Pod Level

Policies can be scoped to a namespace (affecting all pods in that namespace) or to specific pods using label selectors. For a robust security posture, we typically start with a namespace-level default deny, then add pod-specific allow rules.

Prerequisites

Before implementing this guide, ensure you have the following:

  1. Kubernetes Cluster: A running cluster with a CNI plugin that supports NetworkPolicies (e.g., Calico, Cilium). You can check this by creating a test policy; if it doesn’t take effect, your CNI may not support it.
  2. kubectl Access: Administrative access to your cluster.
  3. Namespace: A dedicated namespace for testing, such as secure-net-demo, to avoid affecting production workloads.
  4. Basic YAML Knowledge: Familiarity with reading and writing Kubernetes manifests.

Hands-On Walkthrough

We will build a secure environment step-by-step. We will deploy two services: a web-app and a database. Initially, they will communicate freely. Then, we will lock down the network so only the web-app can talk to the database, and block all other access.

Step 1: Deploy the Application Pods

First, let’s create a namespace and deploy simple containers with labels so we can target them with policies.

apiVersion: v1
kind: Namespace
metadata:
  name: secure-net-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: secure-net-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
  namespace: secure-net-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: postgres
        image: postgres:latest
        ports:
        - containerPort: 5432

Apply this manifest:

kubectl apply -f deployment.yaml

At this stage, both pods are running. You can verify connectivity by exec-ing into the web-app and trying to reach the database pod by its IP address. It should work because there are no policies yet.

Step 2: Implement Default-Deny Ingress

This is the critical step. We want to block all incoming traffic to the database pod by default. We do not want to block the web-app yet, because we need to allow external users to access the website. So, we create a policy that targets only the database.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: secure-net-demo
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Note the podSelector: {}. This empty object selects all pods in the namespace. Combined with policyTypes: [Ingress] and the absence of an ingress rule array, this creates a blanket ban on all incoming traffic for every pod in the namespace.

Wait! If you apply this now, your web-app will also stop receiving incoming connections from the outside world. Users won’t be able to load the site. To fix this, we must immediately add an allow rule for the web-app.

Step 3: Whitelist the Web App

We create a second policy that explicitly allows ingress traffic to the web-app from anywhere (or specifically from the internet/load balancer).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-ingress
  namespace: secure-net-demo
spec:
  podSelector:
    matchLabels:
      app: web-app
  policyTypes:
  - Ingress
  ingress:
  - from: []
    ports:
    - protocol: TCP
      port: 80

Here, from: [] is an empty list. In NetworkPolicy logic, an empty from list implies “from all sources.” This overrides the default deny for the web-app on port 80.

Step 4: Whitelist Database Access for Web App Only

Now, the database is still blocked by the default-deny policy from Step 2. We need to allow the web-app to connect to the database on port 5432. Note that this is an Ingress rule on the database side (since the database is receiving the connection).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-from-web
  namespace: secure-net-demo
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web-app
    ports:
    - protocol: TCP
      port: 5432

This policy says: “For pods labeled ‘database’, allow incoming traffic ONLY from pods labeled ‘web-app’ on port 5432.”

Step 5: Restricting Egress (Advanced)

To truly secure the database, we should also prevent it from making outbound connections. Databases rarely need to initiate connections to the internet. Let’s add a default-deny egress policy for the database.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-egress
  namespace: secure-net-demo
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: web-app
    ports:
    - protocol: TCP
      port: 80

This allows the database to respond to the web-app (return traffic). In many CNI implementations, return traffic is automatically allowed if the initial request was permitted, but explicit egress rules are safer for complex scenarios like DNS resolution or cloud metadata calls.

Verifying It Works

Let’s confirm our security posture. We will use a temporary debug pod to simulate external and internal attacks.

Test 1: Verify Web App is Accessible

Get the ClusterIP of the web-app service (or use the pod IP directly if you exposed it via NodePort). You should be able to curl the web-app from your local machine or another pod.

kubectl get svc -n secure-net-demo
# Use the IP from the output
curl http://<WEB_APP_IP>

This should return the Nginx welcome page. If it fails, check if your load balancer or ingress controller is working correctly.

Test 2: Verify Database is Isolated from External Sources

Create a temporary busybox pod in the same namespace to try and connect to the database.

kubectl run -it --rm debug-pod --image=busybox --restart=Never -n secure-net-demo -- sh

Inside the container, try to ping or telnet to the database pod IP.

wget -qO- --timeout=2 http://<DB_POD_IP>:5432

Expected Result: The command should hang or timeout. No response indicates the traffic is being dropped by the NetworkPolicy. This confirms the default-deny ingress is working for the database.

Test 3: Verify Web App Can Talk to Database

Exec into the web-app pod and try to connect to the database.

kubectl exec -it <WEB_APP_POD_NAME> -n secure-net-demo -- wget -qO- --timeout=2 http://<DB_POD_IP>:5432

Expected Result: Since the database expects PostgreSQL binary protocol, wget might show garbage data or a connection refused error depending on the stack, but critically, it should not timeout due to network drop. If you see a timeout, check your labels. Ensure app: web-app and app: database match exactly in the policy and the deployments.

Test 4: Verify Cross-Namespace Isolation

Create a pod in a different namespace (e.g., default) and try to reach the database. It should fail completely, as NetworkPolicies do not cross namespace boundaries by default.

Common Mistakes

Even with correct syntax, NetworkPolicies are prone to human error. Here are the most common pitfalls and how to avoid them.

1. The DNS Black Hole

If you implement a default-deny Egress policy, your pods will lose the ability to resolve domain names. Kubernetes relies heavily on CoreDNS. If your app tries to look up api.external-service.com, it sends a UDP packet to the kube-dns service. If your egress policy doesn’t explicitly allow traffic to the CoreDNS Service IP (usually <cluster-ip> on port 53 UDP/TCP), the lookup fails, and your application crashes or hangs.

Solution: Always include a rule allowing egress to the CoreDNS service. Find its IP with kubectl get svc -n kube-system kube-dns and add an ipBlock or namespaceSelector rule targeting that IP.

2. Label Mismatches

NetworkPolicies rely entirely on labels. If your deployment has app: web but your policy selects app: web-app, the policy will not apply to the pod. This results in silent failures where the policy seems inactive.

Solution: Use kubectl describe networkpolicy to see which pods are selected. Verify labels match exactly, including case sensitivity.

3. Assuming Layer 7 Protection

Developers often assume NetworkPolicies can filter HTTP paths (e.g., allow /api but block /admin). They cannot. They only filter IPs and Ports. Using NetworkPolicies for path-based filtering will result in all traffic on that port being allowed.

Solution: Use an Ingress Controller (like NGINX or Traefik) or a Service Mesh (like Istio) for Layer 7 routing and filtering. Keep NetworkPolicies for Layer 4 segmentation.

4. Overly Broad Whitelists

A common mistake when fixing connectivity issues is changing a specific podSelector to an empty podSelector or removing the selector entirely, effectively opening the door to everyone. Always restrict sources to the minimum necessary identity.

Going Further

Once you have mastered basic NetworkPolicies, consider these advanced production considerations.

Scaling and Performance

In large clusters with thousands of pods and hundreds of policies, evaluating every packet against every policy can introduce latency. Modern CNIs like Cilium use eBPF to handle this efficiently at the kernel level, minimizing performance overhead. Older CNIs may rely on iptables rules, which can become slow as the rule set grows. If performance is a concern, evaluate your CNI’s implementation of NetworkPolicies.

Alternative Approaches: Service Meshes

While NetworkPolicies are excellent for network-layer security, they lack visibility into application-layer details. A Service Mesh (like Istio or Linkerd) provides mutual TLS (mTLS), detailed traffic metrics, and circuit breaking. Many teams use both: NetworkPolicies for coarse-grained network segmentation and Service Mesh for fine-grained application security and observability. However, note that some Service Meshes require their own sidecar proxies, which adds resource overhead.

GitOps and Policy as Code

Managing dozens of YAML files manually is error-prone. Consider using tools like OPA/Gatekeeper or Kyverno to enforce policies centrally. These tools can validate that new deployments come with appropriate NetworkPolicies attached, ensuring that no new service is deployed without security controls.

Cleanup

To remove all resources created in this tutorial and restore your cluster to a clean state:

kubectl delete namespace secure-net-demo

This command deletes the namespace and all associated pods, services, and network policies within it.

References

This article was written with AI assistance.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *