Exposing TCP and UDP Services in Kubernetes: Beyond HTTP

ByShubham Mishra

Intermediate · 14 min read

The Problem

In the modern DevOps landscape, the vast majority of application traffic is HTTP-based. REST APIs, GraphQL endpoints, and web frontends all rely on the request-response cycle defined by the Hypertext Transfer Protocol. Consequently, Kubernetes native service abstractions—specifically the Service resource with its ClusterIP, NodePort, and LoadBalancer types—are optimized for this paradigm. They handle connection tracking, health checks, and load balancing with a level of sophistication that assumes an understanding of HTTP semantics.

However, not all applications live within the walls of the web. Database servers (MySQL, PostgreSQL), caching layers (Redis, Memcached), message brokers (Kafka, RabbitMQ), and custom IoT protocols often speak raw TCP or UDP. These protocols do not have headers to inspect for health status, nor do they support the concept of a “request” and “response” in the same way HTTP does. When you attempt to expose a PostgreSQL database using a standard Kubernetes Service, you might wonder: How does the kube-proxy know when a connection is healthy? How are packets routed if there is no HTTP header to parse? What happens when latency spikes?

This tutorial addresses the pain point of exposing non-HTTP services effectively. We will move beyond the simple act of creating a YAML file and dive into the mechanics of how Kubernetes handles Layer 4 (Transport Layer) networking. By the end of this guide, you will understand not just how to configure these services, but how the underlying infrastructure manages stateful connections, handles packet loss, and ensures reliability for critical binary protocols.

Core Concepts

To understand how Kubernetes exposes TCP and UDP services, we must first strip away the HTTP-specific assumptions and look at the fundamental networking layers involved. This section defines the key components that make up the transport layer abstraction in Kubernetes.

TCP vs. UDP: The Transport Layer

Kubernetes Services operate at Layer 4 of the OSI model, known as the Transport Layer. The two primary protocols here are Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).

  • TCP (Transmission Control Protocol): A connection-oriented protocol. Before data can be exchanged, a “three-way handshake” occurs to establish a reliable connection between client and server. TCP guarantees delivery, order, and error-checking. It is ideal for databases and file transfers where data integrity is paramount. In Kubernetes, TCP connections are tracked. If a pod dies, the connection is broken, and new connections are routed elsewhere.
  • UDP (User Datagram Protocol): A connectionless protocol. Data is sent in independent packets without establishing a dedicated connection. There is no guarantee of delivery, ordering, or duplicate protection. However, it is significantly faster and has lower overhead. It is used for DNS resolution, streaming video, and real-time gaming. In Kubernetes, UDP is stateless; each packet is treated independently.

The Kubernetes Service Resource

A Service is an abstract way to expose an application running on a set of Pods as a network service. While often associated with HTTP, the Service object itself is protocol-agnostic regarding the payload. It defines:

  • Ports: You specify the port (the port on the Service IP) and the targetPort (the port on the Pod). For TCP/UDP, you also explicitly declare the protocol field, which defaults to TCP but must be set to UDP when necessary.
  • Selectors: Labels that match the Pods backing the service. This creates a dynamic mapping between the stable Virtual IP (VIP) of the Service and the potentially ephemeral IPs of the Pods.

kube-proxy and Service Distribution

The magic of routing happens in kube-proxy. This component runs on every node in the cluster. Its job is to watch the Kubernetes API Server for changes to Service and Endpoint objects. When a change is detected, kube-proxy updates the networking rules on the node to ensure that traffic sent to the Service IP is forwarded to one of the backend Pods.

For TCP and UDP, kube-proxy typically uses two modes:

  1. iptables mode: Uses Linux netfilter rules to redirect traffic. It is efficient but can suffer from rule explosion in large clusters.
  2. ipvs mode: Uses the Linux IPVS kernel module. It provides better load balancing algorithms (like least connection) and handles larger numbers of services more efficiently. This is the recommended mode for production environments handling high-volume TCP/UDP traffic.

Endpoints and EndpointSlices

Historically, Kubernetes used Endpoints objects to map Service IPs to Pod IPs. Modern Kubernetes uses EndpointSlice for improved scalability. An EndpointSlice represents a subset of Pods that are ready to receive traffic. For TCP/UDP services, readiness is determined by the Pod’s readiness probes. If a Pod fails its health check, it is removed from the EndpointSlice, and kube-proxy immediately stops routing traffic to it. This is crucial for non-HTTP services where the application itself cannot easily signal “I am busy” via HTTP status codes.

Connection Tracking (Conntrack)

One of the most critical concepts for TCP services is Connection Tracking. Linux kernels maintain a table (conntrack) of all active network connections. When a TCP connection is established, an entry is created. If the connection is dropped unexpectedly, the conntrack entry times out. In Kubernetes, if a Pod hosting a long-lived TCP connection crashes, the conntrack entry may remain in a stale state on the node until it expires. This can cause issues where traffic continues to be routed to a dead Pod. Understanding this is vital for troubleshooting intermittent connectivity failures in database clusters.

How It Works

Now that we have defined the components, let us examine the lifecycle of a packet traveling through a Kubernetes TCP Service. This mechanical understanding is essential for debugging.

1. The Request Initiation

A client application outside the cluster sends a packet to the Service IP on port 5432 (PostgreSQL). This Service IP is a virtual IP, not assigned to any physical interface. Instead, it is handled by the Linux kernel’s Virtual IP (VIP) mechanism, specifically via ARP (Address Resolution Protocol) or BGP announcements depending on your CNI (Container Network Interface) plugin.

2. DNAT (Destination Network Address Translation)

When the packet hits the node, kube-proxy’s iptables or ipvs rules intercept it. The rule matches the destination IP (Service IP) and port. The rule then performs a DNAT operation, rewriting the destination address from the Service VIP to one of the actual Pod IPs. If using ipvs in rr (round-robin) or lc (least connection) mode, the choice of backend Pod is made dynamically based on current load.

3. Forwarding and SNAT (Source Network Address Translation)

In many configurations, especially with cloud providers, Source NAT is applied. This changes the source IP of the packet from the client’s IP to the Node’s IP. This ensures that the return traffic comes back to the node, where it can be reverse-NATed back to the original client. Without SNAT, asymmetric routing issues can occur, where the response takes a different path than the request, causing firewalls to drop the packet.

4. Delivery to the Pod

The packet arrives at the target Pod’s network namespace. The container runtime (e.g., CRI) forwards it to the specific process listening on the target port. For TCP, the kernel acknowledges the packet. For UDP, if the port is closed, an ICMP port unreachable message might be sent back.

5. Health Check Integration

Unlike HTTP services, TCP and UDP Services do not perform deep packet inspection to verify application health. Kubernetes relies entirely on Readiness Probes. If you define a TCP socket probe or an Exec probe on your Deployment, Kubernetes periodically checks if the port is open. If the probe fails, the Pod is marked as “Not Ready.” Kube-proxy removes this Pod from the EndpointSlices. Subsequent packets routed by kube-proxy will never reach this Pod. This decouples the network layer from the application layer, allowing Kubernetes to isolate faulty nodes without requiring the application to implement complex self-reporting mechanisms.

Prerequisites

To follow this walkthrough, you need the following:

  • Kubernetes Cluster: A working cluster version 1.20 or higher. Ensure your kube-proxy is configured to use ipvs mode if available, as it offers superior performance for TCP/UDP workloads.
  • kubectl Access: Administrative access to the cluster with permissions to create Services, Deployments, and Pods.
  • Network Policy Awareness: Basic understanding of how Network Policies interact with Services. By default, all ingress traffic is allowed, but securing TCP/UDP ports requires explicit policy definitions.
  • Client Tools: A machine capable of sending TCP/UDP traffic to the cluster. For testing, tools like nc (netcat), telnet, or specialized clients for Redis/PostgreSQL are useful.

Hands-On Walkthrough

We will create a simple UDP-based echo service and a TCP-based mock database service to demonstrate both protocols. We will start with the UDP service, as it highlights the stateless nature of the transport layer.

Step 1: Create the Backend Pods

First, we deploy two pods acting as our backend. For simplicity, we will use a public image that listens on a UDP port and echoes data back. Note that for production, you would use your own application images.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: udp-echo-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: udp-echo
  template:
    metadata:
      labels:
        app: udp-echo
    spec:
      containers:
      - name: udp-echo
        image: jmalloc/udp-echo:latest
        ports:
        - containerPort: 8080
          protocol: UDP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tcp-mock-db
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tcp-db
  template:
    metadata:
      labels:
        app: tcp-db
    spec:
      containers:
      - name: tcp-mock
        image: redis:7-alpine
        ports:
        - containerPort: 6379
          protocol: TCP

Apply this manifest to create the deployments. Wait for the pods to enter the Running state.

Step 2: Expose the UDP Service

Create a Service that exposes the UDP echo server. Notice the explicit declaration of protocol: UDP.

apiVersion: v1
kind: Service
metadata:
  name: udp-echo-service
spec:
  type: ClusterIP
  selector:
    app: udp-echo
  ports:
  - port: 8080
    targetPort: 8080
    protocol: UDP

Step 3: Expose the TCP Service

Create a Service for the TCP mock database. Here, the protocol defaults to TCP, but we explicitly state it for clarity.

apiVersion: v1
kind: Service
metadata:
  name: tcp-db-service
spec:
  type: ClusterIP
  selector:
    app: tcp-db
  ports:
  - port: 6379
    targetPort: 6379
    protocol: TCP

Step 4: Verify Endpoints

Check that the Services have correctly identified their backend pods.

kubectl get endpoints udp-echo-service
echo "Expected output should show IP addresses of the two udp-echo pods"

kubectl get endpoints tcp-db-service
echo "Expected output should show IP addresses of the two tcp-db pods"

Step 5: Testing Connectivity

From a debug pod inside the cluster, test the UDP service. Since UDP is connectionless, we send a packet and see if we get a response.


kubectl run -i --tty debug-pod --image=busybox --rm --restart=Never -- sh
# Inside the pod:
nc -u -w 2 <SERVICE_IP> 8080
# Type 'hello' and press enter. You should see 'hello' echoed back.
exit

Now test the TCP service using a Redis CLI client.


kubectl run -i --tty redis-test --image=redis:7-alpine --rm --restart=Never -- sh
# Inside the pod:
redis-cli -h <SERVICE_IP> ping
# Expected output: PONG
exit

Verifying It Works

Verification goes beyond simple connectivity. In production, you must ensure that load balancing is functioning and that failover works correctly.

1. Load Balancing Verification

To verify that traffic is distributed across multiple pods, send multiple requests and observe which pod responds. For the UDP service, since the client-side UDP stack may reuse the same socket, you might need to use a script that opens new sockets for each request.


for i in $(seq 1 10); do
  kubectl exec debug-pod -- nc -u -w 2 <UDP_SERVICE_IP> 8080 <<< "test-$i"
done

If using ipvs mode, you can inspect the load balancing statistics on a node:

kubectl exec -it <node-name> -- ipvsadm -Ln

This command shows the number of connections and packets routed to each backend pod, confirming that the distribution algorithm is active.

2. Failover Verification

Delete one of the backend pods and immediately retry the connection.

kubectl delete pod <pod-name-of-tcp-db>
# Retry the redis-cli ping

The connection should succeed instantly against the remaining pod. If it hangs, check the readinessProbe configuration. Kubernetes waits for the Pod to be marked Not Ready before removing it from the EndpointSlice. If the probe is misconfigured, stale connections may persist.

3. Latency and Packet Loss Monitoring

Use tools like tcpdump on the node to analyze packet retransmissions for TCP services. High retransmission rates indicate network congestion or incorrect MTU settings, which are common issues in overlay networks used by CNIs.

Common Mistakes

Even experienced DevOps engineers fall into traps when dealing with non-HTTP services in Kubernetes.

Mistake 1: Assuming Readiness Probes Work Like HTTP Checks

Why it happens: Developers often omit readiness probes for TCP/UDP services, assuming that if the container is running, the port is open. The Reality: A container can be running while the application inside is hung or failing to bind to the port. Without a TCP Socket Probe or Exec Probe, Kubernetes will continue to route traffic to a broken pod, causing timeouts for clients.

Solution: Always define a tcpSocket readiness probe for critical TCP services.

readinessProbe:
  tcpSocket:
    port: 6379
  initialDelaySeconds: 5
  periodSeconds: 10

Mistake 2: Ignoring Conntrack Limits

Why it happens: TCP services, especially those maintaining persistent connections (like database connection pools), generate a high volume of conntrack entries. The default Linux kernel limit for conntrack tables (often ~65k or ~262k entries) can be exceeded in large clusters with many TCP services.

The Reality: When the conntrack table fills up, new connections are dropped silently, leading to mysterious application errors. Solution: Tune nf_conntrack_max on all nodes and consider increasing the hash table size. Monitor conntrack usage with conntrack -C.

Mistake 3: Using NodePort for Internal Communication

Why it happens: Teams sometimes use NodePort services to allow internal pods to talk to other services, thinking it provides isolation. The Reality: NodePort exposes the service on every node’s IP, creating unnecessary noise and security risks. It also bypasses the efficiency of ClusterIP routing within the same node.

Solution: Use ClusterIP for inter-service communication. Only use NodePort or LoadBalancer when external access is strictly required.

Mistake 4: Misconfiguring UDP Timeout Behavior

Why it happens: UDP has no connection state. Clients may buffer data or assume a connection exists. The Reality: If a pod dies, the client won’t know until it tries to send data and receives an ICMP error or times out. This can lead to long hang times. Solution: Implement application-level keepalives or heartbeat messages. Configure client-side timeouts aggressively.

Going Further

Once you have mastered the basics, several advanced considerations come into play for production-grade TCP/UDP services.

Performance Tuning with IPVS

As mentioned, switch kube-proxy to ipvs mode. IPVS supports advanced scheduling algorithms like lc (Least Connections) and dh (Destination Hashing), which are far superior to round-robin for stateful TCP services. Destination Hashing ensures that a client always connects to the same backend pod, which can be useful for session affinity in certain legacy applications.

Security: Network Policies

With TCP/UDP services, you lose the semantic context of HTTP methods and URLs. Security must be enforced at the port and IP level. Use Kubernetes Network Policies to restrict ingress to only the pods that need to communicate with your database or cache. For example, allow ingress to the PostgreSQL Service only from pods with the label role: backend.

Observability

Standard HTTP metrics (requests per second, error rates) are not available for raw TCP/UDP. To monitor these services, you must instrument the application itself to export metrics via Prometheus counters (e.g., total bytes sent/received, connection count). Alternatively, use eBPF-based tools like Cilium or Tetrate Istio to gain visibility into L4 traffic flows without modifying the application code.

Alternative Approaches: Headless Services

For some stateful TCP services (like ZooKeeper or etcd), you might prefer a Headless Service (clusterIP: None). This returns the individual Pod IPs directly to the client, bypassing kube-proxy load balancing entirely. This gives the client full control over connection management and persistence, which is often required for consensus-based distributed systems.

Cleanup

To remove the resources created in this tutorial, apply the following commands:


kubectl delete service udp-echo-service tcp-db-service
kubectl delete deployment udp-echo-server tcp-mock-db

Verify cleanup by ensuring no endpoints remain for these services:


kubectl get endpoints | grep -E "udp-echo|tcp-db"
# Should return no results

References

This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *