Debugging Kubernetes Cluster DNS Failures: Resolving Pod Service Resolution Errors Step-by-Step

Intermediate · 10 min read

The Problem

In a microservices architecture, services rarely operate in isolation. They constantly communicate with one another over the network. In Kubernetes, this inter-service communication relies heavily on internal Domain Name System (DNS) resolution. When a frontend application needs to call a backend API, it typically uses a logical service name rather than a hardcoded IP address. However, when this DNS resolution fails, the consequences are immediate and severe.

Imagine deploying a new version of your payment processing service. The pods start up, pass their readiness probes, and begin accepting traffic. Suddenly, error rates spike. The logs reveal a cascade of failures: the payment service cannot reach the inventory service, and the order service cannot reach the payment service. The root cause is not a bug in the application code, but a failure in the cluster’s internal DNS. The pods are throwing “Could not resolve host” errors. This pain point halts deployments, causes cascading application failures, and leads to significant downtime. Debugging these DNS failures quickly is a critical skill for any platform engineer.

Core Concepts

Before diving into the debugging steps, it is essential to understand the fundamental components that make Kubernetes DNS work. We will define the key terminology from first principles.

• CoreDNS: The default, flexible DNS server used in Kubernetes. It acts as the authoritative DNS server for the cluster, translating service names into IP addresses.
• Service Discovery: The mechanism by which pods find and communicate with each other. Kubernetes automatically assigns a DNS name to every Service object.
• ClusterIP: A virtual IP address assigned to a Kubernetes Service. It is only reachable from within the cluster. When you resolve a service name, the DNS server returns this ClusterIP.
• resolv.conf: A configuration file located at /etc/resolv.conf inside every Linux container. It tells the operating system’s DNS resolver which nameserver to query and what search domains to append to relative names.
• ndots: A parameter in resolv.conf that defines the threshold of dots in a domain name. If a queried name has fewer dots than the ndots value, the resolver will append the configured search domains before attempting to resolve it.
• kube-dns Service: A standard Kubernetes Service in the kube-system namespace that provides a stable ClusterIP for the CoreDNS pods. Pods are configured to use this ClusterIP as their nameserver.

How It Works

Understanding the mechanics of DNS resolution in Kubernetes requires tracing a query from the application to the CoreDNS pod.

When a pod is scheduled, the kubelet injects DNS configuration into the container’s /etc/resolv.conf file. The nameserver is set to the ClusterIP of the kube-dns service. The search domains are set to include the pod’s namespace, the svc domain, and the cluster.local domain. Crucially, ndots is set to 5.

When an application inside the pod attempts to resolve a relative name like backend-api, the standard C library resolver checks the number of dots. Since backend-api has zero dots, which is less than the ndots value of 5, the resolver does not query the nameserver immediately. Instead, it sequentially appends the search domains. It first queries backend-api.default.svc.cluster.local, then backend-api.svc.cluster.local, and finally backend-api.cluster.local.

These queries are sent to the kube-dns ClusterIP. The cluster’s network proxy (iptables or IPVS) intercepts traffic destined for this ClusterIP and load-balances it across the actual CoreDNS pods. CoreDNS receives the query, checks its internal cache, and if there is a miss, it queries the Kubernetes API server for the Service and Endpoint objects. Once it finds the matching service, it returns the corresponding ClusterIP to the pod.

Prerequisites

To follow along with this tutorial, you will need the following:

• A running Kubernetes cluster (version 1.25 or later is assumed for this guide).
• The kubectl command-line tool installed and configured to communicate with your cluster.
• Basic familiarity with Kubernetes Pods, Services, and Namespaces.
• A container image with DNS debugging tools. We will use nicolaka/netshoot, which includes dig, nslookup, and other networking utilities.

Hands-On Walkthrough

We will now walk through a systematic approach to debugging a DNS failure. We will create a target service, simulate a resolution failure, and trace the issue step-by-step.

First, create a dedicated namespace for our debugging exercise to keep resources isolated.

kubectl create namespace dns-debug

Next, create a dummy backend service that our debug pod will try to resolve. This service does not need actual running pods; we are just testing the DNS resolution of the service name itself.

cat <<EOF | kubectl apply -n dns-debug -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-api
spec:
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
EOF

Now, deploy a debug pod using the netshoot image. This pod will sleep indefinitely, allowing us to exec into it and run diagnostic commands.

cat <<EOF | kubectl apply -n dns-debug -f -
apiVersion: v1
kind: Pod
metadata:
  name: dns-debugger
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:latest
    command: ["sleep", "infinity"]
EOF

Step 1: Inspect the Pod’s DNS Configuration

The first step in any DNS debugging session is to verify what the pod is actually configured to use. Exec into the debug pod and examine the resolv.conf file.

kubectl exec -it -n dns-debug dns-debugger -- cat /etc/resolv.conf

You should see output similar to this:

nameserver 10.96.0.10
search dns-debug.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Verify that the nameserver IP matches the ClusterIP of the kube-dns service in the kube-system namespace. If this IP is incorrect, the kubelet is failing to inject the correct DNS settings, which points to a cluster configuration issue rather than a CoreDNS issue.

Step 2: Test DNS Resolution

Attempt to resolve the target service name from within the debug pod.

kubectl exec -it -n dns-debug dns-debugger -- nslookup backend-api

If this fails with a timeout or “server can’t find”, we have confirmed the DNS failure. To isolate the issue, try resolving the fully qualified domain name (FQDN).

kubectl exec -it -n dns-debug dns-debugger -- nslookup backend-api.dns-debug.svc.cluster.local

If the FQDN resolves but the short name fails, the issue is likely related to the search domains or the ndots configuration. If both fail, the issue is with CoreDNS itself or network connectivity to it.

Step 3: Verify CoreDNS Pod Health

Check if the CoreDNS pods are actually running and ready in the kube-system namespace.

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

Ensure the pods are in the Running state and that the READY column shows 1/1. If a pod is in a CrashLoopBackOff or Pending state, CoreDNS is down, which explains the resolution failure.

Step 4: Analyze CoreDNS Logs

If the pods are running, the next step is to check their logs for errors. CoreDNS logs can reveal API server connectivity issues, plugin errors, or malformed queries.

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Look for errors related to the kubernetes plugin, such as failures to list services or endpoints. If you see “connection refused” to the API server, CoreDNS has lost connectivity to the control plane.

Step 5: Check the kube-dns Service and Endpoints

Even if CoreDNS pods are running, the kube-dns service might not be routing traffic to them correctly. Check the service and its endpoints.

kubectl get svc kube-dns -n kube-system
kubectl get endpoints kube-dns -n kube-system

The ENDPOINTS list in the service output, and the output of the endpoints command, should contain the IP addresses of the running CoreDNS pods. If the endpoints list is empty, the service selector does not match the CoreDNS pod labels, or the pods are not passing their readiness probes.

Verifying It Works

Once you have identified and resolved the underlying issue (for example, by restarting a crashed CoreDNS pod or fixing a network policy), you must verify that DNS resolution is functioning correctly.

Run the nslookup command again from the debug pod:

kubectl exec -it -n dns-debug dns-debugger -- nslookup backend-api

The expected output should show the server address matching the kube-dns ClusterIP, and a non-authoritative answer containing the ClusterIP of the backend-api service.

For a more detailed verification, use the dig command to inspect the query path and timing:

kubectl exec -it -n dns-debug dns-debugger -- dig backend-api.dns-debug.svc.cluster.local +stats

Verify that the Query time is low (typically under 5ms if cached) and that the ANSWER SECTION correctly returns the A record for the service.

Common Mistakes

When debugging and configuring Kubernetes DNS, engineers frequently encounter several pitfalls.

• Ignoring the ndots:5 Performance Penalty: Because ndots is set to 5, a query for an external domain like api.github.com (which has 2 dots) will trigger 5 failed internal searches before finally querying the upstream DNS. This generates massive unnecessary load on CoreDNS. How to avoid: Always use fully qualified domain names (ending with a dot, like api.github.com.) for external traffic, or configure custom dnsConfig in your pods to lower the ndots value.
• Hardcoding ClusterIPs: Developers sometimes hardcode the ClusterIP of a service in their application configuration. ClusterIPs are dynamic and can change if the service is recreated. How to avoid: Always use the DNS service name. If an IP is strictly required, use a Headless Service or configure a static IP via a LoadBalancer.
• Modifying resolv.conf Directly: Attempting to change DNS settings by modifying /etc/resolv.conf inside a running container. These changes are ephemeral and will be lost on pod restart. How to avoid: Use the Pod spec’s dnsConfig field to inject custom DNS settings declaratively.
• Overlooking Network Policies: Assuming CoreDNS is failing when it is actually being blocked by a NetworkPolicy. If pods cannot send UDP port 53 traffic to the CoreDNS pods, resolution will fail. How to avoid: Always ensure your NetworkPolicies explicitly allow egress to the kube-system namespace on port 53 for DNS traffic.

Going Further

Once you have mastered basic DNS debugging, you can explore advanced topics to optimize and secure your cluster’s DNS infrastructure.

NodeLocal DNSCache: In large clusters, the kube-dns ClusterIP can become a bottleneck, and the iptables rules for DNS can introduce latency. NodeLocal DNSCache runs a DNS caching agent on every worker node as a DaemonSet. Pods are configured to point to the local node’s IP for DNS, drastically reducing latency and offloading traffic from the central CoreDNS pods.

CoreDNS Tuning: CoreDNS is highly modular. You can tune its performance by adjusting the cache plugin to increase the Time-To-Live (TTL) for successful and negative responses. Enabling the prometheus plugin allows you to export metrics to monitor query rates, cache hits, and panics, integrating DNS health into your broader observability stack.

Custom Upstream Resolvers: If your pods need to resolve internal corporate domains (e.g., .internal.corp), you can configure CoreDNS to forward specific zones to custom upstream DNS servers using the forward plugin, rather than sending all external traffic to the default public resolvers.

Service Mesh Integration: For advanced traffic management, service meshes like Istio or Linkerd inject sidecar proxies that can intercept and route traffic. While they still rely on Kubernetes DNS for initial service discovery, they add a layer of mTLS and advanced routing logic on top of the standard ClusterIP resolution.

Cleanup

To remove the resources created during this tutorial and keep your cluster clean, delete the dedicated namespace. This will automatically remove all pods, services, and other resources within it.

kubectl delete namespace dns-debug

Architecture

References

• DNS for Services and Pods
• Debugging DNS Resolution

---

This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.