Sherlock Holmes: The Case of the Million NXDomain Errors

If you’ve been following our recent Kubernetes migration blog, you already know the journey has been full of challenges. From configuring pods to tackling networking issues, it’s been a rollercoaster. We’ve explored several tricky problems in previous blogs, and today, we invite you to put on your detective hat and join us as we investigate another Kubernetes mystery.

The Mysterious Case of NXDomain Errors

Imagine this: You’re checking your Kubernetes observability tools, and suddenly, you notice something strange over a million NXDomain errors! What could be causing this? Let’s break it down together.

What Are NXDomain Errors?

Before we jump in, let’s test your DNS knowledge:

Before we jump in — what does an NXDomain error actually indicate?

NXDomain is the resolver telling you the name itself is not there. Not slow, not down — nonexistent. Which makes a million of them very strange indeed.

Unraveling the Clues

We took a closer look at the logs and found something unusual—external domains were mysteriously gaining extra words like .cluster.local or .internal.cloudapp.net. Here are two examples:

gmail.googleapis.com.cluster.local
oauth2.googleapis.com.es52e2p4cafzg4m1it5a.bx.internal.cloudapp.net

Now, let’s put your troubleshooting skills to the test:

Perfectly ordinary Google domains came back with extra words glued onto the end. Who is doing this?

Kubernetes is doing it, and it thinks it is being helpful. The interesting question is no longer who — it is why.

How Kubernetes Handles DNS Queries

To solve this puzzle, we need to understand how Kubernetes resolves DNS queries. When a pod performs a DNS lookup, Kubernetes doesn’t always send the request as-is. Instead, it applies search domains and NDots rules to the query.

Here’s a fun experiment: Try running the following command inside a Kubernetes pod:

cat /etc/resolv.conf

What do you see? You should find an entry for search domains and an ndots value. These settings influence how Kubernetes resolves domain names.

Connecting the Dots

Because the ndots value was set to 5, Kubernetes treated gmail.googleapis.com as an incomplete domain and appended search domains, turning it into:

gmail.googleapis.com.svc.cluster.local
gmail.googleapis.com.cluster.local

These domains don’t exist, leading to the dreaded NXDomain errors!

Fixing the Problem

Now that we’ve cracked the case, let’s apply the fix. Here’s how you can customize DNS settings to prevent Kubernetes from modifying external domains:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 1.2.3.4
    searches:
      - ns1.svc.cluster-domain.example
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "2"
      - name: edns0

The Outcome: A Smooth DNS Experience

By adjusting the DNS configuration, we prevent Kubernetes from mistakenly modifying external queries. This eliminates NXDomain errors and ensures external services resolve correctly.