Right, so you’ve got your cluster humming along, and nginx.default.svc.cluster.local resolves like a charm. But let’s be honest, you don’t want to type that out, and your applications certainly shouldn’t have to. You want to resolve payments.service or magic.internal or database.prod. This is where we stop letting Kubernetes call all the shots and start teaching its DNS system, CoreDNS, some new tricks.

The magic—and the occasional source of frustration—is that CoreDNS is overwhelmingly configured through a single ConfigMap living in the kube-system namespace. It’s a bit like being given the keys to a sports car but being told you can only adjust the steering by editing a single, massive XML file under the hood. Powerful, but handle with care.

The CoreDNS ConfigMap: Your Steering Wheel

First, let’s see what the stock configuration looks like. Run kubectl -n kube-system get configmap coredns -o yaml. You’ll get something wonderfully familiar and slightly intimidating.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

This block of text, specifically the Corefile under data, is the entire universe for your cluster’s DNS server. It’s written in the configuration language of Caddy (which CoreDNS is built upon). Each block, denoted by braces {}, defines a zone (here, the root zone .:53) and the plugins that handle requests for that zone.

The key plugin for Kubernetes magic is kubernetes cluster.local .... This is what automatically creates DNS records for every Service and Pod in your cluster. The fallthrough option is a crucial piece of cleverness: if CoreDNS can’t find a record in the cluster.local zone, it will “fall through” to the next plugin, which is often used to forward the query to your upstream nameservers (e.g., your company’s DNS or a public resolver like 8.8.8.8).

Writing Your Own Custom Domains

Now, the fun part. Let’s say you have an external legacy database that will never run in Kubernetes, living at legacy-db.corporate.com. You want all applications in the cluster to resolve my-database.internal to that address. We do this by adding a new hosts plugin block before the kubernetes block.

Why before? Because CoreDNS executes plugins in the order they are defined. We want it to check our custom mappings first; it’s faster and we avoid the potential weirdness of the kubernetes plugin.

Here’s how you modify the ConfigMap to make it happen:

# coredns-custom-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        # Our custom hosts file mapping
        hosts {
          192.168.100.123 my-database.internal
          10.100.10.5 my-other-service.internal
          fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

The fallthrough directive inside the hosts block is just as important here as it is in the kubernetes plugin. It says, “If you don’t find a match in my custom list, keep going down the plugin chain.” Without it, queries for my-database.internal would work, but queries for nginx.default.svc.cluster.local would mysteriously fail because the request would never make it to the kubernetes plugin.

To apply this: kubectl apply -f coredns-custom-configmap.yaml. CoreDNS pods will automatically reload their configuration within seconds (thanks to the reload plugin), so no need for a pod restart.

Common Pitfalls and the Art of Debugging

This is where the witty friend gets serious for a moment. This system is powerful but brittle.

  1. Order of Plugins Matters: As mentioned, putting your hosts block after the kubernetes block means it will never be reached for any cluster.local query. Think about the flow.

  2. The Dreaded loop: If you mess up your forwarding and CoreDNS ends up forwarding requests to itself, it will detect the loop and halt. You’ll see a Loop detected error in the logs. This is a good safety feature, but it means your DNS is now broken. Test your config changes carefully.

  3. Forgetting fallthrough: This is the most common mistake. If you create a custom zone block (e.g., for internal.), you must include fallthrough if you want non-matching queries in that zone to be resolved by the rest of your Corefile. Otherwise, they just hit a dead end.

  4. Debugging with dig: The best way to see what’s happening is to exec into a pod and use dig. Install it with apt-get update && apt-get install dnsutils if it’s not there.

    kubectl run -it --rm debug --image=busybox:1.35 --restart=Never -- /bin/sh
# Then inside the container:
/ # nslookup my-database.internal

    This will tell you exactly which server responded and with what answer. If you get an answer from a server that isn’t the CoreDNS service IP, you know your fallthrough or forwarding is misconfigured.

Ultimately, wielding this ConfigMap is a core admin skill. It feels a bit janky to be editing a massive string in a YAML file, but it gives you an incredible amount of control. You’re not just configuring DNS; you’re directly shaping the network reality of every pod in your cluster. So make your changes, test them relentlessly, and enjoy the power of making magic.internal actually work.