Service-Discovery

Right, so you’ve got your cluster humming along, and nginx.default.svc.cluster.local resolves like a charm. But let’s be honest, you don’t want to type that out, and your applications certainly shouldn’t have to. You want to resolve payments.service or magic.internal or database.prod. This is where we stop letting Kubernetes call all the shots and start teaching its DNS system, CoreDNS, some new tricks. The magic—and the occasional source of frustration—is that CoreDNS is overwhelmingly configured through a single ConfigMap living in the kube-system namespace. It’s a bit like being given the keys to a sports car but being told you can only adjust the steering by editing a single, massive XML file under the hood. Powerful, but handle with care.

Alright, let’s get our hands dirty. The theory of Kubernetes DNS is all well and good until you’re staring at a misbehaving service and your application is screaming about an UnknownHostException. This is where you stop guessing and start poking the system directly. The most direct way to do this is by running diagnostic tools like nslookup and dig from inside a Pod. Why inside? Because that’s the exact environment your application is running in. The view from your laptop is a lie; the view from the Pod is the truth.

Right, let’s talk about your cluster’s DNS settings, specifically the two things that cause 90% of “but I can’t resolve that!” headaches: ndots and search domains. You’ve probably run a pod, tried to curl another service, and gotten a timeout, only to realize you needed to use the fully qualified name. I’ve been there. It feels dumb. Let’s demystify why it happens. The core issue is that we humans are lazy. We want to say curl api instead of curl api.production.svc.cluster.local. Kubernetes, in its infinite wisdom, tries to help with this by providing search domains. Your Pod’s /etc/resolv.conf isn’t just a list of nameservers; it’s a recipe for how to try and find a name.

Alright, let’s pull back the curtain on one of Kubernetes’ neater party tricks: giving every Pod its own DNS name. You might be thinking, “It’s 2024, why is this impressive?” Because honestly, it should be table stakes. But the way Kubernetes does it is both elegant and, in places, a bit quirky. This isn’t just some load balancer magic; it’s a core part of the system’s service discovery fabric. Here’s the deal: For a Pod to be discoverable via DNS, it needs an identity more stable than its fleeting IP address. Kubernetes provides this by creating a DNS record for each Pod that has one crucial feature: a defined hostname. This doesn’t happen for every Pod automatically. You have to opt-in, and you do it the Kubernetes way: by setting fields in the Pod spec.

Right, let’s talk about the magic trick that makes your Pods find each other without you having to play network detective. It’s the <service>.<namespace>.svc.cluster.local incantation. This isn’t just some random string of jargon; it’s the fully qualified domain name (FQDN) for every Service you create, and it’s the linchpin of service discovery inside your cluster. Think of it like this: a Service is a stable IP address and a logical grouping of Pods. But IP addresses are for computers; we humans (and our applications) prefer names. The Kubernetes DNS system (usually CoreDNS) is the phone book that maps the friendly name of your Service to that stable IP. The .cluster.local part is the default cluster domain, but it’s configurable if you’re feeling fancy and need to change it. The key thing to understand is that this FQDN is always available, for every Service, the moment the Service is created. You don’t have to configure a thing.

Right, let’s talk about CoreDNS. You’ve probably heard it’s the “DNS for Kubernetes,” which is true, but that undersells it. It’s less like a simple phone book and more like the hyper-intelligent, slightly overworked switchboard operator for your entire cluster. It’s the reason your my-app-service.default.svc.cluster.local doesn’t just resolve to a hopeful dream, but to an actual IP address that other pods can talk to. Before we dive in, a moment of silence for its predecessor, kube-dns. It worked, mostly, but it was a bit of a Rube Goldberg device—a dnsmasq container, a kube-dns container, and a sidecar for metrics, all held together with hope and YAML. CoreDNS is a single, blessedly simple Go binary that does everything with far more grace and configurability. The Kubernetes designers made a great call ditching the franken-service.