Right, let’s get our hands dirty and talk about what actually runs your code. The control plane gets all the glamour, but the worker nodes are the grunts doing the real work. They’re the ones sweating in the data center trenches, and they’re made up of three key components that you absolutely must understand: the kubelet, kube-proxy, and the container runtime. If any one of these fails, your pod is basically a fancy paperweight.

The kubelet: The Node’s Belligerent Foreman

Think of the kubelet not as a gentle guide, but as the control plane’s fiercely loyal, slightly obsessive enforcer on the node. Its entire job is to take a set of PodSpecs (YAML manifests that describe a pod) from the API server and scream at the local container runtime until the actual containers match that desired state. It runs as a system service (not in a container) on every node, and it’s the one thing that makes a node a Kubernetes node.

If the API server says, “I need three replicas of nginx running here,” the kubelet is the component that makes it happen. It’s also responsible for reporting back: it constantly tells the control plane the current state of its node (“Node condition: Ready, I have 3.4GB of memory free, and oh, by the way, pod nginx-xyz just crashed”). This two-way communication is the heartbeat of your cluster.

Here’s a classic “oh no” moment: you ssh into a node and manually start a Docker container using docker run. The kubelet has no idea this container exists. It didn’t come from a PodSpec. The kubelet will not manage it, it won’t report on it, and if a health check fails, it might even kill it to make room for a pod it is supposed to be managing. The kubelet is a jealous god. You don’t manually mess with containers on a node; you let the kubelet handle it.

kube-proxy: The Overworked Network Switch Operator

Every pod gets its own IP address. This is brilliant for isolation and a nightmare for networking. How does a request from a pod in Node A find its way to a pod on Node Z? Enter kube-proxy. This component runs on every node and manages the low-level, fiendishly complicated networking rules that make service discovery and load balancing work.

It primarily works by manipulating Linux iptables rules (or increasingly, IPVS) to create a virtual IP for a Kubernetes Service. When you create a Service named my-app, it gets a stable ClusterIP. kube-proxy on every node sees this and sets up a rule that says, “Any traffic destined for IP 10.96.0.12 (the ClusterIP) should be redirected to one of the healthy pods that match this Service’s selector.” It does this by rewriting the destination IP of the packet on the fly.

You can see its handiwork. Let’s say you have a service named my-service. You can peek behind the curtain with iptables:

sudo iptables -t nat -L KUBE-SERVICES | grep my-service

This will show you the complex chain of rules that kube-proxy has built to redirect traffic to the correct backend pods. It’s not pretty to look at, but it’s incredibly effective. The common pitfall here is assuming it’s a traditional proxy—it’s not. It’s a rules manager. This means the actual packet forwarding is handled by the kernel, which is blazingly fast, but debugging can make you want to cry. Remember, kube-proxy handles cluster-internal traffic; it’s not typically your ingress point from the outside world (that’s for an Ingress controller).

The Container Runtime: The Actual Muscle

This is the part that always causes confusion. The kubelet doesn’t run containers itself. It needs a lower-level software to do the grunt work of pulling images, creating container namespaces, and running the containerd process. This is the Container Runtime Interface (CRI)—a plugin API that lets Kubernetes be runtime-agnostic.

You’ve probably heard of Docker. Well, here’s the fun part: as of Kubernetes v1.24, the built-in DockerShim was removed. Docker itself is not a CRI-compatible runtime. It’s too high-level. So what happens now? Typically, you use containerd (which was the core component Docker itself used under the hood) or CRI-O. The kubelet talks to containerd via the CRI, and containerd does the actual work.

This is a good thing! It removes a layer of complexity. But it means old commands break. You used to docker ps on a node to see your containers. Now, you must use the runtime’s native tooling, or better yet, use the Kubernetes abstraction.

# To see containers from a Kubernetes perspective, use the CLI that talks to the API server, not the node.
kubectl get pods -o wide

# To see containers from the node's perspective using containerd
sudo ctr -n k8s.io containers list

The key insight is that the runtime is interchangeable. The kubelet defines what to run, the CRI-compatible runtime defines how to run it. The best practice is to stop thinking about individual containers and start thinking about pods. Let Kubernetes manage the runtime; you manage the declarative state.