Right, let’s talk about upgrade order. You’ve got your cluster, humming along nicely, and you’ve decided it’s time to drag it into the future. The single most important rule, the one you should tattoo on the inside of your eyelids, is this: you upgrade the control plane first, then the worker nodes. Always. This isn’t a suggestion; it’s the law of the land in Kubernetes. Break it, and you’re in for a world of hurt. I’ve seen people try to be clever and do it the other way around. They don’t do it twice.

Why this fanatical devotion to order? Think of the control plane as the brain and central nervous system, and the worker nodes as the limbs. You want to upgrade the brain first so it knows how to properly command the newer, shinier limbs you’re about to attach. An old control plane trying to manage new worker nodes is like trying to run Windows 11 on a 386 processor—it just doesn’t have the vocabulary. The new kubelet on a worker node might be trying to speak API version v1.25, but the old API server in the control plane is still stuck on v1.23. They can’t talk. The result? Your shiny new worker node reports to the API server as NotReady, and you get to spend your afternoon sweating over cryptic error messages instead of looking competent.

The Golden Rule: Control Plane First

The entire upgrade strategy hinges on this sequence. You’ll use your chosen tool (kubeadm, kubectl, your managed cloud console) to upgrade the control plane components one by one: the API server, scheduler, controller manager, etcd (if it’s co-located). Once that’s done and stable, you move on to the worker nodes. This ensures the cluster’s “brain” is always at least as new as, or newer than, its “body.” This is the only way to guarantee API compatibility and avoid the aforementioned circus.

A Practical kubeadm Example

Let’s say you’re moving from Kubernetes 1.27.x to 1.28.y. Here’s how you’d handle the control plane node. First, you drain the node to politely evict its workloads. Yes, even control plane nodes can run pods these days.

# Drain the control plane node, ignoring daemonsets and the static pods
# that actually run the control plane (kubeadm will handle those)
kubectl drain <control-plane-hostname> --ignore-daemonsets --delete-local-data

Now, SSH into the node and do the upgrade. On an apt-based system, it looks like this:

# Fetch the new versions of the kubeadm, kubelet, and kubectl packages
sudo apt-get update
sudo apt-get install kubeadm=1.28.y-00 kubelet=1.28.y-00 kubectl=1.28.y-00

# Now, tell kubeadm to plan the upgrade for the control plane
sudo kubeadm upgrade plan

# If everything looks good, apply it. This is the magic moment.
sudo kubeadm upgrade apply v1.28.y

After this completes, restart the kubelet. The control plane is now upgraded.

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Finally, uncordon the node to mark it schedulable again.

kubectl uncordon <control-plane-hostname>

Why Draining the Control Plane Matters

“You just told me to drain the control plane! Isn’t that… bad?” It feels wrong, I know. But in a highly available (HA) setup, the other control plane nodes will pick up the slack. Even in a single-node control plane setup, the critical control plane pods (api-server, etcd) are run as static pods managed directly by the kubelet on that node. The kubectl drain command cleverly ignores these, so it only evicts your regular workloads. Skipping this drain step is a classic pitfall; you risk corrupting workloads that are running on the node when kubeadm starts messing with the core components.

Handling Multiple Control Planes in an HA Cluster

In an HA cluster, you don’t upgrade all control plane nodes at once. That’s a great way to lose etcd quorum and have a very bad day. You upgrade them one by one, sequentially. Drain -> upgrade -> uncordon the first one. Wait for it to rejoin the cluster and for all components to be healthy. Then move on to the next one. Patience is a virtue, especially here. Rushing this process is a one-way ticket to a restoration-from-backup scenario.

The Worker Node Dance

After all control plane nodes are healthy and running the new version, you repeat a similar—but simpler—process for each worker node.

# Drain the worker node to safely evict all pods
kubectl drain <worker-node-hostname> --ignore-daemonsets --delete-local-data

# SSH to the node, upgrade the kubelet and kubectl
sudo apt-get update
sudo apt-get install kubelet=1.28.y-00 kubectl=1.28.y-00

# Tell the local kubelet to use the new configuration
sudo kubeadm upgrade node

# Restart the kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Uncordon the node to bring it back into the cluster
kubectl uncordon <worker-node-hostname>

The pods you evicted will now be rescheduled by the shiny new control plane onto other nodes (or back onto this one once it’s ready). The kubeadm upgrade node command is crucial—it configures the kubelet on this worker to talk to the newly upgraded control plane.

The One Weird Trick: kubectl convert

Here’s a pro tip for the inevitable “oh crap” moment. Sometimes, you’ll have old manifests lying around that use deprecated API versions. Your new control plane will outright reject them. Before you upgrade, find them! Use kubectl convert (if you still can on the old version) or use a migration tool to update them to the new API group. Finding this out after the upgrade when your CI/CD pipeline starts failing is… suboptimal. Trust me on this one.