Right, so your pod is in a broken state. It’s either crashlooping, stuck in Pending, or just behaving in a way that makes absolutely no sense. Your first instinct is to kubectl exec into it to see what’s going on. But what if the container won’t start? You can’t exec into a container that isn’t running. This is the classic “my car won’t start, and I need to look under the hood but the hood is locked” scenario.

Enter kubectl debug. This is your master key for the hood. It allows you to fire up a temporary, “ephemeral” container inside an existing pod’s namespaces. Think of it as a diagnostic sidecar that lives for exactly as long as you need it to. The magic here is that this new container shares crucial Linux namespaces—specifically the process namespace (PID) and the network namespace (NET)—with the target container. This means you can ps aux and see all the processes from the main container, and you can curl localhost to hit the application in the target container. It’s a brilliantly designed surgical tool.

The Basic Incantation

The most common use case is attaching an ephemeral container with a debugging toolbox image to a pod that’s having a bad day. Let’s say your pod my-broken-app is crashlooping. Here’s how you get in:

kubectl debug -it my-broken-app --image=busybox:latest --target=my-broken-app --share-processes

Let’s break this down because it’s important:

  • -it: Good ol’ interactive and pseudo-TTY. You want a shell session.
  • --image: This is your debugger image. busybox is the Swiss Army knife, but you might prefer nicolaka/netshoot for network issues or a distro like ubuntu for more tools. Always pin a tag. Never use :latest in real life; I’m just being lazy here for the example.
  • --target: This is the crucial bit. It specifies which container in the pod you want to share the namespaces with. If your pod only has one container, you can sometimes omit this, but being explicit saves you from future pain.
  • --share-processes: This is what grants you the magic power to see the target container’s processes. Without it, you’d just see the processes of the ephemeral container itself, which is useless.

Once you’re in, you can run ps aux and see everything. You’ll spot your main application process, probably dead or dying, and you can inspect its environment, check for open files with lsof, or test network connectivity.

When Your Node is the Problem

Sometimes the issue isn’t in a pod but on the node itself. Maybe the kubelet is misbehaving, or you need to check systemd logs. For this, kubectl debug has another trick: it can create a privileged pod on the node itself. This is a massive hammer, so use it carefully.

kubectl debug node/my-node -it --image=ubuntu:jammy

This command does something completely different. It schedules a new pod on the my-node node. This pod is hosted on that node, not inside an existing pod. To give you full access, it mounts the host’s filesystem at /host and runs in the hostPID, hostNetwork, and hostIPC namespaces. You are, for all intents and purposes, on the node. You’ll chroot /host first to truly be in the node’s environment.

# Once inside the node debug pod
chroot /host
head -n 10 /var/log/kubelet.log # Now you're reading the node's kubelet log

The Gotchas and Rough Edges

This power doesn’t come without a few quirks. The designers made some… choices.

  1. It’s Beta and Gated: Ephemeral containers are a beta feature and have to be enabled on the API server. If your kubectl debug command fails with a weird API error, this is probably why. Your cluster admins might have disabled it for security reasons.
  2. SecurityContext Clashes: Your debug container runs with its own security context. If your target pod has a super restrictive securityContext (e.g., running as user 1000) and your debug image (like busybox) defaults to root, the kubectl debug command will fail. It can’t magically override the pod’s security policy. You might need to specify --as-user 1000 or use a debug image that runs as a non-root user. This one trips everyone up.
  3. It’s Ephemeral, So Take Notes: The moment you exit the shell, the ephemeral container is destroyed. All your changes and any files you downloaded inside it are gone forever. This is by design—it’s a debug session, not a fix. You’re there to observe, not to perform open-heart surgery. If you need to make a change, you use the information you gathered to rebuild your main application image properly.
  4. Not for Day-to-Day: This is a troubleshooting tool, not a primary means of interaction. If you find yourself constantly relying on kubectl debug to change configs or restart processes, your image design is broken. The goal is immutable infrastructure; the debug container is there to prove why that immutability is a good idea when something goes wrong.

The bottom line? kubectl debug is the closest thing we have to a silver bullet for Kubernetes troubleshooting. It turns previously impossible debugging scenarios into simple, interactive shell sessions. Just remember it’s a powerful tool, not a crutch. Use it to diagnose, then go fix your manifests and images like a professional.