Right, let’s talk about logging. It’s the duct tape of our industry, and in Kubernetes, it feels like you need a whole lot more of it. You’re not just debugging your application anymore; you’re debugging your application’s entire reality. The first and most crucial thing to burn into your brain is this core Kubernetes design choice: it expects your applications to log to standard output (stdout) and standard error (stderr).

I know, I know. If you come from a world of meticulous log files neatly organized in /var/log/, this can feel a bit… unhygienic. Like just throwing all your laundry into one big pile. But there’s a method to this madness. A container is meant to be an immutable, ephemeral slice of your application. If it logs to a file inside itself, where does that file go when the container crashes and is replaced by a new, pristine one? Exactly. It vanishes into the ether, taking your only clue about the crash with it. By logging to stdout, the logs are captured by the container runtime (like containerd or CRI-O), which is the first step in getting them out of the transient container and into a place where we can actually look at them.

The Container Runtime: Your First Line of Log Defense

So, your app writes a log line to stdout. What now? The container runtime on the node picks it up. Its job is to handle the raw stream of bytes from your container and, crucially, enrich each log entry with some metadata. This is where we get the CONTAINER_ID, the CONTAINER_NAME, and other runtime-specific details. The runtime then typically writes this enriched log data to a file on the node.

Where? It depends. If you’re using containerd (which, let’s be honest, you probably are these days), it doesn’t write a classic text log file. Instead, it writes in a binary format called CRI log format. You’ll find these files in /var/log/pods/. Don’t believe me? Let’s take a look.

First, find your pod’s specifics:

kubectl get pod my-app-pod -o wide
# Note the Node it's running on

Then, if you have access to that node (this is why node-level logging is a pain without tools), you can go exploring:

# Find the directory for your pod's namespace and name
ls /var/log/pods/default_my-app-pod_<random-uuid>/

# You'll see a directory for each container in the pod, symlinked to the actual log files
ls -la /var/log/pods/default_my-app-pod_<random-uuid>/my-app-container/

# Output will look something like:
# 0.log -> /var/lib/containerd/.../log

You can’t just cat these CRI log files directly—they’re not plain text. You need the crictl tool to read them properly:

sudo crictl logs <container-id>

This is the first major “gotcha.” The logs are on the node, in a weird format, and you need node access to read them. This architecture is why we almost never stop here.

The Kubelet and the /var/log/files Fallback

Now, the kubelet’s job is to constantly watch these runtime-specific log files and help make them more accessible. To maintain some backwards compatibility and give us a fighting chance, the kubelet actively maintains symlinks for each container in the old, classic location: /var/log/containers/.

These symlinks point back to the actual log files in /var/log/pods/. This is a small but vital bit of glue. It provides a consistent, predictable path pattern (<pod-name>_<namespace>_<container-name>-<container-id>.log) regardless of the underlying container runtime. This predictable pattern is what allows third-party logging agents to find the logs easily.

Here’s what you’ll see on a node:

ls /var/log/containers/
# my-app-pod_default_my-app-container-<long-id>.log -> /var/log/pods/.../0.log

But—and this is a big but—this is still just a symlink to the same runtime-specific binary log file. The kubelet itself doesn’t do much more than that. It doesn’t ship logs to a central location. It doesn’t rotate them based on size. It’s just a glorified symlink manager for your logs. All the problems remain: the logs are stuck on the node, and if the node dies, catches fire, or is scaled down, those logs are gone forever. Poof.

This is the fundamental limitation of the basic Kubernetes logging architecture. It’s brilliant for local development and debugging with kubectl logs, but it’s a house of cards for any real production system. The designers knew this. They didn’t build this node-level logging to be the final solution; they built it as a stable, consistent interface for something else to hook into. That “something else” is a cluster-level logging agent, which we’ll tackle next. This setup is the reason the DaemonSet is the undisputed champion of Kubernetes logging—it puts a log shipper on every node, right where these files are written.