Alright, let’s talk about journalctl. If systemd is the hyperactive, over-caffeinated stage manager of your system, then journalctl is the stagehand who’s been keeping a meticulous, slightly obsessive diary of everything that’s ever happened. It’s the single command to rule all your system logs, and it’s both brilliant and, at times, infuriatingly clever.

The first thing you’ll probably type is journalctl by itself. Don’t. It will vomit the entire contents of the system journal onto your screen, starting from the dawn of time (or at least the last log rotation). You’ll be paging through boot messages from three weeks ago before you know what hit you. It’s the log equivalent of drinking from a firehose.

Filtering by Time: Your First and Best Trick

The most common thing you want to do is see what happened recently. For this, we have the -e flag, which jumps to the end of the journal. It’s okay, everyone uses it. I use it constantly.

journalctl -e

But we can do better. So much better. Want to see everything since the last boot? systemd tracks boots with a unique ID, and you can easily reference the last one.

journalctl -b

Now, let’s say your website started doing the macarena at 2:14 PM. You can filter the log to a very specific timeframe. The --since and --until flags are your friends here, and they accept wonderfully human-readable strings.

journalctl --since "2023-10-27 14:14:00" --until "2023-10-27 14:20:00"
journalctl --since "2 hours ago"
journalctl --since yesterday --until "1 hour ago"

Zeroing In: The Unit is What Matters

Filtering by time is good, but filtering by the systemd unit is where you’ll live. Ninety percent of your debugging will start here. Use -u (for unit) to see the logs for one specific service. This is non-negotiable and it’s fantastic.

journalctl -u nginx.service

Now, combine it with -f to follow the logs in real-time, just like the old tail -f. This is how you watch a service start up or lose its mind live.

journalctl -fu nginx.service

Priority and PIDs: Getting Surgical

Sometimes a service is fine, but you want to see only the bad stuff. The -p flag filters by priority. Want to see only errors and worse? This is your jam.

journalctl -p err -b

You can use numbers (0=emerg, 3=err, 6=info) or names. Seeing only crit, alert, and emerg messages is usually a sign you’re having a very, very bad day.

Maybe you’re debugging a specific process. You can grab its PID from ps aux or top and then tell journalctl to show only messages from that process. This is incredibly powerful for isolating a single misbehaving thread from a busy application.

journalctl _PID=1234

The _COMM Filter Trap (And How to Avoid It)

Here’s a classic pitfall. You might be tempted to use _COMM to filter by the command name.

journalctl _COMM=nginx

This works… sometimes. The _COMM field stores the executable name, not the unit name. If your process is a Python script named my_cool_server.py, the _COMM will be python3.9 or whatever interpreter it uses. This is almost useless. You’ll get logs from every Python process on the system. Always prefer -u for services and _PID for specific processes over _COMM.

Following and Exporting: For When Things Get Real

We already covered -f for following, which is essential. But what about when you need to send these logs to someone else? Or archive them? The journal is stored in a binary format for efficiency, but you can export it to something humans can read.

The --output flag is key. The default is short, but for a full export with all the metadata, use export. It spits out JSON-like text, which is perfect for feeding into other tools.

journalctl -u nginx.service --since today --output=json > nginx-today.json

You can also use json-pretty if you want to read it yourself, or cat if you just want a continuous blob of text. Need to send a snippet to a colleague? Use --no-pager to prevent journalctl from launching less, so you can pipe it directly to a file.

journalctl -u failed-service.service --no-pager > service-failure.log

Why This All Works: The Journal’s Secret Sauce

All this powerful filtering works because systemd doesn’t just write text to a file. It parses each log message as it’s received and stores it in a structured, indexed binary database. Every log line has a bunch of metadata attached—the PID, UID, executable name, systemd unit, priority—and journalctl lets you query that metadata directly. This is why _PID=1234 works but grep 1234 /var/log/syslog would also match any log line that contained that number, leading to false positives. The old way was like searching for a word in a book by scanning every page; journalctl is like using the book’s index. It’s a fundamental shift, and once you get used to it, you’ll wonder how you ever lived without it. Even if its creator does have a… strong personality.