Right, let’s talk about journald’s split personality when it comes to storage. This isn’t just some academic distinction; it dictates whether your precious logs vanish into the ether after a reboot or stick around for you to autopsy later. It’s the difference between a detective having a crime scene and just having a vague memory of what might have happened.

By default, most distros ship with the journal stored only in memory (/run/log/journal/). This is the “volatile” storage. It’s fast, it doesn’t wear out your SSD with a million tiny writes, and it’s perfect for… well, for situations where you don’t care about logs after a reboot. I can’t think of many of those situations, but they must exist. The moment you shut down, poof—the journal is gone. It’s like a court reporter with amnesia.

The good stuff, the stuff you actually need when things go sideways at 3 AM, is the “persistent” journal. This gets written to /var/log/journal/ and survives reboots, kernel panics, and your questionable decisions to try a custom kernel module.

Making Journal Storage Persistent

Enabling persistence is dead simple. You’re literally just creating a directory and telling systemd to get on with it. The existence of the /var/log/journal/ directory is the trigger.

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

That’s it. No, seriously. The next time your system logs something, it will start appending to the persistent storage in /var/log/journal/. The -p flag in mkdir ensures the parent directories exist, so you don’t have to think about it. After a restart, you’ll see your machine’s boot logs magically appear when you run journalctl -b -1 to see the previous boot. It feels like witchcraft, but it’s just competent design.

Why the Split Exists (And Why It’s Annoying)

The rationale for the volatile default is, admittedly, rooted in practicality. Not all systems have a writable /var/—think embedded devices or read-only root filesystems. For them, a persistent journal is a non-starter. The problem is that this sensible default for a niche case got applied to everyone, including your server that you absolutely need to debug. It’s a classic case of a good intention leading to a collective facepalm for the rest of us. Always check if your distro of choice has already made it persistent (Fedora and others often do); if not, make it your first order of business.

Managing the Beast: Journal Vacuuming

Here’s the catch: logs that live forever eventually consume all your disk space. This is not an if, it’s a when. Unlike logrotate which works on individual files, journald manages its own housekeeping through two settings: SystemMaxUse and SystemMaxFiles. You’ll find these in /etc/systemd/journald.conf.

The journalctl command itself is your tool for manual intervention. The --vacuum family of commands is brilliantly named—it sucks up the old logs.

To shrink the journal to, say, 500MB of the most recent logs:

sudo journalctl --vacuum-size=500M

Or to keep only the last two days of logs:

sudo journalctl --vacuum-time=2days

The best practice? Don’t rely on manual vacuuming. Set sane limits in your journald.conf before you get a panicked email about /var/ being at 100%. Uncomment and set these:

[Journal]
SystemMaxUse=1G
SystemMaxFiles=100
SystemMaxFileSize=100M

This tells journald: “Don’t use more than 1GB total, don’t use more than 100 files, and don’t make any single file bigger than 100MB.” It’s like putting a leash on a friendly, but very hungry, dog.

The Gotcha: Mixing Storage and journalctl -b

Here’s a fun edge case that will ruin your day if you’re not aware. If you boot into a rescue shell or a live CD and mount your root partition, you might try to read the logs from the disk. If you run journalctl --file=/mnt/var/log/journal/*/system.journal, you’ll see all the logs. But if you use the -b flag to filter for a specific boot, it will fail spectacularly.

Why? Because the -b flag relies on the current machine’s boot ID, which, from your live CD, is totally different from the boot IDs in the journal file you’re reading. You have to use the --list-boots flag first to see the boot IDs in that journal file, and then specify the one you want.

# First, see what boots are recorded on the disk
sudo journalctl --file=/mnt/var/log/journal/*/system.journal --list-boots

# Then, use the index from that list (e.g., -1 for the last boot)
sudo journalctl --file=/mnt/var/log/journal/*/system.journal -b -1

It’s a bit clunky, but it makes sense once you realize the journal is a single, structured file, not a series of rotated text files. The designers traded simplicity of file management for a bit of complexity in edge-case forensics. I can live with that.