Right, let’s talk about /var/log. This is where your system’s diary lives, and like any good diary, it’s full of secrets, drama, and a meticulous record of everything that’s ever gone wrong. If your system starts acting weird, this is your first crime scene. Don’t just glance at it; learn to read it like a detective.

The Lay of the Land

First, a quick tour. The /var/log directory is the designated dumping ground, by convention and by the Filesystem Hierarchy Standard (FHS), for all log files. This is brilliant because it means you always know where to start looking. You’ll find everything from the kernel’s deepest mutterings (kern.log) to a user failing to log in for the tenth time (auth.log). The structure is mostly flat, which is both a blessing (simplicity) and a curse (a potential mess of hundreds of files). Some applications, being the special snowflakes they are, create their own subdirectories like /var/log/apt or /var/log/nginx, which is actually a decent practice.

Let’s see what’s in there. Crack open a terminal and have a look.

ls -lah /var/log

You’ll see a mix of familiar names and .gz compressed archives—that’s logrotate doing its job, which we’ll get to in a minute. The main players are almost always text files, so you can use less, tail, cat, or grep to your heart’s content. This is by design; the Unix philosophy loves plain text.

The Usual Suspects: System Logs

Your system generates a staggering amount of logs, but a few files are your core crew.

  • syslog: The old guard. This is often the catch-all for system messages. On modern systems, it’s frequently just a symlink to…
  • auth.log (or secure on Red Hat systems): This is the bouncer’s clipboard. Every single attempt to authenticate, sudo command, SSH login (successful or failed), and user session change gets written here. If you suspect a breach, this is your first stop.
# To see recent authentication attempts
tail -f /var/log/auth.log
# Or on CentOS/RHEL
tail -f /var/log/secure
  • kern.log: The kernel’s internal monologue. Hardware issues, driver failures, and other low-level grumblings end up here. If a new USB device makes your system freeze, this file might know why.
  • dpkg.log / yum.log: Your package manager’s receipt book. Every install, update, or removal is meticulously recorded. Handy for figuring out what change might have broken everything yesterday.
# To see what was installed recently via apt
grep "install " /var/log/dpkg.log

Application-Specific Logs

This is where it gets fun. Most services you install will drop their logs here. They usually name the file after themselves, which is shockingly sensible.

  • /var/log/nginx/ or /var/log/apache2/: Web servers don’t just have one log file; they have a whole directory. You’ll typically find access.log (who asked for what) and error.log (what went wrong serving it). The access.log is a goldmine for web traffic analysis.
  • /var/log/mysql/error.log: MariaDB/MySQL. When your database decides to be dramatic, this is its stage. Connection issues, corruption warnings, and crash details live here.

The syslog Saga and journald

Here’s where the designers made a… choice. For decades, we had the syslog protocol and daemons like rsyslog or syslog-ng handling log routing. Then systemd came along and said, “I’ll take it from here,” and gave us journald with its binary journal.

Why binary? Allegedly for efficiency and richer metadata. The downside? You can’t just grep it with standard tools. You must use journalctl. It’s a classic case of solving a problem that sometimes creates a new one. The good news is that most distributions still run rsyslog alongside journald, pulling messages from the journal and writing them to the good old text files in /var/log for compatibility. It’s a bit schizophrenic, but it works.

# To see the systemd journal (the new hotness)
journalctl -f
# To see only messages from the nginx unit
journalctl -u nginx -f

Taming the Chaos: Logrotate

If nothing ever cleaned up /var/log, your hard drive would fill up with logs from the dawn of time. This is where logrotate earns its keep. It’s a cron job that runs daily, and it’s brutally efficient. It will rotate, compress, and eventually delete old log files based on rules in /etc/logrotate.conf and /etc/logrotate.d/.

A typical rule for an application log might look like this (e.g., /etc/logrotate.d/nginx):

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        invoke-rcmd nginx rotate-logs >/dev/null 2>&1
    endscript
}

This tells logrotate to: rotate these files daily, keep 14 compressed archives, delay compressing the previous one (so troubleshooting tools can still tail it), and then signal nginx to reopen its log files after rotation. It’s a masterpiece of pragmatic system administration.

Best Practices and Pitfalls

  1. Don’t rm active log files. If you need to clear a log that’s currently held open by a process (like nginx), echo "" > file.log is safer than rm. The process will keep writing to the now-deleted inode until it restarts, which is confusing. Better yet, use logrotate or signal the process to reopen its logs.
  2. tail -f is your best friend. Watching a log file in real-time (tail -f /var/log/syslog) is the fastest way to see what’s happening right now when you trigger an event.
  3. grep is your other best friend. Needle in a haystack? grep -i error /var/log/syslog is your magnet. Learn its flags: -C for context, -v to invert, -E for extended regex.
  4. Mind the permissions. Logs often contain sensitive info (usernames, IPs, failed passwords). Their permissions are usually set correctly (640, readable by owner/group, not world-readable), but it’s something to be aware of. auth.log is basically a list of everyone’s failed passwords, after all.
  5. When in doubt, sudo. Non-root users typically can’t read most logs. You’ll need to sudo to cat, less, or tail them. It’s a minor annoyance for a major security win.

The /var/log directory isn’t just a folder; it’s the narrative of your system’s life. Learning to read it fluently is what separates someone who just runs commands from someone who truly understands what’s going on under the hood. Now go get grep-ing.