Right, so you’ve got logs. Lots of them. They’re spewing out of your systems like confetti from a cannon, and right now they’re probably all just piling up in /var/log/syslog, which is about as useful as a screen door on a submarine. We need to bring order to this chaos, and rsyslog is our tool of choice. It’s the venerable workhorse of Linux logging, and it’s powerful enough to make you weep with joy or frustration, sometimes simultaneously. Forget the basic syslog; rsyslog is its modern, plugin-driven, über-powered descendant. Let’s bend it to our will.

The rsyslog.conf Structure: It’s All About the Rules

The main configuration file lives at /etc/rsyslog.conf. Its structure is deceptively simple, divided into three parts: modules, global directives, and rules. The rules are where the magic happens, and they follow a simple pattern:

filter.action

That’s it. The filter part selects a message, and the action part says what to do with it (write to a file, forward it, etc.). The most common, old-school filter is the facility.severity syntax.

  • Facility: Who sent the log? Is it the kernel (kern), an authentication process (auth), a generic system daemon (daemon), or literally any random user-level application (user)? They all have a label.
  • Severity: How urgent is it? From panic-inducing (emerg) down to purely informational (debug). The key thing to remember: selecting a severity means that level and everything higher. mail.err will catch err, crit, alert, and emerg messages from the mail facility.

So, a rule like this:

auth.*    /var/log/auth.log

…means “take every single message from the auth facility, no matter its severity, and append it to /var/log/auth.log.”

The asterisk * is a wildcard. The keyword none means “exclude this facility.” This is useful for excluding chatty debug logs from a specific source.

Property-Based Filters: Where the Real Power Lies

The facility.severity syntax is ancient and, frankly, a bit clunky. It’s like trying to perform surgery with a sledgehammer. Modern rsyslog gives us RainerScript, a more precise and powerful way to filter based on any property of a message.

Want to filter based on the hostname? The program name? Or even the message content? This is how you do it.

# Send all messages from the 'nginx' program to a specific file
if $programname == 'nginx' then /var/log/nginx.log
& stop

# Send any message containing the word 'error' (case-insensitive) to a dedicated error log
if $msg contains 'error' then /var/log/error.log
& stop

# A more complex example: send SSH failed login attempts to a security log
if ($programname == 'sshd') and ($msg contains 'Failed password') then /var/log/ssh-failures.log
& stop

The & stop is crucial here. It means “if this rule matches, process the message with this action and then stop processing it; don’t let it fall through to any other rules.” Without it, your SSH failure message would end up in both ssh-failures.log and auth.log. This is a classic “why are my logs duplicated?!” pitfall. Always ask yourself: “Should this message be processed by other rules after this one?” If the answer is no, use & stop.

Forwarding Logs: Becoming a Log Shipping Master

The single most important thing you can do for basic operational sanity is to forward your logs to a central, remote host. Why? Because when the server hosting /var/log/ is on fire and unreachable, its logs are gone. The remote host is your single source of truth.

On the client (the server generating logs), you need to configure it to send messages. The old-school way uses a single directive:

*.* @192.168.1.100:514

This means “all facilities, all severities, send via UDP to port 514 on 192.168.1.100.” See that @? That means UDP. Using UDP for logging is like sending a postcard: it’s fast, there’s no overhead, but it offers no guarantee the message will arrive. If the network blips or the remote server is overwhelmed, your logs vanish into the ether. Not great.

The modern, reliable way uses a forwarding rule with the omfwd module and explicitly defines the protocol. This goes in your rsyslog.conf:

# Define a custom template for the messages we're sending. This ensures they are RFC 5424 compliant.
template(name="RemoteForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%")

# Now use that template to forward ALL messages via TCP to the remote server
action(
    type="omfwd"
    template="RemoteForwardFormat"
    queue.type="linkedlist"        # Use a disk-assisted queue in case of network outage
    queue.filename="fwdq"          # Base filename for the disk queue
    queue.maxdiskspace="1g"        # Don't use more than 1GB of disk space for the queue
    queue.saveonshutdown="on"      # Save messages to disk if rsyslog shuts down
    queue.timeoutenqueue="on"      # Enable timeouts for the queue
    action.resumeRetryCount="-1"   # Retry indefinitely until the connection comes back
    Protocol="tcp"                 # The key part: use reliable TCP
    Target="log-archiver.example.com"
    Port="10514"                   # Often a non-standard port is used
)

See all that queue stuff? That’s your lifeline. If the central log server goes down for 24 hours, the client will spool logs to its local disk (/var/lib/rsyslog/) until the connection is restored, then ship them all. This is how you achieve reliability. You’re not just sending logs; you’re guaranteeing they get there.

On the server (the central host), you need to uncomment these lines to enable listening:

# Provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="10514")

# Provides UDP syslog reception (if you must, but you really shouldn't for anything important)
# module(load="imudp")
# input(type="imudp" port="514")

Reloading, Not Restarting

You’ve made changes. Now what? Never, ever sudo service rsyslog restart on a production machine. You will lose in-flight messages. The correct incantation is always:

sudo systemctl reload rsyslog

This tells the main rsyslogd process to gracefully re-read its configuration file without killing its active connections and dropping the messages currently in memory. It’s a small thing that separates the pros from the amateurs. Now go forth and corral those logs. Your future self, frantically debugging a 3 AM outage, will thank you.