HackBlue

Tools · Linux command line for security · 20 / 28

How to forward Linux logs to a central server with rsyslog

Send auth and system logs from lab clients to a central rsyslog server over TCP with a disk queue, and verify nothing is lost when the server restarts.

10 min read · practice only with authorization

Logs on the host are not enough

An intruder with root can delete or edit local logs. Logs that leave the host as they are written survive that. A central server also lets you search across many machines at once, which is how you notice the same failed login on twenty hosts.

rsyslog is installed on most distributions and reads the journal’s forwarded messages, so it is the simplest place to start.

The receiving server

On the central VM, enable the TCP input module and a template that writes each client to its own directory. Open port 514/tcp only to your client network. In production, use TLS on port 6514 with certificates; the lab uses plain TCP to keep the moving parts few.

Restart rsyslog and confirm it is listening with ss -tlnp.

Snippet
# /etc/rsyslog.d/10-receiver.conf (server)
module(load="imtcp")
input(type="imtcp" port="514")
template(name="PerHost" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
if $fromhost-ip != "127.0.0.1" then { action(type="omfile" dynaFile="PerHost") stop }

sudo systemctl restart rsyslog && sudo ss -tlnp | grep 514

Lab: client with a disk-assisted queue

On each client, forward authpriv and everything at info or above to the server over TCP with a queue that spools to disk if the server is unreachable. Generate a test line with logger, stop the server, generate more lines, start the server again, and confirm the buffered lines arrive.

That last step matters. Forwarding that silently drops logs during an outage fails exactly when you need it.

Snippet
# /etc/rsyslog.d/90-forward.conf (client)
*.info;authpriv.* action(type="omfwd" target="192.168.56.30" port="514" protocol="tcp"
  queue.type="LinkedList" queue.filename="fwdq" queue.saveOnShutdown="on"
  queue.maxDiskSpace="200m" action.resumeRetryCount="-1")

sudo systemctl restart rsyslog
logger -p authpriv.notice "forwarding test $(date -Is)"
# server
sudo tail /var/log/remote/*/logger.log

Next steps

Add TLS with the gtls driver, restrict which clients may send, and set retention on the server. Monitor for clients that stop sending, since silence can mean a broken agent or a deliberate stop.

Many teams eventually forward from rsyslog into a SIEM. The same client configuration works; only the target changes.

Diagram

Logs leaving the host safely

  1. Client

    rsyslog reads local and journal messages.

  2. Queue

    A disk-assisted queue buffers during outages.

  3. Transport

    TCP in the lab, TLS on 6514 in production.

  4. Server

    Writes per-host files and alerts when a client goes quiet.

    Command
    sudo tail /var/log/remote/*/sshd.log

Check yourself

1. Why forward logs off the host?

2. What prevents losing logs while the central server is down?

3. Which port and transport are typical for production syslog over TLS?