Logging and monitoring basics for Linux servers

Updated November 2026

Logs tell you what happened on a server, but only if they exist when you need them and an attacker could not erase them. Know where your system writes logs, keep them long enough, copy them to a second machine, and look at a few key items every day.

Where logs live

Every current systemd-based distribution collects logs with systemd-journald. It stores messages from services, the kernel and login sessions in a binary journal that you read with journalctl. Some systems also run rsyslog, which receives the same messages and writes plain text files under /var/log.

Default logging by release
ReleaseJournalText files from rsyslog
Debian 12 and 13Yes, on disk in /var/log/journalNot installed by default since Debian 12; no /var/log/syslog or auth.log unless you add rsyslog
Ubuntu 24.04, 26.04YesCheck with dpkg -s rsyslog; if installed: /var/log/syslog, /var/log/auth.log
RHEL 9 and 10 (and rebuilds)YesYes: /var/log/messages, /var/log/secure, /var/log/maillog, /var/log/cron

Some programs write their own files whatever the system logger is: web servers (/var/log/nginx/, /var/log/apache2/ or /var/log/httpd/), databases, mail servers and the audit daemon (/var/log/audit/audit.log).

The journal is readable by root and by members of the systemd-journal, adm and wheel groups. Useful commands:

# is the journal kept on disk? (a list of several boots means yes)
journalctl --list-boots

# errors and worse since the last boot
sudo journalctl -p err -b

# one service, last hour, then follow new lines
sudo journalctl -u nginx --since "-1h"
sudo journalctl -u nginx -f

# kernel messages from the current boot
sudo journalctl -k

# how much space the journal uses
journalctl --disk-usage

The journal uses /var/log/journal if that directory exists. If it does not, logs are kept only in memory under /run/log/journal and are lost at every reboot. To make the journal persistent, create the directory or set Storage=persistent (below).

How long logs are kept

By default journald limits the journal to 10% of the file system, capped at 4 GB, and keeps at least 15% of the disk free. There is no time limit by default (MaxRetentionSec=0). On a busy server, 10% of a small disk may hold only a few days. Set your own limits in a drop-in file instead of editing the main file:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/retention.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=90day
EOF

# RESTARTS journald (logging continues; takes a second)
sudo systemctl restart systemd-journald

To free space right now, sudo journalctl --vacuum-size=1G or sudo journalctl --vacuum-time=30d deletes old archived journal files. Text files under /var/log are rotated by logrotate; see /etc/logrotate.conf and the files in /etc/logrotate.d/ for how many weeks are kept.

How long is enough? Intrusions are often found weeks later, so keep at least 90 days somewhere. Your local copy can be short if the central copy is long. Check any legal or contractual rules that apply to you.

Send logs off the server

An attacker with root can delete or edit local logs. A copy sent as it happens to another machine cannot be changed from the hacked server. It also survives a dead disk. Use one of these two methods.

Services, kernel, sshd, sudo journald rsyslog or journal-upload Central log server TCP + TLS
Logs leave the server as they are written. The central server should have separate credentials.

Option 1: rsyslog

rsyslog speaks the standard syslog protocol, which almost every log server, SIEM and hosted log service accepts. On Debian install it first with sudo apt install rsyslog. On the sending server, create /etc/rsyslog.d/90-forward.conf:

*.* action(type="omfwd"
    target="logs.example.net" port="514" protocol="tcp"
    queue.type="LinkedList" queue.filename="fwd_central"
    action.resumeRetryCount="-1" queue.saveOnShutdown="on")

The queue settings keep messages in memory and spill them to disk while the log server is unreachable, then send them when it returns. Use TCP, not UDP: UDP silently drops messages. Check the syntax, then restart:

sudo rsyslogd -N 1
sudo systemctl restart rsyslog    # RESTARTS rsyslog

On the receiving server, load the TCP input in /etc/rsyslog.d/ with module(load="imtcp") and input(type="imtcp" port="514"), and open that port only to your servers. Plain TCP is not encrypted. Across the internet, use TLS (port 6514 by convention): rsyslog supports it with StreamDriver="gtls" or "ossl", StreamDriverMode="1" and certificate checks, which needs the matching driver package (rsyslog-gnutls or rsyslog-openssl). Follow the rsyslog TLS documentation linked below. On RHEL, a non-standard port also needs an SELinux label (semanage port -a -t syslogd_port_t -p tcp PORT) and a firewalld rule.

Option 2: systemd-journal-upload

If both ends run systemd and you want to keep the full journal fields, install the systemd-journal-remote package on both machines. The sender runs systemd-journal-upload, which pushes journal entries over HTTPS (default port 19532) and remembers where it stopped. The receiver runs systemd-journal-remote, which stores each host's entries under /var/log/journal/remote/.

# sender: set URL= (and certificate files) in /etc/systemd/journal-upload.conf
#   [Upload]
#   URL=https://logs.example.net:19532
sudo systemctl enable --now systemd-journal-upload.service

The receiver needs its own key and certificate, and should trust only certificates you issued. Read the journal-remote manual before exposing it.

Lock down the log server. Do not reuse the web server's SSH keys or passwords on it, accept only the log port from your servers, and patch it like any other machine. If one account can reach both, a single break-in can still erase both copies.

What to look at every day

Five minutes a day, or a script that mails you the results, catches most problems early:

Unusual entries worth acting on: logins at odd hours or from new countries, new user accounts, cron jobs or systemd units you did not create, services restarting in a loop, outbound connection errors from the web server user, and gaps where logs stop for no reason. Signs of compromise lists more.

Alerts and dashboards

Reading logs by hand does not scale past a few servers. Common approaches, from simplest to largest:

Whatever you choose, test it: stop a service or log in from a new address and confirm the alert arrives. An alert channel that nobody reads is the same as no alert.

Sources