Get alerted on SSH logins, new users and sudo use

Updated November 2026

Most break-ins on servers start with a login that nobody expected: a stolen SSH key, a guessed password, a new account, or a key quietly added to authorized_keys. A few small scripts can tell you within seconds, by local mail or a webhook, so you can act while the attacker is still looking around.

This guide builds four pieces that share one helper: an SSH login alert through PAM, a journal watcher for sudo and account changes, a file watcher for /etc/passwd and SSH keys, and an optional auditd rule set. All commands run as root (or with sudo) on a current Debian, Ubuntu or RHEL-family system with systemd.

SSH login (pam_exec) sudo, useradd (journal) File change (path unit, auditd) send-alert log + rate limit Mail to root Webhook (curl)
Every event goes through one helper that logs it, drops repeats, and delivers it.

The alert helper

One script does the sending, so every alert is logged and rate-limited the same way. It writes each alert to the journal, then sends at most one message per "key" every five minutes. Repeats are still logged, so nothing is lost; you just are not flooded.

#!/bin/sh
# /usr/local/sbin/send-alert   (chmod 700, owned by root)
# usage: send-alert KEY SUBJECT   (message text on stdin)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GAP=300                       # seconds between alerts with the same key
STATE=/run/send-alert
umask 077
key=$(printf '%s' "$1" | tr -c 'A-Za-z0-9_.-' '_')
subject=$2
body=$(cat)

printf '%s\n%s\n' "$subject" "$body" | logger -t send-alert

mkdir -p "$STATE"
now=$(date +%s)
last=$(cat "$STATE/$key" 2>/dev/null || echo 0)
[ $((now - last)) -lt "$GAP" ] && exit 0
echo "$now" > "$STATE/$key"

if [ -r /etc/send-alert/curl.conf ]; then
    printf '%s\n\n%s\n' "$subject" "$body" |
        curl -sS -f -m 10 -K /etc/send-alert/curl.conf --data-binary @- >/dev/null
else
    printf '%s\n' "$body" | mail -s "$subject" root
fi

Without a webhook file, alerts go to root's local mailbox, which needs a working mail system (for example Postfix plus the mailx or bsd-mailx package) and an alias that forwards root's mail to a person. To use a webhook instead, put its address in a root-only curl config file. That keeps the secret URL out of the process list and out of the script:

sudo install -d -m 700 /etc/send-alert
sudo tee /etc/send-alert/curl.conf >/dev/null <<'EOF'
url = "https://alerts.example.net/hook/REPLACE-ME"
header = "Content-Type: text/plain; charset=utf-8"
EOF
sudo chmod 600 /etc/send-alert/curl.conf

If your chat service wants JSON, change the pipe to jq -Rs '{text: .}' | curl … --data-binary @- and set the header to application/json. Alerts contain user names, IP addresses and file diffs; never add passwords, keys or tokens to them.

SSH login alerts with pam_exec

pam_exec runs a command from a PAM stack and passes it the login details as environment variables: PAM_USER, PAM_RHOST, PAM_SERVICE, PAM_TTY and PAM_TYPE. PAM_TYPE is open_session when a session starts. Because it hooks the session step, it fires for password and key logins, and for scp and sftp sessions too (as long as UsePAM yes is set in sshd_config, as it is in the configuration Debian, Ubuntu and RHEL ship).

#!/bin/sh
# /usr/local/sbin/login-alert   (chmod 700, owned by root)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[ "$PAM_TYPE" = open_session ] || exit 0
host=$(hostname)
msg="SSH login to $host: user=$PAM_USER from=${PAM_RHOST:-unknown} at $(date '+%F %T %Z')"
# background + no output: a slow mail server or webhook never delays the login
printf '%s\n' "$msg" | send-alert "ssh-$PAM_USER-$PAM_RHOST" "$msg" >/dev/null 2>&1 &
exit 0

The key includes the user and source address, so a login from a new address always alerts even if the same user logged in a minute ago. Add one line at the end of /etc/pam.d/sshd:

session optional pam_exec.so quiet type=open_session /usr/local/sbin/login-alert

Keep a root shell open. Test with a second SSH login before you close your current session. Use optional, never required, so a broken script cannot lock you out. On SELinux systems, if no alert arrives, check sudo ausearch -m avc -ts recent for denials.

OpenSSH can also run /etc/ssh/sshrc at login, but sshd runs a user's own ~/.ssh/rc instead when that file exists and PermitUserRC is on (the default), and ForceCommand skips ~/.ssh/rc. A user can therefore bypass it, which is why PAM is the better hook.

Sudo and account changes

sudo, useradd, usermod, passwd and friends log to the system journal under their own syslog identifiers. A small loop follows the journal with journalctl -f and alerts on the lines that matter:

#!/bin/sh
# /usr/local/sbin/auth-watch   (chmod 700, owned by root)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
host=$(hostname)
journalctl -f -q --since now -o short-iso \
    -t sudo -t useradd -t usermod -t userdel -t groupadd -t gpasswd -t passwd -t chpasswd |
while IFS= read -r line; do
    case $line in
        --*) continue ;;
        *" sudo["*"incorrect password"*|*" sudo["*"NOT in sudoers"*|*" sudo["*"authentication failure"*)
            key=sudo-failed ;;
        *" sudo["*"COMMAND="*)
            key="sudo-used-$(printf '%s' "$line" | awk '{print $4}')" ;;
        *" sudo["*) continue ;;          # session opened/closed noise
        *) key=accounts ;;
    esac
    printf '%s\n' "$line" | send-alert "$key" "$host: $key"
done
# /etc/systemd/system/auth-watch.service
[Unit]
Description=Alert on sudo use and account changes

[Service]
ExecStart=/usr/local/sbin/auth-watch
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now auth-watch.service

If you would rather not touch PAM, add -t sshd -t sshd-session to the journalctl line and a case for *"Accepted "*. Since OpenSSH 9.8, sshd is split into a listener and a per-session sshd-session program, so newer releases log logins under that name.

Watch passwd and SSH keys

A systemd path unit starts a service whenever a watched file is written and closed (PathChanged=). The service compares each file with a saved copy and mails the difference, so you see exactly which user or key was added.

#!/bin/sh
# /usr/local/sbin/account-files-check   (chmod 700, owned by root)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
umask 077
snap=/var/lib/account-files
mkdir -p "$snap"
changes=
for f in /etc/passwd /etc/group /etc/sudoers /etc/ssh/sshd_config /root/.ssh/authorized_keys; do
    copy=$snap/$(printf '%s' "$f" | tr / _)
    src=$f; [ -e "$src" ] || src=/dev/null
    if [ ! -e "$copy" ]; then cp "$src" "$copy"; continue; fi    # first run: snapshot only
    cmp -s "$src" "$copy" && continue
    changes="$changes
== $f ==
$(diff -u "$copy" "$src")"
    cp "$src" "$copy"
done
[ -n "$changes" ] && printf '%s\n' "$changes" | send-alert "files-$(date +%s)" "$(hostname): account or SSH files changed"
exit 0

The key includes a timestamp so file diffs are never rate-limited away: each snapshot is updated after it is reported. /etc/shadow is left out on purpose, because a diff would put password hashes into an email; password changes show up in the journal watcher instead.

# /etc/systemd/system/account-files.path
[Unit]
Description=Watch account and SSH key files

[Path]
PathChanged=/etc/passwd
PathChanged=/etc/group
PathChanged=/etc/sudoers
PathChanged=/etc/sudoers.d
PathChanged=/etc/ssh/sshd_config
PathChanged=/root/.ssh/authorized_keys

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/account-files.service
[Unit]
Description=Report changes to account and SSH key files

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/account-files-check
sudo /usr/local/sbin/account-files-check      # take the first snapshot
sudo systemctl daemon-reload
sudo systemctl enable --now account-files.path

Limits: path units use inotify, which does not watch subdirectories recursively and cannot see changes made by other machines on NFS. You cannot use wildcards, so add a PathChanged= line (and a file in the loop) for each admin's ~/.ssh/authorized_keys. Files under /etc/sudoers.d trigger the alert but are not diffed by the script above.

Wider coverage with auditd

The Linux audit system records which process and which login user changed a file, even across many home directories. Install it with sudo apt install auditd (Debian, Ubuntu); RHEL-family systems ship it. Add rules in the syscall form that audit.rules(7) recommends:

# /etc/audit/rules.d/50-accounts.rules
-a always,exit -F path=/etc/passwd -F perm=wa -k acct-watch
-a always,exit -F path=/etc/shadow -F perm=wa -k acct-watch
-a always,exit -F path=/etc/group -F perm=wa -k acct-watch
-a always,exit -F path=/etc/sudoers -F perm=wa -k acct-watch
-a always,exit -F dir=/etc/sudoers.d -F perm=wa -k acct-watch
-a always,exit -F dir=/root/.ssh -F perm=wa -k acct-watch
sudo augenrules --load
sudo ausearch -k acct-watch --start today -i

ausearch --checkpoint prints only events it has not shown before, which turns it into a cheap alert source. Run this every five minutes from cron (*/5 * * * * root /usr/local/sbin/audit-alert in /etc/cron.d/audit-alert):

#!/bin/sh
# /usr/local/sbin/audit-alert   (chmod 700, owned by root)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
out=$(ausearch --checkpoint /var/lib/audit-alert.checkpoint -k acct-watch -i 2>/dev/null)
[ -n "$out" ] && printf '%s\n' "$out" | send-alert audit "$(hostname): audited account file change"
exit 0

The first run reports everything already in the log. If ausearch later exits with a checkpoint error (codes 10 to 12), delete the checkpoint file and let it start again.

When an alert fires

Alerts on the same machine can be silenced by an attacker who gets root (see privilege escalation). For stronger protection, also ship logs to another host or use a central tool, as described in Intrusion detection and Logging and monitoring.

Sources