Watch processes and outbound connections for signs of compromise

Updated November 2026

An attacker who gets code running on a server almost always does one of two things: opens a way back in (a new listening port) or reaches out (to a mining pool, a command server, or other victims). If you know what normal looks like for your server, both stand out within minutes.

All commands below run as root or with sudo on current Linux distributions. Run them on a healthy server first, so you have something to compare with later.

Take a baseline

ss lists sockets. -t is TCP, -u UDP, -l listening only, -p shows the owning process, -n keeps numbers instead of names, and -H drops the header line.

# what is listening, with process names
sudo ss -tulpn

# save a compact baseline: protocol and local address:port
sudo sh -c "ss -Htuln | awk '{print \$1, \$5}' | sort -u > /root/listen-baseline.txt"

Write down why each listener exists. Anything you cannot explain is worth checking now, before it is an emergency (see Server hardening).

Check outbound connections

Without -l, ss -tupn shows connected sockets. Connections your server started use a local port from the ephemeral range (Linux default 32768 to 60999; check /proc/sys/net/ipv4/ip_local_port_range), so a port filter separates them from visitors connecting in:

# all connected TCP and UDP sockets with processes
sudo ss -tupn

# outbound TCP connections only, with the process that owns each one
sudo ss -Htnp state established '( sport >= :32768 )'

# outbound connections counted by remote port
sudo ss -Htn state established '( sport >= :32768 )' |
    awk '{n = split($4, a, ":"); print a[n]}' | sort | uniq -c | sort -rn

On a typical web server you expect 443 and 80 (updates, APIs), 53 (DNS), maybe 3306 or 5432 to a database host and 587 to a mail relay. A busy mix of other ports is worth a closer look.

Red flags

Common signs of a compromised server
What you see	Often means
Outbound to 6667 or 6697	IRC, which servers rarely need and some botnets use for control
A process with `stratum+tcp://` in its arguments, or a CPU-heavy process with a random name	Cryptocurrency miner talking to a mining pool
Hundreds of outbound connections to port 22 or 25	Your server is scanning others or sending spam
The web server user (`www-data`, `apache`, `nginx`) running `sh`, `curl`, `wget` or `perl`	A webshell or exploited application
A program running from `/tmp`, `/var/tmp` or `/dev/shm`, or whose file was deleted	Dropped malware hiding its file
A new listener on a high port bound to `0.0.0.0`	A backdoor or bind shell

# programs started from temp folders, or whose binary was deleted
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '/tmp/|/var/tmp/|/dev/shm/|\(deleted\)'

# miner-style command lines
ps -eo pid,user,etime,args | grep -Ei '[s]tratum\+|[x]mrig'

# everything the web server user is running (apache on RHEL)
ps -u www-data -o pid,ppid,etime,args

# outbound SSH and SMTP from this host
sudo ss -Htnp state established '( dport = :22 or dport = :25 )'

Deleted is not always bad. After a package update, services that were not restarted still run the old, deleted binary. Check the package name first and see Restarts and reboots. If you find something truly unknown, do not kill it yet: follow Suspicious process or port or Cryptominer.

Live traffic views

nethogs groups bandwidth by process: sudo nethogs. It answers "which program is using the network right now".
iftop shows bandwidth per connection: sudo iftop -i eth0 -n -N -P (no DNS lookups, numeric ports, show ports; use your own interface name).
conntrack prints new connections as the kernel's connection tracker sees them: sudo conntrack -E -e NEW -o timestamp. It needs the conntrack tool (package conntrack on Debian and Ubuntu, conntrack-tools on RHEL) and a stateful firewall that loads connection tracking.

nethogs and iftop are in Debian and Ubuntu; on RHEL-family systems look in EPEL.

Allow-list outbound traffic

Most servers need only a few outbound destinations. Blocking the rest stops many attacks from finishing: the dropper cannot download its payload, the miner cannot reach its pool. This nftables table adds an output filter next to your existing rules:

# /etc/nftables-egress.nft
table inet egress
delete table inet egress
table inet egress {
    chain output {
        type filter hook output priority 0; policy accept;
        oifname "lo" accept
        ct state established,related accept
        meta l4proto { icmp, ipv6-icmp } accept
        udp dport { 53, 123 } accept          # DNS, NTP
        tcp dport { 53, 80, 443 } accept      # DNS, package updates, APIs
        # tcp dport 587 ip daddr 192.0.2.25 accept   # your mail relay
        limit rate 10/minute log prefix "egress-drop: "
        # drop                                # uncomment after a week of clean logs
    }
}

sudo nft -c -f /etc/nftables-egress.nft     # syntax check only
sudo nft -f /etc/nftables-egress.nft        # load it
sudo journalctl -k --since today | grep egress-drop

The first two lines make the file safe to load again after edits. Run in log-only mode first, add rules for every legitimate destination you see in the log, then uncomment drop. To keep it after reboot, include the file from your distribution's nftables config. If you use ufw or firewalld, build the same policy with their tools instead of mixing (see Firewall basics).

Keep ICMP and DNS open. Blocking outbound ICMPv6 breaks IPv6 neighbor discovery, and blocking DNS breaks almost everything. Malware running as root can also remove these rules; egress filtering mainly stops attacks that run as a normal user, such as webshells.

Record every program run

Process accounting makes the kernel write a record each time a process ends: program name (up to 15 characters, no arguments), user, time. It is cheap and useful after the fact.

# Debian, Ubuntu (service acct)
sudo apt install acct
# RHEL family (service psacct, log in /var/account/pacct)
sudo dnf install psacct && sudo systemctl enable --now psacct

sudo lastcomm --user www-data | head -n 30
sudo lastcomm --command curl

auditd records the full command line, the program path and the login user behind each execve call. Rules for the web server user are the most valuable, since a web server rarely needs to start programs:

# /etc/audit/rules.d/60-exec.rules
# web server user: find its number with  id -u www-data  (33 on Debian, 48 for apache on RHEL)
-a always,exit -F arch=b64 -S execve -F euid=33 -k web-exec
-a always,exit -F arch=b32 -S execve -F euid=33 -k web-exec
# commands from people who logged in
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -k user-exec
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=unset -k user-exec

sudo augenrules --load
sudo ausearch -k web-exec --start today -i

Use one rule per architecture as audit.rules(7) advises; drop the b32 lines if the kernel has no 32-bit support. Execve rules on busy users produce many events, so check log rotation in /etc/audit/auditd.conf. The same ausearch --checkpoint alert script from Login alerts works with these keys.

Automate a port check

This script compares listening sockets with the baseline every 15 minutes and also flags outbound connections to ports outside an allow-list. It alerts once per change, using the send-alert helper from Login alerts (or replace that call with mail -s "…" root).

#!/bin/sh
# /usr/local/sbin/port-watch   (chmod 700, owned by root)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ALLOW=" 22 53 80 443 587 "              # expected outbound TCP ports
umask 077
dir=/var/lib/port-watch
mkdir -p "$dir"
ss -Htuln | awk '{print $1, $5}' | sort -u > "$dir/now"

if [ ! -s "$dir/baseline" ]; then          # first run: accept current state
    cp "$dir/now" "$dir/baseline"; cp "$dir/now" "$dir/last"; exit 0
fi
if ! cmp -s "$dir/now" "$dir/baseline" && ! cmp -s "$dir/now" "$dir/last"; then
    { echo "Listening sockets differ from baseline (> new, < missing):"
      diff "$dir/baseline" "$dir/now" | grep '^[<>]'
      echo; ss -tulpn
    } | send-alert "ports-$(date +%s)" "$(hostname): listening ports changed"
fi
cp "$dir/now" "$dir/last"

# skip inbound connections to local services that listen on high ports
lis=" $(ss -Htln | awk '{n = split($4, a, ":"); print a[n]}' | sort -u | tr '\n' ' ')"
odd=$(ss -Htnp state established '( sport >= :32768 )' |
      awk -v ok="$ALLOW" -v lis="$lis" '{n = split($3, l, ":"); m = split($4, r, ":")
          if (index(lis, " " l[n] " ") == 0 && index(ok, " " r[m] " ") == 0) print}')
[ -n "$odd" ] && printf '%s\n' "$odd" | send-alert outbound "$(hostname): unexpected outbound connections"
exit 0

After an intended change (a new service), accept it with sudo cp /var/lib/port-watch/now /var/lib/port-watch/baseline. If a program opens random UDP ports on every start, filter those lines out with grep -v before sort.

# /etc/systemd/system/port-watch.service
[Unit]
Description=Compare listening ports with baseline

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/port-watch

# /etc/systemd/system/port-watch.timer
[Unit]
Description=Run port-watch every 15 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now port-watch.timer
systemctl list-timers port-watch.timer

# without systemd: /etc/cron.d/port-watch
*/15 * * * * root /usr/local/sbin/port-watch

Checks that run on the server can be switched off by an attacker with root, so treat silence with some caution and compare with an outside view now and then: a port scan from another host, or flow logs from your provider. For broader tooling, see Intrusion detection and Signs of compromise.