Found a suspicious process or open port: investigate safely

Updated November 2026

Do not kill an unknown process the moment you see it. Spend ten minutes recording what it is, where its program lives, who started it and what it talks to. Most turn out to be harmless, and for the rest that record is what lets you remove the intruder completely.

Do this first.

  1. Do not kill the process or reboot yet. Its program, network connections and memory disappear with it.
  2. If it is causing harm right now (attacking others, sending spam, encrypting files), cut the server's network at the provider's firewall rather than killing the process.
  3. Write down the time in UTC and every command you run.
  4. Use a fresh SSH session, and do not run or open the suspicious file.

What looks suspicious

1. Set up an evidence folder

Work as root so that saving output works, and keep everything in one folder. Copy it off the server later.

sudo -i
E=/root/ir-$(date -u +%Y%m%dT%H%MZ)
mkdir -m 700 "$E" && cd "$E"
P=12345                                  # the suspicious PID
date -u > time.txt

2. Place it in the process tree

ps -ef --forest > ps-forest.txt
ps -o pid,ppid,user,lstart,etime,stat,cmd -p $P
less ps-forest.txt                       # search for the PID with /

--forest draws the process tree, so you can see the parent: a shell under php-fpm or apache2 means the web application was used to run commands. lstart gives the start time; compare it with your logs.

Real kernel threads show in square brackets, have parent PID 2 (kthreadd) and have no program file. A bracketed name with any other parent, or with a program file, is a disguise.

3. Inspect /proc/<pid>

ls -l /proc/$P/exe /proc/$P/cwd           # program file and working folder
tr '\0' ' ' < /proc/$P/cmdline; echo      # command line
grep -E '^(Name|PPid|Uid|Gid)' /proc/$P/status
tr '\0' '\n' < /proc/$P/environ > environ.txt
ls -l /proc/$P/fd > fd.txt               # open files and sockets
cat /proc/$P/maps > maps.txt             # loaded libraries
cat /proc/$P/cgroup                       # systemd unit or container

The Linux manual says that if the program file has been unlinked, the exe link shows the original path with (deleted) added. Malware often deletes itself after starting. The link can still be opened, so you can save the program even then:

cp /proc/$P/exe exe.bin
chmod 400 exe.bin
sha256sum exe.bin | tee exe.sha256
file exe.bin

A process can change its own command line, so trust exe and cwd more than the name shown in ps.

4. Check its network activity

ss -tulpn > listening.txt               # listening TCP/UDP ports with process
ss -tunap > sockets.txt                 # all sockets, including outbound
grep "pid=$P," sockets.txt
lsof -nP -p $P > lsof.txt               # every file and socket it has open
lsof -nP -i :4444                        # who uses a given port

In ss, -t and -u select TCP and UDP, -l shows listening sockets, -p shows the owning process and -n skips name lookups. A listener on 0.0.0.0 or [::] is reachable from outside unless your firewall blocks it; 127.0.0.1 is local only.

Common, harmless listeners include sshd on 22, a mail server on 25, systemd-resolved on 127.0.0.53:53, chronyd on 323/udp, rpcbind on 111 and your hosting provider's guest agent. Look up anything else before you decide.

5. Find who owns it

systemctl status $P                      # the unit that contains this PID

# Does the program belong to a package, and is the package intact?
dpkg -S "$(readlink /proc/$P/exe)"       # Debian/Ubuntu
dpkg --verify <package>
rpm -qf "$(readlink /proc/$P/exe)"       # RHEL-family, Fedora
rpm -V <package>

ls -la /tmp /var/tmp /dev/shm
crontab -l -u www-data

A program that belongs to a package and passes verification is very likely legitimate. A file that belongs to no package, in a writable temporary folder, run by the web user, is very likely not. An attacker with root can tamper with package databases, so a clean result is strong but not final evidence. Look for how it starts again after a reboot: see Find backdoors.

6. Check the hash and reputation

Search the SHA-256 hash on a malware reputation service such as VirusTotal. Searching by hash does not send the file. Uploading it does share it: VirusTotal says scan reports are shared with its public community and file contents may be shared with premium customers. Never upload scripts or configs that may hold your passwords or customer data. An unknown hash proves nothing either way; custom malware is often new.

To look inside without running it, strings -n 8 exe.bin | less often shows mining pool addresses, IP addresses or wallet IDs.

7. Contain, then stop it

# optional, gdb package: save its memory for analysis
gcore -o "$E/core" $P
kill -STOP $P                            # freeze it: no CPU, memory kept
kill -KILL $P                            # then remove it
systemctl disable --now <unit>           # if a unit starts it (stops the service)

Freezing stops the damage while you finish collecting and decide. If it comes back after being killed, something restarts it: check its parent, systemd units, cron, and shell startup files. Copy $E to another machine with scp -r.

What it means

When tools disagree. If ps, ls /proc and ss show different things, or CPU use has no visible owner, a rootkit may be hiding processes. Commands on that system can no longer be trusted. Examine a disk snapshot from another machine, or boot a rescue system.

How to confirm it is fixed

When to call a professional

Call an incident response team or your host's security team if the process ran as root, if you find a rootkit, if personal data may have been accessed (Personal data breach), or if several servers show the same thing. Give them your evidence folder and timeline. General steps are in Incident response.

Sources