Signs your Linux server has been hacked (and how to check)
Most break-ins leave traces (indicators of compromise) an administrator can find in ten minutes: a process that should not be there, a new account or SSH key, a cron job, a connection to an unknown address, or a changed system file. This page lists the checks in order, with commands. If any check turns up something you cannot explain, stop and follow the first hour of incident response.
Look, do not clean. Do not delete files, kill processes or reboot while you check. That destroys evidence and can warn the attacker. Write down what you find, with the time. Remember that on a fully compromised server (an attacker with root) the tools themselves can lie, so a clean result here is a good sign, not proof.
Before you start
Run the checks as root (or with sudo), otherwise you will not see other users’ processes and files. If you can, log in through your provider’s web console rather than SSH, and do not type passwords you also use elsewhere: assume anything typed on a hacked machine can be recorded.
1. Processes and CPU use
Unexplained high CPU is the classic sign of a cryptominer (cryptojacking). Look for unknown names, processes running as the web server user (www-data, apache, nginx) that start shells, and programs started from /tmp, /var/tmp or /dev/shm.
# top CPU users, with parent PID, owner and run time
ps -eo pid,ppid,user,%cpu,etime,cmd --sort=-%cpu | head -15
# full process tree
ps auxwwf
# programs whose file was deleted after start, or that run from temp dirs
sudo ls -l /proc/*/exe 2>/dev/null | grep -E 'deleted|/tmp/|/dev/shm/'
A deleted executable is not always bad (a service updated but not restarted shows the same), but a deleted binary in /tmp almost always is. See suspicious process and cryptominer for what to do next.
2. New users and UID 0 accounts
Only root should have user ID 0. Attackers often add a second UID 0 account with an innocent name, or give a login shell to a system account.
# every account with UID 0 (should print only "root")
awk -F: '$3 == 0 {print $1}' /etc/passwd
# accounts that can log in (have a real shell)
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}' /etc/passwd
# when the account files last changed
ls -l --time-style=long-iso /etc/passwd /etc/shadow /etc/group
3. SSH authorized_keys
An added public key gives the attacker a password-free way back in, and survives a password change. Check every user, including root and service accounts.
sudo find / \( -path /proc -o -path /sys \) -prune -o \
\( -name authorized_keys -o -name authorized_keys2 \) -type f -print |
while read -r f; do sudo ls -l --time-style=long-iso "$f"; sudo cat "$f"; done
# where sshd actually looks for keys (it can be changed in the config)
sudo sshd -T | grep -i authorizedkeys
Every key should be one you can name. Also look for command= options you did not add.
4. Cron jobs and systemd timers
Scheduled jobs are the most common way malware restarts itself after you kill it (persistence). Look for jobs that download something (curl, wget), decode base64, or run files from temp directories.
sudo ls -la /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /var/spool/cron
for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
# systemd timers and recently changed unit files
systemctl list-timers --all
sudo find /etc/systemd /usr/lib/systemd /lib/systemd -type f -mtime -30 -ls
Find backdoors covers more hiding places: shell profiles, rc.local, ld.so.preload and PAM modules.
5. Outbound connections
A hacked server usually talks to someone: a command server, a mining pool, or other victims it is attacking. ss shows each connection with the process that owns it.
# established TCP connections with process names, no name lookups
sudo ss -tpn
# everything listening for incoming connections
sudo ss -tulpn
Ask of each line: should this program talk to this address? A PHP or web server process with an outgoing connection to an unknown IP, or a listener on a port you never opened, needs an explanation. Do not ping, look up or browse the suspicious addresses: CISA warns that touching attacker infrastructure can tip them off.
6. Modified system binaries
Package managers keep checksums of the files they installed, so you can see which ones changed.
# Debian, Ubuntu (install with: sudo apt install debsums)
sudo debsums -c # list changed files from packages
sudo debsums -l # packages that have no checksums to compare
# RHEL, Alma, Rocky, Fedora, SUSE
sudo rpm -Va
In rpm -Va output, each line has nine test characters: 5 means the file digest differs, S the size, M the mode or permissions, T the modification time; a c marks a configuration file. Changed config files are normal. A 5 on a file in /usr/bin, /usr/sbin or a library directory is not.
Limit. The checksums live on the same disk, so an attacker with root can change them too. The debsums manual itself says it is “of limited use as a security tool” and points to integrity tools that keep their database on safe media. See file integrity monitoring.
7. Web shells in upload folders
A web shell is a small script, often PHP, that lets the attacker run commands through the website. Upload directories should contain images and documents, never code.
# PHP files changed in the last 7 days under the web root
sudo find /var/www -type f -name '*.php*' -mtime -7 -ls
# any PHP file inside an uploads directory (WordPress and others)
sudo find /var/www -path '*/uploads/*' -name '*.php*' -ls
Open suspicious files read-only (less) and look for eval, base64_decode, system or shell_exec on user input. Some legitimate plugins use these too, so compare with a fresh copy of the same version. More in website hacked.
8. Sudo rights
# active lines of sudoers and its include directory
sudo sh -c 'grep -Ev "^[[:space:]]*(#|$)" /etc/sudoers /etc/sudoers.d/*'
sudo ls -l --time-style=long-iso /etc/sudoers.d/
# members of the admin groups (sudo on Debian/Ubuntu, wheel on RHEL/SUSE)
getent group sudo wheel
Look for unknown users, NOPASSWD entries you did not add, and files in /etc/sudoers.d/ with recent dates.
9. Login history
Look for logins at odd hours, from countries you never work from, or for accounts that should never log in.
# successful logins with full times and IP addresses
last -Fi | head -30 # RHEL family, Debian 12, Ubuntu 24.04
wtmpdb last -i | head -30 # Debian 13: needs wtmpdb and libpam-wtmpdb
# failed logins
sudo lastb -Fi | head -30 # where available
sudo lslogins --failed # Debian 13 alternative
# SSH log for the last week (unit is ssh on Debian/Ubuntu, sshd on RHEL/SUSE)
sudo journalctl -u ssh -u sshd --since "7 days ago" | grep -E 'Accepted|Failed'
# with auditd: logins this week, numbers turned into names
sudo ausearch -m USER_LOGIN -ts this-week -i
Debian 13 dropped the old last and lastb because the wtmp file format breaks in 2038. Note that the login records are files the attacker can edit, so a gap in the history is itself suspicious.
10. Disabled or wiped logging
Attackers often turn off logging or empty the files to hide their tracks.
systemctl is-active systemd-journald rsyslog auditd
ls -la /var/log | head -40 # zero-size or missing files?
sudo journalctl --verify # damaged journal files
sudo ls -la /root/.bash_history # a link to /dev/null is a red flag
A history file linked to /dev/null, a log that stops suddenly, or auditd stopped without a reason all point to someone covering their tracks. Sending logs to another machine (see logging and monitoring) is the only reliable defence against this.
What rootkit scanners can and cannot do
Tools like rkhunter and chkrootkit compare the system against lists of known rootkits and suspicious settings. They are worth running, but:
- They find known malware. New or custom tools are missed.
- They run on the suspect system and rely on its kernel. A kernel-level rootkit can hide files and processes from them.
- They raise false alarms, especially after updates, so read each warning before you act on it.
A clean scan does not make a server clean. For a trustworthy check, scan the disk from a clean system (for example, a provider snapshot attached to another machine). See malware scan.
Found something?
- Stop checking and write down what you found, with times.
- Do not delete the evidence or reboot.
- Go to the first hour of incident response: isolate, preserve, then rebuild.
- For a specific problem, start at emergency help.
Sources
- ss(8) manual page, man7.org
- debsums(1), Debian trixie manual pages
- rpm(8) manual page, verify output, RPM project
- last(1), lastb(1), util-linux, man7.org
- wtmpdb(8), Debian trixie manual pages
- Debian 13 release notes: last, lastb and lastlog replaced
- journalctl(1) manual page, man7.org
- ausearch(8) manual page, man7.org
- Technical Approaches to Uncovering and Remediating Malicious Activity (AA20-245A), CISA
- Joint advisory AA20-245A full text, Canadian Centre for Cyber Security