Server slow, high load or out of memory: quick diagnosis
A slow server is short of one of four things: CPU, memory, disk I/O or network. Five commands tell you which one in about a minute. Fix the cause rather than rebooting, because a reboot hides the evidence and the problem usually comes back.
Do this first.
- Before you reboot or kill anything, save a snapshot:
top -b -n 1 > ~/top-$(date +%H%M).txtandsudo dmesg -T | tail -50. - If SSH barely responds, stop the largest non-essential service first (for example a batch job) to get breathing room.
- An unknown process using all CPU is often a cryptominer: do not just kill it, follow Cryptominer.
- If the disk is full, go to Disk full.
Symptoms
- Pages time out, SSH takes seconds to show a prompt, commands hang.
- Monitoring shows a load average far above the number of CPUs.
- Services restart on their own, or a database disappears: the kernel's out-of-memory (OOM) killer may have ended it.
- Your provider warns about CPU or network use.
The first minute
uptime # load average over 1, 5, 15 minutes
nproc # number of CPUs
free -h # memory; read the "available" column
vmstat 1 5 # five one-second samples; ignore the first line
top -b -n 1 | head -20
Load average is not CPU usage. On Linux it counts tasks that are running or waiting to run (state R) and tasks waiting for disk I/O (state D), averaged over 1, 5 and 15 minutes. As a rough rule, a load near the CPU count is busy, and far above it means work is queuing. A load of 40 on 4 CPUs with idle CPU usually points to a disk or storage problem, not a CPU one.
free -h: the available column is "how much memory is available for starting new applications, without swapping". Low free with high buff/cache is normal: Linux uses spare memory as cache.
vmstat's first line shows averages since boot, so read the later lines.
Read the signs
| You see | Likely cause | Next step |
|---|---|---|
High us or sy in vmstat, one process at the top of top | Runaway or busy process, or a miner | CPU section |
High wa, b above 0, processes in state D | Disk or network storage too slow | Disk section |
si/so non-zero every second, low available | Out of memory, swapping hard | Memory section |
High st (steal) | The hypervisor gives your VM less CPU | Contact your host |
| Normal CPU and memory, many connections | Traffic spike, flood or brute force | Attack section |
On kernels with pressure stall information, cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io shows the share of time tasks were stalled on each resource over 10, 60 and 300 seconds. The full line for memory or I/O staying high means the system is thrashing.
CPU: find the busy process
ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -15
top -H -p 1234 # threads of one process
systemctl status 1234 # which service owns this PID
In top, press P to sort by CPU and M by memory. htop is easier to read if it is installed. Note the user and the command: a PHP-FPM pool, a backup job, a search indexer and a stuck cron job are common, harmless causes.
Memory: is the OOM killer at work?
When the kernel cannot reclaim enough memory, it invokes the OOM killer, which picks a process to kill so the rest of the system can continue. By default it logs a table of tasks when it does this.
sudo dmesg -T | grep -iE 'out of memory|oom-kill|killed process'
sudo journalctl -k -b -1 | grep -i 'killed process' # previous boot
ps -eo pid,user,rss,%mem,cmd --sort=-rss | head -15 # biggest users (RSS in KiB)
swapon --show
Typical causes are too many web or PHP worker processes for the RAM, a database cache set larger than the server, a memory leak that grows over days, or a sudden traffic spike. Lower the worker count or cache size and restart that service.
If the server has no swap, a small swap file can stop OOM kills while you fix the cause. The swapon manual calls dd from /dev/zero the most portable way to create one; on Btrfs swap files need special handling, so skip this there.
sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress # 2 GiB, needs free disk
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Swap is a cushion, not a fix. Heavy, constant swapping (high si/so) makes everything slow.
Disk I/O
sudo apt install sysstat # Debian/Ubuntu
sudo dnf install sysstat # RHEL-family, Fedora
iostat -xz 1 3 # ignore the first report (since boot)
ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'
Watch r_await and w_await (milliseconds per request, including queue time) and aqu-sz (queue length). %util near 100% means a single disk is saturated, but the iostat manual warns it does not show the real limit of RAID arrays and modern SSDs. Check for disk errors with sudo dmesg -T | grep -i 'i/o error'; errors mean failing hardware, which is a job for your host. If the disk is nearly full, go to Disk full.
Calm a runaway process
sudo renice -n 19 -p 1234 # lowest CPU priority
sudo ionice -c 3 -p 1234 # disk access only when idle
sudo systemctl restart php8.4-fpm # restarts the service (use your unit name)
sudo kill 1234 # polite stop (SIGTERM)
sudo kill -9 1234 # only if it ignores SIGTERM
A service you recognise can be restarted. A process you do not recognise should be investigated before you stop it: see Suspicious process or port.
When it is an attack
- Cryptominer: an unfamiliar name using nearly all CPU for hours, often run by the web server user, started from
/tmp,/var/tmpor/dev/shm, with outbound connections. See Cryptominer. - Flood or DDoS: CPU in the web server or kernel, very many connections.
ss -sgives totals; see DDoS and floods. - Brute force: many
sshdor login processes; see Brute-force attacks. - Spam or attack traffic from your server: high outbound traffic usually means a compromise; see Signs of compromise.
# Connections per remote IP (top 10)
ss -Htn state established | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head
Look back in time
If the sysstat data collector is enabled, sar shows history, which helps when the problem has already passed: sar -q for load and pressure, sar -r for memory, sar -u for CPU and sar -W for swapping.
How to confirm it is fixed
- The 1-minute load average is back near or below the CPU count, and falling.
free -hshows comfortable available memory andvmstatshowssi/soat 0.- No new OOM messages in
dmesg. - The site and SSH respond normally.
Prevent it
- Size worker pools (PHP-FPM, web server, database connections) to the RAM you have.
- Cap greedy services with systemd:
MemoryHigh=throttles a unit,MemoryMax=is the hard limit, andCPUQuota=caps CPU time. - Alert on load, available memory, disk space and OOM events (Logging and monitoring).
- Patch promptly: many miners arrive through unpatched web apps (Emergency patching).
When to call your host
Contact your provider for high steal time, disk I/O errors, network saturation you cannot filter, or a server that needs a console reboot. If you find signs of an intrusion, follow Incident response.
Sources
- proc_loadavg(5), Linux manual pages
- top(1), procps-ng
- vmstat(8), procps-ng
- free(1), procps-ng
- ps(1), procps-ng
- iostat(1), sysstat
- sar(1), sysstat
- Memory management concepts, Linux kernel documentation
- Documentation for /proc/sys/vm, Linux kernel documentation
- PSI: Pressure Stall Information, Linux kernel documentation
- swapon(8), util-linux
- systemd.resource-control(5), systemd