Disk full on a Linux server: free space safely and fast
A full disk stops databases, mail, logging and package updates, so fix it before anything else. Find out which filesystem is full and what is using it, then remove things that are safe to remove: caches, old logs, old kernels and unused container images. Never free space by deleting files you don't recognise, especially under /var/lib.
Do this first.
- Run
df -handdf -ito see which filesystem is full, and whether it is space or inodes. - Free a little room quickly with safe steps (journal vacuum, package cache) so services and SSH keep working.
- Don't delete files in
/var/lib(databases, package databases, Docker data), and don't delete log files that a running service still has open. - If space is disappearing fast, find the writer first (step 2). Something may be looping, or an attacker may be storing data.
Symptoms
- “No space left on device” in application logs, from
apt/dnf, or when saving a file. - Databases stop accepting writes, mail queues stall, websites return errors.
- Tab completion in the shell fails, or SSH logins hang.
df -hshows 100%, ordf -ishows 100% IUse% while there is still free space (out of inodes).
Step 1: Which filesystem?
df -h
df -i
Look at the Use% and Mounted on columns. /, /var and /boot are the usual suspects. If df -i is full, you have too many small files (often session files, mail queue files or cache fragments), not too much data.
Step 2: What is using it?
Measure the top-level directories of the full filesystem only. -x stays on one filesystem so other mounts don't confuse the totals:
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -15
Repeat on the biggest directory until you find the culprit. ncdu does the same interactively (install it from your distribution's repositories). Run it read-only so a stray key press can't delete anything:
sudo ncdu -x -r /
For an inode problem, count files instead: sudo du -x --inodes --max-depth=1 / | sort -n | tail.
If space keeps shrinking while you look, find the writer with sudo lsof +D /path/to/dir or check which log is growing with ls -lt.
Step 3: Safe quick wins
systemd journal
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=500M
--vacuum-size deletes the oldest archived journal files until the total is below the limit. It doesn't touch active files; add --rotate first if most of the space is in the active file. To make a limit permanent, create /etc/systemd/journald.conf.d/size.conf with:
[Journal]
SystemMaxUse=1G
then sudo systemctl restart systemd-journald (restarts the logging service). By default journald uses at most 10% of the filesystem, capped at 4 GB.
Package caches
# Debian and Ubuntu
sudo apt clean
# RHEL, Rocky, AlmaLinux, Fedora
sudo dnf clean all
Both only remove downloaded package files and metadata, which are fetched again when needed.
Old kernels
Old kernels fill /boot and /lib/modules. Remove them through the package manager, which keeps the running kernel: sudo apt autoremove --purge on Debian and Ubuntu, sudo dnf remove --oldinstallonly on the RHEL family. Never rm files in /boot. If an update already failed because /boot was full, see broken apt or kernel boot failure.
Docker
sudo docker system df
sudo docker system prune
docker system prune removes stopped containers, unused networks, dangling images and unused build cache. It asks before it starts and keeps volumes by default “to prevent important data from being deleted”. -a also removes every image not used by a container, so the next deploy has to pull or rebuild them. Don't add --volumes unless you are sure no stopped container's anonymous volume holds data you need. Never delete files inside /var/lib/docker by hand.
Container logs are another common cause. The default json-file log driver has no size limit. Set max-size and max-file under log-opts in /etc/docker/daemon.json; this needs a Docker restart (restarts the daemon) and only applies to containers created afterwards.
Step 4: Logs and deleted-but-open files
A huge log in /var/log is common. Don't rm it while the service is running: the program keeps the file open, so the space is not freed. Rotate it instead:
sudo logrotate -f /etc/logrotate.conf
-f forces rotation even if logrotate thinks it isn't due. Compressed old copies are then kept according to each file's rotate setting. If one log grows very fast, add maxsize to its configuration so it rotates early, and find out why it is so noisy.
If df still shows the disk full after you deleted files, a process is holding them open. List open files that have been deleted:
sudo lsof +L1
Restart the service named in the output (restarts that service), for example sudo systemctl restart nginx, and the space is released. If you can't restart it right now, you can empty the file through the process's file descriptor (PID and FD come from the lsof output; drop the letter after the FD number): sudo truncate -s 0 /proc/PID/fd/FD. Only do this for log files you are sure you don't need.
Step 5: Other places to check
- /tmp and /var/tmp. Old files here are usually safe to remove, but not files a running program is using right now. On Debian 13,
/tmpis a RAM-backed tmpfs by default, so filling it uses memory, not disk. - Core dumps and crash files in
/var/crashor/var/lib/systemd/coredump: checkcoredumpctl listfirst; remove them once you no longer need them for debugging. - Database files and binary logs. Never delete these by hand. Use the database's own commands to expire old binary or write-ahead logs, and see database recovery if a database stopped.
- Backups and dumps left on the same disk. Move them off the server, which is where they belong anyway.
- Files you don't recognise, such as large archives in web directories or
/tmp, or hidden directories full of data: these can be signs of a compromise. Keep them as evidence and see signs of compromise.
Don't delete random files in /var/lib. It holds the package databases (/var/lib/dpkg, /var/lib/rpm), databases (/var/lib/mysql, /var/lib/postgresql), container storage and service state. Deleting files there can make a system or database unrecoverable. Use the owning tool to clean up.
Confirm it is fixed
df -handdf -ishow the filesystem well below 90%.- Services that failed are running again:
systemctl --failedis empty. Restart any that stopped (restart). - Logs show writes succeeding, and
sudo apt updateorsudo dnf makecacheworks. - Re-run
df -han hour later to make sure usage isn't climbing again.
Prevent it next time
- Alert at 80% and 90% for both space and inodes on every filesystem.
- Set journal and Docker log limits, and check that logrotate covers your application logs.
- Keep data, logs and backups off
/where possible (separate filesystems or volumes). - Let package tools remove old kernels, and keep backups on another system.
When to get help
Call your hosting provider or an administrator if a database won't start after the disk filled, if the filesystem reports errors or remounted read-only, or if you need to grow a disk or volume and aren't sure how. If unknown data appeared quickly, treat it as a possible incident: see incident response.
Sources
- du(1), man7.org
- ncdu manual, Yoran Heling
- journalctl, systemd documentation
- journald.conf, systemd documentation
- logrotate(8), man7.org
- lsof(8), man7.org
- proc_pid_fd(5), man7.org
- apt-get(8), Debian manpages
- DNF command reference, DNF documentation
- docker system prune, Docker Docs
- docker system df, Docker Docs
- JSON File logging driver, Docker Docs
- Issues to be aware of for trixie, Debian 13 release notes