Emergency help for Linux and Unix servers: help yourself first
Something has gone wrong. Stay calm, work through the steps in order, and write down what you do and when. Most problems can be fixed, and quick, careful action limits the damage.
A word you do not know? Every technical term is explained in plain language in the glossary.
Do this first if you think you were hacked
- Do not wipe or reboot the server yet: you may destroy evidence of how they got in.
- Cut its access to the internet (cloud firewall or security group), but keep your own console access.
- Change passwords and SSH keys from a different, clean computer.
- Tell your hosting provider, and your national CERT if data may be affected.
Attacks and malware
- RansomwareFiles encrypted, ransom note
- CryptominerCPU at 100%, unknown process
- Website hackedDefaced, redirects, spam pages
- BackdoorsWhat attackers leave behind
- Brute-force attacksThousands of login attempts
- DDoSTraffic flood
Access and boot problems
- Locked out of SSHTimeout, refused, denied
- Forgot root passwordGRUB and rescue mode
- Will not bootAfter a kernel update
- Leaked passwords or keysRotate in the right order
Broken services
- Website down502, 503, 504 errors
- Disk fullFree space safely
- apt or dpkg brokenDebian and Ubuntu
- dnf or rpm brokenRHEL family and Fedora
- SSL certificate expiredRenewal failed
- Database will not startMySQL, MariaDB, PostgreSQL
All emergency guides are listed below, and the Top 50 questions page answers the most common questions in a few lines each.
- RansomwareLinux or ESXi server encrypted by ransomware? Isolate it, keep evidence, identify the strain, look for a free decryptor, report it and rebuild from clean backups.
- CryptominerServer CPU stuck at 100% from kdevtmpfsi, kinsing or xmrig? Find the miner and hidden processes, remove cron, systemd and ld.so.preload persistence, close the hole.
- Website hackedSite redirecting to spam or flagged by Google? Find web shells and injected code, check WordPress core checksums, restore, update and request a Search Console review.
- Malware scanHow to scan a Linux server with ClamAV, Linux Malware Detect, rkhunter, chkrootkit and YARA, verify packages with debsums and rpm -Va, and what these tools cannot see.
- Find backdoorsFind attacker persistence on Linux: new users, sudoers, SSH keys, cron, systemd units and timers, rc.local, ld.so.preload, PAM, SUID files, kernel modules.
- Locked out of SSHCan't SSH into your server? Tell a timeout from refused or denied, get in via console or rescue system, then fix fail2ban bans, firewall rules, keys and sshd config.
- Reset root passwordReset a lost root password on RHEL 9/10 with rd.break and an SELinux relabel, on Debian or Ubuntu with recovery mode or init=/bin/bash, or from a cloud rescue system.
- Leaked credentialsA password, SSH key or token leaked? Rotate in order: root and admins, SSH keys, API tokens, databases, panels, TLS keys, git secrets. Then check for misuse.
- Brute-force attacksThousands of failed logins? Check nobody got in, make SSH key-only, set up fail2ban for sshd and nginx, rate-limit login pages, add MFA, put panels behind a VPN.
- DDoS and floodsServer slow or unreachable from a traffic flood? Confirm it with ss and nginx logs, call your host, add nginx limit_req and limit_conn, check SYN cookies, cut logging.
- Broken apt and dpkgRecover Debian and Ubuntu from an interrupted update: lock errors, dpkg --configure -a, apt --fix-broken install, held packages, a full /boot and NO_PUBKEY key errors.
- Broken dnf and rpmRepair dnf and yum on RHEL, Rocky, AlmaLinux and Fedora: clean caches, rebuild the rpm database, fix conflicts and duplicates, and undo transactions with dnf history.
- Kernel boot failureBoot the previous kernel from GRUB, make it the default with grubby or update-grub, rebuild the initramfs with update-initramfs or dracut, and fix a full /boot.
- Disk fullFind what filled a Linux disk with df and du, then free space safely: vacuum the journal, rotate logs, release deleted open files, clean caches, old kernels and Docker.
- Roll back an updateUndo a package update that broke a server: apt install pkg=version and apt-mark hold, dnf history undo and downgrade, snapper rollback on SUSE, LVM and cloud snapshots.
- SSL certificate expiredFix an expired TLS certificate: check expiry with openssl, test certbot renew --dry-run, fix port 80, webroot, DNS and rate-limit errors, reload the server.
- IP blacklisted / spamServer sending spam or IP on a blocklist? Find the source in the Postfix queue, stop it, clean the queue, fix the hole, then ask Spamhaus and others to delist.
- Website down (502/503/504)Fix 502, 503 and 504 errors: check services, nginx -t, the PHP-FPM socket and pool, upstream timeouts, error logs, disk, memory and rate limits.
- Database won't startMySQL, MariaDB or PostgreSQL won't start? Read the error log, fix disk and permissions, copy the data, use innodb_force_recovery safely, restore a dump.
- Domain or DNS hijackedDomain or DNS hijacked? Contact your registrar, recover the accounts, check NS and DS records, find rogue certificates on crt.sh, then add locks and 2FA.
- Ransomware-proof backupsServer backups an attacker cannot delete: the 3-2-1 rule, what to back up, database dumps, encrypted restic and Borg with append-only remotes, and restore tests.
- Restore from backupRebuild a hacked or broken Linux server the safe way: fresh OS, patch first, restore data not programs from a clean backup, rotate every secret, verify, go live.
- Recover deleted filesGet back a deleted file on Linux: stop writing to the disk, copy still-open files from /proc, check snapshots and backups, and know the limits of undelete tools.
- High load or out of memoryFind out fast why a Linux server is slow: load vs CPUs, top, vmstat, iostat, free -h, OOM killer messages in dmesg, swap, runaway processes, and attack signs.
- Suspicious process or portCheck an unknown Linux process or listening port without destroying evidence: ps --forest, /proc/PID/exe and cwd, deleted binaries, ss, lsof, systemctl status, hashes.
- End-of-life serverOn a Linux release that no longer gets security fixes? Isolate it, cut exposure, weigh paid extended support (ELTS, Ubuntu Pro, RHEL ELS, TuxCare) and migrate.
- Vulnerability reportA host, scanner or researcher reported a vulnerability? Check it against backported fixes, prioritize with CISA KEV and EPSS, patch, and reply to the reporter.
- Abuse complaintAbuse complaint that your server is scanning, brute-forcing or spamming? It is usually compromised. Find the process, contain it, clean up and reply to the abuse desk.
- Phishing on your serverFake login page on your server? Archive it as evidence, take it offline, find how it got there, report it to Google Safe Browsing, APWG and Netcraft, and look for more.
- Personal data breachPersonal data leaked from your server? Contain it, assess what was exposed, and learn the GDPR Art. 33/34 duties: 72-hour notification, informing people, records.
- Top 50 questionsShort answers to the 50 questions admins ask most about hacked Linux servers, cryptominers, SSH attacks, safe patching, reboots, CVEs and end-of-life systems.