Backups that survive ransomware: the 3-2-1 rule in practice
A backup only protects you from ransomware if the attacker cannot reach it. Keep three copies on two kinds of storage with one off-site, make at least one copy offline or append-only so this server cannot delete it, encrypt it, and prove it works by restoring from it.
Do this first.
- If you have no recent backup at all, make one now: a database dump plus
/etcand your data, copied to a different machine. - Check whether this server can delete its own backups (a mounted backup disk, an rsync job with
--delete, storage keys with delete rights). If it can, so can an attacker. - Store the backup password or key outside the server, for example in a password manager. Without it, an encrypted backup is useless.
- Restore one file from the backup today to prove it works.
Signs your backups would not survive
- The only copy is on the same server, or on a disk or share that is always mounted. The UK NCSC reports ransomware encrypting connected USB and network drives and reaching connected cloud storage.
- The server holds credentials that allow deleting or overwriting old backups.
- Nobody has restored anything from the backup in months.
- Databases are copied as raw files from
/var/lib/mysqlor/var/lib/postgresqlwhile running. Such copies are often inconsistent. - The backup is not encrypted, so a stolen copy is a data breach.
The 3-2-1 rule in practice
- 3copies of your data, counting the live one
- 2different devices or kinds of storage
- 1copy off-site, away from the server
- 1copy offline or append-only
The NCSC points out that 3-2-1 does not require any copy to be offline, so add that rule on top. For a typical rented server, a workable setup is:
- The live data on the server.
- A nightly encrypted backup to a separate backup host or storage service in another data centre, written in append-only mode, so the server can add new backups but not delete or change old ones.
- A weekly copy that is truly offline (a disk you unplug) or held with your provider's immutable or object-lock storage (copies that cannot be changed or deleted until a set date), kept for at least a few weeks.
Better still is a pull design: the backup host logs in to the server and fetches the data, so the server holds no backup credentials at all. The NCSC's advice is the same idea: backup clients should not hold valid credentials while the storage is not in use. CISA advises keeping "offline, encrypted backups of critical data" and testing them regularly. It also says to use immutable storage "with caution": read your provider's terms for retention and early deletion before you rely on it.
What to back up on a server
| Back up | Why |
|---|---|
/etc | All configuration, including web server, mail, firewall, TLS certificates in /etc/letsencrypt and SSH host keys in /etc/ssh |
/var/www, /srv, /home, /root | Websites, uploads, application data, scripts |
/var/spool/cron | User crontabs (Debian keeps them under crontabs/ inside it) |
| Database dumps | A consistent copy made with the database's own dump tool |
| A list of installed packages | Lets you reinstall the same software on a fresh system |
Application data under /var/lib/<app> | Only for applications that keep data there; dump or stop them first |
Skip /proc, /sys, /dev, /run and /tmp. You do not need the operating system's own programs: after an incident you reinstall those from the vendor, as explained in Restore from backup.
Secrets need special care. Keep an offline record of the backup password, database passwords and API keys, because you may need them when the server is gone.
Dump databases first
Run these as root, for example from a nightly script. The PostgreSQL manual says a pg_dump dump is a consistent snapshot of the database from when the dump started, and it does not block normal use. pg_dump skips roles and tablespaces, so also run pg_dumpall --globals-only. For MariaDB and MySQL, --single-transaction gives a consistent dump only for transactional tables such as InnoDB.
# As root
umask 077
mkdir -p /var/backups/db
# MariaDB (on MySQL use mysqldump with the same options)
mariadb-dump --all-databases --single-transaction --routines --events \
> /var/backups/db/mariadb-all.sql
# PostgreSQL: roles first, then each database in custom format
runuser -u postgres -- pg_dumpall --globals-only > /var/backups/db/pg-globals.sql
runuser -u postgres -- pg_dump -Fc mydb > /var/backups/db/mydb.dump
# Installed packages
dpkg --get-selections > /var/backups/db/packages.txt # Debian/Ubuntu
rpm -qa | sort > /var/backups/db/packages.txt # RHEL, Rocky, Alma, Fedora
On Debian and Ubuntu, root can usually log in to MariaDB through the local socket without a password. On other systems put the credentials in /root/.my.cnf with mode 600.
Option 1: restic with an append-only server
restic encrypts everything it stores and needs the repository password for every operation. The restic manual warns that if you lose the password, the data is "irrecoverably lost". On a separate backup host, run restic's rest-server with --append-only, which "allows creation of new backups but prevents deletion and modification of existing backups", and --private-repos, so each server can reach only its own repository. Serve it over HTTPS.
# On the backup host
htpasswd -B -c /srv/restic/.htpasswd web1
rest-server --append-only --private-repos --path /srv/restic
On the server you are protecting:
sudo apt install restic # Debian/Ubuntu
sudo dnf install restic # Fedora; RHEL-family from EPEL
sudo -i # the rest as root
(umask 077; openssl rand -base64 32 > /root/.restic-pass) # copy this password off the server too
cat > /root/restic.env <<'EOF'
export RESTIC_REPOSITORY=rest:https://backup.example.net:8000/web1/
export RESTIC_REST_USERNAME=web1
export RESTIC_REST_PASSWORD=the-htpasswd-password
export RESTIC_PASSWORD_FILE=/root/.restic-pass
EOF
chmod 600 /root/restic.env
. /root/restic.env
restic init
restic backup --one-file-system --exclude-caches \
/etc /root /home /srv /var/www /var/backups/db /var/spool/cron
restic snapshots
List only paths that exist on your server. restic 0.17 and later can also take a dump straight from a command with --stdin-from-command, and cancel the snapshot if the dump fails. Debian 13 ships restic 0.18; Debian 12 ships 0.14, so dumping to a file first, as above, works everywhere.
Old snapshots must be removed somewhere the server cannot reach. Run forget on the backup host itself, against the local path. For append-only repositories the restic manual recommends --keep-within, because count-based rules can let an attacker's fake snapshots push real ones out:
# On the backup host
restic -r /srv/restic/web1 forget --keep-within 90d --prune
Option 2: BorgBackup over SSH
Borg (version 1.4 at the time of writing) encrypts with repokey modes and can be forced into append-only mode by the backup host. In the backup user's ~/.ssh/authorized_keys on the backup host, lock the server's key to one command and one path:
command="borg serve --append-only --restrict-to-path /srv/borg/web1",restrict ssh-ed25519 AAAA... root@web1
sudo apt install borgbackup # Debian/Ubuntu
sudo dnf install borgbackup # Fedora; RHEL-family from EPEL
sudo -i
(umask 077; openssl rand -base64 32 > /root/.borg-pass)
export BORG_REPO='ssh://borg@backup.example.net/srv/borg/web1'
export BORG_PASSCOMMAND='cat /root/.borg-pass'
borg init --encryption=repokey-blake2
borg key export --paper "$BORG_REPO" # print it and keep it with the passphrase, off the server
borg create --stats --one-file-system --exclude-caches ::'{hostname}-{now}' \
/etc /root /home /srv /var/www /var/backups/db /var/spool/cron
borg list
Borg's documentation lists the limits of append-only mode. borg prune and borg delete still run, but they only mark data as deleted, and the data can be recovered until someone compacts the repository in normal mode. Prune and compact from a separate admin key, only after borg check --verify-data, because an attacker could corrupt data quietly. A cron job that prunes and compacts in normal mode makes append-only protection useless.
Test restores
A backup you have never restored is a guess. Restore into a scratch folder, never over live files:
# restic: restore one folder and compare
restic restore latest --target /root/restore-test --include /etc/ssh
diff -r /etc/ssh /root/restore-test/etc/ssh
# restic: read back 5% of the stored data to check it
restic check --read-data-subset=5%
# Borg: paths are stored without the leading slash; extract writes to the current folder
mkdir -p /root/restore-test && cd /root/restore-test
borg extract --list ::web1-2026-11-02T02:15:01 etc/ssh
borg check
Every few months, restore a whole database dump into a throwaway virtual machine and start the application against it. Time it: that is how long your real recovery will take.
Automate and watch it
Put the dump and backup commands in one root-only script that stops on the first error (set -eu), and run it from cron or a systemd timer. Make sure failures reach a person: cron mail, a monitoring check on the age of the last snapshot, or both. Borg's own example script runs daily as root with mode 0700 and reports the exit code.
How to confirm it is fixed
- A backup ran last night and you can see it in
restic snapshotsorborg list. - From the server, deleting an old snapshot fails.
- One copy is off-site and one is offline or append-only.
- The backup password and Borg key are stored outside the server.
- You restored files and a database dump in the last three months.
When to get help
If ransomware has already hit, do not run the backup job: it may back up encrypted files or give the attacker your backup credentials. Follow Ransomware first. If you run many servers or hold regulated data, ask your hosting provider or a backup specialist to review retention and immutability. For damaged databases see Database won't start.
Sources
- Offline backups in an online world, UK NCSC
- #StopRansomware Guide, CISA
- Preparing a new repository, restic documentation
- Backing up, restic documentation
- Removing backup snapshots, restic documentation
- Working with repositories, restic documentation
- rest-server, restic project
- Quick start, BorgBackup documentation
- Additional notes: append-only mode, BorgBackup documentation
- borg key, BorgBackup documentation
- SQL Dump, PostgreSQL documentation
- mariadb-dump, MariaDB documentation