Roll back a bad update on Debian, Ubuntu, RHEL or SUSE
If an update broke a service, put the previous package version back, stop it from updating again for now, and plan to re-apply the update once the bug is fixed. A rollback trades a working service for a known security hole, so keep it short and tracked.
Do this first.
- Find out exactly which packages changed and when (step 1). Don't guess.
- Check whether a configuration change, not the package, is the cause. Package updates may install new config files beside yours (
.dpkg-dist,.rpmnew). - If the update was a security fix for something exposed to the internet, decide with whoever owns the risk before you roll it back, and add other protection meanwhile.
- If you have a snapshot from just before the update and the server holds no new data since, restoring it may be the fastest fix.
Symptoms
- A service fails to start or behaves differently right after
apt upgrade,dnf upgradeor an automatic update. - Errors mention a new library version, a removed option or a changed default.
- The failure started at the same time as an entry in the package logs.
Step 1: Find what changed
# Debian and Ubuntu
less /var/log/apt/history.log
grep -E ' (upgrade|install) ' /var/log/dpkg.log | tail -40
# RHEL, Rocky, AlmaLinux, Fedora
sudo dnf history list
sudo dnf history info 42
Match the times with the start of the failure in journalctl -u SERVICE. Then roll back only the packages involved, not the whole update.
Debian and Ubuntu
Install the previous version
See which versions apt knows about:
apt-cache policy PACKAGE
Install a specific one with =. Downgrade every binary package built from the same source together, or apt will report broken dependencies (for example a library and its tools):
sudo apt install PACKAGE=OLD-VERSION LIBPACKAGE=OLD-VERSION
apt shows the packages it will downgrade and asks for confirmation. Read the list; if it wants to remove other packages, stop and check why. The package scripts usually restart the service.
If the old version is no longer in any repository, look in apt's cache: ls /var/cache/apt/archives/PACKAGE_*, and install a file from there with sudo apt install ./PACKAGE_OLD-VERSION_amd64.deb. For Debian, snapshot.debian.org keeps every package version ever published. Download the exact .deb you need from there, or temporarily add a snapshot date as a source:
deb [check-valid-until=no] https://snapshot.debian.org/archive/debian/20261020T000000Z/ trixie main
Snapshot release files have expired dates, which is why check-valid-until=no is needed. That disables a safety check, so remove the line again as soon as you have the package.
Stop it from updating again
sudo apt-mark hold PACKAGE LIBPACKAGE
apt-mark showhold
A hold keeps the package at its version through upgrades, including automatic ones. For finer control, an apt pin (/etc/apt/preferences.d/) can keep a version range; a pin priority above 1000 even allows apt to downgrade. Holds are easier to see and remove.
RHEL, Rocky, AlmaLinux, Fedora
Undo the transaction
sudo dnf history undo 42
This reverses everything transaction 42 did. To go back further, sudo dnf history rollback 41 undoes every transaction after 41. Both need the old versions in an enabled repository. For a single package:
sudo dnf downgrade PACKAGE
sudo dnf install PACKAGE-OLD-VERSION
downgrade picks the highest available version lower than the installed one; install with a full version string picks an exact one. On DNF5 (Fedora 41 and later), downgrade fails if a name doesn't match an installed package unless you add --skip-unavailable.
Don't downgrade kernel, glibc or SELinux packages. Red Hat doesn't support downgrading core system packages. For a bad kernel, boot and set the previous kernel instead: see kernel boot failure.
Lock the version
# DNF 4 (RHEL 8, 9, 10): install the plugin first
sudo dnf install python3-dnf-plugin-versionlock
sudo dnf versionlock add PACKAGE
sudo dnf versionlock list
DNF5 has versionlock built in, with the same add, list, delete and clear subcommands; it stores locks in /etc/dnf/versionlock.toml. Remove a lock with sudo dnf versionlock delete PACKAGE.
SUSE with snapper
On SUSE Linux Enterprise and openSUSE with a Btrfs root, every zypper transaction creates a pair of snapshots, one before and one after. List them and see what changed:
sudo snapper list -t pre-post
sudo snapper status PRE..POST
To revert only the files that transaction changed: sudo snapper -v undochange PRE..POST. SUSE advises reviewing the list of changed files first and restoring only files that belong to the change you want to undo.
For a full system rollback, reboot (reboot), choose the bootable snapshots entry in the boot menu and pick the snapshot from before the update. If the system works, make it permanent and reboot again (reboot):
sudo snapper rollback
Snapper first saves a snapshot of the current state, so you can go forward again. Directories such as /home, /opt, /srv, /usr/local and /var are excluded from snapshots, so data, logs and databases are not rolled back. For a single package, sudo zypper install --oldpackage PACKAGE=OLD-VERSION downgrades it and sudo zypper addlock PACKAGE locks it.
LVM, Btrfs and cloud snapshots
Snapshots let you undo a whole update at once, but only if you took one before the update. On LVM, with free space in the volume group (sudo vgs):
sudo lvcreate -s -L 10G -n root_preupdate vg0/root
Check it with sudo lvs; if the snapshot's space fills up, you lose it, so size it generously or extend it with lvextend. To roll back, merge it into the original (needs a reboot for a root volume, because the merge waits until the volume is next activated):
sudo lvconvert --merge vg0/root_preupdate
If you don't need it, remove it with sudo lvremove vg0/root_preupdate, because a snapshot keeps using space as the volume changes. Note that a separate /boot partition isn't part of the root snapshot, so kernels there won't match after a merge; check the boot entries before you reboot.
On Btrfs without snapper, sudo btrfs subvolume snapshot -r / /.snapshots/pre-update makes a read-only snapshot (the target directory must exist, and nested subvolumes aren't included). Cloud and virtualisation platforms offer disk snapshots too; take one before patching and restore the whole disk if needed.
A snapshot is not a backup. It lives on the same storage as the original. Restoring an old snapshot also throws away data written since, such as orders, mail or database rows. See restore from backup.
Confirm and re-apply fixes later
- The service works and the expected versions show in
apt-cache policy PACKAGEorrpm -q PACKAGE. - The hold or versionlock is in place and recorded in your ticket or change log with a reason and a review date.
- You know which security fixes you rolled back. Check the vendor's advisory and the vulnerability notes on this site, and add a workaround (firewall rule, disabled feature) if the issue is exposed.
- When a fixed package is released, remove the hold (
sudo apt-mark unhold PACKAGE,sudo dnf versionlock delete PACKAGEorsudo zypper removelock PACKAGE) and update, ideally on a test server first. See patch fast.
Prevent it next time
- Patch a staging server or one node first, then the rest.
- Take a snapshot before every maintenance window and note the package transaction ID.
- Keep configuration in version control so you can tell config changes from package changes.
- Read the release notes for major-version updates of databases, PHP and web servers before applying them.
When to get help
Contact your vendor's support (if you have a subscription), the software's maintainers or an experienced administrator if the old version can't be found, if a rollback breaks dependencies across many packages, or if data formats were upgraded (databases often can't be read by an older version after an upgrade). If nothing short of a rebuild works, see restore from backup.
Sources
- apt-get(8), Debian manpages
- apt-mark(8), Debian manpages
- apt_preferences(5), Debian manpages
- snapshot.debian.org, Debian
- DNF command reference, DNF documentation
- DNF versionlock plugin, dnf-plugins-core documentation
- Versionlock command, DNF5 documentation
- Changes in DNF5 compared to DNF 4, DNF5 documentation
- Managing software with the DNF tool, RHEL 10, Red Hat
- System recovery and snapshot management with Snapper, SUSE documentation
- zypper(8), openSUSE manpages
- lvcreate(8), man7.org
- lvconvert(8), man7.org
- btrfs-subvolume(8), BTRFS documentation