Server won't boot after a kernel update: how to recover
When a server stops booting right after a kernel update, the old kernel is almost always still installed. Boot it from the GRUB menu, make it the default for now, then fix or remove the new kernel. Most failures come from an initramfs that wasn't built (often because /boot was full) or a driver module missing from the new kernel.
Do this first.
- Get console access: your provider's web, serial or VNC console, or IPMI/iDRAC/iLO for physical servers. SSH won't help until the system boots.
- Reboot and pick the previous kernel in the GRUB menu (step 1). Don't reinstall anything yet.
- Don't remove the old, working kernel, and don't run
grub-installor edit partitions unless you are sure GRUB itself is broken. - If the server holds important data and nothing boots, take a disk snapshot through your provider before experimenting.
Symptoms
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0). This usually means the initramfs is missing or broken.- The boot stops at a dracut or initramfs emergency shell, often saying the root device can't be found.
- The system boots, but the network, RAID controller or storage is missing, because a driver didn't load.
- Black screen or reboot loop after GRUB, or GRUB shows only a
grub>prompt.
Step 1: Boot the previous kernel
Reboot (reboot) and watch the console. If the GRUB menu doesn't appear, press Esc or hold Shift as the machine starts (which key works depends on the system); some distributions hide the menu and use a short timeout.
- Debian and Ubuntu: choose “Advanced options for …”, then the entry for the older kernel version (not the “recovery mode” one, unless the normal one fails).
- RHEL, Rocky, AlmaLinux, Fedora: the menu lists each kernel directly. Choose the second entry, which is normally the previous kernel. The
0-rescue-…entry is a generic rescue kernel and initramfs to use if no normal kernel boots.
Once logged in, confirm which kernel runs and read the errors from the failed boot:
uname -r
journalctl --list-boots
journalctl -b -1 -p err
-b -1 means the boot before the current one; it only works if the journal is stored on disk (/var/log/journal exists). If the failure happened before the root filesystem was mounted, the log may be empty and the console message is your best clue.
Step 2: Keep booting the good kernel
RHEL family: grubby
sudo grubby --info=ALL | grep -E '^(index|kernel|title)'
sudo grubby --set-default /boot/vmlinuz-WORKING-VERSION
sudo grubby --default-kernel
For a one-time test of the new kernel later, sudo grub2-reboot INDEX boots that entry only on the next start. Red Hat warns that installing kernels can change index numbers, so check them right before you use it. The next kernel update will normally become the default again, which is what you want once a fixed kernel is out.
Debian and Ubuntu
The simplest durable fix is to remove the broken kernel package so the working one is the newest. Packaging runs update-grub for you. Double-check the version first: it must not be the running kernel.
uname -r
dpkg --list 'linux-image-*' | grep '^ii'
sudo apt purge linux-image-BROKEN-VERSION
On Ubuntu also purge the matching linux-modules-BROKEN-VERSION package if apt doesn't. apt will also offer to remove the kernel meta-package that depends on the broken version (linux-image-amd64 on Debian, linux-image-generic or similar on Ubuntu). That is fine for now, but without it you get no more kernel security updates. Put it back as soon as a fixed kernel is released:
sudo apt install linux-image-amd64
If you prefer to keep the new kernel installed, set GRUB_DEFAULT=saved in /etc/default/grub, run sudo update-grub, then sudo grub-set-default with the entry you want. The GRUB manual recommends using a menu entry's id rather than its title, because titles change.
Step 3: Fix the new kernel
Full /boot
Check df -h /boot. If it is at or near 100%, the new initramfs was probably cut off. Remove old kernels through the package manager, never with rm: sudo apt autoremove --purge on Debian and Ubuntu, or sudo dnf remove --oldinstallonly on the RHEL family (this keeps the running kernel). Then rebuild the initramfs.
Rebuild the initramfs
# Debian and Ubuntu
sudo update-initramfs -u -k NEW-VERSION
# RHEL, Rocky, AlmaLinux, Fedora
sudo dracut -f --kver NEW-VERSION
Use the exact version string from ls /boot (for example the part after vmlinuz-). update-initramfs -u -k all updates every installed kernel that already has an initramfs. dracut's -f overwrites the existing image.
Missing drivers
- Moved VM or new hardware. dracut builds a “host-only” image by default with just the drivers this machine used. If the VM moved to another hypervisor or storage type, the new kernel's initramfs may lack the disk driver. Build a generic one:
sudo dracut -f --no-hostonly --kver NEW-VERSION, or add one module with--add-drivers. - Third-party kernel modules (DKMS packages such as ZFS, GPU or out-of-tree network drivers) must be rebuilt for each new kernel. Check
dkms status; if a build failed, update that package or wait for a version that supports the new kernel. - Secure Boot refuses to load modules that aren't signed with a trusted key. A module that built fine may still not load; check
journalctl -k -b -1for signature errors.
Then test the fixed kernel once: reboot into it (reboot) and fall back to the old one if it still fails.
If no kernel boots
Try rescue mode from GRUB: highlight the entry, press e, add systemd.unit=rescue.target to the end of the line starting with linux, and press Ctrl+X. You need the root password. If that fails too, boot an installer or live image, or your provider's rescue system:
- RHEL family: boot the installation image and choose rescue mode. It finds your installation, mounts it and tells you the path to
chrootinto. - Debian and Ubuntu: mount the root filesystem (and
/boot,/boot/efi) under/mnt, bind-mount the system directories, and chroot in, as on the Debian wiki:
sudo mount /dev/ROOT-PARTITION /mnt
sudo mount /dev/BOOT-PARTITION /mnt/boot # only if /boot is separate
sudo mount /dev/EFI-PARTITION /mnt/boot/efi # UEFI systems
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
sudo chroot /mnt
Inside the chroot, free space in /boot, rebuild the initramfs and run update-grub as above. Only reinstall GRUB (grub-install) if GRUB itself is missing; the Debian wiki page in Sources covers UEFI systems.
Confirm it is fixed
uname -rshows the kernel you intended.systemctl --failedlists no failed units, and network and storage are present.- RHEL family:
sudo grubby --default-kernelshows the intended kernel. Debian and Ubuntu: the GRUB menu shows it first. - A second reboot comes back on its own without console help.
Prevent it next time
- Keep
/bootunder about 70% full and keep at least one older kernel installed. - Reboot after kernel updates in a maintenance window while someone can reach the console, not at 3 a.m. unattended.
- Patch a test server or one node first. See kernel updates and restart after updates.
- Make the journal persistent so the next failed boot leaves logs, and snapshot the disk before kernel updates.
When to get help
Call your hosting provider or an experienced administrator if you have no console access, if the disk or RAID isn't detected by any kernel, if filesystem checks report errors, or if GRUB or the EFI partition is damaged. Restoring from a backup may be faster: see roll back a bad update.
Sources
- Managing, monitoring, and updating the kernel, RHEL 10, Red Hat
- Kernel panic, Rocky Linux documentation
- update-initramfs(8), Debian manpages
- dracut(8), man7.org
- Simple configuration handling, GNU GRUB manual
- GrubEFIReinstall, Debian Wiki
- journalctl, systemd documentation
- kernel-command-line, systemd documentation
- DNF command reference, DNF documentation