Emergency patching playbook: the first hours after a critical CVE
When a critical vulnerability lands, work in a fixed order: decide how urgent it is for you, find every affected host, patch the most exposed ones first, restart what uses the fixed code, and prove the fix is live. If no patch exists yet, reduce exposure until one does.
- Triage: is the vulnerable service reachable, is the flaw being exploited, how likely is exploitation?
- Inventory: which hosts have the affected package, and which version?
- Order: internet-facing and exploited first, then internal, then the rest.
- Snapshot or back up before you change anything.
- Apply the vendor’s fixed package.
- Restart the affected services, or reboot for a kernel fix.
- Verify the installed version and that the running process uses it.
- If there is no patch: mitigate (disable, filter, firewall).
- Tell people what you did and what is still open.
- Check for signs of compromise on hosts that were exposed.
1. Triage in ten minutes
Answer three questions before you touch any server:
- Is it exposed?
- A flaw in a network service that listens on the internet is far more urgent than the same flaw in a tool only administrators run locally. Check whether the affected component runs on your hosts and who can reach it.
- Is it already exploited?
- The US Cybersecurity and Infrastructure Security Agency (CISA) keeps the Known Exploited Vulnerabilities (KEV) catalog, a list of flaws that have been exploited in the wild. If your CVE is on it, treat it as top priority.
- How likely is exploitation?
- The Exploit Prediction Scoring System (EPSS) from FIRST estimates the probability, from 0 to 1, that a CVE will be exploited in the wild in the next 30 days. A score of 0.5 means a 50% estimated chance. Use it to rank issues that are not yet on KEV.
Both data sets are public and can be checked from a shell. Replace the CVE ID with yours:
# Is the CVE in CISA's KEV catalog? (prints a count; 0 = not listed)
curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json | grep -c 'CVE-2024-6387'
# EPSS score and percentile for one CVE
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2024-6387'
Then read your distribution’s page for the CVE. It tells you which releases are affected and which package version fixes it, which may differ from the upstream version in the news. For example, Red Hat states that RHEL 6, 7 and 8 were never affected by CVE-2024-6387 (“regreSSHion” in OpenSSH) because they ship an older OpenSSH. See Security advisories for where each vendor publishes this.
2. Find every affected host
Query the package on each host. With no inventory tool, a loop over SSH works:
# Debian/Ubuntu
for h in web1 web2 db1; do printf '%s: ' "$h"; ssh "$h" dpkg-query -W openssh-server; done
# RHEL, Rocky, AlmaLinux, Fedora, SUSE (RPM-based)
for h in web1 web2 db1; do printf '%s: ' "$h"; ssh "$h" rpm -q openssh-server; done
# Alpine
apk info -v | grep openssh
# FreeBSD (packages; base-system components are covered by FreeBSD advisories)
pkg info openssh-portable
Compare each result with the fixed version from the advisory. Check installed versions explains how, including why a “lower” version number can already contain the fix. Do not forget containers and images, which carry their own packages: see Containers.
3. Decide the patch order
- Internet-facing hosts running the vulnerable service, especially if the CVE is on KEV.
- Hosts reachable from those (jump hosts, internal APIs, databases).
- Everything else, in your normal update window.
Patch one representative host first, check it, then roll out to the rest. If you have 40 web servers, a 15-minute check on one of them is cheap insurance.
4. Snapshot first
Take a VM or storage snapshot, or make sure a recent backup exists, so you can roll back a bad update. On FreeBSD with a ZFS root, create a boot environment you can boot back into:
sudo bectl create before-cve-patch
5. Apply the fix
Refresh the package metadata first, or the tools will not see the new version. Upgrading only the affected package is fastest. A full security upgrade is the better choice if you can afford it.
# Debian/Ubuntu
sudo apt update
sudo apt-get install --only-upgrade openssh-server # just this package
sudo apt upgrade # or everything
# RHEL family (DNF 4)
sudo dnf upgrade --cve=CVE-2024-6387 # packages fixing this CVE
sudo dnf upgrade --advisory=RHSA-XXXX:NNNN # or by advisory ID
sudo dnf upgrade --security # or all security fixes
# SUSE / openSUSE
sudo zypper refresh
sudo zypper patch --cve=CVE-2024-6387 # patches for this CVE
sudo zypper patch --category security # or all security patches
# Alpine (as root)
apk update
apk upgrade openssh # only this package (+ dependencies)
# FreeBSD packages (as root)
pkg upgrade openssh-portable
pkg upgrade -v # all packages with known vulnerabilities
# FreeBSD base system (as root)
freebsd-update fetch
freebsd-update install
Filters need advisory data. dnf --cve, --security and zypper patch --cve only work when the repository publishes advisory metadata. If they find nothing but you know a fixed package exists, upgrade the package by name.
6. Restart what uses the fix
Running programs keep the old code in memory until they restart. Find what needs restarting:
# Debian/Ubuntu (needrestart package): list only, restart nothing
sudo needrestart -r l
# RHEL family: exit code 1 means a reboot is needed; -s lists services
sudo dnf needs-restarting -r
sudo dnf needs-restarting -s
# SUSE: services using deleted files, and whether a reboot is suggested
sudo zypper ps -s
sudo zypper needs-rebooting
# FreeBSD: installed kernel (-k) vs running kernel (-r) vs userland (-u)
freebsd-version -kru
Then restart the affected services. This interrupts the service:
sudo systemctl restart ssh # RESTARTS the service (Debian/Ubuntu name; sshd on RHEL/SUSE)
sudo rc-service sshd restart # Alpine (OpenRC): RESTARTS the service
sudo service sshd restart # FreeBSD: RESTARTS the service
Kernel fixes need a reboot (sudo reboot reboots the machine) or a live patch. See Restarts and reboots and Kernel live patching.
7. Verify
- The installed version matches or exceeds the fixed version in the advisory (
dpkg-query -W,rpm -q,apk info -v,pkg info). - The restart check from step 6 no longer lists the service.
- On FreeBSD,
freebsd-version -kand-rshow the same patch level after the reboot. - The service works: run your health checks, not just “the process is up”.
- A rescan with your vulnerability scanner agrees. See Vulnerability scanning.
8. If there is no patch yet
Reduce exposure until the vendor ships a fix, and remove the workaround afterward:
- Use the vendor’s documented workaround. For CVE-2024-6387, Red Hat documented setting
LoginGraceTime 0insshd_config. Thesshd_configmanual says 0 means “no time limit”, so read the vendor’s notes on side effects before you apply a workaround like this. - Disable the vulnerable feature. Turn off the affected web server module or option. For a vulnerable kernel module you do not need, stop it loading with a line such as
install modulename /bin/falsein a file under/etc/modprobe.d/. - Filter it. A web application firewall rule can block known exploit patterns for web-facing flaws. Treat it as a delay, not a fix, because attackers vary their payloads.
- Firewall it. Limit the port to the addresses that really need it, such as your VPN or admin network.
9. Communicate
Send a short status message to the people who own the affected services. Say what the issue is, which hosts are affected, what you did, what is still open, and when you will update them next. Record the change in your change log, including any workaround you must remove later.
10. Check for compromise
If a host was exposed while the flaw was being exploited, patching closes the door but does not remove an intruder who already got in. Review authentication and service logs from the exposure window. Look for new accounts, SSH keys, cron jobs and unexpected listening ports (sudo ss -tlnp). See Signs of compromise for a full checklist. If you find evidence, follow your incident response process rather than just patching and moving on.
Sources
- Known Exploited Vulnerabilities Catalog, CISA
- Exploit Prediction Scoring System (EPSS), FIRST
- CVE-2024-6387, Red Hat Customer Portal
- apt-get(8) manual page, Debian
- DNF command reference
- dnf needs-restarting, DNF plugins core documentation
- Installing security updates, RHEL 9 documentation, Red Hat
- zypper(8) manual page, openSUSE
- apk-upgrade(8) manual page, apk-tools
- pkg-upgrade(8), FreeBSD manual pages
- freebsd-update(8), FreeBSD manual pages
- freebsd-version(1), FreeBSD manual pages
- bectl(8), FreeBSD manual pages
- needrestart(1) manual page, Debian