Patch many servers at once: Ansible and fleet tools
With more than a handful of servers, patch with a tool that runs in parallel, works in small batches, stops when things go wrong and tells you exactly which hosts failed. Ansible does this over plain SSH; fleet managers such as Landscape, Satellite and Uyuni add dashboards and scheduling on top.
When SSH loops are not enough
A shell loop such as for h in $(cat hosts); do ssh $h sudo apt upgrade -y; done works for five servers. It breaks down as the fleet grows:
- It runs one host at a time, so 200 servers take hours.
- It does not stop when updates start failing, so one bad package can break every host in turn.
- It has no idea of Debian versus RHEL versus SUSE, and no record of which hosts succeeded.
- It cannot decide per host whether a reboot is needed, then wait for the host to come back.
Ansible solves these problems without an agent. You install it on one control machine (for example with pipx install --include-deps ansible). The managed servers need only Python and an SSH account that can use sudo.
Write an inventory
The inventory lists your servers in groups. Group by operating system family, by role and by risk, so you can target "internet-facing Ubuntu web servers" in one word.
# inventory.ini
[ubuntu]
web1.example.com
web2.example.com
[rhel]
db1.example.com
app1.example.com
[suse]
erp1.example.com
[canary]
web1.example.com
[internet_facing]
web1.example.com
web2.example.com
[prod:children]
ubuntu
rhel
suse
Check that Ansible can reach every host before you need it in an emergency:
ansible-inventory -i inventory.ini --graph
ansible all -i inventory.ini -m ansible.builtin.ping -o
Any host that fails the ping will also fail to patch. Fix SSH keys, sudo rights and missing Python now, not during an incident.
Ad-hoc commands for one package
When one package has an urgent fix, an ad-hoc command updates it everywhere in one line. The format is ansible PATTERN -m MODULE -a "OPTIONS"; -b runs the module with sudo, -K asks for the sudo password, and -f sets how many hosts run in parallel (the default is 5).
# 1. which version does each host have? (read-only)
ansible ubuntu -i inventory.ini -m ansible.builtin.command -a 'dpkg-query -W openssh-server' -o
ansible rhel:suse -i inventory.ini -m ansible.builtin.command -a 'rpm -q openssh-server' -o
# 2. update only that package (may restart the service)
ansible ubuntu -i inventory.ini -b -f 20 -m ansible.builtin.apt -a 'name=openssh-server state=latest update_cache=yes'
ansible rhel -i inventory.ini -b -f 20 -m ansible.builtin.dnf -a 'name=openssh-server state=latest update_cache=yes'
ansible suse -i inventory.ini -b -f 20 -m community.general.zypper -a 'name=openssh-server state=latest update_cache=yes'
# 3. RHEL family: which services still run old code?
ansible rhel -i inventory.ini -b -m ansible.builtin.command -a 'dnf needs-restarting -s'
state=latest installs the newest version from the configured repositories; state=present would leave an old version in place. Add --limit canary to try one host first, and -C (check mode) to preview changes. After the update, run step 1 again and compare the versions with the vendor's advisory. For what still needs a restart on each OS, see restarts and reboots.
A rolling update playbook
For routine patching, and for any fix that needs a reboot, use a playbook. This one patches one host first, then five, then a quarter of the fleet at a time. Each batch is checked before the next one starts.
# patch.yml - rolling security update for Debian/Ubuntu, RHEL family and SUSE
# WARNING: installs updates, may restart services, and REBOOTS hosts that need it
- name: Rolling security update
hosts: all
become: true
serial:
- 1
- 5
- "25%"
max_fail_percentage: 10
vars:
min_free_mb: 1024
critical_services: [] # for example: ['nginx', 'postgresql']
pre_tasks:
- name: Measure free space in /var (MB)
ansible.builtin.shell: df -Pm /var | awk 'NR==2 {print $4}'
register: var_free
changed_when: false
check_mode: false
- name: Stop this host if /var is nearly full
ansible.builtin.assert:
that: var_free.stdout | int >= min_free_mb
fail_msg: "Only {{ var_free.stdout }} MB free in /var"
tasks:
# Debian and Ubuntu: unattended-upgrade installs only the origins listed in
# /etc/apt/apt.conf.d/50unattended-upgrades (security by default)
- name: Refresh apt package lists
ansible.builtin.apt:
update_cache: true
when: ansible_facts['os_family'] == 'Debian'
- name: Make sure unattended-upgrades is installed
ansible.builtin.apt:
name: unattended-upgrades
state: present
when: ansible_facts['os_family'] == 'Debian'
- name: Install security updates (Debian, Ubuntu)
ansible.builtin.command: unattended-upgrade -v
when: ansible_facts['os_family'] == 'Debian'
# RHEL, AlmaLinux, Rocky, Oracle Linux, Fedora
- name: Install security updates (RHEL family)
ansible.builtin.dnf:
name: '*'
state: latest
security: true
update_only: true
update_cache: true
when: ansible_facts['os_family'] == 'RedHat'
# SLES and openSUSE; exit code 103 = zypper updated itself, run again
- name: Install security patches (SUSE)
ansible.builtin.command: zypper --non-interactive patch --category security
register: zypper_patch
until: zypper_patch.rc != 103
retries: 3
delay: 5
failed_when: zypper_patch.rc not in [0, 102, 103]
when: ansible_facts['os_family'] == 'Suse'
# Is a reboot required?
- name: Check reboot flag (Debian, Ubuntu)
ansible.builtin.stat:
path: /var/run/reboot-required
register: deb_reboot
when: ansible_facts['os_family'] == 'Debian'
- name: Check reboot need (RHEL family), exit 1 = reboot
ansible.builtin.command: dnf needs-restarting -r
register: rh_reboot
changed_when: false
check_mode: false
failed_when: rh_reboot.rc not in [0, 1]
when: ansible_facts['os_family'] == 'RedHat'
- name: Check reboot need (SUSE), exit 102 = reboot
ansible.builtin.command: zypper needs-rebooting
register: suse_reboot
changed_when: false
check_mode: false
failed_when: suse_reboot.rc not in [0, 102]
when: ansible_facts['os_family'] == 'Suse'
- name: Reboot only if required (REBOOTS THE HOST)
ansible.builtin.reboot:
msg: "Reboot for security updates"
reboot_timeout: 900
when: >-
(deb_reboot.stat is defined and deb_reboot.stat.exists) or
(rh_reboot.rc is defined and rh_reboot.rc == 1) or
(suse_reboot.rc is defined and suse_reboot.rc == 102)
post_tasks:
- name: Critical services are active
ansible.builtin.command: systemctl is-active {{ item }}
loop: "{{ critical_services }}"
changed_when: false
check_mode: false
- name: Health check URL answers 200
ansible.builtin.uri:
url: "{{ health_url }}"
status_code: 200
when: health_url is defined
- name: Record the running kernel
ansible.builtin.command: uname -r
register: running_kernel
changed_when: false
check_mode: false
- name: Show the running kernel
ansible.builtin.debug:
msg: "{{ inventory_hostname }} runs kernel {{ running_kernel.stdout }}"
Run it in check mode first, then for real:
ansible-playbook -i inventory.ini patch.yml --check
ansible-playbook -i inventory.ini patch.yml -e '{"critical_services": ["nginx"]}' --limit ubuntu
How the safety settings work
- serial runs the whole play on one batch before starting the next. With 40 hosts,
[1, 5, "25%"]gives batches of 1, 5, 10, 10, 10 and 4. - max_fail_percentage applies to each batch and must be exceeded, not just reached. With 10, one failure in the first batch (100%) or in the batch of 5 (20%) stops the play. In a batch of 10, one failure (10%) does not stop it, but two (20%) do. Use 0 to stop on any failure.
- Reboot only if required. The playbook reads the same signals you would check by hand:
/var/run/reboot-requiredon Debian and Ubuntu (created by packages such as the kernel, and on Debian by a hook in unattended-upgrades),dnf needs-restarting -ron the RHEL family, andzypper needs-rebootingon SUSE.ansible.builtin.rebootwaits for the host to come back and answer before the play continues; its default timeout is 600 seconds. - Pre- and post-checks stop a host before it fills its disk, and fail the batch if a key service or health URL is down after patching. Take web servers out of the load balancer in a pre-task if your setup allows it.
Security-only filters depend on the repositories. security: true in the dnf module installs only updates marked as security fixes, so it works only where the repositories publish advisory metadata. If dnf updateinfo list --security shows nothing while updates are pending, drop security: true and apply all updates. On Ubuntu 24.04 and later, needrestart restarts affected services automatically after apt runs.
A host that patched but did not reboot may still run old libraries in long-running services. Check them as described in restarts and reboots, and see kernel updates before you reboot a fleet onto a new kernel.
Emergency workflow for a critical CVE
- Find the fixed version. Read each vendor's advisory and note the fixed package version per release. Where to find advisories.
- Find every affected host. Run the read-only version query from the ad-hoc section across
alland save the output. Hosts that are unreachable are not safe; list them separately. Compare versions with an advisory. - Patch the canary. Use the ad-hoc update or the playbook with
--limit canary, then check the service works. - Patch the most exposed hosts next. Run with
--limit internet_facing, then the rest. For a fix that needs a reboot, use the playbook so reboots happen in batches. - Restart what uses the fixed code and reboot where the kernel or a core library changed.
- Prove it. Run the version query again. The
PLAY RECAPat the end lists failed and unreachable hosts; work through each one by hand. - Protect what you cannot patch: restrict access with the firewall, and plan to replace end-of-life systems.
The full first-hours plan is in the emergency patching playbook.
Fleet management tools
If you want a web console, scheduled patch windows, local package mirrors and compliance reports, a fleet manager may suit you better than playbooks alone. These are the main options; each is a large system to run, so try one on a test group first.
- Landscape (Canonical)
- Manages and monitors Ubuntu machines through an agent (Landscape Client) that reports to a central server, with a web portal and API. It can automate updates and apply security patches. Available as SaaS or self-hosted; support is bought with Ubuntu Pro.
- Foreman with Katello
- Open-source projects. Foreman manages the lifecycle of physical and virtual hosts; the Katello plugin adds local yum and deb repositories, "content views" that move tested packages from development to production, and applying package and errata (advisory) updates to one host or a group.
- Red Hat Satellite
- Red Hat's subscription product for RHEL fleets, built from Foreman, Katello, Pulp and Candlepin. It curates repositories and patches per environment and pushes updates on schedules from a central point.
- SUSE Multi-Linux Manager
- SUSE's commercial manager, known as SUSE Manager in earlier versions; version 5.2 became available in July 2026. Based on Uyuni, it patches SLES and openSUSE, the RHEL family (8 to 10), Ubuntu LTS and Debian 13 from one console. Check the release notes for exact client versions.
- Uyuni
- The open-source upstream of SUSE Multi-Linux Manager, part of the openSUSE Project. It uses Salt to manage clients; the Uyuni server acts as the Salt master.
Whatever you choose, the rules stay the same: patch a small group first, patch in batches, reboot only when required, and verify the fixed version on every host.
Sources
- Installing Ansible, Ansible documentation
- How to build your inventory, Ansible documentation
- Introduction to ad hoc commands, Ansible documentation
- ansible command-line reference, Ansible documentation
- ansible-inventory command-line reference, Ansible documentation
- ansible.builtin.apt module, Ansible documentation
- ansible.builtin.dnf module, Ansible documentation
- community.general.zypper module, Ansible documentation
- ansible.builtin.reboot module, Ansible documentation
- Controlling playbook execution: strategies and more, Ansible documentation
- Error handling in playbooks, Ansible documentation
- Conditionals, Ansible documentation
- Automatic updates, Ubuntu Server documentation
- UnattendedUpgrades, Debian Wiki
- zypper(8) manual page, openSUSE
- Landscape documentation, Canonical
- Katello, The Foreman project
- Satellite overview, concepts, and deployment considerations, Red Hat
- Red Hat Satellite, Red Hat
- SUSE Multi-Linux Manager 5.2 release notes, SUSE
- Uyuni project