Ansible Role: System Updates

A production-ready Ansible role for SPIE DataSolutions that automates system package updates across Debian and RedHat-based distributions.

The role is designed to be executed across an entire server stack. It handles regular servers and automatically applies database-aware update procedures when MariaDB is detected.

How It Works

The role follows a single update workflow for the complete server stack.

                    systemupdates role

                           │
                           ▼

                 ┌─────────────────────┐
                 │ Run system updates  │
                 └──────────┬──────────┘
                            │
             ┌──────────────┴──────────────┐
             ▼                             ▼
        regular server                  MariaDB
                                           │
                                           ▼
                                  Cluster detection
                                           │
                              ┌────────────┴────────────┐
                              ▼                         ▼
                         standalone                clustered
                              │                         │
                              ▼                         ▼
                     standard DB update          cluster-safe
                     procedure                   update procedure

Regular Servers

For regular servers, the role performs the configured system package updates using the appropriate package manager:

• Debian: APT
• RedHat-based systems: DNF

At the end of every successful run, the role checks whether the system requires a reboot, including a reboot that was already pending before this run.

The role does not automatically reboot systems.

Database Support

Oracle MySQL is deliberately unsupported by this role.

This role supports:

• MariaDB
• MariaDB Galera / WSREP

There is no Oracle MySQL compatibility layer, detection logic, upgrade procedure, or support path.

MariaDB Servers

When MariaDB is detected on a host, the role performs additional database-aware checks before handling the database update.

The role determines whether the database is running as:

• Standalone
• Clustered / Galera

Standalone Database

For a standalone MariaDB server, the role stops MariaDB, applies the complete operating-system upgrade, and starts the service again.

The role verifies database connectivity. On RPM-based systems it runs mariadb-upgrade only if the MariaDB server version changed. Debian packages run mariadb-upgrade automatically after a server upgrade, as described in the MariaDB upgrade documentation.

If the package update fails while MariaDB is stopped, the role attempts to start the service and validates local connectivity. For Galera, recovery also checks Primary, Synced, readiness, and the configured cluster size. The role then fails with both the original update error and the recovery outcome, and checks/reports any pending reboot. If recovery is unsuccessful, inspect MariaDB/system logs and cluster state and restore service/cluster health manually before retrying the update.

Recovery does not roll back package changes. An APT/DNF transaction may have partially completed before it failed; inspect package-manager status and logs before retrying. Recovery connection checks are bounded by systemupdates_database_recovery_retries and systemupdates_database_recovery_delay (30 attempts, 10 seconds apart by default).

Clustered Database

For a clustered MariaDB/Galera installation, the role uses a cluster-aware update procedure.

The procedure:

1. Stops the MariaDB service gracefully.
2. Ensures no lingering MariaDB server process prevents the update.
3. Applies the complete operating-system upgrade while preventing APT from restarting services.
4. Starts MariaDB again.
5. Verifies that MariaDB is accepting connections.
6. Waits for the Galera node to return to Synced, checks cluster size and readiness, and on RPM-based systems runs mariadb-upgrade if the server version changed.

MariaDB's mariadb-upgrade reference says to run the tool after starting the new server version but documents no Galera-specific option or per-node sequencing. This role runs it on the updated RPM node after that node is Synced; validate major-version Galera upgrades against the release-specific MariaDB guidance.

Clustered systems should always be updated one node at a time.

Reboot Detection

After updates, the role checks whether a reboot is required.

The role checks for reboot requirements on every successful run, including reboots pending from an earlier update. This includes situations such as:

• kernel updates;
• system libraries requiring a restart;
• package-manager reboot indicators;
• other conditions reported by the operating system.

On Debian, the role checks /var/run/reboot-required and compares the running kernel with the newest installed versioned linux-image package. On RedHat, it uses needs-restarting -r.

When a reboot is required, the role reports this explicitly.

The role does not perform an automatic reboot. Reboots should be handled during an appropriate maintenance window.

Quick Start

This role is maintained internally by SPIE DataSolutions and is installed directly from the SPIE Git repository.

Install the role from Git:

ansible-galaxy role install git+https://git.ds.spie-nl.com/datasolutions/ansible-role-systemupdates.git

Then include the role in your playbook:

---
- name: Update all systems
  hosts: all
  become: true
  gather_facts: true
  serial: 1
  any_errors_fatal: true

  roles:
    - role: ansible-role-systemupdates

The role automatically determines the appropriate update procedure for each host.

Regular application and infrastructure servers receive the normal system update procedure.

MariaDB hosts receive additional database detection and handling.

Supported Platforms

• Debian

  • Debian 13 (Trixie)

• Enterprise Linux

  • Rocky Linux 10

The Molecule scenarios run on Debian 13. Rocky Linux 10 is supported but is not covered by the MariaDB or Galera scenarios here; run the regular-host scenario on Rocky Linux 10 and validate database upgrades on a disposable EL test host before production use. Platform support is not a claim that every supported distribution/package-manager combination has been certified.

Requirements

• Ansible Core 2.19+
• Controller and managed-node Python versions supported by the selected Ansible Core version (Debian 13 uses Python 3.13)
• Docker (for local testing with Molecule)
• Python dependencies listed in requirements.txt

Install the pinned and locally validated toolchain with:

python -m pip install -r requirements.txt

Role Variables

Default values are defined in defaults/main.yml.

APT Configuration

systemupdates_upgrade_type: "full"       # safe, full, or dist
systemupdates_cache_valid_time: 3600     # Cache validity in seconds
systemupdates_autoclean: true            # Auto-clean obsolete packages
systemupdates_only_upgrade: false        # Only upgrade existing packages

DNF Configuration

systemupdates_security_only: false       # Only security updates
systemupdates_bugfix: false              # Only bugfix advisories

These RedHat update filters are mutually exclusive; setting both to true fails validation.

General Configuration

systemupdates_autoremove: true
systemupdates_enforce_serial_execution: true
systemupdates_is_database: false
systemupdates_skip_database: false

systemupdates_enforce_serial_execution protects MariaDB hosts by rejecting batches containing more than one database host. The role requires Ansible's default linear strategy so all hosts finish database detection before the serial guard counts them; strategy: free is not supported. Regular hosts may share a batch with one database host. For production Galera rollouts, serial: 1 and any_errors_fatal: true are required playbook settings: serial: 1 updates one cluster member at a time, and any_errors_fatal: true prevents Ansible from proceeding to later batches after a failure. The role cannot enforce these play-level settings itself.

MariaDB is detected automatically. Set systemupdates_is_database: true only when service auto-detection is unavailable; the host must still have a running MariaDB service. Oracle MySQL is not supported.

For a Galera update, set systemupdates_galera_expected_cluster_size to the live wsrep_cluster_size, including any garbd arbitrator. The role intentionally has no default for this safety check. Synchronization waits up to systemupdates_galera_sync_retries attempts, at 10 seconds per attempt.

Set systemupdates_skip_database: true only when a detected MariaDB service is intentionally stopped and the host should receive a regular OS update instead. On Debian, the role passes policy_rc_d: 101 to APT upgrades and autoremove so package scripts do not start MariaDB. DNF has no equivalent service-start policy, so use this override only when that behavior is acceptable on RedHat systems. Do not set it together with systemupdates_is_database.

MariaDB checks use the local mariadb client as root via Unix socket authentication. The managed host must allow this local root access. Galera rolling updates are not supported in Ansible --check mode; standalone and regular-host check mode remains available.

For detailed variable documentation, see:

• defaults/main.yml - Default configuration values
• vars/main.yml - Internal role variables
• meta/argument_specs.yml - Validation and supported values

Dependencies

None.

The role is self-contained and handles the required package-manager operations.

Special Features

• Stack-wide updates - Designed to update an entire server stack using a single role
• Multi-platform - Supports Debian and RedHat-based systems
• Database awareness - Automatically detects MariaDB installations
• Database topology detection - Distinguishes standalone MariaDB from Galera/WSREP clusters
• Cluster-aware updates - Uses a dedicated procedure for clustered MariaDB systems
• Cluster-safe execution - Supports serial execution to prevent simultaneous updates
• Reboot detection - Detects when a system reboot is required after updates
• No automatic reboot - Reports required reboots without restarting systems automatically
• Idempotent - Safe to run repeatedly

Example Playbook

Update an Entire Stack

The role can be applied to all systems in a stack.

---
- name: Update all systems
  hosts: all
  become: true
  gather_facts: true
  serial: 1
  any_errors_fatal: true

  roles:
    - role: ansible-role-systemupdates

The role determines the appropriate update procedure for each host.

Regular application and infrastructure servers receive the normal system update procedure.

MariaDB hosts receive additional database detection and handling.

Security-Focused Updates

---
- name: Apply security-focused updates
  hosts: servers
  become: true
  gather_facts: true
  serial: 1
  any_errors_fatal: true

  roles:
    - role: ansible-role-systemupdates
      vars:
        systemupdates_security_only: true

systemupdates_security_only applies to RedHat-based systems using DNF.

For Debian-based systems, use the appropriate APT upgrade configuration instead.

The exact effect of the configured variables depends on the target operating system and package manager.

Important Notes

⚠️ Cluster environments

For production Galera rollouts, use both required play settings:

serial: 1
any_errors_fatal: true

This updates one cluster member at a time and stops the play after a failure, preventing later batches from proceeding with a potentially unhealthy cluster.

The role checks that each batch contains at most one MariaDB host. Regular hosts may share a batch with one database host. Use Ansible's default linear strategy; strategy: free is unsupported because the guard needs database detection to complete on all hosts before it counts the batch. The role cannot verify whether any_errors_fatal is enabled, so operators must include it in production Galera plays. The serial check can be disabled only by explicitly setting:

systemupdates_enforce_serial_execution: false

⚠️ Database hosts

Hosts running MariaDB are handled using database-aware update procedures.

The role automatically determines whether MariaDB is running standalone or as part of a Galera/WSREP cluster and applies the appropriate procedure.

⚠️ Reboots

The role detects when a system reboot is required after updates and reports this to the operator.

The role does not perform automatic reboots. Required reboots can then be planned and performed during an appropriate maintenance window.

Testing

This role uses Molecule for testing.

Current Molecule scenarios include:

• default: regular-host update workflow on Debian 13 by default; to select another distribution, set both MOLECULE_DISTRO and MOLECULE_IMAGE.
• mariadb: standalone MariaDB update workflow on Debian 13.
• galera: a single-node bootstrapped Galera cluster on Debian 13. It verifies a forced package upgrade, an autoremove-only plan, and a later deliberately failed package transaction while checking that MariaDB is stopped during package changes and that update errors and recovery outcomes are reported.
• mariadb-rocky: standalone MariaDB update workflow on Rocky Linux 10.
• galera-rocky: single-node Galera on Rocky Linux 10; a locally built RPM upgrade verifies the DNF database stop/update/start path and the final cluster health.

The scenario also checks recovery reporting for a MariaDB startup failure, a connection timeout, and a Galera recovery health-check failure. It tests wrong cluster-size and stopped-service rejection, but cannot validate multi-node rejoin/IST/SST behavior, non-Primary refusal, an unreachable cluster peer, loss of the Ansible connection during a transaction, or failure on a later node after an earlier node succeeded. Exercise those cases on a disposable multi-node cluster before production use.

The test scenario also runs a second convergence without package changes, exercising the no-update package-plan path. The regular MariaDB scenario likewise covers repeat-run idempotence. Package-plan differences across APT/DNF repositories and supported Enterprise Linux platforms still require integration coverage on those target platforms.

Running Tests Locally

Prerequisites

Install the Python dependencies:

python -m pip install --upgrade -r requirements.txt

Ensure Docker is running:

sudo systemctl start docker

Running Molecule Tests

Run all Debian scenarios:

molecule test -s default
molecule test -s mariadb
molecule test -s galera

Run the regular-host scenario on Rocky Linux 10:

MOLECULE_DISTRO=rockylinux10 \
MOLECULE_IMAGE=geerlingguy/docker-rockylinux10-ansible@\
sha256:65b18f2ee916a6d0763ff2b17fbe029376008e5d43cc475ee3fc0b6e5b69c467 \
molecule test -s default

Run without destroying containers, for debugging:

molecule converge -s mariadb
molecule verify -s mariadb
molecule converge -s galera
molecule verify -s galera

Linting

Forgejo Actions, GitHub Actions and GitLab CI run YAML linting, production-profile Ansible lint, Molecule syntax checks, Debian scenarios, Rocky Linux 10 regular-host and MariaDB/Galera scenarios, and a packaging-archive check. Forgejo runners need a docker label and Docker access; GitLab runners need privileged Docker-in-Docker support.

Run lint checks locally:

ansible-lint
yamllint .

The CI pipeline should run the Molecule test suite and linting checks for the supported distributions.

Troubleshooting

Molecule Container Won't Start

Ensure Docker is running:

sudo systemctl start docker

Check Docker permissions:

sudo usermod -aG docker $USER

After changing group membership, start a new login session or use:

newgrp docker

Recreate the Molecule containers:

molecule destroy
molecule test

Ansible Connection Errors

• Verify the Ansible connection configuration.
• Check firewall rules.
• Ensure ansible_user is configured correctly.
• Verify that privilege escalation is available when required.

For more help, open an issue or contact:

support.ds.nl@spie.com

License

MIT - See LICENSE for details.

Copyright (c) 2026 SPIE DataSolutions B.V.

Changelog

See CHANGELOG.md for a list of all notable changes to this project.

Security

Please see our Security Policy for reporting security vulnerabilities.

Contributing

Please read our Contributing Guide and Code of Conduct before submitting a Pull Request.

Support

For questions or support, contact:

support.ds.nl@spie.com