Auto-Updates and Manual Rollbacks

Fedora CoreOS provides atomic updates and rollbacks via OSTree deployments.

By default, the OS performs continual auto-updates via two components:

  • rpm-ostree handles multiple on-disk OSTree deployments and can switch between them at boot-time.

  • Zincati continually checks for OS updates and applies them via rpm-ostree.

Wariness to updates

The local Zincati agent periodically checks with a remote service to see when updates are available. A custom "rollout wariness" value (see documentation) can be provided to let the server know how eager, or how risk-averse, the node is to receiving updates.

The rollout_wariness parameter can be set to a floating point value between 0.0 (most eager) and 1.0 (most conservative). In order to receive updates very early in the phased rollout cycle, a node can be configured with a low value (e.g. 0.001). This can be done during provisioning by using the Butane config snippet shown below:

Example: configuring Zincati rollout wariness
variant: fcos
version: 1.5.0
storage:
  files:
    - path: /etc/zincati/config.d/51-rollout-wariness.toml
      contents:
        inline: |
          [identity]
          rollout_wariness = 0.001

OS update finalization

To finalize an OS update, the machine must reboot. As this is an invasive action which may cause service disruption, Zincati allows the cluster administrator to control when nodes are allowed to reboot for update finalization.

The following finalization strategies are available:

  • As soon as the update is downloaded and staged locally, immediately reboot to apply an update.

  • Use an external lock-manager to coordinate the reboot of a fleet of machines.

  • Allow reboots only within configured maintenance windows, defined on a weekly UTC schedule.

A specific finalization strategy can be configured on each node.

The Butane snippet below shows how to define two maintenance windows during weekend days, starting at 22:30 UTC and lasting one hour each:

Example: configuring Zincati updates strategy
variant: fcos
version: 1.5.0
storage:
  files:
    - path: /etc/zincati/config.d/55-updates-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "periodic"
          [[updates.periodic.window]]
          days = [ "Sat", "Sun" ]
          start_time = "22:30"
          length_minutes = 60

For further details on updates finalization, check the Zincati documentation.

Manual Rollbacks

When an update is complete, the previous OS deployment remains on disk. If an update causes issues, you can use it as a fallback. This is a manual operation that requires human intervention and console access.

Temporary rollback

To temporarily boot the previous OS deployment, hold down Shift during the OS boot process. When the bootloader menu appears, select the relevant OS entry in the menu.

Permanent rollback

To permanently revert to the previous OS deployment, log into the target node and run the following commands:

# Stop the service that performs automatic updates
sudo systemctl stop zincati.service

# Mark the previous OS deployment as the default, and immediately reboots into it
sudo rpm-ostree rollback -r

Please note that Zincati will keep looking for updates and upgrade to any new available OS deployment, other than the one you just reverted.

If you prefer, you can temporarily turn off auto-updates. Later on, you can re-enable them in order to let the machine catch up with the usual flow of updates:

# Disable Zincati in order to opt-out from future auto-updates
sudo systemctl disable --now zincati.service

[...]

# At a later point, re-enable it to re-align with the tip of stream
sudo systemctl enable --now zincati.service

Want to help? Learn how to contribute to Fedora Docs ›