Auto-Updates and Manual Rollbacks

Fedora CoreOS provides atomic updates and rollbacks via OSTree deployments.

By default, the OS performs continual auto-updates via two components:

  • rpm-ostree handles multiple on-disk OSTree deployments and can switch between them at boot-time.

  • Zincati continually checks for OS updates and applies them via rpm-ostree.

Wariness to updates

The local Zincati agent periodically checks with a remote service to see when updates are available. A custom "rollout wariness" value (see documentation) can be provided to let the server know how eager, or how risk-averse, the node is to receiving updates.

The rollout_wariness parameter can be set to a floating point value between 0.0 (most eager) and 1.0 (most conservative). In order to receive updates very early in the phased rollout cycle, a node can be configured with a low value (e.g. 0.001). This can be done during provisioning by using the FCCT configuration snippet shown below:

Example: configuring Zincati rollout wariness
variant: fcos
version: 1.0.0
storage:
  files:
    - path: /etc/zincati/config.d/51-rollout-wariness.toml
      contents:
        inline: |
          [identity]
          rollout_wariness = 0.001

OS update finalization

To finalize an OS update, the machine must reboot. As this is an invasive action which may cause service disruption, Zincati allows the cluster administrator to control when nodes are allowed to reboot for update finalization.

The following finalization strategies are available:

  • As soon as the update is downloaded and staged locally, immediately reboot to apply an update.

  • Use an external lock-manager to coordinate the reboot of a fleet of machines.

A specific finalization strategy can be configured on each node.

The FCCT snippet below shows how to use a custom lock-manager:

Example: configuring Zincati updates strategy
variant: fcos
version: 1.0.0
storage:
  files:
    - path: /etc/zincati/config.d/55-updates-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "fleet_lock"
          [updates.fleet_lock]
          base_url = "https://example.com/lock-manager/"

For further details on updates finalization, check the Zincati documentation.

Manual Rollbacks

When an update is complete, the previous OS deployment remains on disk. If an update causes issues, you can use it as a fallback. This is a manual operation that requires human intervention and console access.

Temporary rollback

To temporarily boot the previous OS deployment, hold down Shift during the OS boot process. When the bootloader menus appears, select the relevant OS entry in the menu.

Permanent rollback

To permanently revert to the previous OS deployment, log into the target node, and run rpm-ostree rollback -r. This operation marks the previous OS deployment as the default, and immediately reboots into it.