Auto-Updates and Manual Rollbacks
Fedora CoreOS provides atomic updates and rollbacks via OSTree deployments.
By default, the OS performs continual auto-updates via two components:
The local Zincati agent periodically checks with a remote service to see when updates are available. A custom "rollout wariness" value (see documentation) can be provided to let the server know how eager, or how risk-averse, the node is to receiving updates.
rollout_wariness parameter can be set to a floating point value between
0.0 (most eager) and
1.0 (most conservative). In order to receive updates very early in the phased rollout cycle, a node can be configured with a low value (e.g.
0.001). This can be done during provisioning by using the Butane config snippet shown below:
variant: fcos version: 1.4.0 storage: files: - path: /etc/zincati/config.d/51-rollout-wariness.toml contents: inline: | [identity] rollout_wariness = 0.001
To finalize an OS update, the machine must reboot. As this is an invasive action which may cause service disruption, Zincati allows the cluster administrator to control when nodes are allowed to reboot for update finalization.
The following finalization strategies are available:
As soon as the update is downloaded and staged locally, immediately reboot to apply an update.
Use an external lock-manager to coordinate the reboot of a fleet of machines.
Allow reboots only within configured maintenance windows, defined on a weekly UTC schedule.
A specific finalization strategy can be configured on each node.
The Butane snippet below shows how to define two maintenance windows during weekend days, starting at 22:30 UTC and lasting one hour each:
variant: fcos version: 1.4.0 storage: files: - path: /etc/zincati/config.d/55-updates-strategy.toml contents: inline: | [updates] strategy = "periodic" [[updates.periodic.window]] days = [ "Sat", "Sun" ] start_time = "22:30" length_minutes = 60
For further details on updates finalization, check the Zincati documentation.
When an update is complete, the previous OS deployment remains on disk. If an update causes issues, you can use it as a fallback. This is a manual operation that requires human intervention and console access.
To temporarily boot the previous OS deployment, hold down
Shift during the OS boot process. When the bootloader menu appears, select the relevant OS entry in the menu.
To permanently revert to the previous OS deployment, log into the target node and run the following commands:
# Stop the service that performs automatic updates sudo systemctl stop zincati.service # Mark the previous OS deployment as the default, and immediately reboots into it sudo rpm-ostree rollback -r
Please note that Zincati will keep looking for updates and upgrade to any new available OS deployment, other than the one you just reverted.
If you prefer, you can temporarily turn off auto-updates. Later on, you can re-enable them in order to let the machine catch up with the usual flow of updates:
# Disable Zincati in order to opt-out from future auto-updates sudo systemctl disable --now zincati.service [...] # At a later point, re-enable it to re-align with the tip of stream sudo systemctl enable --now zincati.service