Testing Fedora CoreOS updates
Make sure that you have completed the steps described in the initial setup page before starting this tutorial. |
In this tutorial, we will not focus on provisioning but on what happens during updates and the options that are available in case of failures.
Downloading an older Fedora CoreOS release
One of the defining features of Fedora CoreOS is automatic updates. To see them in action, we have to download an older Fedora CoreOS release. Start by picking an older release from the Fedora CoreOS release page:
# replace "XX.XXXXXXXX.X.X" with one of older release IDs from the release page above (e.g. "34.20210919.3.0")
RELEASE="XX.XXXXXXXX.X.X"
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig
Once the archive has been downloaded, make sure to verify its integrity:
gpg --verify fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig
Look for "Good signature from" in the output. |
Once you have verified the archive, you can extract it with:
unxz fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz
To make the tutorial simpler, you should rename this image to a shorter name:
mv fedora-coreos-$RELEASE-qemu.x86_64.qcow2 fedora-coreos-older.qcow2
Writing the Butane config and converting to Ignition
We will create a Butane config that:
-
Sets up console autologin.
-
Raise kernel console logging level to hide audit messages from the console.
-
Add an SSH Key for the
core
user.
Let’s write this Butane config to a file called updates.bu
:
variant: fcos
version: 1.4.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAA...
systemd:
units:
- name: serial-getty@ttyS0.service
dropins:
- name: autologin-core.conf
contents: |
[Service]
# Override Execstart in main unit
ExecStart=
# Add new Execstart with `-` prefix to ignore failure
ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
TTYVTDisallocate=no
storage:
files:
- path: /etc/profile.d/systemd-pager.sh
mode: 0644
contents:
inline: |
# Tell systemd to not use a pager when printing information
export SYSTEMD_PAGER=cat
- path: /etc/sysctl.d/20-silence-audit.conf
mode: 0644
contents:
inline: |
# Raise console message logging level from DEBUG (7) to WARNING (4)
# to hide audit messages from the interactive console
kernel.printk=4
Optionally you can replace the SSH pubkey in the yaml file with your own public key so you can log in to the booted instance. If you choose not to do this you’ll still be auto logged in to the serial console. |
Run Butane to convert that to an Ignition config:
butane --pretty --strict updates.bu --output updates.ign
Startup and initial update
Now let’s provision it. Make sure that you are starting from the older Fedora CoreOS image in this step:
# Setup the correct SELinux label to allow access to the config
chcon --verbose --type svirt_home_t updates.ign
# Start a Fedora CoreOS virtual machine
virt-install --name=fcos --vcpus=2 --ram=2048 --os-variant=fedora-coreos-stable \
--import --network=bridge=virbr0 --graphics=none \
--qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${PWD}/updates.ign" \
--disk=size=20,backing_store=${PWD}/fedora-coreos-older.qcow2
As the system is not up to date, Zincati will notice this and will start updating the system. You should see the update process happening right away:
All necessary network services may not be up and running during the initial check. In such case Zincati will check for updates again in about 5 minutes. |
[core@localhost ~]$ systemctl status --full zincati.service ● zincati.service - Zincati Update Agent Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-11-30 13:56:05 UTC; 5min ago Docs: https://github.com/coreos/zincati Main PID: 1026 (zincati) Status: "found update on remote: 35.20211029.3.0" Tasks: 11 (limit: 2262) Memory: 20.0M CPU: 192ms CGroup: /system.slice/zincati.service ├─1026 /usr/libexec/zincati agent -v └─2308 rpm-ostree deploy --lock-finalization revision=ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c --disallow-downgrade Nov 30 13:56:04 localhost zincati[1026]: [INFO zincati::cincinnati] Cincinnati service: https://updates.coreos.fedoraproject.org Nov 30 13:56:04 localhost zincati[1026]: [INFO zincati::cli::agent] agent running on node '0d817dcff294404895a7a33727980fe8', in update group 'default' Nov 30 13:56:04 localhost zincati[1026]: [INFO zincati::update_agent::actor] registering as the update driver for rpm-ostree Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::update_agent::actor] initialization complete, auto-updates logic enabled Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::strategy] update strategy: immediate Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::update_agent::actor] reached steady state, periodically polling for updates Nov 30 13:56:05 localhost systemd[1]: Started Zincati Update Agent. Nov 30 13:56:05 localhost zincati[1026]: [ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?os_checksum=6334e11901fba7403600522286d817b520eeb15b66b341b392e0c08ed2576b74&stream=stable&os_version=34.20210919.3.0&basearch=x86_64&group=default&node_uuid=0d817dcff294404895a7a33727980fe8&platform=qemu): error trying to connect: dns error: failed to lookup address information: Name or service not known Nov 30 14:01:12 localhost zincati[1026]: [INFO zincati::cincinnati] current release detected as not a dead-end Nov 30 14:01:13 localhost zincati[1026]: [INFO zincati::update_agent::actor] target release '35.20211029.3.0' selected, proceeding to stage it [core@localhost ~]$ rpm-ostree status State: busy AutomaticUpdatesDriver: Zincati DriverState: active; found update on remote: 35.20211029.3.0 Transaction: deploy --lock-finalization revision=ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c --disallow-downgrade Initiator: caller :1.28 Deployments: ● fedora:fedora/x86_64/coreos/stable Version: 34.20210919.3.0 (2021-10-04T17:11:11Z) Commit: 6334e11901fba7403600522286d817b520eeb15b66b341b392e0c08ed2576b74 GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39 [core@localhost ~]$ systemctl status --full zincati.service ● zincati.service - Zincati Update Agent Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-11-30 13:56:05 UTC; 7min ago Docs: https://github.com/coreos/zincati Main PID: 1026 (zincati) Status: "update staged: 35.20211029.3.0; reboot delayed due to active user sessions" Tasks: 5 (limit: 2262) Memory: 16.6M CPU: 229ms CGroup: /system.slice/zincati.service └─1026 /usr/libexec/zincati agent -v Nov 30 13:56:04 localhost zincati[1026]: [INFO zincati::cli::agent] agent running on node '0d817dcff294404895a7a33727980fe8', in update group 'default' Nov 30 13:56:04 localhost zincati[1026]: [INFO zincati::update_agent::actor] registering as the update driver for rpm-ostree Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::update_agent::actor] initialization complete, auto-updates logic enabled Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::strategy] update strategy: immediate Nov 30 13:56:05 localhost zincati[1026]: [INFO zincati::update_agent::actor] reached steady state, periodically polling for updates Nov 30 13:56:05 localhost systemd[1]: Started Zincati Update Agent. Nov 30 13:56:05 localhost zincati[1026]: [ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?os_checksum=6334e11901fba7403600522286d817b520eeb15b66b341b392e0c08ed2576b74&stream=stable&os_version=34.20210919.3.0&basearch=x86_64&group=default&node_uuid=0d817dcff294404895a7a33727980fe8&platform=qemu): error trying to connect: dns error: failed to lookup address information: Name or service not known Nov 30 14:01:12 localhost zincati[1026]: [INFO zincati::cincinnati] current release detected as not a dead-end Nov 30 14:01:13 localhost zincati[1026]: [INFO zincati::update_agent::actor] target release '35.20211029.3.0' selected, proceeding to stage it Nov 30 14:02:55 localhost zincati[1026]: [INFO zincati::update_agent::actor] update staged: 35.20211029.3.0
Shortly after the update has been staged, the system should reboot to apply the latest update. You may need to stop the autologin service to initiate the reboot.
Broadcast message from Zincati at Tue 2021-11-30 14:02:55 UTC: New update 35.20211029.3.0 is available and has been deployed. If permitted by the update strategy, Zincati will reboot into this update when all interactive users have logged out, or in 10 minutes, whichever comes earlier. Please log out of all active sessions in order to let the auto-update process continue. [core@localhost ~]$ sudo systemctl stop serial-getty@ttyS0.service
When we log back in we can view the current version of Fedora CoreOS is now the latest one. The rpm-ostree status
output will also show the older version, which still exists in case we need to rollback:
[core@localhost ~]$ rpm-ostree status State: idle AutomaticUpdatesDriver: Zincati DriverState: active; periodically polling for updates (last checked Tue 2021-11-30 14:22:41 UTC) Deployments: ● fedora:fedora/x86_64/coreos/stable Version: 35.20211029.3.0 (2021-11-17T23:45:08Z) Commit: ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F fedora:fedora/x86_64/coreos/stable Version: 34.20210919.3.0 (2021-10-04T17:11:11Z) Commit: 6334e11901fba7403600522286d817b520eeb15b66b341b392e0c08ed2576b74 GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
The currently booted deployment is denoted by the ● character.
|
You can view the differences between the two versions by running an rpm-ostree db diff
command:
[core@localhost ~]$ rpm-ostree db diff ostree diff commit from: rollback deployment (6334e11901fba7403600522286d817b520eeb15b66b341b392e0c08ed2576b74) ostree diff commit to: booted deployment (ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c) Upgraded: NetworkManager 1:1.30.6-1.fc34 -> 1:1.32.12-2.fc35 NetworkManager-cloud-setup 1:1.30.6-1.fc34 -> 1:1.32.12-2.fc35 NetworkManager-libnm 1:1.30.6-1.fc34 -> 1:1.32.12-2.fc35 NetworkManager-team 1:1.30.6-1.fc34 -> 1:1.32.12-2.fc35 NetworkManager-tui 1:1.30.6-1.fc34 -> 1:1.32.12-2.fc35 WALinuxAgent-udev 2.3.1.1-1.fc34 -> 2.3.1.1-2.fc35 ...
Reverting to the previous version
If the system is not functioning fully for whatever reason we can go back to the previous version:
[core@localhost ~]$ sudo rpm-ostree rollback --reboot
After logging back in after reboot we can see we are now booted back into the old deployment from before the upgrade occurred:
[core@localhost ~]$ rpm-ostree status State: idle AutomaticUpdatesDriver: Zincati DriverState: active; periodically polling for updates (last checked Tue 2021-11-30 15:17:47 UTC) Deployments: ● fedora:fedora/x86_64/coreos/stable Version: 34.20211031.3.0 (2021-11-08T23:55:31Z) Commit: 056a1508fad0eecb740cf10a949d2292e11084195f1defbf455f418a5991daa2 GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39 fedora:fedora/x86_64/coreos/stable Version: 35.20211029.3.0 (2021-11-17T23:45:08Z) Commit: ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
And you can also verify that Zincati will not try to update to the new version we just rollbacked from:
[core@localhost ~]$ systemctl status --full zincati.service ● zincati.service - Zincati Update Agent Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-11-30 15:36:04 UTC; 19s ago Docs: https://github.com/coreos/zincati Main PID: 802 (zincati) Status: "found update on remote: 34.20211031.3.0" Tasks: 10 (limit: 2262) Memory: 23.4M CPU: 169ms CGroup: /system.slice/zincati.service ├─ 802 /usr/libexec/zincati agent -v └─2814 rpm-ostree deploy --lock-finalization revision=056a1508fad0eecb740cf10a949d2292e11084195f1defbf455f418a5991daa2 --disallow-downgrade Nov 30 15:36:03 localhost zincati[802]: [INFO zincati::cli::agent] agent running on node '8d67840e25884927aaeb72f64e100046', in update group 'default' Nov 30 15:36:03 localhost zincati[802]: [INFO zincati::update_agent::actor] registering as the update driver for rpm-ostree Nov 30 15:36:04 localhost zincati[802]: [INFO zincati::update_agent::actor] found 1 other finalized deployment Nov 30 15:36:04 localhost zincati[802]: [INFO zincati::update_agent::actor] deployment 35.20211029.3.0 (ccdac241ed4b83b55e4d72e864923679c00a15804fbea7d1e638e67c2a08dd5c) will be excluded from being a future update target Nov 30 15:36:04 localhost zincati[802]: [INFO zincati::update_agent::actor] initialization complete, auto-updates logic enabled
Cleanup
Now let’s take down the instance for the next test. Disconnect from the serial console by pressing CTRL
+ ]
or from SSH and then destroy the machine:
virsh destroy fcos virsh undefine --remove-all-storage fcos
Conclusion
In these tutorials we have learned a little bit about Fedora CoreOS. We have learned how it is delivered as a pre-created disk image, how it is provisioned in an automated fashion via Ignition, and also how automated updates are configured and achieved via Zincati and rpm-ostree. The next step is to try out Fedora CoreOS for your own use cases and join the community!