Testing Fedora CoreOS updates

Make sure that you have completed the steps described in the initial setup page before starting this tutorial.

In this tutorial, we will not focus on provisioning but on what happens during updates and the options that are available in case of failures.

Downloading an older Fedora CoreOS release

One of the defining features of Fedora CoreOS is automatic updates. To see them in action, we have to download an older Fedora CoreOS release. Start by picking an older release from the Fedora CoreOS release page:

# replace "XX.XXXXXXXX.X.X" with one of older release IDs from the release page above (e.g. "34.20210919.3.0")
RELEASE="XX.XXXXXXXX.X.X"
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig

Once the archive has been downloaded, make sure to verify its integrity:

curl https://fedoraproject.org/fedora.gpg | gpg --import
gpg --verify fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig
Look for "Good signature from" in the output.

Once you have verified the archive, you can extract it with:

unxz fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz

To make the tutorial simpler, you should rename this image to a shorter name:

mv fedora-coreos-$RELEASE-qemu.x86_64.qcow2 fedora-coreos-older.qcow2

Writing the Butane config and converting to Ignition

We will create a Butane config that:

  • Sets up console autologin.

  • Add an SSH Key for the core user.

Let’s write this Butane config to a file called updates.bu:

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA...
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
          TTYVTDisallocate=no
storage:
  files:
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat
Optionally you can replace the SSH pubkey in the yaml file with your own public key so you can log in to the booted instance. If you choose not to do this you’ll still be auto logged in to the serial console.

Run Butane to convert that to an Ignition config:

butane --pretty --strict updates.bu --output updates.ign

Startup and initial update

Now let’s provision it. Make sure that you are starting from the older Fedora CoreOS image in this step:

# Setup the correct SELinux label to allow access to the config
chcon --verbose --type svirt_home_t updates.ign

# Start a Fedora CoreOS virtual machine
virt-install --name=fcos --vcpus=2 --ram=2048 --os-variant=fedora-coreos-stable \
    --import --network=bridge=virbr0 --graphics=none \
    --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${PWD}/updates.ign" \
    --disk=size=20,backing_store=${PWD}/fedora-coreos-older.qcow2

As the system is not up to date, Zincati will notice this and will start updating the system. You should see the update process happening right away:

All necessary network services may not be up and running during the initial check. In such case Zincati will check for updates again in about 5 minutes. 
[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-08-21 02:32:06 UTC; 5min ago
       Docs: https://github.com/coreos/zincati
   Main PID: 1360 (zincati)
     Status: "found update on remote: 36.20220723.3.1"
      Tasks: 12 (limit: 2254)
     Memory: 22.0M
        CPU: 407ms
     CGroup: /system.slice/zincati.service
             ├─ 1360 /usr/libexec/zincati agent -v
             └─ 1828 rpm-ostree deploy --lock-finalization --skip-branch-check revision=e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45 --disallow-downgrade

Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::cincinnati] Cincinnati service: https://updates.coreos.fedoraproject.org
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::cli::agent] agent running on node 'ff3024bf1d764dee887eb7f8c69f9952', in update group 'default'
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] registering as the update driver for rpm-ostree
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] initialization complete, auto-updates logic enabled
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::strategy] update strategy: immediate
Aug 21 02:32:06 localhost.localdomain systemd[1]: Started zincati.service - Zincati Update Agent.
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] reached steady state, periodically polling for updates
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?os_checksum=4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f&node_uuid=ff3024bf1d764dee887eb7f8c69f9952&os_version=36.20220716.3.1&basearch=x86_64&group=default&platform=qemu&stream=stable): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Aug 21 02:37:12 localhost.localdomain zincati[1360]: [INFO  zincati::cincinnati] current release detected as not a dead-end
Aug 21 02:37:13 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] target release '36.20220723.3.1' selected, proceeding to stage it

[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; update staged: 36.20220723.3.1; reboot delayed due to active user sessions
Deployments:
  fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220723.3.1 (2022-08-08T17:18:13Z)
                    Commit: e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
                      Diff: 26 upgraded

● fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220716.3.1 (2022-07-25T18:45:58Z)
                    Commit: 4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4


[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-08-21 02:32:06 UTC; 7min ago
       Docs: https://github.com/coreos/zincati
   Main PID: 1360 (zincati)
     Status: "update staged: 36.20220723.3.1; reboot delayed due to active user sessions"
      Tasks: 5 (limit: 2254)
     Memory: 18.2M
        CPU: 487ms
     CGroup: /system.slice/zincati.service
             └─ 1360 /usr/libexec/zincati agent -v

Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] registering as the update driver for rpm-ostree
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] initialization complete, auto-updates logic enabled
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::strategy] update strategy: immediate
Aug 21 02:32:06 localhost.localdomain systemd[1]: Started zincati.service - Zincati Update Agent.
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] reached steady state, periodically polling for updates
Aug 21 02:32:06 localhost.localdomain zincati[1360]: [ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?os_checksum=4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f&node_uuid=ff3024bf1d764dee887eb7f8c69f9952&os_version=36.20220716.3.1&basearch=x86_64&group=default&platform=qemu&stream=stable): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Aug 21 02:37:12 localhost.localdomain zincati[1360]: [INFO  zincati::cincinnati] current release detected as not a dead-end
Aug 21 02:37:13 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] target release '36.20220723.3.1' selected, proceeding to stage it
Aug 21 02:37:53 localhost.localdomain zincati[1360]: [INFO  zincati::update_agent::actor] update staged: 36.20220723.3.1
Aug 21 02:37:53 localhost.localdomain zincati[1360]: [WARN  zincati::update_agent] interactive sessions detected, entering grace period (maximum 10 minutes)

Shortly after the update has been staged, the system should reboot to apply the latest update. You may need to stop the autologin service to initiate the reboot.

Broadcast message from Zincati at Sun 2022-08-21 02:37:53 UTC:
New update 36.20220723.3.1 is available and has been deployed.
If permitted by the update strategy, Zincati will reboot into this update when
all interactive users have logged out, or in 10 minutes, whichever comes
earlier. Please log out of all active sessions in order to let the auto-update
process continue.
[core@localhost ~]$ sudo systemctl stop serial-getty@ttyS0.service

When we log back in we can view the current version of Fedora CoreOS is now the latest one. The rpm-ostree status output will also show the older version, which still exists in case we need to rollback:

[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Sun 2022-08-21 02:41:21 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220723.3.1 (2022-08-08T17:18:13Z)
                    Commit: e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

  fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220716.3.1 (2022-07-25T18:45:58Z)
                    Commit: 4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
The currently booted deployment is denoted by the ● character.

You can view the differences between the two versions by running an rpm-ostree db diff command:

[core@localhost ~]$ rpm-ostree db diff
ostree diff commit from: rollback deployment (4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f)
ostree diff commit to:   booted deployment (e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45)
Upgraded:
  clevis 18-6.fc36 -> 18-8.fc36
  clevis-dracut 18-6.fc36 -> 18-8.fc36
  clevis-luks 18-6.fc36 -> 18-8.fc36
  clevis-systemd 18-6.fc36 -> 18-8.fc36
  ...

Reverting to the previous version

If the system is not functioning fully for whatever reason we can go back to the previous version:

[core@localhost ~]$ sudo rpm-ostree rollback --reboot

After logging back in after reboot we can see we are now booted back into the old deployment from before the upgrade occurred:

[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Sun 2022-08-21 02:42:48 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220716.3.1 (2022-07-25T18:45:58Z)
                    Commit: 4e2f304fe4af38272772514b2bd685c4b4981422f553b53e469f8df3c01d883f
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

  fedora:fedora/x86_64/coreos/stable
                   Version: 36.20220723.3.1 (2022-08-08T17:18:13Z)
                    Commit: e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

And you can also verify that Zincati will not try to update to the new version we just rollbacked from:

[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-08-21 02:42:48 UTC; 35s ago
       Docs: https://github.com/coreos/zincati
   Main PID: 844 (zincati)
     Status: "periodically polling for updates (last checked Sun 2022-08-21 02:42:48 UTC)"
      Tasks: 5 (limit: 2254)
     Memory: 20.5M
        CPU: 245ms
     CGroup: /system.slice/zincati.service
             └─ 844 /usr/libexec/zincati agent -v

Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::cincinnati] Cincinnati service: https://updates.coreos.fedoraproject.org
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::cli::agent] agent running on node 'ff3024bf1d764dee887eb7f8c69f9952', in update group 'default'
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::update_agent::actor] registering as the update driver for rpm-ostree
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::update_agent::actor] found 1 other finalized deployment
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::update_agent::actor] deployment 36.20220723.3.1 (e0b2bf7e843d71f2aff103705b491c3c28c1416cedcb093646fd44037b7b6a45) will be excluded from being a future update target
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::update_agent::actor] initialization complete, auto-updates logic enabled
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::strategy] update strategy: immediate
Aug 21 02:42:48 localhost.localdomain systemd[1]: Started zincati.service - Zincati Update Agent.
Aug 21 02:42:48 localhost.localdomain zincati[844]: [INFO  zincati::update_agent::actor] reached steady state, periodically polling for updates
Aug 21 02:42:49 localhost.localdomain zincati[844]: [INFO  zincati::cincinnati] current release detected as not a dead-end

Cleanup

Now let’s clean up the instance. Disconnect from the serial console by pressing CTRL + ] or from SSH and then destroy the machine:

virsh destroy fcos
virsh undefine --remove-all-storage fcos

Conclusion

In these tutorials we have learned a little bit about Fedora CoreOS. We have learned how it is delivered as a pre-created disk image, how it is provisioned in an automated fashion via Ignition, and also how automated updates are configured and achieved via Zincati and rpm-ostree. The next step is to try out Fedora CoreOS for your own use cases and join the community!

Want to help? Learn how to contribute to Fedora Docs ›