Testing Fedora CoreOS updates

Make sure that you have completed the steps described in the initial setup page before starting this tutorial.

In this tutorial, we will not focus on provisioning but on what happens during updates and the options that are available in case of failures.

Downloading an older Fedora CoreOS release

One of the defining feature of Fedora CoreOS is automatic updates. To see them in action, we have to download an older Fedora CoreOS release. Start by picking an older release from the Fedora CoreOS release page:

RELEASE="32.20200629.3.0"
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz
curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/$RELEASE/x86_64/fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig

Once the archive has been downloaded, make sure to verify its integrity:

gpg --verify fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz.sig

Look for "Good signature from" in the output.

Once you have verified the archive, you can extract it with:

unxz fedora-coreos-$RELEASE-qemu.x86_64.qcow2.xz

To make the tutorial simpler, you should rename this image to a shorter name:

mv fedora-coreos-$RELEASE-qemu.x86_64.qcow2 fedora-coreos-older.qcow2

Writing the Butane config and converting to Ignition

We will create a Butane config that:

Sets up console autologin.
Raise kernel console logging level to hide audit messages from the console.
Add an SSH Key for the core user.

Let’s write this Butane config to a file called updates.bu:

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA...
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
          TTYVTDisallocate=no
storage:
  files:
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # to hide audit messages from the interactive console
          kernel.printk=4

Optionally you can replace the SSH pubkey in the yaml file with your own public key so you can log in to the booted instance. If you choose not to do this you’ll still be auto logged in to the serial console.

Run Butane to convert that to an Ignition config:

butane --pretty --strict updates.bu --output updates.ign

Startup and initial update

Now let’s provision it. Make sure that you are starting from the older Fedora CoreOS image in this step:

# Setup the correct SELinux label to allow access to the config
chcon --verbose --type svirt_home_t updates.ign

# Start a Fedora CoreOS virtual machine
virt-install --name=fcos --vcpus=2 --ram=2048 --os-variant=fedora-coreos-stable \
    --import --network=bridge=virbr0 --graphics=none \
    --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${PWD}/updates.ign" \
    --disk=size=20,backing_store=${PWD}/fedora-coreos-older.qcow2

As the system is not up to date, Zincati will notice this and will immediately start updating the system. If you are fast enough, you should see the update process happening right away:

[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-08-07 17:42:14 UTC; 24s ago
       Docs: https://github.com/coreos/zincati
   Main PID: 889 (zincati)
      Tasks: 6 (limit: 2290)
     Memory: 17.9M
     CGroup: /system.slice/zincati.service
             ├─ 889 /usr/libexec/zincati agent -v
             └─1889 rpm-ostree deploy --lock-finalization revision=a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869 --disallow-downgrade

Aug 07 17:42:14 localhost systemd[1]: Started Zincati Update Agent.
Aug 07 17:42:14 localhost zincati[889]: [INFO ] starting update agent (zincati 0.0.11)
Aug 07 17:42:17 localhost zincati[889]: [INFO ] Cincinnati service: https://updates.coreos.fedoraproject.org
Aug 07 17:42:17 localhost zincati[889]: [INFO ] agent running on node '2ce419a50b0a4a4f8b10e5e9cf487b68', in update group 'default'
Aug 07 17:42:17 localhost zincati[889]: [INFO ] initialization complete, auto-updates logic enabled
Aug 07 17:42:18 localhost zincati[889]: [INFO ] target release '32.20200715.3.0' selected, proceeding to stage it

[core@localhost ~]$ rpm-ostree status
State: busy
Transaction: deploy --lock-finalization revision=a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869 --disallow-downgrade
  Initiator: caller :1.18
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-08-07 17:42:14 UTC; 51s ago
       Docs: https://github.com/coreos/zincati
   Main PID: 889 (zincati)
      Tasks: 5 (limit: 2290)
     Memory: 17.8M
     CGroup: /system.slice/zincati.service
             ├─ 889 /usr/libexec/zincati agent -v
             └─4463 rpm-ostree finalize-deployment a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869

Aug 07 17:42:14 localhost systemd[1]: Started Zincati Update Agent.
Aug 07 17:42:14 localhost zincati[889]: [INFO ] starting update agent (zincati 0.0.11)
Aug 07 17:42:17 localhost zincati[889]: [INFO ] Cincinnati service: https://updates.coreos.fedoraproject.org
Aug 07 17:42:17 localhost zincati[889]: [INFO ] agent running on node '2ce419a50b0a4a4f8b10e5e9cf487b68', in update group 'default'
Aug 07 17:42:17 localhost zincati[889]: [INFO ] initialization complete, auto-updates logic enabled
Aug 07 17:42:18 localhost zincati[889]: [INFO ] target release '32.20200715.3.0' selected, proceeding to stage it
Aug 07 17:43:05 localhost zincati[889]: [INFO ] staged deployment '32.20200715.3.0' available, proceeding to finalize it

Shortly after the update has been staged, the system should reboot to apply the latest update.

When we log back in we can view the current version of Fedora CoreOS is now 32.20200715.3.0. The rpm-ostree status output will also show the older version, which still exists in case we need to rollback:

[core@localhost ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200715.3.0 (2020-07-27T11:36:29Z)
                    Commit: a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

The currently booted deployment is denoted by the ● character.

You can view the differences between the two versions by running an rpm-ostree db diff command:

[core@localhost ~]$ rpm-ostree db diff
ostree diff commit from: rollback deployment (6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890)
ostree diff commit to:   booted deployment (a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869)
Upgraded:
  btrfs-progs 5.6.1-1.fc32 -> 5.7-1.fc32
  console-login-helper-messages 0.18.1-1.fc32 -> 0.18.2-1.fc32
  console-login-helper-messages-issuegen 0.18.1-1.fc32 -> 0.18.2-1.fc32
  console-login-helper-messages-motdgen 0.18.1-1.fc32 -> 0.18.2-1.fc32
  console-login-helper-messages-profile 0.18.1-1.fc32 -> 0.18.2-1.fc32
  crun 0.13-2.fc32 -> 0.14.1-1.fc32
  crypto-policies 20200610-1.git7f9d474.fc32 -> 20200619-1.git781bbd4.fc32
  dbus 1:1.12.18-1.fc32 -> 1:1.12.20-1.fc32
  dbus-common 1:1.12.18-1.fc32 -> 1:1.12.20-1.fc32
  dbus-libs 1:1.12.18-1.fc32 -> 1:1.12.20-1.fc32
  ...

Reverting to the previous version

If the system is not functioning fully for whatever reason we can go back to the previous version:

[core@localhost ~]$ sudo rpm-ostree rollback --reboot

After logging back in after reboot we can see we are now booted back into the old 32.20200629.3.0 deployment from before the upgrade occurred:

[core@localhost ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200715.3.0 (2020-07-27T11:36:29Z)
                    Commit: a3b08ee51b1d950afd9d0d73f32d5424ad52c7703a6b5830e0dc11c3a682d869
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

And you can also verify that Zincati will not try to update to the new version we just rollbacked from:

[core@localhost ~]$ systemctl status --full zincati.service
● zincati.service - Zincati Update Agent
     Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-08-08 15:55:15 UTC; 2min 19s ago
       Docs: https://github.com/coreos/zincati
   Main PID: 674 (zincati)
      Tasks: 2 (limit: 2290)
     Memory: 17.8M
     CGroup: /system.slice/zincati.service
             └─674 /usr/libexec/zincati agent -v

Aug 08 15:55:15 localhost systemd[1]: Started Zincati Update Agent.
Aug 08 15:55:15 localhost zincati[674]: [INFO ] starting update agent (zincati 0.0.11)
Aug 08 15:55:19 localhost zincati[674]: [INFO ] Cincinnati service: https://updates.coreos.fedoraproject.org
Aug 08 15:55:19 localhost zincati[674]: [INFO ] agent running on node '817ccb9a75ec4f2b845e74fdb81e8850', in update group 'default'
Aug 08 15:55:19 localhost zincati[674]: [INFO ] initialization complete, auto-updates logic enabled

Cleanup

Now let’s take down the instance for the next test. Disconnect from the serial console by pressing CTRL + ] or from SSH and then destroy the machine:

virsh destroy fcos
virsh undefine --remove-all-storage fcos

Conclusion

In these tutorials we have learned a little bit about Fedora CoreOS. We have learned how it is delivered as a pre-created disk image, how it is provisioned in an automated fashion via Ignition, and also how automated updates are configured and achieved via Zincati and rpm-ostree. The next step is to try out Fedora CoreOS for your own use cases and join the community!