Setup Koji Builder SOP

Contents

Upgrades Builders

It is recommended to upgrade or reinstall all builders after each release. To do it, usually the builders are upgraded and virtual machines reinstalled. First they should be upgraded in staging environment to make sure everything is working properly before starting on production.

Staging Environment

Follow the process to upgrade builders in staging environment:

  1. Update the virt-install path on the builders in ansible repository, like in this example

  2. Each builder virtual machine is removed, either via virt-inst-dest or manually, and the buildvm.yml playbook is run to deploy them.

  3. The hub is re-installed or upgraded with dnf --refresh --releasever <release> distro-sync.

  4. Test builds are done to make sure everything is working.

Production Environment

For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From buildvm-s390x-01 to buildvm-s390x-14 are zvm instances, they have to be updated with dnf --releasever NAME distro-sync. In production builders, are usually done per bvmhost at a time:

  1. To do this process the user should be koji admin

  2. Find all the builders on the host looking in /var/log/virthost-lists.out on batcave01 or look for vmhost variable in inventory/host_vars/ on ansible repository

  3. In a loop, disable all those builders using koji disable-host <vmname>. Like the example below:

    for i in seq -w 01 35
do
  echo $i
  koji disable-host buildvm-s390x-$i.s390.fedoraproject.org
done

  4. In a loop check for running builds on them with koji list-tasks --host <vmname>

  5. Either wait for all of them to finish or optionally run koji free-task NAME to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers

  6. Once they are all empty, destroy them as above and reinstall them with the buildvm playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders

  7. After installed, check that each builder is checking in with koji list-hosts | grep <vmname>

  8. In a loop, reenable them

Want to help? Learn how to contribute to Fedora Docs ›