Setup Koji Builder SOP
It is recommended to upgrade or reinstall all builders after each release. To do it, usually the builders are upgraded and virtual machines reinstalled. First they should be upgraded in staging environment to make sure everything is working properly before starting on production.
Follow the process to upgrade builders in staging environment:
Update the virt-install path on the builders in ansible repository, like in this example
Each builder virtual machine is removed, either via
virt-inst-destor manually, and the
buildvm.ymlplaybook is run to deploy them.
The hub is re-installed or upgraded with
dnf --refresh --releasever <release> distro-sync.
Test builds are done to make sure everything is working.
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From buildvm-s390x-01 to buildvm-s390x-14 are zvm instances, they have to be updated with
dnf --releasever NAME distro-sync.
In production builders, are usually done per bvmhost at a time:
To do this process the user should be koji admin
Find all the builders on the host looking in
/var/log/virthost-lists.outon batcave01 or look for
inventory/host_vars/on ansible repository
In a loop, disable all those builders using
koji disable-host <vmname>. Like the example below:
for i in seq -w 01 35 do echo $i koji disable-host buildvm-s390x-$i.s390.fedoraproject.org done
In a loop check for running builds on them with
koji list-tasks --host <vmname>
Either wait for all of them to finish or optionally run
koji free-task NAMEto free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
Once they are all empty, destroy them as above and reinstall them with the
buildvmplaybook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders
After installed, check that each builder is checking in with
koji list-hosts | grep <vmname>
In a loop, reenable them