Bodhi Infrastructure Release and Deployment SOP

Bodhi is used by Fedora developers to submit potential package updates for releases and to manage buildroot overrides. From here, bodhi handles all of the dirty work, from sending around emails, dealing with Koji, to composing the repositories.

Bodhi production instance: https://bodhi.fedoraproject.org

Bodhi project page: https://github.com/fedora-infra/bodhi

See also: Bodhi Infrastructure - Releng SOP

Performing a bodhi upgrade

Necessary permissions

Groups to be member in:

In addition you should be named as app-owner in the bodhi os playbook.

Have the release in the official Bodhi repository ready

Following steps were taken from https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst up to the point of building a beta-release.

Changing the specfile manually is tedious and error-prone, it is more useful to just do several patch releases on staging, before we commit to one to deploy on production.

Releasing a patch-release

Make sure you have all of the code you want to release in the branch. I.e. for 5.7.3 release, you would be looking at https://github.com/fedora-infra/bodhi/tree/5.7

Easiest way to incorporate additional changes from PRs targetting develop, is to tag them with *-backports tag, i.e. 5.7-backports. Mergify will automatically create a cherry-picked PR targetting the release branch.

When you are ready to denote the new patch version:

  1. bump version in ./bodhi/init.py

  2. run the get-authors.py in ./news

  3. run towncrier

  4. review the docs/user/release_notes.rst

  5. commit, push to your branch and make a PR against develop with the corect *-backports label

After all of the tests successfully passed in the PRs, and both of them are merged, tag the release: 1. git checkout 5.7 2. git pull origin 5.7 3. git tag 5.7.3 4. git push origin refs/tags/5.7.3

If you push the wrong tag, you can delete it by git push origin :refs/tags/5.7.3

Beware, mergify creates the bacport PR only after the pr to develop was merged.

Testing deployed Bodhi without making a release

If the Bodhi changes don’t involve any of

  • Composer code

  • Doesn’t include any upgrades to database with alembic

You can opt to use the hotfix-style container-image that can be found among the templates: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix

Change the env-vars in the docker-file-base-hotfix to reflect the commit to be built. The image definition is not buildable locally, as it needs the openshift environment to configure koji and kerberos.

Change the buildconfig to point to the new definition:

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14]

Then commit and push the changes, to be able to deploy from batcave with:

sudo rbac-playbook openshift-apps/bodhi.yml -l staging

Make sure to check os.stg.fedoraproject.org to see if the build completed successfully.

Build Bodhi

Bodhi is deployed from the infrastructure Koji repositories. At the time of this writing, it is deployed from the f34-infra and f34-infra-stg (for staging) repositories. Bodhi is built for these repositories from the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[ ]bodhi dist-git repository.

You can use fedpkg to download the package:

fedpkg clone rpms/bodhi

Curently we are interested in f34 builds as all of the bodhi infra runs on f34 servers:

fedpkg switch-branch f34

Edit the bodhi.spec specfile to have the correct release version.

Download the sources to the SOURCES folder (if it doesn’t exist, just mkdir -p /rpmbuild/SOURCES/)

wget https://github.com/fedora-infra/bodhi/archive/refs/tags/5.7.3.tar.gz -o /rpmbuild/SOURCES/bodhi-5.7.3.tar.gz

As an example, to build a Bodhi  for the f34-infra-stg repository, you can use this command:

$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec

Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm

$ koji build f34-infra ~/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm

When building a Bodhi release that is intended for production, we should build from the production dist-git repo instead of uploading an SRPM:

$ koji build f34-infra git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37

All builds against the f34-infra build target will go into the f34-infra-stg repository. If you wish to promote a build from staging to production, you can do something like this command:

$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra

Staging

The upgrade playbook will apply configuration changes after running the alembic upgrade. Sometimes you may need changes applied to the Bodhi systems in order to get the upgrade playbook to succeed. If you are in this situation, you can apply those changes by running the bodhi-backend playbook:

sudo rbac-playbook -l staging groups/bodhi-backend.yml

In the https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg, edit the bodhi_version setting to the version you wish to deploy to staging. For example, to deploy version 5.7.3 to staging, I would set that varible like this:

bodhi_version:  ‘5.7.3’

The package itself will then be installed in container build by ‘dnf install -y bodhi-server-\{{bodhi_version}}’

Beware, when running the playbooks on the backend vm, it currently installs the latest bodhi package available in the respective infra/infra-stg tag.

It can be useful to verify correct version is available on the backend, i.e. for staging run

ssh bodhi-backend01.stg.iad2.fedoraproject.org
dnf list bodhi-server --refresh --all --showduplicates

In future we will probably pin these in the VM as well.

Make sure you have the correct docker-image set in the configuration. in the ansible repo:

vim roles/openshift-apps/bodhi/templates/buildconfig.yml

After you checked, and have the correct version, commit and create a PR for ansible repo.

After it has been merged, run these commands:

# Synchronize the database from production to staging
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
# Upgrade the Bodhi backend on staging
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
# Upgrade the Bodhi frontend on staging
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging

Tests to run

For compose test you can follow https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta

For automated health-check, you can run https://pagure.io/fedora-ci/monitor-gating,

It should be sufficient to check the openshift project https://os.fedoraproject.org/console/project/monitor-gating where it is being run every few hours. There is a dashboard available https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating

Production

After testing in staging, you should be able to move the rpm from staging to production:

koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34

Even if the rpm version is pinned in your deploy, double check the availability in the tag

The upgrade playbook will apply configuration changes after running the alembic upgrade. Sometimes you may need changes applied to the Bodhi systems in order to get the upgrade playbook to succeed. If you are in this situation, you can apply those changes by running the bodhi-backend playbook:

sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend

In thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[ ]os_masters inventory, edit the bodhi_version setting to the version you wish to deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to production, I would set that varible like this:

bodhi_version: "5.7.3"

To update the bodhi RPMs in production:

# Update the backend VMs (this will also run the migrations, if any)
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
# Update the frontend
$ sudo rbac-playbook openshift-apps/bodhi.yml

All of the playbooks should finish with no failures.

Afterwards, check https://os.fedoraproject.org/console/project/bodhi/overview to see if the image has been built and if it has been properly deployed.

Things to watchout for

  • Correct rpm not being available for container build

    • This is the reason to check the infra-tag for bodhi

  • Failure to start the pod due to backend db mismatch

    • Sometimes the alembic upgrade that is supposed to run as part of playbooks/manual/upgrade/bodhi.yml fails silently

  • Some issues resurface only after few days in production, resulting in visible stack-traces in logs.

Syncing the production database to staging

This can be useful for testing issues with production data in staging:

$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging

Want to help? Learn how to contribute to Fedora Docs ›