EPEL Minor Version Mass Branching

Description

Originally the EPEL repository had packages that built against the latest RHEL minor, but since EPEL 10 the EPEL repository now builds against the latest CentOS Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process lets the developers continue to maintain their packages against CentOS Stream without risking to break packages that already work on the current active RHEL minor target.

Preparatory steps

Send announcement

Announce the upcoming mass branching one week ahead of time. You can use this email as a reference.

Prepare ansible changes

Most of the branching process is controlled by ansible playbooks. Before these are run some changes will be needed in the ansible repo. Set these up ahead of time in a pull request, which will be merged on branching day. These changes include:

  • change the epel_minor variable to 3

  • change the epel_branched_minor variable to 2

  • add the epel10.3 release to new-updates-sync

  • disable syncing of the centos-10-snapshot directory

You can use this pull request as a reference.

Branching day

Send announcement

Reply to the previous announcement to state that the mass branching is starting.

Remove koji targets

Remove the epel10 koji targets to prevent builds during the branching process. These targets will later be recreated using 10.3 tags instead of 10.2 tags.

$ koji remove-target epel10-candidate
$ koji remove-target epel10

Cancel all running build for EPEL 10.2

List all running tasks and select only those relevant for current branching.

$ koji list-builds --state=0 --type=rpm | grep el10_2 | awk '{print $1}'

Cancel each of those tasks

$ koji cancel

Merge ansible changes

Merge the pending ansible pull request.

Koji

Collect lists of builds that have been created for epel10.2 but have not yet reached stable. These can later be manually tagged for epel10.3.

$ koji list-tagged --quiet epel10.2-testing-candidate
$ koji list-tagged --quiet epel10.2-signing-pending
$ koji list-tagged --quiet epel10.2-testing-pending
$ koji list-tagged --quiet epel10.2-testing
$ koji list-tagged --quiet epel10.2-pending

Once these lists have been captured, run the following playbook on your local machine to create the epel10.3 tags and perform related tasks. Leaving out the koji_profile variable will cause it to be run with the stg koji profile.

$ ansible-playbook \
      -e koji_profile=koji \
      -e "release=10.3" \
      playbooks/manual/epel-minor-release/koji-release-tags.yml

Dist-git

Run the following playbook on batcave to create epel10.2 dist-git branches for every package with an active (not retired) epel10 branch. Leaving out the target variable will cause it to be run in staging.

$ sudo rbac-playbook \
      -e target=pkgs \
      -e release=10.2 \
      -e branch_from=epel10 \
      manual/epel-minor-release/branch-distgit-packages.yml

Bodhi

Bodhi and new-updates-sync expect certain directories and symlinks to already exist, so we need to create those before creating the release in bodhi. Run the following playbook on batcave. Leaving out the target variable will cause it to be run in staging.

$ sudo rbac-playbook \
      -e target=bodhi_backend \
      -e "release=10.3" \
      manual/epel-minor-release/prepare-bodhi-repos.yml

Other playbooks

Run the following playbooks to apply the remaining changes made in ansible.

Make sure to make releng aware that we need to run the autosign playbook, since it requires for the server to be accessible from ssh.

 
$ sudo rbac-playbook groups/batcave.yml
$ sudo rbac-playbook groups/koji-hub.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook manual/autosign.yml

Bodhi releases

Create the new EPEL 10.3 bodhi release.

$ bodhi releases create \
      --name EPEL-10.3 \
      --long-name "Fedora EPEL 10.3" \
      --version 10.3 \
      --branch epel10 \
      --id-prefix FEDORA-EPEL \
      --dist-tag epel10.3 \
      --stable-tag epel10.3 \
      --testing-tag epel10.3-testing \
      --candidate-tag epel10.3-testing-candidate \
      --pending-signing-tag epel10.3-signing-pending \
      --pending-testing-tag epel10.3-testing-pending \
      --pending-stable-tag epel10.3-pending \
      --override-tag epel10.3-override \
      --mail-template fedora_epel_legacy_errata_template \
      --state current \
      --released-on $(date +%F) \
      --composed-by-bodhi \
      --no-create-automatic-updates

Adjust branch field of the previous EPEL 10.2 release.

$ bodhi releases edit \
      --name EPEL-10.2 \
      --branch epel10.2

The new release will not show up in the bodhi web interface until the pods are re-deployed and some services are restarted.

Run the following playbook on batcave.

$ sudo rbac-playbook openshift-apps/bodhi.yml

Run the following command on bodhi-backend01.

$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service

Recreate koji targets

Once mass-branching is complete and the new release exists in bodhi, re-create these targets with the following commands:

$ koji add-target epel10           epel10.3-build epel10.3-testing-candidate
$ koji add-target epel10-candidate epel10.3-build epel10.3-testing-candidate

MirrorManager

MirrorManager should automatically map the new repo. Verify this with the following command:

$ curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-z-10.3'

We need to update the repo redirects that are used in metalinks. This is done in the MirrorManager admin panel web interface. Once complete, verify the repo redirects with the following command:

$ curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-10'

Koschei

Create EPEL 10.3 in koschei.

Send announcement

Reply to the previous announcement to state that the mass branching has concluded.

Want to help? Learn how to contribute to Fedora Docs ›