Mass Branching

Description

At each release’s "branching point" we branch the pending release away from devel/ which allows Rawhide (currently F45) to move on while the pending release goes into bugfix and polish mode.

Below is a list of steps to follow to branch a new Fedora release.

Consider Before Running

Please proactively check for blockers with the infrastructure team and the community.

Mass Re-Signing (One Week Before Branching Day)

This work should be initiated about one week before the branching day.

When we branch off of Rawhide, the branched release packages are already signed by the F45 key, but we need to re-sign everything in Rawhide with the new F46 key.

We also need to re-sign all of the ELN packages as they use the same key as Rawhide.

The signing key has to be updated in the ELN override file and has to be listed first, i.e.:

sigkeys = ['NEW_KEY', 'OLD_KEY']

Add a new config for the new key to robosignatory. Something like:

            [[consumer_config.koji_instances.primary.tags]]
            from = "f45"
            to = "f45"
            key = "{{ (env == 'production')|ternary('fedora-46', 'testkey') }}"
            keyid = "{{ (env == 'production')|ternary('a15B79cc', 'd300e724') }}"
            {% if env == "production" %}
            file_signing_key = "fedora-46-ima"
            {% endif %}

            [[consumer_config.koji_instances.primary.tags]]
            from = "eln"
            to = "eln"
            key = "{{ (env == 'production')|ternary('fedora-46', 'testkey') }}"
            keyid = "{{ (env == 'production')|ternary('a15B79cc', 'd300e724') }}"
            {% if env == "production" %}
            file_signing_key = "fedora-46-ima"
            {% endif %}

This allows robosignatory to sign packages in the F45 tag with the F46 key.

git clone https://forge.fedoraproject.org/releng/tooling.git
Confirm the new key fingerprint is in release-process/signing/koji_rpm_signer.py.
Run koji_rpm_signer.py to gather the list of packages to sign:

./release-process/signing/koji_rpm_signer.py --just-list --tag f45 fedora-46 | grep ".src.rpm$" | sed -e 's|.src.rpm$||' > unsigned-packages

You should get a list of all of the source packages by name.

Copy unsigned-packages list to autosign01.rdu3.fedoraproject.org.
On autosign01 run in a tmux session:

sudo -su robosignatory
passphrase=$(systemd-ask-password "Please enter passphrase for 'autosign' key: ")
(enter the autosign passphrase)
keyctl add user "sigul:autosign" "${passphrase}" @s
for i in `cat unsigned-packages`
do
  echo $i;
  robosignatory sign-tag primary $i f45;
  sleep 1;
done

This will iterate over all of the packages and sign them with the new F46 key. Once complete, re-run the ./release-process/signing/koji_rpm_signer.py command to confirm all are signed.

Repeat the process with the 'eln' tag.

On branching day, remove the robosignatory config for this re-signing.

The OpenH264 packages are not included in the general mass re-signing process since they live outside of the main f45 tag. These must be handled separately. More on this later.

Preparatory Steps (A Few Days Before Branching Day)

This section covers changes that must be in place before the actual branching day process can start. It is best to prepare them in advance so that there is sufficient time for other Release Engineering team members to review and approve them.

Create A Ticket In The Release Engineering Tracker

Tracker: https://forge.fedoraproject.org/releng/tickets

TITLE:
F45 Mass Branching: Preparatory PRs

Create a table with the links to the pull requests.
This makes it less difficult to keep track of all of the moving pieces.
Consider using previous branching tickets for reference.

Send Announcement

Mailing list: devel-announce@lists.fedoraproject.org

SUBJECT:
Fedora 45 Mass Branching Notification - Scheduled for YYYY-MM-DD

BODY:
Hello all,

Per the release schedule [1], we will start the mass branching for Fedora 45 on YYYY-MM-DD.

We have a tracker ticket [2] open on the Release Engineering issue tracker. In case of any questions and/or requests feel free to leave a comment on the ticket.

Rawhide builds in Koji will be disabled for the duration of Mass Branching. All builds running at that time will be canceled and can be resubmitted by maintainers after the Mass Branching. All Rawhide updates that are pending will be unpushed.

Once Fedora 45 is branched we will re-enable builds in Koji with a notification to this list.

You can contact us through the Release Engineering Matrix room [3], by sending an email to our mailing list [4], or by opening a ticket on our issue tracker [5].

Regards,
YOUR NAME
Fedora Release Engineering

[1] https://fedorapeople.org/groups/schedule/f-45/f-45-key-tasks.html
[2] https://forge.fedoraproject.org/releng/tickets/issues/XXXXX
[3] https://matrix.to/#/#releng:fedoraproject.org
[4] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[5] https://forge.fedoraproject.org/releng/tickets/issues

Create Bugzilla Component

To ensure accurate tracking and management of issues for each Fedora release, a new Bugzilla component should be created for the branched version.

A user with access to the fedora-pm Bugzilla group needs to perform the following actions:

Log in to Bugzilla.
Navigate to the "Administration" section.
Under "Components," locate "Fedora."
Select "Edit Versions" to view the existing components.
Add the new branched version to the list.

This process allows for proper issue categorization and ensures all bugs related to the new Fedora release are properly tracked.

Repository Changes

The following repositories need to be updated, including adding a new branch for the branched release and updating the Rawhide branch with new release values.

https://forge.fedoraproject.org/releng/pungi-fedora
https://forge.fedoraproject.org/infra/ansible
https://pagure.io/fedora-kiwi-descriptions
https://pagure.io/fedora-comps
https://pagure.io/fedora-lorax-templates/
https://pagure.io/workstation-ostree-config/
- Follow the procedure detailed in the repo README under the Branching instructions for new Fedora releases section.
https://src.fedoraproject.org/rpms/fedora-release
https://src.fedoraproject.org/rpms/fedora-repos
https://github.com/fedora-infra/statusfpo

Release Engineering Tooling Updates

Repository: https://forge.fedoraproject.org/releng/tooling

`find_unblocked_orphans.py`

Location: packages/orphaned/orphaned-packages-process/find_unblocked_orphans.py
Key changes:
- Update koji_tag from f45 to f46.
- Update pagure_branch from f44 to f45.
- Ensure koji_hub references remain consistent across updates.

`get_active_packages.sh`

Location: release-process/mass-branching/fedora/get_active_packages.sh
- Update OUTPUT_FINAL variable in script to components_f45.txt.

Ansible

Repository: https://forge.fedoraproject.org/infra/ansible

Bodhi

Bodhi needs to be updated to add a new release. This needs to be done in the bodhi2 role:

roles/bodhi2/backend/files/new-updates-sync
roles/bodhi2/backend/tasks/main.yml
roles/bodhi2/backend/templates/pungi.rpm.conf.j2
roles/bodhi2/backend/templates/koji_sync_listener.toml

Poddlers

Add a new entry to the template with the correct bug_fixes and security_fixes dates (EOL date in the release schedule).

Fedora Branched

Set FedoraBranched to True.
Set FedoraBranchedBodhi to preenable.

Koji Hub

Update the roles/koji_hub/templates/hub.conf.j2 file.
- In the sidetag section, add tag f46-build :: allow.
Check the roles/koji_hub/templates/koji-gc.conf.j2 file.
- List the new release key and retention.

Robosignatory

Update the robosignatory config.
- Add the new release (45).
- Disable branched signing so that we can freeze branched until we get a compose.

Fedora Release

Dist-git repository: https://src.fedoraproject.org/rpms/fedora-release
File: fedora-release.spec
On rawhide branch update relevant lines:
- %define eol_date YYYY-MM-DD
  - End Of Life date from release schedule.
- %define dist_version 46
- Version: 46
On f45 branch update relevant lines:
- %define release_name Forty Five
- %define is_rawhide 0

Fedora Repos

Dist-git repository: https://src.fedoraproject.org/rpms/fedora-repos
File: fedora-repos.spec
On rawhide branch update relevant lines:
- %global rawhide_release 46
- Version: 46
- Release: 0.1%{?eln:.eln%{eln}}
- Add a %changelog entry:

%changelog
* Day Mon DD YYYY Name Email - 46-0.1
- Rawhide is now F46

On f45 branch update relevant lines:
- %global rawhide_release 46
- global updates_testing_enabled 1
- Release: 0.3%{?eln:.eln%{eln}}
  - Increment whatever number there is by one.
- Add a %changelog entry:

%changelog
* Day Mon DD YYYY Name Email - 45-0.3
- Rawhide is now F46
- `updates-testing` repository for F45 enabled

Make Sure That `mock-core-configs` Is Updated

Repository: https://github.com/rpm-software-management/mock

If it is not updated then file an issue.

OpenH264

Create a set of empty repositories under /srv/web/codecs.fedoraproject.org/openh264/46.

After the mass branching is finished we will need to get a new Rawhide (F46) build, send it to Cisco, and update the empty repository with metadata once Cisco publishes the binaries.

Branching Day

This section is for actions on the actual branching day.

Update `statusfpo`

Repository: https://github.com/fedora-infra/statusfpo
Add an outage as per the project’s documentation.

Unpush Testing Updates In Bodhi

Find out which updates need to be unpushed.
- CLI: $ bodhi updates query --releases F45 --status testing
- Web: https://bodhi.fedoraproject.org/updates/?releases=F45&status=testing
Unpush the updates one by one.
- $ bodhi updates request <id> unpush
For each of the unpushed updates, add a comment about why it was unpushed.
- $ bodhi updates comment <id> "This update was unpushed as part of Fedora 45 Mass Branching because it failed gating tests. Please resubmit these changes for F45 and/or F46/Rawhide."

TODO: A script could be written to automate this step.

Koji

Disable External HTTP Submissions

Disable external HTTP submissions to Koji by blocking proxy01/proxy10 on both koji01.rdu3.fedoraproject.org and koji02.rdu3.fedoraproject.org. Do this on both hosts so that one is not left open to external users.

SSH to each Koji host.

$ ssh koji01.rdu3.fedoraproject.org
and in a second tmux window:
$ ssh koji02.rdu3.fedoraproject.org

Make sure the toggle script is available from the releng tooling repository.

$ git clone https://forge.fedoraproject.org/releng/tooling.git
$ cd tooling/release-process/mass-branching/fedora
$ sudo install -m 0755 koji_http_toggle.sh /usr/local/sbin/koji-http-toggle.sh

Block external HTTP submissions.

$ sudo /usr/local/sbin/koji-http-toggle.sh block

Verify the block before continuing.

$ sudo /usr/local/sbin/koji-http-toggle.sh status

Verify that external users cannot submit builds: a remote koji build submission or fkinit should fail while the block is active.

To re-enable external submissions later:

$ sudo /usr/local/sbin/koji-http-toggle.sh unblock

Cancel All Running Builds For Rawhide

Host: compose-x86-01.rdu3.fedoraproject.org
1. List all running tasks and select only those that are relevant to the current branching.
  $ koji list-builds --state=0 --type=rpm | grep fc45 | awk '{print $1}'
2. Cancel each of those tasks.
  $ koji cancel <build>

Dist-Git

Host: pkgs01.rdu3.fedoraproject.org which has to be accessed via SSH from batcave01.rdu3.fedoraproject.org
Merge the prepared releng PR.

Dist-git needs to be updated in two steps:

Create new git branches.
Update gitolite.conf to allow users to push to the new branches.

For both of these actions, we will first need to obtain a list of all active packages.

Generate The Active Packages List

Access the host.

$ ssh batcave01.rdu3.fedoraproject.org
$ sudo ssh pkgs01.rdu3.fedoraproject.org  # For production
$ sudo ssh pkgs01.stg.rdu3.fedoraproject.org  # For testing

Clone the Release Engineering repository and make sure the script is executable.

$ git clone https://forge.fedoraproject.org/releng/tooling.git
$ cd tooling/release-process/mass-branching/fedora/
$ chmod +x get_active_packages.sh

Run the script.
```
$ ./get_active_packages.sh
```

This will generate the file components_f45.txt which contains all valid, active, and imported packages, prefixed with rpm/.

Create New Git Branches

On pkgs01.stg (for testing) and pkgs01 (for production), run:

$ sudo -u pagure python /usr/local/bin/mass-branching-git.py 45 INPUT_FILE

Where INPUT_FILE is the path to the file generated by the get_active_packages.sh script.

Update `gitolite.conf`

TODO

Ansible

Host: batcave01.rdu3.fedoraproject.org
Merge the prepared Ansible PR.

Apply The Changes

Run the playbooks.

$ sudo rbac-playbook groups/koji-hub.yml -t koji_hub
$ sudo rbac-playbook groups/releng-compose.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook groups/openqa.yml -t testcase_stats
$ sudo rbac-playbook openshift-apps/bodhi.yml -t all,rollout
$ sudo rbac-playbook openshift-apps/greenwave.yml -t all,rollout
$ sudo rbac-playbook openshift-apps/poddlers.yml -t all,build,rollout
$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2

Run the autosign playbook.
- Someone with access to autosign01.rdu3.fedoraproject.org needs to log in there.
- Start sshd.
- Run playbooks/manual/autosign.yml.
- Stop sshd.
- Enter the passphrase when it restarts due to changes.

Make sure to run the koji-hub.yaml playbook with the -t koji_hub option to only apply the hub configuration changes. Running the playbook without this option will overwrite the manual changes done by the koji-http-toggle.sh script.

Create Koji Release Tags

Host: compose-x86-01.rdu3.fedoraproject.org

The Koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from Rawhide go.

Clone the Release Engineering repository.

$ git clone https://forge.fedoraproject.org/releng/tooling.git
$ cd tooling/release-process/mass-branching/fedora/

Run the script.
```
$ ./create_koji_release_tags.sh 45
```

Bodhi

Host: bodhi-backend01.rdu3.fedoraproject.org

Create Empty Repositories

Clone the Release Engineering repository and copy the script.

$ git clone https://forge.fedoraproject.org/releng/tooling.git
$ cp tooling/release-process/mass-branching/fedora/create_empty_repositories.sh /var/tmp`

Run the script as the ftpsync user.

$ sudo -u ftpsync sh /var/tmp/create_empty_repositories.sh 45

Link Empty Repositories

Link the empty repositories so that new-updates-sync does not complain about it.

$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f45-updates
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f45-updates-testing

Set The End Of Life Date For The New Release

Set the EOL date based on the Fedora release schedule.

$ bodhi releases edit --name F45 --eol YYYY-MM-DD
$ bodhi releases edit --name F45F --eol YYYY-MM-DD
$ bodhi releases edit --name F45 --branch f45

Create Rawhide Release

To create a new Rawhide release in Bodhi, you need to run:

$ bodhi releases create \
  --name "F46" --long-name "Fedora 46" \
  --id-prefix FEDORA --version 46 --branch rawhide \
  --dist-tag f46 \
  --stable-tag f46 \
  --testing-tag f46-updates-testing \
  --candidate-tag f46-updates-candidate \
  --pending-stable-tag f46-updates-pending \
  --pending-testing-tag f46-updates-testing-pending \
  --pending-signing-tag f46-signing-pending \
  --state pending \
  --override-tag f46-override \
  --create-automatic-updates \
  --not-composed-by-bodhi

To create a Flatpak release for F45 in Bodhi, you need to run:

$ bodhi releases create \
  --name "F45F" --long-name "Fedora 45 Flatpaks" \
  --id-prefix FEDORA-FLATPAK --version 45 --branch f45 \
  --dist-tag f45-flatpak \
  --stable-tag f45-flatpak-updates \
  --testing-tag f45-flatpak-updates-testing \
  --candidate-tag f45-flatpak-updates-candidate \
  --pending-stable-tag f45-flatpak-updates-pending \
  --pending-testing-tag f45-flatpak-updates-testing-pending \
  --state pending \
  --override-tag f45-flatpak-override

Build `fedora-repos` And `fedora-release` Packages

Host: compose-x86-01.rdu3.fedoraproject.org

Merge the prepared PRs.

Clone the packages.

$ fedpkg clone -B -a fedora-release
$ fedpkg clone -B -a fedora-repos

Do the builds for both packages for Rawhide.

$ fedpkg request-side-tag --base-tag f45-build
$ cd fedora-release/f45
$ fedpkg build --nowait --target=SIDETAG_ID
$ cd ../../fedora-repos/f45
$ fedpkg build --nowait --target=SIDETAG_ID

Do the builds for both packages for F44.

$ fedpkg request-side-tag --base-tag f44-build
$ cd fedora-release/f44
$ fedpkg build --nowait --target=SIDETAG_ID
$ cd ../../fedora-repos/f44
$ fedpkg build --nowait --target=SIDETAG_ID

Create the two updates in Bodhi.

$ bodhi updates new --from-tag f46-build-side-XXXXXX --notes "fedora-release and fedora-repos for new Rawhide (F46)"
$ bodhi updates new --from-tag f45-build-side-XXXXXX --notes "fedora-release and fedora-repos for new Branched (F45)"

These two updates must be in stable before doing a compose.

Update `fedora-appstream-metadata`

Dist-git repository: https://src.fedoraproject.org/rpms/fedora-appstream-metadata

TODO

Update `fedfind` Release Metadata

We need to update the /srv/groups/qa/metadata/release.json file on fedorapeople (people01). It is used by fedfind as a source of truth about release numbers.

openQA, fedora-image-uploader and various other things will behave incorrectly if it is not updated. The new branched release number should be added to the branched list in this file. Before the edit, the list should be empty.

If you do not have access to edit this file, ask a member of the Quality team to do so.

Create The First Composes

Host for Rawhide: compose-rawhide01.rdu3.fedoraproject.org
Host for Branched: compose-branched01.rdu3.fedoraproject.org
These PRs must be merged before the composes are fired off:
- pungi-fedora
- fedora-kiwi-descriptions
- fedora-comps
- workstation-ostree-config

Once the other changes are all done, we must create post-branching composes for both Rawhide and the newly branched F45 as soon as possible.

Process

SSH into the appropriate compose host.

$ ssh compose-rawhide01.rdu3.fedoraproject.org
or
$ ssh compose-branched01.rdu3.fedoraproject.org

Navigate to the directory containing cronjobs.
```
$ cd /etc/cron.d
```
Manually edit the cronjob so that it runs the next minute. For example if it is currently 14:27, change the first two numbers to 28 14.
Once the compose has started, remember to change the numbers back to their original values.

It is done this way so that everything is exactly the same as when the cronjob usually runs.

Please verify the repository permissions that are created under /pub/fedora/linux/development/45 and /pub/fedora-secondary/development/45. They should be owned by ftpsync:ftpsync. Check directory permissions (should be "0755") to ensure new composes synchronize correctly.

Update MirrorManager

MirrorManager needs to be updated so that it will point Rawhide to the new Rawhide release.

MirrorManager now runs in OpenShift.

SSH to batcave, become root, then reach the OpenShift control host.

$ ssh batcave01.rdu3.fedoraproject.org
$ sudo su
$ ssh os-control01.rdu3.fedoraproject.org

Find the MirrorManager namespace and enter a frontend pod.

$ oc -n mirrormanager get pods
$ oc -n mirrormanager rsh <pod_name> -- /bin/bash

Open psql and update redirects.

-- Remove redirects for the previous Rawhide (F45)
delete from repository_redirect where from_repo like '%45%';

-- Add redirects for the new Rawhide (F46)
insert into repository_redirect (from_repo, to_repo) values ('fedora-46', 'rawhide');
insert into repository_redirect (from_repo, to_repo) values ('fedora-source-46', 'rawhide-source');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-f46', 'rawhide');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-debug-f46', 'rawhide-debug');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-source-f46', 'rawhide-source');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-f46', 'rawhide');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-debug-f46', 'rawhide-debug');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-source-f46', 'rawhide-source');
insert into repository_redirect (from_repo, to_repo) values ('fedora-debug-46', 'rawhide-debug');
UPDATE repository_redirect SET to_repo = regexp_replace(to_repo, '-45$', '-46') WHERE from_repo ILIKE '%openh264%';

-- if modular redirects still exist in your environment, add these too
insert into repository_redirect (from_repo, to_repo) values ('fedora-modular-46', 'rawhide-modular');
insert into repository_redirect (from_repo, to_repo) values ('fedora-modular-debug-46', 'rawhide-modular-debug');
insert into repository_redirect (from_repo, to_repo) values ('fedora-modular-source-46', 'rawhide-modular-source');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-modular-f46', 'rawhide-modular');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-modular-f46', 'rawhide-modular');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-modular-debug-f46', 'rawhide-modular-debug');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-modular-debug-f46', 'rawhide-modular-debug');
insert into repository_redirect (from_repo, to_repo) values ('updates-testing-modular-source-f46', 'rawhide-modular-source');
insert into repository_redirect (from_repo, to_repo) values ('updates-released-modular-source-f46', 'rawhide-modular-source');

Use psql mirrormanager2 if you need to connect to the database explicitly.

Enable Autosigning On Branched Release

Once the branched compose is finished we need to re-enable robosignatory on the branched release.

ELN related work

Should this section be in the Create Release Signing Key SOP instead of here?

Add the new Rawhide key to ELN Pungi config. For example, look at this pungi eln config commit

Change the trigger notification for DistroBuildSync to the new Rawhide version. For example, look at this commit.

Branch New Rawhide In Koschei

Follow this SOP: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release

Documentation Attributes Update

Repository: https://forge.fedoraproject.org/infra/docs
Location: modules/release_guide/pages/_partials/attributes.adoc
1. Update the attributes to new values.
2. Open a pull request.
3. Merge the pull request.

Resources

Fedora 45 release schedule.
Fedora 44 preparatory PRs ticket (for reference).
Fedora 43 mass branching tracker ticket (for reference).

Want to help? Learn how to contribute to Fedora Docs ›

Mass Branching

Description

Consider Before Running

Mass Re-Signing (One Week Before Branching Day)

Preparatory Steps (A Few Days Before Branching Day)

Create A Ticket In The Release Engineering Tracker

Send Announcement

Create Bugzilla Component

Repository Changes

Release Engineering Tooling Updates

find_unblocked_orphans.py

get_active_packages.sh

Ansible

Bodhi

Poddlers

Fedora Branched

Koji Hub

Robosignatory

Fedora Release

Fedora Repos

Make Sure That mock-core-configs Is Updated

OpenH264

Branching Day

Update statusfpo

Unpush Testing Updates In Bodhi

Koji

Disable External HTTP Submissions

Cancel All Running Builds For Rawhide

Dist-Git

Generate The Active Packages List

Create New Git Branches

Update gitolite.conf

Ansible

Apply The Changes

Create Koji Release Tags

Bodhi

Create Empty Repositories

Link Empty Repositories

Set The End Of Life Date For The New Release

Create Rawhide Release

Build fedora-repos And fedora-release Packages

Update fedora-appstream-metadata

Update fedfind Release Metadata

Create The First Composes

Process

Update MirrorManager

Enable Autosigning On Branched Release

ELN related work

Branch New Rawhide In Koschei

Documentation Attributes Update

Resources

`find_unblocked_orphans.py`

`get_active_packages.sh`

Make Sure That `mock-core-configs` Is Updated

Update `statusfpo`

Update `gitolite.conf`

Build `fedora-repos` And `fedora-release` Packages

Update `fedora-appstream-metadata`

Update `fedfind` Release Metadata