Mass Rebuild

Description

Periodically we do mass rebuilds of rawhide during the development cycle. This SOP outlines the necessary steps.

Assumptions

This assumes that the mass rebuild has already been approved and scheduled via Release Engineering and FESCo. Coordinate with Infrastructure as well for any needed Koji updates.

This also assumes that the mass rebuild does not need to be done in dependency order and that the mass rebuild does not involve an ABI change.

Considerations

  • The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.

  • Check in on scripts frequently to avoid a long-stalled command from adding significant delays in finishing the rebuild.

  • Check with secondary arches, whether they are up-to-date enough with primary, and create the rebuild tag and build target when they are. It will then take care of rebuilds of the arch specific packages in appropriate Koji instances.

  • Please make sure you have right ACLs to run the script mentioned below. FAS membership in at least one of sysadmin-releng/releng-team/cvsadmin/sysadmin is needed in order to get root access on the machine if required.

Actions

Preparatory Steps

The following steps may be completed in the weeks leading up to the scheduled mass rebuild.

  1. Create the mass rebuild tracker issue.

    Create an issue on the Release Engineering issues page that points at the schedule for the current release.

    See the Fedora 43 mass rebuild issue example.

  2. Set up the mass rebuild wiki page.

    The mass rebuild wiki page should answer the following questions for maintainers:

    • Why is the mass rebuild happening?

    • How to opt out of the mass rebuild?

    • Make sure to inform people either via mail or via the wiki page about the PKG_SKIP_LIST which allows packages to be skipped from rebuilding as well as the noautobuild file that they can add to the root of their distgit repository which will cause the mass rebuild script to skip it.

    • What changes from changesets are taken into account for Fedora 44?

    See the Fedora 43 Wiki example.

  3. Send out the mass rebuild notice.

    Send out the same information posted on the wiki to the devel-announce@lists.fedoraproject.org mailing list.

    Wait for moderators to approve the post. If they have not, after some time ping them and ask them to review it.

    SUBJECT:
Fedora 44 Mass Rebuild Notification - Scheduled for YYYY-MM-DD

BODY:
Hello all,

Per the Fedora Linux 44 release schedule [1], we will start a mass rebuild
for Fedora 44 on YYYY-MM-DD.

We have a tracker ticket [2] open on the Release Engineering issue tracker.
In case of any questions and/or requests feel free to leave a comment on
the ticket.

This mass rebuild will be done in a side tag (`f44-rebuild`) and merged
when completed. FTBFS (Fails To Build From Source) bugs will be filed
shortly after the mass rebuild.

Note that maintainers can continue to do their work as normal.
Builds done by maintainers will be newer than the start of the mass rebuild and used instead.
There is no need to build into the rebuild tag or the like.

If you need to exclude any packages from the rebuild, you can use
`PKG_SKIP_LIST` or add a `noautobuild` file to the root of your dist-git
repository. This will instruct the mass rebuild script to skip these packages.

If you encounter any bugs or issues in the reports, please let Release
Engineering know.

You can contact us through the Release Engineering Matrix room [3], by
sending an email to our mailing list [4], or by opening a ticket on our
issue tracker [5].

Regards,
YOUR NAME
Fedora Release Engineering

[1] https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html
[2] https://forge.fedoraproject.org/releng/tickets/issues/ISSUE_NUMBER
[3] https://matrix.to/#/#releng:fedoraproject.org
[4]
https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[5] https://forge.fedoraproject.org/releng/tickets/issues

  4. Create a tag to contain the mass rebuild.

    Mass rebuilds require their own tag to contain all related builds. The example assumes we are doing a rebuild for Fedora 44.

    $ koji add-tag f44-rebuild --parent f44

  5. Request package autosigning for new mass rebuild tag.

    File a ticket with Fedora Infrastructure requesting the new mass rebuild tag be enabled for package autosigning.

  6. Create the Koji target for the mass rebuild.

    Using the same f44-rebuild tag created in the previous example:

    $ koji add-target f44-rebuild f44-build

    koji add-target target-name buildroot-tag destination-tag describes the syntax format above. If the destination-tag is not specified then it will be the same as the target-name.

  7. Update relevant scripts.

    The mass rebuild depends on four main scripts from the Release Engineering repository. Each one requires some changes in variables for each new mass rebuild cycle.

Change the following items:

  • The build tag, holding tag, and target tag should be updated to reflect the Fedora release you’re building for.

  • The epoch should be updated to the point at which all features that the mass rebuild is for have landed in the build system (and a newRepo task completed with those features).

  • The comment which is inserted into spec changelogs.

  • For the references of the above mentioned changes refer to the F40 mass rebuild PR.

Starting The Mass Rebuild

The mass_rebuild.py script takes care of:

  • Discovering available packages in Koji.

  • Trimming out packages which have already been rebuilt.

  • Checking out packages from dist-git.

  • Bumping the spec file.

  • Committing the change.

  • Git tagging the change.

  • Submitting the build request to Koji.

  1. Connect to the mass rebuild host.

    $ ssh compose-branched01.rdu3.fedoraproject.org

  2. Log in as root.

    $ sudo -i

  3. Start a terminal multiplexer (this ensures the script will continue running even if the user gets interrupted).

    # tmux

  4. Clone or checkout the latest copy of the Release Engineering repository.

  5. Run the mass_rebuild.py script.

    # cd tooling/release-process/mass-rebuilds
# ./mass_rebuild.py 2>&1 | tee ~/massbuild.out

  6. Monitor the output for a few minutes to ensure commits and builds are working.

Monitoring The Mass Rebuild

The community has a very high interest in the status of rebuilds and many maintainers will want to know if their build failed right away. The find_failures.py and need_rebuild.py scripts are designed to update publicly available URLs for stakeholders to monitor.

  1. Connect to a compose host (or the monitoring scripts can be run on the same machine where the mass rebuild is happening).

    $ ssh compose-x86-01.rdu3.fedoraproject.org
or
$ ssh compose-branched01.rdu3.fedoraproject.org:22

  2. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session).

    $ tmux

  3. Clone or checkout the latest copy of the Release Engineering repository and move to the location that contains the find_failures.py script.

    $ cd tooling/release-process/mass-rebuilds

  4. Set up the rebuild failures notification website. The find_failures.py script discovers attempted builds that have failed. It lists those failed builds and sorts them by package owner.

    $ while true; do ./find_failures.py > f44-failures.html && cp f44-failures.html /mnt/koji/mass-rebuild/f44-failures.html; sleep 600; done

  5. Start a second pane in the terminal emulator.

  6. Set up the site for packages that need to be rebuilt. The need_rebuild.py script discovers packages that have not yet been rebuilt and generates an html file that lists them sorted by package owner. This gives external stakeholders a rough idea of how much work is remaining in the mass rebuild.

    $ cd tooling/release-process/mass-rebuilds/
$ while true; do ./need_rebuild.py > f44-need-rebuild.html && cp -f f44-need-rebuild.html /mnt/koji/mass-rebuild/f44-need-rebuild.html; sleep 600; done

Post Mass Rebuild Tasks

Once the mass rebuild script completes, and all the pending builds have finished, the builds will need to be tagged. The mass_tag.py script will accomplish this task. The script will:

  • Discover completed builds.

  • Trim out builds that are older than the latest build for a given package.

  • Tag remaining builds into their final destination (without generating email).

  1. Clone the latest copy of the Release Engineering repository.

  2. Make sure there is no autosign config for f44 to f44 in place. If there is then remove it.

  3. Run the mass_tag.py script (requires Koji Kerberos authentication).

    $ cd tooling/release-process/mass-rebuilds/
$ ./mass_tag.py --source f44-rebuild --target f44

  4. Remove the Koji target for the rebuild.

    $ koji remove-target f44-rebuild

  5. Send the final notification to the devel-announce@lists.fedoraproject.org list.

    SUBJECT:
Fedora 44 Mass Rebuild Completed

BODY:
Hello all,

Per the Fedora Linux 44 release schedule [1], we started a mass rebuild
for Fedora 44 on YYYY-MM-DD. This is now completed.

The mass rebuild was run for the changes listed here:
https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Driving_Features

The rebuild was done in a side tag (f44-rebuild) that has been merged into f44.

Failures from the mass rebuild can be found here:
https://kojipkgs.fedoraproject.org/mass-rebuild/f44-failures.html

Packages that still need to be rebuilt can be found here:
https://kojipkgs.fedoraproject.org/mass-rebuild/f44-need-rebuild.html

A total of X packages were identified as needing to be rebuilt as part of this efford.
At the end of the mass rebuild, X builds failed and will need to be addressed by package maintainers.
In total, X builds were successfully tagged into f44.

FTBFS bugs will be filed shortly for the remaining failures.

Discussions related to the Fedora 44 mass rebuild took place in the tracker ticket [2].

Please let Release Engineering know if you notice any issues with the reporting.

You can contact us through the Release Engineering Matrix room [3], by
sending an email to our mailing list [4], or by opening a ticket on our
issue tracker [5].

Regards,
YOUR NAME
Fedora Release Engineering

[1] https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html
[2] https://forge.fedoraproject.org/releng/tickets/issues/ISSUE_NUMBER
[3] https://matrix.to/#/#releng:fedoraproject.org
[4]
https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[5] https://forge.fedoraproject.org/releng/tickets/issues

Want to help? Learn how to contribute to Fedora Docs ›