Mass Rebuild

Description

Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.

Assumptions

This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.

This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.

Considerations

  • The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.

  • Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild.

  • Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.

  • Please make sure you have right ACLs to run the script mention below, one needs to be atleast in sysadmin-releng/releng-team/cvsadmin/sysadmin in order to get root access of the machine if required.

Actions

Preparatory Steps

The following steps may be completed in the weeks leading up to the scheduled mass rebuild.

  1. Create the Mass Rebuild Pagure Issue

    Create an issue on the Release Engineering issues page that points at the schedule for the current release.

    See the Fedora 27 mass rebuild issue example.

  2. Set up the Mass Rebuild Wiki Page

    The mass rebuild wiki page should answer the following questions for maintainers:

    • Why the mass rebuild is happening?

    • How to opt out of the mass rebuild?

    • Make sure to inform people either on mail, or in the wiki page about the PKG_SKIP_LIST which allows packages to be skipped from rebuilding as well as noautobuild file that they can add it to there root of a distgit repository and the mass rebuild script will skip it.

    • What changes from changesets are taken in account for F41?

    See the Fedora 41 Wiki example.

  3. Send out the Mass Rebuild Notice

    Send out the same information posted on the wiki to the devel-announce@lists.fedoraproject.org mailing list. Please check if the email is through, if stuck in moderation please ping respective members.

    See the template in releng repository template.

  4. Create a Tag to Contain the Mass Rebuild

    Mass rebuilds require their own tag to contain all related builds. The example assumes we are doing a rebuild for Fedora 42.

    $ koji add-tag f42-rebuild --parent f42

  5. Request Package Auto-Signing for New Mass-Rebuild Tag

    File a ticket with Fedora Infrastructure requesting the new mass-rebuild tag be enabled for package auto-signing.

  6. Create the Koji Target for the Mass Rebuild

    Using the same f42-rebuild tag created in the previous example:

    $ koji add-target f42-rebuild f42-build

    koji add-target target-name buildroot-tag destination-tag describes the syntax format above. If the destination-tag is not specified then it will be the same as the target-name.

  7. Update Scripts

    The mass rebuild depends on four main scripts from the releng git repository. Each one requires some changes in variables for each new mass rebuild cycle.

    • mass-rebuild.py

      • buildtag

      • targets

      • epoch

      • comment

      • target

    • find-failures.py

      • buildtag

      • desttag

      • epoch

    • mass-tag.py

    • need-rebuild.py

      • buildtag

      • target

      • updates

      • epoch

Change the following items:

  • the build tag, holding tag, and target tag should be updated to reflect the Fedora release you’re building for

  • the epoch should be updated to the point at which all features that the mass rebuild is for have landed in the build system (and a newRepo task completed with those features)

  • the comment which is inserted into spec changelogs

  • for the references of the above mentioned changes refer to the F40 mass rebuild PR.

Starting the Mass Rebuild

The mass-rebuild.py script takes care of:

  • Discovering available packages in koji

  • Trimming out packages which have already been rebuilt

  • Checking out packages from git

  • Bumping the spec file

  • Committing the change

  • git tagging the change

  • Submitting the build request to Koji

  1. Connect to the mass-rebuild Machine

    $ ssh compose-branched01.iad2.fedoraproject.org

  2. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)

    $ tmux

  3. Clone or checkout the latest copy of the releng git repository.

  4. Run the mass-rebuild.py script from releng/scripts

    $ cd path/to/releng_repo/scripts
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out

Monitoring Mass Rebuilds

The community has a very high interest in the status of rebuilds and many maintainers will want to know if their build failed right away. The find-failures.py and need-rebuild.py scripts are designed to update publicly available URLs for stakeholders to monitor.

  1. Connect to a Compose Machine or the monitoring scripts can be run on the same machine where the mass rebuild is happening.

    $ ssh compose-x86-02.iad2.fedoraproject.org / compose-branched01.iad2.fedoraproject.org:22

  2. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)

    $ tmux

  3. Clone or checkout the latest copy of the releng git repository

  4. Set Up the Rebuild Failures Notification Web Site

    The find_failures.py script discovers attempted builds that have failed. It lists those failed builds and sorts them by package owner. +

$ while true; do ./find_failures.py > f42-failures.html && cp f42-failures.html /mnt/koji/mass-rebuild/f42-failures.html; sleep 600; done

  1. Start a second pane in the terminal emulator

  2. Set up the Site for Packages that Need Rebuilt

    The need-rebuild.py script discovers packages that have not yet been rebuilt and generates an html file listing them sorted by package owner. This gives external stakeholders a rough idea of how much work is remaining in the mass rebuild. +

$ while true; do ./need-rebuild.py > f42-need-rebuild.html && cp f42-need-rebuild.html /mnt/koji/mass-rebuild/f42-need-rebuild.html; sleep 600; done

Post Mass Rebuild Tasks

Once the mass rebuild script completes, and all the pending builds have finished, the builds will need to be tagged. The mass-tag.py script will accomplish this task. The script will:

  • Discover completed builds

  • Trim out builds that are older than the latest build for a given package

  • Tag remaining builds into their final destination (without generating email)

  1. Clone or checkout the latest copy of the releng git repository

  2. Run the mass-tag.py script (requires koji kerberos authentication)

  3. Remove the koji target for the rebuild: $ koji remove-target f42-rebuild

    $ cd path/to/releng_repo/scripts
$ ./mass-tag.py --source f42-rebuild --target f42

  4. Send the final notification to the devel-announce@lists.fedoraproject.org list

    For the contents use this template.

Want to help? Learn how to contribute to Fedora Docs ›