Release Process SOPs

This section covers program management procedures related to the process of getting a release out the door. The table of contents below is organized by category. The document beyond that is in chronological order.

You may choose to create a wiki page to track the status of housekeeping tasks, as has been done historically.

Wiki and website creation

There are so many places in the wiki to create things. This is probably not an exhaustive list. Hopefully in the future we’ll have improved this process by consolidating and automating.

  • Update the index.html page on the web view of the schedule.

  • Create a release page (Releases/N) in the wiki that collects the pages in that category. You can use {{Special:PrefixIndex/Releases/N/}} to make it automagic (replacing "N" with the release number). This is a good target for de-wikifying.

  • Create a release schedule page (Releases/N/Schedule) in the wiki. For Fedora 31 and previous, this included a manually-transcribed copy of key milestones. For Fedora 32 and beyond, Ben decided that’s a good way to introduce human error and made that page point to the web view of the schedule instead.

  • Create a ChangeSet page (Releases/N/ChangeSet) in the wiki. The output of the change processing scripts gives you the content for this page.

  • Create a Spins page (Releases/N/Spins) in the wiki. Copy this from the previous version and make any changes as appropriate. This is another good target for de-wikifying.

  • Create a Release blocking deliverables page (Releases/N/ReleaseBlocking) in the wiki. Again, copy this from the previous version and make changes as appropriate. Stop me if you’ve heard this before, but this is a good target for de-wikifying.

  • Add the new release to the Releases wiki page.

  • Add the new release to the Changes wiki page.

Spins keepalive

In order to keep from shipping spins that no one is maintaining anymore, FESCo approved a process where the Program Manager will check with spin owners to make sure they’re still around and wanting to produce the spin. The deadline corresponds with the Self-Contained Change deadline.

A few weeks before the deadline, create Pagure tickets for the spins and tag the appropriate owner. Then you can send this email to devel-announce and spins mailing lists (update the date and versions):

FESco previously approved a requirement that Spin/Labs owners send a
keepalive request in order to keep building the spin or lab. I have
opened Pagure issues[1] for all Spins and Labs listed on the wiki[2].

If you are the owner of one of those spins and labs, please reply in
the appropriate ticket by30 January 2019 to indicate the spin should
continue to be produced. If there is a spin or lab that does not have
an open ticket, please create one[3].

The reasoning for this is to not ship spins that are not actively
maintained. Future improvements to the release process that will allow
for teams to self-publish solutions will eventually remove the need
for these keepalives.

[1] https://pagure.io/fedora-project-schedule/issues
[2] https://fedoraproject.org/wiki/Releases/30/Spins
[3] https://pagure.io/fedora-project-schedule/new_issue

Rawhide Rebase Warning

Greetings,

This e-mail is intended to inform you about the upcoming Bugzilla changes
happening on TKTK (Rawhide bug rebase) and what you need to do, if anything.

We will be automatically changing the version for most rawhide bugs to Fedora TKTK.
This will result in regular bugs reported against rawhide during the Fedora TKTK
development cycle being changed to version 'TKTK' instead of their current
assignment, ‘rawhide’.  This is to align with the branching of Fedora TKTK from
rawhide and to more accurately tell where in the lineage of releases the bug was
last reported.

Note that this procedure does not apply to bugs that are open for the ‘Package
Review’ or 'kernel' components or bugs that have the ''FutureFeature'' or ''Tracking'' keywords
set. These will stay open as rawhide bugs indefinitely.

If you do not want your bugs changed to version ‘TKTK‘, add the ''FutureFeature''
keyword. If you need help changing a large amount of bugs manually, we’d be glad
to help.

The process was re-approved by FESCo https://pagure.io/fesco/issue/1096 .

Release blocking deliverables

Some deliverables are release blockers — we do not ship the release if they are not successfully composed. FESCo approved a proposal to use a static list of release-blocking deliverables. Formerly, this was a manual audit process each release.

A week before the branch point, create the release-blocking page in the wiki (see the F31 page as an example). Email this to the devel mailing list. Any edits should have been included in Change proposals, but it’s a good opportunity to catch anything that got overlooked.

Branch Day

Update product description

Adjust wording of the Fedora product in Bugzilla to reflect the branch.

Bugs related to the components of the Fedora distribution. If you are reporting a bug
against a stable release or a branched pre-release version please select that
version number. The currently maintained released versions are: Fedora N-1, Fedora N.
The branched pre-released version is Fedora N+1. If you have a bug to report against
the daily development tree (rawhide) please choose 'rawhide' as the version.

For more information about filing a bug against Fedora packages, see
https://docs.fedoraproject.org/en-US/quick-docs/howto-file-a-bug/

Create N+1 version

Rawhide Rebase

All of the following condiditions must be met:

  • version == rawhide

  • component != "Package Review"

  • component != "kernel"

  • opened against rawhide before 2019-08-13 (Rawhide Branching date is 2019-08-13)

  • keyword ''FutureFeature'' is '''not''' present

  • keyword ''Tracking'' is '''not''' present

  • the string ''RFE'' is '''not''' present in the summary

  • status != CLOSED

  • You can use this Bugzilla query to select bugs that meet the criteria above.

Save the results as a CSV file and run the fedora_rebase.pl script against it. For example:

fedora_rebase.pl 31 bugs-2019-08-13.csv

This script will take a while to run. Execute it from a machine with stable power and networking.

Go/No-Go meeting

Before each public release, FESCo, QA, and Release Engineering meet to determine if the release criteria are met for a particular release. This meeting is called the Go/No-Go Meeting. FPgM is responsibile for scheduling and leading this important meeting.

Occurrence

This meeting is held for every public release (Beta and Final) and may reoccur for each milestone if the release slips. Meetings are generally held the Thursday before a target release date.

SOP

Pre-meeting

  • Schedule the meeting in Fedocal’s Fedora release calendar. The Go/No-Go meeting is traditionally held on the Thursday prior to the release date from 17:00–19:00 UTC. Select an available meeting channel. #fedora-meeting-1 has generally been available at this slot. It is best to set the meeting to recur weekly for 3 weeks in case of multiple slips.

  • Send an email 5–7 calendar days before the meeting to the devel-announce, logistics, and test-announce mailing lists.

During the meeting

The IRC script below provides a general framework for running the Go/No-Go meeting. The general flow is to review blocker bugs, verify that a compose is available, and that the test matrices are completed. If there are proposed or accepted blocker bugs, the meeting becomes a blocker review meeting to evaluate them. This portion of the meeting can be led by a member of the QA team or by the Program Manager. It is not uncommon to defer some bugs until later in the meeting so that last-minute testing can be performed.

The compose section is generally very quick. Release Engineering or QA will confirm that a candidate compose exists.

In the test matrices section, QA will review the results of tests. Some optional tests may not be complete.

Finally, the Program Manager polls each of the constituent teams. If they all vote "go", then the release is "go" and will be released on the coming Tuesday. The Program Manager’s work is done (at least as far as this goes). If the release is "no-go", then a follow-up meeting must happen. Traditionally, we wait until the following Thursday. However for small issues that can be fixed for the next compose, we may choose to reconvene in 24 hours (Friday). If the release is still no-go at that point, the next meeting should happen on Thursday as regularly scheduled.

IRC script
#startmeeting F31 Beta Go/No-Go meeting
#meetingname F31-Beta-Go_No_Go-meeting

#topic Roll Call

#topic Purpose of this meeting
#info Purpose of this meeting is to check whether or not F31 Beta is ready for shipment, according to the release criteria.
#info This is determined in a few ways:
#info 1. No remaining blocker bugs
#info 2. Release candidate compose is available
#info 3. Test matrices for Beta are fully completed

#topic Current status — Blocker bugs
#link https://qa.fedoraproject.org/blockerbugs/milestone/31/beta/buglist

#topic Current status — Candidate compose

#topic Current status — Test matrix coverage
#link https://fedoraproject.org/wiki/Category:Fedora_31_Test_Results

#topic Go/No-Go decision
I will poll each team. Please reply “go” or “no-go”

FESCO?

Releng?

QA?

#agreed Fedora 31 Beta is GO
#info Fedora 31 Beta will release on 2019-09-17
 --- or ---
#agreed Fedora 31 Beta is NO-GO
#info The next F31 Beta Go/No-Go meeting will be Thursday, 2019-09-19 at 1700 UTC

#action bcotton to announce decision

#endmeeting

Post-meeting

  • If release status is Go, FPgM has to

    • announce the result to the devel-announce, test-announce, and logistics lists

    • remove the subsequent release dates from the schedule

  • If release status is No-Go, FPgM has to

    • announce the result and the next meeting details to the devel-announce, test-announce, and logistics lists

    • update the schedule to reflect the new target date

      • For a Beta slip of one or two weeks, we do not adjust the Final release date. For slippages of more than two or three weeks, you may need to propose a change to the subsequent schedules. A final release slip likely does not require slipping the subsequent release unless it is longer than a month.

Release Readiness meeting

Before each public release all of the groups participating the development of Fedora’s next release meet to make sure the release is well-coordinated. This meeting is called the Release Readiness Meeting. The FPgM is responsible for scheduling and leading this meeting.

Occurrence

This meeting is held for every public release (Beta and Final), but only once for each milestone.

SOP

The current process is not great. It requires teams to have a representative present at a particular time and if something is wrong, it may be too late to solve the problem before the release. This meeting needs some re-evaluation in order to ensure it is useful to the community.

Pre-meeting

  • Schedule the meeting in Fedocal’s Fedora release calendar. The Go/No-Go meeting is traditionally held on the Thursday prior to the first scheduled release date from 19:00–21:00 UTC. Select an available meeting channel. #fedora-meeting-1 has generally been available at this slot. It is best to set the meeting to recur weekly for 3 weeks in case of multiple slips.

  • Send an email 5–7 calendar days before the meeting to as many mailing lists as you can think of plus Discussion in order to reach as many stakeholder groups as possible.

During the meeting

Go through each area and check for their status. (You will probably get a lot of no-answers.) Use the #info command to record statuses for the minutes. If an area needs help, use the #help command to raise it and coordinate among attendees to resolve the issue or follow up on it.

After the meeting is over, award participants the Readiness badge.

IRC script
#startmeeting F[version] [milestone = Beta, Final] Readiness Meeting
#meetingname f[version]-[milestone = beta, final]-readiness-meeting

#topic Roll Call

#chair [xxx]

#topic Purpose of this meeting
#info "Before each public release all of the groups participating the development of Fedora's next release meet to make sure the release is well coordinated."
#link https://fedoraproject.org/wiki/Release_Readiness_Meetings

#topic Current status
#topic Ambassadors
#topic Design
#topic Documentation
#topic FESCo
#topic Fedora Engineering Manager
#topic Fedora Project Leader
#topic Marketing
#topic Infrastructure
#topic QA
#topic Release Engineering
#topic Translations
#topic Websites
#topic Open floor

Release Day

Update product description

Adjust wording of the Fedora product in Bugzilla to reflect the branch.

Bugs related to the components of the Fedora distribution. If you are reporting a bug
against a stable release or a branched pre-release version please select that
version number. The currently maintained released versions are: Fedora N-2, Fedora N-1,
and Fedora N. If you have a bug to report against the daily development tree (rawhide)
please choose 'rawhide' as the version.

For more information about filing a bug against Fedora packages, see
https://docs.fedoraproject.org/en-US/quick-docs/howto-file-a-bug/

Cleanup of tracking bugs

Review all the tracking bugs for Changes and close those which are already implemented and delivered. Tracking bugs of delivered Changes to be closed as CURRENTRELEASE resolution.

Wiki and website edits

EOL Closure reminder

The EOL closure reminder is posted to all bugs for Fedora N-2 a day or two after the GA date of a new release. Select all bugs for Fedora version N-2 that are not closed and save the results as a CSV file.

Run the fedora_eol_warning.pl script against it. For example, to warn that Fedora 29 will be EOL on 19 November 2019:

fedora_eol_warning.pl 29 2019-11-19 bugs-2019-10-22.csv

This script will take a while to run. Execute it from a machine with stable power and networking.
By default, the script will check to see that the version of the bug matches the EOL version you specfify. You will need to add the version field to your bugzilla query for this to work. If for some reason you like to live dangerously, you can set CHECK_VERSION = 0 in the script.

EOL day

Bug closure

All open bugs are closed on the EOL date. Select all bugs for Fedora version N-2 that are not closed and save the results as a CSV file.

Run the fedora_close_eol.pl script against it. For example, to close Fedora 29 bugs with an EOL date of 19 November 2019:

fedora_close_eol.pl 29 2019-11-19 bugs-2019-10-22.csv

This script will take a while to run. Execute it from a machine with stable power and networking.
By default, the script will check to see that the version of the bug matches the EOL version you specfify. You will need to add the version field to your bugzilla query for this to work. If for some reason you like to live dangerously, you can set CHECK_VERSION = 0 in the script.

Update product description

Adjust wording of the Fedora product in Bugzilla to reflect the EOL.

Bugs related to the components of the Fedora distribution. If you are reporting a bug
against a stable release or a branched pre-release version please select that
version number. The currently maintained released versions are: Fedora N-1, Fedora N.
If you have a bug to report against the daily development tree (rawhide) please choose
'rawhide' as the version.

For more information about filing a bug against Fedora packages, see
https://docs.fedoraproject.org/en-US/quick-docs/howto-file-a-bug/

Disable EOL version

Wiki and website edits