End Of Life

Description

Each release of Fedora is maintained as laid out in the maintenance schedule. At the conclusion of the maintenance period, a Fedora release enters end of life status. This procedure describes the tasks necessary to move a release to that status. To get the correct EOL date visit release schedule for the last release.

Actions

Reminder announcement

Around a week before the EOL date, end an email to devel@, devel-announce@, test@, test-announce@, announce@ lists as remainder about the release EOL. Use the template from release engineering repo.

Please ensure that the reminder email is scheduled to be sent one week before the end of life.

fedora-appstream-metadata package update

If necessary, update the fedora-appstream-metadata package with the new EOL date. Follow the instructions from the repo README.

Build the package, create an update and make sure it lands in the stable repo before proceeding.

This is not necessary if the package already has the correct EOL date.

Koji tasks

  • Disable builds by removing targets

koji remove-target f39
koji remove-target f39-candidate
koji remove-target f39-container-candidate
koji remove-target f39-flatpak-candidate
koji remove-target f39-infra
koji remove-target f39-coreos-continuous
koji remove-target f39-rebuild

  • Remove sidetags

Remove any existing sidetags from the eol release.

for i in `koji list-sidetags --basetag f39`
do
	koji remove-sidetag $i
done

You need to do this differently because just removing the target of a sidetag will leave it in a weird state where it cannot be removed.

  • Purge from disk the signed copies of rpms that are signed with the EOL’d release key. To achieve this, add the release key to koji_cleanup_signed.py script in releng repo and run the script on compose-branched01.iad2.fedoraproject.org:

tmux
sudo su
cd /root/releng
./scripts/koji_cleanup_signed.py

Running this script in tmux or screen is recommended as it takes a long time to run.

Final stable push

Do a final stable push for the release in Bodhi, following the Pushing Updates SOP.

Bodhi tasks

Run the following bodhi commands to set the releases state to archived

bodhi releases edit --name "F39" --state archived
bodhi releases edit --name "F39C" --state archived
bodhi releases edit --name "F39F" --state archived

Fedora Infra Ansible Changes

We need to make some changes and then run several roles in the ansible repo.

Update FedoraPreviousPrevious.yaml in ansible repository

Edit /vars/all/FedoraPreviousPrevious.yaml and set the value to False.

Adjust the EOL for gnome-software

Edit /roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json in ansible and set the release to status "EOL".

Adjust the greenwave gating policy

Edit /roles/openshift-apps/greenwave/templates/fedora.yaml and find the policies with _upgrade_ in their name, near the bottom. You should see a comment noting these are not run on "the oldest stable release". The oldest release listed in the product_versions section for each of these policies should be one higher than the release you are EOLing. Remove the lines for these releases, so the oldest release listed is now two higher than the release you are EOLing. You may also remove all lines for the release you are EOLing, but this is not critical.

Run the playbooks

First, push the above edits. Then run the associated playbooks on batcave01:

sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/greenwave.yml

Another way to run the playbook is using rbac-playbook, in case you don’t have sysadmin-main rights or can’t become root. Syntax: sudo rbac-playbook groups/bodhi-backend.yml

Final announcement

Send the final announcement to devel@, devel-announce@, test@, test-announce@, announce@ lists. Use the template from release engineering repo.

Other tasks

Remove EOL release from Koschei

The release going EOL must be removed from Koschei to prevent a flood of failed scratch builds. See the Koschei sysadmin guide for details.

Remove release from fedfind source of truth

The Quality team maintains a source of truth for release information used by the fedfind tool on fedorapeople (people01) in /srv/groups/qa/metadata/release.json. If you have access to this file, you can remove the release going EOL from the file’s list of stable releases. Otherwise, alert the quality team to do so.

Stage the EOL release to archive

  1. Log into to bodhi-backend01 and become ftpsync, via root

    ssh bodhi-backend01.iad2.fedoraproject.org
sudo su
su - ftpsync

  2. Then change into the releases directory.

    cd /pub/fedora/linux/releases

  3. Check to see that the target directory doesn’t already exist.

    ls /pub/archive/fedora/linux/releases/

  4. Do a recursive rsync to update any changes in the trees since the previous copy.

    rsync -avAXSHP ./39/ /pub/archive/fedora/linux/releases/39/

  5. Similarly, sync updates and updates/testing.

    cd ../updates/
rsync -avAXSHP 39/ /pub/archive/fedora/linux/updates/39/
cd testing
rsync -avAXSHP 39/ /pub/archive/fedora/linux/updates/testing/39/

  6. Do the same with fedora-secondary.

    cd /pub/fedora-secondary/releases/
rsync -avAXSHP ./39/ /pub/archive/fedora-secondary/releases/39/
cd ../updates/
rsync -avAXSHP 39/ /pub/archive/fedora-secondary/updates/39/
cd testing
rsync -avAXSHP 39/ /pub/archive/fedora-secondary/updates/testing/39/

  7. Announce to the mirror list this has been done and that in 2 weeks you will move the old trees to archives.

Move the EOL release to archive

  1. In two weeks, run the following playbook from batcave:

sudo ansible-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='Fedora' version='39'"

  1. If there are problems, the postgres DB may have issues and so you need to get a DBA to update the backend to fix items.

  2. Wait an hour or so then you can remove the files from the main tree.

We need to do a cleanup here for both fedora and fedora-secondary for releases, updates, and updates/testing for the 39 release.

 
ssh bodhi-backend01
cd /pub/fedora/linux/releases/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora/linux/updates/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora/linux/updates/testing/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .

#for fedora-secondary
cd /pub/fedora-secondary/releases/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora-secondary/updates/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora-secondary/updates/testing/39
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .

Consider Before Running

  • Resource contention in infrastructure, such as outages

  • Extenuating circumstances for specific planned updates, if any

  • Send the reminder announcement, if it isn’t sent already

Want to help? Learn how to contribute to Fedora Docs ›