COVID-19 risk assessment

The primary concern of the Fedora leadership remains the health and safety of the Fedora community. However, we are tracking the risk to Fedora release deliverables as the situation evolves. This document is maintained by the Fedora Program Manager (Ben Cotton) with the Fedora Project Leader (Matthew Miller) as a backup.

Status

Table 1. Project status
Project Status

Fedora 32

On Track

Data center move

At Risk

Fedora 33

Not tracking yet

Changelog

  • 2020-03-20 - bcotton - Initial version

Points of contact

Table 2. Points of contact
Area Name Email Location/time zone

Program Management

Ben Cotton

bcotton AT redhat DOT com

Indiana, US (Eastern)

Fedora Project Leader

Matthew Miller

mattdm AT redhat DOT com

Massachusetts, US (Eastern)

Infrastructure

Kevin Fenzi

kfenzi AT redhat DOT com

Oregon, US (Pacific)

Release Engineering

Mohan Boddu

mboddu AT redhat DOT com

Massachusetts, US (Eastern)

Data center move

Stephen Smoogen

ssmoogen AT redhat DOT com

North Carolina, US (Eastern)

QA

Adam Williamson

awilliam AT redhat DOT com

British Columbia, CA (Pacific)

Fedora 32 release

Key Dates:

  • 2020-04-07 - Final freeze begins

  • 2020-04-21 - Fedora 32 final release preferred target date

  • 2020-04-28 - Fedora 32 final release target date #1

Table 3. Fedora 32 risks
Risk area Impact Likelihood Mitigation & Notes

Release engineering staff

High

Low

Release Engineering knowledge is duplicated across multiple people with large geographic separation

Infrastucture staff

High

Low

Infrastructure knowledge is duplicated across multiple people with large geographic separation

QA test staff

High

Low

Testing staff (both Red Hat and non-Red Hat contributors) are widely distributed across the globe

openQA test platform

Medium

Medium

Adam Williamson is the primary admin for openQA. However, other members of the team have knowledge and could keep the platform going if Adam is unavailable.

QA test hardware

Medium

High

Some QA tests require physical access to hardware located in Red Hat offices (see table below). Some of these can be done with contributor’s personally-owned hardware and others may be skipped if Red Hat offices remain closed through the release.

Test cases at risk

The table below represents test cases identified by the QA team which may be impacted by lack of access to Red Hat offices. Where “use contributor-owned hardware” is listed in the mitigation section, someone has volunteered to run those tests. For others, we will put out public calls for testing and may decide to exclude those tests for this release if office shutdown continues.

Table 4. Fedora 32 at-risk test cases
Test case Mitigation

Testcase_Boot_default_install

Use contributor-owned hardware. May require a longer period between release candidate compose and go/no-go decision

Testcase_Anaconda_User_Interface_Basic_Video_Driver

Use contributor-owned hardware.

Testcase_audio_basic

Testcase_desktop_menus

Testcase_desktop_automount

Testcase_Printing_New_Printer

Use contributor-owned hardware.

Data center move

Key dates:

  • 2020-04-13 - Move 1: Essential hardware in IAD2 datacenter

  • 2020-04-13 - Move 2: Communishift hardware to RDU-CC datacenter

  • 2020-05-01 - CPE management Go/No-Go decision for move

  • 2020-05-11 - Move 3: QA equipment to IAD2 datacenter

  • 2020-05-20–2020-07-01: Reduced build capacity

  • 2020-06-01 - Move 4: Remaining hardware to IAD2 datacenter

  • 2020-06-15 - Hard deadline for hardware to be out of PHX2

  • 2020-07-22 - Mass rebuild begins

Table 5. Data center move risks
Risk area Impact Likelihood Mitigation & Notes

Delayed delivery of Move 1 hardware to IAD2

High

Medium

Some delays have happened with delivery of new hardware. Team is exploring mitigation with alternative shelving and zip ties. CPE team working with Red Hat IT and vendors on capacity plans. If it is delayed to Move 3 or 4, we will not have build infrastructure during the early part of the Fedora development cycle.

Delayed delivery of Move 3 hardware to IAD2 after leaving PHX2

High

Medium

QA infrastructure remains offline. Impact is lessened unless the outage continues into late July. Otherwise, it may impact Fedora 33 schedule.

Delayed delivery of Move 4 hardware to IAD2 after leaving PHX2

Medium

Medium

Reduced build capacity remains in place longer than expected. This may end up impacting the Fedora 33 mass rebuild window.

Datacenter staff availability

High

Medium

Datacenters have implemented quarantine policies to mitigate the risk of staff infection. If staff become unavailable during racking/de-racking times, schedule will be delayed.

Infrastructure staff availability

High

Medium

1.5 sysadmins are assigned to perform remote work. They are following the guidance of Red Hat and local health officials. If their capacity is reduced, we can try to find additional help, but reinforcements may lack the requisite knowledge.

Hardware remains in PHX2 after 15 June

Medium

Medium

Hardware that remains in service will be removed by datacenter staff and stored for later shipment. Fedora will continue with reduced build capacity beyond the scheduled window, potentially impacting the mass rebuild.