Fedora CI

In order to clearly distinguish test from the CI system running it the Standard Test Interface was introduced. It clearly defines essential terms such as test, test subject, test suite, test framework, test result, test artifact, test system and describes what are their responsibilities and requirements.

This general approach gives a nice flexibility as it does not enforce any specific tools or frameworks to be used. Basically it only describes how tests are discovered and where the testing results should be stored to be processed by the automation.

When a test fails, CI can prevent the broken change from affecting other packages. That gating happens in Bodhi. Greenwave is used together with ResultsDB and WaiverDB to make the gating decision.

Fedora Notifications have been adjusted to notify by default every packager when any step of the CI pipeline fails on one of the package they maintain. So if you are a kernel maintainer and a commit made to the kernel dist-git repository fails to compose an OSTree, FMN will notify you of it.

Bodhi includes the CI results in its update page, just as it already includes tests results from taskotron.

Various tools involved in the CI automation are sending updates about the progress of the testing to the message bus. Consistent format of the CI messages allows to build simpler services which will provide useful features for everyone.

Standard Test Roles were implemented to enable both automation tools and developers in their local environments to easily execute tests. This set of ansible roles supports various frameworks and allows to execute tests against different test subjects (such as classic rpm package, docker container or Atomic Host).

As Standard Test Interface does not define test framework which should be used for testing it is possible to choose the most suitable framework for your project. Here are some examples:

Standard Test Interface defines only a very simple metadata for selecting which tests should be run. For more complex scenarios Flexible Metadata Format can be used:

The testing Pipeline detects tests for enabled packages, executes the test coverage and gathers the results.

Test results from the CI pipeline are displayed in Pagure web interface. See the commits page and pull request pages of respective package to see the results. Pagure integration with COPR provides auto-rebuilding and pull request/commit flagging on new changes.

In general it makes sense to store tests as close to the upstream as possible. So what are the appropriate test types recommended for testing the Always Ready Operating System?

For unit tests it usually makes more sense to store them directly within the upstream project repository. However, in some cases it might be worth to fetch tests for Fedora CI from the upstream repository as well.

Tests may be written all sorts of different ways, but have to be exposed and invoked in a standard way. Tests are enabled by including the tests/tests.yml file in the package dist-git repository as defined by the Standard Test Interface. Test code itself can be stored directly in the dist-git or fetched from another repository. Shared tests namespace can be used for storing test code relevant for multiple packages.

Executing a test written with the use of Standard Test Roles is as simple as running an ansible playbook

However, a set of environment variables needs to be set properly in order to execute the test against the desired test subject. The Tests wiki contains detailed instructions about running tests and adding new test coverage.

There are statistics pages to track how many Fedora dist-git repositories have tests in the Standard Test Format. These pages are updated automatically.

There is an active effort to open source existing internal Red Hat tests to Fedora called Upstream First. Tests being migrated can be found in the upstream first repository:

The ownership and maintenance of tests should be shared between QE & Devel. For tests that reside in the rpm namespace, QE can use pull requests to create/update tests. Likewise in the tests namespace, both QE and Devel will have commit rights, both QE and Devel should review and sign-off with each commit.