Share Test Code

Motivation

In order to make the CI workflow reliable and efficient it is crucial to keep the test coverage in a good shape at all times. Sharing test code between several packages (even within multiple branches of the same package) may significantly help to:

  • Prevent test code duplication

  • Minimize test maintenance

  • Catch incompatibilities early

In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.

See the whole ci-list discussion for some more context.

Implementation

Store test code in your preferred repository and reference the tests from the dist-git yaml file. There is also a special tests namespace dedicated for storing Fedora CI integration tests:

Use fedpkg to quickly clone repositories from the tests namespace:

fedpkg clone tests/shell

tmt

Enabling tests from a remote repository using tmt is straightforward:

discover:
    how: fmf
    url: https://src.fedoraproject.org/tests/shell.git

See the discover step documentation for more details.

Examples

Here are some real-life examples where sharing test code can increase long-term efficiency.

SELinux

Several SELinux user space components are sharing test coverage in a single selinux test repository:

Ruby

Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task.

Currently the shared tests/ruby repository hosts these three ruby integration tests:

  • systemtap-static-probes-in-ruby - exercising ruby’s systemtap api

  • tls-minimal-version - ensure Ruby OpenSSL respects crypto-policies

  • run-basic-rails-application - run a simple rails application

Shell

There are several shells which implement the POSIX specification:

  • bash

  • ksh

  • mksh

  • zsh

  • dash

All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).

Shell tests repository:

Bash plans/shell.fmf:

summary:
    Run relevant tests from the shell tests repository
discover:
    how: fmf
    url: https://src.fedoraproject.org/tests/shell
    filter: component:bash
execute:
    how: tmt
environment:
    PACKAGES: bash
    SH_BIN: bash

Start

In order to create a new repository in the tests namespace use the fedpkg’s request-tests-repo command. For example to create a shared test repository with the name foo, which will be available at https://src.fedoraproject.org/tests/foo.git

  • Setup authentication to pagure according to the help in request-repo command

    fedpkg request-repo -h

  • Request a new repository with a sensible decription

    fedpkg request-tests-repo foo "Description of the repository"

Want to help? Learn how to contribute to Fedora Docs ›