On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI.
For each component it makes sense to enable even the most simple test such as running the binary with
--help or using an internal smoke test. Here’s an example from the did component:
- hosts: localhost roles: - role: standard-test-basic tags: - classic tests: - smoke: dir: . run: did this year --test required_packages: - did
That’s it. As you see above, executing a single command as a test is very easy with the help of the Basic role.
There are multiple versions of Python programming language available in Fedora and a number of related subpackages. As all of them should be tested (including their various combinatios) we share test coverage for them in the
Once the test is avaible in the share test repository it can be easily linked from supported Python versions:
We test additional Python implementations as well:
Plus we ensure that essential tools for venv and virtualnv, such as
virtualenv itself correctly work with all supported versions:
Note that for the last set of examples we run the same test several times with modified environment. For example:
- smoke36: dir: python/smoke run: VERSION=3.6 ./venv.sh - smoke37: dir: python/smoke run: VERSION=3.7 ./venv.sh - smoke26: dir: python/smoke run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh - smoke27: dir: python/smoke run: VERSION=2.7 METHOD=virtualenv ./venv.sh - smoke34_virtualenv: dir: python/smoke run: VERSION=3.4 METHOD=virtualenv ./venv.sh
In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance.
There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). Thus we store test code in the
These tests are then linked from all relevant
Flexible Metadata Format filter is used to select appropriate tests instead of listing individual tests manually. Environment variables
SH_BIN are used to specify which shell implementation is being tested:
- hosts: localhost roles: - role: standard-test-beakerlib tags: - classic repositories: - repo: "https://src.fedoraproject.org/tests/shell.git" dest: "shell" fmf_filter: "tier: 1, 2 & tags: classic" environment: PACKAGES: ksh SH_BIN: ksh required_packages: - ksh - expect # login requires expect - which # smoke requires which
Some of the tests might be relevant only for selected components. This can be handled easily by additional
repositories: - repo: "https://src.fedoraproject.org/tests/shell.git" dest: "shell" fmf_filter: "tier: 1, 2 & component: dash"
See the Metadata page for the full list of so far drafted attributes.
There are several components related to SELinux. They are tightly connected so change in one of them can cause problems in other. That’s why their tests are shared and executed together:
Instead of listing relevant tests to be executed manually in each dist git rpms repository Flexible Metadata Format is used:
- hosts: localhost roles: - role: standard-test-beakerlib tags: - classic repositories: - repo: "https://src.fedoraproject.org/tests/selinux.git" dest: "selinux" fmf_filter: "tier: 1 | component: selinux-policy"
fmf_filter selects all tests relevant for the
selinux-policy component plus all Tier 1 selinux tests:
tier: 1 | component: selinux-policy
The following six components are covered:
fmf command line tool to quickly check which tests will be scheduled:
# dnf install -y fmf # fedpkg clone -a tests/selinux # cd selinux # fmf ls --filter "tier: 1 | component: checkpolicy" /selinux-policy/policy-rpm-macros /checkpolicy/sedispol /checkpolicy/checkmodule /checkpolicy/sedismod /checkpolicy/checkpolicy /checkpolicy/checkpolicy-docs /libsepol/sepol_check_context /libsemanage/verify-options-in-semanage-conf /libselinux/getsebool /policycoreutils/booleans
See the Flexible Metadata Format documentation for other options how to install fmf.