How to add simple STI test for a package
There is a package somepackage, which contains a binary: /usr/bin/somebinary
The most simple and obvious way to test this binary is to run it: somebinary --help and check the exit status of the result.
How to add this test for the package?
Standard Test Roles framework provides solution for the task. It is fully supported in Fedora Rawhide.
tests/tests.yml in the dist-git of the package:
. ├── 0001-something.patch ├── somepackage.spec ├── sources └── tests └── tests.yml
tests.yml is an ansible playbook, where you describe test environment and steps to run your tests.
There are many options and examples available, but let’s focus on the task at hand: we want to run just one command.
For this case the content of the file should look as follows:
- hosts: localhost roles: - role: standard-test-basic (1) tags: - classic tests: - simple: dir: . run: "somebinary --help" (2)
|1||this is a standard test role, it takes care of the test environment, logging, archiving results, etc|
|2||this is your test command, its exit code defines the outcome of the test|
Submit your change as a pull request to see test results in Pagure interface, or push it directly to dist-git and get new test results every time you build package in Koji.
|Test will be running in a non-blocking mode until you configure gating for it.|
There is a package
somepackage, and there is an integration test suite
for it packaged in a separate
sometests package, which provides the
run_some_tests binary in the system path.
There goal is to trigger this binary as a test for a main package.
Same as above, you need to create a
tests.yml configuration file with
the following content:
--- - hosts: localhost roles: - role: standard-test-basic tags: - classic required_packages: - sometests (1) tests: - integration_tests: (2) dir: . run: run_some_tests (3)
|1||additional package which needs to be installed in the test environment|
|2||any string, will be used as identifier for artifacts and test results|
|3||test execution command|
Let’s look into slightly more complicated setup now.
Suppose there is a package
somepackage which we are going to
test. There is an integration test suite for it, which is (sadly) not
yet packaged and located in a separate git repository
https://somewhere/sometests.git. Test suite has a dependency on some
sometool. And in test repository there is a
run_some_tests script which triggers test execution.
There goal is to trigger the execution of the test suite for a package.
We need to create a
tests.yml configuration file with the following
--- - hosts: localhost roles: - role: standard-test-basic (1) tags: - classic required_packages: - sometool (2) repositories: - repo: "https://somewhere/sometests.git" (3) dest: "sometests" (4) tests: - integration_tests: (5) dir: "sometests" (6) run: "run_some_tests --all" (7)
|1||same basic test role as usual|
|2||additional package which needs to be installed in the test environment|
|3||path to remote git repository|
|4||local path where the repository will be checked out|
|5||any string, will be used as identifier for artifacts and test results|
|6||same folder as in
|7||test execution command|
Put a bash script in tests/scripts/ folder and run it from the playbook.
. ├── 0001-something.patch ├── somepackage.spec ├── sources └── tests ├── scripts │ └── run_tests.sh (1) └── tests.yml
|1||your custom test scenario|
Configure dist-git test to run this script:
- hosts: localhost roles: - role: standard-test-basic (1) tags: - classic tests: - simple: dir: scripts (2) run: ./run_tests.sh (3)
|1||same standard role|
|2||switch to subfolder (path is relative to
|3||this is the test script, its exit code is the outcome of the test|
To test the build we:
checkout dist-git repo
take latest qcow image of Fedora Rawhide
install all packages from the koji build on it
run ansible playbook defined in tests.yml
It is possible to run and debug standard test roles locally. But we highly recommend to use the pull-request workflow for it: simply create a pull-request and wait for CI to react on it.
We trigger almost the same CI machinery for PR testing as it is used for gating of new builds.
And as soon as result of the test is ready, it will appear on the pull request page in Fedora Pagure
To restart the test add a comment to PR in Pagure, with the following