How to add simple STI test for a package

Test as a single command

Task

There is a package somepackage, which contains a binary: /usr/bin/somebinary

The most simple and obvious way to test this binary is to run it: somebinary --help and check the exit status of the result.

How to add this test for the package?

Standard Test Roles framework provides solution for the task. It is fully supported in Fedora Rawhide.

Solution

Create file tests/tests.yml in the dist-git of the package:

rpms/somepackage.git:

.
├── 0001-something.patch
├── somepackage.spec
├── sources
└── tests
    └── tests.yml

tests.yml is an ansible playbook, where you describe test environment and steps to run your tests.

There are many options and examples available, but let’s focus on the task at hand: we want to run just one command.

For this case the content of the file should look as follows:

rpms/somepackage.git:tests/tests.yml

- hosts: localhost
  roles:
  - role: standard-test-basic                         (1)
    tags:
    - classic
    tests:
    - simple:
        dir: .
        run: "somebinary --help"                      (2)

1	this is a standard test role, it takes care of the test environment, logging, archiving results, etc
2	this is your test command, its exit code defines the outcome of the test

Submit your change as a pull request to see test results in Pagure interface, or push it directly to dist-git and get new test results every time you build package in Koji.

Test will be running in a non-blocking mode until you configure gating for it.

Tests in a (sub)package

Task

There is a package somepackage, and there is an integration test suite for it packaged in a separate sometests package, which provides the run_some_tests binary in the system path.

There goal is to trigger this binary as a test for a main package.

Solution

Same as above, you need to create a tests.yml configuration file with the following content:

rpms/somepackage.git:tests/tests.yml

---
- hosts: localhost
  roles:
    - role: standard-test-basic
      tags:
        - classic
      required_packages:
        - sometests                    (1)

      tests:
        - integration_tests:            (2)
            dir: .
            run: run_some_tests         (3)

1	additional package which needs to be installed in the test environment
2	any string, will be used as identifier for artifacts and test results
3	test execution command

Tests in the source tarball

Tests in external repository

Task

Let’s look into slightly more complicated setup now.

Suppose there is a package somepackage which we are going to test. There is an integration test suite for it, which is (sadly) not yet packaged and located in a separate git repository https://somewhere/sometests.git. Test suite has a dependency on some packaged tool sometool. And in test repository there is a run_some_tests script which triggers test execution.

There goal is to trigger the execution of the test suite for a package.

Solution

We need to create a tests.yml configuration file with the following content:

rpms/somepackage.git:tests/tests.yml

---
- hosts: localhost
  roles:
    - role: standard-test-basic                        (1)
      tags:
        - classic

      required_packages:
        - sometool                                     (2)

      repositories:
        - repo: "https://somewhere/sometests.git"      (3)
          dest: "sometests"                            (4)

      tests:
        - integration_tests:                           (5)
            dir: "sometests"                           (6)
            run: "run_some_tests --all"                (7)

1	same basic test role as usual
2	additional package which needs to be installed in the test environment
3	path to remote git repository
4	local path where the repository will be checked out
5	any string, will be used as identifier for artifacts and test results
6	same folder as in `<4>`, contains the checked out external repository
7	test execution command

Questions

What if I want to run not one but a sequence of commands?

Put a bash script in tests/scripts/ folder and run it from the playbook.

rpms/somepackage.git:

.
├── 0001-something.patch
├── somepackage.spec
├── sources
└── tests
    ├── scripts
    │   └── run_tests.sh      (1)
    └── tests.yml

1	your custom test scenario

Configure dist-git test to run this script:

- hosts: localhost
  roles:
  - role: standard-test-basic      (1)
    tags:
    - classic
    tests:
    - simple:
        dir: scripts               (2)
        run: ./run_tests.sh        (3)

1	same standard role
2	switch to subfolder (path is relative to `tests/` folder)
3	this is the test script, its exit code is the outcome of the test

What is under the hood?

To test the build we:

checkout dist-git repo
take latest qcow image of Fedora Rawhide
install all packages from the koji build on it
run ansible playbook defined in tests.yml

How do I verify my configuration?

It is possible to run and debug standard test roles locally. But we highly recommend to use the pull-request workflow for it: simply create a pull-request and wait for CI to react on it.

We trigger almost the same CI machinery for PR testing as it is used for gating of new builds.

And as soon as result of the test is ready, it will appear on the pull request page in Fedora Pagure

To restart the test add a comment to PR in Pagure, with the following content: [citest]

Want to help? Learn how to contribute to Fedora Docs ›