Test Management Tool
tmt tool aims to provide an efficient and comfortable way to create, execute, debug and enable tests in the Continuous Integration.
It implements the Test Metadata Specification which allows to store all needed test execution data directly within a git repository. The same configuration can be used for enabling tests in the Fedora CI, RHEL CI and Packit. Tests can be easily executed in your preferred environment, e.g. in virtual machine, container or directly on the localhost.
Install tmt on your laptop:
sudo dnf install -y tmt # basic features, executing tests on localhost sudo dnf install -y tmt-all # install all available tmt subpackages including all dependencies
You can also install selected provision plugins only:
sudo dnf install -y tmt-provision-container # additional dependencies for executing tests in containers sudo dnf install -y tmt-provision-virtual # support for running tests in a virtual machine using testcloud
See the tmt install section for more installation options.
Check out the desired dist git branch using fedpkg:
fedpkg clone -a bash cd bash git checkout f32
Or clone your GitHub project repository:
git clone https://github.com/psss/tmt/ cd tmt git checkout -b enable-tests
Let’s enable a simple smoke test using the minimal plan template:
$ tmt init --template mini Tree '/tmp/bash' initialized. Applying template 'mini'. Directory '/tmp/bash/plans' created. Plan '/tmp/bash/plans/example.fmf' created.
Edit the newly created plan as needed, for example like this:
summary: Basic smoke test for bash execute: script: bash --version
Execute all available tests safely in a virtual machine:
Run only tests matching given name or located under the current directory:
tmt run test --name smoke tmt run test --name .
Show detailed test results from the latest tmt run executed by current user:
tmt run --last report -fvvv
Executing tests enabled using the Standard Test Interface in tests/tests.yml is not supported yet but we are working on it.
Explicitly choose which steps should be run:
tmt run discover
This will provide an overview of tests which would be run. To list individual tests enable the verbose mode:
tmt run discover --verbose tmt run discover -v
local as the provision method but run
tmt run --all provision --how local
Execute inside a container or virtual machine:
tmt run --all provision --how container --image fedora tmt run --all provision --how virtual --image fedora-32
Check all available provision plugins:
tmt run provision --help
Install additional packages on the guest:
tmt run --all prepare --how install --package httpd
Get the latest package from provided copr repository:
tmt run --all prepare --how install --copr psss/tmt --package tmt
Use the freshly build local rpm or all rpms from provided local directory:
tmt run --all prepare --how install --package tmp/RPMS/noarch/tmt-0.20-1.fc32.noarch.rpm tmt run --all prepare --how install --directory tmp/RPMS/noarch
Check all available prepare options:
tmt run prepare --help
In order to create more complex tests let’s use the base plan template:
tmt plan create /plans/basic --template base tmt plan create /plans/basic -t base
Update summary as needed, keep discover method to
fmf and choose whether tests should be executed as
shell scripts (just check the exit code) or
beakerlib tests (investigate journal for test results):
summary: Check basic bash features discover: how: fmf execute: how: tmt
In order to create a simple shell test skeleton use the shell template:
$ tmt test create /tests/smoke Template (shell or beakerlib): shell Directory '/tmp/bash/tests/smoke' created. Test metadata '/tmp/bash/tests/smoke/main.fmf' created. Test script '/tmp/bash/tests/smoke/test.sh' created.
Update metadata file:
summary: Check bash version contact: Petr Šplíchal <firstname.lastname@example.org> test: ./test.sh
Adjust the test script as desired:
#!/bin/sh -eux tmp=$(mktemp) bash --version > $tmp grep 'GNU bash' $tmp grep 'Free Software Foundation' $tmp rm $tmp
tmt run to verify the test is working as expected.
Use beakerlib template to create a new beakerlib test:
$ tmt test create /tests/smoke -t beakerlib Directory '/tmp/bash/tests/smoke' created. Test metadata '/tmp/bash/tests/smoke/main.fmf' created. Test script '/tmp/bash/tests/smoke/test.sh' created.
Update test metadata and code as needed, use
tmt run to verify everything is working fine.
When creating the pull request make sure you add all created files including the special
git add . git commit
In order to test your changes in Fedora CI no additional configuration is needed. Make sure you push the changes into your forked repository as fedora rpms/tests namespace does not allow force-push or branch removal.
git remote add fork ssh://email@example.com/forks/psss/rpms/tmt.git git push fork -u enable-tests
In order to test a pull request on GitHub enable the Packit-as-a-Service integration and add a
.packit.yaml configuration file:
jobs: - job: tests trigger: pull_request metadata: targets: - fedora-all
For more details see the Testing Farm documentation. Once the integration is enabled push the branch, create a new pull request as ususal and wait for results:
git push origin -u enable-tests
When creating a pull request to enable tests in a repository with no
include a couple of hints and links for those who are not familiar with the new tooling:
This pull request enables tests in the Fedora CI using `tmt` which also allows to easily execute and debug tests from your laptop: Run tests directly on your localhost: sudo dnf install -y tmt tmt run --all provision --how local Run tests in a virtual machine: sudo dnf install -y tmt-provision-virtual tmt run Check the documentation to learn more about the tool: https://docs.fedoraproject.org/en-US/ci/tmt/
Explore available tests, convert old metadata, share test code.
In order to see which tests are available:
tmt test ls
To show more details about individual tests:
tmt test show
To see an overview of all metadata:
Explore all available options and commands using
Test code does not have to reside in the same git repository (e.g. dist git rpms namespace). It is possible to store tests in a dedicated repository and share them across components or product versions. You only need to reference the repository in the discover step. Use the full plan template to get quickly started:
tmt plan create /plans/upstream -t full
Update the repository url to point to the right place:
summary: Essential command line features discover: how: fmf url: https://github.com/psss/tmt execute: how: tmt
Now you will be able to run tests from the remote repository. See the discover step documentation for details.
Multiple shell commands can be provided under the
script attribute as well:
summary: Basic smoke test for bash execute: script: - bash --version - bash -c 'echo $((1+1+1))' | grep 3
See the script method documentation for details.
In the discover step it is possible to reference multiple repositories as well. In this way you can for example easily execute both upstream and fedora tests as part of a single plan:
discover: - name: fedora how: fmf url: https://src.fedoraproject.org/tests/selinux.git - name: upstream how: fmf url: https://github.com/SELinuxProject/selinux-testsuite
See also multiple config example in tmt repo to get a better idea.
It is possible to use multiple plans to group relevant tests together or to be able to easily run a subset of tests.
For example, let’s have a
/plans/features plan which covers all functionality tests from the local git repository:
discover: how: fmf execute: how: tmt
And a separate
/plans/integration plan to enable integration testing with another component:
discover: how: fmf url: https://src.fedoraproject.org/rpms/ltrace.git execute: how: tmt
Running all tests from given plan is then very easy:
tmt run plan --name /plans/features
When run in the CI, results from such plans are reported as a single resultsdb testcase and are shown in pull requests as a single flag.
In order to enable separate result for each plan, create a
ci.fmf file in the git repository root with the following content:
Once the separate reporting is enabled, you can turn on gating for selected plans only.
Plan name becomes part of the resultsdb testcase name which is used in the
See the gating documentation on Using Multiple Plans for more details.
Here is an example of a minimal test creation path:
dnf install -y tmt-all git clone https://src.fedoraproject.org/rpms/bash cd bash tmt init -t mini vim plans/example.fmf tmt run
A slightly extended example with custom test and plan template and executing test directly on the local host:
dnf install -y tmt-all git clone https://src.fedoraproject.org/rpms/bash cd bash tmt init tmt plan create --template base plans/smoke tmt test create --template beakerlib tests/smoke vim plans/smoke.fmf tests/smoke/* tmt run --all provision -h local git add . git commit -m "Enable basic tests" git push
In order to safely run tests under a virtual machine started on your laptop you only need to install the
By default the session connection is used so no other steps should be needed, just execute tests using the
tmt run command.
See the upstream Virtualization Tips for more options.
Example projects with tmt tests:
See the tmt examples page for more inspiration.
- Does the tool replace/deprecate STI?
No, currently there is no plan to decommission STI. Both
stiapproach to CI configuration can be used in parallel.
- Are these tests supported in Fedora CI?
Yes, Fedora CI support is enabled for all active branches and the tests namespace as well.
- Which Linux distributions does the tool support?
As a system under test (on which the tests are executed) all supported Fedora versions, Centos 6+ and Red Hat Enterprise Linux 6+ can be used. For the test runner (where tmt command is run) all supported Fedora versions, Centos 8+ or Red Hat Enterprise Linux 8+ are required.