Test Management Tool
Summary
The tmt
tool aims to provide an efficient and comfortable way to create, execute, debug and enable tests in the Continuous Integration.
It implements the Test Metadata Specification which allows to store all needed test execution data directly within a git repository. The same configuration can be used for enabling tests in the Fedora CI, RHEL CI and Packit. Tests can be easily executed in your preferred environment, e.g. in virtual machine, container or directly on the localhost.
Premiers pas
Install
Install tmt on your laptop:
sudo dnf install -y tmt # basic features, executing tests on localhost sudo dnf install -y tmt+all # install all available tmt subpackages including all dependencies
You can also install selected provision plugins only:
sudo dnf install -y tmt+provision-container # additional dependencies for executing tests in containers sudo dnf install -y tmt+provision-virtual # support for running tests in a virtual machine using testcloud
See the tmt install section for more installation options.
Git Repo
Check out the desired dist git branch using fedpkg:
fedpkg clone -a bash cd bash git checkout f32
Or clone your GitHub project repository:
git clone https://github.com/teemtee/tmt/ cd tmt git checkout -b enable-tests
Smoke Test
Let’s enable a simple smoke test using the minimal plan template:
$ tmt init --template mini Tree '/tmp/bash' initialized. Applying template 'mini'. Directory '/tmp/bash/plans' created. Plan '/tmp/bash/plans/example.fmf' created.
Edit the newly created plan as needed, for example like this:
summary: Basic smoke test for bash execute: script: bash --version
Execute Tests
Run Tests
Execute all available tests safely in a virtual machine:
tmt run
Run only tests matching given name or located under the current directory:
tmt run test --name smoke tmt run test --name .
Show detailed test results from the latest tmt run executed by current user:
tmt run --last report -fvvv
Executing tests enabled using the Standard Test Interface in tests/tests.yml is not supported yet but we are working on it. |
Select Steps
Explicitly choose which steps should be run:
tmt run discover
This will provide an overview of tests which would be run. To list individual tests enable the verbose mode:
tmt run discover --verbose tmt run discover -v
Provision Options
Choose local
as the provision method but run --all
steps:
tmt run --all provision --how local
Execute inside a container or virtual machine:
tmt run --all provision --how container --image fedora tmt run --all provision --how virtual --image fedora-32
Check all available provision plugins:
tmt run provision --help
Prepare Options
Install additional packages on the guest:
tmt run --all prepare --how install --package httpd
Get the latest package from provided copr repository:
tmt run --all prepare --how install --copr @teemtee/tmt --package tmt
Use the freshly build local rpm or all rpms from provided local directory:
tmt run --all prepare --how install --package tmp/RPMS/noarch/tmt-0.20-1.fc32.noarch.rpm tmt run --all prepare --how install --directory tmp/RPMS/noarch
Check all available prepare options:
tmt run prepare --help
Create Test
In order to create more complex tests let’s use the base plan template:
tmt plan create /plans/basic --template base tmt plan create /plans/basic -t base
Update summary as needed, keep discover method to fmf
and choose whether tests should be executed as shell
scripts (just check the exit code) or beakerlib
tests (investigate journal for test results):
summary: Check basic bash features discover: how: fmf execute: how: tmt
Shell Test
In order to create a simple shell test skeleton use the shell template:
$ tmt test create /tests/smoke Template (shell or beakerlib): shell Directory '/tmp/bash/tests/smoke' created. Test metadata '/tmp/bash/tests/smoke/main.fmf' created. Test script '/tmp/bash/tests/smoke/test.sh' created.
Update metadata file:
summary: Check bash version contact: Petr Šplíchal <psplicha@redhat.com> test: ./test.sh
Adjust the test script as desired:
#!/bin/sh -eux tmp=$(mktemp) bash --version > $tmp grep 'GNU bash' $tmp grep 'Free Software Foundation' $tmp rm $tmp
Use tmt run
to verify the test is working as expected.
BeakerLib Test
Use beakerlib template to create a new beakerlib test:
$ tmt test create /tests/smoke -t beakerlib Directory '/tmp/bash/tests/smoke' created. Test metadata '/tmp/bash/tests/smoke/main.fmf' created. Test script '/tmp/bash/tests/smoke/test.sh' created.
Update test metadata and code as needed, use tmt run
to verify everything is working fine.
Pull Requests
When creating the pull request make sure you add all created files including the special .fmf
directory.
git add . git commit
Fedora
In order to test your changes in Fedora CI no additional configuration is needed. Make sure you push the changes into your forked repository as fedora rpms/tests namespace does not allow force-push or branch removal.
git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/tmt.git git push fork -u enable-tests
GitHub
In order to test a pull request on GitHub enable the Packit-as-a-Service integration and add a .packit.yaml
configuration file:
jobs: - job: tests trigger: pull_request metadata: targets: - fedora-all
For more details see the Testing Farm documentation. Once the integration is enabled push the branch, create a new pull request as ususal and wait for results:
git push origin -u enable-tests
Templates
When creating a pull request to enable tests in a repository with no tmt
configuration, include a couple of hints and links for those who are not familiar with the new tooling:
This pull request enables tests in the Fedora CI using `tmt` which also allows to easily execute and debug tests from your laptop: Run tests directly on your localhost: sudo dnf install -y tmt tmt run --all provision --how local Run tests in a virtual machine: sudo dnf install -y tmt+provision-virtual tmt run Check the documentation to learn more about the tool: https://docs.fedoraproject.org/en-US/ci/tmt/
Manage Tests
Explore available tests, convert old metadata, share test code.
Explore Tests
In order to see which tests are available:
tmt test ls
To show more details about individual tests:
tmt test show
To see an overview of all metadata:
tmt
Explore all available options and commands using --help
.
Share Tests
Test code does not have to reside in the same git repository (e.g. dist git rpms namespace). It is possible to store tests in a dedicated repository and share them across components or product versions. You only need to reference the repository in the discover step. Use the full plan template to get quickly started:
tmt plan create /plans/upstream -t full
Update the repository url to point to the right place:
summary: Essential command line features discover: how: fmf url: https://github.com/teemtee/tmt execute: how: tmt
Now you will be able to run tests from the remote repository. See the discover step documentation for details.
Various Hints
Multiple Commands
Multiple shell commands can be provided under the script
attribute as well:
summary: Basic smoke test for bash execute: script: - bash --version - bash -c 'echo $((1+1+1))' | grep 3
See the script method documentation for details.
Installing Dependencies
Required packages can be installed using the prepare
attribute:
summary: Basic smoke test for python3-m2crypto prepare: how: install package: - python3-setuptools - python3-m2crypto execute: script: python3 -c "import M2Crypto"
See the prepare step documentation for details.
Multiple Repositories
In the discover step it is possible to reference multiple repositories as well. In this way you can for example easily execute both upstream and fedora tests as part of a single plan:
discover: - name: fedora how: fmf url: https://src.fedoraproject.org/tests/selinux.git - name: upstream how: fmf url: https://github.com/SELinuxProject/selinux-testsuite
See also multiple config example in tmt repo to get a better idea.
Multiple Plans
It is possible to use multiple plans to group relevant tests together or to be able to easily run a subset of tests. For example, let’s have a /plans/features
plan which covers all functionality tests from the local git repository:
discover: how: fmf execute: how: tmt
And a separate /plans/integration
plan to enable integration testing with another component:
discover: how: fmf url: https://src.fedoraproject.org/rpms/ltrace.git execute: how: tmt
Running all tests from given plan is then very easy:
tmt run plan --name /plans/features
When run in the CI, results from such plans are reported as a single resultsdb testcase and are shown in pull requests as a single flag. In order to enable separate result for each plan, create a ci.fmf
file in the git repository root with the following content:
resultsdb-testcase: separate
Once the separate reporting is enabled, you can turn on gating for selected plans only. Plan name becomes part of the resultsdb testcase name which is used in the gating.yaml
config. See the gating documentation on Using Multiple Plans for more details.
Minimal Path
Here is an example of a minimal test creation path:
dnf install -y tmt+all git clone https://src.fedoraproject.org/rpms/bash cd bash tmt init -t mini vim plans/example.fmf tmt run
A slightly extended example with custom test and plan template and executing test directly on the local host:
dnf install -y tmt+all git clone https://src.fedoraproject.org/rpms/bash cd bash tmt init tmt plan create --template base plans/smoke tmt test create --template beakerlib tests/smoke vim plans/smoke.fmf tests/smoke/* tmt run --all provision -h local git add . git commit -m "Enable basic tests" git push
Virtualization Tips
In order to safely run tests under a virtual machine started on your laptop you only need to install the tmt+provision-virtual
package. By default the session connection is used so no other steps should be needed, just execute tests using the tmt run
command. See the upstream Virtualization Tips for more options.
More Info
Questions
- Does the tool replace/deprecate STI?
-
No, currently there is no plan to decommission STI. Both
tmt
andsti
approach to CI configuration can be used in parallel. - Are these tests supported in Fedora CI?
-
Yes, Fedora CI support is enabled for all active branches and the tests namespace as well.
- Which Linux distributions does the tool support?
-
As a system under test (on which the tests are executed) all supported Fedora versions, Centos 6+ and Red Hat Enterprise Linux 6+ can be used. For the test runner (where tmt command is run) all supported Fedora versions, Centos 8+ or Red Hat Enterprise Linux 8+ are required.
Want to help? Learn how to contribute to Fedora Docs ›