Test Management Tool

Summary

The tmt tool aims to provide an efficient and comfortable way to create, execute, debug and enable tests in the Continuous Integration.

It implements the Test Metadata Specification which allows to store all needed test execution data directly within a git repository. The same configuration can be used for enabling tests in the Fedora CI, RHEL CI and Packit. Tests can be easily executed in your preferred environment, e.g. in virtual machine, container or directly on the localhost.

First Steps

Install

Install tmt on your laptop:

sudo dnf install -y tmt       # basic features, executing tests on localhost
sudo dnf install -y tmt+all   # install all available tmt subpackages including all dependencies

You can also install selected provision plugins only:

sudo dnf install -y tmt+provision-container   # additional dependencies for executing tests in containers
sudo dnf install -y tmt+provision-virtual     # support for running tests in a virtual machine using testcloud

See the tmt install section for more installation options.

Git Repo

Check out the desired dist git branch using fedpkg:

fedpkg clone -a bash
cd bash
git checkout f32

Or clone your GitHub project repository:

git clone https://github.com/teemtee/tmt/
cd tmt
git checkout -b enable-tests

Smoke Test

Let’s enable a simple smoke test using the minimal plan template:

$ tmt init --template mini
Tree '/tmp/bash' initialized.
Applying template 'mini'.
Directory '/tmp/bash/plans' created.
Plan '/tmp/bash/plans/example.fmf' created.

Edit the newly created plan as needed, for example like this:

summary:
    Basic smoke test for bash
execute:
    script: bash --version

Execute Tests

Run Tests

Execute all available tests safely in a virtual machine:

tmt run

Run only tests matching given name or located under the current directory:

tmt run test --name smoke
tmt run test --name .

Show detailed test results from the latest tmt run executed by current user:

tmt run --last report -fvvv

Executing tests enabled using the Standard Test Interface in tests/tests.yml is not supported yet but we are working on it.

Select Steps

Explicitly choose which steps should be run:

tmt run discover

This will provide an overview of tests which would be run. To list individual tests enable the verbose mode:

tmt run discover --verbose
tmt run discover -v

Provision Options

Choose local as the provision method but run --all steps:

tmt run --all provision --how local

Execute inside a container or virtual machine:

tmt run --all provision --how container --image fedora
tmt run --all provision --how virtual --image fedora-32

Check all available provision plugins:

tmt run provision --help

Prepare Options

Install additional packages on the guest:

tmt run --all prepare --how install --package httpd

Get the latest package from provided copr repository:

tmt run --all prepare --how install --copr @teemtee/tmt --package tmt

Use the freshly build local rpm or all rpms from provided local directory:

tmt run --all prepare --how install --package tmp/RPMS/noarch/tmt-0.20-1.fc32.noarch.rpm
tmt run --all prepare --how install --directory tmp/RPMS/noarch

Check all available prepare options:

tmt run prepare --help

Create Test

In order to create more complex tests let’s use the base plan template:

tmt plan create /plans/basic --template base
tmt plan create /plans/basic -t base

Update summary as needed, keep discover method to fmf and choose whether tests should be executed as shell scripts (just check the exit code) or beakerlib tests (investigate journal for test results):

summary:
    Check basic bash features
discover:
    how: fmf
execute:
    how: tmt

Shell Test

In order to create a simple shell test skeleton use the shell template:

$ tmt test create /tests/smoke
Template (shell or beakerlib): shell
Directory '/tmp/bash/tests/smoke' created.
Test metadata '/tmp/bash/tests/smoke/main.fmf' created.
Test script '/tmp/bash/tests/smoke/test.sh' created.

Update metadata file:

summary: Check bash version
contact: Petr Šplíchal <psplicha@redhat.com>
test: ./test.sh

Adjust the test script as desired:

#!/bin/sh -eux
tmp=$(mktemp)
bash --version > $tmp
grep 'GNU bash' $tmp
grep 'Free Software Foundation' $tmp
rm $tmp

Use tmt run to verify the test is working as expected.

BeakerLib Test

Use beakerlib template to create a new beakerlib test:

$ tmt test create /tests/smoke -t beakerlib
Directory '/tmp/bash/tests/smoke' created.
Test metadata '/tmp/bash/tests/smoke/main.fmf' created.
Test script '/tmp/bash/tests/smoke/test.sh' created.

Update test metadata and code as needed, use tmt run to verify everything is working fine.

Pull Requests

When creating the pull request make sure you add all created files including the special .fmf directory.

git add .
git commit

Fedora

In order to test your changes in Fedora CI no additional configuration is needed. Make sure you push the changes into your forked repository as fedora rpms/tests namespace does not allow force-push or branch removal.

git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/tmt.git
git push fork -u enable-tests

GitHub

In order to test a pull request on GitHub enable the Packit-as-a-Service integration and add a .packit.yaml configuration file:

jobs:
- job: tests
  trigger: pull_request
  metadata:
    targets:
    - fedora-all

For more details see the Testing Farm documentation. Once the integration is enabled push the branch, create a new pull request as ususal and wait for results:

git push origin -u enable-tests

Templates

When creating a pull request to enable tests in a repository with no tmt configuration, include a couple of hints and links for those who are not familiar with the new tooling:

This pull request enables tests in the Fedora CI using `tmt` which
also allows to easily execute and debug tests from your laptop:

Run tests directly on your localhost:

    sudo dnf install -y tmt
    tmt run --all provision --how local

Run tests in a virtual machine:

    sudo dnf install -y tmt+provision-virtual
    tmt run

Check the documentation to learn more about the tool:
https://docs.fedoraproject.org/en-US/ci/tmt/

Manage Tests

Explore available tests, convert old metadata, share test code.

Explore Tests

In order to see which tests are available:

tmt test ls

To show more details about individual tests:

tmt test show

To see an overview of all metadata:

tmt

Explore all available options and commands using --help.

Share Tests

Test code does not have to reside in the same git repository (e.g. dist git rpms namespace). It is possible to store tests in a dedicated repository and share them across components or product versions. You only need to reference the repository in the discover step. Use the full plan template to get quickly started:

tmt plan create /plans/upstream -t full

Update the repository url to point to the right place:

summary:
    Essential command line features
discover:
    how: fmf
    url: https://github.com/teemtee/tmt
execute:
    how: tmt

Now you will be able to run tests from the remote repository. See the discover step documentation for details.

Various Hints

Multiple Commands

Multiple shell commands can be provided under the script attribute as well:

summary:
    Basic smoke test for bash
execute:
    script:
        - bash --version
        - bash -c 'echo $((1+1+1))' | grep 3

See the script method documentation for details.

Installing Dependencies

Required packages can be installed using the prepare attribute:

summary: Basic smoke test for python3-m2crypto
prepare:
    how: install
    package:
      - python3-setuptools
      - python3-m2crypto
execute:
    script: python3 -c "import M2Crypto"

See the prepare step documentation for details.

Multiple Repositories

In the discover step it is possible to reference multiple repositories as well. In this way you can for example easily execute both upstream and fedora tests as part of a single plan:

discover:
  - name: fedora
    how: fmf
    url: https://src.fedoraproject.org/tests/selinux.git
  - name: upstream
    how: fmf
    url: https://github.com/SELinuxProject/selinux-testsuite

See also multiple config example in tmt repo to get a better idea.

Multiple Plans

It is possible to use multiple plans to group relevant tests together or to be able to easily run a subset of tests. For example, let’s have a /plans/features plan which covers all functionality tests from the local git repository:

discover:
    how: fmf
execute:
    how: tmt

And a separate /plans/integration plan to enable integration testing with another component:

discover:
    how: fmf
    url: https://src.fedoraproject.org/rpms/ltrace.git
execute:
    how: tmt

Running all tests from given plan is then very easy:

tmt run plan --name /plans/features

When run in the CI, results from such plans are reported as a single resultsdb testcase and are shown in pull requests as a single flag. In order to enable separate result for each plan, create a ci.fmf file in the git repository root with the following content:

resultsdb-testcase: separate

Once the separate reporting is enabled, you can turn on gating for selected plans only. Plan name becomes part of the resultsdb testcase name which is used in the gating.yaml config. See the gating documentation on Using Multiple Plans for more details.

Minimal Path

Here is an example of a minimal test creation path:

dnf install -y tmt+all
git clone https://src.fedoraproject.org/rpms/bash
cd bash
tmt init -t mini
vim plans/example.fmf
tmt run

A slightly extended example with custom test and plan template and executing test directly on the local host:

dnf install -y tmt+all
git clone https://src.fedoraproject.org/rpms/bash
cd bash
tmt init
tmt plan create --template base plans/smoke
tmt test create --template beakerlib tests/smoke
vim plans/smoke.fmf tests/smoke/*
tmt run --all provision -h local
git add .
git commit -m "Enable basic tests"
git push

Virtualization Tips

In order to safely run tests under a virtual machine started on your laptop you only need to install the tmt+provision-virtual package. By default the session connection is used so no other steps should be needed, just execute tests using the tmt run command. See the upstream Virtualization Tips for more options.

More Info

Test Examples

Example projects with tmt tests:

See the tmt examples page for more inspiration.

Questions

Does the tool replace/deprecate STI?: No, currently there is no plan to decommission STI. Both tmt and sti approach to CI configuration can be used in parallel.
Are these tests supported in Fedora CI?: Yes, Fedora CI support is enabled for all active branches and the tests namespace as well.
Which Linux distributions does the tool support?: As a system under test (on which the tests are executed) all supported Fedora versions, Centos 6+ and Red Hat Enterprise Linux 6+ can be used. For the test runner (where tmt command is run) all supported Fedora versions, Centos 8+ or Red Hat Enterprise Linux 8+ are required.

Want to help? Learn how to contribute to Fedora Docs ›