Test Management Tool

Summary

The tmt tool aims to provide an efficient and comfortable way to create, execute, debug and enable tests in the Continuous Integration.

It implements the Test Metadata Specification which allows to store all needed test execution data directly within a git repository. The same configuration can be used for enabling tests in the Fedora CI (near future), RHEL CI and Packit. Tests can be easily executed in your preferred environment, e.g. in virtual machine, container or directly on the localhost.

First Steps

Install

Install tmt on your laptop:

sudo dnf install -y tmt       # basic features, executing tests on localhost
sudo dnf install -y tmt-all   # install all available tmt subpackages including all dependencies

You can also install selected provision plugins only:

sudo dnf install -y tmt-provision-container   # additional dependencies for executing tests in containers
sudo dnf install -y tmt-provision-virtual     # support for running tests in a virtual machine using testcloud

See the tmt install section for more installation options.

Git Repo

Check out the desired dist git branch using fedpkg:

fedpkg clone -a bash
cd bash
git checkout f32

Or clone your GitHub project repository:

git clone https://github.com/psss/tmt/
cd tmt
git checkout -b enable-tests

Smoke Test

Let’s enable a simple smoke test using the minimal plan template:

$ tmt init --template mini
Tree '/tmp/bash' initialized.
Applying template 'mini'.
Directory '/tmp/bash/plans' created.
Plan '/tmp/bash/plans/example.fmf' created.

Edit the newly created plan as needed, for example like this:

summary:
    Basic smoke test for bash
execute:
    script: bash --version

Execute Tests

Run Tests

Execute all available tests safely in a virtual machine:

tmt run

Run only tests matching given name or located under the current directory:

tmt run test --name smoke
tmt run test --name .

Show detailed test results from the latest tmt run executed by current user:

tmt run --last report -fvvv

Executing tests enabled using the Standard Test Interface in tests/tests.yml is not supported yet but we are working on it.

Select Steps

Explicitly choose which steps should be run:

tmt run discover

This will provide an overview of tests which would be run. To list individual tests enable the verbose mode:

tmt run discover --verbose
tmt run discover -v

Provision Options

Choose local as the provision method but run --all steps:

tmt run --all provision --how local

Execute inside a container or virtual machine:

tmt run --all provision --how container --image fedora
tmt run --all provision --how virtual --image fedora-32

Check all available provision plugins:

tmt run provision --help

Prepare Options

Install additional packages on the guest:

tmt run --all prepare --how install --package httpd

Get the latest package from provided copr repository:

tmt run --all prepare --how install --copr psss/tmt --package tmt

Use the freshly build local rpm or all rpms from provided local directory:

tmt run --all prepare --how install --package tmp/RPMS/noarch/tmt-0.20-1.fc32.noarch.rpm
tmt run --all prepare --how install --directory tmp/RPMS/noarch

Check all available prepare options:

tmt run prepare --help

Pull Requests

In order to test a pull request on GitHub enable the Packit-as-a-Service integration and add a .packit.yaml configuration file:

jobs:
- job: tests
  trigger: pull_request
  metadata:
    targets:
    - fedora-all

For more details see the Testing Farm documentation. Once the integration is enabled push the branch, create a new pull request as ususal and wait for results:

git push origin -u enable-tests

Fedora CI support for tmt tests is coming in August/September 2020.

Create Test

In order to create more complex tests let’s use the base plan template:

tmt plan create /plans/basic --template base
tmt plan create /plans/basic -t base

Update summary as needed, keep discover method to fmf and choose whether tests should be executed as shell scripts (just check the exit code) or beakerlib tests (investigate journal for test results):

summary:
    Check basic bash features
discover:
    how: fmf
execute:
    how: beakerlib

Shell Test

In order to create a simple shell test skeleton use the shell template:

$ tmt test create /tests/smoke
Template (shell or beakerlib): shell
Directory '/tmp/bash/tests/smoke' created.
Test metadata '/tmp/bash/tests/smoke/main.fmf' created.
Test script '/tmp/bash/tests/smoke/test.sh' created.

Update metadata file:

summary: Check bash version
contact: Petr Šplíchal <psplicha@redhat.com>
test: ./test.sh

Adjust the test script as desired:

#!/bin/sh -eux
tmp=$(mktemp)
bash --version > $tmp
grep 'GNU bash' $tmp
grep 'Free Software Foundation' $tmp
rm $tmp

Use tmt run to verify the test is working as expected.

BeakerLib Test

Use beakerlib template to create a new beakerlib test:

$ tmt test create /tests/smoke -t beakerlib
Directory '/tmp/bash/tests/smoke' created.
Test metadata '/tmp/bash/tests/smoke/main.fmf' created.
Test script '/tmp/bash/tests/smoke/test.sh' created.

Update test metadata and code as needed, use tmt run to verify everything is working fine.

Manage Tests

Explore available tests, convert old metadata, share test code.

Explore Tests

In order to see which tests are available:

tmt test ls

To show more details about individual tests:

tmt test show

To see an overview of all metadata:

tmt

Explore all available options and commands using --help.

Share Tests

Test code does not have to reside in the same git repository (e.g. dist git rpms namespace). It is possible to store tests in a dedicated repository and share them across components or product versions. You only need to reference the repository in the discover step. Use the full plan template to get quickly started:

tmt plan create /plans/upstream -t full

Update the repository url to point to the right place:

summary:
    Essential command line features
discover:
    how: fmf
    url: https://github.com/psss/tmt
execute:
    how: beakerlib

Now you will be able to run tests from the remote repository. See the discover step documentation for details.

Various Hints

Multiple Commands

Multiple shell commands can be provided under the script attribute as well:

summary:
    Basic smoke test for bash
execute:
    script:
        - bash --version
        - bash -c 'echo $((1+1+1))' | grep 3

See the shell method documentation for details.

Multiple Repositories

In the discover step it is possible to reference multiple repositories as well. In this way you can for example easily execute both upstream and fedora tests as part of a single plan:

discover:
  - name: fedora
    how: fmf
    repository: https://src.fedoraproject.org/tests/selinux.git
  - name: upstream
    how: fmf
    repository: https://github.com/SELinuxProject/selinux-testsuite

See also multiple config example in tmt repo to get a better idea.

Minimal Path

Here is an example of a minimal test creation path:

dnf install -y tmt-all
git clone https://src.fedoraproject.org/rpms/bash
cd bash
tmt init -t mini
vim plans/example.fmf
tmt run

A slightly extended example with custom test and plan template and executing test directly on the local host:

dnf install -y tmt-all
git clone https://src.fedoraproject.org/rpms/bash
cd bash
tmt init
tmt plan create --template base plans/smoke
tmt test create --template beakerlib tests/smoke
vim plans/smoke.fmf tests/smoke/*
tmt run --all provision -h local
git add .
git commit -m "Enable basic tests"
git push

Virtualization Tips

Here’s just a couple of hints how to get the virtualization quickly working on your laptop. See the Getting started with virtualization docs to learn more.

Make sure libvirtd is running:

sudo systemctl start libvirtd

Add your user account to the libvirt group. You might need to restart your desktop session to get it fully working.

sudo usermod -a -G libvirt $(whoami)
newgrp libvirt

Here you can find vm images for download.

Questions

Does the tool replace/deprecate STI?

No, currently there is no plan to decommission STI. Both tmt and sti approach to CI configuration can be used in parallel.

Are these tests supported in Fedora CI?

Fedora CI support is coming in August/September 2020.

Which Linux distributions does the tool support?

As a system under test (on which the tests are executed) all supported Fedora versions, Centos 6+ and Red Hat Enterprise Linux 6+ can be used. For the test runner (where tmt command is run) all supported Fedora versions, Centos 8+ or Red Hat Enterprise Linux 8+ are required.