Debugging with Toolbx

The FCOS image is kept minimal by design to reduce the image size and the attack surface. This means that it does not include every troubleshooting tools that a normal OS may include. Instead, the recommended approach is to leverage containers with the toolbox utility included in the image.

What is Toolbx?

Toolbx is a utility that allows you to create privileged containers meant to debug and troubleshoot your instance. It is a wrapper around podman which starts long running containers with default mounts and namespaces to facilitate debugging the host system.

These containers can then be used to install tools that you may need for troubleshooting.

Using Toolbx

You can create a new toolbox by running the command below. On the first run, this will ask you if you want to use the image registry.fedoraproject.org/fedora-toolbox:35.

toolbox create

You can then list all the running toolboxes running on the host. This should show you your newly created toolbox. In this case, it is named fedora-toolbox-35.

toolbox list

As pointed out by the output of the toolbox create command, you can enter the following command to enter your toolbox.

toolbox enter

Now that you’re in the container, you can use the included dnf package manager to install packages. For example, let’s install strace to look at read syscall done by the host’s toolbox utility.

sudo dnf install strace
# Some hosts directories are mounted at /run/host
strace -eread /run/host/usr/bin/toolbox list

Once done with your container, you can exit the container and then remove it from the host with the following command.

toolbox rm --force fedora-toolbox-35
Toolbx allows you to create toolboxes with your custom images. You can find more details in the toolbox manpages.