Running containerized workloads in systemd is a simple yet powerful means for reliable and rock-solid deployments. Podman has an excellent integration with systemd in the form of Quadlet. Quadlet is a tool for running Podman containers in systemd in an optimal and declarative way. Workloads can be declared in the form of systemd-unit-like files extended with Podman-specific functionality.

You might be wondering about the benefits of running containerized workloads in systemd. First, systemd is the central control instance on modern Linux systems. Among other things, it manages system and user services and the dependencies among them. It has tons of capabilities such as elaborate restart policies. Hence, Podman’s integration with systemd was an important milestone to integrate traditional Linux sysadmin craftmanship with modern container technologies.

Second, Podman’s daemonless architecture integrates perfectly with systemd. The sophisticated process management of systemd allows it to monitor a container and restart it if needed. The combination of systemd and Podman allowed us to tackle new use cases where human intervention is not always possible, for instance edge computing or IoT. On top, Quadlet is a seamless extension for systemd, which makes it very approachable for sysadmins.

So let’s take a closer look at Quadlet, and take the following example Quadlet .container file:

As mentioned, Quadlet extends systemd-units with Podman-specific features. Quadlet .container files, for instance, add a [Container] table where we can declare container-specific options such as the image, command, and name of the container, but also which volumes and networks it should use. Quadlet is a systemd-generator that is being executed on boot or when reloading the systemd daemon. If you want to test the upper example, you can create the file in your home directory ($HOME/.config/containers/systemd/test.container) and run systemctl --user daemon-reload. Reloading the daemon will fire Quadlet and generate a systemd service named test.service that you can then start with systemctl --user start test.service.

You can think of Quadlet like Docker Compose, but for running containers in systemd. The declarative nature of Quadlet makes it a perfect candidate for installing and embedding workloads on a bootc system. Quadlet also supports running Pods and Kubernetes-compliant YAML definitions, can manage volumes, networks and images and further supports building images. For detailed documentation on Quadlet and more examples, please refer to the upstream documentation.

Container images can be pulled on-demand whenever a Quadlet is started and the referenced image is not yet present in the local containers storage.

The big advantage of this model is the ease-of-use as we just need to install and run a Quadlet and let Podman deal with the pulling. Podman containers started by Quadlet and those created by the user or other tools can all just share the same store.

A major disadvantage of on-demand pulls is that it delays starting the workloads until the individual images have been pulled down. This may even impact boot time when Quadlets are started at boot. Moreover, disconnected environments cannot make use of on-demand pulls at all due to the lack of network connectivity.

Logically-bound images are an improvement over on-demand pulls as they allow images to be pre-pulled at install time and on updates. This way, logically-bound images must not be pulled when a Quadlet starts as those images are already present on the system.

Logically-bound images can be specified in the /usr/lib/bootc/bound-images.d directory in the form of symlinks. bootc will automatically pull the images on bootc install, bootc upgrade and bootc switch. The symlinks in the directory point to Quadlet files on the system. Currently, those Quadlets must either be .container or .image files.

An example Containerfile using logically-bound images may look as follows:

To access logically-bound images, .container Quadlets need to add the following like to the [Container] table:

GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage

This setting allows Podman to access the so-called additional image store of bootc. Please note that the presented solution of using GlobalArgs is preferable over a system-wide configuration in storage.conf — unless all containers run in Quadlets. For more details on logically-bound images, please refer to the upstream documentation of bootc and the Fedora examples.

Some use cases require the entire boot image to be fully self contained. That means that everything needed to execute the workloads is shipped with the bootc image, including container images of the application containers and Quadlets. Such images are also referred to as “physically-bound images”.

The underlying mechanism is very similar to the one of logically-bound images in that physically-bound images can be pre-pulled during image build time and made available at runtime. Let’s first dive into how we can achieve such physical embedding and explain later on why we recommend doing it as follows.

The instruction in a Containerfile to physically embed an image at build time may look like that:

RUN skopeo copy --preserve-digests docker://<IMAGE> dir:/usr/lib/containers-image-cache/<DIRECTORY>

At runtime, the image can be moved into Podman’s mutable store as follows:

RUN skopeo copy --preserve-digests dir:/usr/lib/containers-image-cache/<DIRECTORY> containers-storage:<IMAGE>

Since the embedded images may change with each system update, we cannot use an additional image store for this purpose as it was not designed to be swapped out. Instead we make use of the “dir” transport of the container tools which allows for storing one or more images in the same directory. The “dir” transport further allows to preserve the digest of the image which is crucial for the image to be referenced by its original digest. However, there is one caveat: we need to keep track of the name of the image.

To improve the experience of physically embedding container images, we propose two scripts that you can find in the upstream Fedora bootc examples.

An example to physically embed a number of images in a Containerfile may look as follows:

To copy the images into Podman’s mutable store at runtime, just run /usr/bin/copy_embedded_images.sh. Note that the images must be copied over before any container or service (e.g., Quadlet) depending on such an image is started. It could be moved into a systemd unit that starts before any Quadlet, for instance. For more information, please see the upstream upstream Fedora bootc examples. In the meantime, we are working on improving the user experience when using physically-embedded images.

An important aspect of embedding images is the way they are referenced. Images can be referenced by tag, by digest, or a mix of both. While digests always point to exactly one image, a tag may be updated on a registry. Choosing the right way of referencing an image can impact the quality and robustness of workloads, so we should be intentional about it. If you are interested in this topic, please see an article on how to name, version, and reference container images.