Upgrade Paths in Modularity

Since modularity consists of modules and packages, there two levels of upgrade.

On the modular level

When DNF have multiple module builds of a stream available in the repositories, it will first separate the builds by a module context. Then in each context DNF will sort the builds by a version. Finally DNF will pick as an upgrade the highest version of a context which matches the currently installed context.

There are two algorithms for separating modules by a context depending on a type of the context:

Static context

Module builds with static_context: true keyword are called static-context. A value of context field is the only factor considered when establishing an upgrade path.

For instance, having installed foo:stream:0:A which requires bar:x and the following four static-context modules in a repository:

Upgrade path A      Upgrade path B
────────────────    ─────────────────
foo:stream:2:A      foo:stream:2:B     ╮
  requires bar:y      requires: bar:z  │ in
foo:stream:1:A  <┐  foo:stream:1:B     │ a repository
  requires bar:x │    requires: bar:z  ╯
foo:stream:0:A   ┘                     ╮ already
  requires bar:x                       ╯ installed

DNF will create two lineages A and B, picks lineage A because foo:stream is already installed with a static context A, warn that foo:stream:2:A cannot be installed because it conflicts with an already enabled stream bar:x, and will upgrade to foo:stream:1:A because version 1 is the highest number which has met run-time dependencies.

Dynamic context

Module builds without static_context: true keyword are called dynamic-context (or nonstatic). Establishing an upgrade path actually does not consider the context value, but run-time dependencies of the modules. This algorithm is deprecated because it’s cumbersome, does not support changing run-time dependencies, and adding and removing contexts.

For instance, having installed foo:stream:0:Z which requires bar:x and the following four dynamic-context modules in a repository:

Upgrade path 1      Upgrade path 2
────────────────    ────────────────
foo:stream:2:A  <┐  foo:stream:2:C   ╮
  requires bar:x │    requires bar:y │ in
foo:stream:1:B   │  foo:stream:1:D   │ a repository
  requires bar:x │    requires bar:y ╯
foo:stream:0:Z   ┘                   ╮ already
  requires bar:x                     ╯ installed

DNF will create two lineages 1 and 2, and will upgrade to foo:stream:2:A. This is because foo:stream:2:A has the highest version in a lineage whose modules require bar:x as the already installed foo:stream:0:Z.

Observe that with the dynamic context a module maintainer cannot express that foo:stream:0:Z or foo:stream:1:B should upgrade to foo:stream:2:C if the maintainer decided to change a dependency from bar:x to bar:y and were unable to build foo:stream:2:A (e.g. an upstream rewrote the application from Ruby 2.4 to Ruby 2.6).

Latest version overrides the properties of a stream

Once DNF identifies the latest module, all modular metadata defined in that module build will apply and replace modular metadata occurring in the older module versions. That includes a module description shown with dnf module list command, a list of packages installed when installing a profile, or a list of demodularized packages.

The only data which are used from older versions of (a context of) the stream are RPM packages. That enables users to downgrade a package in case the newer module build contained a bug.

Adding a context

For the reasons explained above, if you want to add a new context (e.g. you want to add a support for a new Perl 5.32 to your perl-DBI module), you need to use a static context:

Upgrade path A          Upgrade path B
────────────────────    ────────────────────
perl-DBI:stream:2:A  ↑  perl-DBI:stream:2:B  ↑
  requires perl:5.30 ↑    requires perl:5.32 ↑
perl-DBI:stream:1:A  ↑
  requires perl:5.30 ↑
╰──────────────────╯    ╰──────────────────╯
     Old context             New context

It’s not possible to achieve this by standard means with a dynamic context. DNF would report a warning or an error. If you ever needed to achieve this with nonstatic contexts, the key point how to indulge DNF is fill a matrix of contexts and versions with artificial modules which have never been built (i.e. manually writing the modulemd documents):

Upgrade path 1          Upgrade path 2
────────────────────    ────────────────────
perl-DBI:stream:2:A  ↑  perl-DBI:stream:2:B  ↑
  requires perl:5.30 ↑    requires perl:5.32 ↑
                     ↑ ┌────────────────────┐
perl-DBI:stream:1:A  ↑ │perl-DBI:stream:1:B │  An artificial module build
  requires perl:5.30 ↑ │  requires perl:5.32│  filling a gap in this matrix
                       └────────────────────┘

On the other hand, with a static context DNF will understand that modules from context B should not mix with module from a context A. When installing perl-DBI:stream for the first time, DNF will pick the right context based on an already installed perl stream. When upgrading perl-DBI:stream, DNF will preserve the context based on an already installed perl-DBI:stream context. Thus using static-context modules is recommended.

Removing a context

The same constrains of dynamic context and a strong recommendation for static context apply here. With static context you can stop building modules for that context whenever you want. With dynamic context you cannot and need to maintain the full matrix forever.

Migrating to a different stream

With a static context it’s possible to use a modulemd-obsoletes document to switch to a completely different stream. This is useful when you want to stop supporting an existing stream, or a context, and you provide a replacement. When user has installed an obsolete stream, or a context, DNF will offer switching to the replacement.

Upgrade path A              Upgrade path B
────────────────────────    ──────────────
                         ┌> perl:5.32:1:B
perl:5.30                │
  obsoleted by perl:5.32 │
perl:5.30:1:A            ┘

This feature is currently experimental and needs to be enabled in DNF configuration with module_stream_switch=True option.

On the package level

On the package level, upgrades of a modular system work the same way as on a traditional system — using NEVRA comparison to determine which packages are the newest. There is, however, one additional step right before the NEVRA comparison that needs to happen on a modular system — limiting which packages are going to be part of the comparison based on what modules are enabled — and that is the key difference.

There are up to three classes of RPM packages available to a modular system:

  1. Nonmodular packages — packages not being part of a module. In Fedora, these are coming from the Everything repository.

  2. Modular packages — packages being part of a module. In Fedora, these are coming from the Modular, Modular Updates, and Modular Test Updates repositories.

  3. Hotfixes — nonmodular packages created on-demand by users or vendors meant to fix a critical issue before an official upgrade comes from the distribution. These need to be provided in a separate repository with a hotfix flag set. Fedora doesn’t provide such packages.

To determine the limited set of packages for the NEVRA comparison, the following algorithm is used:

  1. For all enabled and default streams:

    1. Collect all modular packages, i.e. NEVRAs, that are part of an active context of that stream and add them to a pile of available packages for DNF. To do this, look at artifacts section of all module versions belonging to the same context of the stream. That includes a currently installed version, new versions coming into Modular Updates repository, as well as historical versions already presented in the Modular repository.

    2. Add names of these packages onto a list of modularly filtered names with an exception of names recorded as removed from a module in the latest version of the module. This list of modularly filtered names will be used for preferring modular packages over nonmodular ones.

  2. Add all nonmodular packages whose name is not on the list and which do not provide a name from the list to the pile of available packages. This step ensures that modular packages always have a higher priority than nonmodular ones. In other words, nonmodular packages can never upgrade modular packages unless explicitly unlisted by the stream.

  3. Add all hotfix packages to the pile. These are just added, exempted from the modular filtering. That means hotfixes can potentially upgrade both traditional and modular packages.

The next step is to take this pile of packages, and run a NEVRA comparison to determine the highest version of each package. The highest versions are then installed as a part of the upgrade process.

Example 1

A system has enabled streams bar:1 and loo:1; a repository contains:

Package  From a module   Requires a module
──────────────────────────────────────────
foo-1                                       ← a nonmodular package
foo-2    bar:1:2022:a    loo:1              ╮
foo-3    bar:1:2023:a    loo:1              │ modular
foo-4    bar:1:2023:b    loo:2              │ packages
foo-5    bar:2:2023:a                       ╯
         loo:1:2000:c                       ╮ empty
         loo:2:2000:c                       ╯ modules

dnf install foo command will install foo-3.

A reasoning is following: Active contexts are bar:1::a and loo:1::c. (It is not bar:1::b because loo:1 is enabled.) Hence packages available to DNF are foo-2 and foo-3. (It is not foo-1 because it’s nonmodular and has the same name as modular foo-2. It is not foo-4 because bar:1::b has an unmet modular dependency. It is not foo-5 because bar:2 is not enabled.) Highest NEVRA among the two available packages is foo-3. Therefore DNF will install foo-3.

Example 2

Next day a new update appears:

Package  From a module   Requires a module  Unlisted package
────────────────────────────────────────────────────────────
         bar:1:2024:a    loo:1                               ╮ empty
         bar:1:2024:a                       foo              ╯ modules

dnf upgrade foo command will not upgrade anything, reporting foo-3 is already latest and installed.

A reasoning is that the pile of available packages contains foo-1, foo-2, and foo-3 and foo-3 is already installed.

Please observe that foo-1 package became available because foo is unlisted name now. However, unlisting that name does not disappear the modular packages foo-2 and foo-3. Modular and and nonmodular foo packages are now handled equally.

Example 3

Another day a next update appears:

Package  From a module
──────────────────────
foo-6                   ← a nonmodular package

dnf upgrade foo command will upgrade to foo-6.

A reasoning is that foo-6 is the highest one from a pile of available packages among foo-1, foo-2, foo-3, and foo-6.

This demonstrates that a successful demodularization of a package requires both unlisting that package name and building that package, this time out of any module, with a new, highest version.