Module Stream

A module stream is the cornerstone of the Modularity project. A module stream represents a new entity which can be a part of a RPM repository. With the inclusion of module streams in a RPM repository the repository will be extended to a modular RPM repository.

A module stream is a group of RPM packages which are built, installed and shipped together. The RPM packages are bound together by modular metadata. The artifacts of a module stream are binary and source RPM files and a module metadata (modulemd) yaml file.

Table of Contents

Modular Metadata
- Input modulemd-packager
- Output modulemd
Module profiles

Modular Metadata

A standard RPM package consists of binary and source RPM files which are carrying all the necessary metadata inside the RPM file header. With the Modularity project we introduced a lot of additional metadata which describes how a module stream should be built, how it should be installed, but also how it fits and interacts within a modular repository.

There are two basic types of modular metadata. The modulemd yaml file which we are using as an input for building a module stream and a modulemd yaml file which is an artifact of the module stream build process. Of course we have more types of modulemd but for now we will only focus on the input modulemd-packager and the output modulemd files.

For the sake of space, both examples of module streams were shortened. For example a Perl module stream consists of 170-180 components. If you want to check out the un-shortened version please check the Perl module dist-git repository.

The specification of all the metadata formats used by Modularity can be found in the libmodulemd git repository. libmodulemd is a C library which provides an API (with python bindings) which enables us to work with module metadata.

Input modulemd-packager

Example 1. modulemd-packager metadata of a Perl module stream

---
document: modulemd-packager (1)
version: 3
data:
    stream: '5.32' (2)
    summary: Practical Extraction and Report Language
    description: >
        Perl is a high-level programming language with roots in C, sed, awk
        and shell scripting. Perl is good at handling processes and files, and
        is especially good at handling text. Perl's hallmarks are practicality
        and efficiency. While it is used to do a lot of different things,
        Perl's most common applications are system administration utilities
        and web programming.
    license: [MIT]
    configurations: (3)
        - context: '866eb18a' (4)
          platform: f34
          buildrequires: (5)
              perl-bootstrap: ['5.32']
          buildopts: (6)
              rpms:
                  macros: |
                      %_with_perl_enables_groff 1
                      %_without_perl_enables_syslog_test 1
                      %_with_perl_enables_systemtap 1
                      %_without_perl_enables_tcsh 1
                      %_without_perl_Archive_Tar_enables_optional_test 1
                      %_without_perl_autodie_enables_optional_test 1
                      %_without_perl_Compress_Bzip2_enables_optional_test 1
                      ⋮
        - context: 'ab7e7bfe'
        ⋮
        - context: '553f8e7a'
        ⋮
        - context: 'd5eb079e'
        ⋮
    references:
        community: https://www.perl.org/
        documentation: https://www.perl.org/docs.html
        tracker: https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&product=Fedora%20Modules&component=perl
    profiles: (7)
        common:
            description: Interpreter and all Perl modules bundled within upstream Perl.
            rpms:
                - perl
        minimal:
            description: Only the interpreter as a standalone executable.
            rpms:
                - perl-interpreter
    api: (8)
        rpms:
            - perl
            - perl-Archive-Tar
            - perl-Attribute-Handlers
            - perl-autodie
            - perl-AutoLoader
            - perl-AutoSplit
            - perl-autouse
            - perl-B
            - perl-base
            - perl-Benchmark
          ⋮
    filter: (9)
        rpms:
            - perl-tests
            - perl-Compress-Raw-Bzip2-tests
            - perl-Compress-Raw-Zlib-tests
            - perl-Digest-tests
            - perl-ExtUtils-MakeMaker-tests
            - perl-IO-Compress-tests
            - perl-Module-CoreList-tests
            - perl-Storable-tests
            - perl-URI-tests
    components: (10)
        rpms:
            # We can rebuild in a random order because we have all binary
            # packages available from perl-bootstrap.
            perl:
                rationale: The Perl interpreter.
                ref: f33
            perl-Algorithm-Diff:
                rationale: A run-time dependency.
                ref: f33
            perl-Archive-Tar:
                rationale: Core Perl API.
                ref: f33
            perl-Archive-Zip:
                rationale: A run-time dependency.
                ref: f33
            perl-autodie:
                rationale: Core Perl API.
                ref: f33
            perl-bignum:
                rationale: Core Perl API.
                ref: f33
          ⋮
...

1	the definition of the type of modulemd yaml document
2	the name of the stream of a module stream
3	definition of context configurations for different releases or build options
4	each name of a context is defined by the user and uniquely identifies the buildtime and runtime configuration combination of a module stream
5	with modular dependencies a module stream can depend on other modules in buildtime or runtime
6	with a context configuration we can set options which we want to be passed done to the build process for all the components of module stream
7	profiles provide us with a feature to be able to install different combinations of components at runtime
8	a list of binary RPM names which are provided by the module stream
9	a list of binary RPM names which should be filtered out after building from the module stream
10	a list of RPM packages which should be build, distributed, enabled and installed by the module stream

The modulemd-packager document is used for defining the whole of a module stream. The document is used as an input file to the build system you are using to build modules in. In our example we have a Perl module stream with the name perl:5.32. The name of a module stream consists of a combination of a module’s name and the name of its stream. As you can see in the example, only the stream name (5.32) is defined. This is because, if you are building your module streams in Fedora you don’t need to provide the stream name or the module name. Both are taken at build time from the name of the dist-git (module name) repository and its branch (stream name). For more information on the definition of modulemd-packager please refer to the modulemd yaml file specifications.

The example has four context configurations. This means that this definition file will produce four different module stream builds for each context. Each context defines how, where and for what purpose a module stream is built. You can find more information on contexts in the Module context section.

At the time of writing a standard Perl module stream needs to build around 180 RPM packages. Those RPM packages build around 400 source and binary RPM files. So when building four context configurations you get to around 1600 source and binary RPM files.

When the input modulemd-packager file is processed by the build system it is split into output modulemd files which are copies of the modulemd-packager, but are updated with the context configuration of each context and the build process of the build system. In our example this would equal to four output modulemd yaml files.

Output modulemd

Example 2. output modulemd metadata of a Perl module stream created by the build system

---
document: modulemd
version: 2
data:
  name: perl (1)
  stream: "5.32"
  version: 3520210922083143 (2)
  context: ab7e7bfe (3)
  static_context: true (4)
  arch: x86_64 (5)
  summary: Practical Extraction and Report Language
  description: >
    Perl is a high-level programming language with roots in C, sed, awk and shell
    scripting. Perl is good at handling processes and files, and is especially good
    at handling text. Perl's hallmarks are practicality and efficiency. While it is
    used to do a lot of different things, Perl's most common applications are system
    administration utilities and web programming.
  license:
    module:
    - MIT
    content:
    - (Copyright only) and (Artistic or GPL+)
    - (GPL+ or Artistic) and (GPLv2+ or Artistic) and MIT
    - (GPL+ or Artistic) and Artistic 2.0 and UCD
    - (GPL+ or Artistic) and BSD
    ⋮
  xmd: {} (6)
  dependencies: (7)
  - buildrequires:
      perl-bootstrap: [5.32]
      platform: [f35]
    requires:
      platform: [f35]
  references:
    community: https://www.perl.org/
    documentation: https://www.perl.org/docs.html
    tracker: https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&product=Fedora%20Modules&component=perl
  profiles:
    common:
      description: Interpreter and all Perl modules bundled within upstream Perl.
      rpms:
      - perl
    minimal:
      description: Only the interpreter as a standalone executable.
      rpms:
      - perl-interpreter
  api:
    rpms:
    - perl
    - perl-Archive-Tar
    - perl-Attribute-Handlers
    - perl-AutoLoader
    - perl-AutoSplit
    - perl-B
    - perl-Benchmark
    - perl-CPAN
    ⋮
  filter: (8)
    rpms:
    - perl-Compress-Raw-Bzip2-tests
    - perl-Compress-Raw-Zlib-tests
    - perl-Digest-tests
    - perl-ExtUtils-MakeMaker-tests
    - perl-IO-Compress-tests
    - perl-Module-CoreList-tests
    - perl-Storable-tests
    - perl-URI-tests
    - perl-tests
  buildopts:
    rpms:
      macros: >
        %_with_perl_enables_groff 1
        %_without_perl_enables_syslog_test 1
        %_with_perl_enables_systemtap 1
        %_without_perl_enables_tcsh 1
        ⋮
  components:
    rpms:
      perl:
        rationale: The Perl interpreter.
        ref: f33
        arches: [aarch64, armv7hl, i686, ppc64le, s390x, x86_64]
      perl-Algorithm-Diff:
        rationale: A run-time dependency.
        ref: f33
        arches: [aarch64, armv7hl, i686, ppc64le, s390x, x86_64]
        ⋮
  artifacts: (9)
    rpms:
    - perl-4:5.32.1-471.module_f35+12589+8a7d3254.src
    - perl-4:5.32.1-471.module_f35+12589+8a7d3254.x86_64
    - perl-Algorithm-Diff-0:1.1903-17.module_f35+11299+2c136bae.noarch
    - perl-Algorithm-Diff-0:1.1903-17.module_f35+11299+2c136bae.src
    - perl-Archive-Tar-0:2.38-3.module_f35+11299+2c136bae.noarch
    - perl-Archive-Tar-0:2.38-3.module_f35+11299+2c136bae.src
    - perl-Archive-Zip-0:1.68-3.module_f35+11299+2c136bae.noarch
    - perl-Archive-Zip-0:1.68-3.module_f35+11299+2c136bae.src
    - perl-Attribute-Handlers-0:1.01-471.module_f35+12589+8a7d3254.noarch
    ⋮
...

1	the module name added by the build system
2	the module version added by the build system
3	the module context name added from the context configuration of the input `modulemd-packager`
4	marks this contexts as a static contexts. More on this in the Module Context section.
5	the system architecture for which this modules stream was build
6	`xmd` or extra metadata which can be added by the build system
7	the modular runtime and build time dependencies of a module stream
8	filter property filters out binary RPMs which should not be provided in the final module stream
9	the list of source and binary RPM artifact filenames which are associated with this module

Every modulemd-packager yaml file is split into modulemd files at build time. For each context defined we create its own modulemd file. The modulemd files describe the final form of a module stream build. The output modulemd file is the file to be included in the RPM repository together with the source and binary RPM files associated with the metadata.

The modulemd file does not contain the context configuration section as the file is the result artifact of build of a module stream context combination. Each modulemd yaml file is processed and configured to its respective context configuration from its input modulemd-packager yaml file.

The artifacts section is a list of source and binary RPM filenames which where built by the module stream build process. This section specifically describes which RPM files should be included into the DNF transaction when enabling and installing modules.

Module profiles

Example 3. the profiles property of postgres:10 module stream

⋮
    profiles:
        server:
            rpms:
                - postgres-server
        client:
            rpms:
                - postgres
⋮

The profile of a module stream describes the combination of RPM packages that can be installed together. For example the postgres:10 module stream provides at the time of writing, two installation profiles client and server.

Want to help? Learn how to contribute to Fedora Docs ›