Python Packaging Guidelines

Python Version Support

In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that’s one for python3.x and one for python2.7. However the Python 2 stack will be removed from Fedora and is deprecated. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged into Fedora without FESCo exception.

For guidelines on maintaining already existing python2 packages, see the appendix.

Multiple Python Runtimes

On Fedora /usr/bin/python is, if it is installed, a symbolic link to /usr/bin/python3. It was a symbolic link to /usr/bin/python2 on previous releases.

Packages in Fedora MUST NOT use /usr/bin/python. Instead packages for Python 3 MUST use /usr/bin/python3 (even if upstream supports both Python 2 and 3). As a result of that /usr/bin/python (as well as /usr/bin/env python and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed (for example by using the %py3_shebang_fix macro in the spec file). If it is necessary to disable the checks, please see the information in Shebang lines.

All Python runtimes have a virtual provide for python(abi) = $MAJOR.$MINOR. For example, the Python 3.7 runtime package has:

$ rpm -q --provides python3 | grep abi
python(abi) = 3.7

Python modules using these runtimes should have a corresponding "Requires" line on the Python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.

Naming

The source package for a Python library MUST be named with the python- prefix. A built package however must include the Python major version in the name, using the python3- prefix. This is accomplished by adding a subpackage. See example below.

This rule does not apply to applications.

The character + in names of built packages (i.e. non-SRPM) that include .dist-info or .egg-info directories is reserved for Python Extras and MUST NOT be used for any other purpose. The + character triggers the automatic dependency generator for extras. Replace any + signs in the upstream name with -, or omit them when at the beginning of the name. As an exception, + characters are permitted at the end of the name.

Dependencies

Packages building for Python 3 will need BuildRequires: python3-devel. Most of them will also need BuildRequires: python3-setuptools. When in doubt, inspect the setup.py file for setuptools import.

Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python- prefix. Dependencies on Python packages instead MUST use names beginning with python3-.

Automatically generated dependencies

Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as setuptool’s install_requires) to determine what the package should depend on. The generator parses the installed metadata from /usr/lib(64)?/pythonX.Y/site-packages/[^/]+++.(egg|dist)-info/requires.txt, so it will not work with software that uses plain distutils.

This generates run time requires in the form of pythonX.Ydist(foo). If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the setup.py file) in the %prep section of the specfile or fall back to filtering those dependencies.

The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.

As an example, the upstream notebook package has (as of version 5.6.0):

install_requires = [
    'jinja2',
    'tornado>=4',
    'pyzmq>=17',
    'ipython_genutils',
    'traitlets>=4.2.1',
    'jupyter_core>=4.4.0',
    'jupyter_client>=5.2.0',
    'nbformat',
    'nbconvert',
    'ipykernel',
    'Send2Trash',
    'terminado>=0.8.1',
    'prometheus_client'
],

And the resulting dependencies:

python3.7dist(ipykernel)
python3.7dist(ipython-genutils)
python3.7dist(jinja2)
python3.7dist(jupyter-client) >= 5.2
python3.7dist(jupyter-core) >= 4.4
python3.7dist(nbconvert)
python3.7dist(nbformat)
python3.7dist(prometheus-client)
python3.7dist(pyzmq) >= 17
python3.7dist(send2trash)
python3.7dist(terminado) >= 0.8.1
python3.7dist(tornado) >= 4
python3.7dist(traitlets) >= 4.2.1

Note that any .0 suffixes are removed from version numbers to match the behavior of Python tools. (PEP 440 specifies that X.Y and X.Y.0 are treated as equal.)

This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding:

%{?python_disable_dependency_generator}

Although this statement can be used anywhere in the spec, we recommend putting it just before the main package’s %description declaration.

Python Extras

Python extras are a way for Python projects to declare that extra dependencies are required for additional functionality.

For example, requests has several standard dependencies (e.g. urllib3). But it also declares an extra named requests[security], which lists additional dependencies (e.g. cryptography). Unlike RPM subpackages, extras can only specify additional dependencies, not additional files. The main package will work if the optional dependency is not installed, but it might have limited functionality.

Python tools treat extras as virtual packages. For example, if a user runs pip install requests[security], or installs a project that depends on requests[security], both requests and cryptography will be installed.

Starting with Fedora 33, extras are usually provided by packages with no files. Instead of square brackets, Fedora package names conventionally use the + character to separate the package name and the extra name, e.g. the package would be named python3-requests+security. The plus sign is valid in RPM package names, but not in Python canonical project names nor in extras identifiers.

Python packages SHOULD have Provides for all extras the upstream project specifies, except those that are not useful for other packages (for example build/development requirements, commonly named dev, doc or test).

A package that provides a Python extra MUST provide python3dist(…[…]) and python3.Xdist(…[…]), for example, python3.9dist(requests[security]). These requirements SHOULD be generated using the automatic dependency generator.

A package that provides a Python extra MUST require the extra’s main package with exact NEVR.

A subpackage that primarily provides one Python extra SHOULD be named by appending + and the extra name to the main package name. For example, python3-requests+security.

The most straightforward way to provide an extra is with a dedicated subpackage containing no files (a "metapackage"). This case can be automated with the %python_extras_subpkg macro.

Alternative approach: when some extra is always useful in a distro, it can be provided by the main package; when several extras are related, they may be provided by a single subpackage. However, having one dedicated subpackage per extra allows you to use the automatic dependency generator to ensure that the extras' requirements will stay in sync with upstream. If you create a dedicated subpackage and want it to be always/usually installed, you MAY Require/Recommend/Suggest it from the main package.

The dependency generator for extras activates if the following holds:

  • The package must contain the .egg-info/.dist-info directory, usually as %ghost.

  • The package name must end with +EXTRA (where EXTRA is the extra name).

As an example, the extra subpackage for requests[security] can be specified using the %python_extras_subpkg convenience macro as follows. The macro takes the main package name and name(s) of the extra(s) as well as path to the .egg-info or .dist-info directory:

%{?python_extras_subpkg:%python_extras_subpkg -n python3-requests -i %{python3_sitelib}/*.egg-info security}

For this case, the extras dependency generator will read upstream metadata from the .egg-info directory. If it finds that the security extra has a dependency on cryptography, it will generate Requires: python3.Xdist(cryptography), Provides: python3dist(requests[security]) (and the corresponding python3.Xdist variant).

If you need additional features that the %python_extras_subpkg macro does not cover, you will need to write the subpackage sections manually. Such features can be, for example:

  • Obsoleting/providing other names (e.g. obsoleted extras packages)

  • Manual strong or weak dependencies on other (possibly non-Python) packages

  • Including files excluded from the main package (if such files only make sense with the extra and the base package does not fail without them)

As an example of what you need to write in these cases, the %python_extras_subpkg macro invocation above expands to the following:

%package -n python3-requests+security
Summary: Metapackage for python3-requests: security extras
Requires: python3-requests = %{?epoch:%{epoch}:}%{version}-%{release}
%description -n python3-requests+security
This is a metapackage bringing in security extras requires for python3-requests.
It contains no code, just makes sure the dependencies are installed.

%files -n python3-requests+security
%ghost %{python3_sitelib}/*.egg-info

Note that the dependency generator does not add a dependency on the main package (the Requires: python3-setuptools_scm = ... above). If you are not using the %python_extras_subpkg macro, you need to add it manually.

The %python_extras_subpkg can take multiple extras names to generate multiple packages. For more options, see the change proposal which introduced this.

Provides

For any module foo intended to be used in Python 3 with import foo, the package that includes it should provide python3-foo. This is of course always the case if the subpackage is named python3-foo (as in the examples below). If the subpackage has some other name, then Provides: python3-foo should be added explicitly (via %py_provides python3-foo, see below).

The %py_provides macro

All packages that provide python3-... (for any ...) SHOULD also provide python-... and python3.X-.... Starting from Fedora 33, most of the Python packages named python3-... will provide such names automatically via the dependency generator in /usr/lib/rpm/fileattrs/pythonname.attr.

Any manually added virtual provides of python3-... SHOULD be done via the %py_provides macro.

Instead of:

Provides: python3-pkg_resources = %{version}-%{release}

Do:

%py_provides python3-pkg_resources

Optionally, supply a custom epoch:version-release as a second argument to %py_provides.

On releases older than Fedora 33, or (for technical limitations) for packages without files, it is necessary to use %py_provides even for package names:

%package -n python3-%{srcname}
Summary: %{summary}
%py_provides python3-%{srcname}

Packagers SHOULD try to remove explicit %py_provides calls for package names, but MAY preserve them if they aim for compatibility with older releases or packages without files.

Historically, there was %python_provide macro with similar but different semantics. It still works for compatibility reasons but it is deprecated and SHOULD NOT be used and packagers SHOULD replace is with appropriate %py_provides call.

Automatic Provides with a standardized name

When building a Python package, RPM looks for .dist-info and .egg-info files or directories in the %files sections of all packages. If one or more are found, RPM parses them to find the standardized name (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two Provides: tags in the following format:

Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: python3dist(CANONICAL_STANDARDIZED_NAME)

The 3.Y is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a canonical format used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.

Requires and BuildRequires with standardized names

These Provides tags can be used to list Requires and BuildRequires of a package using the standardized names (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the %{py3_dist} macro that accept one or more parameters: the standardized name(s) of the desired Python software. It will convert the name(s) to the canonical format and create the proper python3dist(...) tag(s).

In addition, you can use the %{py_dist_name} macro that simply transforms any standardized name to the canonical format.

For example:

BuildRequires: %{py3_dist PyMySQL} >= 0.7.5
# => BuildRequires: python3dist(pymysql) >= 0.7.5

Requires: %{py3_dist virtualenv pyPEG2}
# => Requires: python3dist(virtualenv) python3dist(pypeg2)

%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0}
# => 0-0

Source Files from PyPI

When packaging software which is available from PyPI, you can make use of the %pypi_source macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are:

  1. The name of the PyPI project. Defaults to %srcname if defined, or to %pypi_name if defined, or to %name (the package name).

  2. The version of the PyPI project. Defaults to %version (the package version) with any ~ characters removed (used for alpha/beta/dev versions in RPM version but not in Python package version).

  3. The file extension to use. Defaults to tar.gz.

In most cases it is not necessary to specify any arguments.

Macros

The following macros are defined for you in all supported Fedora and EPEL releases:

Macro Expanded path Notes

%{__python}

(Error)

Don’t use this macro without redefining it. Defining it changes the meaning of other "unversioned" Python macros such as %{python} or %{python_sitelib}.

%{__python3}

/usr/bin/python3

Python 3 interpreter. Redefining this macro changes all the %{python3...} macros.

%{python3}

%{__python3}

Python 3 interpreter. Use this macro in spec files.

%py_provides

(Lua script)

See The %py_provides macro for detailed explanation.

%{python3_sitelib}

/usr/lib/python3.X/site-packages

Where pure python3 modules are installed.

%{python3_sitearch}

/usr/lib64/python3.X/site-packages on 64bit architectures (e.g. x86_64) and /usr/lib/python3.X/site-packages on 32bit.

Where python3 extension modules (e.g. C compiled) are installed.

%{py_byte_compile}

(script)

See byte-compiling section for usage.

%{python3_version}

3.X

Python 3 version. Useful when running programs with Python version in filename, such as nosetests-%{python3_version}.

%{python3_version_nodots}

3X

Python 3 version without dots. Useful when listing files explicitly in %files section, such as %{python3_sitearch}/foo/_speedups.cpython-%{python3_version_nodots}*.so

%{python3_platform}

linux-x86_64 on x86_64

The platform name used in Python, useful for specifying $PYTHONPATH, such as PYTHONPATH=build/lib.%{python3_platform}-%{python3_version}

%py3_build

%{__python3} setup.py build …

See %py3_install for passing arguments to setup.py build or directly to setup.py.

%py3_install

%{__python3} setup.py install --skip-build …

Various flags are passed to setup.py install, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. To add extra flags/arguments to setup.py install, separate them with --, for example: %py3_install -- --install-scripts %{_libexecdir}. To pass custom command line arguments directly to setup.py, define %py_setup_args.

%__pytest

/usr/bin/pytest

The pytest command used in %pytest. Don’t use this macro directly, but feel free to redefine it for usage in %pytest if desired.

%pytest

PATH=… PYTHONPATH=… … %{__pytest}

Various environment variables are set to ensure the packaged version is tested. Use this macro instead of direct pytest calls in %check. Pass additional argument as if passed to pytest, e.g. %pytest -m "not network" to deselect tests marked as network.

%{py_dist_name}

(Lua script)

Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See Automatic Provides with a standardized name for more information.

%{py3_dist}

(Lua script)

Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to python3dist(CANONICAL_NAME), which is useful when listing dependencies. See Automatic Provides with a standardized name for more information.

%{pypi_source}

(Lua script)

Evaluates to the appropriate URL for the package. See above for more information.

%pycached ….py

(Lua script)

Given a Python file, lists the file and the files with its bytecode cache. See Byte compiling for more information.

%{py3_shebang_flags}

s

The default set of flags for Python shebangs. Redefine or undefine this to change the set. Used by %py3_shebang_fix.

%py3_shebang_fix …

(Python script)

Given paths for Python files or directories with them, it changes Python shebangs to #! %{__python3}, preserves any existing flags (if found) and adds flags defined in %{py3_shebang_flags} (if not already present).

During %install or when listing %files you can use the %{python3_sitearch} and %{python3_sitelib} macros to specify where the installed modules are to be found. For instance:

%files
# A pure python3 module
%{python3_sitelib}/foomodule/
# A compiled python3 extension module
%{python3_sitearch}/barmodule/

Use of the macros has several benefits:

  • It ensures that the packages are installed correctly on multilib architectures.

  • Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed Python version even if the directory structure changes radically (for instance, if python3_sitelib moves into %{_datadir}).

Packages using Cython

A great amount of extension modules for Python (Python modules written in a compiled language such as C or C++) are written using the Cython language and compiler.

Majority of such packages contains the generated C (or C++) sources in the source tarball.

Tightening the general Fedora policy, packages MUST NOT use pre-generated Cython sources. They MUST be deleted in %prep and regenerated during the build.

Any exception to this rule should be considered a bootstrapping.

Files to include

When packaging Python modules, several types of files are included:

  • *.py source files because they are used when generating tracebacks.

  • *.pyc byte compiled files.

    • Python will try to create them at runtime if they don’t exist which leads to spurious SELinux AVC denials in the logs.

    • If the system administrator invokes Python with -OO, they will be created with no docstrings. This can break some programs.

  • *.egg-info or *.dist-info files or directories. If these are generated by the module’s build scripts they must be included in the package because they might be needed by other applications and modules at runtime.

The source files MUST be included in the same package as the byte compiled versions.

Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used:

  • %{python3_sitelib}/*

  • %{python3_sitearch}/*

  • %{python_sitelib}/*

  • %{python_sitearch}/*

And packages MUST NOT include the top-level __pycache__ directory (see below).

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled Python). These files will be located inside a directory named __pycache__.

The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, Python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds in %{python3_sitelib} or %{python3_sitearch} (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them).

You must include the .pyc files in your package. If the build process creates a __pycache__ directory in a subdirectory of %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory. You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.

All that you need to do is include the files in the %files section (replacing %{python3_sitelib} with the appropriate macro for your package):

%files
%{python3_sitelib}/foo/

or, if the Python code installs directly into %{python3_sitelib}, use the %pycached macro to include the bytecode cache files:

%files
%pycached %{python3_sitelib}/foo.py

That evaluates roughly to

%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/foo.cpython-%{python3_version_nodots}{,.opt-?}.pyc
The %pycached macro only supports Python 3.5+, so for older Python versions (such as 3.4 in EPEL 6 or 7), you need to list the files manually.
In case you need to use other macros with the %pycached macro, such as %exclude or %ghost, pass the other macro as part of the argument to %pycached. For example: %pycached %exclude /path/to/foo.py Using the macros in wrong order would only apply %exclude to the first entry that %pycached generates.

Manual byte compilation

For more details on the internals of byte compilation, please see the appendix.

Example Python spec file

The following is a very simple spec file for a Python module.

python-example.spec
%global srcname example

Name:           python-%{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        Example python module

License:        MIT
URL:            https://pypi.python.org/pypi/example
Source0:        %{pypi_source}

BuildArch:      noarch

%global _description %{expand:
A python module which provides a convenient example. This is the
rest of the description that provides more details.}

%description %_description

%package -n python3-%{srcname}
Summary:        %{summary}
BuildRequires:  python3-devel
BuildRequires:  python3-setuptools

%description -n python3-%{srcname} %_description

%prep
%autosetup -n %{srcname}-%{version}

%build
%py3_build

%install
%py3_install

%check
%{python3} setup.py test

# Note that there is no %%files section for the unversioned python module
%files -n python3-%{srcname}
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}-*.egg-info/
%{python3_sitelib}/%{srcname}/
%{_bindir}/sample-exec

Packaging eggs

Please see the Python eggs guidelines for information specific to Python eggs.

Reviewer checklist

The following briefly summarizes the guidelines for reviewers to go over:

  • Must: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See prebuilt binaries Guidelines for details).

  • Must: Python modules must not download any dependencies during the build process.

  • Must: When building a compat package, it must install using easy_install -m so it won’t conflict with the main package.

  • Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.

  • Should: Additional python3-... provides should be accomplished via a %py_provides call.

  • Should: A package which is used by another package via an egg interface should provide egg info.