Handling PEP 517 (pyproject.toml) packages in Gentoo

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We’ve found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.

The design is quite simple — it reads pyproject.toml and calls setuptools’ setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works — it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.