{"id":1072,"date":"2019-12-24T23:59:58","date_gmt":"2019-12-24T22:59:58","guid":{"rendered":"https:\/\/blogs.gentoo.org\/mgorny\/?p=1072"},"modified":"2019-12-25T00:37:12","modified_gmt":"2019-12-24T23:37:12","slug":"handling-pep-517-pyproject-toml-packages-in-gentoo","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2019\/12\/24\/handling-pep-517-pyproject-toml-packages-in-gentoo\/","title":{"rendered":"Handling PEP 517 (pyproject.toml) packages in\u00a0Gentoo"},"content":{"rendered":"

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools<\/a>. All those solutions provided a setup.py<\/code> script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass<\/code>. PEP 517<\/a> changed that.<\/p>\n

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml<\/code> file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.<\/p>\n

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.
\n
\nInstead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml<\/code> files into setup.py<\/code> scripts. Since the new formats are declarative, this should not be that hard.<\/p>\n

We’ve found poetry-setup<\/a> project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell<\/a>. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.<\/p>\n

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy<\/a> is a small-ish (<100 SLOC) pyproject.toml<\/code>-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.<\/p>\n

The design is quite simple \u2014 it reads pyproject.toml<\/code> and calls setuptools’ setup()<\/code> function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py<\/code> script in other packages. In fact, this is how its own setup.py<\/code> works \u2014 it carries flit-compatible pyproject.toml<\/code> and uses itself to install itself via setuptools.<\/p>\n

dev-python\/pyproject2setuppy<\/code> is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml<\/code> packages should be as simple as adding the following declaration into ebuilds:<\/p>\n

DISTUTILS_USE_SETUPTOOLS=pyproject.toml\n<\/code><\/pre>\n
This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world.  It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.<\/p>\n","protected":false},"excerpt":{"rendered":"
So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that. Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: … Continue reading “Handling PEP 517 (pyproject.toml) packages in\u00a0Gentoo”<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1072"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=1072"}],"version-history":[{"count":15,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1072\/revisions"}],"predecessor-version":[{"id":1087,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1072\/revisions\/1087"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=1072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=1072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=1072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}