{"id":1249,"date":"2020-10-21T21:52:38","date_gmt":"2020-10-21T19:52:38","guid":{"rendered":"http:\/\/blogs.gentoo.org\/mgorny\/?p=1249"},"modified":"2021-10-01T10:35:42","modified_gmt":"2021-10-01T08:35:42","slug":"distutils_use_setuptools-qa-spam-and-more-qa-spam","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2020\/10\/21\/distutils_use_setuptools-qa-spam-and-more-qa-spam\/","title":{"rendered":"DISTUTILS_USE_SETUPTOOLS, QA spam and\u2026 more QA spam?"},"content":{"rendered":"<p><ins>Update: the information provided in this post is out of date.  As of today, Python 3.7 is no longer relevant from DISTUTILS_USE_SETUPTOOLS perspective, and &#8216;rdepend&#8217; is no longer valid when when entry points are used.<\/ins><\/p>\n<p>I suppose that most of the Gentoo developers have seen at least one of the \u2018uses a probably incorrect DISTUTILS_USE_SETUPTOOLS value\u2019 bugs by now.  Over 350 have been filed so far, and new ones are filed practically daily.  The truth is, I&#8217;ve never intended for this QA check to result in bugs being filed against packages, and certainly not that many bugs.<\/p>\n<p>This is not an important problem to be fixed immediately.  The vast majority of Python packages depend on setuptools at build time (this is why the build-time dependency is the eclass&#8217; default), and being able to unmerge setuptools is not a likely scenario.  The underlying idea was that the QA check would make it easier to update <kbd>DISTUTILS_USE_SETUPTOOLS<\/kbd> when bumping packages.<\/p>\n<p>Nobody has asked me for my opinion, and now we have hundreds of bugs that are not very helpful.  In fact, the effort involved in going through all the bugmail, updating packages and closing the bugs greatly exceeds the negligible gain.  Nevertheless, some people actually did it.  I have bad news for them: setuptools upstream has changed entry point mechanism, and most of the values will have to change again.  Let me elaborate on that.<br \/>\n<!--more--><\/p>\n<h2>The current logic<\/h2>\n<p>The current eclass logic revolves around three primary values:<\/p>\n<ul>\n<li><kbd>no<\/kbd> indicating that the package does not use setuptools<\/li>\n<li><kbd>bdepend<\/kbd> indicating that the package uses setuptools at build time only<\/li>\n<li><kbd>rdepend<\/kbd> indicating that the package uses setuptools at build- and runtime<\/li>\n<\/ul>\n<p>There&#8217;s also support for <kbd>pyproject.toml<\/kbd> but it&#8217;s tangential to the problem at hand, so let&#8217;s ignore it.<\/p>\n<p>The setuptools package \u2014 besides the build system \u2014 includes a <kbd>pkg_resources<\/kbd> sub-package that can be used to access package&#8217;s metadata and resources.  The two primary uses of <kbd>rdepend<\/kbd> revolves around this.  These are:<\/p>\n<ol>\n<li><kbd>console_scripts<\/kbd> entry points \u2014 i.e. autogenerated executable scripts that call a function within the installed package rather than containing the program code itself.<\/li>\n<li>Direct uses of <kbd>pkg_resources<\/kbd> in the modules installed by the package.<\/li>\n<\/ol>\n<p>Both of these cases were equivalent from dependency standpoint.  Well, not anymore.<\/p>\n<h2>Entry points via importlib.metadata<\/h2>\n<p>Well, the big deal is the <a rel=\"external\" href=\"https:\/\/docs.python.org\/3.8\/library\/importlib.metadata.html\">importlib.metadata<\/a> module that was added in Python 3.8 (there&#8217;s also a relevant <a rel=\"external\" href=\"https:\/\/docs.python.org\/3.7\/library\/importlib.html#module-importlib.resources\">importlib.resources<\/a> module since Python 3.7).  It is a built-in module that provides routines to access the installed package metadata, and therefore renders another part of <kbd>pkg_resources<\/kbd> redundant.<\/p>\n<p>The big deal is that the new versions of setuptools have embraced it, and no longer require <kbd>pkg_resources<\/kbd> to run entry points.  To be more precise, the new logic selects the built-in module as the first choice, with fallback to the <kbd>importlib_metadata<\/kbd> backport and finally to <kbd>pkg_resources<\/kbd>.<\/p>\n<p>This means that the vast majority of packages that used to depend on setuptools at runtime, no longer does strictly that.  With Python 3.8 and newer, they have no additional runtime dependencies and just require setuptools at build time.  With older versions of Python, they prefer importlib_metadata over it.  In both cases, the packages can still use pkg_resources directly though.<\/p>\n<h2>How to resolve it via the eclass?<\/h2>\n<p>Now, technically speaking this means replacing <kbd>rdepend<\/kbd> with three new variants:<\/p>\n<ul>\n<li><kbd>scripts<\/kbd> \u2014 that means build-time dependency on setuptools + runtime impl-conditional dep on importlib_metadata, for pure entry point usage.<\/li>\n<li><kbd>rdepend<\/kbd> \u2014 that means runtime dependency on setuptools, for pure pkg_resources usage.<\/li>\n<li><kbd>scripts+rdepend<\/kbd> \u2014 for packages that combine both.<\/li>\n<\/ul>\n<p>Of course, this means that the existing packages would get a humongous number of new bug reports, often requesting a change to the value that was updated recently.  The number could be smaller if we changed the existing meaning of <kbd>rdepend<\/kbd> to mean importlib.metadata, and introduced a new value for pkg_resources.<\/p>\n<p>Still, that&#8217;s not the best part.  The real fun idea is that once we remove Python 3.7, all Python versions would have importlib.metadata built-in and the distinction will no longer be necessary.  Eventually, everyone would have to update the value again, this time to <kbd>bdepend<\/kbd>.  Great, right?<\/p>\n<h2>\u2026or not to resolve it?<\/h2>\n<p>Now that we&#8217;ve discussed the solution recommended to me, let&#8217;s consider an alternative.  For the vast majority of packages, the runtime dependency on setuptools is unnecessary.  If the user uses Python 3.8+ or has importlib_metadata installed (which is somewhat likely, due to direct dependencies on it), pkg_resources will not be used by the entry points.  Nevertheless, setuptools is still pretty common as a build-time dependencies and, as I said before, it makes little sense to uninstall it.<\/p>\n<p>We can simply keep things as-is.  Sure, the dependencies will not be 100% optimal.  Yet, the dependency on setuptools will ensure that entry points continue working even if the user does not have importlib_metadata installed.  We will eventually want to update DISTUTILS_USE_SETUPTOOLS logic but we can wait for it till Python versions older than 3.8 become irrelevant, and we are back to three main variants.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Update: the information provided in this post is out of date. As of today, Python 3.7 is no longer relevant from DISTUTILS_USE_SETUPTOOLS perspective, and &#8216;rdepend&#8217; is no longer valid when when entry points are used. I suppose that most of the Gentoo developers have seen at least one of the \u2018uses a probably incorrect DISTUTILS_USE_SETUPTOOLS &hellip; <a href=\"https:\/\/blogs.gentoo.org\/mgorny\/2020\/10\/21\/distutils_use_setuptools-qa-spam-and-more-qa-spam\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;DISTUTILS_USE_SETUPTOOLS, QA spam and\u2026 more QA spam?&#8221;<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[11],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1249"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=1249"}],"version-history":[{"count":21,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1249\/revisions"}],"predecessor-version":[{"id":1643,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1249\/revisions\/1643"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=1249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=1249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=1249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}