{"id":2375,"date":"2025-07-26T15:29:41","date_gmt":"2025-07-26T13:29:41","guid":{"rendered":"https:\/\/blogs.gentoo.org\/mgorny\/?p=2375"},"modified":"2025-07-26T15:29:41","modified_gmt":"2025-07-26T13:29:41","slug":"epytest_plugins-and-other-goodies-now-in-gentoo","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2025\/07\/26\/epytest_plugins-and-other-goodies-now-in-gentoo\/","title":{"rendered":"EPYTEST_PLUGINS and other goodies now in Gentoo"},"content":{"rendered":"

If\u00a0you are following the\u00a0gentoo-dev<\/a> mailing list, you may have noticed that there’s been a\u00a0fair number of\u00a0patches sent for\u00a0the\u00a0Python eclasses recently. Most of\u00a0them have been centered on\u00a0pytest support. Long story short, I’ve came\u00a0up with what I\u00a0believed to\u00a0be\u00a0a\u00a0reasonably good design, and\u00a0decided it’s time to\u00a0stop manually repeating all the\u00a0good practices in\u00a0every ebuild separately.<\/p>\n

In\u00a0this post, I\u00a0am going to\u00a0shortly summarize all the\u00a0recently added options. As\u00a0always, they are all also documented in\u00a0the\u00a0Gentoo Python Guide<\/a>.
\n<\/p>\n

The\u00a0unceasing fight against plugin autoloading<\/h2>\n

The\u00a0pytest<\/a> test loader defaults to\u00a0automatically loading all the\u00a0plugins installed to\u00a0the\u00a0system. While this is usually quite convenient, especially when you’re testing in\u00a0a\u00a0virtual environment, it\u00a0can get quite messy when you’re testing against system packages and\u00a0end\u00a0up with\u00a0lots of\u00a0different plugins installed. The\u00a0results can range from\u00a0slowing tests down to\u00a0completely breaking the\u00a0test suite.<\/p>\n

Our initial attempts to\u00a0contain the\u00a0situation were based on\u00a0maintaining a\u00a0list of\u00a0known-bad plugins and\u00a0explicitly disabling their autoloading. The\u00a0list of\u00a0disabled plugins<\/a> has gotten quite long by\u00a0now. It\u00a0includes both plugins that were known to\u00a0frequently break tests, and\u00a0these that frequently resulted in\u00a0automagic dependencies.<\/p>\n

While the\u00a0opt-out approach allowed us to\u00a0resolve the\u00a0worst issues, it\u00a0only worked when we\u00a0knew about a\u00a0particular issue. So naturally we’d miss some rarer issue, and\u00a0learn only when\u00a0arch testing workflows were failing, or\u00a0users reported issues. And\u00a0of\u00a0course, we would still be\u00a0loading loads of\u00a0unnecessary plugins at\u00a0the\u00a0cost of\u00a0performance.<\/p>\n

So, we\u00a0started disabling autoloading entirely, using PYTEST_DISABLE_PLUGIN_AUTOLOAD<\/kbd> environment variable. At\u00a0first we\u00a0only used it when we\u00a0needed to, however over time we’ve started using it almost everywhere \u2014 after all, we\u00a0don’t want the\u00a0test suites to\u00a0suddenly start failing because of\u00a0a\u00a0new pytest plugin installed.<\/p>\n

For a\u00a0long time, I\u00a0have been hesitant to\u00a0disable autoloading by\u00a0default. My\u00a0main concern was that it’s easy to\u00a0miss a\u00a0missing plugin. Say, if\u00a0you ended\u00a0up failing to\u00a0load pytest-asyncio<\/kbd> or\u00a0a\u00a0similar plugin, all the\u00a0asynchronous tests would simply be\u00a0skipped (verbosely, but\u00a0it’s still easy to\u00a0miss among the\u00a0flood of\u00a0warnings). However, eventually we\u00a0started treating this warning as\u00a0an\u00a0error (and\u00a0then pytest started doing the\u00a0same upstream), and\u00a0I\u00a0have decided that going opt-in is\u00a0worth the\u00a0risk. After all, we\u00a0were already disabling it all over the\u00a0place anyway.<\/p>\n

EPYTEST_PLUGINS<\/h2>\n

Disabling plugin autoloading is\u00a0only the\u00a0first part of\u00a0the\u00a0solution. Once you disabled autoloading, you need to\u00a0load the\u00a0plugins explicitly \u2014 it’s not\u00a0sufficient anymore to\u00a0add them as\u00a0test dependencies, you also need to\u00a0add a\u00a0bunch of\u00a0-p<\/kbd> switches. And\u00a0then, you need to keep maintaining both dependencies and\u00a0pytest switches in\u00a0sync. So\u00a0you’d end\u00a0up with\u00a0bits like:<\/p>\n

BDEPEND=\"\r\n  test? (\r\n    dev-python\/flaky[${PYTHON_USEDEP}]\r\n    dev-python\/pytest-asyncio[${PYTHON_USEDEP}]\r\n    dev-python\/pytest-timeout[${PYTHON_USEDEP}]\r\n  )\r\n\"\r\n\r\ndistutils_enable_tests pytest\r\n\r\npython_test() {\r\n  local -x PYTEST_DISABLE_PLUGIN_AUTOLOAD=1\r\n  epytest -p asyncio -p flaky -p timeout\r\n}<\/pre>\n
Not\u00a0very efficient, right?  The\u00a0idea then is to\u00a0replace all that with a\u00a0single EPYTEST_PLUGINS<\/kbd> variable:<\/p>\n
EPYTEST_PLUGINS=( flaky pytest-{asyncio,timeout} )\r\ndistutils_enable_tests pytest<\/pre>\n
And\u00a0that’s it!  EPYTEST_PLUGINS<\/kbd> takes a\u00a0bunch of\u00a0Gentoo package names (without category \u2014 almost all of\u00a0them reside in\u00a0dev-python\/<\/kbd>, and\u00a0we can special-case the\u00a0few that do\u00a0not), distutils_enable_tests<\/kbd> adds the\u00a0dependencies and\u00a0epytest<\/kbd> (in\u00a0the\u00a0default python_test()<\/kbd> implementation) disables autoloading and\u00a0passes the\u00a0necessary flags.<\/p>\n
Now, what’s really cool is\u00a0that the\u00a0function will automatically determine the\u00a0correct argument values!  This can be\u00a0especially important if\u00a0entry point names change between package versions \u2014 and\u00a0upstreams generally don’t consider this an\u00a0issue, since autoloading isn’t affected.<\/p>\n
Going towards no\u00a0autoloading by\u00a0default<\/h2>\n
Okay, that gives us a\u00a0nice way of\u00a0specifying which plugins to\u00a0load.  However, weren’t we\u00a0talking of\u00a0disabling autoloading by\u00a0default?<\/p>\n
Well, yes \u2014 and\u00a0the\u00a0intent is\u00a0that it’s going to\u00a0be\u00a0disabled by\u00a0default in\u00a0EAPI\u00a09.  However, until then there’s a\u00a0simple solution we\u00a0encourage everyone to\u00a0use: set an\u00a0empty EPYTEST_PLUGINS<\/kbd>.  So:<\/p>\n
EPYTEST_PLUGINS=()\r\ndistutils_enable_tests pytest<\/pre>\n
\u2026and\u00a0that’s it.  When it’s set to\u00a0an\u00a0empty list, autoloading is disabled.  When it’s unset, it is enabled for backwards compatibility.  And\u00a0the\u00a0next pkgcheck release is\u00a0going to\u00a0suggest it:<\/p>\n
dev-python\/a2wsgi\r\n  EPyTestPluginsSuggestion: version 1.10.10: EPYTEST_PLUGINS can be used to control pytest plugins loaded\r\n<\/pre>\n
EPYTEST_PLUGIN* to\u00a0deal with\u00a0special cases<\/h2>\n
While the\u00a0basic feature is\u00a0neat, it\u00a0is not\u00a0a\u00a0golden bullet.  The\u00a0approach used is\u00a0insufficient for\u00a0some packages, most notably pytest plugins that run a\u00a0pytest subprocesses without\u00a0appropriate -p<\/kbd> options, and\u00a0expect plugins to be\u00a0autoloaded there.  However, after some more fiddling we arrived at\u00a0three helpful features:<\/p>\n
    \n
  1. EPYTEST_PLUGIN_LOAD_VIA_ENV<\/kbd> that switches explicit plugin loading from\u00a0-p<\/kbd> arguments to\u00a0PYTEST_PLUGINS<\/kbd> environment variable.  This greatly increases the\u00a0chance that subprocesses will load the\u00a0specified plugins as\u00a0well, though it\u00a0is more likely to\u00a0cause issues such as\u00a0plugins being loaded twice (and\u00a0therefore is\u00a0not\u00a0the\u00a0default).  And\u00a0as\u00a0a\u00a0nicety, the\u00a0eclass takes care of\u00a0finding\u00a0out the\u00a0correct values, again.<\/li>\n
  2. EPYTEST_PLUGIN_AUTOLOAD<\/kbd> to\u00a0reenable autoloading, effectively making EPYTEST_PLUGINS<\/kbd> responsible only for\u00a0adding dependencies.  It’s really intended to\u00a0be\u00a0used as\u00a0a\u00a0last resort, and\u00a0mostly for\u00a0future EAPIs when autoloading will be\u00a0disabled by\u00a0default.<\/li>\n
  3. Additionally, EPYTEST_PLUGINS<\/kbd> can accept the\u00a0name of\u00a0the\u00a0package itself (i.e.\u00a0${PN}<\/kbd>) \u2014 in\u00a0which case it will not\u00a0add a\u00a0dependency, but\u00a0load the\u00a0just-built plugin.<\/li>\n<\/ol>\n
    How useful is that?  Compare:<\/p>\n
    BDEPEND=\"\r\n  test? (\r\n    dev-python\/pytest-datadir[${PYTHON_USEDEP}]\r\n  )\r\n\"\r\n\r\ndistutils_enable_tests pytest\r\n\r\npython_test() {\r\n  local -x PYTEST_DISABLE_PLUGIN_AUTOLOAD=1\r\n  local -x PYTEST_PLUGINS=pytest_datadir.plugin,pytest_regressions.plugin\r\n  epytest\r\n}<\/pre>\n
    \u2026and:<\/p>\n
    EPYTEST_PLUGINS=( \"${PN}\" pytest-datadir )\r\nEPYTEST_PLUGIN_LOAD_VIA_ENV=1\r\ndistutils_enable_tests pytest<\/pre>\n
    Old and\u00a0new bits: common plugins<\/h2>\n
    The\u00a0eclass already had some bits related to\u00a0enabling common plugins.  Given that EPYTEST_PLUGINS<\/kbd> only takes care of\u00a0loading plugins, but\u00a0not\u00a0passing specific arguments to\u00a0them, they are still meaningful.  Furthermore, we’ve added EPYTEST_RERUNS<\/kbd>.<\/p>\n
    The\u00a0current list is:<\/p>\n
      \n
    1. EPYTEST_RERUNS=...<\/kbd> that takes a\u00a0number of\u00a0reruns and\u00a0uses pytest-rerunfailures<\/a> to\u00a0retry failing tests the\u00a0specified number of\u00a0times.<\/li>\n
    2. EPYTEST_TIMEOUT=...<\/kbd> that takes a\u00a0number of\u00a0seconds and\u00a0uses pytest-timeout<\/a> to\u00a0force a\u00a0timeout if\u00a0a\u00a0single test does not\u00a0complete within the\u00a0specified time.<\/li>\n
    3. EPYTEST_XDIST=1<\/kbd> that enables parallel testing using pytest-xdist<\/a>, if\u00a0the\u00a0user allows multiple test jobs.  The\u00a0number of\u00a0test jobs can be\u00a0controlled (by\u00a0the\u00a0user) by\u00a0setting EPYTEST_JOBS<\/kbd> with a\u00a0fallback to\u00a0inferring from\u00a0MAKEOPTS<\/kbd> (setting to\u00a01 disables the\u00a0plugin entirely).<\/li>\n<\/ol>\n
      The\u00a0variables automatically add the\u00a0needed plugin, so\u00a0they do\u00a0not\u00a0need to\u00a0be\u00a0repeated in\u00a0EPYTEST_PLUGINS<\/kbd>.<\/p>\n
      JUnit XML output and\u00a0gpy-junit2deselect<\/h2>\n
      As\u00a0an\u00a0extra treat, we\u00a0ask pytest to\u00a0generate a\u00a0JUnit-style XML output for\u00a0each test run that can be\u00a0used for\u00a0machine processing of\u00a0test results.  gpyutils<\/a> now supply a\u00a0gpy-junit2deselect<\/kbd> tool that can parse this XML and\u00a0output a\u00a0handy EPYTEST_DESELECT<\/kbd> for\u00a0the\u00a0failing tests:<\/p>\n
      $ gpy-junit2deselect \/tmp\/portage\/dev-python\/aiohttp-3.12.14\/temp\/pytest-xml\/python3.13-QFr.xml\r\nEPYTEST_DESELECT=(\r\n  tests\/test_connector.py::test_tcp_connector_ssl_shutdown_timeout_nonzero_passed\r\n  tests\/test_connector.py::test_tcp_connector_ssl_shutdown_timeout_passed_to_create_connection\r\n  tests\/test_connector.py::test_tcp_connector_ssl_shutdown_timeout_zero_not_passed\r\n)<\/pre>\n
      While it\u00a0doesn’t replace due diligence, it\u00a0can help you update long lists of\u00a0deselects.  As\u00a0a\u00a0bonus, it automatically collapses deselects to\u00a0test functions, classes and\u00a0files when all matching tests fail.<\/p>\n
      hypothesis-gentoo to\u00a0deal with\u00a0health check nightmare<\/h2>\n
      Hypothesis<\/a> is\u00a0a\u00a0popular Python fuzz testing library.  Unfortunately, it\u00a0has one feature that, while useful upstream, is\u00a0pretty annoying to\u00a0downstream testers: health checks.<\/p>\n
      The\u00a0idea behind health checks is\u00a0to\u00a0make sure that fuzz testing remains efficient.  For\u00a0example, Hypothesis is\u00a0going to\u00a0fail if\u00a0the\u00a0routine used to\u00a0generate examples is\u00a0too\u00a0slow.  And\u00a0as\u00a0you can guess, “too slow” is\u00a0more likely to happen on\u00a0a\u00a0busy Gentoo system than on\u00a0dedicated upstream CI.  Not to\u00a0mention some upstreams plain ignore health check failures if\u00a0they happen rarely.<\/p>\n
      Given how often this broke for\u00a0us, we\u00a0have requested an\u00a0option to\u00a0disable Hypothesis health checks<\/a> long ago.  Unfortunately, upstream’s answer can be\u00a0summarized as: “it’s up to\u00a0packages using Hypothesis to\u00a0provide such an\u00a0option, and\u00a0you should not\u00a0be\u00a0running fuzz testing downstream anyway”.  Easy to\u00a0say.<\/p>\n
      Well, obviously we are\u00a0not\u00a0going to\u00a0pursue every single package using Hypothesis to\u00a0add a\u00a0profile with health checks disabled.  We did report health check failures sometimes, and\u00a0sometimes got no\u00a0response at\u00a0all.  And\u00a0skipping these tests is\u00a0not\u00a0really an\u00a0option, given that often there are no\u00a0other tests for\u00a0a\u00a0given function, and\u00a0even if\u00a0there are \u2014 it’s just going to\u00a0be\u00a0a\u00a0maintenance nightmare.<\/p>\n
      I’ve finally figured out that we can create a\u00a0Hypothesis plugin \u2014 now hypothesis-gentoo<\/a> \u2014 that provides a\u00a0dedicated “gentoo” profile with\u00a0all\u00a0health checks disabled, and\u00a0then we\u00a0can simply use this\u00a0profile in\u00a0epytest<\/kbd>.  And\u00a0how do we\u00a0know that Hypothesis is used?  Of\u00a0course we\u00a0look at\u00a0EPYTEST_PLUGINS<\/kbd>!  All pieces fall into\u00a0place.  It’s not\u00a0100% foolproof, but\u00a0health check problems aren’t that common either.<\/p>\n
      Summary<\/h2>\n
      I\u00a0have to\u00a0say that I\u00a0really like what we\u00a0achieved here.  Over the\u00a0years, we\u00a0learned a\u00a0lot about pytest, and\u00a0used that knowledge to\u00a0improve testing in\u00a0Gentoo.  And\u00a0after repeating the\u00a0same patterns for\u00a0years, we\u00a0have finally replaced them with\u00a0eclass functions that can\u00a0largely work out\u00a0of\u00a0the\u00a0box.  This is\u00a0a\u00a0major step forward.<\/p>\n","protected":false},"excerpt":{"rendered":"
      If\u00a0you are following the\u00a0gentoo-dev mailing list, you may have noticed that there’s been a\u00a0fair number of\u00a0patches sent for\u00a0the\u00a0Python eclasses recently. Most of\u00a0them have been centered on\u00a0pytest support. Long story short, I’ve came\u00a0up with what I\u00a0believed to\u00a0be\u00a0a\u00a0reasonably good design, and\u00a0decided it’s time to\u00a0stop manually repeating all the\u00a0good practices in\u00a0every ebuild separately. In\u00a0this post, I\u00a0am going to\u00a0shortly … Continue reading “EPYTEST_PLUGINS and other goodies now in Gentoo”<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[11,3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2375"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=2375"}],"version-history":[{"count":26,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2375\/revisions"}],"predecessor-version":[{"id":2401,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2375\/revisions\/2401"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=2375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=2375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=2375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}