10 Years’ Perspective on Python in Gentoo

I’m a Gentoo developer for over 10 years already. I’ve been doing a lot of different things throughout that period. However, Python was pretty much always somewhere within my area of interest. I don’t really recall how it all started. Maybe it had something to do with Portage being written in Python. Maybe it was the natural next step after programming in Perl.

I feel like the upcoming switch to Python 3.9 is the last step in the prolonged effort of catching up with Python. Over the last years, we’ve been working real hard to move Python support forward, to bump neglected packages, to enable testing where tests are available, to test packages on new targets and unmask new targets as soon as possible. We have improved the processes a lot. Back when we were switching to Python 3.4, it took almost a year from the first false start attempt to the actual change. We started using Python 3.5 by default after upstream dropped bugfix support for it. In a month from now, we are going to start using Python 3.9 even before 3.10 final is released.

I think this is a great opportunity to look back and see what changed in the Gentoo Python ecosystem, in the last 10 years.

Python package ebuilds 10 years ago

Do you know how a Python package ebuild looked like 10 years ago? Let’s take gentoopm-0.1 as an example (reformatted to fit the narrow layout better):


# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.1.ebuild,v 1.1 2011/07/15 19:05:26 mgorny Exp $

EAPI=3

PYTHON_DEPEND='*:2.6'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS='2.4 2.5'
DISTUTILS_SRC_TEST=setup.py

inherit base distutils

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/gentoopm/"
SRC_URI="http://cloud.github.com/downloads/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.8.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="dev-python/epydoc"
PDEPEND="app-admin/eselect-package-manager"

src_prepare() {
  base_src_prepare
  distutils_src_prepare
}

src_compile() {
  distutils_src_compile

  if use doc; then
    "$(PYTHON -2)" setup.py doc || die
  fi
}

src_install() {
  distutils_src_install

  if use doc; then
    dohtml -r doc/* || die
  fi
}

This ebuild is actually using the newer API of python.eclass that is enabled via SUPPORT_PYTHON_ABIS. It provides support for installing for multiple implementations (like the modern python-r1 eclass). PYTHON_DEPEND is used to control the dependency string added to ebuild. The magical syntax here means that the ebuild supports both Python 2 and Python 3, from Python 2.6 upwards. RESTRICT_PYTHON_ABIS opts out support for Python versions prior to 2.6. Note the redundancy — PYTHON_DEPEND controls the dependency, specified as a range of Python 2 and/or Python 3 versions, RESTRICT_PYTHON_ABIS controls versions used at build time and needs to explicitly exclude all unsupported branches.

Back then, there were no PYTHON_TARGETS to control what was built. Instead, the eclass defaulted to using whatever was selected via eselect python, with the option to override it via setting USE_PYTHON in make.conf. Therefore, there were no cross-package USE dependencies and you had to run python-updater to verify whether all packages are built for the current interpreter, and rebuild these that were not.

Still, support for multiple ABIs, as the eclass called different branches/implementations of Python, was a major step forward. It was added around the time that the first releases of Python 3 were published, and our users have been relying on it to support a combination of Python 2 and Python 3 for years. Today, we’re primarily using it to aid developers in testing their packages and to provide a safer upgrade experience.

The python.eclass stalemate

Unfortunately, things at the time were not all that great. Late 2010 marks a conflict between the primary Python developer and the rest of the community, primarily due to the major breakage being caused by the changes in Python support. By mid-2011, it was pretty clear that there is no chance to resolve the conflict. The in-Gentoo version of python.eclass was failing to get EAPI 4 support for 6 months already, while an incompatible version continued being developed in the (old) Python overlay. As Dirkjan Ochtman related in his mail:

I guess by now pretty much everyone knows that the python eclass is rather complex, and that this poses some problems. This has also been an important cause for the disagreements between Arfrever and some of the other developers. Since it appears that Arfrever won’t be committing much code to gentoo-x86 in the near future, I’m trying to figure out where we should go with the python.eclass. […]

Dirkjan Ochtman, 2011-06-27, [gentoo-dev] The Python problem

Eventually, some of the changes from the Python overlay were backported to the eclass and EAPI 4 support was added. Nevertheless, at this point it was pretty clear that we need a new way forward. Unfortunately, the discussions were leading nowhere. With the primary eclass maintainer retired, nobody really comprehended most of the eclass, nor were able to afford the time to figure it out. At the same time, involved parties wanted to preserve backwards compatibility while moving forward.

The tie breaker: python-distutils-ng

Some of you might find it surprising that PYTHON_TARGETS are not really a python-r1 invention. Back in March 2012, when Python team was still unable to find a way forward with python.eclass, Krzysztof Pawlik (nelchael) has committed a new python-distutils-ng.eclass. It has never grown popular, and it has been replaced by the python-r1 suite before it ever started being a meaningful replacement for python.eclass. Still, it served an important impulse that made what came after possible.

Here’s a newer gentoopm ebuild using the new eclass (again reformatted):


# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.2.5-r1.ebuild,v 1.1 2012/05/26 10:11:21 mgorny Exp $

EAPI=4
PYTHON_COMPAT='python2_6 python2_7 python3_1 python3_2'

inherit base python-distutils-ng

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI="mirror://github/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~mips ~x86 ~x86-fbsd"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.10.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="doc? ( dev-python/epydoc )"
PDEPEND="app-admin/eselect-package-manager"

python_prepare_all() {
  base_src_prepare
}

src_compile() {
  python-distutils-ng_src_compile
  if use doc; then
    "${PYTHON}" setup.py doc || die
  fi
}

python_install_all() {
  if use doc; then
    dohtml -r doc/*
  fi
}

Just looking at the code, you may see that python-r1 has inherited a lot after this eclass. python-distutils-ng in turn followed some of the good practices introduced before in the ruby-ng eclass. It introduced PYTHON_TARGETS to provide explicit visible control over implementations used for the build — though notably it did not include a way for packages to depend on matching flags (i.e. the equivalent of ${PYTHON_USEDEP}). It also used the sub-phase approach that makes distutils-r1 and ruby-ng eclasses much more convenient than the traditional python.eclass approach that roughly resembled using python_foreach_impl all the time.

What’s really important is that python-distutils-ng carved a way forward. It’s been a great inspiration and a proof of concept. It has shown that we do not have to preserve compatibility with python.eclass forever, or have to learn its inner workings before starting to solve problems. I can’t say how Python would look today if it did not happen but I can say with certainly that python-r1 would not happen so soon if it weren’t for it.

python-r1

In October 2012, the first version of python-r1 was committed. It combined some of the very good ideas of python-distutils-ng with some of my own.  The goal was not to provide an immediate replacement for python.eclass. Instead, the plan was to start simple and add new features as they turned out to be necessary. Not everything went perfectly but I dare say that the design has stood the test of time. While I feel like the eclasses ended up being more complex than I wished they would be, they still work fine with no replacement in sight and they serve as inspiration to other eclasses.

For completeness, here’s a 2017 gentoopm live ebuild that uses a pretty complete distutils-r1 feature set:


# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6
PYTHON_COMPAT=( python{2_7,3_4,3_5,3_6} pypy )

EGIT_REPO_URI="https://github.com/mgorny/gentoopm.git"
inherit distutils-r1 git-r3

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI=""

LICENSE="BSD-2"
SLOT="0"
KEYWORDS=""
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/pkgcore-0.9.4[${PYTHON_USEDEP}]
    >=sys-apps/portage-2.1.10.3[${PYTHON_USEDEP}]
    >=sys-apps/paludis-3.0.0_pre20170219[python,${PYTHON_USEDEP}]
  )"
DEPEND="
  doc? (
    dev-python/epydoc[$(python_gen_usedep python2_7)]
  )"
PDEPEND="app-eselect/eselect-package-manager"

REQUIRED_USE="
  doc? ( $(python_gen_useflags python2_7) )"

src_configure() {
  use doc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 )
  distutils-r1_src_configure
}

python_compile_all() {
  use doc && esetup.py doc
}

python_test() {
  esetup.py test
}

python_install_all() {
  use doc && local HTML_DOCS=( doc/. )
  distutils-r1_python_install_all
}

The new eclasses have employed the same PYTHON_TARGETS flags for the general implementation choice but also added PYTHON_SINGLE_TARGET to make choosing the implementation more convenient (and predictable at the same time) when the package did not permit choosing more than one. The distutils-r1 eclass reused the great idea of sub-phases to make partial alterations to the phases easier.

The unique ideas included:

  • the split into more eclasses by functionality (vs switching the mode via variables as done in python.eclass)
  • exposing dependencies and REQUIRED_USE constraints via variables to be used in ebuild instead of writing elaborate mechanisms for adding dependencies
  • avoiding command substitution in global scope to keep metadata regeneration fast

The eclasses evolved a lot over the years. Some of the original ideas turned out pretty bad, e.g. trying to run sub-phases in parallel (which broke a lot of stuff for minor performance gain) or the horribly complex original interaction between PYTHON_TARGETS and PYTHON_SINGLE_TARGET. Ebuilds often missed dependencies and REQUIRED_USE constraints, until we finally made the pkgcheck-based CI report that.

The migration to new eclasses took many years. Initially, python-r1 ebuilds were even depending on python.eclass ebuilds. The old eclass was not removed until March 2017, i.e. 4.5 years after introducing the new eclasses.

Testing on new Python targets

We went for the opt-in approach with PYTHON_COMPAT. This means that for every new Python target added, we start with no packages supporting it and have to iterate over all of the packages adding the support. It’s a lot of work and it has repeatedly caused users pain due to packages not being ported in time for the big switch to the next version. Some people have complained about that and suggested that we should go for opt-out instead. However, if you think about it, opt-in is the only way to go.

The big deal is that for any particular package to support a new implementation, all of its Python dependencies need to support it as well. With the opt-in approach, it means that we’re doing the testing dependency-first, and reaching the bigger packages only when we confirm that there’s at least a single version of every dependency that works for it. If we do things right, users don’t even see any regressions.

If we went for the opt-out approach, all packages would suddenly claim to support the new version. Now, this wouldn’t be that bad if we were actually able to do a big CI run for all packages — but we can’t since a lot of them do not have proper tests, and Python version incompatibility often can’t be detected statically. In the end, we would be relying on someone (possibly an user) reporting that something is broken. Then we’d have to investigate where in the dependency chain the culprit is, and either restrict the new target (and then restrict it in all its reverse dependencies) or immediately fix it.

So in the end, opt-out would be worse for both users and developers. Users would hit package issues first hand, and developers would have to spend significant time on the back-and-forth effort of removing support for new targets, and then adding it again. If we are to add new targets early (which is a worthwhile goal), we have to expect incompatible packages. My experience so far shows that Gentoo developers sometimes end up being the first people to submit patches fixing the incompatibility. This can be a real problem given that many Python packages have slow release cycles, and are blocking their reverse dependencies, and these in turn block their reverse dependencies and so on.

Now, things did not always go smoothly. I have prepared a Python release and Gentoo packaging timeline that puts our packaging work into perspective. As you can see, we were always quite fast in packaging new Python interpreters but it took a significant time to actually switch the default targets to them — in fact, we often switched just before or even after upstream stopped providing bug fixes to the version in question.

Our approach has changed over the years. Early on, we generally kept both the interpreter and the target in ~arch, and stabilized them in order to switch targets. The prolonged stable-mask of the new target has resulted in inconsistent presence of support for the new target in stable, and this in turn involved a lot of last-minute stabilization work. Even then, we ended up switching targets before stable was really ready for that. This was particularly bad for Python 3.4 — the period seen as ‘stable’ on the timeline is actually a period following the first unsuccessful switch of the default. It took us over half a year to try again.

Then (around Python 3.6, if I’m not mistaken) we switched to a different approach. Instead of delaying till the expected switch, we’ve tried to unmask the target on stable systems as soon as possible. This way, we started enforcing dependency graph consistency earlier and were able to avoid big last minute stabilizations needed to unmask the target.

Eventually, thanks to pkgcheck’s superior StableRequestCheck, I’ve started proactively stabilizing new versions of Python packages. This was probably the most important improvement of all. The stabilization effort was streamlined, new versions of packages gained stable keywords sooner and along them did the support for new Python implementations.

The effort in package testing and stabilizations have finally made it possible to catch up with upstream. We have basically gone through a target switching sprint, moving through 3.7 and 3.8 in half a year each. The upcoming switch to Python 3.9 concludes this effort. For the first time in years, our default will be supported upstream for over 6 months.

That said, it is worth noting that things are not becoming easier for us. Over time, Python packages keep getting new dependencies, and this means that every new Python version will involve more and more porting work. Unfortunately, some high profile packages such as setuptools and pytest keep creating bigger and bigger dependency loops. At this point, it is no longer reasonable to attempt to port all the cyclic dependencies simultaneously. Instead, I tend to temporarily disable tests for the initial ports to reduce the number of dependencies. I’ve included a list of suggested initial steps in the Python Guide to ease future porting efforts.

Packaging the Python interpreter

CPython is not the worst thing to package but it’s not the easiest one either. Admittedly, the ebuilds were pretty good back when I joined. However, we’ve already carried a pretty large and mostly undocumented set of patches, and most of these patches we carry up to this day, with no chance of upstreaming them.

Some of the involved patches are build fixes and hacks that are either specific to Gentoo, or bring the flexibility Gentoo cares about (such as making some of the USE flags possible). There are also some fixes for old bugs that upstream has not shown any interest in fixing.

Another part of release management is resolving security bugs. Until recently, we did not track vulnerabilities in CPython very well. Thanks to the work of new Security team members, we have started being informed of vulnerabilities earlier. At the same time, we realized that CPython’s treatment of vulnerabilities is a bit suboptimal.

Admittedly, when very bad things happen upstream releases fixes quickly. However, non-critical vulnerability fixes are released as part of the normal release cycle, approximately every two months. For some time already, every new release contained some security fixes and had to be stabilized quickly. At the same time it contained many other changes with their own breakage potential. Old Python branches were even worse — absurdly, even though these versions received security fixes only, the releases were even rarer.

In the end, I’ve decided that it makes more sense to backport security fixes to our patchset as soon as I become aware of them, and stabilize the patch bumps instead. I do this even if upstream makes a new release at the same time, since patch bumps are safer stable targets. Even then, some of the security fixes actually require changing the behavior. To name a few recent changes:

  • urllib.parse.parse_qsl() historically used to split the URL query string on either & or ;. This somewhat surprising behavior was changed to split only on &, with a parameter to change the separator (but no option to restore the old behavior).
  • urllib.parse.urlparse() historically preserved newlines, CRs and tabs. In the latest Python versions this behavior was changed to follow a newer recommendation of stripping these characters. As a side effect, some URL validators (e.g. in Django) suddenly stopped rejecting URLs with newlines.
  • The ipaddress module recently stopped allowing leading zeros in IPv4 addresses. These were accepted before but some of the external libraries were incidentally interpreting them as octal numbers.

Some of these make you scratch your head.

Unsurprisingly, this is also a lot of work. At this very moment, we are maintaining six different slots of CPython (2.7, 3.6, 3.7, 3.8, 3.9, 3.10). For every security backport set, I start by identifying the security-related commits on the newest release branch. This is easy if they’re accompanied by a news entry in Security category — unfortunately, some vulnerability fixes were treated as regular bug fixes in the past. Once I have a list of commits, I cherry-pick them to our patched branch and make a patchset out of that. This is the easy part.

Now I have to iterate over all the older versions. For maintained branches, the first step is to identify if upstream has already backported their fixes. if they did, I cherry-pick the backport commits from the matching branch. If they did not, I need to verify if the vulnerability applies to this old version, and cherry-pick from a newer version. Sometimes it requires small changes.

The hardest part is Python 2.7 — it is not supported upstream but still used as a build-time dependency by a few projects. The standard library structure in Python 2 differs a lot from the one in Python 3 but many of Python 3 vulnerabilities stem from code dating back to Python 2. In the end, I have to find the relevant piece of original code in Python 2, see if it is vulnerable and usually rewrite the patch for the old code.

I wish this were the end. However, in Gentoo we are also packaging PyPy, the alternative Python implementation. PyPy follows its own release cycle. PyPy2 is based on now-unmaintained Python 2, so most of the vulnerability fixes from our Python 2.7 patchset need to be ported to it. PyPy3 tries to follow Python 3’s standard library but is not very fast at it. In the end, I end up cherry-picking the changes from CPython to PyPy for our patchsets, and at the same time sending them upstream’s way so that they can fix the vulnerabilities without having to sync with CPython immediately.

Historically, we have also supported another alternative implementation, Jython. However, for a very long time upstream’s been stuck on Python 2.5 and we’ve eventually dropped support for Jython as it became unmaintainable. Upstream has eventually caught up with Python 2.7 but you can imagine how useful that is today.

There are other interesting projects out there but I don’t think we have the manpower (or even a very good reason) to work on them.

Now, a fun fact: Victor Stinner indicates in his mail Need help to fix known Python security vulnerabilities that the security fixes CPython receives are only a drop in the ocean. Apparently, at the time of writing there were 78 open problems of various severity.

A bus and a phoenix

Any project where a single developer does a sufficiently large portion of work has a tendency towards a bus factor of one. The Python project was no different. Ten years ago we were relying on a single person doing most of the work. When that person retired, things started going out of hand. Python ended up with complex eclasses that nobody wholly understood and that urgently needed updates. Many packages were getting outdated.

I believe that we did the best that we could at the time. We’ve started the new eclass set because it was much easier to draw a clear line and start over. We ended up removing many of the less important or more problematic packages, from almost the whole net-zope category (around 200 packages) to temporarily removing Django (only to readd it later on). Getting everything at least initially ported took a lot of time. Many of the ports turned out buggy, a large number of packages were outdated, unmaintained, missing test suites.

Today I can finally say that the Python team’s packages standing is reasonably good. However, in order for it to remain good we need to put a lot of effort every day. Packages need to be bumped. Bumps often add new dependencies. New Python versions require testing. Test regressions just keep popping up, and often when you’re working on something else. Stabilizations need to be regularly tracked, and they usually uncover even more obscure test problems. Right now it’s pretty normal for me to spend an hour of my time every day just to take care of the version bumps.

I have tried my best to avoid returning to a bus factor of one. I have tried to keep the eclasses simple (but I can’t call that a complete success), involve more people in the development, keep things well documented and packages clean. Yet I still end up doing a very large portion of work in the team. I know that there are other developers who can stand in for me but I’m not sure if they will be able to take up all the work, given all their other duties.

We really need young blood. We need people who would be able to dedicate a lot of time specifically to Python, and learn all the intricacies of Python packaging. While I’ve done my best to document everything I can think of in the Python Guide, it’s still very little. The Python ecosystem is a very diverse one, and surprisingly hard to maintain without burning out. Of course, there are many packages that are pleasure to maintain, and many upstreams that are pleasure to work with. Unfortunately, there are also many projects that make you really frustrated — with bad quality code, broken tests, lots of NIH dependencies and awful upstreams that simply hate packagers.

Over my 10 (and a half) years as a developer, I have done a lot of different things. However, if I were to point one area of Gentoo where I put most of my effort, that area would be Python. I am proud of all that we’ve been able to accomplish, and how great our team is. However, there are still many challenges ahead of us, as well as a lot of tedious work.

6 Replies to “10 Years’ Perspective on Python in Gentoo”

  1. Let me be the first one here to thank you for all the hard work you’ve put into Python stuff in Gentoo over the years. It’s downright Herculean =)

    And also thanks for this gist of Python ebuilding history, that `$PYTHON_DEPEND` definitely brings some memories!

  2. Thanks. And I mean it.

    Bloody youth for silly snakes and boring coffee beans, where are thou?

  3. To be honest, Gentoo’s handling of Python is still the reason today I still run Gentoo.

    Some places, I’ve moved to Debian/Ubuntu (mainly my work PC and my netbook; the former to mirror what my colleagues are running, the latter because a Atom N450 is no speed demon), and in others, AlpineLinux is slowly creeping in on servers (OpenRC for the win) and OpenBSD on routers/firewalls, but my main workhorse has been Gentoo since about 2005.

    A big reason for this is its flexibility… another big reason that became important when I started doing Python code at work was how Gentoo handles Python.: *Every* version of Python is treated as a first-class citizen and installing a Python module from the Portage tree results in a build for every installed and compatible Python interpreter.

    In Debian, OpenBSD, AlpineLinux, and everywhere else, you have “python2” packages and “python3” packages. pypy is available on Debian/Ubuntu, but hardly any modules are packaged for it, and with no support in tools like stdeb, it’s a lot of piss-farting about to build it yourself.

    Gentoo: it’s almost so natural you don’t even think about it. `emerge dev-python/foobar`… done… for python2.7, python3.6-3.9, pypy3… whatever you have installed.

    It’s the Python team, including people like yourself, that I have to thank for that. Please keep up the good work.

  4. I’m impressed by where Python support in Gentoo went. I… *cough* might have had a bit of a forcing hand in the early days of Python 3 support (not my shiniest moments, let’s be honest), but it was never a positive addition.

    And congratulations on ten years in the project! It seems like yesterday that we have butted heads about breakages 😀

  5. you need to spend few days and write a script to automate the bumping process. It should be possible to detect a new version, “cp” the ebuild, verify dependencies, run “test” and so *automatically*.

    1. I honestly doubt the Python ecosystem is suitable for that. Even if you get it to work reliably 90% of the time, the time needed to verify stuff and fix the mistakes would exceed the time spent on bumping things manually.

Leave a Reply

Your email address will not be published. Required fields are marked *