Category: Gentoo

Optimizing distutils-r1.eclass via wheel reuse

Yesterday I’ve enabled a new distutils-r1.eclass optimization: wheel reuse. Without this optimization, the eclass would build a separate wheel for every Python implementation enabled, and then install every one of these wheels. In many cases, this meant repeatedly building the same thing. With the optimization enabled, under some circumstances the eclass will be able to build one (or two) wheels, and install them for all implementations.

This change brings the eclass behavior closer to the behavior of package managers such as pip. While this will cause no change for users who build packages for a single Python version only, it can bring some nice speedup when building for multiple interpreters. Particularly, pure Python packages using setuptools will no longer incur the penalty of having to start setuptools multiple times (which is quite slow), and packages using the stable ABI won’t have to build roughly identical extensions multiple times.

In this post, I’m going to shortly go over a few design considerations of the new feature.

Continue reading “Optimizing distutils-r1.eclass via wheel reuse”

The review-work balance, and other dilemmas

One of the biggest problems of working in such a large project as Gentoo, is that there’s always a lot of work to be done. Once you get engaged deeply enough, no matter how hard you’re going to try, the backlog will just keep growing. There are just so many things that need to be done, and someone has to do them.

Sooner or later, you are going to start facing some dilemmas, such as:

  • This befell me, because nobody else was willing to do that. Should I continue overburdening myself with this, or should I leave it and let it rot?
  • I have more time than other people on the team. Should I continue doing the bulk of the work, or leave more of it to them?
  • What is the right balance between reviewing contributions, and doing the work myself?

In this post, I’d like to discuss these problems from my perspective as a long-time Gentoo developer.
Continue reading “The review-work balance, and other dilemmas”

The dead weight of packages in Gentoo

You’ve probably noticed it already: Gentoo developers are overwhelmed. There is a lot of unresolved bugs. There is a lot of unmaintained packages. There is a lot of open pull requests. This is all true, but it’s all part of a larger problem, and a problem that doesn’t affect Gentoo alone.

It’s a problem that any major project is going to face sooner or later, and especially a project that’s almost entirely relying on volunteer work. It’s a problem of bitrot, of different focus, of energy deficit. And it is a very hard problem to solve.
Continue reading “The dead weight of packages in Gentoo”

Optimizing parallel extension builds in PEP517 builds

The distutils (and therefore setuptools) build system supports building C extensions in parallel, through the use of -j (--parallel) option, passed either to build_ext or build command. Gentoo distutils-r1.eclass has always passed these options to speed up builds of packages that feature multiple C files.

However, the switch to PEP517 build backend made this problematic. While the backend uses the respective commands internally, it doesn’t provide a way to pass options to them. In this post, I’d like to explore the different ways we attempted to resolve this problem, trying to find an optimal solution that would let us benefit from parallel extension builds while preserving minimal overhead for packages that wouldn’t benefit from it (e.g. pure Python packages). I will also include a fresh benchmark results to compare these methods.
Continue reading “Optimizing parallel extension builds in PEP517 builds”

The story of distutils build directory in Gentoo

The Python distutils build system, as well as setuptools (that it was later merged into), used a two-stage build: first, a build command would prepare a built package version (usually just copy the .py files, sometimes compile Python extensions) into a build directory, then an install command would copy them to the live filesystem, or a staging directory. Curious enough, distutils were an early adopter of out-of-source builds — when used right (which often enough wasn’t the case), no writes would occur in the source directory and all modifications would be done directly in the build directory.

Today, in the PEP517 era, two-stage builds aren’t really relevant anymore. Build systems were turned into black boxes that spew wheels. However, setuptools still internally uses the two-stage build and the build directory, and therefore it still remains relevant to Gentoo eclasses. In this post, I’d like to shortly tell how we dealt with it over the years.
Continue reading “The story of distutils build directory in Gentoo”