Handling PEP 517 (pyproject.toml) packages in Gentoo

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.
Continue reading “Handling PEP 517 (pyproject.toml) packages in Gentoo”

A distribution kernel for Gentoo

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Continue reading “A distribution kernel for Gentoo”

A better ebuild workflow with pure git and pkgcheck

Many developers today continue using repoman commit as their primary way of committing to Gentoo. While this tool was quite helpful, if not indispensable in times of CVS, today it’s a burden. The workflow using a single serial tool to check your packages and commit to them is not very efficient. Not only it wastes your time and slows you down — it discourages you from splitting your changes into more atomic commits.

Upon hearing the pkgcheck advocacy, many developers ask whether it can commit for you. It won’t do that, that’s not its purpose. Not only it’s waste of time to implement that — it would actually make it a worse tool. With its parallel engine pkgcheck really shines when dealing with multiple packages — forcing it to work on one package is a waste of its potential.

Rather than trying to proliferate your bad old habits, you should learn how to use git and pkgcheck efficiently. This post aims to give you a few advices.

Continue reading “A better ebuild workflow with pure git and pkgcheck”

Improving distfile mirror structure

The Gentoo distfile mirror network is essential in distributing sources to our users. It offloads upstream download locations, improves throughput and reliability, guarantees distfile persistency.

The current structure of distfile mirrors dates back to 2002. It might have worked well back when we mirrored around 2500 files but it proved not to scale well. Today, mirrors hold almost 70 000 files, and this number has been causing problems for mirror admins.

The most recent discussion on restructuring mirrors started in January 2015. I have started the preliminary research in January 2017, and it resulted in GLEP 75 being created in January 2018. With the actual implementation effort starting in October 2019, I’d like to summarize all the data and update it with fresh statistics.

Continue reading