One jobserver to rule them all

A common problem with running Gentoo builds is concurrency. Many packages include extensive build steps that are either fully serial, or cannot fully utilize the available CPU threads throughout. This problem becomes less pronounced when running building multiple packages in parallel, but then we are risking overscheduling for packages that do take advantage of parallel builds.

Fortunately, there are a few tools at our disposal that can improve the situation. Most recently, they were joined by two experimental system-wide jobservers: guildmaster and steve. In this post, I’d like to provide the background on them, and discuss the problems they are facing.
Continue reading “One jobserver to rule them all”

How we incidentally uncovered a 7-year old bug in gentoo-ci

“Gentoo CI” is the service providing periodic linting for the Gentoo repository. It is a part of the Repository mirror and CI project that I’ve started in 2015. Of course, it all started as a temporary third-party solution, but it persisted, was integrated into Gentoo Infrastructure and grew organically into quite a monstrosity.

It’s imperfect in many ways. In particular, it has only some degree of error recovery and when things go wrong beyond that, it requires a manual fix. Often the “fix” is to stop mirroring a problematic repository. Over time, I’ve started having serious doubts about the project, and proposed sunsetting most of it.

Lately, things have been getting worse. What started as a minor change in behavior of Git triggered a whole cascade of failures, leading to me finally announcing the deadline for sunsetting the mirroring of third-party repositories, and starting ripping non-critical bits out of it. Interesting enough, this whole process led me to finally discover the root cause of most of these failures — a bug that has existed since the very early version of the code, but happened to be hidden by the hacky error recovery code. Here’s the story of it.

Continue reading “How we incidentally uncovered a 7-year old bug in gentoo-ci”

EPYTEST_PLUGINS and other goodies now in Gentoo

If you are following the gentoo-dev mailing list, you may have noticed that there’s been a fair number of patches sent for the Python eclasses recently. Most of them have been centered on pytest support. Long story short, I’ve came up with what I believed to be a reasonably good design, and decided it’s time to stop manually repeating all the good practices in every ebuild separately.

In this post, I am going to shortly summarize all the recently added options. As always, they are all also documented in the Gentoo Python Guide.
Continue reading “EPYTEST_PLUGINS and other goodies now in Gentoo”

Poetry(-core), or the ultimate footgun

I’ve been complaining about the Poetry project a lot, in particular about its use (or more precisely, the use of poetry-core) as a build system. In fact, it pretty much became a synonym of a footgun for me — and whenever I’m about to package some project using poetry-core, or switching to it, I’ve learned to expect some predictable mistake. I suppose the time has come to note all these pitfalls in a single blog post.

Continue reading “Poetry(-core), or the ultimate footgun”

The peculiar world of Gentoo package testing

While discussing uv tests with Fedora developers, it occurred to me how different your average Gentoo testing environment is — not only from these used upstream, but also from these used by other Linux distributions. This article will be dedicated exactly to that: to pointing out how it’s different, what does that imply and why I think it’s not a bad thing.
Continue reading “The peculiar world of Gentoo package testing”