In November 2025, I’ve started working on steve, the jobserver for Gentoo. I wrote about it already, in the “One jobserver to rule them all” post. Back then, my main focus was on the motivation for a system-wide jobserver, and the technical details of getting it working. I have also mentioned a few client bugs we’ve discovered along the way.
Since then, we’ve found a few more bugs, as well as problematic design patterns. I think they’re kind of interesting, so I’ve decided to dedicate this post specifically to them. As a disclaimer, my aim is not to pick on specific projects; I’m bringing them up as real examples of what we’ve hit, and how that impacts jobserver operation.
Missing token release on error
As I’ve mentioned in the previous post already, the POSIX jobserver implementation relies on clients fully accounting for tokens. Clients “acquire” (read) tokens when they need to start a job, and they “release” them (write them back) when the job’s done; and they must eventually release all the tokens, or they’re going to be lost and other clients won’t be able to acquire them. The documentation is pretty explicit on that:
Your tool should be sure to write back the tokens it read, even under error conditions. This includes not only errors in your tool but also outside influences such as interrupts (
SIGINT), etc. You may want to install signal handlers to manage this write-back.
Is this hard? Well, it depends. For a start, there’s a lot of signals to cover. There are the more obvious ones, like SIGINT and SIGTERM. There are the less obvious ones, like SIGSEGV and SIGILL; I mean, do you really need to catch them? I’d say a robust implementation has to. And then there is SIGKILL that you really can’t catch.
Given that you can’t catch SIGKILL, an indiscriminate OOM-kill could easily result in tokens being lost. Hence, the whole architecture of steve is built upon the idea that we can’t rely on clients releasing tokens reliably and we need to work around that. Still, as a feature it can log whenever a client exits without releasing all the tokens. That way, I was quite surprised to learn that GNU make itself does not return jobserver tokens on SIGINT. Well, bugs happen.
But then, there are harder cases. For example, jobserver-rs can’t install signal handlers, because it’s a library. Any program using it needs to take care of that, and they do not necessarily do.
Missing error handling
Missing handling for signals is one thing. However, GCC did not implement error handling in jobserver code. It just assumes that the named pipe will open successfully, that reads will succeed and so on. If anything bad happens, GCC will crash with an ICE somewhere later on.
If this could be bad with a plain named pipe, having it backed by steve makes things even worse. Steve can literally crash (or be stopped by the user), and then suddenly all the clients connected to it can explode in a variety of ways. Admittedly, this is a corner case and I find it hard to argue what’s the best behavior here; though I suppose, say, disabling jobserver support and proceeding with the implicit slot is still better than suddenly crashing.
Overzealous file type checks
The jobserver protocol was originally built using named pipes. However, to the best of my knowledge, it is impossible to build proper token accounting on top of that, and therefore avoid the risk of losing tokens. For this reason, steve is using a character device governed by CUSE. Some other implementations are using files governed by FUSE.
While GNU make itself never checked the underlying file type, many implementations do: we’ve removed the named pipe requirement in LLVM and in pytest-jobserver. This is not strictly a bug; rather an implementation choice that’s impossible for us to satisfy.
Alexander Monakov proposed an interesting suggestion why that is the case: the original GNU make implementation checks the file descriptor type when using the older jobserver variation based on anonymous pipes. That implementation relies simultaneously on the child process inheriting the file descriptor, and the environment variable specifying its number. In a pathological case, the file descriptor could be closed and replaced by some other open file; in which case checking if it’s a pipe is a last resort protection against starting to read or write a random file. However, when passing an explicit path rather than file descriptor, such a check is not really necessary.
Acquiring tokens in a child process
Traditionally, jobserver integration is part of the job scheduler. The scheduler acquires tokens, starts new jobs and releases them once finished. However, there is at least one case where that’s not the case: pytest-jobserver plugin hooks into child processes rather than the scheduler.
Such an implementation is much easier, since it seamlessly integrates with pytest-xdist, requiring no changes to the scheduler. However, it also has a major limitation: you can’t dynamically adjust the job count. The xdist plugin starts the specified number of jobs (which effectively becomes the upper bound), and the jobserver plugin throttles the actual test execution based on the availability of job tokens.
From steve’s point of view, the limitation of this approach is that every job token is associated with a different process. Features such as per-process job limits or round-robin token delivery can’t work reliably, because steve can’t really associate tokens across processes (at least for the time being).
By the way, another side effect of this design was that originally pytest-jobserserver did not obtain job tokens for test collection.
Problems with implicit slot
Another problem I’ve mentioned already is implicit slots. Per the documentation:
Second, every command make starts has one implicit job slot reserved for it before it starts. Any tool which wants to participate in the jobserver protocol should assume it can always run one job without having to contact the jobserver at all.
This design makes sense. The job starting GNU make (or any other jobserver client) should be covered by a job slot already, either implicit or using a job token. While starting nested jobs, the client uses little CPU time, and it makes no sense for it to block another job slot. Besides, if not for that, every nested make invocation would consume one more job token, and you’d soon run out of tokens.
However, it is easy to miss this fine point and actually try to acquire a token for every job started. As a result, you end up running one job too few. On top of that, it’s not always actually easy to implement that. For example, pytest-jobserver implementation ended up special-casing xdist job “gw0”, which is imperfect and could lead to locking: “gw0” uses the implicit slot even if it has nothing else to do, and any other job will have to acquire another token to finish its tasks.
On the other hand, nasm-rs implemented the implicit slot by writing an extra job token to the pipe. I mean, for named pipe it is sound: by adding one more token to the pool, you account for the parent process not needing one, the children get as many tokens as they need, and the total job count matches. However, for steve it meant we actually had to allow processes to temporarily hold −1 token. Doable but kinda ugly.
Writing back the wrong token
There’s another curious point in GNU make documentation:
It’s important that when you release the job slot, you write back the same character you read. Don’t assume that all tokens are the same character; different characters may have different meanings to GNU
make.
Can you guess which implementation did not comply?
Yes, of course GNU make did not write back the same character. GCC also does not preserve the token. Admittedly, I suppose steve is the first jobserver to actually use characters with different meanings. I’ve used that to distinguish different jobs, and therefore be able to tell how long a particular job is running. It’s not a critical feature, it can be helpful when debugging though.
Holding on to tokens throughout multiple jobs
What would you debug, actually? Well, for example you could notice that whenever Cargo is building something large, everything else stops. For example, you start five more emerge processes, and not one of them is able to start. You SIGUSR1 steve and confirm that Cargo is holding all the tokens for a few minutes now.
So you double-check the code, and file an issue: Cargo does not return tokens immediately after finishing a single job. But is that a bug? Arguably, the specification doesn’t prohibit that. Apparently it works out better when building Firefox. For building multiple packages in parallel, it is much worse, though.
What’s to come?
Jobservers aren’t anything new. However, the widespread interest in jobserver support seems to have appeared recently. If I were to hazard a guess, that’d be due to a combination of large projects becoming more heterogeneous, and toolchains becoming more complex. It’s perfectly normal to be building a large project comprising of C, C++ and Rust sources, and possibly including some bundled libraries that add both ninja and make to the mix. On top of that, LTO can overlay its own concurrency. The jobserver protocol is the obvious solution to throttling, and it stood the test of time. It’s supported by ninja, by Cargo, by GCC… and the list continues to be growing.
The idea of using a shared jobserver isn’t new either. Jobserver integration was proposed for Gentoo a long time ago. NixOS experimented with jobservers as well. However, steve is pretty new and by design inevitably different from GNU make. We’ve found many issues. And as testing continues and even more client implementations emerge, we’re bound to find more.
As a footnote, many different people have been repeatedly pointing out the limitations of the protocol, as well as announcing that they’re working on something new and better. I don’t hold my hopes high, and honestly, I don’t want to have to bother with another big transition. The GNU make protocol may be ugly, but it’s well supported and there are reasonably good ways to make it work. Steve has proven that.