A\u00a0common problem with running Gentoo builds is\u00a0concurrency. Many packages include extensive build steps that are either fully serial, or\u00a0cannot fully utilize the\u00a0available CPU threads throughout. This problem becomes less pronounced when running building\u00a0multiple packages in\u00a0parallel, but\u00a0then we are risking overscheduling for\u00a0packages that do\u00a0take advantage of\u00a0parallel builds.<\/p>\n
Fortunately, there are a\u00a0few tools at\u00a0our disposal that can improve the\u00a0situation. Most recently, they were joined by\u00a0two experimental system-wide jobservers: guildmaster<\/a> and\u00a0steve<\/a>. In\u00a0this post, I’d like to\u00a0provide the\u00a0background on\u00a0them, and\u00a0discuss the\u00a0problems they are facing. You can use the\u00a0MAKEOPTS<\/kbd> variable to\u00a0specify a\u00a0number of\u00a0parallel jobs to\u00a0run:<\/p>\n This is used not\u00a0only by\u00a0GNU make, but\u00a0it is also recognized by\u00a0a\u00a0plethora of\u00a0eclasses and\u00a0ebuilds, and\u00a0converted into appropriate options for\u00a0various builders, test runners and\u00a0other tools that can benefit from\u00a0concurrency. So\u00a0far, that’s good news; whenever we can, we’re going to\u00a0run 12\u00a0jobs and\u00a0utilize all the\u00a0CPU threads.<\/p>\n The\u00a0problems start when we’re running multiple builds in\u00a0parallel. This could be\u00a0either due to\u00a0running emerge --jobs<\/kbd>, or\u00a0simply needing to\u00a0start another emerge<\/kbd> process. The\u00a0latter happens to\u00a0me quite often, as I\u00a0am testing multiple packages simultaneously.<\/p>\n For\u00a0example, if\u00a0we end\u00a0up building four packages simultaneously, and\u00a0all of\u00a0them support -j<\/kbd>, we\u00a0may end\u00a0up spawning 48\u00a0jobs. The\u00a0issue isn’t just saturating the\u00a0CPU; imagine you’re running 48\u00a0memory-hungry C++ compilers simultaneously!<\/p>\n One possible workaround is to\u00a0use the\u00a0--load-average<\/kbd> option, e.g.:<\/p>\n This causes tools supporting the\u00a0option not to\u00a0start new jobs if\u00a0the\u00a0current load exceeds 13, which roughly approximates 13\u00a0processes running simultaneously. However, the\u00a0option isn’t universally supported, and\u00a0the\u00a0exact behavior differs from\u00a0tool to\u00a0tool. For\u00a0example, CTest doesn’t start any\u00a0jobs when the\u00a0load is\u00a0exceeded, effectively stopping test execution, whereas GNU\u00a0make and\u00a0Ninja throttle themselves down to\u00a0one job.<\/p>\n Of\u00a0course, this is a\u00a0rough approximation. While GNU\u00a0make attempts to\u00a0establish the\u00a0current load from\u00a0\/proc\/loadavg<\/kbd>, most tools just use the\u00a0one-minute average from\u00a0getloadavg()<\/kbd>, suffering from\u00a0some lag. It\u00a0is entirely possible to\u00a0end\u00a0up with\u00a0interspersed periods of\u00a0overscheduling while the\u00a0load is\u00a0still ramping up, followed by\u00a0periods of\u00a0underscheduling before it decreases again. Still, it is better than nothing, and\u00a0can become especially useful for\u00a0providing background load for\u00a0other tasks: a\u00a0build process that can utilize the\u00a0idle CPU threads, and\u00a0back down when other builds need them.<\/p>\n Nested Makefiles are\u00a0processed by\u00a0calling make<\/kbd> recursively, and\u00a0therefore face a\u00a0similar problem: if\u00a0you run multiple make processes in\u00a0parallel, and\u00a0they run multiple jobs simultaneously, you end\u00a0up overscheduling. To\u00a0avoid this, GNU\u00a0make introduces a\u00a0jobserver. It ensures that the\u00a0specified job number is\u00a0respected across multiple make invocations.<\/p>\n At\u00a0the\u00a0time of\u00a0writing, GNU\u00a0make supports three kinds of\u00a0the\u00a0jobserver protocol:<\/p>\n All these variants follow roughly the\u00a0same design principles, and\u00a0are peer-to-peer protocols for\u00a0using shared state rather than true servers in\u00a0the\u00a0network sense. The\u00a0jobserver’s role is mostly limited to\u00a0initializing the\u00a0state and\u00a0seeding it with an\u00a0appropriate number of\u00a0job tokens. Afterwards, clients are\u00a0responsible for\u00a0acquiring a\u00a0token whenever they are\u00a0about to\u00a0start a\u00a0job, and\u00a0returning it once the\u00a0job finishes. The\u00a0availability of\u00a0job tokens therefore limits the\u00a0total number of\u00a0processes started.<\/p>\n The\u00a0flexibility of\u00a0modern protocols permitted more tools to\u00a0support them. Notably, the\u00a0Ninja build system recently started supporting the\u00a0protocol, therefore permitting proper parallelism in\u00a0complex build systems combining Makefiles and\u00a0Ninja. The\u00a0jobserver protocol is\u00a0also supported by\u00a0Cargo and\u00a0various Rust tools, GCC and\u00a0LLVM, where it can be\u00a0used to\u00a0limit the\u00a0number of\u00a0parallel LTO\u00a0jobs.<\/p>\n With a\u00a0growing number of\u00a0tools becoming capable of\u00a0parallel processing, and\u00a0at\u00a0the\u00a0same time gaining support for\u00a0the\u00a0GNU\u00a0make jobserver protocol, it\u00a0starts being an\u00a0interesting solution to\u00a0the\u00a0overscheduling problem. If\u00a0we could run one jobserver shared across all build processes, we\u00a0could control the\u00a0total number of\u00a0jobs running simultaneously, and\u00a0therefore have all the\u00a0simultaneously running builds dynamically adjust one to\u00a0another!<\/p>\n In\u00a0fact, this is not\u00a0a\u00a0new idea. A\u00a0bug requesting jobserver integration<\/a> has been filed for\u00a0Portage back in\u00a02019. NixOS jobserver effort<\/a> dates back at\u00a0least to\u00a02021, though it has not\u00a0been merged yet. Guildmaster<\/a> and\u00a0steve<\/a> joined the\u00a0effort very recently.<\/p>\n There are\u00a0two primary problems with using a\u00a0system-wide jobserver: token release reliability, and\u00a0the\u00a0“implicit slot” problem.<\/p>\n The\u00a0first problem is\u00a0more important. As\u00a0noted before, the\u00a0jobserver protocol relies entirely on\u00a0clients releasing the\u00a0job tokens they acquired, and\u00a0the\u00a0documentation explicitly emphasizes that they must be\u00a0returned even in\u00a0error conditions. Unfortunately, this is not\u00a0always possible: if\u00a0the\u00a0client gets killed, it\u00a0cannot run any cleanup code and\u00a0therefore return the\u00a0tokens! For\u00a0scoped jobservers like GNU\u00a0make’s this usually isn’t that much of\u00a0a\u00a0problem, since make normally terminates upon a\u00a0child being killed. However, a\u00a0system jobserver could easily be left with no\u00a0job tokens in\u00a0the\u00a0queue this way!<\/p>\n This problem cannot really be\u00a0solved within the\u00a0strict bounds of\u00a0the\u00a0jobserver protocol. After all, it is just a\u00a0named pipe, and\u00a0there are\u00a0limits to\u00a0how much you can monitor what’s happening to\u00a0the\u00a0pipe buffer. Fortunately, there is a\u00a0way around that: you can implement a\u00a0proper server for\u00a0the\u00a0jobserver protocol using FUSE, and\u00a0provide it in\u00a0place of\u00a0the\u00a0named pipe. Good news is, most of\u00a0the\u00a0tools don’t actually check the\u00a0file type, and\u00a0these that do can easily be\u00a0patched.<\/p>\n The\u00a0current draft of\u00a0NixOS jobserver<\/a> provides a\u00a0regular file with special behavior via\u00a0FUSE, whereas guildmaster and\u00a0steve both provide a\u00a0character device via\u00a0its\u00a0CUSE API. NixOS jobserver and\u00a0guildmaster both return unreleased tokens once the\u00a0process closes the\u00a0jobserver file, whereas steve returns them once the\u00a0process acquiring them exits. This way, they can guarantee that a\u00a0process that either can’t release its tokens (e.g.\u00a0because it’s been killed), or\u00a0one that doesn’t because of\u00a0implementation issue (e.g.\u00a0Cargo), doesn’t end up effectively locking other builds. It\u00a0also means we can provide live information on\u00a0which processes are\u00a0holding the\u00a0tokens, or\u00a0even implement additional features such as\u00a0limiting token provision based on\u00a0the\u00a0system load, or\u00a0setting per-process limits.<\/p>\n The\u00a0second problem is\u00a0related to\u00a0the\u00a0implicit assumption that a\u00a0jobserver is\u00a0inherited from\u00a0a\u00a0parent GNU\u00a0make process that already acquired a\u00a0token to\u00a0spawn the\u00a0subprocess. Since the\u00a0make subprocess doesn’t really do any work itself, it can “use” the\u00a0token to\u00a0spawn another job instead. Therefore, every GNU\u00a0make process running under a\u00a0jobserver has one implicit slot that runs jobs without consuming any tokens. If\u00a0the\u00a0jobserver is\u00a0running externally and\u00a0no\u00a0job tokens were\u00a0acquired while\u00a0running the\u00a0top make process, it\u00a0ends up running an\u00a0extra process without a\u00a0job token: so\u00a0steve -j12<\/kbd> permits 12\u00a0jobs, plus one extra job for\u00a0every package being built.<\/p>\n Fortunately, the\u00a0solution is\u00a0rather simple: one needs to\u00a0implement token acquisition at\u00a0Portage level. Portage acquires a\u00a0new token prior to\u00a0starting a\u00a0build job, and\u00a0releases it\u00a0once the\u00a0job finishes. In\u00a0fact, this solves two problems: it accounts for\u00a0the\u00a0implicit slot in\u00a0builders implementing the\u00a0jobserver protocol, and\u00a0it limits the\u00a0total number of\u00a0jobs run for\u00a0parallel builds.<\/p>\n However, this is\u00a0a\u00a0double-edged sword. On\u00a0one hand, it\u00a0limits the\u00a0risk of\u00a0overscheduling when running parallel build jobs. On\u00a0the\u00a0other, it\u00a0means that a\u00a0new emerge<\/kbd> job may not be\u00a0able to\u00a0start immediately, but\u00a0instead wait for\u00a0other jobs to\u00a0free up job tokens first, negatively affecting interactivity.<\/p>\n A\u00a0semi-related issue is\u00a0that acquiring a\u00a0single token doesn’t properly account for\u00a0processes that are\u00a0parallel themselves but\u00a0do not\u00a0implement the\u00a0jobserver protocol, such as\u00a0pytest-xdist<\/kbd> runs. It may be\u00a0possible to\u00a0handle these better by\u00a0acquiring multiple tokens prior to\u00a0running them (or\u00a0possibly while running them), but\u00a0in\u00a0the\u00a0former case one needs to be\u00a0careful to\u00a0acquire them atomically, and\u00a0not\u00a0end\u00a0up with the\u00a0equivalent of\u00a0lock contention: two processes acquiring part of\u00a0the\u00a0tokens they require, and\u00a0waiting forever for\u00a0more.<\/p>\n The\u00a0implicit slot problem also causes issues in\u00a0other clients. For\u00a0example, nasm-rs writes an\u00a0extra token to\u00a0the\u00a0jobserver pipe<\/a> to\u00a0avoid special-casing the\u00a0implicit slot. However, this violates the\u00a0protocol and\u00a0breaks clients with per-process tokens. Steve carries a\u00a0special workaround for\u00a0that package.<\/p>\n A\u00a0growing number of\u00a0tools is\u00a0capable of\u00a0some degree of\u00a0concurrency: from\u00a0builders traditionally being able to\u00a0start multiple parallel jobs, to\u00a0multithreaded compilers. While they provide some degree of\u00a0control over\u00a0how many jobs to\u00a0start, avoiding overscheduling while running multiple builds in\u00a0parallel is\u00a0non-trivial. Some builders can use load average to\u00a0partially mitigate the\u00a0issue, but\u00a0that’s far from a\u00a0perfect solution.<\/p>\n Jobservers are our best bet right now. Originally designed to\u00a0handle job scheduling for\u00a0recursive GNU\u00a0make invocations, they are being extended to\u00a0control other parallel processes throughout the\u00a0build, and\u00a0can be further\u00a0extended to\u00a0control the\u00a0job numbers across different builds, and\u00a0even across different build containers.<\/p>\n While NixOS seems to\u00a0have dropped the\u00a0ball, Gentoo is\u00a0now finally actively pursuing global jobserver support. Guildmaster and\u00a0steve both prove that the\u00a0server-side implementation is\u00a0possible, and\u00a0integration is just around the\u00a0corner. At\u00a0this point, it’s not clear whether a\u00a0jobserver-enabled systems are going to\u00a0become the\u00a0default in\u00a0the\u00a0future, but\u00a0certainly it’s\u00a0an\u00a0interesting experiment to\u00a0carry.<\/p>\n","protected":false},"excerpt":{"rendered":" A\u00a0common problem with running Gentoo builds is\u00a0concurrency. Many packages include extensive build steps that are either fully serial, or\u00a0cannot fully utilize the\u00a0available CPU threads throughout. This problem becomes less pronounced when running building\u00a0multiple packages in\u00a0parallel, but\u00a0then we are risking overscheduling for\u00a0packages that do\u00a0take advantage of\u00a0parallel builds. Fortunately, there are a\u00a0few tools at\u00a0our disposal that can … Continue reading “One jobserver to rule them all”<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2439"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=2439"}],"version-history":[{"count":45,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2439\/revisions"}],"predecessor-version":[{"id":2484,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/2439\/revisions\/2484"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=2439"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=2439"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=2439"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n<\/p>\nThe\u00a0job multiplication problem<\/h2>\n
MAKEOPTS=\"-j12\"<\/pre>\n
Load-average scheduling to\u00a0the\u00a0rescue<\/h2>\n
MAKEOPTS=\"-j12 -l13\"<\/pre>\n
The\u00a0nested Makefile problem and\u00a0GNU\u00a0Make jobserver<\/h2>\n
\n
A\u00a0system-wide jobserver<\/h2>\n
The\u00a0token release problem<\/h2>\n
The\u00a0implicit slot problem<\/h2>\n
Summary<\/h2>\n