Implement GNU Make jobserver client protocol support in Ninja

[digit-google]

This PR implements the GNU Make jobserver client protocol in Ninja. This implements all client modes for Posix (pipe and fifo) and Windows (semaphore).

This protocol allows several participating processes to coordinate parallel build tasks / processing work.
For example, GNU Make, and now Ninja, use it to control how many parallel commands they dispatch at any given time. The Rust compiler and linker, and some C++ compilers (e.g. Clang has -flto=jobserver), use that to control how many parallel threads in a single invocation. The protocol is also implemented by the cargo Rust tool (but the latter only sets the CARGO_MAKEFLAGS environment variable).

Client mode is useful when Ninja is invoked as part of a more complex build, that launches several build tasks in parallel (e.g. recursive Make or CMake invocations). In this mode, Ninja detects that MAKEFLAGS contains --jobserver-auth or --jobserver-fds options, and uses the job slot pool to control its own dispatch of parallel build commands. It also passes the MAKEFLAGS value to child processes to let them participate in the coordination protocol.

This also includes a new script misc/jobserver_pool.py that can be used as a standalone job slot pool implementation, which can be used any client directly for testing.

This has been tested on large Fuchsia build plans, with certain build configurations that launch 24 sub-Ninja builds from a top-level Ninja build plan. With remote builders enabled, this reduces the total time from 22minutes to 12minutes.

This work is inspired by contributions from many other developers, including @hundeboll (see PR #2450), @mcprat (see PR #2474) and @stefanb2 (PR #1140) to name a few.

EDIT: (Removed mention of server mode as this has been pushed to a future PR).

[kalvdans]

Some minor comments on the code; I haven't actually tried your branch.

[Neustradamus]

To follow this important PR.

[nanonyme]

Does this work together with -flto=auto such that ninja limits flto threads when working as job server?

[nekopsykose]

Does this work together with -flto=auto such that ninja limits flto threads when working as job server?

for gcc, it should yes. the gcc lto wrapper basically emits a makefile and calls make on it to run the ltrans jobs, and that sub-make will take jobserver tokens from the ninja jobserver in this pr.

for clang, there is no equivalent- no part of the thinlto process or lld itself takes jobserver tokens (but maybe i missed something), that would require implementing the protocol (reading env vars, having lld and perhaps also the linker plugin used in other linkers spawn threads only based on grabbed tokens, ..)

[kaspar030]

Just to report: I'm developing laze, a build system calling out to Ninja for actual building.

One of laze's core features is its native support for large build matrizes of mostly similar build configurations (e.g., build something for dozens to hundreds of slightly different embedded devices).

In one project, there's a test application that calls out to cmake for building a library, and cmake calls Ninja recursively. So when building this application for 10 boards at the same time (one laze command resulting in one Ninja call), previously, this would on my laptop call 10 cmake+Ninja simultaneously, and each sub-Ninja would run up to 10 gcc instance. While there's enough RAM so this actually finishes, my laptop becomes unresponsive, typing laggy, ...

Using the Ninja binary as built by this PRs CI output and with NINJA_JOBSERVER=1, the number of gcc instances stays at or below 10. Which IMO is the expected behavior and this PR finally adds this to Ninja.

I gave this a hyperfine run to see if total build time is affected, and it seems like the jobserver version is slightly slower, within margin of error:

tests/pkg/relic on  add_laze_buildfiles took 40s371ms 
❯ hyperfine -p "rm -Rf ../../../build" "laze b"
Benchmark 1: laze b
  Time (mean ± σ):     41.419 s ±  1.242 s    [User: 241.710 s, System: 39.031 s]
  Range (min … max):   38.262 s … 42.548 s    10 runs
 
tests/pkg/relic on  add_laze_buildfiles took 6m57s586ms 
❯ NINJA_JOBSERVER=0 hyperfine -p "rm -Rf ../../../build" "laze b"
Benchmark 1: laze b
  Time (mean ± σ):     41.102 s ±  0.968 s    [User: 245.940 s, System: 38.790 s]
  Range (min … max):   38.645 s … 41.908 s    10 runs

(but, the jobserver version did not make typing here laggy ... 🙂)

[sw]

Using --jobserver without an argument doesn't work for me.

c:\>ninja --jobserver
ninja: fatal: invalid -j parameter

c:\>ninja --jobserver=0
ninja: error: loading 'build.ninja': The system cannot find the file specified.

c:\>ninja --jobserver=1
ninja: error: loading 'build.ninja': The system cannot find the file specified.

Also, setting NINJA_JOBSERVER and then trying to limit the number of parallel builds on the command line doesn't work, but that seems to be intended ("Explicit parallelism (-j), ignoring NINJA_JOBSERVER environment variable."). Maybe that's what @thesamesam alluded to in #1139. I don't really see a reason for this - is it so the child processes don't accidentally try to be jobservers as well?

With ninja -jX --jobserver=1, it seems to work as expected. We are using Ninja alongside CMake on Windows in a project with many ExternalProjects, which up to now would cause the X*N problem. So I hope that a solution can finally be merged.

[robUx4]

I gave a test on the VLC contrib build which builds in parallel more than a hundred libraries using autotools (make), CMake (ninja) and meson (ninja).

The maximum number of threads seems to be respected on my local machine.

In the CI things are working properly as well in Debian and Ubuntu.

On one machine it logs:

ninja: Jobserver mode detected: k -j48 --jobserver-auth=4,5

On the other:

ninja: Jobserver mode detected: k -j64 -Orecurse --jobserver-auth=3,4

[nanonyme]

Using --jobserver without an argument doesn't work for me.

c:\>ninja --jobserver
ninja: fatal: invalid -j parameter

c:\>ninja --jobserver=0
ninja: error: loading 'build.ninja': The system cannot find the file specified.

c:\>ninja --jobserver=1
ninja: error: loading 'build.ninja': The system cannot find the file specified.

Also, setting NINJA_JOBSERVER and then trying to limit the number of parallel builds on the command line doesn't work, but that seems to be intended ("Explicit parallelism (-j), ignoring NINJA_JOBSERVER environment variable."). Maybe that's what @thesamesam alluded to in #1139. I don't really see a reason for this - is it so the child processes don't accidentally try to be jobservers as well?

With ninja -jX --jobserver=1, it seems to work as expected. We are using Ninja alongside CMake on Windows in a project with many ExternalProjects, which up to now would cause the X*N problem. So I hope that a solution can finally be merged.

This sounds like an inconvenient limitation. Why couldn't ninja be jobserver with environment variable when explicit -jN is set?

[digit-google]

Thanks, the reason why --jobserver does not work on Windows for @sw is interesting. On this platform, we use our own src/getopt.c implementation which, apparently, only supports optional arguments for short options, not long one.

Besides that, the getopt_long() manpage states that arguments for long options should be provided as --option=arg or --option arg only, but does not say how optional arguments should be processed. This means that something like ninja --jobserver <target> is ambiguous, as it would technically be interpreted as equivalent to ninja --jobserver=<target> which will likely fail.

I am going to get rid of the problem by making --jobserver a simple flag, and adding --jobserver-mode=<mode> to specify the mode instead (so --jobserver-mode=0 will be needed to disable the feature even if NINJA_JOBSERVER is defined in the environment).

Apart from that, @nanonyme is correct that this was to avoid child processes to become jobserver themselves by accident. However, this can be solved by ensuring that NINJA_JOBSERVER is never defined in the these processes, which is simpler, so I'll change this too.

Quick question regarding behavior: Currently:

• Using an explicit -j1 disables jobserver client mode, as well as pool mode, as this is interpreted by the client not wanting parallel dispatch).

• Using an explicit -j0 disables jobserver pool mode, but not client mode, as this is interpreted by the client asking for "infinite parallelism", which to me seems only useful to see how bad the system reacts under heavy load.

If anyone thinks this is not reasonable or would create a problem for their workflow, let me know. I selected these conditions on a hunch since I never use these myself (well except -j1 in very rare cases).

[kepstin]

Is there any chance that jobserver mode could be made "automatic" by default? I.e. act as a jobserver client if a jobserver is available from the environment, otherwise (if -j is set) start a jobserver pool?

This would simplify the use of ninja quite a bit - you don't have to worry about remembering to set an extra ninja command-line parameter or environment variable to ensure that recursive builds, rust, gcc lto., etc. parallelize properly.

[jhasse]

Hm ... for automatic detection of the client: Should this be done by checking MAKEFLAGS?
And whether to automatically spawn a server: Build edges could specify if they are a jobserver client and if one edge with that option is part of the build, ninja would activate the jobserver automatically.

Some more general comments about this PR:

1. I don't like the new environment variable NINJA_JOBSERVER, we should keep them to a minimum and I don't see why it is needed.
2. I would only implement the newer (better) fifo mode on Linux. That way the --jobserver-mode wouldn't be needed.

[digit-google]

To answer @kepstin's question. Client mode is already setup automatically by inspecting the MAKEFLAGS variable. If it detects that Ninja is called within the context of another jobserver pool, it will automatically use it to control how it dispatches parallel jobs, and will also pass the variable to sub-commands as well to let them participate in the coordination protocol.

On the other hand, starting the pool in Ninja automatically seems risky to me, because it can change much more than Ninja's behavior (all sub-commands as well). For example, there are rare cases (unbalanced builds) where this will result in a slower build overall (e.g. as described here).

That's why I believe the NINJA_JOBSERVER environment variable is a good compromise here. You can set it to 1 and Ninja will automatically start a pool (if not already running in a client context). It is something that also allows this to work without modifying build systems and scripts, or wrappers such a cmake --build or meson compile and many others).

I know @jhasse dislikes environment variables, but I believe that this is one of the rare cases where the benefits outweigh the annoyances.

[digit-google]

To answer @jhasse's latest comment now, client mode is already started automatically by looking at MAKEFLAGS, there is nothing to do to use it, and NINJA_JOBSERVER is only here to control when Ninja implements the pool itself.

Adding a special syntax to the Ninja build plan to indicate that specific actions support the protocol is doable, but it will mean the feature won't be usable until all Ninja generators support it, which may be a veeeeery long time. I don't think it's really useful here. That might be useful to explicitly disable it for certain commands (though on Posix one can simply start the command with MAKEFLAGS= ...., Win32 is more complicated).

For Posix, I think it is far preferrable to use --jobserver-mode=pipe as the default at the moment as many multi-build systems run on older distributions that do not have GNU Make 4.4 yet (which is the one which implements fifo mode). Even my Debian 12-based Linux distribution at work is only providing GNU Make 4.3 today.

I imagine we could switch to fifo transparently in a few years when GNU Make 4.4+ is more widespread. But since there is absolutely no performance difference between the two modes, and that our client implementation supports both transparently, when Ninja is not the pool implementation, pipe seems a reasonable choice for now.

[eli-schwartz]

That might be useful to explicitly disable it for certain commands (though on Posix one can simply start the command with MAKEFLAGS= ...., Win32 is more complicated).

This is why ninja should support a dedicated syntax to run a build rule with specific environment variables set -- trivial on POSIX, annoying on Win32 so would require invoking processes two different ways depending on whether the build plan has demanded an environment variable, iirc.

Not all ninja generators permit specifying arbitrary shell command syntax for build rules. For precisely the reason of predictable cross platform behavior.

[jhasse]

Even my Debian 12-based Linux distribution at work is only providing GNU Make 4.3 today.

That's not because GNU Make 4.4 is brand new (it's 2 years old actually), but because Debian is such a bad distribution. If we wanted to be pragmatic we could have merged the first PR 8 years ago.

[digit-google]

That's not because GNU Make 4.4 is brand new (it's 2 years old actually), but because Debian is such a bad distribution. If we wanted to be pragmatic we could have merged the first PR 8 years ago.

Well, just like build systems there are only two types of Linux distributions: those that people complain about, and those that nobody uses ;-)

[htot]

People: the world is bigger than just your distributions packages. The demand for jobserver support becomes relevant when you are using a build system that builds many different packages using combinations of make, ninja and possibly other methods, like Yocto's bitbake (if you are unfamiliar with that just imagine the build system that builds all of debian's packages). This is why it is unrealistic to convert everything to ninja, there are 1000's of packages to be built.

Now Yocto installs it's own make and ninja version, master contains make 4.4.1. To get the jobserver working in bitbake I patched it based on a suggestion from the Yocto developers, and forked one of stephan's versions. This has been working for at least a year.

I would expect a similar need for jobserver in buildroot.

The importance is that if you have a bitbake build machine that does m parallel makes/ninja's on n cores (+ ht), you are going to need about m x n x 1GB RAM (per user building world).

[nanonyme]

In any case I wouldn't expect stable distributions to pick up 1.12.0 or above as update. The scheduling changes means that some legacy builds break. So this PR is not going to apply to historic but future distro releases. In orher words, parties who get this are highly likely to have new enough make.

[ArsenArsen]

For Posix, I think it is far preferrable to use --jobserver-mode=pipe as the default at the moment as many multi-build systems run on older distributions that do not have GNU Make 4.4 yet (which is the one which implements fifo mode). Eveno I'm not convinced. my Debian 12-based Linux distribution at work is only providing GNU Make 4.3 today.

Odds are that time will start passing, even for Debian, enough
to get make 4.4 at around the same time or before the hypothetical Ninja release that'd include this patch, though.

So I'm not sure it's actually advantageous to default to pipe. I'm exporting NINJA_JOBSERVER=fifo for my testing.

[jdrouhard]

As for not being able to go beyond the nproc + 2 number, there's an issue with RealCommandRunner::CanRunMore(). It returns a capacity that only ever takes into account config_.parallelism which isn't actually updated/used in jobserver mode, it's guessed (if no -j is passed) as nproc + 2.

[digit-google]

@jdrouhard , thank you very much. I think I fixed the issues you described in the latest push. Some explanations:

• First, I have finally been able to reproduce the default parallelism overtaking the jobserver pool. Turns out I was using the wrong binary for my most recent testing! My mistake, really sorry. The bug was introduced in one of the previous rebases, where the code that sets the capacity to INT_MAX in RealCommandRunner::CanRunMore() when a jobserver is enabled was removed. This is now back.

  Surprisingly, this was not caught by the existing regression tests, so I added a new one to check for this specific condition. It is Linux specific though, as it uses taskset, but that should be good enough to ensure the feature works everywhere.

• Second, thanks again for the crash diagnostic and suggestions. I fixed it by creating the JobserverClient instance before the builder. The SetJobserverClient() method was added to avoid changing the Builder constructor, which would either require a default argument (which in my experience introduce subtle bugs over time), or modifying many unit tests that don´t really care about the feature. We could pass the ownership of the JobserverClient instance to the Builder in that function, but for now, I prefer the simpler fix. I don't have a good regression test for this specific case though :-/ Any suggestion welcome.

[jdrouhard]

Awesome, thanks. Just to guard against future refactorings reordering things, could you change Builder::SetJobServerClient(...) to take a unique_ptr (by value), and let the Builder own the unique_ptr<JobServer>? You would just move the jobserver_client ptr from there instead of passing a raw pointer and wouldn't need to check if it's null or not before doing so.

[digit-google]

I just uploaded a new version of the PR that, compared to the previous one:

• Uses std::unique_ptr<Jobserver::Client> to move ownership of the value to the Builder as suggested by @jdrouhard . However we still need to check for null as this feature is completely optional :-)

• Removes support for the pipe-based protocol in Ninja, as requested by @jhasse . Note that jobserver_pool.py still supports it, as this is used in a regression test to verify that Ninja prints a warning when it detects it in MAKEFLAGS. That script now sets up a FIFO pool by default now. This warning is also why the MAKEFLAGS parser in src/jobserver.cc still recognizes the pipe-based format, as it allows giving a better error message, which looks like:

ninja: warning: Ignoring jobserver: Pipe-based protocol is not supported! [ -j128 --jobserver-fds=3,4 --jobserver-auth=3,4]

• Updates the commit messages, and doc/manual.asciidoc to clarify that pipe-based protocol is not supported.

If you are interested in the old code, I have created a JOBSERVER-with-pipe-support tag on my github fork. However, I will likely not work on this in the future.

[jhasse]

Thanks again! Almost good to go.

[jhasse]

Remove this paragraph.

[nanonyme]

I just uploaded a new version of the PR that, compared to the previous one:

* Uses `std::unique_ptr<Jobserver::Client>` to move ownership of the value to the `Builder` as suggested by @jdrouhard . However we still need to check for null as this feature is completely optional :-)

* Removes support for the pipe-based protocol in Ninja, as requested by @jhasse . Note that jobserver_pool.py still supports it, as this is used in a regression test to verify that Ninja prints a warning when it detects it in `MAKEFLAGS`. That script now sets up a FIFO pool by default now. This warning is also why the MAKEFLAGS parser in src/jobserver.cc still recognizes the pipe-based format, as it allows giving a better error message, which looks like:

ninja: warning: Ignoring jobserver: Pipe-based protocol is not supported! [ -j128 --jobserver-fds=3,4 --jobserver-auth=3,4]

* Updates the commit messages, and `doc/manual.asciidoc` to clarify that pipe-based protocol is not supported.

If you are interested in the old code, I have created a JOBSERVER-with-pipe-support tag on my github fork. However, I will likely not work on this in the future.

Would it be possible to host the standalone Python tool that creates job server with some license compatible with this project? I would be quite interested in including it in our build stack and wrapping ninja both in cmake and meson projects to run under it. I guess this project could even use that as git submodule for tests if project manager agrees.

[digit-google]

@nanonyme, the jobserver_pool.py script is under the Apache 2.0 license, so it's already compatible.

I have no interest in hosting it into a separate Github repository, if that's what you are asking for, but feel free to create one if you want. In its current form, it is only meant to run Ninja regression tests.

[kaspar030]

🎇

[h-vetinari]

I was looking for an issue that captures the pieces that have been descoped here but couldn't seem to find one. FWICT, it would be at least:

• pipe support
• server mode

I don't feel I'm well-placed to open an issue on this, but if it helps I can do it.

[digit-google]

For server mode, I implemented a new --jobserver-pool flag, now available in PR ##2634. Please take a look.

I don't have plans to support the pipe-based scheme, either in client or pool mode.

[Ext3h]

That was the wrong cpu_count - it does not correctly reflect which CPU cores are masked for this process. os.process_cpu_count (for Python 3.13 and above) gives the correct available core count.

[thesamesam]

There's many, many people subscribed to this PR, maybe file a bug with the suggestions instead? Thanks.

[Ext3h]

Illegal argument for jobs_count == 1, argument lMaximumCount must be non-zero by definition.