Optimize child process spawns by skipping job server configs - #2827
Open
rnk wants to merge 1 commit into
Open
Conversation
rnk
force-pushed
the
jobserver-posix-spawn
branch
from
August 28, 2026 05:50
2d2b264 to
38a715b
Compare
Collaborator
|
Nice to see you here :) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2827 +/- ##
==========================================
+ Coverage 73.71% 73.73% +0.01%
==========================================
Files 72 72
Lines 37932 37976 +44
==========================================
+ Hits 27963 28002 +39
- Misses 9969 9974 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`AsyncCommand::spawn` calls `jobserver::Client::configure` on every child it
spawns. On Unix that registers a `pre_exec` closure to clear `CLOEXEC` on the
jobserver's two file descriptors -- and the presence of *any* `pre_exec` makes
`std` abandon its `posix_spawn` fast path and fall back to `fork` + `exec`.
That is a bad trade for a server process. `fork` duplicates the parent's page
tables, and sccache's whole job is to be a long-lived process holding a large
cache; the child then throws the copy away microseconds later in `exec`. The
cost scales with how much memory the server has touched, so it grows over the
life of a build.
An empty closure is enough to trigger it:
let mut c = Command::new("/bin/true");
if pre_exec { unsafe { c.pre_exec(|| Ok(())); } }
c.spawn()
without: clone3({flags=CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND, ...})
with: clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|...|SIGCHLD)
`CLONE_VM|CLONE_VFORK` shares the address space and copies nothing. The second
form is a real fork, and it is what shows up in a profile as `dup_mmap` ->
`copy_page_range` -> `copy_pte_range`.
Almost nothing sccache spawns can use a jobserver. It reaches a child through
`CARGO_MAKEFLAGS`, and only `rustc` reads it: preprocessors, version probes and
C/C++ compilers all ignore it. So stop sharing by default and let the callers
that spawn `rustc` ask, via `RunCommand::share_jobserver`.
For the local compile, which is one code path shared by every frontend, the
opt-in is a field on `SingleCompileCommand` rather than a builder call. That
makes the compiler ask every frontend the question, so a new one cannot get it
wrong by omission -- and getting it wrong in this direction is what matters,
since a `rustc` without a jobserver spawns as many codegen threads as there
are CPUs, per concurrent `rustc`, which is the oversubscription the jobserver
exists to prevent.
Measured on a 2456-file LLVM build (X86 only, Release, clang, `-j16`, 16
cores). Two scenarios, interleaved rounds:
All compiles are cache hits, with `SCCACHE_DIRECT=false` so the preprocessor
still runs -- this isolates the spawn cost, since it is nearly all the server
does (4 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 40.7 s (+-0.7) | 28.2 s (+-0.4) | 22.8 s |
| after | 33.5 s (+-0.2) | 10.6 s (+-0.1) | 5.8 s |
All compiles are cache misses, so each one both preprocesses and compiles
(2 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 336.5 s | 96.3 s | 45.8 s |
| after | 333.1 s | 70.1 s | 21.2 s |
The saving is almost entirely system time, which is what a page-table copy
costs. Wall clock barely moves on the miss build because it is dominated by the
compiler itself; the win there is 26 s of a core given back, not a faster
build.
`strace` on a 1300-compile miss build confirms the mechanism, and shows why
both spawn sites had to be covered:
| build | fork | posix_spawn |
|------------------|------|-------------|
| before | 2617 | 0 |
| preprocess only | 1319 | 1300 |
| this change | 0 | 2619 |
Under `perf record -p <server>` on the cache-hit build, the address-space
symbols (`copy_pte_range`, `copy_present_ptes`, `zap_pte_range`,
`smp_call_function_many_cond` and friends) go from 4.20% of the server's
samples to 0.05%.
One behaviour change worth naming: the client-side fallback in `commands.rs`,
which runs the compiler itself when the server declines the job, no longer
shares its jobserver either. That jobserver is private to a single short-lived
client process with exactly one child, so it never limited anything across
compiles; the server's jobserver is the one that does real work.
rnk
force-pushed
the
jobserver-posix-spawn
branch
from
August 28, 2026 18:22
38a715b to
3e3d27e
Compare
rnk
marked this pull request as ready for review
August 28, 2026 18:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Configuring the job server so that all child processes can access it causes Rust's subprocess library to switch from
posix_spawntofork, which supports running arbitrary code in a pre-exec context. Fork is, unfortunately, very expensive. You really want to use vfork or posix_spawn, which IIRC uses that under the hood, in order to get fast process launching.This really doesn't matter in the grand scheme of things because sccache direct mode (the defatult) doesn't launch a ton of processes when it gets a cache hit, but if you turn it off, it will launch may pre-processing jobs, and the overhead is observable in a profiler, which is how I (well, Claude) found this:
It seemed like a reasonably small contribution that might make a good first PR.
The code is AI generated, and I'll take another pass to try to simplify it. I think it has the wrong job server default.