Problem
Four run tests fail during the full Nix package suite on one Apple Silicon workstation,
all on the same production constant — PTY_LIST_TIMEOUT (2s, src/run.rs:46):
metadata_patch_uses_exact_id_and_one_json_stdin_payload
→ `pty metadata patch --id` failed: timed out after 2.0s
removed_and_recreated_pty_root_is_indeterminate_not_absent
→ `pty list --json` failed: timed out after 2.0s
pty_task_observation_preserves_exact_generation_and_closed_states
→ `pty list --json` failed: timed out after 2.0s
running_pty_without_complete_generation_is_indeterminate
→ index out of bounds: the len is 0 but the index is 0 (downstream of the same empty result)
Every one of these drives a stub #!/bin/sh script, not the real pty binary. So the 2s
budget expired while a shell ran three trivial builtins — which means the budget is being
spent on getting the child running, not on the work the child does.
The same constant governs production: patch_presentation and list_entries both pass
PTY_LIST_TIMEOUT to a child process. If 2s can expire around a stub shell, it can expire
around a real pty list --json on a reconcile pass, which surfaces as a spurious list
failure and a skipped pass.
This is the same defect class as #356 and #255 — a budget that has to cover getting the
child running, not just the behaviour being measured — but landing on a production constant
rather than a test-local deadline.
What is NOT the cause
Reported alongside the failure because the obvious explanations were tested and do not hold.
Whoever picks this up should not re-run these.
Not #358 / not a recent regression. The change that surfaced it is test-only — its entire
diff to src/run.rs is inside mod tests, and it touches neither PTY_LIST_TIMEOUT nor
run_captured's logic.
Not CPU load. Standalone fork+exec latency stays far below 2s even under deliberate
oversubscription:
| Host |
Load |
trivial /bin/sh |
pty list --json (empty root) |
| 16-core workstation |
10 |
p90 0.011s, max 0.369s |
p90 0.187s, max 0.231s |
| 16-core workstation |
68 |
p90 0.004s, max 0.261s |
p90 0.063s, max 0.101s |
| 10-core builder |
10 |
p90 0.027s, max 0.176s |
p90 0.230s, max 0.295s |
| 10-core builder |
50–58 |
p90 0.039s, max 0.223s |
p90 0.569s, max 0.710s |
Not memory pressure. 93% free at the time of the failing run.
Not simply in-suite spawn concurrency. run_captured installs a pre_exec hook
(setsid), which forces real fork() instead of posix_spawn(), and cargo test runs the
suite as ~ncpu threads of one large binary all spawning children. fork() cost does scale
with parent heap on Darwin — 16 threads at load 56 measured p90 0.121s / max 0.537s with a
2 GB heap, versus p90 0.041s / max 0.131s at 256 MB — but that is an order of magnitude short
of 2s. More decisively, the hosted 3-core macOS CI runner runs the same suite and passes;
if concurrent spawning were the mechanism, fewer cores should be worse, not better.
Where it has and has not been seen
| Host |
Cores |
Condition |
Result |
Hosted macos-15 CI runner |
3 |
CI |
pass ×2 |
| 10-core builder |
10 |
load 18–56 |
pass ×4 |
| 16-core workstation |
16 |
load ~99, a file-provider daemon pegged at 100% |
these 4 fail |
| 16-core workstation |
16 |
load ~75, same host ~30 min later |
these 4 pass |
Intermittent, not deterministic. A later full-suite run on the same host did not
reproduce them, so this is a tail event tied to transient host state rather than a property
of the host. Load average was ~99 while CPU demand was low and memory was 93% free, which is
consistent with processes blocked in the filesystem layer rather than starved of CPU — but a
standalone probe on that same host during that same state still could not produce a slow
child, so this is a lead, not a conclusion.
Why this is worth fixing anyway
The mechanism is unestablished, but the shape of the budget is wrong regardless: 2s is
charged for the child's scheduling, execve, and interpreter startup as well as the actual
pty work, so the margin available to the operation depends on host state the caller cannot
see. That is the same design error #356 fixed in the tests.
Deliberately not proposing a new number here. Raising the constant buys margin
proportional to nothing — the in-tree comments already call that the wrong answer — and the
measurements above are not a basis for choosing one.
Suggested next step
Reproduce inside the suite context rather than from a standalone probe: instrument
run_captured to record spawn→exit latency per child and run the full suite on the affected
host, so the distribution is captured at the moment it goes bad. That distinguishes "the
child was slow to start" from "the child was slow to run" — which decides whether the fix is
a separate startup allowance or something else entirely.
References
Posted on behalf of @schickling
| field |
value |
agent_identity |
unknown |
agent_persona |
generalist |
agent_supervisor |
unavailable |
agent_tool |
Claude Code |
agent_tool_version |
2.1.237 |
agent_runtime |
Claude Code 2.1.237 |
tooling_profile |
dotfiles@11eaf2d-dirty |
Problem
Four
runtests fail during the full Nix package suite on one Apple Silicon workstation,all on the same production constant —
PTY_LIST_TIMEOUT(2s,src/run.rs:46):Every one of these drives a stub
#!/bin/shscript, not the realptybinary. So the 2sbudget expired while a shell ran three trivial builtins — which means the budget is being
spent on getting the child running, not on the work the child does.
The same constant governs production:
patch_presentationandlist_entriesboth passPTY_LIST_TIMEOUTto a child process. If 2s can expire around a stub shell, it can expirearound a real
pty list --jsonon a reconcile pass, which surfaces as a spurious listfailure and a skipped pass.
This is the same defect class as #356 and #255 — a budget that has to cover getting the
child running, not just the behaviour being measured — but landing on a production constant
rather than a test-local deadline.
What is NOT the cause
Reported alongside the failure because the obvious explanations were tested and do not hold.
Whoever picks this up should not re-run these.
Not #358 / not a recent regression. The change that surfaced it is test-only — its entire
diff to
src/run.rsis insidemod tests, and it touches neitherPTY_LIST_TIMEOUTnorrun_captured's logic.Not CPU load. Standalone
fork+execlatency stays far below 2s even under deliberateoversubscription:
/bin/shpty list --json(empty root)Not memory pressure. 93% free at the time of the failing run.
Not simply in-suite spawn concurrency.
run_capturedinstalls apre_exechook(
setsid), which forces realfork()instead ofposix_spawn(), andcargo testruns thesuite as ~ncpu threads of one large binary all spawning children.
fork()cost does scalewith parent heap on Darwin — 16 threads at load 56 measured p90 0.121s / max 0.537s with a
2 GB heap, versus p90 0.041s / max 0.131s at 256 MB — but that is an order of magnitude short
of 2s. More decisively, the hosted 3-core macOS CI runner runs the same suite and passes;
if concurrent spawning were the mechanism, fewer cores should be worse, not better.
Where it has and has not been seen
macos-15CI runnerIntermittent, not deterministic. A later full-suite run on the same host did not
reproduce them, so this is a tail event tied to transient host state rather than a property
of the host. Load average was ~99 while CPU demand was low and memory was 93% free, which is
consistent with processes blocked in the filesystem layer rather than starved of CPU — but a
standalone probe on that same host during that same state still could not produce a slow
child, so this is a lead, not a conclusion.
Why this is worth fixing anyway
The mechanism is unestablished, but the shape of the budget is wrong regardless: 2s is
charged for the child's scheduling,
execve, and interpreter startup as well as the actualptywork, so the margin available to the operation depends on host state the caller cannotsee. That is the same design error #356 fixed in the tests.
Deliberately not proposing a new number here. Raising the constant buys margin
proportional to nothing — the in-tree comments already call that the wrong answer — and the
measurements above are not a basis for choosing one.
Suggested next step
Reproduce inside the suite context rather than from a standalone probe: instrument
run_capturedto record spawn→exit latency per child and run the full suite on the affectedhost, so the distribution is captured at the moment it goes bad. That distinguishes "the
child was slow to start" from "the child was slow to run" — which decides whether the fix is
a separate startup allowance or something else entirely.
References
codex_app_serverPosted on behalf of @schickling
agent_identityagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimetooling_profile