fix(tests): synchronize load-sensitive Darwin tests on the event they measure, add Darwin CI

[schickling-assistant]

Problem

Four st2 tests spend their behavioral deadline on fixture scheduling. On a loaded
aarch64-darwin host the deadline expires before the fixture reaches the state under
test, so the test fails while asserting nothing about the behavior it names. They pass
in isolation, which is why this reads as flakiness rather than as a defect in the tests.

Test	Budget	What the budget also had to pay for	Failure under load
run::tests::input_write_failure_terminates_and_reaps_the_child	1s	fork+exec of the fixture shell, then exec 0<&-	unexpected write error: timed out after 1.0s
run::tests::the_group_kill_reaps_a_descendant_that_outlives_the_direct_child	2s	the child forking a descendant and publishing its pid	the child never forked a descendant, so this case tested nothing: NotFound
ding::tests::deferred_delivery_backoff_bounds_short_lived_pty_attempts	250ms wall clock	the DING loop reaching its first poll at all	left: 0, right: 1
ding::tests::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke	one 3s budget shared by two waits	the first wait, leaving nothing for the second	left: 1, right: 2

fork+exec of /bin/sh measured 0.554s on an idle 10-core M1 Max — over half of
the first test's entire budget before the behavior under test has begun.

The first two are #356. The third is #255, open since 2026-08-18. The fourth was found by
the loaded Darwin rebuild added in this branch — it had never been reported, which is the
argument for the CI job in one line. #255 already named the cause of its own invisibility: "CI is ubuntu-latest only, so every green check on this
repository proves Linux and says nothing about Darwin." This PR closes that gap too.

Goal

Each test synchronizes on the lifecycle event it intends to measure, then applies a
bounded assertion — and a hosted Darwin job gates the surface these failures live on, so
the next one is caught by CI rather than by someone running the suite locally.

Decisions

The barrier goes in on_spawn, because run_captured already starts the child deadline
after on_spawn returns. A test blocks there until its fixture published a marker, so
fork+exec is paid outside the deadline instead of out of it. No production
semantics change: both real call sites pass a no-op observer, and the seam already existed.

That ordering was load-bearing and proven by nothing — a future harmless-looking reorder
would put every barrier test back on a race with the scheduler while leaving the suite
green. the_spawn_observer_runs_before_the_child_deadline_starts now pins it. Its barrier
outlasts the timeout by construction, so the outcome depends on the order alone and on
nothing the host's scheduler does.

Markers are published by atomic rename, not polled until parseable. A shell redirection
truncates the target before the bytes land, so a barrier on bare existence can read an empty
file. mv from a sibling temp file makes the marker atomic. #318 solved the same shape by
retrying the read until it parsed; the rename removes the retry rather than bounding it,
and a genuinely bad pid stays a parse error instead of becoming a confusing timeout.

The DING test stops on the loop's own poll counter. session_alive runs exactly once
per iteration, so probes counts polls directly and load-independently. The test now
stops once the first poke landed and the loop polled five more times — precisely when a
missing backoff would poke again — instead of wherever a 250ms window happened to land. An
added assertion that the loop really did poll again keeps the counter-test non-vacuous:
one poke across one poll would otherwise prove nothing about backoff.

A barrier ceiling is not a deadline. await_fixture_ready and its DING counterpart wait
30s. That number bounds a fixture that never ran at all — it is not the behaviour under
test, and waiting longer costs nothing when the fixture does run. startup_backlog_… had
the right barriers already but gave both of them one shared 3s budget, so the first could
spend it; each now has its own. Those barriers report rather than panic, because an unwind
from inside thread::scope would leave run_ding without the stop flag that ends it,
turning a failed assertion into a hang.

Rejected: widening the deadlines. It buys margin proportional to nothing, leaves the
tests measuring scheduling plus behavior, and the existing in-tree comments already
identify widening as the wrong answer. The barrier removes the dependency instead.

Darwin CI scope is nix build .#st2, not nix flake check. That builds the package
and runs its hermetic suite via doCheck — exactly the surface these failures live on —
without also demanding cold aarch64-darwin builds of the pty and otelite inputs that
checks.parked-recovery and checks.otel-export pull in. Widening once it is stable is a
follow-up, not a prerequisite. Hosted macos-15 rather than a self-hosted Mac: the repo is
public so hosted arm64 runners are free, and the fleet's self-hosted runners are currently
down.

Verification

Root cause proven by shrinking the deadline — if the budget is what is being consumed,
shrinking it must reproduce both failures deterministically on an idle Linux box. It does:

# test 1 deadline 1s -> 1ms
unexpected write error: timed out after 0.0s
# test 2 deadline 2s -> 0
the child never forked a descendant, so this case tested nothing: NotFound

Identical to the messages the loaded Darwin host produces.

Reproduced on Apple Silicon, first attempt, packaged suite at 19485cf:

test ding::tests::deferred_delivery_backoff_bounds_short_lived_pty_attempts ... FAILED
test run::tests::input_write_failure_terminates_and_reaps_the_child ... FAILED
test run::tests::the_group_kill_reaps_a_descendant_that_outlives_the_direct_child ... FAILED
test result: FAILED. 461 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out

Counter-tests confirmed by mutation. Each mutation was applied to the fixed tree and
must fail its exact test — the whole reason these tests exist:

Mutation	Expected to fail	Result
drop kill(-pid, SIGKILL) from terminate_and_reap_before	the_group_kill_reaps_a_descendant_…	FAILED (only that test)
drop terminate_and_reap_before on the stdin-failure path	input_write_failure_terminates_…	FAILED (only that test)
move the deadline above on_spawn	the_spawn_observer_runs_before_…	FAILED: the child deadline started before on_spawn returned: timed out after 0.2s
replace the DING backoff with None	deferred_delivery_backoff_bounds_…	FAILED: left: 6, right: 1
stop queueing new arrivals for delivery	startup_backlog_gets_one_generic_recovery_…	FAILED: the ding loop never delivered both notices

The ordering guard caught a real gap while being written: its first version asserted on a
child that exits, and run_captured checks try_wait before the deadline, so it passed
under the reorder. Rewriting it around the stdin path makes it non-vacuous — the FAILED
row above is the rewritten version.

Linux, full lib+bin suite: 482 passed; 0 failed (was 481 — one test added).

CI, on the final commit — both platforms green:

Job	Scope	Result
check-x86_64-linux	full nix flake check	pass, 37m33s
check-aarch64-darwin	nix build .#st2	pass, 5m51s

nix build .#st2 runs the suite via doCheck, so a green Darwin build is every gated test
passing on a 3-core runner that is a stricter scheduler than any Mac in the fleet.

Repeated rebuilds on Apple Silicon under sustained load — 8 spinners on 10 cores plus a
known fseventsd defect holding a further core, load average 21–38. A single green run
proves little about a load-sensitive failure, so this was run repeatedly:

Rebuild	Branch state	Result
baseline	19485cf (no fix), load 3.5	461 passed; 3 failed — the repro, first attempt
1	first three fixed, load 27–38	all three pass; 1 unrelated failure (see Concerns)
2	first three fixed, load 18–32	465 passed; 0 failed
3	first three fixed, load 24–32	464 passed; 1 failed — surfaced the fourth defect
4	all four fixed, load 24–30	465 passed; 0 failed
5	all four fixed, load 22–29	465 passed; 0 failed

Each row is a build that actually re-ran the suite. Grading is on the test result: lines,
not on the exit status: nix build --rebuild also compares output bytes, and st2's Darwin
build is not bit-reproducible, so a run whose tests all pass can still exit non-zero with
may not be deterministic: output ... differs. Conversely a plain nix build is a cache
hit once the output is valid and runs nothing at all. Both traps produced misleading rows
during this work before the loop was corrected.

Linux, full lib+bin suite: 482 passed; 0 failed.

cargo fmt/clippy introduce no new findings in the touched files. Pre-existing drift in
src/lib.rs, src/main.rs, src/message.rs, src/metrics.rs, src/omp_session.rs,
src/service.rs, tests/otel_export.rs and two untouched spots in src/run.rs is left
alone — the flake deliberately does not gate either.

Complexity

Two test helpers (await_fixture_ready and await_ding_progress, ~15 lines each) and one
guard test. No production code paths added or changed; the two doc comments on
run_captured state an ordering the code already had.

Concerns

• The ordering is still a convention, now with a test. on_spawn blocking is not
  obvious from its signature. The guard test is the mitigation; a type that made the
  barrier explicit would be more machinery than the two call sites justify.
• The Darwin job roughly doubles CI wall-clock for this repo and hosted macOS minutes
  are metered differently from Linux. nix build .#st2 rather than nix flake check keeps
  that bounded, and concurrency (added here) stops superseded pushes from stacking runs.
• Expect the Darwin job to surface more of these. It already did once inside this PR —
  the fourth test above was found by the loaded rebuild, not by a report. That is the job
  working, but it means the first weeks may cost some triage.
• The most likely next one is already known.
  codex_app_server::tests::process_group_cleanup_reaps_a_native_launcher_descendant failed
  on my highest-load Darwin rebuild with the launcher did not create its native descendant: NotFound. fix(codex-app-server): deflake darwin-parallel test races #318 addresses it by raising that deadline from 1s to 5s and polling until the
  pid parses — a wider window rather than a barrier on the event — so it may still go red on
  a 3-core runner. Worth watching once both land; not a reason to hold either.

Friction & bottlenecks

• The reproduction cost a full ~10-minute Nix rebuild per iteration because the failure
  only reproduces inside the packaged suite. Shrinking the deadline locally on Linux gave
  the same evidence in seconds; that trick is worth reaching for first on any
  "load-sensitive test" report.
• Re-running a Nix build repeatedly to hunt a load-sensitive test has two silent traps and
  no good primitive. --rebuild refuses when the output is not already valid — including
  after a failed run — and its exit status also folds in a bit-reproducibility comparison
  that st2's Darwin build does not satisfy. Plain nix build is a cache hit that runs no
  tests. Three separate verification loops produced meaningless rows before I stopped
  trusting the exit status and graded on test result: lines instead.
• st2's aarch64-darwin build is not bit-reproducible (output ... differs on rebuild).
  Pre-existing and unrelated to this PR, but it is why --rebuild cannot be used as a
  pass/fail signal here.
• I restarted one verification loop on the shared Mac with pkill -f verify.sh, a pattern
  kill on a host others use. It matched only my own processes, but killing the recorded PID
  was the correct move and I should have done that.

Follow-ups

• Widen the Darwin job toward nix flake check once pty and otelite build reliably for
  aarch64-darwin in CI.
• Production patch_presentation and list_entries give the pty child a 2s budget that
  likewise spans fork+exec. On a loaded Darwin host that can time out spuriously; the
  effect is a logged error and a retry on the next reconcile pass, not a crash, so it is not
  addressed here. Worth a separate look — flagged rather than folded in.
• fix(codex-app-server): deflake darwin-parallel test races #318 / codex_app_server lib tests are load-flaky on macOS (pass isolated) #315 fix the same defect class in src/codex_app_server.rs. No file overlap with
  this PR, and both can land independently.

References

• Closes Darwin lifecycle tests consume behavior deadlines during fixture scheduling #356
• Closes st2 main does not pass its own test suite on macOS #255
• Refs fix(codex-app-server): deflake darwin-parallel test races #318, codex_app_server lib tests are load-flaky on macOS (pass isolated) #315 — same defect class, different module

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	Claude Code
agent_tool_version	2.1.237
agent_runtime	Claude Code 2.1.237
tooling_profile	dotfiles@11eaf2d-dirty

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c20ae4327

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".