What happened
The deterministic CI job failed once on Linux. It did not fail an assertion. The
step timeout 10m cargo test --locked --lib exited with code 124. The timeout
command killed the whole suite.
One test hung. The test harness printed the over-60-seconds warning for exactly
one name:
daemon::tests::half_closed_local_input_releases_guard_and_still_delivers_output
The warning appeared at 21:48:52Z. The kill happened at 21:57:20Z. The test
therefore stalled for 8.5 minutes or more.
Evidence: run 31843918056, job 94906377962, commit 02a176f, 2026-08-14.
The stall is in an unbounded await
Every wait inside the test body has a bound already:
read_shell_marker bounds itself at 20 seconds.
- PROOF 2 bounds the guard release at 500 milliseconds.
- PROOF 3 bounds the output read at 45 seconds.
A stall in any of those fails the test in under a minute. It cannot hold the job
for 8.5 minutes. The stall is therefore in one of the awaits that has no bound:
state.dial_alpn(...),
recycle_endpoint_if_generation(...), which first takes endpoint_recycle, or
client.shutdown() and server.shutdown().
FabricNode::shutdown cancels the token and then awaits the task with no bound.
A daemon task that misses the cancel hangs that call forever. This test recycles
the endpoint before it shuts down, so shutdown after a recycle is the first
candidate to examine.
It is not deterministic
A re-run of the identical commit passed in 3 minutes 32 seconds. The same test
passed 3 times out of 3 on macOS in about 3.9 seconds each.
macOS is a weak signal here. The test's own comment records that the remote tears
down in about 300 milliseconds on macOS, but takes over 40 seconds on Linux. The
platform that showed the hang is the platform with the long teardown.
Why this is worth an issue
The failure mode gives no diagnostic. timeout kills the process group, so there
is no panic, no backtrace, and no name for the stuck await. The reader only sees
exit code 124. A second occurrence should land here rather than be rediscovered.
Two directions are worth considering:
- Bound the shutdown await in the test, so a stall fails one test with a message
instead of killing the suite.
- Decide whether
FabricNode::shutdown should bound its own await. A daemon that
cannot stop within a bound is a production concern, not only a test concern.
I did not change either one. A single unreproduced observation is not enough
evidence to choose, and a bound added in the wrong place would hide the fault
rather than report it.
What happened
The
deterministicCI job failed once on Linux. It did not fail an assertion. Thestep
timeout 10m cargo test --locked --libexited with code 124. Thetimeoutcommand killed the whole suite.
One test hung. The test harness printed the over-60-seconds warning for exactly
one name:
The warning appeared at 21:48:52Z. The kill happened at 21:57:20Z. The test
therefore stalled for 8.5 minutes or more.
Evidence: run 31843918056, job 94906377962, commit 02a176f, 2026-08-14.
The stall is in an unbounded await
Every wait inside the test body has a bound already:
read_shell_markerbounds itself at 20 seconds.A stall in any of those fails the test in under a minute. It cannot hold the job
for 8.5 minutes. The stall is therefore in one of the awaits that has no bound:
state.dial_alpn(...),recycle_endpoint_if_generation(...), which first takesendpoint_recycle, orclient.shutdown()andserver.shutdown().FabricNode::shutdowncancels the token and then awaits the task with no bound.A daemon task that misses the cancel hangs that call forever. This test recycles
the endpoint before it shuts down, so shutdown after a recycle is the first
candidate to examine.
It is not deterministic
A re-run of the identical commit passed in 3 minutes 32 seconds. The same test
passed 3 times out of 3 on macOS in about 3.9 seconds each.
macOS is a weak signal here. The test's own comment records that the remote tears
down in about 300 milliseconds on macOS, but takes over 40 seconds on Linux. The
platform that showed the hang is the platform with the long teardown.
Why this is worth an issue
The failure mode gives no diagnostic.
timeoutkills the process group, so thereis no panic, no backtrace, and no name for the stuck await. The reader only sees
exit code 124. A second occurrence should land here rather than be rediscovered.
Two directions are worth considering:
instead of killing the suite.
FabricNode::shutdownshould bound its own await. A daemon thatcannot stop within a bound is a production concern, not only a test concern.
I did not change either one. A single unreproduced observation is not enough
evidence to choose, and a bound added in the wrong place would hide the fault
rather than report it.