Skip to content

A lib test hangs on Linux and kills the whole suite with no diagnostic #54

Description

@myobie

What happened

The deterministic CI job failed once on Linux. It did not fail an assertion. The
step timeout 10m cargo test --locked --lib exited with code 124. The timeout
command killed the whole suite.

One test hung. The test harness printed the over-60-seconds warning for exactly
one name:

daemon::tests::half_closed_local_input_releases_guard_and_still_delivers_output

The warning appeared at 21:48:52Z. The kill happened at 21:57:20Z. The test
therefore stalled for 8.5 minutes or more.

Evidence: run 31843918056, job 94906377962, commit 02a176f, 2026-08-14.

The stall is in an unbounded await

Every wait inside the test body has a bound already:

  • read_shell_marker bounds itself at 20 seconds.
  • PROOF 2 bounds the guard release at 500 milliseconds.
  • PROOF 3 bounds the output read at 45 seconds.

A stall in any of those fails the test in under a minute. It cannot hold the job
for 8.5 minutes. The stall is therefore in one of the awaits that has no bound:

  • state.dial_alpn(...),
  • recycle_endpoint_if_generation(...), which first takes endpoint_recycle, or
  • client.shutdown() and server.shutdown().

FabricNode::shutdown cancels the token and then awaits the task with no bound.
A daemon task that misses the cancel hangs that call forever. This test recycles
the endpoint before it shuts down, so shutdown after a recycle is the first
candidate to examine.

It is not deterministic

A re-run of the identical commit passed in 3 minutes 32 seconds. The same test
passed 3 times out of 3 on macOS in about 3.9 seconds each.

macOS is a weak signal here. The test's own comment records that the remote tears
down in about 300 milliseconds on macOS, but takes over 40 seconds on Linux. The
platform that showed the hang is the platform with the long teardown.

Why this is worth an issue

The failure mode gives no diagnostic. timeout kills the process group, so there
is no panic, no backtrace, and no name for the stuck await. The reader only sees
exit code 124. A second occurrence should land here rather than be rediscovered.

Two directions are worth considering:

  1. Bound the shutdown await in the test, so a stall fails one test with a message
    instead of killing the suite.
  2. Decide whether FabricNode::shutdown should bound its own await. A daemon that
    cannot stop within a bound is a production concern, not only a test concern.

I did not change either one. A single unreproduced observation is not enough
evidence to choose, and a bound added in the wrong place would hide the fault
rather than report it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions