Skip to content

fix(codex-app-server): deflake darwin-parallel test races - #318

Open
schickling wants to merge 1 commit into
mainfrom
fix/codex-app-server-darwin-test-flakes
Open

fix(codex-app-server): deflake darwin-parallel test races#318
schickling wants to merge 1 commit into
mainfrom
fix/codex-app-server-darwin-test-flakes

Conversation

@schickling

Copy link
Copy Markdown
Contributor

Fixes #315

Problem

Two codex_app_server lib tests fail intermittently under full-suite parallel load on macOS arm64 while passing deterministically in isolation (rate table + panics in #315):

  • process_group_cleanup_reaps_a_native_launcher_descendantParseIntError { kind: Empty }
  • subscribed_control_pump_delivers_a_typed_reference_to_the_real_fifo_headAttempted != Accepted

Baseline on main (d29c37f2), five full-suite runs across two days: 3 of 5 failed, always exactly one of these tests.

Root causes

  1. The launcher shell does printf "%s" "$$" > pidfile: the file is created before the bytes land. The test waited for is_file() then read once — an empty read under load panics on parse.
  2. The test asserted delivery-state == Accepted immediately after joining pump and fake-server threads. The pump persists Accepted synchronously, but its final websocket frames race the fake server's exit under descheduling, so the pump can terminate before processing item/completed.

Fix (tests only, no production changes)

  1. Wait for parseable pid content within a bounded deadline instead of a single read after existence.
  2. Poll for the Accepted outcome (bounded 10s), bail early if the pump dies, then join and assert. Same contract — delivery must reach Accepted — observed load-tolerantly.

Verification

macOS arm64, fix commit, three consecutive full-suite runs:

RUN1: test result: ok. 349 passed; 0 failed
RUN2: test result: ok. 349 passed; 0 failed
RUN3: test result: ok. 349 passed; 0 failed

Linux x86_64: cargo test -p st2 --lib → 355 passed; 0 failed.
Isolation loops post-fix: both targets 10/10 pass.

Complexity

Test-only; ~20 lines across two tests, no new abstractions.

Two lib tests failed intermittently under full-suite parallel load on
macOS while passing deterministically in isolation (#315):

- process_group_cleanup_reaps_a_native_launcher_descendant: the shell
  redirection creates the pidfile before the pid bytes land, so reading
  immediately after is_file() observed an empty file and panicked with
  ParseIntError { kind: Empty }. Wait for parseable content within a
  bounded deadline instead of a single read.

- subscribed_control_pump_delivers_a_typed_reference_to_the_real_fifo_head:
  asserted delivery-state == Accepted immediately after joining the pump,
  but the pump's final websocket frames race the fake server's exit under
  load, so it could exit before persisting Accepted. Poll for the outcome
  (bounded), bail early if the pump dies, then join and assert.

No production changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codex_app_server lib tests are load-flaky on macOS (pass isolated)

1 participant