Skip to content

fix(tests): drain pty master so macOS integration tests cannot hang; bound all waits - #254

Merged
bkrabach merged 1 commit into
mainfrom
fix/macos-integration-tests
Aug 3, 2026
Merged

fix(tests): drain pty master so macOS integration tests cannot hang; bound all waits#254
bkrabach merged 1 commit into
mainfrom
fix/macos-integration-tests

Conversation

@bkrabach

@bkrabach bkrabach commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What

The pytest -m integration job added in #251 and wired up in #253 hangs forever on the macOS runner (18-24+ min before I cancelled it; the ubuntu job finishes in 39s). #253 was merged believing that job had passed. It had not — it had never completed on macOS, and it never passed.

Root-caused on real macOS hardware (26.6, Darwin 25.6.0, arm64). Both symptoms are harness bugs. The product is correct on macOSgit diff --stat -- amplifier_app_cli/ for this PR is empty.

Root cause: an un-drained pty master wedges the child on macOS

macOS wedges an exiting pty child whose output queue is never drained — not slowly, unreapably:

linux   child wrote  10000B, parent_reads=False -> reaped@0.02s
darwin  child wrote  10000B, parent_reads=False -> *** NEVER REAPED ***  SIGKILL reaped=False
darwin  child wrote 100000B, parent_reads=True  -> reaped normally

A wedged child lands in ps state ?Es — Exiting, session leader, controlling terminal already revoked. That one state produces both failures:

1. OSError: [Errno 5] in 3 echo tests — the EIO is on a write, not a read (test_terminal_echo_integration.py:260, inside send()). The platform difference:

linux   slave CLOSED (child exited)  write=OK          read=EOF
darwin  slave CLOSED (child exited)  write=OSError EIO read=EOF

Caught in the act with ps sampled per write: w1@1.11s ps='Ss+' OK | w2@1.30s ps='Z' EIO. The child had already wedged and died before the parent's hardcoded sleep(0.6)+sleep(0.3) fired. On Linux the child is still alive at 0.9s, so the byte lands and the race is invisible.

Note what the tempting fix would have done: swallowing the EIO would make macOS green while never delivering the Ctrl-C — a vacuously passing test. Fixed properly instead: drain the pty so the child never wedges, and replace fixed sleeps with a readiness handshake so the send lands inside the child's live window.

2. The hang--faulthandler-timeout pinned it exactly:

test_single_ctrlc_requests_graceful_cancellation Timeout (0:00:25)!
  File "tests/test_ctrlc_functional_integration.py", line 302 in _run_pty_scenario
301    os.kill(pid, signal.SIGKILL)
302    os.waitpid(pid, 0)        # blocks forever; a ?Es child can never be reaped

The product is acquitted, with evidence

Ctrl-C cancellation works on macOS — the wedged child had already written its result before dying:

{"scenario": "single_ctrlc", "elapsed": 4.07, "state": "graceful", "is_cancelled": true,
 "messages": ["\n[yellow]Stopping after current operation completes... (Ctrl+C again to force)[/yellow]"]}

That is exactly what the test asserts.

Termios restoration works on macOS. Same driver, only difference is a drain thread:

darwin drain=False  every scenario -> exited=False, ps_final='?Es'
darwin drain=True   all 6 scenarios -> exited=True, termios={'ECHO': True, 'ICANON': True, 'ISIG': True}

The fix

New tests/pty_harness.py; both drivers converted (124 insertions, 174 deletions in the two test files — net smaller).

  • Drain thread reads a dup() of the master, so the caller's master_fd keeps blocking-write semantics and termios.tcgetattr() still reads real state.
  • send() / signal() raise AssertionError on undeliverable input — a test can never silently verify nothing.
  • No blocking waitpid anywhere. wait() and kill() are bounded WNOHANG polls. A test may fail; a test must never hang.
  • wait_for_marker() readiness handshake replaces parent-side sleeps.
  • Same unbounded waitpid(pid, 0) removed from test_stdout_offload_freeze_integration.py:198 — that test deliberately builds the un-drained-pty + SIGKILL condition, so it was the next landmine.

CI hardening: timeout-minutes: 10 on both jobs. Typical runtime is 20-30s. A job stuck at in_progress reads as "not done yet" rather than "broken" — which is precisely how a false green got merged. Now a hang fails loudly in 10 minutes instead of burning runner hours toward the 6h default on runners that bill at 10x.

Deliberate-breakage proof (tests still catch regressions)

break linux darwin
raw_mode.__exit__ never restores (the literal user-facing symptom) 6 failed, 1 passed (3/3 reps) 6 failed, 1 passed (3/3 reps)
Ctrl-C forwarding neutered (pre-fix behavior) 3 failed — assert 'none' == 'graceful'
orphan-prevention backstop removed 6/6 caught 1/6 caught

The Ctrl-C break failing on macOS also directly disproves vacuity: the \x03 byte really does reach the child's _CtrlCInterrupt path there.

The 1/6 on the last row is a property of the bug, not the probe — it depends on asyncio shutdown sweep order. When the last orphan to restore happens to be the one holding pristine attrs, the terminal comes out healthy by luck: exit_order=[1,3,0,2] -> CAUGHT, exit_order=[2,1,3,0] -> missed.

Verification — all four combinations

                              integration          default suite
linux   no controlling tty    13 passed / 25.9s    1282 passed, 1 skipped, 13 deselected, 1 xfailed
linux   WITH controlling tty  13 passed / 16.8s    1282 passed, 1 skipped, 13 deselected, 1 xfailed
darwin  no controlling tty    13 passed / 29.0s    1283 passed, 13 deselected, 1 xfailed
darwin  WITH controlling tty  13 passed / 18.5s    1283 passed, 13 deselected, 1 xfailed

macOS: 5 consecutive integration runs, all green, no stray ?Es children left behind. (Before this fix, one was still parked hours after the run that spawned it.)

Follow-up to #251 and #253.

…bound all waits

The `pytest -m integration` job added in #251 and wired up in #253 hangs
forever on the macOS runner. Root-caused on real macOS hardware to a test-harness
bug (not a product bug): an un-drained pty master wedges the child on macOS.

Both symptoms are harness bugs. The product is correct on macOS.

Root cause: macOS wedges an exiting pty child whose output queue is never
drained — not slowly, unreapably. A wedged child lands in ps state `?Es`
(Exiting, session leader, controlling terminal already revoked).

The fix introduces a shared pty harness (`tests/pty_harness.py`) that:
- Drains the pty master via a dedicated thread on a dup() so the caller's
  master_fd keeps blocking-write semantics
- Replaces parent-side sleeps with `wait_for_marker()` readiness handshake
  so sends land inside the child's live window
- Bounds all `waitpid` calls — no blocking waits that can hang forever
- Removes the un-drained-pty + SIGKILL condition that was a landmine in
  test_stdout_offload_freeze_integration.py:198

CI hardening: `timeout-minutes: 10` on both jobs. Typical runtime is 20-30s.
A hung job now fails loudly in 10 minutes rather than burning runner hours.

Verification: 5 consecutive integration runs on macOS, all green, no stray
`?Es` children left behind.

Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
bkrabach marked this pull request as ready for review August 3, 2026 14:13
@bkrabach
bkrabach merged commit 9c5a58f into main Aug 3, 2026
7 checks passed
@bkrabach
bkrabach deleted the fix/macos-integration-tests branch August 3, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants