Skip to content

flaky: engine TestAdoptRootReparentsGrandchildPastSettledIntermediateWithoutCommittedOutcome can hang the package for the full 10m timeout #275

Description

@andybons

Failure

TestAdoptRootReparentsGrandchildPastSettledIntermediateWithoutCommittedOutcome blocks forever on a channel receive and takes the whole engine package to its 10-minute test timeout. The package reports FAIL with no test-level failure message, so the cause is only visible in the panic's goroutine dump.

Observed once in CI on 2026-09-09 (run 34395100933), on a provider/openai and docs-only change that touches no engine/ file:

panic: test timed out after 10m0s
  running tests:
    TestAdoptRootReparentsGrandchildPastSettledIntermediateWithoutCommittedOutcome (9m49s)

goroutine 2923 [chan receive, 9 minutes]:
github.com/majorcontext/harness/engine.(*resumeClaims).wantClaim(0xc000dbc7b0, 0xc000573500, {0xc000e0c5c0, 0x1e})
	engine/recovery_harness_test.go:212 +0x46
github.com/majorcontext/harness/engine.(*resumeClaims).waitSettled(0xc000dbc7b0, 0xc000573500, 0xc00061e160, {0xc000e0c5c0, 0x1e})
	engine/recovery_harness_test.go:226 +0x5e
github.com/majorcontext/harness/engine.TestAdoptRootReparentsGrandchildPastSettledIntermediateWithoutCommittedOutcome(0xc000573500)
	engine/session_manager_delivery_test.go:3653 +0x1e28

FAIL	github.com/majorcontext/harness/engine	600.069s

resumeClaims.wantClaim (engine/recovery_harness_test.go:212) waits on an unbuffered claim channel with no deadline of its own, so a claim that never arrives becomes a package-wide hang rather than a named assertion failure.

Reproduction rate

Rare, and it did not reproduce locally:

what base result
CI, full go test -race ./... 088261a (diff identical to merged #268) hang, 9m49s
local, 6 × go test -count=1 -race ./engine/ main @ 0045923 6/6 green, 79–81s each
local, 5 × isolated -run of this test main @ 155ede5 5/5 green, 1.1s

So it appears to need full-suite concurrency and load. -count=1 matters when reproducing: without it, Go's test cache silently serves repeat runs and a loop measures nothing.

Not the other flake

This is distinct from the TestChildTurnStartAndEndObserversConcurrentAcrossManyChildren flake fixed by #269, which touches engine/session_manager_child_turn_start_test.go. The base here already includes #269. It is the same shape as #137 (a hang to the full 10-minute timeout) but in a different package.

Why it is worth a fix rather than a retry

A hang costs the full 10-minute budget and reports no cause at the test level, so it reads as a broken build on an unrelated PR. Per AGENTS.md, an in-process wait like this should block on a production notification seam with a failure bound, so a missing claim fails fast and names itself.

Found while investigating Codex WebSocket lineage (#268); filed so it has a tracked owner rather than living only in a dead CI log.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions