Skip to content

test(monitor): TestRealEndToEnd flakes on the Thinking… indicator and on turn-mark assertions #162

Description

@andybons

Problem

tools/monitor/e2e's TestRealEndToEnd (tools/monitor/e2e/real_e2e.mjs)
fails intermittently in CI with at least three different assertions:

Error: timed out waiting for: the Thinking… pending indicator appears while the turn is busy with no content yet
AssertionError: a freshly created, never-prompted session must open with no turn marks: 2
AssertionError: the operator entry must also precede the turn's streaming assistant reply, not just the pending indicator that preceded it

Observed on PR CI runs across several unrelated commits. It passes 3 of 3
locally (go test -race -count=1 ./tools/monitor/e2e/), so it is
load-sensitive rather than a real regression, and it forces a CI re-run on
otherwise-green PRs.

Cause (partial)

The failing assertions all observe a live UI state that only exists while a
turn is mid-flight:

  • The Thinking… indicator exists only between "turn started" and "first
    token rendered". The script already treats one such observation as
    opportunistic (if (pendingIndicatorEl), with a NOTE branch and a
    comment explaining that a fast runner dismisses the indicator first), but
    another site still hard-waits for it.
  • "a freshly created, never-prompted session must open with no turn marks"
    reads a session the harness has already driven in an earlier scenario, so
    it depends on the earlier turn's marks having settled.

The pattern is the same in each: the assertion needs a state that a fast
or slow runner can move past, and it is not derived from a condition that
stays true.

Fix

Make each of these observe a stable condition instead of a transient one,
the same way the DOM-order check already does: it verifies the operator
entry precedes the streaming reply, which holds whether or not the
indicator was ever seen. Where a transient state genuinely must be
observed, gate the whole assertion on having caught it (the existing
if (pendingIndicatorEl) shape) rather than hard-waiting.

The rule this keeps: a waitFor must wait for a condition that stays true
once reached.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions