Skip to content

Parameterize run-timeout coverage across benchmark lifecycle stages #465

Description

@viraatc

Context

PR #409 adds integration coverage for settings.timeouts.run_timeout_s during service launch, live performance, and metrics drain. Coverage is not yet organized as one sharp lifecycle-stage matrix equivalent to the SIGINT integration coverage.

Keep this as a test-focused follow-up rather than expanding PR #409 further. That PR is already approximately 1,510 non-test changed lines against the 1,300-line review target; this work would not add production behavior, but it would increase an already large review.

Lifecycle matrix

Parameterize the same run-watchdog contract across:

  1. Service startup.
  2. Endpoint-worker startup/connect.
  3. Live session execution.
  4. In-flight response drain.
  5. Metrics tokenization drain.

Accuracy scoring/finalization is intentionally excluded and tracked separately in #464.

Acceptance criteria

  • Each stage deterministically blocks at a known lifecycle boundary before the watchdog fires.
  • Every case surfaces ExecutionError attributed to the run timeout.
  • No case yields a complete: true run-level report.
  • Child services and endpoint workers are not leaked.
  • Overlapping one-off timeout tests are removed or folded into the parameterized matrix.
  • Test synchronization uses observable stage boundaries rather than arbitrary sleeps.
  • No production timeout behavior changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions