Context
PR #409 consolidates the existing benchmark timeout configuration. accuracy_drain_timeout_s only bounds outstanding endpoint responses from the accuracy load phase. It does not cover score_accuracy() or finalization, and scorers with SKIP_ENDPOINT_PHASE bypass that response drain entirely.
The run watchdog is currently cancelled before finalize_benchmark(). External-service scorers such as SWE-bench and subprocess-based scorers such as VBench/LiveCodeBench can therefore wait without a benchmark-level deadline.
Scope
Add an explicit deadline for accuracy scoring and finalization in a follow-up MR. Define how it composes with run_timeout_s rather than silently changing the existing response-drain timeout.
Acceptance criteria
- Scoring/finalization has an explicit, documented deadline.
- External-service and subprocess scorers have defined cancellation/termination behavior.
SKIP_ENDPOINT_PHASE scorers are covered.- Expiry produces honest incomplete/interrupted artifacts and a non-zero exit.
- No scorer process or service is leaked after expiry.
- Integration coverage exercises service-backed and subprocess-backed timeout paths.
Context
PR #409 consolidates the existing benchmark timeout configuration.
accuracy_drain_timeout_sonly bounds outstanding endpoint responses from the accuracy load phase. It does not coverscore_accuracy()or finalization, and scorers withSKIP_ENDPOINT_PHASEbypass that response drain entirely.The run watchdog is currently cancelled before
finalize_benchmark(). External-service scorers such as SWE-bench and subprocess-based scorers such as VBench/LiveCodeBench can therefore wait without a benchmark-level deadline.Scope
Add an explicit deadline for accuracy scoring and finalization in a follow-up MR. Define how it composes with
run_timeout_srather than silently changing the existing response-drain timeout.Acceptance criteria
SKIP_ENDPOINT_PHASEscorers are covered.