fix(automationbench): record the harness's exit status on the k8s surface too

[yoavkatz]

What broke

Run 20260910-091433-61fe has 30 tasks that ran for ~38 minutes, burned real tokens, and produced multi-megabyte traces — then were killed at the wall clock before the verifier could grade them. The dashboard could only label them incomplete, never timeout, because the signal that separates the two was never recorded.

Task 10, for example: 150 model turns, 117,248 tokens, $0.08, a 6.5 MB trace whose last span lands mid-conversation at 09:54:43, and no result.json. Every incomplete task in the run is cut off within seconds of the same wall-clock moment while the graded ones finished early on their own — a deadline expiring, not 30 independent failures.

Why

Only run-agent writes /output/agent/.exit-code. automationbench is model-only: its preset replaces the standard three-phase pipeline with its own harness, so run-agent never runs — and python3 /app/run_automationbench.py || true swallowed the status on top of that. write-result coerces the missing file to JSON null:

EXIT_CODE=$(head -1 /output/agent/.exit-code 2>/dev/null | tr -d '[:space:]' || true)
[[ "$EXIT_CODE" =~ ^-?[0-9]+$ ]] || EXIT_CODE=null

Null is not a failure code, so a wall-clock kill and a task that never launched arrive at the dashboard identically. Its classifier reaches timeout only via isinstance(ec, int) && ec == 124, so these fall through to incomplete — correct given its inputs, and unfixable downstream.

compose.yaml was fixed in 82f9c01; the preset was not. The two surfaces therefore disagreed on a rule-16 field — a rule 24b lockstep break ("changes to the compose base or the chart MUST be reflected in the other in the same commit") and a rule 24 violation of byte-equivalent result.json across surfaces.

The fix

Both surfaces now bound the harness the way run-agent bounds an agent (timeout -k 30 $TIMEOUT, rule 14) and record $?.

The bound is load-bearing, not decoration. Without it the only remaining limit is the pod's activeDeadlineSeconds (timeout + deadlineGrace = 2400s here), whose SIGKILL takes down the whole pod and leaves no shell alive to write .exit-code — so recording alone would still produce null at exactly the moment it matters. The inner 1800s bound fires first, the harness dies with 124, and the dashboard labels it timeout with no dashboard change.

Why CI missed it

Rule 29's gates are per-surface: each renders and parses fine alone. The drift lives in runnerArgs — the one per-benchmark command a preset may still define — so the rules' "one chart and one container recipe cannot drift from themselves" reasoning does not extend to it.

Two static gates now close that:

• a_preset_that_replaces_the_agent_phase_records_its_exit_status — any preset overriding runnerArgs must record a status (.exit-code, or tau-bench's exit $rc).
• a_preset_that_replaces_the_agent_phase_bounds_its_harness — any preset recording its own status must bound the harness with timeout -k.

tau-bench already satisfied both; automationbench was the only gap across all nine presets. The first gate fails on the pre-fix preset (verified by reverting it).

Verification

• helm template --set benchmark=automationbench renders the 1800s inner bound inside the 2400s pod deadline, $? untouched by Helm.
• docker compose config passes (rule 27); renders $$? so the container shell receives $?.
• Static suite green (34 check tests). The task_inspection failure (14 red trace-fixture findings) is pre-existing on main — confirmed on a clean tree, unrelated.
• Not run: a live on-cluster eval confirming a 124 end-to-end. Worth one timeout-bound run before trusting the label.

Follow-up

The 30 incomplete tasks in that run are rerunnable, but this model wasn't converging inside 30 minutes — a rerun wants a larger EVAL_TIMEOUT (--set timeoutOverride=), not just a relaunch.

🤖 Generated with Claude Code