What happened
The fixed-prompt headless controller treats a structured verifier pass as authoritative, but not a structured verifier failure.
When a Harbor cell has:
cell.status === "failed"
cell.errorClass === "runtime_error" or "infra_failed"
harbor.reward === 0
harbor.verifier.outcome === "failed"
- a valid final verifier attempt with
classification === "failed" and reward === 0
taskCompletedEvent() records the cell with scored: false.
The asymmetric control case is already accepted: when the same failed cell has a structured verifier pass with positive reward, structuredVerifierPassed makes it scored and passed.
This leaves a valid structured verifier failure replacement-eligible even though the verifier already produced a terminal pass/fail decision. It can cause unnecessary paid replacement attempts and can bias Pass@1 by retrying a result that should have been locked.
Two observed examples:
- An OpenCode
compile-compcert cell ended with errorClass: "infra_failed" and a structured verifier failure, but was recorded unscored.
- A Maka
qemu-startup cell ended with errorClass: "runtime_error" and a structured verifier failure, but was recorded unscored.
This is separate from the runtime trace-serialization defect addressed by #1558.
How to reproduce
- Exercise the fixed-prompt controller with a
TaskRunOutput whose cell has:
status: "failed"
errorClass: "runtime_error" (repeat with "infra_failed")
- Give the output a valid Harbor result:
reward: 0
verifier.outcome: "failed"
- final attempt
{ classification: "failed", reward: 0 }
- Observe that the emitted
task_completed event has scored: false.
- Change the verifier outcome to a valid pass with positive reward.
- Observe that the failed cell is now accepted as
scored: true.
The relevant logic is taskCompletedEvent() in packages/headless/src/fixed-prompt-controller.ts: structuredVerifierPassed participates in verifierGraded, but the corresponding structured verifier failure does not.
Expected behavior:
Once the structured verifier produces a valid passed or failed outcome, that result should be scored and locked regardless of the agent cell's terminal status. Only missing, malformed, or infrastructure-failed verifier outcomes should remain replacement-eligible.
Regression coverage should include:
- structured pass on a failed agent cell → scored pass;
- structured failure on a failed agent cell → scored fail;
- verifier infrastructure failure → unscored;
- a scored verifier failure is not eligible for replacement.
Environment
- Maka observed subject commit:
ee7e4ba5d
- Confirmed affected logic remains at current commit:
c561fed4e1
- Surface: Headless / Harbor Terminal-Bench harness
- macOS: 26.5.2
- Node.js: v26.5.0
Logs, screenshots, or additional context
No provider HTTP 429 responses were involved in the observed examples. The issue concerns verifier authority and result classification, not provider availability.
What happened
The fixed-prompt headless controller treats a structured verifier pass as authoritative, but not a structured verifier failure.
When a Harbor cell has:
cell.status === "failed"cell.errorClass === "runtime_error"or"infra_failed"harbor.reward === 0harbor.verifier.outcome === "failed"classification === "failed"andreward === 0taskCompletedEvent()records the cell withscored: false.The asymmetric control case is already accepted: when the same failed cell has a structured verifier pass with positive reward,
structuredVerifierPassedmakes it scored and passed.This leaves a valid structured verifier failure replacement-eligible even though the verifier already produced a terminal pass/fail decision. It can cause unnecessary paid replacement attempts and can bias Pass@1 by retrying a result that should have been locked.
Two observed examples:
compile-compcertcell ended witherrorClass: "infra_failed"and a structured verifier failure, but was recorded unscored.qemu-startupcell ended witherrorClass: "runtime_error"and a structured verifier failure, but was recorded unscored.This is separate from the runtime trace-serialization defect addressed by #1558.
How to reproduce
TaskRunOutputwhose cell has:status: "failed"errorClass: "runtime_error"(repeat with"infra_failed")reward: 0verifier.outcome: "failed"{ classification: "failed", reward: 0 }task_completedevent hasscored: false.scored: true.The relevant logic is
taskCompletedEvent()inpackages/headless/src/fixed-prompt-controller.ts:structuredVerifierPassedparticipates inverifierGraded, but the corresponding structured verifier failure does not.Expected behavior:
Once the structured verifier produces a valid
passedorfailedoutcome, that result should be scored and locked regardless of the agent cell's terminal status. Only missing, malformed, or infrastructure-failed verifier outcomes should remain replacement-eligible.Regression coverage should include:
Environment
ee7e4ba5dc561fed4e1Logs, screenshots, or additional context
No provider HTTP 429 responses were involved in the observed examples. The issue concerns verifier authority and result classification, not provider availability.