Skip to content

fix(headless): lock valid structured verifier failures as scored results #1585

Description

@Astro-Han

What happened

The fixed-prompt headless controller treats a structured verifier pass as authoritative, but not a structured verifier failure.

When a Harbor cell has:

  • cell.status === "failed"
  • cell.errorClass === "runtime_error" or "infra_failed"
  • harbor.reward === 0
  • harbor.verifier.outcome === "failed"
  • a valid final verifier attempt with classification === "failed" and reward === 0

taskCompletedEvent() records the cell with scored: false.

The asymmetric control case is already accepted: when the same failed cell has a structured verifier pass with positive reward, structuredVerifierPassed makes it scored and passed.

This leaves a valid structured verifier failure replacement-eligible even though the verifier already produced a terminal pass/fail decision. It can cause unnecessary paid replacement attempts and can bias Pass@1 by retrying a result that should have been locked.

Two observed examples:

  • An OpenCode compile-compcert cell ended with errorClass: "infra_failed" and a structured verifier failure, but was recorded unscored.
  • A Maka qemu-startup cell ended with errorClass: "runtime_error" and a structured verifier failure, but was recorded unscored.

This is separate from the runtime trace-serialization defect addressed by #1558.

How to reproduce

  1. Exercise the fixed-prompt controller with a TaskRunOutput whose cell has:
    • status: "failed"
    • errorClass: "runtime_error" (repeat with "infra_failed")
  2. Give the output a valid Harbor result:
    • reward: 0
    • verifier.outcome: "failed"
    • final attempt { classification: "failed", reward: 0 }
  3. Observe that the emitted task_completed event has scored: false.
  4. Change the verifier outcome to a valid pass with positive reward.
  5. Observe that the failed cell is now accepted as scored: true.

The relevant logic is taskCompletedEvent() in packages/headless/src/fixed-prompt-controller.ts: structuredVerifierPassed participates in verifierGraded, but the corresponding structured verifier failure does not.

Expected behavior:

Once the structured verifier produces a valid passed or failed outcome, that result should be scored and locked regardless of the agent cell's terminal status. Only missing, malformed, or infrastructure-failed verifier outcomes should remain replacement-eligible.

Regression coverage should include:

  • structured pass on a failed agent cell → scored pass;
  • structured failure on a failed agent cell → scored fail;
  • verifier infrastructure failure → unscored;
  • a scored verifier failure is not eligible for replacement.

Environment

  • Maka observed subject commit: ee7e4ba5d
  • Confirmed affected logic remains at current commit: c561fed4e1
  • Surface: Headless / Harbor Terminal-Bench harness
  • macOS: 26.5.2
  • Node.js: v26.5.0

Logs, screenshots, or additional context

No provider HTTP 429 responses were involved in the observed examples. The issue concerns verifier authority and result classification, not provider availability.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions