feat(terminal-bench): selectable agent harness plus full control and tracking

[abhinav-pola]

Why

Terminal Bench hardcoded the pi coding agent, so a result measured pi + model with no way to compare a different agent on the same tasks. This PR makes the agent selectable across the Harbor benchmarks, converges Terminal Bench onto the Harbor sandbox it should always have used, and fills in the control knobs and result columns that were missing.

Twelve commits, each reviewable on its own.

Agent selection

Every agent is launched through an Ori Harness passthrough command (ori pi, ori claude). Ori is not a proxy: it injects OpenRouter auth and execs the real CLI, resolving credentials from the OPENROUTER_API_KEY the sandbox already receives, and installs as a standalone Linux binary.

benchmark	agent values	default
terminal_bench	pi, claude	pi
swe_atlas_{qa,tw,rf}, deep_swe	mini_swe, pi, claude	mini_swe

mini_swe is the pre-existing in-harness loop, named for what it is — a mini-swe-agent scaffold (MINI_SWE_SYSTEM_MESSAGE, one bash tool, COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT sentinel). It was briefly called native in this branch, which described nothing and became actively misleading once the genuinely native pi path was retired.

wandr is excluded: it injects OpenRouter server tools into its own requests, which an external CLI replaces with its own, so it would stop measuring what it is for.

bun src/cli/index.ts --benchmark terminal_bench --model anthropic/claude-opus-5 \
  --solver-config '{"agent":"claude","taskSubset":["fix-git"]}'

bun src/cli/index.ts --benchmark deep_swe --model anthropic/claude-opus-5 \
  --solver-config '{"agent":"pi","taskSubset":["abs-stepped-slices"]}'

Control

One agentReasoningEffort (none…max) goes to ori --reasoning-effort, which each harness translates natively (pi --thinking, claude --effort). Verified on stable ori: none yields 0 reasoning tokens, high yields 58.

Also available on every agent: systemPrompt, appendSystemPrompt, allowedTools, disallowedTools, isolateAgentConfig, agentPackage, oriChannel.

The pi-only thinking and claude-only effort fields are removed, not aliased. An alias took precedence over the new field, so a config carrying both — likely on replay, since thinking was defaulted on main and the whole validated config is stored in the parquet benchmark_config column — would silently run at the legacy level.

Removing them alone would only have relocated the hazard, because zod strips unknown keys: {"thinking":"high"} was silently discarded, and so was a typo like agentReasoningEfort while the real field quietly took its default. buildSchemaValidatedConfig now rejects any solver-config key absent from the benchmark's own options schema or the shared model base, naming the offenders. This gives BENCHMARK_OPTIONS_SCHEMAS its first runtime consumer and covers new benchmarks automatically.

Tracking

generation_ids resolves against the OpenRouter generations API for either agent, so spend is reconcilable from stored results.

field	pi	claude
generation_ids	responseId	message.id
cost	usage.cost.total	total_cost_usd
reasoning tokens	usage.reasoning	thinking_tokens
generation_time_ms	wall-clock	duration_ms
agentTurns / agentToolCalls	event counts	num_turns / tool_use blocks
response_items	stream events	stream events

ori-parquet.test.ts runs the solver through the real parquet writer, reads the file back, and asserts no unexpectedly null columns. request_body, extra_scores and primary_score stay null: there is no single request body for an external CLI, and terminal_bench is binary pass/fail so accuracy already is the score.

Session attribution goes out as X-Session-Id via ORI_OPENROUTER_SESSION_ID, confirmed through a local TLS-terminating proxy on ori 0.7.1. Ori replaces a value containing a control character with a fresh UUID silently, which would detach a run from its generations, so both the CLI and the solver reject such a value rather than letting attribution vanish.

Harbor convergence, and a dropped-settings bug

Terminal Bench is a Harbor benchmark — its tasks come from harbor-framework/terminal-bench-2-1 and its task.toml is the format deep-swe parses — but it carried a parallel reimplementation of the harbor sandbox that silently dropped per-task settings.

TaskTomlSchema requires cpus, memory_mb and gpus, but the dataset never read them and the local Modal layer passed only a timeout, workdir and command. Every task ran at Modal's defaults: 22 of 90 request more than 2048 MB and 6 request more than 1 CPU. An under-provisioned task OOMs or thrashes and scores 0, which reads as model failure.

terminal-bench/sandbox.ts and terminal-bench/modal-sandbox.ts are deleted. session.ts builds harbor's CreateSessionInput with the declared resources and the task files as uploads, and runTerminalBenchVerifier replaces the sandbox-owned runTests(). Reward parsing is unchanged in practice: all 90 tasks write echo 1 or echo 0, so harbor's parseReward and the old strict === "1" agree on this dataset.

Running as root is correct here

Claude Code refuses --dangerously-skip-permissions as root, and Modal sandboxes run as root, so the first real run of the claude path failed every task before the agent did any work — zero tokens, zero cost, reward 0. The run script sets IS_SANDBOX=1, after which the same task completed cleanly.

Anthropic documents running Claude Code as a non-root user in containers. That is the wrong answer for this benchmark: across the 90 tasks, 122 shell files run root-requiring commands (114 apt-get install, 27 pip install, 2 useradd, 1 systemctl restart). A non-root agent could not install packages and would fail those tasks for environmental reasons; restoring the ability with passwordless sudo trips the identical refusal. The check exists to stop an agent wrecking a real system, and this is a disposable single-task container where root is the task.

Residual risk is that IS_SANDBOX is undocumented (anthropics/claude-code#58150 asks for it to be specified). If it stops working the failure is loud rather than silent — exit 1 with the exact message captured in testOutput and agentExitCode in metadata.

Network policy on the CLI path

The agent runs inside the sandbox and must reach OpenRouter. deep_swe declares allow_internet = false by default, which the Modal layer maps to blockNetwork: true, so a CLI agent there would have failed every task with no route to the model. The agent session now gets egress when a CLI agent is selected; where the verifier has its own sandbox (deep-swe) it keeps the task's declared policy so scoring stays offline. swe-atlas and terminal-bench share one sandbox between agent and verifier, so egress covers both — moot in practice, since swe-atlas defaults to true and all 90 terminal-bench tasks declare true. When the policy is overridden, agentNetworkForced and taskAllowInternet land in sample metadata.

The grading contract reaches the agent

swe_atlas and deep_swe encode how work is handed in inside buildInstanceMessage, not in instruction.md: qa must write /logs/agent/answer.txt in <<FINAL_ANSWER>> tags, tw must write /logs/agent/manifest.txt in <<TEST_MANIFEST>> tags, rf starts at / and must locate the repo root without touching test files, and deep-swe is graded from git commits with uncommitted changes discarded. The CLI path uploaded only the raw instruction, so an agent could do the work correctly and score zero. Each benchmark now passes a CLI-appropriate variant of its contract as the appended system prompt, preserving any caller-supplied prompt alongside.

Real Modal runs

agent	exit	gen_ids	turns	cost	reasoning	time_ms
pi	137	224	223	$1.203	88	780,168
claude (before IS_SANDBOX)	1	0	–	$0.000	0	2,048
claude (after)	0	8	15	$0.376	757	38,404

pi's 0 is honest: it used its full 750s budget and was SIGKILLed, the timeout firing 168ms past the configured bound. claude's 0 is a clean completion that did not solve the task. Cost reconciliation checks out to the digit — pi's reported per-generation cost matched OpenRouter's billing on every sampled generation — and prompt caching is working (99.98% cache reads on the pi run).

Behavior changes

• Honoring declared resources will move scores on the 28 tasks requesting more than default cpus or memory. Presumably upward, but not comparable to previous runs.
• terminalBenchScorer emits a verifier_log trajectory, filling scorer_trajectory for both agents. Additive; score values unchanged.
• pi parquet columns change value: generation_time_ms and reasoning_tokens from hardcoded 0 to real numbers, response_items from null to the full event stream, so files get larger.
• CLI-path results measure a different system than mini_swe and are not comparable to existing harbor baselines. mini_swe remains the default everywhere it existed.
• A pi request failing at the API used to exit 0 with zero usage, indistinguishable from a cheap successful run. It is now surfaced. Past pi runs showing reward 0 with zero cost are worth re-checking.

isolateAgentConfig defaults to false, preserving current pi behavior. With the default, pi reads AGENTS.md/CLAUDE.md from the task working directory, so task-shipped files can influence the agent; the knob exists to rule that out, but flipping the default would move scores.

Breaking change

./benchmarks/terminal-bench/sandbox and ./benchmarks/terminal-bench/modal-sandbox, added in #22, are removed — the convergence deletes those modules. ./benchmarks/harbor/sandbox and ./benchmarks/harbor/modal-sandbox are exported in their place, plus ./benchmarks/agent-cli/schema for the agent enums. A re-export shim was rejected: the interfaces genuinely differ (runTests() and a task-file triple versus uploads, resources and attach), so it could not be type-compatible.

Verification

All five gate commands exit 0: format:check, check, typecheck, bun test (1321 pass, 0 fail), build. Rebased on current main.

Flags and parsers were checked against the real CLIs, not just fixtures: --reasoning-effort verified to change reasoning-token counts, parsers run against captured success and failure streams from both agents, generation ids resolved against /api/v1/generation, generated run scripts pass bash -n in minimal and all-knobs-on forms, and a dataset test walks all 90 real task files asserting every declared cpus and memory survives into metadata.

Known gaps

• Response caching is not applied on the CLI path. mini_swe gets it through responses-client.ts; agent CLIs make their own requests. fix(response-cache): send cache salt as header #36 moved the salt from a body field to x-openrouter-cache-salt, so all three pieces are now headers and injection through pi's models.json headers has become feasible — not done here.
• swe_atlas and deep_swe have never run on Modal. Both the network fix and the grading-contract fix were written for them and are unit-tested only. Given that this branch's two worst bugs were findable only by running, those paths should get one real task each before their numbers are trusted.
• stepLimit is inert on the CLI path. Neither CLI exposes a turn limit, so wall-clock is the only bound. One haiku task cost $1.20 over 223 turns; a 90-task sweep is $100+ on haiku alone.
• oriInstallUrl and agentPackage are interpolated into Dockerfile RUN steps without validation, as flagged by Devin Review. Config reaches them only via --solver-config from a caller who already has a shell, so this is hardening rather than a live boundary; deliberately left out by decision.
• Claude's X-Session-Id forwarding is unconfirmed. Claude Code ignores HTTP_PROXY, so it bypassed the observation method that confirmed pi.

🤖 Generated with Claude Code

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 4 additional findings in Devin Review.