Xal Benchmarks

Reproducible evaluations and public run artifacts for Xal, a terminal coding harness with a headless agent core.

Terminal-Bench 2.0

Our first published evaluation runs Xal against all 89 tasks in Terminal-Bench 2.0 through Harbor.

Setting	Value
Agent	Xal
Dataset	terminal-bench/terminal-bench-2
Dataset digest	sha256:c6fc2e2382c1dbae99b2d5ecd2f4f4a60c3c01e0d84642d69b4afd92e99d078b
Provider	Xal openai-chatgpt (ChatGPT subscription)
Model	gpt-5.6-luna
Thinking	medium
Attempts	1 per task
Concurrency	4
Harbor	0.16.1
Xal revision	13a1ce2
Date	2026-08-22

Result

45 / 89 tasks passed — 50.56% accuracy.

• 45 rewards of 1.0
• 44 rewards of 0.0
• 4 AgentTimeoutError exceptions, scored as 0.0
• 2h 1m 30s total Harbor runtime

View the native job on Harbor Hub. See RESULTS.md for the per-task breakdown and jobs/xal-gpt-5.6-luna-medium-20260822T112326Z for the Harbor configuration, results, verifier output, and Xal trajectories.

Important

This is a one-attempt preliminary result, not an official Terminal-Bench leaderboard entry. Terminal-Bench 2.0 requires at least five trials per task. The official submission process is currently closed while the maintainers introduce a new integrity process. We intend to submit a qualifying verified run when submissions reopen.

Publication sanitization

The public artifacts preserve rewards, timestamps, configuration, verifier output, and agent trajectories. Before publication, we made only the following safety redactions:

• Local host paths were replaced with generic paths.
• Secret-shaped fixture strings from the sanitize-git-repo benchmark task were replaced with explicit redaction placeholders.

No rewards, exceptions, task outputs, model events, or token counts were changed.

Reproduce the benchmark

This repository includes a standalone Harbor adapter and runner for benchmarking a neighboring Xal checkout using a connected ChatGPT subscription.

Prerequisites

• Linux with Docker running
• Bun and Cargo, matching Xal's development prerequisites
• A neighboring Xal checkout at ../xal, or --xal-root PATH
• A ChatGPT subscription connected in Xal with xal connect chatgpt

The runner installs uv, Harbor 0.16.1, Zig, and cargo-zigbuild under this repository on first use. It does not install them globally. Run setup separately if desired:

./benchmark.sh setup

Run

Start the full benchmark in detached mode:

./benchmark.sh

The initial command stays attached while it installs local tooling, builds a portable Xal binary from the current checkout, and validates the ChatGPT connection. The benchmark and Harbor viewer then run detached.

Open the live UI at:

http://<machine-ip>:4444

Binding to 0.0.0.0 exposes the viewer to every reachable interface. Use firewall rules or --host 127.0.0.1 when it should only be local.

Check or stop both detached processes:

./benchmark.sh status
./benchmark.sh stop

Stopping sends SIGTERM to the benchmark and viewer process groups, waits up to 30 seconds, then force-stops anything still running.

Smoke test

Run one task before committing to the full suite:

./benchmark.sh --limit 1

Run a named task or a filtered subset:

./benchmark.sh --include build-cython-ext
./benchmark.sh --include 'build-*' --exclude build-pov-ray

Model and thinking

Select any model exposed by the connected ChatGPT subscription and one of Xal's thinking efforts:

./benchmark.sh --model gpt-5.6-luna --thinking high
./benchmark.sh --model gpt-5.6-sol --thinking low

Supported thinking values are none, low, medium, high, xhigh, and max. Xal falls back to the model's default if the selected model does not expose the requested effort.

If Xal has more than one ChatGPT connection, select one by profile name:

./benchmark.sh --connection personal

Subscription-backed runs can hit account limits. Concurrency defaults to one for that reason. Increase it cautiously:

./benchmark.sh --concurrency 2

Leaderboard-eligible run

Terminal-Bench 2.0 requires at least five attempts for every task. To produce a candidate job:

./benchmark.sh \
  --attempts 5 \
  --model gpt-5.6-luna \
  --thinking medium \
  --concurrency 4 \
  --job-name xal-gpt-5.6-luna-medium-k5

Do not modify timeout or resource settings for a leaderboard candidate. Follow the current instructions linked from the Terminal-Bench 2.0 leaderboard before running or submitting, because its integrity requirements and submission process may change.

Other options

--attempts N
--dataset NAME
--job-name NAME
--xal-root PATH
--credentials PATH
--host HOST
--port PORT
--no-build

Arguments after -- pass directly to harbor run:

./benchmark.sh --limit 5 -- --max-retries 1

Run ./benchmark.sh --help for the complete interface.

Files

• xal_agent.py implements the Harbor installed-agent adapter for Xal.
• benchmark.sh builds Xal and manages the detached benchmark and viewer.
• jobs/ contains published Harbor jobs, trial results, trajectories, and verifier output.
• logs/benchmark.log contains detached Harbor output and is ignored.
• logs/viewer.log contains viewer output and is ignored.
• build/xal is the portable benchmark binary and is ignored.
• .state/, .tools/, and .venv/ are local runtime/tooling directories and are ignored.

Review every new job before publication. Task instructions and model-generated terminal output become public when committed.

Credential handling

The runner reads Xal's existing credentials.json and validates it by refreshing the ChatGPT model catalog before starting Harbor. The custom Harbor agent selects only the requested openai-chatgpt OAuth profile and uploads that profile plus the benchmark binary into each isolated task container. Credentials are not copied into this repository or Harbor's downloaded agent logs.

Each trial receives its own temporary Xal home inside its container. Xal runs headlessly in yolo mode so benchmark tasks can execute without approval prompts. A copied OAuth credential can still be refreshed inside a long-running trial, but that refreshed token remains in the trial container rather than being written back to the host.

License

The runner and adapter are available under the MIT License. Terminal-Bench task content and downloaded benchmark artifacts remain subject to their respective upstream terms.