Skip to content

feat(adapters): add the task-source adapter foundation and clawbench-sources CLI - #345

Open
vaibhavdabas16 wants to merge 3 commits into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:feat/task-source-adapters
Open

feat(adapters): add the task-source adapter foundation and clawbench-sources CLI#345
vaibhavdabas16 wants to merge 3 commits into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:feat/task-source-adapters

Conversation

@vaibhavdabas16

Copy link
Copy Markdown
Contributor

Advances #72. This is step 1 of the incremental plan in that issue — _base.py + schema.py + native.py + __init__.py + the sources CLI — with no behaviour change for bundled tasks. The benchmark adapters themselves (claw-eval, wild-claw-bench, webarena, mind2web, webvoyager) follow in later PRs, so this does not close the issue.

What's here

src/clawbench/adapters/:

  • schema.pyClawBenchTask, the shared target type. A superset of test-cases/task.schema.json plus provenance (source, source_id), with ScoringLayer, AdapterWarning, and ExtraInfo.
  • _base.pyAdapterBase, the @register registry, source-spec parsing (name or name:/path), cache-dir resolution, and the CLAWBENCH_OFFLINE gate.
  • native.py — identity loader for the bundled corpora; the reference implementation for the field-mapping docstring convention.
  • cli.pyclawbench-sources.

Decisions worth reviewing

Two deliberate deviations from the issue text:

  1. CLI shape. The issue sketches clawbench sources, but clawbench is the TUI and every non-interactive command here is its own clawbench-<verb> entrypoint. This adds clawbench-sources with list / show / cases rather than growing a subcommand tree on the TUI. sources refresh is deliberately absent — it needs the _pins.yaml fetch machinery, which is step 5.

  2. Dataclasses, not Pydantic. The issue specifies a Pydantic ClawBenchTask. ClawBench has no Pydantic dependency, and its other registry (run_support/harness_registry.py) validates frozen dataclasses with hand-rolled checks and explicit error messages. This follows that rather than adding a runtime dependency for one module. Happy to switch if you'd rather take the dependency.

The scoring-layer contract, from §4 of the issue, is the part most worth a look. An adapter declares which layers its tasks can be judged by; a layer a source cannot support is meant to score null rather than 0, so leaderboard aggregation never conflates "the agent failed" with "this task was never scored on that axis". ClawBenchTask.to_task_json() enforces the corresponding invariant on the write side: it refuses to render a native task.json for a task with no interception contract instead of inventing an eval_schema. Wiring null through the recording itself belongs with the first adapter that actually needs it.

Time limits. ClawBenchTask.time_limit is in minutes, matching task.json and the container watchdog. Most upstream schemas use seconds, so each adapter converts — the docstring says so explicitly to keep that from being silently wrong.

Native default_path() is the bundled test-cases/ root, and load() accepts either a suite directory (*/task.json) or a corpus root (*/*/task.json), so clawbench-sources cases clawbench-native --path test-cases/v2 and the bare form both work.

Testing

tests/test_task_source_adapters.py — 26 cases covering the registry and its error message, source-spec parsing (including that a bare Windows path isn't mistaken for name:path), cache-dir precedence, the offline flag, schema validation and task.json rendering checked against the committed task.schema.json, warning formatting, the native loader on a suite dir, a corpus root, and the real bundled V2 corpus, plus each CLI subcommand. clawbench.adapters.cli is added to the test_cli_entrypoints help-module list.

Full suite passes locally (301 passed, 10 skipped); ruff check, ruff format --check, and pyright are clean on the new files.

First step of the incremental plan in TIGER-AI-Lab#72: the shared target type, the
registry, and an identity adapter, with no behaviour change for bundled
tasks.

`ClawBenchTask` is a superset of test-cases/task.schema.json plus the
provenance an imported task needs — which source it came from and that
source's own id — so a leaderboard can partition rows by origin. Time
limits stay in minutes, matching task.json and the container watchdog,
because most upstream schemas use seconds and the conversion has to live
somewhere explicit.

Adapters declare which scoring layers they can honour. A layer a source
cannot support is meant to score null rather than 0, so aggregation never
confuses "the agent failed" with "never scored on that axis";
to_task_json() therefore refuses to render a task with no interception
contract instead of inventing one.

Fields with no 1:1 mapping surface as an AdapterWarning naming the field,
the fallback, and the pinned upstream revision the mapping was written
against — a task is never dropped silently.

Validation is hand-rolled rather than Pydantic: the package has no
Pydantic dependency and its other registries (harness_registry) validate
frozen dataclasses exactly this way.
`clawbench-sources` lists the registered adapters with their upstream,
pin, declared scoring layers, and whether a checkout is present; `show`
prints one source's status and its module field-mapping table; `cases`
lists the tasks a source exposes, with any load warnings.

TIGER-AI-Lab#72 sketches this as `clawbench sources`, but `clawbench` is the TUI and
every non-interactive command in this repo is its own `clawbench-<verb>`
entrypoint, so it follows that convention instead.

A source is addressable as `<name>:<path>` to pin it to an explicit
clone; otherwise it resolves under CLAWBENCH_SOURCES_DIR, XDG_CACHE_HOME,
or ~/.cache. CLAWBENCH_OFFLINE is honoured up front so a later fetching
adapter fails loudly instead of cloning.
Adds docs/task-sources.md covering the CLI, the shared task type, the
scoring-layer contract, field-mapping warnings, and how to write an
adapter, plus the CHANGELOG entry and a docs/cli.md row.

Tests cover the registry, source-spec parsing and cache resolution, the
schema's validation and task.json rendering, the native loader against
both a suite directory and a corpus root, and each CLI subcommand.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant