feat(adapters): add the task-source adapter foundation and clawbench-sources CLI - #345
Open
vaibhavdabas16 wants to merge 3 commits into
Open
feat(adapters): add the task-source adapter foundation and clawbench-sources CLI#345vaibhavdabas16 wants to merge 3 commits into
vaibhavdabas16 wants to merge 3 commits into
Conversation
First step of the incremental plan in TIGER-AI-Lab#72: the shared target type, the registry, and an identity adapter, with no behaviour change for bundled tasks. `ClawBenchTask` is a superset of test-cases/task.schema.json plus the provenance an imported task needs — which source it came from and that source's own id — so a leaderboard can partition rows by origin. Time limits stay in minutes, matching task.json and the container watchdog, because most upstream schemas use seconds and the conversion has to live somewhere explicit. Adapters declare which scoring layers they can honour. A layer a source cannot support is meant to score null rather than 0, so aggregation never confuses "the agent failed" with "never scored on that axis"; to_task_json() therefore refuses to render a task with no interception contract instead of inventing one. Fields with no 1:1 mapping surface as an AdapterWarning naming the field, the fallback, and the pinned upstream revision the mapping was written against — a task is never dropped silently. Validation is hand-rolled rather than Pydantic: the package has no Pydantic dependency and its other registries (harness_registry) validate frozen dataclasses exactly this way.
`clawbench-sources` lists the registered adapters with their upstream, pin, declared scoring layers, and whether a checkout is present; `show` prints one source's status and its module field-mapping table; `cases` lists the tasks a source exposes, with any load warnings. TIGER-AI-Lab#72 sketches this as `clawbench sources`, but `clawbench` is the TUI and every non-interactive command in this repo is its own `clawbench-<verb>` entrypoint, so it follows that convention instead. A source is addressable as `<name>:<path>` to pin it to an explicit clone; otherwise it resolves under CLAWBENCH_SOURCES_DIR, XDG_CACHE_HOME, or ~/.cache. CLAWBENCH_OFFLINE is honoured up front so a later fetching adapter fails loudly instead of cloning.
Adds docs/task-sources.md covering the CLI, the shared task type, the scoring-layer contract, field-mapping warnings, and how to write an adapter, plus the CHANGELOG entry and a docs/cli.md row. Tests cover the registry, source-spec parsing and cache resolution, the schema's validation and task.json rendering, the native loader against both a suite directory and a corpus root, and each CLI subcommand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances #72. This is step 1 of the incremental plan in that issue —
_base.py+schema.py+native.py+__init__.py+ the sources CLI — with no behaviour change for bundled tasks. The benchmark adapters themselves (claw-eval, wild-claw-bench, webarena, mind2web, webvoyager) follow in later PRs, so this does not close the issue.What's here
src/clawbench/adapters/:schema.py—ClawBenchTask, the shared target type. A superset oftest-cases/task.schema.jsonplus provenance (source,source_id), withScoringLayer,AdapterWarning, andExtraInfo._base.py—AdapterBase, the@registerregistry, source-spec parsing (nameorname:/path), cache-dir resolution, and theCLAWBENCH_OFFLINEgate.native.py— identity loader for the bundled corpora; the reference implementation for the field-mapping docstring convention.cli.py—clawbench-sources.Decisions worth reviewing
Two deliberate deviations from the issue text:
CLI shape. The issue sketches
clawbench sources, butclawbenchis the TUI and every non-interactive command here is its ownclawbench-<verb>entrypoint. This addsclawbench-sourceswithlist/show/casesrather than growing a subcommand tree on the TUI.sources refreshis deliberately absent — it needs the_pins.yamlfetch machinery, which is step 5.Dataclasses, not Pydantic. The issue specifies a Pydantic
ClawBenchTask. ClawBench has no Pydantic dependency, and its other registry (run_support/harness_registry.py) validates frozen dataclasses with hand-rolled checks and explicit error messages. This follows that rather than adding a runtime dependency for one module. Happy to switch if you'd rather take the dependency.The scoring-layer contract, from §4 of the issue, is the part most worth a look. An adapter declares which layers its tasks can be judged by; a layer a source cannot support is meant to score
nullrather than0, so leaderboard aggregation never conflates "the agent failed" with "this task was never scored on that axis".ClawBenchTask.to_task_json()enforces the corresponding invariant on the write side: it refuses to render a nativetask.jsonfor a task with no interception contract instead of inventing aneval_schema. Wiringnullthrough the recording itself belongs with the first adapter that actually needs it.Time limits.
ClawBenchTask.time_limitis in minutes, matchingtask.jsonand the container watchdog. Most upstream schemas use seconds, so each adapter converts — the docstring says so explicitly to keep that from being silently wrong.Native
default_path()is the bundledtest-cases/root, andload()accepts either a suite directory (*/task.json) or a corpus root (*/*/task.json), soclawbench-sources cases clawbench-native --path test-cases/v2and the bare form both work.Testing
tests/test_task_source_adapters.py— 26 cases covering the registry and its error message, source-spec parsing (including that a bare Windows path isn't mistaken forname:path), cache-dir precedence, the offline flag, schema validation andtask.jsonrendering checked against the committedtask.schema.json, warning formatting, the native loader on a suite dir, a corpus root, and the real bundled V2 corpus, plus each CLI subcommand.clawbench.adapters.cliis added to thetest_cli_entrypointshelp-module list.Full suite passes locally (
301 passed, 10 skipped);ruff check,ruff format --check, andpyrightare clean on the new files.