feat(runner): record code, corpus, and agent revisions in run-meta.json - #347
feat(runner): record code, corpus, and agent revisions in run-meta.json#347vaibhavdabas16 wants to merge 6 commits into
Conversation
run-meta.json recorded what was run — task, model, harness name, image ids — but not the revisions those names resolved to. Two runs of "openclaw on v2" a month apart are indistinguishable in the artifact even when the agent, the corpus, and ClawBench itself have all moved, which makes a published leaderboard row labelled rather than reproducible. This collects the ClawBench version, commit, branch and dirty state; the corpus suite and the revision of the commit that last touched it; and the agent and plugin versions pinned by the harness Dockerfile — reading the pins the image was built from rather than asking a running container. Every lookup is best-effort and returns None rather than raising: a PyPI install has no git repository and a container host may have no git at all, and a missing provenance field must never fail a run that otherwise succeeded. `dirty: None` deliberately means "the lookup failed", which is not the same claim as False. Lookups are cached because a batch run builds one block per task and the answers cannot change within a process.
Adds `provenance` alongside `runtime`, reusing the harness image id that _runtime_meta already resolves rather than inspecting the image twice.
Adds a Provenance section to the trace cookbook covering the block's shape, what corpus.revision and harness.pinned_versions actually mean, and why every field can be null — with the filter to apply before comparing runs across code versions. Tests cover pin extraction for each bundled harness, that base-image and COPY lines are not mistaken for agent pins, pip-style pins, corpus resolution for bundled and external case dirs, and that a missing git checkout or git binary yields nulls rather than an exception.
| "branch": _git(repo, "rev-parse", "--abbrev-ref", "HEAD"), | ||
| # `git status` succeeding with no output means a clean tree; a failed | ||
| # lookup returns None, which is not the same claim as "clean". | ||
| "dirty": (status != "") if status is not None else None, |
There was a problem hiding this comment.
dirty would never be False.
Even on a clean checkout, the _git() returns None since it converts empty string to None.
|
|
||
|
|
||
| @lru_cache(maxsize=32) | ||
| def harness_pins(harness: str) -> dict[str, str]: |
There was a problem hiding this comment.
This checks the version pinned in the Dockerfile and setup scripts, but they're not necessarily the version being ran due to the existence of --no-build flag.
This should be set to either Unknown of None under such cases.
| # `pkg==1.2.3` for pip. This reads the pins the image was built from rather | ||
| # than asking the agent for its version, which would need a running container. | ||
| _NPM_PIN_RE = re.compile(r"(?<![\w@/])((?:@[\w.-]+/)?[\w.-]+)@(\d[\w.+-]*)") | ||
| _PIP_PIN_RE = re.compile(r"([\w.-]+)==(\d[\w.+-]*)") |
There was a problem hiding this comment.
It is not handling cases with [] such as litellm[proxy].
| # Version pins declared in a harness Dockerfile: `pkg@1.2.3` for npm and | ||
| # `pkg==1.2.3` for pip. This reads the pins the image was built from rather | ||
| # than asking the agent for its version, which would need a running container. | ||
| _NPM_PIN_RE = re.compile(r"(?<![\w@/])((?:@[\w.-]+/)?[\w.-]+)@(\d[\w.+-]*)") |
There was a problem hiding this comment.
This only handles @Version but not @revisions.
Four defects Perry2004 caught, all real: `dirty` could never be False. `_git()` collapsed empty output into None, so a clean `git status --porcelain` — which succeeds with no output — was indistinguishable from a failed lookup. `_git()` now returns None only when the command could not run, and "" when it ran and said nothing; callers that want a non-empty value ask for it explicitly. Dockerfile pins were reported as fact even under --no-build, where the image can be arbitrarily older than the Dockerfile on disk. Pins are now claimed only when this run built the image, with a `pins_source` field of "dockerfile" or "unverified" saying which. The signal is set by docker_build() rather than read off the --no-build flag, because clawbench-batch builds once and then runs every child with --no-build — keying off the flag would have marked the entire batch path unverified. Pip extras dropped the pin entirely: `litellm[proxy]==1.77.3` matched nothing at all, so the pin vanished silently rather than being recorded. Only `name@version` was recognised, so revision pins — `pkg@github:o/r#sha`, `pkg@git+https://…#ref`, and pip's PEP 508 `pkg @ git+…@ref` — were missed. That is exactly how a preview-stage agent like DeepSeek Harness is pinned, which is the case TIGER-AI-Lab#309 needs. A floating dist-tag (`@next`) is still not collected: it names a moving target, so recording it as a pin would be a false claim.
The clean/dirty test monkeypatched `_git` to return "", bypassing the very `or None` conversion that made `dirty` unable to ever be False — the mock asserted the intended behaviour while the real code did the opposite. It now runs against a real temporary repository, and fails against the old implementation. Adds coverage for pip extras, all three revision-pin spellings, dist-tags being excluded, URL userinfo not being mistaken for a pin, and pins being claimed only for a harness whose image this run built.
Records which pin spellings are collected and why a floating dist-tag is not, and explains when pinned_versions describes the image that actually ran — including that a batch run reports "dockerfile" while a bare clawbench-run --no-build reports "unverified".
|
Thanks — all four were real, and the first two were the interesting ones. Fixed in b7bc2e0…78754ab.
So the signal is set by
A batch run reports Extras. Revisions. Now handles One deliberate exclusion: a floating dist-tag ( On "and setup scripts": I checked, and no bundled harness pins versions in its CI is green; the full suite passes locally (300 passed, 10 skipped). |
2 participants
Advances #309 §1. Does not close it — see the scoping note below, which is the part I'd most like a maintainer's read on.
Why this, and not the
dshadapter#309 §1 asks for two things: a
deepseek-harnessadapter, and "Record the dsh package version or Git commit, plugin set, model configuration, corpus revision, and ClawBench commit inrun-meta.json."I did the second and deliberately stopped short of the first. Adding a harness here is mechanical — a directory of scripts plus a
harnesses.yamlentry — but thesetup-*.shandrun-*.shcontents depend entirely ondsh's actual CLI surface, config format, and transcript shape, and DeepSeek Harness is in developer preview. I can't build the image to check, so I'd be writing a plausible-lookingrun-deepseek-harness.shthat would probably be wrong in ways nobody could see from the diff. That seemed worse than not writing it. Happy to do the adapter as a follow-up if someone can share a workingdshinvocation, or point me at the preview docs.The provenance half turned out to be a real gap, and a general one — it applies to every harness, not just
dsh.The gap
run-meta.jsonrecords what ran — task, model, harness name, image ids — but not the revisions those names resolved to. Two runs of "openclaw on v2" a month apart are indistinguishable in the artifact even when the agent, the corpus, and ClawBench itself have all moved. That makes a published leaderboard row labelled rather than reproducible, and it blocks §2's "fixed task IDs, corpus commit, dsh revision" and §4's "document the exact reproduction commands and environment" regardless of which harness is being added.What lands
A
provenanceblock alongsideruntime:corpus.revisionis the last commit that touched that suite — two runs sharing it saw the same task text.harness.pinned_versionsis parsed from the version pins in the harness Dockerfile (opencode-ai@1.4.4,@playwright/mcp@0.0.70,pip install x==1.2). It reads what the image was built from rather than shelling into a running container, so it costs nothing at run time and works for a harness that is already gone. This answers the issue's "package version and plugin set" for any harness. Verified against the bundled Dockerfiles: openclaw, opencode, claude-code, browser-use all resolve;null(no pins) correctly reports nothing.Failure behaviour, which is most of the design
Every lookup is best-effort and returns
Nonerather than raising. A PyPI install has no git checkout; a container host may have nogit; a task from an explicit--cases-dirhas a history that isn't ClawBench's to claim. A missing provenance field must never fail a run that otherwise succeeded.One distinction worth flagging:
dirty: nullmeans the lookup failed, which is not the same claim asfalse. Anyone filtering runs by "clean tree" should checkis False, not truthiness — the cookbook section says so explicitly.Lookups are
lru_cached because a batch builds one block per task and the answers cannot change inside a process.Testing
tests/test_provenance.py— 16 cases: pin extraction for each bundled harness, thatFROM node:24-slimandCOPY --from=...uv:0.11.6are not mistaken for agent pins, pip-style pins, corpus resolution for bundled and external case dirs, and that a missing checkout or missinggitbinary yields nulls rather than an exception.test_results_and_metadata.pyasserts the block reachesrun-meta.jsonand reuses the same image id asruntime.Full suite passes locally (
291 passed, 10 skipped);ruffandpyrightclean.