feat(cli): list what the fleet publishes, and --local what the checkout has - #515
Closed
elronbandel wants to merge 1 commit into
Closed
feat(cli): list what the fleet publishes, and --local what the checkout has#515elronbandel wants to merge 1 commit into
elronbandel wants to merge 1 commit into
Conversation
…ut has `list` ran `docker images` + `docker inspect`, so it answered only for images already pulled to the local daemon — nothing at all on a clean machine, and never the question a listing is asked: what can I run? The release already publishes that answer (#497). So `list` now reads evals/catalog:latest — an anonymous token, a manifest, one blob — and needs no docker, no credential and no checkout. `--local` answers the other question, what this checkout declares, straight from containers/*/*/Dockerfile, so a benchmark that has never been published still lists where it exists. Both fill the same Catalog and feed one renderer: the two views differ in source, never in shape. `list evals` is one row per family x agent. swe-bench is one benchmark with 500 tasks behind it; 500 x 7 rows is not a listing. fleet-catalog.sh carries the labels those columns show — description, tasks, env and internet for a benchmark, description and runtime for an agent, provider for a model — read from the tree at publish time, where they already are, under a new `meta` key. `agents` and `families` are untouched: they are the launch contract a consumer picks a pair from. Models are probed directly (`models/<name>`), since they publish on their own rather than through a combo. Every field of the parsed catalog defaults, so a catalog published before `meta` existed lists one column short instead of failing to parse. Closes #514 Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
elronbandel
force-pushed
the
elron/cli-list-published
branch
from
September 9, 2026 10:58
26db395 to
8fb42a2
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #514. Depends on #513 (rules), which permits reading the published
catalog — rule 14 today forbids it.
Problem
listrandocker images+docker inspect, so it answered only for imagesalready pulled to the local daemon: nothing at all on a clean machine, and never
the question a listing is asked — what can I run?
The release already publishes that answer (#497).
What it does now
No docker, no credential, no checkout.
--local: this checkout'scontainers/*/*/Dockerfile, so a benchmark thathas never been published still lists where it exists.
Catalogand feed one renderer: the two views differ insource, never in shape.
list evalsis one row per family × agent — swe-bench is one benchmarkwith 500 tasks behind it; 500 × 7 rows is not a listing.
The catalog carries the labels the columns show
fleet-catalog.shnow also writes, under a newmetakey, the labels thecomponents already declare:
description/tasks/env/internetfor abenchmark,
description/runtimefor an agent,providerfor a model — readfrom the tree at publish time, where they already are, at no extra registry cost.
Models are probed directly (
models/<name>), since they publish on their ownrather than through a combo.
agentsandfamiliesare untouched — they are the launch contract thedashboard picks pairs from, so
metais purely additive and no consumer moves.Verification
cargo test -p eval-containers— 10 tests inlist.rs, all offline: aper-task family is one row sized by its tasks; a shared-env one by its dataset
label; a catalog with no
metastill lists (one column short, never a parsefailure); the
evalsfilters narrow to pairs; labels come offLABELlinesonly (a
RUN echocannot invent one);tasks.txtheadings are skipped;--localoutside a checkout says so.cargo clippy --all-targets -- -D warningsandcargo fmt --checkclean;shellcheckclean onfleet-catalog.sh.lists all 99 published families (columns blank until the catalog is
republished with
meta);--locallists 107, with descriptions, task countsand types.
Rules checked against
.agents/src/RULES.md14 as amended by rules(cli): a listing reads the published catalog, not only what you pulled #513 (labels, from the publishedcatalog or the checkout; no index of its own) and the underlying tools
table, which rules(cli): a listing reads the published catalog, not only what you pulled #513 extends with
curl..agents/RULES.md8 (minimal — one renderer for both sources, no newcrate dependency: the fetch is a
curllike the rest of the CLI shells out)..agents/delivery/RULES.md14 (the catalog sweep still fails dirty: anunreadable registry aborts rather than publishing a shrunken catalog).
.agents/contributing/RULES.md1 (resolves feat(cli):listshould answer for the published fleet, and--localfor the checkout #514), 2 (code only), 3(this section), 4 (no tool attribution).
After merge
The catalog is republished by the next fleet run — until then the published mode
lists names and task counts with the label columns blank, which is exactly what
the "catalog without
meta" test pins.