Skip to content

feat(cli): list what the fleet publishes, and --local what the checkout has - #515

Closed
elronbandel wants to merge 1 commit into
mainfrom
elron/cli-list-published
Closed

feat(cli): list what the fleet publishes, and --local what the checkout has#515
elronbandel wants to merge 1 commit into
mainfrom
elron/cli-list-published

Conversation

@elronbandel

Copy link
Copy Markdown
Contributor

Closes #514. Depends on #513 (rules), which permits reading the published
catalog — rule 14 today forbids it.

Problem

list ran docker images + docker inspect, so it answered only for images
already pulled to the local daemon: nothing at all on a clean machine, and never
the question a listing is asked — what can I run?

The release already publishes that answer (#497).

What it does now

$ eval-containers list benchmarks
$ ghcr.io/exgentic/evals/catalog:latest
BENCHMARK                    DESCRIPTION                                       TASKS TYPE       INTERNET
acpbench                     ACPBench - Action, Change, and Planning reasoni…   1040 shared-env false
swe-bench                    SWE-bench Verified                                 500 per-task   false
  • Default: the published catalog — an anonymous token, a manifest, one blob.
    No docker, no credential, no checkout.
  • --local: this checkout's containers/*/*/Dockerfile, so a benchmark that
    has never been published still lists where it exists.
  • Both fill the same Catalog and feed one renderer: the two views differ in
    source, never in shape.
  • list evals is one row per family × agent — swe-bench is one benchmark
    with 500 tasks behind it; 500 × 7 rows is not a listing.

The catalog carries the labels the columns show

fleet-catalog.sh now also writes, under a new meta key, the labels the
components already declare: description/tasks/env/internet for a
benchmark, description/runtime for an agent, provider for a model — read
from the tree at publish time, where they already are, at no extra registry cost.
Models are probed directly (models/<name>), since they publish on their own
rather than through a combo.

agents and families are untouched — they are the launch contract the
dashboard picks pairs from, so meta is purely additive and no consumer moves.

Verification

  • cargo test -p eval-containers — 10 tests in list.rs, all offline: a
    per-task family is one row sized by its tasks; a shared-env one by its dataset
    label; a catalog with no meta still lists (one column short, never a parse
    failure); the evals filters narrow to pairs; labels come off LABEL lines
    only (a RUN echo cannot invent one); tasks.txt headings are skipped;
    --local outside a checkout says so.
  • cargo clippy --all-targets -- -D warnings and cargo fmt --check clean;
    shellcheck clean on fleet-catalog.sh.
  • Ran both modes against the live registry and a checkout: the published mode
    lists all 99 published families (columns blank until the catalog is
    republished with meta); --local lists 107, with descriptions, task counts
    and types.

Rules checked against

After merge

The catalog is republished by the next fleet run — until then the published mode
lists names and task counts with the label columns blank, which is exactly what
the "catalog without meta" test pins.

…ut has

`list` ran `docker images` + `docker inspect`, so it answered only for images
already pulled to the local daemon — nothing at all on a clean machine, and
never the question a listing is asked: what can I run?

The release already publishes that answer (#497). So `list` now reads
evals/catalog:latest — an anonymous token, a manifest, one blob — and needs no
docker, no credential and no checkout. `--local` answers the other question,
what this checkout declares, straight from containers/*/*/Dockerfile, so a
benchmark that has never been published still lists where it exists. Both fill
the same Catalog and feed one renderer: the two views differ in source, never in
shape.

`list evals` is one row per family x agent. swe-bench is one benchmark with 500
tasks behind it; 500 x 7 rows is not a listing.

fleet-catalog.sh carries the labels those columns show — description, tasks, env
and internet for a benchmark, description and runtime for an agent, provider for
a model — read from the tree at publish time, where they already are, under a
new `meta` key. `agents` and `families` are untouched: they are the launch
contract a consumer picks a pair from. Models are probed directly
(`models/<name>`), since they publish on their own rather than through a combo.
Every field of the parsed catalog defaults, so a catalog published before `meta`
existed lists one column short instead of failing to parse.

Closes #514

Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
@elronbandel

Copy link
Copy Markdown
Contributor Author

Superseded: the fleet now publishes an index (#519), so the rule and the code both point at that instead of the catalog — #553.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): list should answer for the published fleet, and --local for the checkout

1 participant