Skip to content

rules(cli): a listing reads the published catalog, not only what you pulled - #513

Closed
elronbandel wants to merge 1 commit into
mainfrom
elron/rules-list-published
Closed

rules(cli): a listing reads the published catalog, not only what you pulled#513
elronbandel wants to merge 1 commit into
mainfrom
elron/rules-list-published

Conversation

@elronbandel

Copy link
Copy Markdown
Contributor

Closes #512. Rules only — the code that depends on it lands separately
(contributing rule 2).

The rule today

  1. List. eval-containers list benchmarks|agents|models — MUST read Docker image labels. No separate database or index.

That has one workable reading, and it is what the code does: docker images to
find them, docker inspect to read them. So list answers only for images
already pulled to the local daemon — nothing at all on a clean machine, and
never the question a listing is asked: what can I run?

What changed

  • Rule 14 now says where the labels may be read from: the fleet's published
    catalog (evals/catalog:latest, written by the release from those same LABEL
    lines), or the checkout's containers/*/*/Dockerfile under --local. The
    prohibition that matters is kept verbatim in spirit — the CLI keeps no index
    of its own
    .
  • Underlying tools: curl added, for the three HTTP calls that read the
    catalog (anonymous token, manifest, one blob). Adding a tool is itself a rule
    change and is recorded in the changelog, as the table requires.

Why the catalog is not the index the rule forbids

The concern behind "no separate database or index" is a hand-maintained list
that drifts from the components. The catalog is derived — by the publisher, at
publish time, from the components' own labels — so nothing is stored that a
component does not declare, and there is nothing to keep in step by hand. Same
source of truth, published.

Rules checked against

Follow-up

The code PR makes list read the catalog by default and --local read the
checkout, and extends fleet-catalog.sh to carry the per-component labels a
listing shows.

Noticed while writing this, not in scope: cli/src/build.rs shells out to
bash (to run a benchmark's build.sh), which is not in the tools table
either — that is drift, and wants its own issue.

…pulled

Rule 14 said `list` MUST read Docker image labels, with no separate database or
index. The one workable reading is what the code does — `docker images` plus
`docker inspect` — so `list` answers only for images already pulled to the local
daemon: nothing at all on a clean machine, and never the question a listing is
asked, which is what the fleet publishes.

The release now writes that answer down (#497): evals/catalog:latest, derived by
containers/scripts/fleet-catalog.sh from the components' own LABEL lines. The
sentence that blocked reading it guards against a second source of truth that
drifts from the components — which this is not. It is the same source, published
by the thing that publishes the images, so nothing is stored that a component
does not declare.

So rule 14 now says where the labels may be read from — the published catalog,
or the checkout under `--local` — and keeps the prohibition that matters: the
CLI keeps no index of its own. `curl` joins the underlying tools for the three
HTTP calls that read the catalog (anonymous token, manifest, one blob).

Closes #512

Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
@elronbandel
elronbandel force-pushed the elron/rules-list-published branch from e63fba7 to 93f9166 Compare September 9, 2026 10:58
@elronbandel

Copy link
Copy Markdown
Contributor Author

Superseded: the fleet now publishes an index (#519), so the rule and the code both point at that instead of the catalog — #552.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rfc: .agents/src/RULES.md#14 — a listing should read the published catalog, not only images you pulled

1 participant