Skip to content

feat(release): publish an index of every artifact the fleet has published - #521

Merged
elronbandel merged 1 commit into
mainfrom
elron/fleet-index
Sep 9, 2026
Merged

feat(release): publish an index of every artifact the fleet has published#521
elronbandel merged 1 commit into
mainfrom
elron/fleet-index

Conversation

@elronbandel

@elronbandel elronbandel commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Closes #519. The catalog is untouched — this publishes a second artifact beside
it, ghcr.io/exgentic/evals/index:latest.

Why

evals/catalog:latest answers "what can I launch" as a model: a family list
times an agent lineup. The lineup is copied verbatim from the workflow and never
swept, and each family is proved by probing a single agent — on the measured
assumption that gaps are whole-family, never per-agent.

#516 ended that
assumption. Measured against the live registry:

agent published combos
claude-code, codex, gemini-cli, openclaw, opencode, pi, zerostack 759 each
openhands 1 (handbook--openhands)

So the catalog offers openhands against 99 benchmarks, 98 of which are an
ImagePullBackOff waiting for whoever picks them — the failure #81 fixed on the
family axis, reopened on the agent axis. A model of the fleet cannot be trusted;
only an enumeration can.

What it publishes

{"source": {...}, "generated": "2026-09-09T...Z", "registry": "ghcr.io/exgentic",
 "images": {"evals/handbook--openhands": ["latest"], "benchmarks/aime": ["latest"], ...},
 "labels": {"benchmarks/aime": {"eval.benchmark.description": "American Invitational…",
                                "eval.benchmark.env": "shared-env", ...}}}

Every artifact, with its tags, plus the eval.* labels each component declares.
Search and filter it; don't re-derive it.

The labels come from each component's Dockerfile: at the commit a release
publishes from, that is the source of truth the rest of the repo already treats
it as (benchmarks/RULES.md 24f makes the LABEL line canonical for
per-task-ness, and the chart's per-task.json is derived from it). Reading them
back out of each published image instead costs a manifest→index→config walk per
component for a copy that can only lag — measured at 95s against 89s, for the
same line count.

Named, then proved — because the registry can't be listed

/v2/_catalog returns all of GHCR with no way to seek to one org (last= is
rejected), and the packages REST listing is hard-capped at 10000 results — this
org is at 9900 already
(1876 of them buildcache/*), so that listing is
truncating today. So names come from the tree — every component, every
benchmark × agent combo, every baked task, standalone bundles included — and each
is proved by its own tags/list. An untagged package reads as absent, since
listing one as available is how a launch becomes an ImagePullBackOff.

Measured

89s against the live registry: 8797 candidates, 6805 published (evals
5986, benchmarks 759, agents 25, core 17, models 9, gateways 3), 130 labelled
components, ~770 KB. One scoped token per 100 repositories, 32-way parallel.

Verification

  • Ran end to end against the live registry four times (89–95s), valid JSON every
    time; the numbers above are from the last run.
  • Both label sources measured against each other: Dockerfiles 89s, published
    image configs 95s, identical output for every component — the tree version is
    the one here.
  • The openhands case above is a query over the produced index, not a claim.
  • pre-commit run — ruff, ruff-format, actionlint, yaml, codespell all pass.
  • No unit tests: this repo has no Python test target (tests/static is Rust +
    shell sweeps, and its rules require offline tests), and every branch here is a
    registry call. Say the word if you want a pytest target added and I'll cover
    candidates() and the failure taxonomy offline.

Rules checked against

  • .agents/delivery/RULES.md 14 (fail dirty — a registry that never answered
    aborts the sweep instead of shrinking the index), 16 (it describes the
    latest channel), 17 (verified publish — the artifact is pulled back and
    cmpd, not trusted from a zero exit).
  • .agents/RULES.md 8 (minimal — one new script and one step; the catalog
    path is unchanged. 277 lines, of which 187 are code: the bulk is GHCR's
    per-repository auth, the batching and parallelism that make 8797 refs take 89
    seconds, and retries that tell "never heard of it" apart from a blip).
  • .agents/contributing/RULES.md 1 (resolves feat(release): publish an index of every artifact the fleet has published #519), 2 (code only), 3
    (this section), 4 (no tool attribution).

Follow-ups

…shed

evals/catalog:latest answers "what can I launch" as a model — a family list
times an agent lineup — and a model of the fleet cannot be trusted. The lineup
is copied verbatim from the workflow and never swept, and each family is proved
by probing a single agent, on the measured assumption that gaps are whole-family
rather than per-agent. #516 ended that: openhands joined the lineup with one
published pair (handbook--openhands) against the other agents' 759 each, so the
catalog now offers it against 99 benchmarks, 98 of them an ImagePullBackOff
waiting for whoever picks them.

So publish the enumeration beside it. containers/scripts/fleet-index.py names
every artifact this tree can publish — every component, every benchmark x agent
combo, every baked task, standalone bundles included — and proves each one
against the registry with its own tags/list. What comes out is a list a consumer
can search and filter: 6805 published of 8797 candidates, 130 labelled
components, in 89s against the live registry.

The eval.* labels come from each component's Dockerfile, which at the commit a
release publishes from is the source of truth the rest of the repo already
treats it as (benchmarks/RULES.md 24f makes the LABEL line canonical for
per-task-ness, and the chart's per-task.json is derived from it). Reading them
back out of each published image instead cost a manifest-index-config walk per
component for a copy that can only lag: measured at 95s against 89s, and the
same line count.

Named-then-proved rather than listed, because the registry cannot be enumerated
from outside: /v2/_catalog returns all of GHCR with no way to seek to one org,
and the packages REST listing is capped at 10000 results — this org is at 9900
already, 1876 of them buildcache. A cap that truncates silently is worse than a
sweep that costs 89 seconds.

Python rather than bash like its neighbours: a few thousand parallel HTTP calls
whose failures must be told apart from empty answers is what a shell script does
badly. A repo the registry never answered for aborts the run; only a clean
"never heard of it" reads as unpublished, and an untagged package reads as
absent, because listing one as available is how a launch becomes an
ImagePullBackOff.

The catalog is untouched. Both artifacts are published by the same job, and the
index is read back after the push (delivery/RULES.md 17). Once the dashboard and
the CLI read the index, the two become one artifact under one name.

Closes #519

Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
@elronbandel
elronbandel merged commit a13401b into main Sep 9, 2026
6 checks passed
@elronbandel
elronbandel deleted the elron/fleet-index branch September 9, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(release): publish an index of every artifact the fleet has published

1 participant