feat(release): publish an index of every artifact the fleet has published - #521
Merged
Conversation
…shed evals/catalog:latest answers "what can I launch" as a model — a family list times an agent lineup — and a model of the fleet cannot be trusted. The lineup is copied verbatim from the workflow and never swept, and each family is proved by probing a single agent, on the measured assumption that gaps are whole-family rather than per-agent. #516 ended that: openhands joined the lineup with one published pair (handbook--openhands) against the other agents' 759 each, so the catalog now offers it against 99 benchmarks, 98 of them an ImagePullBackOff waiting for whoever picks them. So publish the enumeration beside it. containers/scripts/fleet-index.py names every artifact this tree can publish — every component, every benchmark x agent combo, every baked task, standalone bundles included — and proves each one against the registry with its own tags/list. What comes out is a list a consumer can search and filter: 6805 published of 8797 candidates, 130 labelled components, in 89s against the live registry. The eval.* labels come from each component's Dockerfile, which at the commit a release publishes from is the source of truth the rest of the repo already treats it as (benchmarks/RULES.md 24f makes the LABEL line canonical for per-task-ness, and the chart's per-task.json is derived from it). Reading them back out of each published image instead cost a manifest-index-config walk per component for a copy that can only lag: measured at 95s against 89s, and the same line count. Named-then-proved rather than listed, because the registry cannot be enumerated from outside: /v2/_catalog returns all of GHCR with no way to seek to one org, and the packages REST listing is capped at 10000 results — this org is at 9900 already, 1876 of them buildcache. A cap that truncates silently is worse than a sweep that costs 89 seconds. Python rather than bash like its neighbours: a few thousand parallel HTTP calls whose failures must be told apart from empty answers is what a shell script does badly. A repo the registry never answered for aborts the run; only a clean "never heard of it" reads as unpublished, and an untagged package reads as absent, because listing one as available is how a launch becomes an ImagePullBackOff. The catalog is untouched. Both artifacts are published by the same job, and the index is read back after the push (delivery/RULES.md 17). Once the dashboard and the CLI read the index, the two become one artifact under one name. Closes #519 Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
elronbandel
force-pushed
the
elron/fleet-index
branch
from
September 9, 2026 12:04
f2f3683 to
eba9f26
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #519. The catalog is untouched — this publishes a second artifact beside
it,
ghcr.io/exgentic/evals/index:latest.Why
evals/catalog:latestanswers "what can I launch" as a model: a family listtimes an agent lineup. The lineup is copied verbatim from the workflow and never
swept, and each family is proved by probing a single agent — on the measured
assumption that gaps are whole-family, never per-agent.
#516 ended that
assumption. Measured against the live registry:
handbook--openhands)So the catalog offers
openhandsagainst 99 benchmarks, 98 of which are anImagePullBackOff waiting for whoever picks them — the failure #81 fixed on the
family axis, reopened on the agent axis. A model of the fleet cannot be trusted;
only an enumeration can.
What it publishes
{"source": {...}, "generated": "2026-09-09T...Z", "registry": "ghcr.io/exgentic", "images": {"evals/handbook--openhands": ["latest"], "benchmarks/aime": ["latest"], ...}, "labels": {"benchmarks/aime": {"eval.benchmark.description": "American Invitational…", "eval.benchmark.env": "shared-env", ...}}}Every artifact, with its tags, plus the
eval.*labels each component declares.Search and filter it; don't re-derive it.
The labels come from each component's Dockerfile: at the commit a release
publishes from, that is the source of truth the rest of the repo already treats
it as (
benchmarks/RULES.md24f makes theLABELline canonical forper-task-ness, and the chart's
per-task.jsonis derived from it). Reading themback out of each published image instead costs a manifest→index→config walk per
component for a copy that can only lag — measured at 95s against 89s, for the
same line count.
Named, then proved — because the registry can't be listed
/v2/_catalogreturns all of GHCR with no way to seek to one org (last=isrejected), and the packages REST listing is hard-capped at 10000 results — this
org is at 9900 already (1876 of them
buildcache/*), so that listing istruncating today. So names come from the tree — every component, every
benchmark × agent combo, every baked task, standalone bundles included — and each
is proved by its own
tags/list. An untagged package reads as absent, sincelisting one as available is how a launch becomes an ImagePullBackOff.
Measured
89s against the live registry: 8797 candidates, 6805 published (evals
5986, benchmarks 759, agents 25, core 17, models 9, gateways 3), 130 labelled
components, ~770 KB. One scoped token per 100 repositories, 32-way parallel.
Verification
time; the numbers above are from the last run.
image configs 95s, identical output for every component — the tree version is
the one here.
pre-commit run— ruff, ruff-format, actionlint, yaml, codespell all pass.tests/staticis Rust +shell sweeps, and its rules require offline tests), and every branch here is a
registry call. Say the word if you want a pytest target added and I'll cover
candidates()and the failure taxonomy offline.Rules checked against
.agents/delivery/RULES.md14 (fail dirty — a registry that never answeredaborts the sweep instead of shrinking the index), 16 (it describes the
latestchannel), 17 (verified publish — the artifact is pulled back andcmpd, not trusted from a zero exit)..agents/RULES.md8 (minimal — one new script and one step; the catalogpath is unchanged. 277 lines, of which 187 are code: the bulk is GHCR's
per-repository auth, the batching and parallelism that make 8797 refs take 89
seconds, and retries that tell "never heard of it" apart from a blip).
.agents/contributing/RULES.md1 (resolves feat(release): publish an index of every artifact the fleet has published #519), 2 (code only), 3(this section), 4 (no tool attribution).
Follow-ups
the lineup — what actually closes the openhands case.
eval-containers listfilters the index (feat(cli):listshould answer for the published fleet, and--localfor the checkout #514/feat(cli): list what the fleet publishes, and --local what the checkout has #515).name.