You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The repository includes an experimental reproducibility card for the founding Inspect Evals case, but card structure is currently documented only by example. Contributors need machine-readable validation before the registry can accept multiple frameworks and review statuses consistently.
Proposed outcome
Define a small versioned card schema and add a CLI command such as:
Problem
The repository includes an experimental reproducibility card for the founding Inspect Evals case, but card structure is currently documented only by example. Contributors need machine-readable validation before the registry can accept multiple frameworks and review statuses consistently.
Proposed outcome
Define a small versioned card schema and add a CLI command such as:
The first implementation should validate structure and conservative evidence claims. It does not need to fetch external URLs or verify remote content.
Acceptance criteria
fork-validated,upstream-reviewed, orupstream-merged.upstream-reviewed/upstream-mergedcard without an upstream evidence URL.evalrepro card validatewith clear exit codes and error messages.registry/**/*.jsoncard in CI.Design boundary
Do not build a generic remote attestation system in this issue. Signature and artifact verification belong to a later design proposal.
Likely files
src/evalrepro/cards.py(new)src/evalrepro/cli.pytests/test_cards.py(new).github/workflows/ci.ymlregistry/README.mdDifficulty
Intermediate. A contributor can begin by proposing the schema/error model in a comment before writing code.