Skip to content

Registry: add a versioned reproducibility-card validator #7

Description

@seva9523

Problem

The repository includes an experimental reproducibility card for the founding Inspect Evals case, but card structure is currently documented only by example. Contributors need machine-readable validation before the registry can accept multiple frameworks and review statuses consistently.

Proposed outcome

Define a small versioned card schema and add a CLI command such as:

evalrepro card validate registry/inspect/datasets-4.8.5-to-5.0.1.json

The first implementation should validate structure and conservative evidence claims. It does not need to fetch external URLs or verify remote content.

Acceptance criteria

  • Document card schema v1 and its compatibility policy.
  • Validate required fields: ID, date, framework, baseline, candidate, verdict, review status, evidence, and limitations.
  • Restrict review status to fork-validated, upstream-reviewed, or upstream-merged.
  • Reject an upstream-reviewed/upstream-merged card without an upstream evidence URL.
  • Reject invalid record counts, malformed URLs, unsupported verdicts, duplicate evaluation names, and missing limitations.
  • Add evalrepro card validate with clear exit codes and error messages.
  • Validate every registry/**/*.json card in CI.
  • Add positive and mutation-based negative tests.
  • Keep validation offline and dependency-light.

Design boundary

Do not build a generic remote attestation system in this issue. Signature and artifact verification belong to a later design proposal.

Likely files

  • src/evalrepro/cards.py (new)
  • src/evalrepro/cli.py
  • tests/test_cards.py (new)
  • .github/workflows/ci.yml
  • registry/README.md
  • the founding card, if clarification is required

Difficulty

Intermediate. A contributor can begin by proposing the schema/error model in a comment before writing code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions