An opt-in pattern proposal, with a runnable example rather than a spec dump.
The gap it covers: an autoevals score answers "how did the model do?".
It cannot answer the adversarial follow-up: "was the pass bar fixed before
the results were seen, or adjusted after?" For teams whose evals feed
procurement, audits, or public claims, that second question is starting to
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
The pattern: lock the success criteria (metric, comparator, threshold,
dataset SHA-256, seed) to a canonical hash before the run; compute the
verdict against the locked bar; refuse to score if the criteria changed
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
fields, four MIT reference implementations, byte-identical canonicalization
across Python/JS/Go/Rust.
Runnable example (offline, no keys):
https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
FAIL / PASS paths all exercised.
Precedent: the same locking pattern was merged onto the UK AI Safety
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
Analysis category, alongside the WandB and MLflow integrations), and an
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
What opt-in could look like here: nothing in autoevals core needs to
change. A cookbook entry or an examples/ link would already serve users who
need pre-registered runs; a metadata convention for attaching the locked
hash to logged experiments would be the deeper version. Happy to write
either, adjust the shape, or take this to the cookbook repo if that is the
better home.
Spec: https://spec.falsify.dev/v0.1
An opt-in pattern proposal, with a runnable example rather than a spec dump.
The gap it covers: an autoevals score answers "how did the model do?".
It cannot answer the adversarial follow-up: "was the pass bar fixed before
the results were seen, or adjusted after?" For teams whose evals feed
procurement, audits, or public claims, that second question is starting to
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
The pattern: lock the success criteria (metric, comparator, threshold,
dataset SHA-256, seed) to a canonical hash before the run; compute the
verdict against the locked bar; refuse to score if the criteria changed
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
fields, four MIT reference implementations, byte-identical canonicalization
across Python/JS/Go/Rust.
Runnable example (offline, no keys):
https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
FAIL / PASS paths all exercised.
Precedent: the same locking pattern was merged onto the UK AI Safety
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
Analysis category, alongside the WandB and MLflow integrations), and an
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
What opt-in could look like here: nothing in autoevals core needs to
change. A cookbook entry or an examples/ link would already serve users who
need pre-registered runs; a
metadataconvention for attaching the lockedhash to logged experiments would be the deeper version. Happy to write
either, adjust the shape, or take this to the cookbook repo if that is the
better home.
Spec: https://spec.falsify.dev/v0.1