The gap
The eval harness's process rule 2 (crates/b2-embed/evals/README.md) is binding on every corpus edit and ends: "The audit is a ten-line script; run it, don't eyeball it." No such script is committed anywhere — scripts/ holds only doctor.sh and compare-embed-device.sh, and nothing under crates/b2-embed/evals/ runs a token audit.
The audits clearly were run (the queries.json description records their verdicts: teaching/training rejected for the parrot negative because teach/train land in campus-life.md and running-and-aquarium.md; castle, square, boat, routine, human all rejected; the date-shaped block's numerics verified at zero occurrences corpus-wide, word-boundary and stem-prefix both). But each editor currently has to re-invent the tool a binding rule requires, which is exactly the failure mode the rule exists to prevent — an eyeballed audit.
Ask
Commit the audit as scripts/token-audit (or a small model-free example beside the harness), checking both directions of rule 2, word-boundary and stem-prefix both:
- for an edited/added note: no existing query's content tokens newly land in it;
- for a new query: its content tokens do not split toward a rival note (report per-note occurrence counts so the "split evenly" judgement has numbers).
A useful shape: token-audit note <path> and token-audit query "<text>" against evals/corpus/ (and --corpus corpus-dense/ for the within-corpus form the dense labels use). Then cite it from the guide so a corpus-edit commit can name the audit run it passed.
The gap
The eval harness's process rule 2 (
crates/b2-embed/evals/README.md) is binding on every corpus edit and ends: "The audit is a ten-line script; run it, don't eyeball it." No such script is committed anywhere —scripts/holds onlydoctor.shandcompare-embed-device.sh, and nothing undercrates/b2-embed/evals/runs a token audit.The audits clearly were run (the
queries.jsondescription records their verdicts:teaching/trainingrejected for the parrot negative becauseteach/trainland incampus-life.mdandrunning-and-aquarium.md;castle,square,boat,routine,humanall rejected; the date-shaped block's numerics verified at zero occurrences corpus-wide, word-boundary and stem-prefix both). But each editor currently has to re-invent the tool a binding rule requires, which is exactly the failure mode the rule exists to prevent — an eyeballed audit.Ask
Commit the audit as
scripts/token-audit(or a small model-free example beside the harness), checking both directions of rule 2, word-boundary and stem-prefix both:A useful shape:
token-audit note <path>andtoken-audit query "<text>"againstevals/corpus/(and--corpus corpus-dense/for the within-corpus form the dense labels use). Then cite it from the guide so a corpus-edit commit can name the audit run it passed.