Wells Bench — a deduction benchmark that verifies itself

Every instance is generated, not curated. Every instance is proved to have exactly one solution. Every instance is proved solvable by deduction alone, with no guessing. Every answer is graded by a program, with no human judgement and no partial credit.

No results have been collected. This repository is a generator, a verifier, a grader and a results format. What any model scores on it is an open question, and the answer is not in here because nobody has run it. If you run it, publish what you got in the format below.

---

The genre, and why it suits text

Every cell holds one digit: the number of orthogonal steps from that cell to the nearest cell holding 0. That is the whole rule.

It rests on one equivalence, which is what makes the whole thing work:

A grid of non-negative integers is the step-distance field of its own zeros exactly when (a) orthogonal neighbours differ by at most 1, and (b) every cell above 0 touches a cell one smaller.

Four properties follow, and each is why this genre and not another:

• It is a rectangle of digits in and a rectangle of digits out. The question and the answer have the same shape, so nothing has to be translated into coordinates, described in prose, or drawn. The benchmark measures deduction rather than the ability to describe a grid. A loop puzzle, a region puzzle or anything with edges between cells would have spent most of its difficulty on notation.
• Both conditions are local, so the puzzle can be finished by looking at one cell and its four neighbours at a time. That is what makes "solvable without guessing" a checkable property rather than a hope.
• In an open rectangle, step distance is Manhattan distance, which gives a second, completely different way to reason about a grid — used by the verifier to disagree with the solver.
• There is no corpus. As far as a search of published pencil-puzzle genres goes, distance-to-nearest has not been used as a puzzle rule. There are no worked examples to have memorised, and the difficulty is not in knowing a technique.

The genre is shared with Daily/, a one-a-day puzzle for humans; the generator here is a generalisation of that one to arbitrary size, with the difficulty controls a benchmark needs rather than the ones a daily needs.

Contamination

The argument is not that the answers are secret. Secrets leak, and a benchmark whose value depends on one dies the first time somebody pastes it into a forum. The argument is:

• a set is a pure function of (name, seed phrase, tier layout);
• a fresh seed gives a fresh set with the same proved properties in well under a second;
• so a solver that has memorised the published set gains nothing on the set you make in the next thirty seconds — and you can prove your set has the same properties before you use it, with bin/verify.mjs.

node bin/build.mjs --name my-set --seed "any phrase at all" --layout t1=10,t3=10,t5=10 --out bench
node bin/verify.mjs bench/my-set.json

The published set carries a SHA-256 digest of its instances, and a results file that does not name the right digest is refused rather than scored.

What is in here

src/            pure logic. No I/O, no DOM, no network, no clock, no Math.random.
  rng.js        seeded PRNG
  grid.js       geometry, BFS distance field, legality, the value domain
  solver.js     three human techniques, difficulty profile, solution COUNTING
  generate.js   build a board, carve clues while it still solves by logic; tiers
  manifest.js   named sets, canonical JSON, digests, key binding
  render.js     instance -> the exact text a solver is shown
  parse.js      whatever the solver said -> a grid, or an honest refusal
  grade.js      instance + reply -> one bit, computed two ways
  results.js    the results file format, its validation, and the report
  api.js        request building for two endpoint shapes. No network, no keys.
bin/
  build.mjs     write a set, its key, its prompts and a blank results file
  run.mjs       score a run; optionally collect one if YOU have a key
  verify.mjs    INDEPENDENT re-proof of a built set
tools/
  independent.mjs      the second derivation of every property. Imports no src/.
  prove-tests-fail.mjs sabotage harness: 19 deliberate breakages
  build-site.mjs       refresh site/js and site/bench from the tested sources
  serve.mjs            local static server
  e2e.mjs              drive the real page in headless Chromium
site/           the deployable artefact: explanation, worked example, grader
bench/          the published set: public-001
results/        published runs
test/           node --test, no browser, no network

Commands

npm test          # 71 tests, ~8s
npm run build     # rebuild bench/public-001 (deterministic)
npm run verify    # independently re-prove every instance in it
npm run sabotage  # break 19 load-bearing things, confirm each is caught
npm run site      # refresh site/js and site/bench from src/ and bench/
npm run serve     # http://localhost:8788
npm run e2e       # drive the real page in headless Edge/Chrome (needs serve)

Node 24 (node --test, global WebSocket for the e2e driver).

Running the benchmark

None of these assume you have an API key. Nothing in this repository assumes a key exists, and no request is made unless you ask for one.

By hand. bench/public-001.prompts.txt holds every prompt. Paste them into whatever you already pay for, collect the replies into bench/public-001.answers-template.json, and score it:

node bin/run.mjs --set bench/public-001.json --answers my-answers.json --out my-report.json

or drop the file onto the web page, which grades in the browser with no upload.

With your own harness. bench/public-001.prompts.jsonl holds one {id, prompt} per line.

With your own key.

node bin/run.mjs --set bench/public-001.json --call \
  --shape openai-chat --endpoint https://…/v1/chat/completions \
  --model some-model --key-env YOUR_ENV_VAR --out-answers my-answers.json

--shape is openai-chat or anthropic-messages, which between them cover almost every hosted or local endpoint including Ollama and llama.cpp. If --key-env names an empty variable the runner stops and says so rather than sending a request with no key.

The results format

{
  "wellsbench": 1,
  "set":    { "name": "public-001", "setDigest": "a5544f3e…" },
  "solver": { "name": "…", "vendor": "…", "version": "…", "access": "api | chat-ui | human" },
  "conditions": {
    "date": "2026-09-08",
    "attemptsPerInstance": 1,
    "toolsAllowed": false,
    "temperature": null,
    "maxOutputTokens": null,
    "systemPrompt": null,
    "notes": "anything a reader needs in order to compare this with another run"
  },
  "answers": { "public-001-t1-000": "the model's reply, verbatim", "…": "…" }
}

date, attemptsPerInstance and toolsAllowed are required; a file that omits them is refused rather than scored, because a number nobody can interpret is worse than no number. Instances with no answer are scored as wrong, never skipped — a harness that drops the instances it failed to run inflates its own score. Model output is kept verbatim so a third party can re-derive your number.

What is verified, and how

Everything below is a test, not a claim in a readme.

• Exactly one solution, proved three ways. The generator counts solutions by backtracking search, stopping at two. tools/independent.mjs proves uniqueness by a set-cover argument over well positions with no search in it at all. test/crosscheck.test.mjs compares both against exhaustive enumeration of every possible zero-set on grids small enough to enumerate, over 1,500 random clue-sets — and asserts that impossible, unique and ambiguous cases all occurred, so a checker hardwired to "unique" fails rather than passes.
• No guessing. Every instance is completed by three named techniques alone, by two independently written propagators, both reasoning over the digits 0–9 and nothing more — the same information the prompt gives.
• No decorative clues. Carving runs to exhaustion, and the verifier re-tests minimality by removing each clue in turn.
• The grader cannot pass a wrong answer. Every single-cell mutation of every solution in every digit is submitted and must be rejected — thousands of near-misses. Separately, thousands of grids that are legal Wells boards but not this board are submitted and must be rejected too.
• The grader refuses a broken instance. The bit is computed twice — key equality and re-checking the rule — and because each instance has exactly one solution the two must agree. If they disagree the grader emits an integrity failure instead of a score, because a wrong key that silently marks correct solvers wrong is worse than no benchmark.
• The prompt never leaks the solution, checked on every instance in flat form, in row form, and row by row.
• The page a visitor receives, not the files on this machine. tools/e2e.mjs drives real headless Chromium at a 390×844 phone viewport, reads three prompts out of the live DOM, and grades a correct answer, a one-digit error and a refusal through the page's own button.
• The worked example on the front page is parsed back out of the HTML and re-proved unique and deducible, including the three specific cells its narrative names.

Proving the tests can fail

npm run sabotage breaks 19 load-bearing behaviours one at a time and requires the relevant check to fail. All 19 are caught. Among them: the grader marking everything correct; the grader trusting the key without re-checking the rule; hasUniqueSolution hardwired to true; the solution counter never finding a second solution; the set-cover argument hardwired to "unique"; the legality test forgetting that every value needs a neighbour one smaller; carving removing clues whether or not the puzzle survives; the prompt leaking the answer; the parser taking the first answer block instead of the last, or inventing an answer by trimming an over-long block; results validation accepting anything; the set digest ceasing to depend on the puzzles; a shipped puzzle altered after its digest was computed; and site/js drifting from src/.

Results

Not collected. Nothing here claims that any model finds this hard or easy.

results/2026-09-08-claude-opus-5-illustrative.json is one illustrative data point and is not a finding: the model that wrote this generator solved 4 of the 30 instances in a single session — the first instance of tiers t1, t1, t2 and t3 — and got all 4 right. It had just written the code, it knew the genre intimately, it had unlimited deliberation and no time limit, and it used a global counting argument rather than the step-by-step deduction the puzzle is built around. One model, one session, four instances, no controls. It shows the harness works end to end. It shows nothing about any model.

Known limits

• Tiers are defined by size and deduction depth, not by measured difficulty. Whether t5 is harder than t1 for any particular solver is exactly the question this repository refuses to answer without data.
• Answer extraction is a judgement call, however carefully specified. The grader reports which strategy it needed for each reply, and reports "no answer found" separately from "wrong", so a run whose answers all needed the loosest strategy is visible as such. A solver could still be penalised for formatting.
• One genre. A score here is a score at one kind of reasoning.
• The public set is a sample, not the benchmark. The benchmark is the generator. Publishing a score against public-001 is only meaningful alongside a score against a set you made yourself.

Deploying (not done — Rod's call)

site/ is static: no build step, no server, no database, no accounts, no analytics, no third-party request, and no way for anyone to contact anybody.

npx wrangler pages deploy site --project-name=wells-bench

Run npm run site first, so what deploys is what the tests exercised.

Licence

MIT.