feat: expand benchmark matrix and external evidence model

.github/workflows/validate.yml

@@ -22,7 +22,7 @@ jobs:
         run: skills-ref validate ./practical-coding
       - name: Run benchmark harness tests
         working-directory: practical-coding
-        run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability
+        run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog
       - name: Check Codex default_prompt references the skill as $skill-name
         run: grep -qF '$practical-coding' practical-coding/agents/openai.yaml
       - name: Ensure legacy local graph runtime is not reintroduced

benchmarks/README.md

@@ -2,7 +2,7 @@
 
 This chain runs isolated Codex sessions directly against `gpt-5.6-luna`, preserves every prompt/transcript/workspace, applies mechanical graders, and writes JSON plus Markdown summaries. It follows the mature evaluation shape used by Agent Skills and Ponytail: realistic cases, fixed sources, clean sessions, repeated paired arms, deterministic assertions where possible, tokens/time, and raw evidence.
 
-For prerequisites, pinned revisions, exact reproduction commands, evidence boundaries, and the published v1.11 calibration results, see [`REPRODUCING.md`](REPRODUCING.md).
+For prerequisites, pinned revisions, exact reproduction commands, evidence boundaries, and the published v1.11 calibration results, see [`REPRODUCING.md`](REPRODUCING.md). For the external benchmark landscape and the public-regression/external/held-out evidence model, see [`../docs/evaluations/2026-08-24-benchmark-landscape.md`](../docs/evaluations/2026-08-24-benchmark-landscape.md).
 
 ## Run
 
@@ -15,6 +15,8 @@ pwsh -File benchmarks/run.ps1 -Profile smoke -Suite router -Case direct-artifact
 pwsh -File benchmarks/run.ps1 -Rescore D:\path\to\benchmark-results\20260824-203839
 ```
 
+`run.ps1` is the canonical entrypoint. It loads the core runner through `run_catalog.py`, which installs the extended public case catalog before execution. This keeps benchmark mechanics separate from the evolving task corpus. `manifest.json` fingerprints the complete benchmark runtime bundle (core runner + case catalog + canonical wrapper), so task/scorer changes cannot masquerade as the same benchmark revision.
+
 For a result that will be presented as a stable ranking, opt into the evidence gate:
 
 ```powershell
@@ -30,11 +32,15 @@ python benchmarks/check_stability.py benchmark-results\v111-delivery-n1-core-rev
 
 That command intentionally reports the published v1.11 Delivery `n=1` artifact as `PROVISIONAL`; it must not be used for a stable ranking until the same cells are rerun with at least three distinct repetitions.
 
-Profiles:
+## Profiles
+
+| Profile | Delivery | Router | Decision | Debug | Default runs | Cells without previous/no-Skill arm |
+|---|---:|---:|---:|---:|---:|---:|
+| `smoke` | 3 | 4 | 1 | 1 | 1 | 14 |
+| `standard` | 9 | 28 | 6 | 8 | 3 | 222 |
+| `full` | 18 | 28 | 10 | 12 | 3 | 324 |
 
-- `smoke`: one run by default; quick harness and model sanity check.
-- `standard`: three runs by default; balanced release comparison.
-- `full`: three runs by default; expanded Ponytail delivery, router, multi-round Decision, and debugging matrix.
+`standard` is the normal release gate. `full` carries the complete public regression matrix. The extra Router cases span all six routes; the expanded Debug set covers twelve cases across parsing, normalization, tenant isolation, pagination, units, row handling, state invariants, TTL semantics, URL handling, and the upstream transfer/amount tasks. Decision grows from four to ten two-turn decisions in `full`.
 
 Useful options:
 
@@ -44,38 +50,38 @@ Useful options:
 - `-SourcesRoot <directory>` reuses pinned competitor checkouts. Without it, sources are cached under the user-local application data directory and cloned as needed.
 - `-IncludeBaseline` adds a no-skill delivery arm.
 - `-NoBuilds` skips runner-owned frontend production builds. It is rejected for a stable Delivery ranking.
-- `-SelfTest` runs the local harness regression tests and validates fixtures, upstream scorers, source pins, and reporting without model calls.
+- `-SelfTest` runs the local harness regression tests and validates fixtures, upstream scorers, expanded custom scorers, source pins, and reporting without model calls.
 - `-FailOnCellFailure` makes any behavioral cell failure return exit code 2. By default only harness/infrastructure failures are non-zero, because a valid comparison may intentionally expose competitor or candidate failures.
 - `-RequireStableRanking` requires at least three distinct repetitions per selected suite/case/arm and rejects incomplete or infrastructure-failed runs before they are called stable.
 - `-Rescore <run-directory>` reapplies the current mechanical graders to saved workspaces/transcripts without another model call; the manifest records the new runner hash and rescore time.
 
 By default, run artifacts are written under `benchmark-results/` and ignored by Git, so transcripts and generated workspaces remain inspectable across commands without entering commits. Use `-Output` for an explicit location.
 
-The fast harness regression suite is also runnable without sources or model access:
+The fast harness regression suite is also runnable without model calls:
 
 ```powershell
-python -m unittest benchmarks.test_benchmarks benchmarks.test_stability
+python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog
 ```
 
 The output directory contains:
 
 ```text
-manifest.json          fixed model, commits, profile, cases, and skill hashes
-results.json           one record per cell
-summary.json           grouped rates, medians, means, and standard deviations
-comparisons.json       Practical-minus-comparator behavioral and efficiency deltas
-rollups.json           suite/arm totals across cases
+manifest.json           fixed model, commits, profile, cases, and skill hashes
+results.json            one record per cell
+summary.json            grouped rates, medians, means, and standard deviations
+comparisons.json        Practical-minus-comparator behavioral and efficiency deltas
+rollups.json            suite/arm totals across cases
 rollup-comparisons.json suite-level Practical-minus-comparator deltas
-report.md              human-readable comparison and Practical deltas
-cells/                 prompt, raw JSONL, stderr, answer, and code workspace per cell
+report.md               human-readable comparison and Practical deltas
+cells/                  prompt, raw JSONL, stderr, answer, and code workspace per cell
 ```
 
 ## Suites and scoring
 
 - `delivery`: Ponytail's published agentic tasks and deterministic scorer. Reports correctness, safety, production LOC, test LOC, files, tokens, time, tool calls, and optional frontend build result.
 - `router`: exact classification across Direct, Decision, Debugging, Implementation, Exploration, and Verification, including overlap and negative-boundary cases.
 - `decision`: Practical versus Matt Pocock `grilling`. Uses a real resumed second turn and gates on frontier questions, one recommendation per question, no premature implementation, and convergence after scripted user decisions. Trade-off language is reported diagnostically but is not a declared grilling contract gate.
-- `debug`: shared-root-cause tasks scored on the repaired invariant and sibling callers. Tests/TDD process receives no bonus.
+- `debug`: shared-root-cause tasks scored on the repaired invariant and sibling callers. Tests/TDD process receives no bonus. Each Practical-owned Debug seed is required to fail its deterministic scorer, and a separate oracle fixture must pass it before the case is accepted into the catalog.
 
 `total_tokens` includes cached input because that is how Codex reports turn input. The report therefore also separates cached input, uncached input, output, and reasoning tokens. `duration_seconds` is per-cell process duration; suite elapsed time is recorded separately and is not obtained by summing concurrent cell durations.
 
@@ -84,3 +90,5 @@ cells/                 prompt, raw JSONL, stderr, answer, and code workspace per
 Use repeated paired results. A candidate is not accepted merely because its prose matches a Skill contract. Require no correctness/build regression, then compare delivered code and behavior. Treat LOC, tokens, and time as secondary within equally correct artifacts. `n=1` is a smoke result, not a stable ranking.
 
 A published stable ranking must pass `benchmarks/check_stability.py` with the default minimum `n=3`. The gate checks distinct repetition IDs, complete-run metadata, and infrastructure errors. Behavioral or build failures remain valid benchmark observations and therefore do not invalidate the sample by themselves.
+
+The public catalog is a **regression suite**, not a hidden generalization test. Once a case has influenced Skill wording, its future 100% score should be treated as a ceiling check. External benchmarks and a private held-out set are required for stronger claims.

benchmarks/REPRODUCING.md

@@ -12,10 +12,12 @@ The chain deliberately separates three evidence types:
 | Router | Practical-owned exact classification corpus covering Direct, Decision, Debugging, Implementation, Exploration, and Verification | Current and previous Practical | Project regression benchmark |
 | Decision | Practical-owned two-turn scenarios and mechanical contract grader | Practical and Matt Pocock `grilling`, plus optional previous Practical | Comparative benchmark; `grilling` has no declared upstream behavior benchmark |
 | Debug | Ponytail `trace-transfer`/`trace-amount` plus Practical-owned shared-boundary cases; deterministic invariant and sibling-caller grading | Practical and Superpowers, plus optional previous Practical | Mixed upstream/custom comparative benchmark |
-| Harness tests | Six Python tests for question/recommendation counting, delta direction, fixed Luna configuration, route coverage, and Decision-module injection | Runner only | Local benchmark-infrastructure regression tests |
+| Harness tests | Python unit tests for runner mechanics, stability gating, catalog breadth, duplicate detection, scorer seed rejection, and oracle acceptance | Runner only | Local benchmark-infrastructure regression tests |
 
 The Decision and Debug comparisons must not be described as official Matt Pocock or Superpowers benchmark results. They are controlled Codex/Luna comparisons against those Skills' relevant behavior. Tests, TDD phases, planning prose, and workflow completeness receive no quality points; only the delivered behavior, safety invariant, build, and artifact metrics are scored.
 
+The public Router/Decision/Debug catalog is a regression corpus. Cases that have influenced Skill wording remain useful for preventing regressions but are no longer independent evidence of generalization. See [`../docs/evaluations/2026-08-24-benchmark-landscape.md`](../docs/evaluations/2026-08-24-benchmark-landscape.md) for the external and held-out evidence plan.
+
 ## Pinned upstream sources
 
 The runner verifies these exact commits before spending model calls:
@@ -55,7 +57,7 @@ Set-Location practical-coding
 pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest
 ```
 
-`-SelfTest` makes no model calls. It runs the six harness regression tests, verifies every profile case ID, then proves that the Ponytail and custom scorers accept their good fixtures and reject their bad fixtures.
+`-SelfTest` makes no model calls. It runs the harness/stability/catalog unit tests, verifies every profile case ID, proves the pinned Ponytail scorers reject their bad references, proves every Practical-owned Debug seed fails, and proves each expanded Debug oracle passes its deterministic scorer.
 
 ## 2. Run the standard comparison
 
@@ -66,19 +68,28 @@ pwsh -NoProfile -File benchmarks/run.ps1 `
   -Workers 3
 ```
 
-This runs 150 isolated cells when no previous-version or no-Skill arm is requested:
+The current public `standard` profile runs 222 isolated cells when no previous-version or no-Skill arm is requested:
 
-- 9 Delivery cases × 2 arms × 3 runs;
-- 16 Router cases × 1 arm × 3 runs;
-- 4 Decision cases × 2 arms × 3 runs;
-- 4 Debug cases × 2 arms × 3 runs.
+- 9 Delivery cases × 2 arms × 3 runs = 54;
+- 28 Router cases × 1 arm × 3 runs = 84;
+- 6 Decision cases × 2 arms × 3 runs = 36;
+- 8 Debug cases × 2 arms × 3 runs = 48.
 
-For the expanded 18-case Ponytail delivery matrix:
+For the complete public matrix:
 
 ```powershell
 pwsh -NoProfile -File benchmarks/run.ps1 -Profile full -Runs 3 -Workers 3
 ```
 
+`full` runs 324 cells without a previous-version or no-Skill arm:
+
+- 18 Delivery cases × 2 arms × 3 runs = 108;
+- 28 Router cases × 1 arm × 3 runs = 84;
+- 10 Decision cases × 2 arms × 3 runs = 60;
+- 12 Debug cases × 2 arms × 3 runs = 72.
+
+The `smoke` profile intentionally remains small and defaults to one repetition. It is for harness/model sanity only.
+
 ## 3. Run a before/after candidate gate
 
 While editing a dirty candidate, compare it with the checked-in version:
@@ -102,6 +113,8 @@ pwsh -NoProfile -File benchmarks/run.ps1 `
 
 The materialized baseline Skill is copied into the run directory. Both entrypoint and complete Skill-bundle hashes are recorded in `manifest.json`, preventing an ambiguous "previous version" comparison.
 
+For a comparison that will be published as a stable ranking, add `-RequireStableRanking`. It rejects effective `n<3`, incomplete runs, infrastructure failures, and Delivery rankings without production-build evidence.
+
 ## 4. Run a focused regression
 
 Selectors are repeatable:
@@ -114,9 +127,9 @@ pwsh -NoProfile -File benchmarks/run.ps1 `
   -BaselineRef HEAD
 
 pwsh -NoProfile -File benchmarks/run.ps1 `
-  -Profile smoke `
-  -Suite decision `
-  -Case service-boundary `
+  -Profile full `
+  -Suite debug `
+  -Case trace-cache-tenant `
   -Runs 3
 ```
 
@@ -148,18 +161,20 @@ The rescore timestamp and current runner hash are written to the manifest. Raw r
 
 ## Published v1.11 calibration
 
-The following results were produced on Windows 11 with Python 3.13.14, Codex CLI 0.145.0, `gpt-5.6-luna`, reasoning `medium`, and three parallel workers. They are calibration results, not claims about every repository or model.
+The following table is a **historical calibration produced before the public catalog expansion in this document**. It remains the evidence for the v1.11 iteration and must not be relabeled as results on the new 28/10/12 matrix.
+
+The run used Windows 11 with Python 3.13.14, Codex CLI 0.145.0, `gpt-5.6-luna`, reasoning `medium`, and three parallel workers.
 
-| Matrix | Practical v1.11 | Frozen v1.10 | Comparator | Main difference |
+| Historical matrix | Practical v1.11 | Frozen v1.10 | Comparator | Main difference |
 |---|---:|---:|---:|---|
-| Router, 16 cases × n=3 | 48/48 | 48/48 | — | Both arms reached the current harness ceiling; the new negative rules caused no regression but did not prove a gain |
+| Router, 16 cases × n=3 | 48/48 | 48/48 | — | Both arms reached that harness ceiling; the new negative rules caused no regression but did not prove a gain |
 | Debug, 4 cases × n=3 | 12/12 | 12/12 | Superpowers 10/12 | Superpowers twice repaired only the named caller and missed the sibling/shared invariant |
 | Decision, 4 cases × n=3 | 12/12 | 12/12 | grilling 10/12 | Both Practical versions converged after the scripted reply; grilling reopened `api-migration` twice |
 | Delivery, 6 differentiating cases × n=1 | 5/6 | 5/6 | Ponytail 5/6 | All arms scored 6/6 correct/safe; production builds separated the pass rate and remain unstable at `n=1` |
 
-For Debug, suite median time was 39.1 seconds for v1.11, 44.5 seconds for v1.10, and 78.8 seconds for Superpowers; median total tokens were 80,940, 88,053, and 245,966 respectively. These secondary efficiency metrics matter only after correctness and safety. Delivery total LOC at `n=1` was 376 for v1.11, 363 for v1.10, and 343 for Ponytail, so the current data does **not** support a claim that v1.11 matches Ponytail's compactness.
+For historical Debug, suite median time was 39.1 seconds for v1.11, 44.5 seconds for v1.10, and 78.8 seconds for Superpowers; median total tokens were 80,940, 88,053, and 245,966 respectively. These secondary efficiency metrics matter only after correctness and safety. Historical Delivery total LOC at `n=1` was 376 for v1.11, 363 for v1.10, and 343 for Ponytail, so those data do **not** support a claim that v1.11 matches Ponytail's compactness.
 
-The published comparison used `-BaselineSkill docs/evaluations/snapshots/practical-v1.10`, not commit `75d5013`. See [`../docs/evaluations/2026-08-24-practical-v111-iteration.md`](../docs/evaluations/2026-08-24-practical-v111-iteration.md) for the complete per-case tables and acceptance decisions.
+The published comparison used `-BaselineSkill docs/evaluations/snapshots/practical-v1.10`, not commit `75d5013`. See [`../docs/evaluations/2026-08-24-practical-v111-iteration.md`](../docs/evaluations/2026-08-24-practical-v111-iteration.md) for the complete historical per-case tables and acceptance decisions.
 
 ## Reproducibility limits
 
@@ -168,4 +183,5 @@ The published comparison used `-BaselineSkill docs/evaluations/snapshots/practic
 - `input_tokens` already includes cached input. Compare `uncached_input_tokens`, `output_tokens`, and reasoning tokens separately when discussing cost.
 - A successful Ponytail-derived Delivery score proves the reused deterministic contract, not equivalence with Ponytail's original Claude runtime.
 - The runner disables normal user Skills, plugins, apps, memories, and multi-agent behavior, then embeds exactly one selected Skill arm. This isolates the comparison but differs from a user's fully configured interactive session.
-- Do not generalize the focused v1.11 calibration rows into a whole-Skill ranking. Run the standard or full matrix after each material prompt change.
+- The public Practical-owned cases are visible regression tests. Do not use their saturation as proof of unseen-task generalization.
+- Do not generalize the focused v1.11 historical calibration rows into a whole-Skill ranking. Run the current standard/full matrix and an independent external or held-out suite after material prompt changes.