loop: the differential tool counted unmeasured files as agreement

docs/NOW.md

@@ -1,3 +1,16 @@
+# NOW -- the differential tool counted 91 of 150 unmeasured files as agreement (2026-08-15)
+
+Last updated: 2026-08-15
+
+## loop: six categories, reason codes, and printed coverage (Closes #2166)
+
+- **Same binaries, same 150 files, two versions of the same tool.** Five categories printed `150 unchanged, 0 field-loss, 0 unknown` -- total agreement. Six categories print `59 unchanged, 91 not-evaluated (both-error)`, coverage **39.3 %**. Nothing about the compiler changed between those two lines, and the first is the line that had been quoted in pull requests
+- **`unchanged` was carrying two statements**: "we compared and found no difference" and "neither binary parsed the file". Both incremented the same counter, so no reviewer and no threshold could separate them. On the 634-spec library the split was 330 measured against 286 unmeasured -- a 52 % base reported as 100 %
+- Six mutually exclusive categories now, with the partition **asserted at runtime**; every `not-evaluated` row carries a reason code (`both-error`, `base-timeout`, `candidate-timeout`, `environment-failure`, `excluded-source-loss`); no `PASS` while any `unknown` remains; coverage printed on every run, in the same sentence as any "no regressions" claim, because a caveat in a neighbouring paragraph does not travel with the number
+- **Full corpus, single completed run, uniform 12 s threshold, 1089 files** `[measured]`: 524 unchanged/ok, 343 unchanged/fail, **0 regressions**, 1 strict-improvement, 221 not-evaluated (195 both-timeout, 26 candidate-timeout). **Coverage 868/1089 = 79.7 %**
+- **The 26 `ok -> timeout` files are the boundary, not a slowdown.** Timed directly, 3 runs each way: median candidate/base ratio **1.010** (min 0.985, max 1.026), on files taking 10.8-11.7 s against a 12 s wall. A 1-3 % jitter is enough to move them across it, so the count difference measures the threshold and not the compiler
+- Rules R15 and R16 added to `docs/loop/LOOP-RULES.md` and resealed. `tri corpus-parse`, `corpus-status`, `diffmodes`, `loop-rules` registered in `scripts/ci/loop-tools-tracked.sh`, which fails on an untracked tool -- the state that already destroyed two of these scripts along with every number they produced
+
 # NOW -- BNF: the control that measures what ternary is worth (2026-08-09)
 
 Last updated: 2026-08-09

docs/loop/LOOP-RULES.sha256

@@ -0,0 +1 @@
+fc74dccf1f53c220fd2ffcd279b41255df538db02817a26ad58be8865eb4ee6b  docs/loop/LOOP-RULES.md

scripts/ci/loop-tools-tracked.sh

@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+# Loss of a loop tool must be impossible, not merely regrettable.
+#
+# Two scripts -- scripts/tri_loop/cost.py and scripts/tri_loop/diffbin.py -- were
+# written, used to produce numbers that were then quoted in a pull request, and
+# lost. They were never committed to any branch, and when the working copy was
+# re-cloned they went with it. Six recovery routes were checked and all came back
+# empty: dangling git objects, the reflog, shell history, CI artifacts, PR and
+# issue comments, and the session snapshot. Every number they had produced became
+# unreproducible in one stroke.
+#
+# The failure was not carelessness at the keyboard. It was that nothing in the
+# repository could tell the difference between a tool that exists and a tool that
+# exists only in one untracked working directory. This script is that difference.
+#
+# It fails when:
+#   1. a required loop tool is missing from the working tree
+#   2. a required loop tool exists but git does not track it -- the exact state
+#      that preceded the loss
+#   3. any file under scripts/tri_loop/ is untracked, so a NEW tool cannot be
+#      quietly added and then lost the same way
+#   4. a required tool is not executable via the dispatcher, i.e. `tri <cmd>`
+#      does not resolve
+#
+# Not claimed: that this makes the tools correct, or that a tracked file cannot
+# be deleted in a later commit. It closes the specific hole that swallowed these
+# two files -- work that exists only outside version control -- and nothing wider.
+set -uo pipefail
+
+cd "$(dirname "$0")/../.." || exit 2
+
+REQUIRED_TOOLS=(
+  "scripts/tri_loop/triage.py"
+  "scripts/tri_loop/cost.py"
+  "scripts/tri_loop/diffbin.py"
+  "scripts/tri_loop/damage.py"
+  "scripts/tri_loop/damage_freeze.py"
+  "scripts/tri_loop/damage_repair.py"
+  "scripts/tri_loop/corpus_parse.py"
+  "scripts/tri_loop/corpus_status.py"
+  "scripts/tri_loop/diffmodes.py"
+  "scripts/tri_loop/loop_rules.py"
+)
+REQUIRED_SUBCOMMANDS=(triage cost diffbin damage damage-freeze damage-repair \
+                      corpus-parse corpus-status diffmodes loop-rules)
+
+fail=0
+note() { printf '  %s\n' "$1"; }
+
+echo "loop-tools-tracked: the tools the measurement loop depends on"
+echo
+
+echo "1. present in the working tree"
+for f in "${REQUIRED_TOOLS[@]}"; do
+  if [[ -f "$f" ]]; then
+    note "ok       $f"
+  else
+    note "MISSING  $f"
+    fail=1
+  fi
+done
+
+echo
+echo "2. tracked by git (this is the check that would have caught the loss)"
+for f in "${REQUIRED_TOOLS[@]}"; do
+  [[ -f "$f" ]] || continue
+  if git ls-files --error-unmatch "$f" >/dev/null 2>&1; then
+    note "tracked  $f"
+  else
+    note "UNTRACKED $f -- exists here and nowhere else; one re-clone loses it"
+    fail=1
+  fi
+done
+
+echo
+echo "3. nothing under scripts/tri_loop/ is untracked"
+untracked=$(git ls-files --others --exclude-standard -- scripts/tri_loop/ 2>/dev/null)
+if [[ -z "$untracked" ]]; then
+  note "ok       no untracked files under scripts/tri_loop/"
+else
+  while IFS= read -r u; do
+    [[ -n "$u" ]] && note "UNTRACKED $u"
+  done <<<"$untracked"
+  fail=1
+fi
+
+echo
+echo "4. reachable through the dispatcher"
+# The dispatcher resolves `tri <cmd>` generically, to tri_loop/<cmd with - as _>.py.
+# So the honest check is that the generic exec line is still there AND that the
+# file each subcommand resolves to exists. An earlier draft of this check grepped
+# scripts/tri for the literal command name, which passed for `tri triage` only
+# because the word appears in a comment -- a check that reports the presence of
+# its own documentation is worth nothing.
+if grep -q 'exec python3 "\$LOOP_DIR/\${cmd//-/_}.py"' scripts/tri; then
+  note "ok       generic loop dispatch line present"
+else
+  note "BROKEN   generic loop dispatch line is gone from scripts/tri"
+  fail=1
+fi
+for c in "${REQUIRED_SUBCOMMANDS[@]}"; do
+  target="scripts/tri_loop/${c//-/_}.py"
+  if [[ -f "$target" ]]; then
+    note "ok       tri $c -> $target"
+  else
+    note "UNROUTED tri $c -> $target does not exist"
+    fail=1
+  fi
+done
+
+echo
+if [[ $fail -eq 0 ]]; then
+  echo "PASS: every loop tool is present, tracked, and routed."
+  echo "This says nothing about whether any of them is correct."
+else
+  echo "FAIL: a loop tool is missing, untracked, or unrouted."
+  echo "An untracked tool is the state that already destroyed two of these"
+  echo "scripts and every number they produced. Commit it before relying on"
+  echo "its output for any claim."
+fi
+exit $fail

scripts/tri

@@ -2,6 +2,33 @@
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+# Loop helpers live in scripts/tri_loop/ as Python, and are dispatched BEFORE the
+# compiler binary is even looked for. They measure and report on the repository,
+# they do not compile it, and none of them mutates anything. Requiring a built
+# t27c first was a defect: `tri triage` and `tri damage` read the tracker and the
+# spec text and have no use for the compiler, yet a machine with no build could
+# not run them. Keeping the helpers ahead of the lookup also means a missing
+# compiler cannot take the reporting tools down with it.
+cmd="${1:-}"
+LOOP_DIR="$SCRIPT_DIR/tri_loop"
+if [[ "$cmd" == "loop-help" ]]; then
+  echo "tri loop helpers (report only, never mutate):"
+  for f in "$LOOP_DIR"/*.py; do
+    [[ -e "$f" ]] || continue
+    name="$(basename "$f" .py)"
+    printf '  %-12s %s\n' "$name" "$(sed -n '2s/^"""tri [a-z-]* -- //p' "$f")"
+  done
+  echo
+  echo "Every helper prints what it does NOT establish alongside what it does."
+  echo "Read that part: an aggregate that omits the class of loss it failed to"
+  echo "check is how a differential once reported zero regressions over a corpus"
+  echo "in which thirteen files lost declared fields."
+  exit 0
+fi
+if [[ -n "$cmd" && -f "$LOOP_DIR/${cmd//-/_}.py" ]]; then
+  exec python3 "$LOOP_DIR/${cmd//-/_}.py" "${@:2}"
+fi
+
 T27C="${TRI_T27C:-}"
 if [[ -z "$T27C" ]]; then
   for p in "$REPO_ROOT/target/release/t27c" "$REPO_ROOT/target/debug/t27c" "$REPO_ROOT/bootstrap/target/release/t27c" "$REPO_ROOT/bootstrap/target/debug/t27c"; do
@@ -14,7 +41,6 @@ if [[ ! -x "$T27C" ]]; then
 fi
 # `t27c suite` is the only subcommand that accepts --repo-root.
 # Map `tri test` to the suite subcommand; pass other commands through unchanged.
-cmd="${1:-}"
 if [[ "$cmd" == "test" || "$cmd" == "suite" ]]; then
     # Forward everything after the subcommand to t27c suite, including --json.
     exec "$T27C" suite --repo-root "$REPO_ROOT" "${@:2}"

scripts/tri_loop/corpus_parse.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""Parse every spec in a tree with a given t27c binary and record a per-file verdict.
+
+Why this exists as a separate tool: a whole-corpus parse run is the only way to
+tell a parser change apart from a corpus change, and the naive shell loop that
+preceded it silently hung. The hang was not mysterious: within the
+`specs/scratch/*_bench_NNd_aos_call_dedup.t27` family, parse time grows
+multiplicatively per added dimension and NOT with file size (11d is 317 KB and
+4.5 s while 10d is 2.2 MB and 2.0 s). A per-file timeout is therefore mandatory,
+and files that hit it are reported as `timeout`, never silently dropped: an
+excluded file is a fact about the measurement, not an absence of one.
+
+Verdicts, one per file, exhaustive and mutually exclusive:
+  ok       - exit 0
+  fail     - non-zero exit within the time budget
+  timeout  - exceeded the per-file budget, verdict unknown
+
+Usage:
+  corpus_parse.py <t27c-binary> <spec-dir> --out results.json [--timeout SEC]
+                  [--jobs N] [--exclude GLOB]...
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import fnmatch
+import json
+import os
+import subprocess
+import sys
+import time
+
+
+def find_specs(root: str, excludes: list[str]) -> list[str]:
+    out: list[str] = []
+    for dirpath, _dirnames, filenames in os.walk(root):
+        for name in sorted(filenames):
+            if not name.endswith(".t27"):
+                continue
+            path = os.path.join(dirpath, name)
+            if any(fnmatch.fnmatch(path, pat) for pat in excludes):
+                continue
+            out.append(path)
+    return sorted(out)
+
+
+def parse_one(binary: str, path: str, budget: float) -> dict:
+    started = time.monotonic()
+    try:
+        proc = subprocess.run(
+            [binary, "parse", path],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.PIPE,
+            timeout=budget,
+        )
+    except subprocess.TimeoutExpired:
+        return {
+            "path": path,
+            "verdict": "timeout",
+            "exit_code": None,
+            "ms": round((time.monotonic() - started) * 1000, 1),
+            "first_error": None,
+        }
+    elapsed_ms = round((time.monotonic() - started) * 1000, 1)
+    if proc.returncode == 0:
+        return {
+            "path": path,
+            "verdict": "ok",
+            "exit_code": 0,
+            "ms": elapsed_ms,
+            "first_error": None,
+        }
+    text = proc.stderr.decode("utf-8", "replace")
+    first = None
+    for line in text.splitlines():
+        if "rror" in line:
+            first = line.strip()[:400]
+            break
+    return {
+        "path": path,
+        "verdict": "fail",
+        "exit_code": proc.returncode,
+        "ms": elapsed_ms,
+        "first_error": first,
+    }
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("binary")
+    ap.add_argument("spec_dir")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--timeout", type=float, default=20.0)
+    ap.add_argument("--jobs", type=int, default=max(1, (os.cpu_count() or 2)))
+    ap.add_argument("--exclude", action="append", default=[])
+    args = ap.parse_args()
+
+    if not os.path.isfile(args.binary):
+        print(f"binary not found: {args.binary}", file=sys.stderr)
+        return 2
+
+    specs = find_specs(args.spec_dir, args.exclude)
+    results: list[dict] = []
+    with concurrent.futures.ThreadPoolExecutor(max_workers=args.jobs) as pool:
+        futures = [
+            pool.submit(parse_one, args.binary, p, args.timeout) for p in specs
+        ]
+        for fut in concurrent.futures.as_completed(futures):
+            results.append(fut.result())
+
+    results.sort(key=lambda r: r["path"])
+    counts = {"ok": 0, "fail": 0, "timeout": 0}
+    for r in results:
+        counts[r["verdict"]] += 1
+
+    payload = {
+        "binary": args.binary,
+        "spec_dir": args.spec_dir,
+        "timeout_sec": args.timeout,
+        "excluded_globs": args.exclude,
+        "total": len(results),
+        "counts": counts,
+        "results": results,
+    }
+    with open(args.out, "w", encoding="utf-8") as fh:
+        json.dump(payload, fh, indent=2, sort_keys=True)
+
+    print(f"total {len(results)}: " + " ".join(f"{k}={v}" for k, v in counts.items()))
+    print(f"written {args.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())