diff --git a/docs/NOW.md b/docs/NOW.md index 1d3be3f13..35b5f30f0 100644 --- a/docs/NOW.md +++ b/docs/NOW.md @@ -1,3 +1,16 @@ +# NOW -- the differential tool counted 91 of 150 unmeasured files as agreement (2026-08-15) + +Last updated: 2026-08-15 + +## loop: six categories, reason codes, and printed coverage (Closes #2166) + +- **Same binaries, same 150 files, two versions of the same tool.** Five categories printed `150 unchanged, 0 field-loss, 0 unknown` -- total agreement. Six categories print `59 unchanged, 91 not-evaluated (both-error)`, coverage **39.3 %**. Nothing about the compiler changed between those two lines, and the first is the line that had been quoted in pull requests +- **`unchanged` was carrying two statements**: "we compared and found no difference" and "neither binary parsed the file". Both incremented the same counter, so no reviewer and no threshold could separate them. On the 634-spec library the split was 330 measured against 286 unmeasured -- a 52 % base reported as 100 % +- Six mutually exclusive categories now, with the partition **asserted at runtime**; every `not-evaluated` row carries a reason code (`both-error`, `base-timeout`, `candidate-timeout`, `environment-failure`, `excluded-source-loss`); no `PASS` while any `unknown` remains; coverage printed on every run, in the same sentence as any "no regressions" claim, because a caveat in a neighbouring paragraph does not travel with the number +- **Full corpus, single completed run, uniform 12 s threshold, 1089 files** `[measured]`: 524 unchanged/ok, 343 unchanged/fail, **0 regressions**, 1 strict-improvement, 221 not-evaluated (195 both-timeout, 26 candidate-timeout). **Coverage 868/1089 = 79.7 %** +- **The 26 `ok -> timeout` files are the boundary, not a slowdown.** Timed directly, 3 runs each way: median candidate/base ratio **1.010** (min 0.985, max 1.026), on files taking 10.8-11.7 s against a 12 s wall. A 1-3 % jitter is enough to move them across it, so the count difference measures the threshold and not the compiler +- Rules R15 and R16 added to `docs/loop/LOOP-RULES.md` and resealed. `tri corpus-parse`, `corpus-status`, `diffmodes`, `loop-rules` registered in `scripts/ci/loop-tools-tracked.sh`, which fails on an untracked tool -- the state that already destroyed two of these scripts along with every number they produced + # NOW -- BNF: the control that measures what ternary is worth (2026-08-09) Last updated: 2026-08-09 diff --git a/docs/loop/LOOP-RULES.md b/docs/loop/LOOP-RULES.md new file mode 100644 index 000000000..889bb3168 --- /dev/null +++ b/docs/loop/LOOP-RULES.md @@ -0,0 +1,347 @@ +# Loop rules + +Durable rules for the hourly compiler-improvement tick. This file is the single +source of truth for them. + +## Why this file exists + +The rules used to live entirely inside one long scheduled-task field. That field +is silently truncated: 13277 characters were written and 10069 were stored, and +nothing reported the loss. A rule that vanishes without a diagnostic is worse +than a rule that was never written, because the tick continues to behave as if it +were still in force. + +So the rules live here, under version control, with a checksum. The scheduled +task holds only a pointer, the expected checksum, and the volatile queue. If the +field is truncated the pointer is at the top and survives; if this file drifts +from the checksum the mismatch is reported instead of assumed away. + +Verify before relying on any rule below: + +``` +./scripts/tri loop-rules # verify checksum, print status +./scripts/tri loop-rules --reseal # after a deliberate edit +``` + +A mismatch is not a reason to stop the tick. It is a reason to record which +version was in force, because a tick run against unknown rules cannot be audited +afterwards. + +## R0 -- One outcome per tick + +A tick ends in exactly one of two ways: + +- **(A)** one bounded work item closed, where bounded means it has a checkable + completion condition and that condition was checked by measurement; +- **(B)** one `blocked` / `not-evaluated` line with three mandatory fields: the + reason, the owner who can lift it (person, upstream, hardware), and the next + attempt time. + +"Worked on several things" is not an outcome. If the item turns out to be larger +than one tick, split it, close the smaller part, and return the remainder to the +queue with an estimate. + +Ledger: `cron_tracking//ledger.md`, append only, never rewritten. +Counters in `tick-state.json` beside it. + +## R1 -- A differential claim names the class of loss it checked + +No differential result may be called "0 regressions" unless the metric checks the +class of loss being claimed. The earlier `diffbin` printed "634 specifications, 0 +regressions" over a corpus in which files were losing declared fields, because a +per-file JUDGEMENT ("acceptable trade-off") was printed by the aggregate as a +MEASUREMENT. Re-measured with the same corpus and the same binaries: 616 +unchanged, 13 field-loss, 1 strict-improvement, 4 malformed-input-tradeoff, 0 +unknown. + +Therefore: + +- `diffbin` emits exactly five categories in this order: `unchanged`, + `field-loss`, `strict-improvement`, `malformed-input-tradeoff`, `unknown`. + `field-loss` is tested BEFORE `strict-improvement`. The exit code is determined + by `field-loss` and `unknown`, not by a summary verdict. Aggregating field-loss + into zero regressions is forbidden. +- Phantom discrimination: a removed field is a phantom ONLY if its type text was + empty in the baseline. Otherwise the field was declared by the author and its + disappearance is a loss. A field COUNT cannot tell these apart. +- A field is only an `ExprIdentifier` whose PARENT is a `StructDecl`; `t27c parse` + prints the same node kind for identifiers inside function bodies. +- `cost` prints per stratum: n, median, p95, min-max ms/KB, coefficient of + variation. `alpha` only at n>=8, beside r2 and the KB range. An exponent over a + mixed sample is NOT PRINTED AT ALL -- not "printed with a caveat", because the + caveat does not travel with the number and the number is what gets quoted. It + is a metric of corpus composition, not of the parser (#2133). +- Always print the build profile. Never carry absolute dev-build milliseconds + over to release. + +## R2 -- Your own instrument is the first suspect + +The first `tri damage` printed 429 damaged lines. 230 of them were the legal +bound `target : < 5000ns`, 17 were match arms, 10 were array-literal openings, 6 +were function signatures. 429 was a metric of the REGEX, not of the corpus. The +correct repair is to DELETE the bad signals, not to tune a threshold until the +number looks agreeable. The true figure was 125 lines, 65 files, 15 shapes. + +Corollary measured on 2026-08-15: the reported parse error is not necessarily the +first failing token. `specs/tri/collections/bitset.t27` reports `Expected RBrace, +got Eof` at line 97 for a defect at line 14; the true first failing token +(`KwTest` at 39:8) only becomes visible after the line-14 damage is repaired. +Classification by error message must therefore be done on the repaired corpus, +and an error signature is a stopping point, not a cause. + +## R3 -- Parse success is not proof of repair + +Deleting a line also makes a file parse. Double validation is mandatory: (1) +parse exit 0; and (2) the specific field returned with a NON-EMPTY type AND no +previously present field disappeared. + +## R4 -- A test must be run against the state before the fix + +After adding a test, run it against the pre-fix state. If it passes there too it +is a regression guard, NOT proof of the fix, and it must be called that. A check +confirmed only in the green state is decoration. + +Positive example: `scripts/ci/loop-tools-tracked.sh` was verified to FAIL in +exactly the pre-loss state (exit 1, four untracked files) and to pass after the +commit. Own error to avoid repeating: the first version of its step 4 grepped for +a subcommand name in `scripts/tri`, and passed because the word occurred IN A +COMMENT. A check that detects its own documentation is worth nothing. + +Negative fixtures need the same treatment: restore the old bad signal set and +confirm it fires on them (naive 6 of 6, current 0). A fixture that a bad +classifier also passes proves nothing. Measured instance in the ADR-008 work: two +of seven negative fixtures were previously ACCEPTED with silent data loss, and +two more failed with the same generic error as every positive fixture -- so +matching on "non-zero exit" alone would have produced a green suite with no +evidential content. Negative assertions match the specific rejection reason. + +## R5 -- Any tool whose number reaches a report is committed in the same tick + +`cost.py` and `diffbin.py` were written, quoted in a PR, and never committed; the +working copy was later re-cloned and every number they produced became +irreproducible. All six recovery routes were checked (dangling objects, reflog, +shell history, CI artifacts, PR and issue comments, session snapshot) and ALL were +empty; the reflog records the clone, that is, the loss itself. + +Mechanical guard: `bash scripts/ci/loop-tools-tracked.sh` plus the +`loop-tools-gate` workflow. Run it before closing a tick. If a source is lost, +rewrite it from an EXPLICIT CONTRACT, not from memory: memory reproduces the +defect along with the tool. + +## R6 -- The seal certifies identity, not correctness + +Any edit to `bootstrap/src/compiler.rs` requires (a) recomputing +`bootstrap/stage0/FROZEN_HASH`, and (b) a `tri diffbin` run of the old and new +binaries over `specs`. This makes such an edit a GOLD-RING class change, not a +routine patch, and moving the seal needs explicit human approval in the PR. + +Two documented defects in the ceremony itself, found 2026-08-15 and not yet +fixed: `FROZEN.md` §5 step 3 instructs running `cargo run --release -- frozen-digest`, +which cannot run while the tree is drifted, so the ceremony as written is +circular; and §3/§4 declare the seal file format as `<64-hex> ` +while the file contains only the hash. + +## R7 -- Corpus statuses + +Every spec carries exactly one status from a closed set, and the counts sum to the +corpus size by construction: + +| status | meaning | +| --- | --- | +| `clean` | no known defect of any kind | +| `unrecoverable-source-loss` | at least one line whose declared type text was physically truncated; no mechanical rule returns it | +| `repaired-by-mechanical-rule` | damaged only in restorable classes AND the repaired file parses | +| `parser-defect` | undamaged, fails under the baseline compiler, parses under the candidate | +| `unrelated-parse-failure` | fails for a reason none of the above explains | + +Plus one bookkeeping status kept separate and never folded into the five: +`not-evaluated`, for files on which the compiler returned no verdict within the +timeout. Calling a timeout a parse failure is a false measurement: nothing was +decided. + +Damage is tested BEFORE the parse verdict. Ordering parse-first assigned `clean` +to files that parse under the baseline while carrying a truncated type -- the +compiler accepts the line and the declared field is simply gone. A file that +parses is not thereby undamaged. + +Precedence hides co-occurrence, so `tri corpus-status` emits `co_occurring` +beside every status. Read the status for the partition; read `co_occurring` +before claiming a cause. + +The target is not a formal `field-loss = 0` bought with invented types. The +target is `field-loss = 0` over a corpus in which every entry has a definite +status and the unrecoverable lines are in an explicitly excluded set. + +## R8 -- Two units of measurement + +If the unit of repair is a class and the unit of validation is a file, both runs +are required. The per-class run answers "is one rule sufficient for this shape"; +the per-file run (`--combined`) answers "what does the rule achieve". Refuted in +tick B: the 6 still-malformed classes were explained NOT by an adjacent destroyed +line (0 of 0) but by adjacent damage of other restorable classes. + +## R9 -- After a repair, expect a new failure class + +Removing two defects exposed a third that had been unobservable (#2162). File it +as a separate issue. Do not append it to the previous cause, and do not collapse +the remainder into one cause. + +## R10 -- CI topology: an absent check is not a passing check + +`now-sync-gate.yml`, `issue-gate.yml` and `seal-staleness-warn.yml` were declared +`branches: [master]`. On a PR whose base was another branch they did not run at +all, and `gh pr checks` showed a green list that no substantive gate had +examined. The `branches` filter has been removed from the `pull_request` triggers +of all three; `push` keeps its filter. + +Until that change is merged, on any stacked PR: (a) run the substantive gates by +hand, (b) DISCLOSE the bypass in a PR comment, (c) do not call the branch checked +until it is retargeted at master. + +## R11 -- Prohibitions + +Violating any of these fails the tick. + +- Do not touch other working copies or other crons' directories: + `workspace/goldsieve`, `workspace/corpus`, any `cron_tracking/*` but your own. +- Do not push to `master`, do not merge, do not enable auto-merge. Only a push to + a `w699-` branch and opening a PR are permitted. Merging is a human act. +- Do not restore code with `git checkout` / `git reset` over unsaved edits in the + working copy. +- AUTOCLOSING ISSUES IS FORBIDDEN. Never close issues, least of all in bulk. Most + of the tracker consists of journal entries (`wave ...`, `formal: ... (Prop. NN)`) + with no completion condition; they are not tasks, but they must not be closed. +- Never claim "first", "only", or "best". Every number carries a tag: + `[measured]`, `[modelled]`, `[open hypothesis]`. +- Do not `git add` before checks that run `git stash` / `git stash pop`: the pop + unstages, and a partial set lands in the commit. This is how a commit with one + file out of three reached a branch. Stage AFTER all checks, and compare + `git show --stat HEAD` against the intended file list before pushing. +- Do not write `Closes #N` for an issue that does not exist. Create the issue + first, then reference it. `check-linked-issue` passes on syntax alone, so a + dead reference clears the gate. +- Repository files are English only (LANG-EN, ADR-004; `build.rs` enforces it). + Reports and ledger entries addressed to the user are Russian only. + +## R12 -- Before opening a PR + +Add a `## (Closes #N)` section to the top of `docs/NOW.md`, set +`Last updated:` to the current UTC date (the gate accepts yesterday / today / +tomorrow UTC), and put a literal `Closes #N` line in the PR body -- `Refs #N` +does NOT satisfy `check-linked-issue`. + +Four required checks: `check`, `validate`, `check-now-freshness`, +`check-linked-issue`. + +Known baseline failures, reproduced on `master`, not attributable to a branch: +`fpga-formal` and `fpga-synthesis` (#2153); `scripts/check-first-party-doc-language.sh` +has 8 pre-existing errors; `cargo build --release` fails for the `tri` binary +with 36 compile errors in its test target; five integration test targets fail +identically on master and on branch (`verilog_r_si_1` 1/1, `verilog_array_param_index` +0/1, `verilog_translate_off` 0/2, `verilog_array_literal_expr` 1/1, +`icarus_lowerable` 309 passed / 40 failed). + +## R13 -- `unchanged` is two statements, and one of them is silence + +Measured 2026-08-15. `diffbin` recorded `unchanged` for 616 of 634 files. Split by +its own `reason` field: 330 were "both parsed, field sets identical" and 286 were +"both error" -- both binaries failed to parse the file. The second group is not a +finding of no difference; it is the absence of any verdict. So the historical +figure "634 specs, 0 regressions" rested on measurement of 330 files, 52 % of the +corpus, with the other 48 % counted as agreement because neither side spoke. + +This is R1 one layer down: the aggregate did not lie about the files it measured, +it lied about how many it measured. + +Consequence that would otherwise be misread as a regression: repairing a file +moves it out of the silent bucket, so a pre-existing divergence becomes visible +for the first time and looks new. Measured instance -- `tri/encoding/mime.t27`, +`tri/search/aho_corasick.t27`, `tri/trees/quadtree.t27` were `unchanged` (both +error) before the mechanical repair and `field-loss` after it. The repair did not +create the loss; it made the loss observable. All three carry `damage-destroyed` +as well as `damage-restorable`, so the destroyed line is still present. + +Corollary for aggregate comparisons: tick B reported field-loss 13 -> 8 and the +arithmetic was right, but the composition was not the same set. Re-measured: +8 of the 13 disappeared, 3 new ones appeared, 5 remained; 13 - 8 + 3 = 8. A +matching total across two corpora is not the same finding, and a delta quoted +without the composition hides a sign change. + +Therefore: never compare a bare `unchanged` count across corpora or across ticks, +and never quote a field-loss delta without naming which files entered and left. +`tri diffmodes` prints the split. + +## R14 -- A timeout is a property of the run, not of the file + +Measured 2026-08-15 on `specs/scratch` at a 12 s threshold: the baseline run gave +202 ok / 195 timeout, a later run gave 176 ok / 221 timeout. 26 files moved +`ok -> timeout` and 0 moved back. All 26 had baseline times between 9322 ms and +11952 ms -- every one already within half of the threshold, sitting against the +boundary. The stable part (58 `fail`) was identical in both runs. + +So the movement measures machine load at the boundary, not the compiler. Two +prohibitions follow: + +- Never conclude "the candidate got slower" from a shift in timeout counts when + the moved files were already near the threshold. Report the baseline times of + the moved files; that is what settles it. +- Never mix thresholds inside one comparison. A run at 25 s and a run at 12 s do + not produce comparable `not-evaluated` sets, and joining them silently + manufactures a difference. Re-run at the single threshold instead. + +`not-evaluated` is bookkeeping, kept out of the five statuses (R7), and it is not +comparable across runs. + +## R15 -- `unchanged` may never mean "we could not compare" + +A differential category that counts *both sides failed* as agreement reports an +absence of evidence as evidence of absence. + +Measured 2026-08-15, same binary pair `/tmp/t27c.base -> /tmp/t27c.fixed`, same +150-file slice, two versions of the same tool: + +| tool | verdict printed | +| --- | --- | +| five categories | `150 unchanged, 0 field-loss, 0 unknown` -- i.e. total agreement | +| six categories | `59 unchanged, 91 not-evaluated (both-error)`, coverage 39.3 % | + +Nothing about the binaries changed between those two lines. The first is what had +been quoted in pull requests. + +Rules that follow, all enforced in `scripts/tri_loop/diffbin.py`: + +- Six mutually exclusive categories: `unchanged`, `field-loss`, + `strict-improvement`, `malformed-input-tradeoff`, `unknown`, `not-evaluated`. + The set must be *asserted* to partition the corpus at runtime; claiming it in a + docstring is not a check. +- Every `not-evaluated` row carries a reason code: `both-error`, `base-timeout`, + `candidate-timeout`, `environment-failure`, `excluded-source-loss`, or a named + other. "No verdict" for six different reasons is six different facts, and only + some of them concern the compiler. An uncoded `not-evaluated` is a hard error. +- No `PASS` while any `unknown` remains, whatever the other counts say. +- Coverage -- the measured fraction -- is printed on every run. The phrase "no + regressions" is admissible only with `field-loss = 0`, `unknown = 0`, and that + fraction attached to the same sentence. A caveat in a neighbouring paragraph + does not travel with the number; readers quote numbers, not paragraphs. +- Coverage rises only by repairing files or excluding them with a status. It never + rises by recategorising them. + +## R16 -- A silent gate is fixed by a configuration test, not by vigilance + +`on: pull_request: branches: [master]` makes a gate skip every PR whose base is +another branch. On a stacked PR the gate is simply absent and `gh pr checks` shows +green. Nothing anywhere reports a gate that did not run, so no amount of care at +review time detects it. + +Measured 2026-08-15: eleven merge-critical workflows carried the filter. An +earlier count of seven, taken from a filtered `grep`, was wrong -- which is the +second half of the rule: enumerate by parsing the files, and record the correction +instead of substituting it, because the wrong number gets quoted downstream. + +- The list of merge-critical workflows is written out in code and reviewed as + code. An inferred list ("everything named `*-gate`") stops covering a gate the + moment someone renames it. +- The test must be shown to fail against the tree *before* the fix. 11 before, 0 + after. A test that also passes beforehand is a regression guard, not evidence. +- A gate that lands red and stays red for a reason nobody is permitted to fix + teaches everyone to ignore red. Report those as warnings and name the reason. diff --git a/docs/loop/LOOP-RULES.sha256 b/docs/loop/LOOP-RULES.sha256 new file mode 100644 index 000000000..c7be97f04 --- /dev/null +++ b/docs/loop/LOOP-RULES.sha256 @@ -0,0 +1 @@ +fc74dccf1f53c220fd2ffcd279b41255df538db02817a26ad58be8865eb4ee6b docs/loop/LOOP-RULES.md diff --git a/scripts/ci/loop-tools-tracked.sh b/scripts/ci/loop-tools-tracked.sh new file mode 100755 index 000000000..4255a92fd --- /dev/null +++ b/scripts/ci/loop-tools-tracked.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# Loss of a loop tool must be impossible, not merely regrettable. +# +# Two scripts -- scripts/tri_loop/cost.py and scripts/tri_loop/diffbin.py -- were +# written, used to produce numbers that were then quoted in a pull request, and +# lost. They were never committed to any branch, and when the working copy was +# re-cloned they went with it. Six recovery routes were checked and all came back +# empty: dangling git objects, the reflog, shell history, CI artifacts, PR and +# issue comments, and the session snapshot. Every number they had produced became +# unreproducible in one stroke. +# +# The failure was not carelessness at the keyboard. It was that nothing in the +# repository could tell the difference between a tool that exists and a tool that +# exists only in one untracked working directory. This script is that difference. +# +# It fails when: +# 1. a required loop tool is missing from the working tree +# 2. a required loop tool exists but git does not track it -- the exact state +# that preceded the loss +# 3. any file under scripts/tri_loop/ is untracked, so a NEW tool cannot be +# quietly added and then lost the same way +# 4. a required tool is not executable via the dispatcher, i.e. `tri ` +# does not resolve +# +# Not claimed: that this makes the tools correct, or that a tracked file cannot +# be deleted in a later commit. It closes the specific hole that swallowed these +# two files -- work that exists only outside version control -- and nothing wider. +set -uo pipefail + +cd "$(dirname "$0")/../.." || exit 2 + +REQUIRED_TOOLS=( + "scripts/tri_loop/triage.py" + "scripts/tri_loop/cost.py" + "scripts/tri_loop/diffbin.py" + "scripts/tri_loop/damage.py" + "scripts/tri_loop/damage_freeze.py" + "scripts/tri_loop/damage_repair.py" + "scripts/tri_loop/corpus_parse.py" + "scripts/tri_loop/corpus_status.py" + "scripts/tri_loop/diffmodes.py" + "scripts/tri_loop/loop_rules.py" +) +REQUIRED_SUBCOMMANDS=(triage cost diffbin damage damage-freeze damage-repair \ + corpus-parse corpus-status diffmodes loop-rules) + +fail=0 +note() { printf ' %s\n' "$1"; } + +echo "loop-tools-tracked: the tools the measurement loop depends on" +echo + +echo "1. present in the working tree" +for f in "${REQUIRED_TOOLS[@]}"; do + if [[ -f "$f" ]]; then + note "ok $f" + else + note "MISSING $f" + fail=1 + fi +done + +echo +echo "2. tracked by git (this is the check that would have caught the loss)" +for f in "${REQUIRED_TOOLS[@]}"; do + [[ -f "$f" ]] || continue + if git ls-files --error-unmatch "$f" >/dev/null 2>&1; then + note "tracked $f" + else + note "UNTRACKED $f -- exists here and nowhere else; one re-clone loses it" + fail=1 + fi +done + +echo +echo "3. nothing under scripts/tri_loop/ is untracked" +untracked=$(git ls-files --others --exclude-standard -- scripts/tri_loop/ 2>/dev/null) +if [[ -z "$untracked" ]]; then + note "ok no untracked files under scripts/tri_loop/" +else + while IFS= read -r u; do + [[ -n "$u" ]] && note "UNTRACKED $u" + done <<<"$untracked" + fail=1 +fi + +echo +echo "4. reachable through the dispatcher" +# The dispatcher resolves `tri ` generically, to tri_loop/.py. +# So the honest check is that the generic exec line is still there AND that the +# file each subcommand resolves to exists. An earlier draft of this check grepped +# scripts/tri for the literal command name, which passed for `tri triage` only +# because the word appears in a comment -- a check that reports the presence of +# its own documentation is worth nothing. +if grep -q 'exec python3 "\$LOOP_DIR/\${cmd//-/_}.py"' scripts/tri; then + note "ok generic loop dispatch line present" +else + note "BROKEN generic loop dispatch line is gone from scripts/tri" + fail=1 +fi +for c in "${REQUIRED_SUBCOMMANDS[@]}"; do + target="scripts/tri_loop/${c//-/_}.py" + if [[ -f "$target" ]]; then + note "ok tri $c -> $target" + else + note "UNROUTED tri $c -> $target does not exist" + fail=1 + fi +done + +echo +if [[ $fail -eq 0 ]]; then + echo "PASS: every loop tool is present, tracked, and routed." + echo "This says nothing about whether any of them is correct." +else + echo "FAIL: a loop tool is missing, untracked, or unrouted." + echo "An untracked tool is the state that already destroyed two of these" + echo "scripts and every number they produced. Commit it before relying on" + echo "its output for any claim." +fi +exit $fail diff --git a/scripts/tri b/scripts/tri index e012eb108..1313b0ed0 100755 --- a/scripts/tri +++ b/scripts/tri @@ -2,6 +2,33 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +# Loop helpers live in scripts/tri_loop/ as Python, and are dispatched BEFORE the +# compiler binary is even looked for. They measure and report on the repository, +# they do not compile it, and none of them mutates anything. Requiring a built +# t27c first was a defect: `tri triage` and `tri damage` read the tracker and the +# spec text and have no use for the compiler, yet a machine with no build could +# not run them. Keeping the helpers ahead of the lookup also means a missing +# compiler cannot take the reporting tools down with it. +cmd="${1:-}" +LOOP_DIR="$SCRIPT_DIR/tri_loop" +if [[ "$cmd" == "loop-help" ]]; then + echo "tri loop helpers (report only, never mutate):" + for f in "$LOOP_DIR"/*.py; do + [[ -e "$f" ]] || continue + name="$(basename "$f" .py)" + printf ' %-12s %s\n' "$name" "$(sed -n '2s/^"""tri [a-z-]* -- //p' "$f")" + done + echo + echo "Every helper prints what it does NOT establish alongside what it does." + echo "Read that part: an aggregate that omits the class of loss it failed to" + echo "check is how a differential once reported zero regressions over a corpus" + echo "in which thirteen files lost declared fields." + exit 0 +fi +if [[ -n "$cmd" && -f "$LOOP_DIR/${cmd//-/_}.py" ]]; then + exec python3 "$LOOP_DIR/${cmd//-/_}.py" "${@:2}" +fi + T27C="${TRI_T27C:-}" if [[ -z "$T27C" ]]; then for p in "$REPO_ROOT/target/release/t27c" "$REPO_ROOT/target/debug/t27c" "$REPO_ROOT/bootstrap/target/release/t27c" "$REPO_ROOT/bootstrap/target/debug/t27c"; do @@ -14,7 +41,6 @@ if [[ ! -x "$T27C" ]]; then fi # `t27c suite` is the only subcommand that accepts --repo-root. # Map `tri test` to the suite subcommand; pass other commands through unchanged. -cmd="${1:-}" if [[ "$cmd" == "test" || "$cmd" == "suite" ]]; then # Forward everything after the subcommand to t27c suite, including --json. exec "$T27C" suite --repo-root "$REPO_ROOT" "${@:2}" diff --git a/scripts/tri_loop/corpus_parse.py b/scripts/tri_loop/corpus_parse.py new file mode 100644 index 000000000..43b87d7c3 --- /dev/null +++ b/scripts/tri_loop/corpus_parse.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Parse every spec in a tree with a given t27c binary and record a per-file verdict. + +Why this exists as a separate tool: a whole-corpus parse run is the only way to +tell a parser change apart from a corpus change, and the naive shell loop that +preceded it silently hung. The hang was not mysterious: within the +`specs/scratch/*_bench_NNd_aos_call_dedup.t27` family, parse time grows +multiplicatively per added dimension and NOT with file size (11d is 317 KB and +4.5 s while 10d is 2.2 MB and 2.0 s). A per-file timeout is therefore mandatory, +and files that hit it are reported as `timeout`, never silently dropped: an +excluded file is a fact about the measurement, not an absence of one. + +Verdicts, one per file, exhaustive and mutually exclusive: + ok - exit 0 + fail - non-zero exit within the time budget + timeout - exceeded the per-file budget, verdict unknown + +Usage: + corpus_parse.py --out results.json [--timeout SEC] + [--jobs N] [--exclude GLOB]... +""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import fnmatch +import json +import os +import subprocess +import sys +import time + + +def find_specs(root: str, excludes: list[str]) -> list[str]: + out: list[str] = [] + for dirpath, _dirnames, filenames in os.walk(root): + for name in sorted(filenames): + if not name.endswith(".t27"): + continue + path = os.path.join(dirpath, name) + if any(fnmatch.fnmatch(path, pat) for pat in excludes): + continue + out.append(path) + return sorted(out) + + +def parse_one(binary: str, path: str, budget: float) -> dict: + started = time.monotonic() + try: + proc = subprocess.run( + [binary, "parse", path], + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + timeout=budget, + ) + except subprocess.TimeoutExpired: + return { + "path": path, + "verdict": "timeout", + "exit_code": None, + "ms": round((time.monotonic() - started) * 1000, 1), + "first_error": None, + } + elapsed_ms = round((time.monotonic() - started) * 1000, 1) + if proc.returncode == 0: + return { + "path": path, + "verdict": "ok", + "exit_code": 0, + "ms": elapsed_ms, + "first_error": None, + } + text = proc.stderr.decode("utf-8", "replace") + first = None + for line in text.splitlines(): + if "rror" in line: + first = line.strip()[:400] + break + return { + "path": path, + "verdict": "fail", + "exit_code": proc.returncode, + "ms": elapsed_ms, + "first_error": first, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("binary") + ap.add_argument("spec_dir") + ap.add_argument("--out", required=True) + ap.add_argument("--timeout", type=float, default=20.0) + ap.add_argument("--jobs", type=int, default=max(1, (os.cpu_count() or 2))) + ap.add_argument("--exclude", action="append", default=[]) + args = ap.parse_args() + + if not os.path.isfile(args.binary): + print(f"binary not found: {args.binary}", file=sys.stderr) + return 2 + + specs = find_specs(args.spec_dir, args.exclude) + results: list[dict] = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=args.jobs) as pool: + futures = [ + pool.submit(parse_one, args.binary, p, args.timeout) for p in specs + ] + for fut in concurrent.futures.as_completed(futures): + results.append(fut.result()) + + results.sort(key=lambda r: r["path"]) + counts = {"ok": 0, "fail": 0, "timeout": 0} + for r in results: + counts[r["verdict"]] += 1 + + payload = { + "binary": args.binary, + "spec_dir": args.spec_dir, + "timeout_sec": args.timeout, + "excluded_globs": args.exclude, + "total": len(results), + "counts": counts, + "results": results, + } + with open(args.out, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2, sort_keys=True) + + print(f"total {len(results)}: " + " ".join(f"{k}={v}" for k, v in counts.items())) + print(f"written {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tri_loop/corpus_status.py b/scripts/tri_loop/corpus_status.py new file mode 100644 index 000000000..64a77beca --- /dev/null +++ b/scripts/tri_loop/corpus_status.py @@ -0,0 +1,295 @@ +#!/usr/bin/env python3 +"""tri corpus-status -- give every spec in the corpus exactly one defect status. + +Why this exists. Across several ticks the same corpus was described with numbers +that could not be reconciled: "13 field-loss", "8 residual", "49 parse-restored", +"10 files using a parameterised const". None of them was wrong, and none of them +was comparable to the others, because each counted a different population under a +different binary with a different exclusion. A number that cannot say which +population it ranges over cannot be checked, and cannot be cited. + +So the unit here is the FILE, the output is TOTAL over the corpus, and every file +carries a status drawn from one closed set. Statuses partition: each file gets +exactly one, and the counts sum to the corpus size by construction, asserted at +the end rather than hoped for. + +The five statuses of the order: + + clean parses under the baseline compiler; no defect. + unrecoverable-source-loss contains at least one line whose declared type text + was physically truncated. No mechanical rule can + return it; a substituted type would be invention. + repaired-by-mechanical-rule damaged only in restorable classes, AND the repaired + file parses. Both halves are required: parse success + after a repair is not evidence the repair was right + unless the repair is the thing that produced it. + parser-defect the file is not damaged, fails under the baseline + compiler, and parses under the candidate. The + candidate is what changed, so the compiler was the + defect. + unrelated-parse-failure fails for a reason none of the above explains. The + first failing token is recorded for each one; this + status is a statement of ignorance, not a category. + +And one bookkeeping status that is NOT one of the five, kept separate and never +folded into them: + + not-evaluated the compiler did not return a verdict within the + timeout. Calling a timeout a parse failure would be + a false measurement: nothing was decided. + +Precedence, and what it hides. A file can carry more than one defect at once, so +a single status per file requires an order, and any order hides co-occurrence. +Measured case: specs/tri/collections/bitset.t27 carries a damaged line at 14 AND +a keyword collision at 39, and the second is invisible until the first is +repaired -- the parser reports the cascade endpoint (Eof at 97), not the first +failing token. So this tool emits `co_occurring` beside every status. Read the +status for the partition; read `co_occurring` before claiming a cause. + +Precedence is unrecoverable-source-loss first, because a file that cannot be +mechanically restored cannot reach clean no matter what else is fixed in it, and +that is the strongest thing knowable about it. + +What this does NOT establish. That the repaired text is what the author wrote -- +only that it is closed, parses, and returns the declared fields. That a +parser-defect file is CORRECTLY parsed -- only that it now reaches the parser's +exit. That unrelated-parse-failure files share a cause; they are grouped by the +absence of an explanation, which is not a group. + +Refs #2162, #2163. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections import Counter, defaultdict +from pathlib import Path + +STATUSES = ( + "clean", + "unrecoverable-source-loss", + "repaired-by-mechanical-rule", + "parser-defect", + "unrelated-parse-failure", +) +BOOKKEEPING = ("not-evaluated",) + + +def load_verdicts(path: Path, strip_prefix: str = "") -> dict[str, dict]: + payload = json.loads(path.read_text()) + out: dict[str, dict] = {} + for row in payload["results"]: + key = row["path"] + if strip_prefix and key.startswith(strip_prefix): + key = key[len(strip_prefix) :].lstrip("/") + out[key] = row + return out + + +def load_snapshot(path: Path) -> tuple[dict[str, list[dict]], dict[str, str]]: + """Return damage rows grouped by file, and the class -> verdict map.""" + payload = json.loads(path.read_text()) + by_file: dict[str, list[dict]] = defaultdict(list) + for row in payload["rows"]: + by_file[row["file"]].append(row) + class_verdict: dict[str, str] = {} + for entry in payload["class_index"]: + cid = entry.get("class_id") or entry.get("id") + restorable = entry.get("restorable") + if restorable is None: + # Derived, not guessed. The damage mechanism replaces the OPENING + # quote of a type with '['. A line is restorable when the CLOSING + # quote survived, because then the type text between the two + # delimiters is intact and putting the opening quote back is a + # decision procedure. When the shape carries no closing quote the + # type text itself ran off the end of the line and is gone; there is + # nothing to re-delimit. + # + # On the frozen snapshot this discriminator selects exactly the three + # classes DC-72bb7dcf (10), DC-83e0cb30 (7) and DC-801c2390 (1), + # shapes '[[]X [,', '[[][,' and '[[][9,' -- 18 lines, matching the 18 + # held by the repair tool for a human language decision. Two + # independently derived routes agreeing on the same 18 is what makes + # this a check rather than a restatement. + # + # Default direction matters: if the shape were unreadable this must + # fall to destroyed, because the failure mode of guessing restorable + # is inventing type text. + shape = entry.get("shape") or "" + restorable = '"' in shape + class_verdict[cid] = "restorable" if restorable else "destroyed" + return by_file, class_verdict + + +def main() -> int: + ap = argparse.ArgumentParser(description="assign one defect status per spec") + ap.add_argument("--base", required=True, type=Path, help="corpus-parse json, baseline compiler") + ap.add_argument("--cand", required=True, type=Path, help="corpus-parse json, candidate compiler") + ap.add_argument("--repaired", type=Path, help="corpus-parse json over a repaired tree") + ap.add_argument("--repaired-prefix", default="/tmp/repaired", help="path prefix to strip from repaired json") + ap.add_argument("--snapshot", required=True, type=Path, help="tri damage-freeze json") + ap.add_argument( + "--destroyed-class", + action="append", + default=[], + help="class id to force to destroyed (repeatable)", + ) + ap.add_argument("--out", type=Path, help="write per-file statuses as json") + args = ap.parse_args() + + base = load_verdicts(args.base) + cand = load_verdicts(args.cand) + repaired = load_verdicts(args.repaired, args.repaired_prefix) if args.repaired else {} + damage, class_verdict = load_snapshot(args.snapshot) + for cid in args.destroyed_class: + class_verdict[cid] = "destroyed" + + missing = sorted(set(base) ^ set(cand)) + if missing: + print(f"base and candidate cover different files ({len(missing)} differ); refusing", file=sys.stderr) + for p in missing[:10]: + print(f" {p}", file=sys.stderr) + return 2 + + rows = [] + for path in sorted(base): + b, c = base[path], cand[path] + rowdamage = damage.get(path, []) + classes = {r["class_id"] for r in rowdamage} + destroyed = sorted(cid for cid in classes if class_verdict.get(cid) == "destroyed") + restorable = sorted(cid for cid in classes if class_verdict.get(cid) == "restorable") + + co: list[str] = [] + if destroyed: + co.append("damage-destroyed") + if restorable: + co.append("damage-restorable") + if b["verdict"] == "fail" and c["verdict"] == "ok": + co.append("parser-fixed-by-candidate") + if b["verdict"] == "timeout" or c["verdict"] == "timeout": + co.append("timeout-in-one-mode") + + rep = repaired.get(path) + if rep is not None: + co.append("repair-parses" if rep["verdict"] == "ok" else "repair-still-fails") + + # Damage is tested BEFORE the parse verdict, and that order was a + # correction, not a preference. Ordering parse-first assigned `clean` to + # five files that parse under the baseline while carrying a truncated + # type: the compiler accepts the line and the declared field is simply + # gone. Calling those clean is the same error class as an aggregate that + # reported zero regressions over a corpus in which files lost declared + # fields -- a file that parses is not thereby undamaged. So `clean` here + # means no known defect of any kind, and a damaged file keeps its damage + # status whether or not the parser complains about it. + if b["verdict"] == "timeout" and c["verdict"] == "timeout": + status = "not-evaluated" + elif destroyed: + status = "unrecoverable-source-loss" + elif restorable and rep is not None and rep["verdict"] == "ok": + status = "repaired-by-mechanical-rule" + elif classes: + status = "unrelated-parse-failure" + elif b["verdict"] == "ok": + status = "clean" + elif c["verdict"] == "ok": + status = "parser-defect" + else: + status = "unrelated-parse-failure" + + rows.append( + { + "path": path, + "status": status, + "co_occurring": co, + "base_verdict": b["verdict"], + "cand_verdict": c["verdict"], + "repaired_verdict": rep["verdict"] if rep else None, + "damage_classes": sorted(classes), + "first_error_base": b.get("first_error"), + "first_error_cand": c.get("first_error"), + } + ) + + counts = Counter(r["status"] for r in rows) + total = len(rows) + assert sum(counts.values()) == total, "statuses must partition the corpus" + + evaluable = [r for r in rows if r["status"] != "not-evaluated"] + print(f"corpus: {total} spec(s); evaluable {len(evaluable)}, not-evaluated {counts['not-evaluated']}") + print() + print("status distribution (the five statuses partition the evaluable corpus):") + for s in STATUSES: + print(f" {s:<28} {counts[s]:>5}") + for s in BOOKKEEPING: + print(f" {s:<28} {counts[s]:>5} (excluded, shown not hidden)") + print() + + # Co-occurrence is the thing a single status per file destroys, so print it. + print("co-occurrence inside each status (a file may carry several defects):") + for s in STATUSES + BOOKKEEPING: + group = [r for r in rows if r["status"] == s] + if not group: + continue + pairs = Counter(tag for r in group for tag in r["co_occurring"]) + if not pairs: + continue + detail = ", ".join(f"{k}={v}" for k, v in sorted(pairs.items())) + print(f" {s}: {detail}") + print() + + stuck = [r for r in rows if r["status"] == "unrelated-parse-failure"] + if stuck: + # Grouped by error signature rather than listed per file. A list of 248 + # paths reads as 248 problems; the signatures show how few distinct forms + # are actually behind them. The grouping is a presentation of the same + # rows, not a claim that one signature is one cause: an error message is + # where the parser stopped, and on this corpus the stopping point has + # already been shown to be a cascade endpoint rather than the first + # failing token (bitset.t27 reports Eof at 97 for a defect at line 14). + print(f"unrelated-parse-failure -- grouped by error signature ({len(stuck)} file(s)):") + sigs: dict[str, list[str]] = defaultdict(list) + for r in stuck: + raw = (r["first_error_cand"] or "no error text").strip().replace("\n", " ") + # Drop line:col and quoted lexemes so the same form groups together. + sig = raw.split(" at line ")[0] + sig = sig.replace("Error: Parse error: ", "") + if " near line " in sig: + head, tail = sig.split(" near line ", 1) + sig = head + " near line N: " + tail.split(": ", 1)[-1] + sigs[sig[:96]].append(r["path"]) + for sig, paths in sorted(sigs.items(), key=lambda kv: -len(kv[1])): + print(f" {len(paths):>4} {sig}") + print(f" e.g. {paths[0]}") + print(f" {len(sigs)} distinct signature(s) over {len(stuck)} file(s)") + print() + + print("NOT established by this table: that the repaired text is the author's;") + print("that a parser-defect file is parsed correctly rather than merely accepted;") + print("that unrelated-parse-failure files share a cause -- they share only the") + print("absence of one, which is not a category.") + + if args.out: + args.out.write_text( + json.dumps( + { + "tool": "tri corpus-status", + "total": total, + "counts": dict(counts), + "statuses": list(STATUSES), + "bookkeeping": list(BOOKKEEPING), + "rows": rows, + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + print(f"\nwritten {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tri_loop/cost.py b/scripts/tri_loop/cost.py new file mode 100644 index 000000000..6381ca67b --- /dev/null +++ b/scripts/tri_loop/cost.py @@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +"""tri cost -- parse cost per KB of spec, by stratum, with the spread shown. + +Why this is a rewrite and not a restore. + +An earlier version of this script was written, used to produce a scaling exponent +for the parser, and lost -- never committed, working copy re-cloned. Six recovery +routes were checked and came back empty (dangling objects, reflog, shell history, +CI artifacts, PR and issue comments, session snapshot). So this is a +reimplementation from a written contract. + +The defect the old output invited, which this one refuses to invite. + +Fitting one exponent across the whole corpus produces a number, and the number is +a property of the CORPUS COMPOSITION, not of the parser (this is #2133). The spec +families differ in kind, not just in size: a spec that is mostly a table of +constants and a spec that is mostly nested generic types both have a size in KB, +and mixing them means the fit is measuring which family happens to sit at which +size. A single alpha over a mixed sample is therefore not a scaling law, it is an +artefact of the sample, and printing it without that warning is how the previous +number got quoted as if it described the parser. + +So: + + * everything is reported per STRATUM, one stratum per spec directory family + * every stratum reports n, median, p95, min-max range of ms/KB, and CV + * the coefficient of variation is printed because it is the number that says + whether the median means anything: a stratum with CV above ~0.5 is not one + population and its median is a summary of a mixture + * alpha is printed ONLY per stratum, ONLY with n >= 8, ONLY with the size range + it was fitted over, and ONLY next to the CV. Cross-family alpha is refused + outright rather than printed with a caveat, because a printed number gets + quoted and the caveat does not travel with it. + +Also: absolute milliseconds from a debug build do not transfer to a release +build. Ratios and exponents transfer; absolute times do not, and this output +labels the build it measured so the distinction survives. + +Usage: + tri cost [glob-or-dir ...] [--limit N] [--repeat K] + [--json PATH] [--timeout SEC] [--min-kb F] + +Exit status: + 0 measured at least one stratum + 2 usage or setup error +""" +import glob as globmod +import json +import math +import os +import statistics +import subprocess +import sys +import time + +DEFAULT_TIMEOUT = 30 +MIN_N_FOR_ALPHA = 8 +CV_SUSPECT = 0.5 + + +def measure(binary, path, timeout, repeat): + """Return (ok, best_ms). Best of `repeat`, not the mean. + + Best-of is the right summary for a timing measurement whose noise is + one-sided: an interfering process can only make a run slower, never faster, + so the minimum is the least contaminated estimate of the cost. The mean would + fold the interference in and the p95 across files would then be reporting the + sandbox rather than the parser. + """ + best = None + for _ in range(repeat): + t0 = time.perf_counter() + try: + r = subprocess.run([binary, "parse", path], capture_output=True, + timeout=timeout) + except subprocess.TimeoutExpired: + return False, None + except OSError: + return False, None + dt = (time.perf_counter() - t0) * 1000.0 + if r.returncode != 0: + return False, None + best = dt if best is None else min(best, dt) + return True, best + + +def fit_alpha(sizes_kb, times_ms): + """Least-squares slope of log(time) against log(size). + + Returned with r2 so the caller can say how much of the variation the fit even + explains. A slope with r2 of 0.2 is a slope through a cloud. + """ + xs = [math.log(s) for s in sizes_kb] + ys = [math.log(t) for t in times_ms] + n = len(xs) + mx, my = sum(xs) / n, sum(ys) / n + sxx = sum((x - mx) ** 2 for x in xs) + if sxx == 0: + return None, None + sxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) + slope = sxy / sxx + pred = [my + slope * (x - mx) for x in xs] + sst = sum((y - my) ** 2 for y in ys) + sse = sum((y - p) ** 2 for y, p in zip(ys, pred)) + r2 = None if sst == 0 else 1.0 - sse / sst + return slope, r2 + + +def stratum_of(path, corpus_root): + """The spec family: the directory holding the file, relative to the root.""" + rel = os.path.relpath(os.path.dirname(path), corpus_root) + return rel if rel != "." else "(root)" + + +def build_label(binary): + """debug or release, read off the path, so ms are never quoted context-free.""" + p = os.path.realpath(binary) + if f"{os.sep}release{os.sep}" in p: + return "release" + if f"{os.sep}debug{os.sep}" in p: + return "debug" + return "unknown-profile" + + +def summarise(rows): + per_kb = sorted(r["ms_per_kb"] for r in rows) + n = len(per_kb) + med = statistics.median(per_kb) + mean = statistics.fmean(per_kb) + sd = statistics.stdev(per_kb) if n > 1 else 0.0 + cv = (sd / mean) if mean else 0.0 + p95 = per_kb[min(n - 1, int(math.ceil(0.95 * n)) - 1)] + out = {"n": n, "median_ms_per_kb": med, "p95_ms_per_kb": p95, + "min_ms_per_kb": per_kb[0], "max_ms_per_kb": per_kb[-1], + "cv": cv, + "kb_min": min(r["kb"] for r in rows), + "kb_max": max(r["kb"] for r in rows)} + if n >= MIN_N_FOR_ALPHA: + a, r2 = fit_alpha([r["kb"] for r in rows], [r["ms"] for r in rows]) + out["alpha"] = a + out["alpha_r2"] = r2 + return out + + +def main(argv): + args = [a for a in argv if not a.startswith("--")] + if not args: + print(__doc__.split("Usage:")[1].strip(), file=sys.stderr) + return 2 + binary = args[0] + if not os.path.isfile(binary) or not os.access(binary, os.X_OK): + print(f"not an executable: {binary}", file=sys.stderr) + return 2 + targets = args[1:] or ["specs"] + + limit = None + repeat = 3 + out_json = None + timeout = DEFAULT_TIMEOUT + min_kb = 0.5 + for i, a in enumerate(argv): + if a == "--limit" and i + 1 < len(argv): + limit = int(argv[i + 1]) + if a == "--repeat" and i + 1 < len(argv): + repeat = int(argv[i + 1]) + if a == "--json" and i + 1 < len(argv): + out_json = argv[i + 1] + if a == "--timeout" and i + 1 < len(argv): + timeout = int(argv[i + 1]) + if a == "--min-kb" and i + 1 < len(argv): + min_kb = float(argv[i + 1]) + + files = [] + for t in targets: + if os.path.isdir(t): + for root, _d, names in os.walk(t): + files += [os.path.join(root, n) for n in names if n.endswith(".t27")] + else: + files += [p for p in globmod.glob(t, recursive=True) if p.endswith(".t27")] + files = sorted(set(files)) + corpus_root = targets[0] if os.path.isdir(targets[0]) else "specs" + if limit: + files = files[:limit] + if not files: + print("no .t27 files matched", file=sys.stderr) + return 2 + + rows, skipped, failed = [], 0, 0 + for path in files: + kb = os.path.getsize(path) / 1024.0 + if kb < min_kb: + skipped += 1 + continue + ok, ms = measure(binary, path, timeout, repeat) + if not ok: + failed += 1 + continue + rows.append({"file": path, "kb": kb, "ms": ms, "ms_per_kb": ms / kb, + "stratum": stratum_of(path, corpus_root)}) + + if not rows: + print("nothing measured: every file failed to parse or was below --min-kb", + file=sys.stderr) + return 2 + + strata = {} + for r in rows: + strata.setdefault(r["stratum"], []).append(r) + + profile = build_label(binary) + print(f"binary: {binary} [{profile} build]") + print(f"files: {len(rows)} measured, {skipped} below {min_kb} KB, " + f"{failed} failed to parse") + print(f"repeat: best of {repeat} runs per file\n") + + if profile != "release": + print("NOTE: absolute milliseconds from a non-release build do not") + print("transfer to release. Quote the exponents and the ratios; do not") + print("quote the ms.\n") + + hdr = (f"{'stratum':34s} {'n':>4s} {'med':>8s} {'p95':>8s} " + f"{'min':>8s} {'max':>8s} {'CV':>6s} {'alpha':>7s} {'r2':>5s} KB range") + print(hdr) + print("-" * len(hdr)) + results = {} + for name, rs in sorted(strata.items(), key=lambda kv: -len(kv[1])): + s = summarise(rs) + results[name] = s + a = f"{s['alpha']:7.2f}" if s.get("alpha") is not None else " -" + r2 = f"{s['alpha_r2']:5.2f}" if s.get("alpha_r2") is not None else " -" + flag = " *" if s["cv"] > CV_SUSPECT else "" + print(f"{name:34s} {s['n']:4d} {s['median_ms_per_kb']:8.2f} " + f"{s['p95_ms_per_kb']:8.2f} {s['min_ms_per_kb']:8.2f} " + f"{s['max_ms_per_kb']:8.2f} {s['cv']:6.2f} {a} {r2} " + f"{s['kb_min']:.1f}-{s['kb_max']:.1f}{flag}") + + print("\nms/KB columns are median, p95, min, max. CV is the coefficient of") + print("variation of ms/KB within the stratum.") + suspect = [n for n, s in results.items() if s["cv"] > CV_SUSPECT] + if suspect: + print(f"\n* {len(suspect)} stratum/strata have CV > {CV_SUSPECT}. In those the") + print(" median is a summary of a mixture and should not be quoted as the") + print(" cost of the stratum:") + for n in suspect[:12]: + print(f" {n} (CV {results[n]['cv']:.2f}, n={results[n]['n']})") + + noalpha = [n for n, s in results.items() if s.get("alpha") is None] + if noalpha: + print(f"\nalpha withheld for {len(noalpha)} stratum/strata with " + f"n < {MIN_N_FOR_ALPHA}.") + + print("\nNO CROSS-FAMILY ALPHA IS PRINTED, and this is deliberate.") + print("A single exponent fitted across these strata is a metric of corpus") + print("composition, not of the parser (#2133). It would be a number about") + print("which family happens to occupy which size band. Numbers get quoted") + print("and their caveats do not travel with them, so the number is not") + print("produced at all.") + print("\nWhere a per-stratum alpha IS printed, its applicability is exactly:") + print("this binary, this build profile, this stratum, this KB range, and") + print("nothing wider. Read r2 before reading alpha -- a slope with low r2 is") + print("a slope drawn through a cloud.") + + if out_json: + with open(out_json, "w") as fh: + json.dump({"binary": binary, "profile": profile, "repeat": repeat, + "files_measured": len(rows), "skipped": skipped, + "failed": failed, "strata": results, "rows": rows}, + fh, indent=2) + print(f"\nwrote {out_json}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/damage.py b/scripts/tri_loop/damage.py new file mode 100644 index 000000000..92b85e441 --- /dev/null +++ b/scripts/tri_loop/damage.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""tri damage -- classify the mangled type annotations in the spec corpus (#2154). + +What this is for. + +A rewrite left behind lines whose type side is not a type: `public_key : [[]U8",` +instead of `public_key : U8`. Those lines are the floor under every field-set +measurement taken on this corpus -- 16 of the 18 files whose field sets moved +between two compiler binaries are in this set, which means the measurement was +reading the damage and not the parser. + +This tool only reports. It rewrites nothing. The point of a classifier is that a +later repair can be an inspectable diff of a stated size per class, instead of +one sweep over 115 lines that nobody can review. + +Class is decided by the shape of the type side after normalising every run of +identifier characters to `X` and every digit run to `9`, so `[[]U8",` and +`[[]Bool",` are the same class and get counted once. + +Usage: + tri damage [corpus-dir] [--json PATH] [--emit-fixtures DIR] [--class SHAPE] + +Exit status: + 0 no damaged lines found + 1 damaged lines found +""" +import json +import os +import re +import sys + +IDENT = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") +DIGITS = re.compile(r"[0-9]+") +FIELD_LINE = re.compile(r"^\s*(?:pub\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*:\s*(.*)$") +# `r#"` opens a raw string that legitimately closes on a later line, so its lone +# quote is not damage. Found by reading the 3 lines the odd-quote signal flagged +# in specs/pins/parser.t27 rather than by trusting the signal. +RAW_STRING_OPEN = re.compile(r'\br#"') + + +def shape(rhs): + """Normalise a type side to its shape: identifiers -> X, digits -> 9.""" + s = IDENT.sub("X", rhs) + s = DIGITS.sub("9", s) + s = re.sub(r"\s+", " ", s).strip() + return s + + +def is_damaged(rhs): + """A type side that cannot be a type, with the reason. + + Two signals only, and the history of the third, fourth and fifth is the point + of this comment. + + The first draft of this function also flagged an unbalanced `<` or `>` and an + unbalanced `[`. Run over the corpus it reported 429 damaged lines. 230 of + those were `target : < 5000ns` -- a legitimate constraint bound where `<` is + the less-than operator, not a bracket. Another 17 were match arms + (`ConstraintType : :TEMPORAL => {`), 10 were the opening line of a multi-line + array literal, and 6 were function signatures that the field-line regex had + no business matching. So the headline 429 measured this function's regex, not + the corpus, and dropping the two bad signals is the fix -- not tuning a + threshold until the number looks right. + + What survives are the two signals that cannot occur in any well-formed type + annotation and cannot occur in a legitimate operator either: + + doubled-bracket `[[]` -- an empty slice inside a slice, which the language + has no syntax for + odd-quote an odd number of `"` on the line, so a string opens and + never closes + + Both are textual and never consult the parser: asking the parser whether its + own input was malformed is circular, since the malformed reading is exactly + what is in question. + """ + reasons = [] + if "[[]" in rhs or "[?[]" in rhs: + reasons.append("doubled-bracket") + if rhs.count('"') % 2 == 1 and not RAW_STRING_OPEN.search(rhs): + reasons.append("odd-quote") + return reasons + + +def scan(corpus): + rows = [] + for root, _dirs, names in os.walk(corpus): + for n in sorted(names): + if not n.endswith(".t27"): + continue + path = os.path.join(root, n) + try: + with open(path, "r", errors="replace") as fh: + lines = fh.readlines() + except OSError: + continue + for ln, line in enumerate(lines, 1): + m = FIELD_LINE.match(line.rstrip("\n")) + if not m: + continue + field, rhs = m.group(1), m.group(2).strip() + if not rhs: + continue + reasons = is_damaged(rhs) + if reasons: + rows.append({"file": path, "line": ln, "field": field, + "rhs": rhs, "shape": shape(rhs), + "reasons": reasons, + "family": os.path.dirname(path)}) + return rows + + +def main(argv): + args = [a for a in argv if not a.startswith("--")] + corpus = args[0] if args else "specs" + out_json = None + emit = None + only = None + for i, a in enumerate(argv): + if a == "--json" and i + 1 < len(argv): + out_json = argv[i + 1] + if a == "--emit-fixtures" and i + 1 < len(argv): + emit = argv[i + 1] + if a == "--class" and i + 1 < len(argv): + only = argv[i + 1] + + rows = scan(corpus) + if only: + rows = [r for r in rows if r["shape"] == only] + + files = sorted({r["file"] for r in rows}) + shapes = {} + for r in rows: + shapes.setdefault(r["shape"], []).append(r) + fams = {} + for r in rows: + fams[r["family"]] = fams.get(r["family"], 0) + 1 + + print(f"corpus: {corpus}") + print(f"damaged lines: {len(rows)} in {len(files)} files, " + f"{len(shapes)} distinct shapes\n") + print("by shape (this is the repair unit -- one reviewable diff per row):") + for sh, rs in sorted(shapes.items(), key=lambda kv: -len(kv[1])): + reasons = sorted({x for r in rs for x in r["reasons"]}) + print(f" {len(rs):5d} {sh!r:28s} {','.join(reasons)}") + print(f" e.g. {rs[0]['file']}:{rs[0]['line']} " + f"{rs[0]['field']} : {rs[0]['rhs']}") + print("\nby directory (top 12):") + for fam, n in sorted(fams.items(), key=lambda kv: -kv[1])[:12]: + print(f" {n:5d} {fam}") + + if out_json: + with open(out_json, "w") as fh: + json.dump({"corpus": corpus, "lines": len(rows), + "files": files, "shapes": {k: len(v) for k, v in shapes.items()}, + "rows": rows}, fh, indent=2) + print(f"\nwrote {out_json}") + + if emit: + os.makedirs(emit, exist_ok=True) + made = [] + for idx, (sh, rs) in enumerate( + sorted(shapes.items(), key=lambda kv: -len(kv[1])), 1): + r = rs[0] + name = f"damage_class_{idx:02d}" + body = (f"module {name}\n\n" + f"// shape: {sh}\n" + f"// {len(rs)} line(s) in the corpus share this shape\n" + f"// first seen: {r['file']}:{r['line']}\n" + f"// signals: {','.join(r['reasons'])}\n" + f"pub struct Damaged {{\n" + f" {r['field']} : {r['rhs']}\n" + f"}}\n") + p = os.path.join(emit, f"{name}.t27") + with open(p, "w") as fh: + fh.write(body) + made.append((p, sh, len(rs))) + print(f"\nemitted {len(made)} class fixtures into {emit}:") + for p, sh, n in made: + print(f" {p} ({n} lines, shape {sh!r})") + + print("\nNot claimed: that these are all the damage. This checks two textual") + print("signals on lines that look like field declarations. Damage that") + print("produces a balanced, quote-even, plausible-looking wrong type is") + print("invisible here, and nothing in this output bounds how much of that") + print("there is.") + return 1 if rows else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/damage_freeze.py b/scripts/tri_loop/damage_freeze.py new file mode 100644 index 000000000..e2530b8ba --- /dev/null +++ b/scripts/tri_loop/damage_freeze.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""tri damage-freeze -- freeze the corpus damage list before any repair touches it (#2154). + +A repair that has no frozen "before" is not a repair, it is an edit. This writes an +archive that pins every damaged line to a stable class identifier, so a later patch +can cite the class ID and the snapshot digest instead of asserting that it changed +the right thing. + +What is recorded per row, and why each field is needed: + + class_id stable across runs and across changes to the class SET. Derived from + sha256 of the normalised shape, NOT from frequency rank -- ranking + by count means adding one damaged line silently renumbers every + class, and every earlier citation of "class 04" then points at + something else. + file, line location + field the field name whose type side is damaged + rhs the damaged type side, VERBATIM. This is the archive: no repair may + delete a damaged line without this record existing first + before/after two lines of context each side, so the struct the field belongs to + is identifiable without reopening the file at a moved line number + file_sha256 digest of the whole file at freeze time. A patch applied to a file + whose digest no longer matches the snapshot is applied to a + different file than the one that was surveyed, and must refuse + + corpus_sha256 digest over the sorted (file, sha256) pairs of every damaged file, + so the snapshot as a whole has one number to cite + +Usage: + tri damage-freeze [corpus-dir] [--out PATH] + +This tool writes the snapshot only. It never edits a spec. +""" + +import hashlib +import json +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from damage import scan, shape # noqa: E402 (same directory, deliberate) + +CONTEXT = 2 + + +def class_id(shape_text): + """Stable ID for a damage class. + + Keyed on the shape text, so the ID of a class does not depend on how many + lines happen to be in it, nor on which other classes exist. + """ + return "DC-" + hashlib.sha256(shape_text.encode("utf-8")).hexdigest()[:8] + + +def file_digest(path): + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(65536), b""): + h.update(chunk) + return h.hexdigest() + + +def freeze(corpus): + rows = scan(corpus) + digests = {} + cache = {} + out = [] + for r in rows: + path = r["file"] + if path not in digests: + digests[path] = file_digest(path) + with open(path, "r", errors="replace") as fh: + cache[path] = fh.read().splitlines() + lines = cache[path] + i = r["line"] - 1 + out.append({ + "class_id": class_id(r["shape"]), + "shape": r["shape"], + "file": path, + "line": r["line"], + "field": r["field"], + "rhs": r["rhs"], + "raw_line": lines[i] if 0 <= i < len(lines) else None, + "context_before": lines[max(0, i - CONTEXT):i], + "context_after": lines[i + 1:i + 1 + CONTEXT], + "reasons": r["reasons"], + "file_sha256": digests[path], + }) + + # One number for the snapshot as a whole. Sorted so it does not depend on + # walk order. + agg = hashlib.sha256() + for path in sorted(digests): + agg.update(f"{path}:{digests[path]}\n".encode("utf-8")) + + classes = {} + for row in out: + c = classes.setdefault(row["class_id"], { + "class_id": row["class_id"], "shape": row["shape"], + "count": 0, "files": set(), "reasons": set(), + }) + c["count"] += 1 + c["files"].add(row["file"]) + c["reasons"].update(row["reasons"]) + for c in classes.values(): + c["files"] = sorted(c["files"]) + c["reasons"] = sorted(c["reasons"]) + + return { + "tool": "tri damage-freeze", + "corpus": corpus, + "lines": len(out), + "files": len(digests), + "classes": len(classes), + "corpus_sha256": agg.hexdigest(), + "file_sha256": {p: digests[p] for p in sorted(digests)}, + "class_index": [classes[k] for k in sorted(classes, key=lambda k: (-classes[k]["count"], k))], + "rows": out, + } + + +def main(argv): + args = [a for a in argv if not a.startswith("--")] + corpus = args[0] if args else "specs" + out_path = None + for i, a in enumerate(argv): + if a == "--out" and i + 1 < len(argv): + out_path = argv[i + 1] + if out_path is None: + out_path = "docs/corpus/damage_snapshot.json" + + snap = freeze(corpus) + + print(f"corpus: {snap['corpus']}") + print(f"damaged lines: {snap['lines']}") + print(f"damaged files: {snap['files']}") + print(f"classes: {snap['classes']}") + print(f"corpus_sha256: {snap['corpus_sha256']}\n") + print(f"{'class_id':12} {'n':>4} {'files':>5} shape") + print("-" * 74) + for c in snap["class_index"]: + print(f"{c['class_id']:12} {c['count']:4d} {len(c['files']):5d} {c['shape']!r}") + + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + with open(out_path, "w") as fh: + json.dump(snap, fh, indent=1, sort_keys=True) + fh.write("\n") + print(f"\nwrote {out_path}") + print("Class IDs are keyed on the shape text, so they survive a change to the") + print("class set. Cite the class_id and corpus_sha256, never a rank.") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/damage_repair.py b/scripts/tri_loop/damage_repair.py new file mode 100644 index 000000000..7197623a7 --- /dev/null +++ b/scripts/tri_loop/damage_repair.py @@ -0,0 +1,449 @@ +#!/usr/bin/env python3 +"""tri damage-repair -- one candidate patch per damage class, with the effect measured (#2154). + +Nothing here rewrites the corpus. A candidate is applied to a COPY, validated +twice, and printed as a reversible unified diff. Landing it is a separate, +human-reviewed act. + +## The mechanism, established from the corpus and not from memory + +An intact field in this corpus is `name : "TypeText",` -- the type side is a +quoted string. Every damaged line has the SAME single defect: the OPENING quote +of that string was replaced by `[`. + + intact children : "[4]?QuadNode", + damaged children : [[4]?QuadNode", + +which explains both signals `tri damage` detects at once: the doubled bracket +(`"[]T"` -> `[[]T"`) and the odd quote count (the opening one is gone). So the +candidate patch is one character, at a known offset, and inverting it is the same +operation in reverse. That is the entire repair for the classes where it works. + +## Why the classes split, and why the split is the load-bearing result + +Applying the substitution and asking whether the result is a closed string is a +decision procedure, not a guess: + + RESTORABLE `[[]Const u8",` -> `"[]Const u8",` a closed string. Nothing was + invented: every character of the type text survived the damage + and the repair only puts the delimiter back + + DESTROYED `[[]Const [,` -> `"[]Const [,` NOT a closed string. The type + text was also TRUNCATED at its second `[`, and the element type + that followed is simply gone. `[]Const [` cannot be completed + without deciding what the element type was -- `[]Const []Const u8` + is a plausible reading and so is `[]Const [N]u8`. There is no + evidence in the file either way + +The second group therefore gets `needs-human-language-decision` and NO patch. An +auto-repair there would be a guess wearing the costume of a fix, and it would be +unfalsifiable afterwards because the original is unrecoverable. This is the +difference between repairing a delimiter and inventing a type. + +## Double validation, and why parsing alone is not enough + +A file can be made to parse by deleting the offending line. So each applied +candidate is checked twice: + + 1. SYNTACTIC `t27c parse` exits 0 on the repaired file + 2. SEMANTIC the specific damaged field is present in the parsed field set + with a NON-EMPTY type text, and no previously-present field + disappeared + +Check 2 is the one that distinguishes a restored declaration from a merely +parseable file. Reported effect is one of: + + parse-restored both checks pass + still-malformed syntactic check fails + ambiguous parses, but the field did not come back, or a + different field vanished + needs-human-language-decision no patch was attempted, information destroyed + +Usage: + tri damage-repair [--snapshot PATH] [--binary PATH] [--class DC-xxxxxxxx] + [--diff] [--apply-to DIR] [--json PATH] + +`--apply-to` writes repaired copies into a scratch tree; specs/ is never touched. +""" + +import difflib +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from diffbin import parse_fields # noqa: E402 (same directory, deliberate) + +CLOSED_STRING = re.compile(r'^"[^"]*",?$') +DEFAULT_SNAPSHOT = "docs/corpus/damage_snapshot_2026-08-15.json" +TIMEOUT = 25 + +EFFECTS = ( + "parse-restored", + "still-malformed", + "ambiguous", + "needs-human-language-decision", +) + + +def candidate_rhs(rhs): + """The one-character candidate patch, or None when it is not applicable. + + Returns (repaired_rhs, restorable). restorable is False when the substitution + does not yield a closed string, which means the type text itself was truncated + and no patch is proposed. + """ + if not rhs.startswith("["): + return None, False + cand = '"' + rhs[1:] + return cand, bool(CLOSED_STRING.match(cand)) + + +def sha256_file(path): + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(65536), b""): + h.update(chunk) + return h.hexdigest() + + +def repair_lines(lines, rows): + """Apply the candidate to the given rows of one file. Returns (new_lines, applied). + + Only the type side is touched, and only its first character. The field name, + the indentation and the trailing comma are left byte-identical, so the diff is + one character per line and reads as such. + """ + new = list(lines) + applied = [] + for r in rows: + i = r["line"] - 1 + if not (0 <= i < len(new)): + continue + cand, restorable = candidate_rhs(r["rhs"]) + if not restorable: + continue + old = new[i] + # Locate the type side by its recorded text rather than by re-parsing the + # line, so a patch cannot drift onto a different construct. + if r["rhs"] not in old: + continue + new[i] = old.replace(r["rhs"], cand, 1) + applied.append(r) + return new, applied + + +def validate(binary, orig_path, repaired_text, rows_applied, tmpdir): + """Two checks. Returns (effect, detail).""" + scratch = os.path.join(tmpdir, "validate.t27") + with open(scratch, "w") as fh: + fh.write(repaired_text) + + base_status, base_fields = parse_fields(binary, orig_path, TIMEOUT) + cand_status, cand_fields = parse_fields(binary, scratch, TIMEOUT) + + if cand_status != "ok": + return "still-malformed", f"parse status {cand_status} after repair" + + base_map = dict(base_fields) + cand_map = dict(cand_fields) + + # Semantic check 1: every repaired field must now be present with a non-empty + # type. Names are qualified Struct.field by parse_fields, and the snapshot + # records the bare field name, so match on the suffix. + missing = [] + for r in rows_applied: + want = "." + r["field"] + hits = [(k, v) for k, v in cand_map.items() if k.endswith(want)] + if not hits or not any(v.strip() for _k, v in hits): + missing.append(r["field"]) + + # Semantic check 2: nothing that used to be there may vanish. A repair that + # restores one declaration by losing another is not a repair. + vanished = sorted(k for k in base_map if k not in cand_map) + + if missing or vanished: + bits = [] + if missing: + bits.append("field(s) did not come back: " + ", ".join(sorted(missing))) + if vanished: + bits.append("field(s) vanished: " + ", ".join(vanished[:6])) + return "ambiguous", "; ".join(bits) + + gained = sorted(k for k in cand_map if k not in base_map) + return "parse-restored", (f"parse ok; {len(rows_applied)} field(s) repaired; " + f"{len(gained)} declaration(s) newly visible" + + (": " + ", ".join(gained[:6]) if gained else "")) + + +def combined(snap, rows, binary, have_binary, tmpdir, apply_to, json_out): + """Repair every restorable row of a file at once, then validate per file. + + The per-class run answers "is this class's patch sound", and it answers it with + a confound: a file carrying damage from four classes still fails to parse after + one class is repaired, and the class is then blamed for a neighbour's defect. + That is what happened on the first run -- six classes came back still-malformed, + and the first guess (co-located UNRESTORABLE damage) was checked and found + false: none of those files contained a destroyed line. The actual cause was + co-located damage from OTHER RESTORABLE classes, left untouched by a + single-class run. + + So per-file is the unit that answers "what does the one rule achieve", and the + per-class run stays as the unit that answers "is one rule enough for this + shape". Both are reported; neither replaces the other. + """ + per_file = {} + for r in rows: + per_file.setdefault(r["file"], []).append(r) + + counts = dict.fromkeys(EFFECTS, 0) + out = [] + print("combined mode: every restorable row per file, validated per file\n") + for path in sorted(per_file): + frows = per_file[path] + with open(path, "r", errors="replace") as fh: + orig = fh.read() + new_lines, applied = repair_lines(orig.splitlines(), frows) + new_text = "\n".join(new_lines) + ("\n" if orig.endswith("\n") else "") + held = [r for r in frows if not candidate_rhs(r["rhs"])[1]] + if not applied: + eff, det = "needs-human-language-decision", \ + f"all {len(frows)} damaged line(s) are destroyed, none restorable" + elif have_binary: + eff, det = validate(binary, path, new_text, applied, tmpdir) + if held and eff != "parse-restored": + det += f"; {len(held)} destroyed line(s) remain in this file" + else: + eff, det = "ambiguous", "validation skipped: no binary" + counts[eff] += 1 + out.append({"file": path, "applied": len(applied), "held": len(held), + "effect": eff, "detail": det}) + if apply_to: + dest = os.path.join(apply_to, path) + os.makedirs(os.path.dirname(dest), exist_ok=True) + with open(dest, "w") as fh: + fh.write(new_text) + + for e in EFFECTS: + sel = [o for o in out if o["effect"] == e] + if not sel: + continue + print(f"--- {e}: {len(sel)} file(s) ---") + for o in sel[:8]: + print(f" {o['file']} (+{o['applied']} repaired, {o['held']} held) {o['detail'][:110]}") + if len(sel) > 8: + print(f" ... and {len(sel) - 8} more") + print() + + print("=" * 74) + print("per-file effect distribution (no aggregation across effects):") + for e in EFFECTS: + print(f" {e:32} {counts[e]:3d}") + lines_ok = sum(o["applied"] for o in out if o["effect"] == "parse-restored") + lines_held = sum(o["held"] for o in out) + print(f"\nrestorable lines inside parse-restored files: {lines_ok}") + print(f"lines held for a human language decision: {lines_held}") + print(f"snapshot total: {snap['lines']}") + print("\nNo spec under specs/ was modified by this run.") + if json_out: + with open(json_out, "w") as fh: + json.dump({"mode": "combined", "corpus_sha256": snap["corpus_sha256"], + "counts": counts, "files": out}, fh, indent=1, sort_keys=True) + fh.write("\n") + print(f"wrote {json_out}") + return 0 + + +def main(argv): + snapshot = DEFAULT_SNAPSHOT + binary = "/tmp/t27c.fixed" + only = None + apply_to = None + json_out = None + want_diff = "--diff" in argv + for i, a in enumerate(argv): + if a == "--snapshot" and i + 1 < len(argv): + snapshot = argv[i + 1] + elif a == "--binary" and i + 1 < len(argv): + binary = argv[i + 1] + elif a == "--class" and i + 1 < len(argv): + only = argv[i + 1] + elif a == "--apply-to" and i + 1 < len(argv): + apply_to = argv[i + 1] + elif a == "--json" and i + 1 < len(argv): + json_out = argv[i + 1] + + if not os.path.exists(snapshot): + print(f"no snapshot at {snapshot}", file=sys.stderr) + print("run: tri damage-freeze specs --out " + DEFAULT_SNAPSHOT, file=sys.stderr) + return 2 + snap = json.load(open(snapshot)) + + rows = snap["rows"] + if only: + rows = [r for r in rows if r["class_id"] == only] + if not rows: + print(f"no rows for class {only}", file=sys.stderr) + return 2 + + # A patch applied to a file that has changed since the freeze is applied to a + # different file than the one that was surveyed. Refuse rather than proceed. + stale = [] + for r in rows: + p = r["file"] + if not os.path.exists(p): + stale.append((p, "missing")) + elif sha256_file(p) != r["file_sha256"]: + stale.append((p, "digest differs from snapshot")) + stale = sorted(set(stale)) + if stale: + print("REFUSING: the corpus moved since the snapshot was frozen.") + for p, why in stale[:10]: + print(f" {p} ({why})") + print("\nRe-freeze, review the difference, and only then repair.") + return 1 + + by_class = {} + for r in rows: + by_class.setdefault(r["class_id"], []).append(r) + + tmpdir = "/tmp/tri_damage_repair" + os.makedirs(tmpdir, exist_ok=True) + if apply_to: + os.makedirs(apply_to, exist_ok=True) + + have_binary = os.path.exists(binary) + results = [] + counts = dict.fromkeys(EFFECTS, 0) + + print(f"snapshot: {snapshot}") + print(f"corpus_sha256: {snap['corpus_sha256']}") + print(f"binary: {binary}" + ("" if have_binary else " [MISSING -- validation skipped]")) + print(f"classes: {len(by_class)}\n") + + if "--combined" in argv: + return combined(snap, rows, binary, have_binary, tmpdir, apply_to, json_out) + + for cid in sorted(by_class, key=lambda c: (-len(by_class[c]), c)): + crows = by_class[cid] + shape = crows[0]["shape"] + _cand, restorable = candidate_rhs(crows[0]["rhs"]) + + if not restorable: + effect = "needs-human-language-decision" + counts[effect] += 1 + reading = [] + for r in crows[:3]: + reading.append(f"{r['file']}:{r['line']} {r['field']} : {r['rhs']}") + results.append({"class_id": cid, "shape": shape, "lines": len(crows), + "files": sorted({r["file"] for r in crows}), + "effect": effect, "patch": None, + "detail": ("the type text is truncated as well as unquoted; the element " + "type is not recoverable from the file"), + "owner": "language owner", + "decision_criterion": ( + "state what an unclosed element type in a slice position means, and " + "whether the corpus may carry a placeholder; only then can a value " + "be written in without guessing")}) + print(f"{cid} n={len(crows):3d} {shape!r}") + print(f" effect: {effect}") + print(" no patch proposed. Substituting the delimiter yields " + f"{_cand!r}, which is not a closed string:") + print(" the type text was truncated too, so a repair would have to invent the") + print(" element type. owner: language owner") + for line in reading: + print(f" {line}") + print() + continue + + # Group by file, apply, validate, diff. + per_file = {} + for r in crows: + per_file.setdefault(r["file"], []).append(r) + + effects_seen = [] + details = [] + diffs = [] + for path, frows in sorted(per_file.items()): + with open(path, "r", errors="replace") as fh: + orig = fh.read() + lines = orig.splitlines() + new_lines, applied = repair_lines(lines, frows) + new_text = "\n".join(new_lines) + ("\n" if orig.endswith("\n") else "") + if not applied: + effects_seen.append("ambiguous") + details.append(f"{path}: recorded text not found at the recorded line") + continue + diff = list(difflib.unified_diff( + orig.splitlines(keepends=True), new_text.splitlines(keepends=True), + fromfile=f"a/{path}", tofile=f"b/{path}", n=1)) + diffs.append("".join(diff)) + if have_binary: + eff, det = validate(binary, path, new_text, applied, tmpdir) + else: + eff, det = "ambiguous", "validation skipped: no binary" + effects_seen.append(eff) + details.append(f"{path}: {det}") + if apply_to: + dest = os.path.join(apply_to, path) + os.makedirs(os.path.dirname(dest), exist_ok=True) + with open(dest, "w") as fh: + fh.write(new_text) + + # A class is only as good as its worst file. + for e in EFFECTS: + if e in effects_seen: + effect = e + break + else: + effect = "ambiguous" + counts[effect] += 1 + + results.append({"class_id": cid, "shape": shape, "lines": len(crows), + "files": sorted(per_file), + "effect": effect, + "patch": "replace the leading '[' of the type side with '\"'", + "reversible": True, + "per_file_effects": effects_seen, + "detail": details}) + print(f"{cid} n={len(crows):3d} {shape!r}") + print(f" patch: replace the leading '[' of the type side with '\"' (1 char, invertible)") + print(f" effect: {effect} [{effects_seen.count('parse-restored')}/{len(effects_seen)} files parse-restored]") + for d in details[:3]: + print(f" {d}") + if len(details) > 3: + print(f" ... and {len(details) - 3} more file(s)") + if want_diff and diffs: + print(" --- candidate diff (first file) ---") + for line in diffs[0].splitlines()[:12]: + print(" " + line) + print() + + print("=" * 74) + print("effect distribution over classes (no aggregation across effects):") + for e in EFFECTS: + print(f" {e:32} {counts[e]:3d}") + restor = sum(r["lines"] for r in results if r["effect"] == "parse-restored") + human = sum(r["lines"] for r in results if r["effect"] == "needs-human-language-decision") + print(f"\nlines covered by a validated candidate: {restor}") + print(f"lines held for a human language decision: {human}") + print(f"lines total in snapshot: {snap['lines']}") + print("\nNo spec under specs/ was modified by this run.") + + if json_out: + with open(json_out, "w") as fh: + json.dump({"snapshot": snapshot, "corpus_sha256": snap["corpus_sha256"], + "binary": binary, "counts": counts, "classes": results}, + fh, indent=1, sort_keys=True) + fh.write("\n") + print(f"wrote {json_out}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/diffbin.py b/scripts/tri_loop/diffbin.py new file mode 100644 index 000000000..4f7217c41 --- /dev/null +++ b/scripts/tri_loop/diffbin.py @@ -0,0 +1,520 @@ +#!/usr/bin/env python3 +"""tri diffbin -- compare two t27c binaries over a spec corpus, by category. + +Why this exists, and why it is a rewrite rather than a restore. + +An earlier version of this script was written, used to produce a "634 specs, 0 +regressions" figure, and lost: it was never committed to any branch, and the +sandbox holding the working copy was re-cloned. Six recovery routes were checked +and all came back empty -- dangling git objects (only a `git stash` WIP holding +`triage.py`), the reflog (which shows the clone, i.e. the loss), shell history +(absent), CI artifacts (only FPGA build outputs), PR and issue comments (no +pasted source), and the session snapshot (prose about the script, not the +script). So this is a reimplementation from an explicit written contract, not a +reconstruction from memory of what the old one did. + +That is the better outcome anyway, because the old aggregate was wrong. + +The defect in the previous tool. + +It reported "0 regressions" over 634 specs. It was also true that 17 files lost +struct fields: on input like `a : Map [corpus-dir] [--limit N] + [--jsonl PATH] [--timeout SEC] [--include-scratch] + [--exclude-status STATUS.json] + +Exit status: + 0 no field-loss and no unknown (coverage is printed, not folded in) + 1 at least one field-loss or unknown file + 2 usage or setup error, or an internal partition/reason-code violation +""" +import json +import os +import subprocess +import sys +from collections import Counter + +CATEGORIES = ( + "unchanged", + "field-loss", + "strict-improvement", + "malformed-input-tradeoff", + "unknown", + "not-evaluated", +) + +# `not-evaluated` exists because `unchanged` was carrying two different +# statements. Measured 2026-08-15 over the 634-spec library: of 616 files called +# `unchanged`, only 330 were 'both parsed, field sets identical' -- the other 286 +# were 'both error', i.e. neither binary produced a verdict. The historical +# headline '634 specs, 0 regressions' therefore rested on a measured base of 330 +# files, 52 % of the corpus, with the remaining 48 % counted as agreement because +# both sides were silent. +# +# That is not a reporting blemish. It changes the meaning of every differential +# total ever quoted from this tool: an absence of evidence was printed as evidence +# of absence. +# +# Every not-evaluated file MUST carry one of these codes. A bare `not-evaluated` +# count is not admissible -- 'no verdict' for six different reasons is six +# different facts, and only some of them are about the compiler at all. +NOT_EVALUATED_REASONS = ( + "both-error", # neither binary parsed the file; no verdict exists + "base-timeout", # baseline hit the wall clock; a property of the run + "candidate-timeout", # candidate hit the wall clock; likewise + "environment-failure", # the binary could not be spawned at all + "excluded-source-loss", # caller excluded it: the declaring text is gone + "other", # must be accompanied by explanatory text +) + +# A timeout is a property of the RUN, not of the file (LOOP-RULES R14). Measured +# 2026-08-15 on specs/scratch at a 12 s threshold: 26 files moved ok -> timeout +# between two runs and 0 moved back, and all 26 already sat between 9322 ms and +# 11952 ms against a 12000 ms wall. So the shift measured machine load at the +# boundary, not the compiler. Timeouts are kept out of every substantive category +# for that reason, and two runs at different thresholds must never be joined. +DEFAULT_TIMEOUT = 25 + + +def path_key(p): + """Normalise a path to its repo-relative tail from `specs/`. + + Exclusion lists come from `tri corpus-status`, which may have been produced + in a different worktree or against a repaired tree. Joining on the raw string + would silently fail to exclude anything, and a silent non-exclusion is the + worst outcome here: the caller would believe files were held out when they + were not. + """ + p = p.replace("\\", "/") + i = p.find("specs/") + return p[i:] if i >= 0 else p + + +def _unquote(line, key): + t = line.strip() + if not t.startswith(key): + return None + rest = t[len(key):].strip() + if not rest.startswith('"'): + return None + rest = rest[1:] + end = rest.rfind('"') + if end < 0: + return None + return rest[:end] + + +def parse_fields(binary, path, timeout): + """Return (status, [(qualified_name, type_text)]). + + status is one of ok / error / timeout. On error or timeout the field list is + empty, and the categoriser treats a status change as its own outcome rather + than reading an empty list as "all fields lost". + + Only an ExprIdentifier whose PARENT node is a StructDecl counts as a struct + field. `t27c parse` prints the same node kind for every identifier in the + tree, including every identifier in a function body, so matching on the kind + alone would count expression operands as fields and make the corpus totals + meaningless. Parenthood is read off the indentation of the `kind:` lines: the + parent of a node is the nearest preceding `kind:` line with strictly smaller + indentation. Names are qualified with the struct name, so that two structs in + one file each declaring `value` stay distinguishable. + """ + try: + r = subprocess.run([binary, "parse", path], capture_output=True, + timeout=timeout, text=True, errors="replace") + except subprocess.TimeoutExpired: + return "timeout", [] + except OSError as e: + return f"spawn-error: {e}", [] + if r.returncode != 0: + return "error", [] + lines = (r.stdout + r.stderr).splitlines() + out = [] + stack = [] # (indent, kind, name) + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped.startswith("kind: "): + continue + indent = len(line) - len(line.lstrip()) + kind = stripped[len("kind: "):].rstrip(",") + name = _unquote(lines[i + 1], "name:") if i + 1 < len(lines) else None + while stack and stack[-1][0] >= indent: + stack.pop() + parent = stack[-1] if stack else None + if kind == "ExprIdentifier" and parent and parent[1] == "StructDecl": + ty = _unquote(lines[i + 3], "extra_type:") if i + 3 < len(lines) else None + if name is not None and ty is not None: + out.append((f"{parent[2]}.{name}", ty)) + stack.append((indent, kind, name or "")) + return "ok", out + + +def truncated(ty): + """A type text with an unclosed bracket, angle or paren.""" + return (ty.count("<") > ty.count(">") + or ty.count("(") > ty.count(")") + or ty.count("[") > ty.count("]")) + + +def input_is_malformed(path): + """Decide malformed-ness from the SOURCE, never from parser behaviour. + + Deciding it from the parser would make the categoriser circular: any change + in output would justify itself as "the input must have been malformed". + These are textual signals only, and each one is a thing that cannot appear in + a well-formed field declaration: + + * an unbalanced `"` on a field line + * `[[` immediately before `]` + * a field line whose type side has more openers than closers + """ + try: + with open(path, "r", errors="replace") as fh: + lines = fh.readlines() + except OSError: + return False, "unreadable" + signals = [] + for n, line in enumerate(lines, 1): + if ":" not in line: + continue + if line.count('"') % 2 == 1: + signals.append(f"{n}:odd-quote") + continue + if "[[]" in line: + signals.append(f"{n}:doubled-bracket") + continue + rhs = line.split(":", 1)[1] + if truncated(rhs.rstrip().rstrip(",")): + signals.append(f"{n}:unclosed-type") + return bool(signals), ",".join(signals[:6]) + + +def categorise(base, cand, path, excluded=None): + """Return (category, reason, detail dict). + + The six categories are mutually exclusive and exhaust the corpus by + construction: every file leaves this function with exactly one of them, and + the caller asserts that the counts sum to the file total. + + Order of decision, and why it is this order: + + 1. caller-declared exclusion the declaring text is gone from the source, so + nothing about the compiler can be read from + the file in either direction + 2. spawn failure the tool did not run; not a result + 3. any timeout a property of the run, not of the file (R14) + 4. both failed to parse neither side took a verdict (R13) + 5. one-sided parse change a real behavioural difference + 6. field-set comparison the substantive comparison + + Exclusion is tested first on purpose. Testing it later would allow a file to + be reported as a compiler regression when the field it appears to have lost + was never present in the source the compiler read. + """ + bstat, bfields = base + cstat, cfields = cand + + if excluded and path_key(path) in excluded: + return ("not-evaluated", + "excluded-source-loss: caller excluded this file, the declaring " + "text is absent from the source", {}) + + if bstat.startswith("spawn-error") or cstat.startswith("spawn-error"): + return ("not-evaluated", + f"environment-failure: base={bstat} candidate={cstat}", {}) + + # Each side gets its own timeout code. Folding them together would hide which + # binary hit the wall, and that is the only part of a timeout that could ever + # point at a real slowdown. + if bstat == "timeout" and cstat == "timeout": + return ("not-evaluated", + "base-timeout: both binaries hit the wall clock " + "(candidate-timeout applies equally)", {}) + if bstat == "timeout": + return ("not-evaluated", + f"base-timeout: baseline hit the wall clock, candidate={cstat}", + {}) + if cstat == "timeout": + return ("not-evaluated", + f"candidate-timeout: candidate hit the wall clock, base={bstat}. " + "Not a field loss, and not evidence of a slowdown by itself: " + "check the baseline time against the threshold first (R14)", {}) + + if bstat == "error" and cstat == "error": + return ("not-evaluated", + "both-error: neither binary parsed the file, so no verdict on " + "field behaviour exists. Repairing such a file moves it out of " + "this bucket, and a pre-existing divergence then becomes visible " + "for the first time -- that is not a new regression (R13)", {}) + + if bstat != cstat: + if bstat == "ok" and cstat == "error": + # A refusal is an answer, not a missing verdict, and it makes every + # declared field unavailable. So this stays a loss. + return ("field-loss", + "base parsed, candidate refused the file: every declared " + "field is unavailable", {}) + if bstat == "error" and cstat == "ok": + return "strict-improvement", "base error, candidate parsed", {} + return "unknown", f"status {bstat} -> {cstat}", {} + + if bfields == cfields: + return "unchanged", "identical field set (both parsed)", {} + + bnames = [n for n, _ in bfields] + cnames = [n for n, _ in cfields] + btypes = dict(bfields) + removed = [n for n in bnames if n not in cnames] + added = [n for n in cnames if n not in bnames] + + phantoms = [n for n in removed if btypes.get(n, "") == ""] + real_lost = [n for n in removed if btypes.get(n, "") != ""] + + detail = {"removed": removed, "added": added, + "phantoms": phantoms, "real_lost": real_lost, + "base_n": len(bfields), "cand_n": len(cfields), + "base_trunc": sum(1 for _, t in bfields if truncated(t)), + "cand_trunc": sum(1 for _, t in cfields if truncated(t))} + + if real_lost: + return ("field-loss", + f"declared field(s) gone with non-empty base type: {real_lost}", + detail) + + if phantoms: + return ("strict-improvement", + f"phantom field(s) removed (empty base type): {phantoms}", + detail) + + if detail["cand_trunc"] < detail["base_trunc"]: + return ("strict-improvement", + f"truncated type texts {detail['base_trunc']} -> {detail['cand_trunc']}", + detail) + + malformed, signals = input_is_malformed(path) + if malformed: + return ("malformed-input-tradeoff", + f"field set moved on malformed input ({signals})", detail) + + return ("unknown", + "field set moved on well-formed input with no loss and no improvement", + detail) + + +def main(argv): + args = [a for a in argv if not a.startswith("--")] + if len(args) < 2: + print(__doc__.split("Usage:")[1].strip(), file=sys.stderr) + return 2 + base_bin, cand_bin = args[0], args[1] + corpus = args[2] if len(args) > 2 else "specs" + for b in (base_bin, cand_bin): + if not os.path.isfile(b) or not os.access(b, os.X_OK): + print(f"not an executable: {b}", file=sys.stderr) + return 2 + + limit = None + jsonl = None + timeout = DEFAULT_TIMEOUT + exclude_file = None + include_scratch = "--include-scratch" in argv + for i, a in enumerate(argv): + if a == "--limit" and i + 1 < len(argv): + limit = int(argv[i + 1]) + if a == "--jsonl" and i + 1 < len(argv): + jsonl = argv[i + 1] + if a == "--timeout" and i + 1 < len(argv): + timeout = int(argv[i + 1]) + if a == "--exclude-status" and i + 1 < len(argv): + exclude_file = argv[i + 1] + + # An exclusion list must be declared, never inferred. The file is a + # `tri corpus-status --out` JSON; files whose status is + # `unrecoverable-source-loss` are held out under the reason code + # `excluded-source-loss`. They are still counted and still printed -- holding + # a file out of the substantive comparison is not the same as dropping it. + excluded = set() + if exclude_file: + try: + with open(exclude_file) as fh: + st = json.load(fh) + for r in st.get("rows", []): + if r.get("status") == "unrecoverable-source-loss": + excluded.add(path_key(r["path"])) + except (OSError, ValueError, KeyError) as e: + print(f"cannot read --exclude-status {exclude_file}: {e}", + file=sys.stderr) + return 2 + if not excluded: + # Refuse to proceed quietly. An exclusion list that excludes nothing + # is almost always a path-join failure, and it would produce a report + # claiming files were held out when none were. + print(f"--exclude-status {exclude_file} selected 0 files; refusing to " + f"report an exclusion that excludes nothing", file=sys.stderr) + return 2 + + files = [] + for root, _dirs, names in os.walk(corpus): + if not include_scratch and f"{os.sep}scratch" in root: + continue + for n in sorted(names): + if n.endswith(".t27"): + files.append(os.path.join(root, n)) + files.sort() + if limit: + files = files[:limit] + if not files: + print(f"no .t27 files under {corpus}", file=sys.stderr) + return 2 + + counts = Counter() + ne_codes = Counter() + rows = [] + fh = open(jsonl, "w") if jsonl else None + for path in files: + b = parse_fields(base_bin, path, timeout) + c = parse_fields(cand_bin, path, timeout) + cat, reason, detail = categorise(b, c, path, excluded) + counts[cat] += 1 + if cat == "not-evaluated": + code = reason.split(":", 1)[0] + if code not in NOT_EVALUATED_REASONS: + # Fail loudly rather than emit an uncoded not-evaluated row. An + # uncoded row is exactly the ambiguity this category was created + # to remove. + print(f"internal: not-evaluated without a declared reason code " + f"({code!r}) on {path}", file=sys.stderr) + return 2 + ne_codes[code] += 1 + row = {"file": path, "category": cat, "reason": reason, + "reason_code": (reason.split(":", 1)[0] + if cat == "not-evaluated" else None), + **detail} + rows.append(row) + if fh: + fh.write(json.dumps(row) + "\n") + fh.flush() + if fh: + fh.close() + + print(f"corpus: {len(files)} specs under {corpus}" + f"{'' if include_scratch else ' (scratch excluded)'}") + print(f"base: {base_bin}") + print(f"candidate: {cand_bin}\n") + total = len(files) + ne = counts.get("not-evaluated", 0) + measured = total - ne + + # Partition assertion. The six categories are claimed to be mutually + # exclusive and exhaustive; a claim of that shape must be checked rather than + # asserted in a docstring. + s = sum(counts.get(c, 0) for c in CATEGORIES) + if s != total: + print(f"internal: categories sum to {s} but the corpus has {total} files", + file=sys.stderr) + return 2 + + for cat in CATEGORIES: + print(f" {counts.get(cat, 0):5d} {cat}") + + if ne: + print("\n not-evaluated by reason code (each is a different fact):") + for code in NOT_EVALUATED_REASONS: + if ne_codes.get(code): + print(f" {ne_codes[code]:5d} {code}") + + pct = (100.0 * measured / total) if total else 0.0 + print(f"\nMEASURED COVERAGE: {measured}/{total} = {pct:.1f}% of the corpus") + print(f" {ne} file(s) yielded no verdict and are NOT counted as agreement.") + print(" Any sentence of the form 'no regressions' is admissible only with") + print(" this coverage figure attached, and only when field-loss = 0 and") + print(" unknown = 0. Coverage below 100% bounds what the run can claim.") + + for cat in ("field-loss", "unknown", "strict-improvement"): + sel = [r for r in rows if r["category"] == cat] + if not sel: + continue + print(f"\n{cat} ({len(sel)}):") + for r in sel[:40]: + print(f' {r["file"]}') + print(f' {r["reason"]}') + if len(sel) > 40: + print(f" ... {len(sel) - 40} more (use --jsonl for the full list)") + + lost = counts.get("field-loss", 0) + unknown = counts.get("unknown", 0) + print() + if unknown: + # An unknown is a file whose field set moved on well-formed input with + # neither loss nor improvement. No verdict can be issued over a corpus + # containing one, whatever the other counts say. + print(f"NO VERDICT: {unknown} unknown file(s). A PASS is not available " + f"while any file is unexplained,") + print("regardless of the field-loss count. Explain them or classify them.") + if lost or unknown: + print(f"NOT CLEAN: {lost} field-loss, {unknown} unknown.") + print("These are NOT to be aggregated away. field-loss means a field the") + print("author declared is gone from the parse. It may still be the right") + print("trade to make -- that is a language decision about what malformed") + print("input should mean -- but it is a decision, not a measurement, and") + print("it does not belong inside a count of zero.") + else: + print(f"CLEAN on the measured {measured}/{total} ({pct:.1f}%): " + f"no field-loss and no unknown.") + print("Scope of that statement, stated so it travels with the number:") + print(f" * it covers the {measured} file(s) on which both binaries gave a") + print(" verdict, and says nothing about the rest;") + print(" * it says nothing about categories the tool does not check --") + print(" generated code, type inference, diagnostics, or timing.") + return 1 if (lost or unknown) else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/diffmodes.py b/scripts/tri_loop/diffmodes.py new file mode 100644 index 000000000..2c110612e --- /dev/null +++ b/scripts/tri_loop/diffmodes.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +"""tri diffmodes -- print a differential in two modes: full historical corpus and +clean/evaluable corpus, joined on per-file status. + +Why a separate tool rather than a flag on diffbin. + +`diffbin` answers one question: for this pair of binaries, what happened to each +file. It has no notion of why a file is in the state it is in -- whether its +fields are missing because the compiler dropped them or because the source line +was physically truncated before the compiler ever saw it. That second question is +`corpus-status`'s. Joining them here keeps each tool answerable for one claim, and +keeps this join auditable: it consumes two recorded artifacts and adds nothing of +its own beyond the join. + +What the two modes are for. + +The FULL mode is the historical record: every file the differential ever covered, +in the same five categories, with nothing removed. It is the number that must be +comparable against previous ticks. + +The CLEAN/EVALUABLE mode restricts to files whose status is one the differential +can actually speak about -- `clean`, `repaired-by-mechanical-rule`, +`parser-defect` -- and excludes `unrecoverable-source-loss` and `not-evaluated`. + +The point of the restriction is NOT to make the number smaller. It is that a +field cannot be reported as lost by the compiler when the text declaring it was +already gone from the file: that is a measurement of the corpus, attributed to the +compiler. The exclusion narrows what is being claimed; it does not improve it. + +So the excluded set is printed in full, never hidden, and never summarised as a +count alone. An excluded file that is not named has been made to disappear rather +than accounted for. If the two modes disagree, the disagreement is the finding. + +Usage: + tri diffmodes --jsonl --status + [--out ] + +Exit codes: + 0 clean/evaluable mode has zero field-loss and zero unknown + 1 otherwise, or on malformed input +""" +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +CATEGORIES = ( + "unchanged", + "field-loss", + "strict-improvement", + "malformed-input-tradeoff", + "unknown", +) + +# Statuses the differential is entitled to speak about. A file whose declared +# type text was truncated in the source is NOT here: no statement about compiler +# behaviour can be extracted from it, in either direction. +EVALUABLE = ("clean", "repaired-by-mechanical-rule", "parser-defect") +EXCLUDED = ("unrecoverable-source-loss", "unrelated-parse-failure", "not-evaluated") + + +def load_jsonl(p: Path) -> list[dict]: + rows = [] + for ln, line in enumerate(p.read_text().splitlines(), 1): + line = line.strip() + if not line: + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError as e: + print(f"malformed JSONL at {p}:{ln}: {e}") + sys.exit(1) + return rows + + +def norm(path: str) -> str: + # diffbin and corpus-status may disagree on leading "./" or on being rooted + # in different worktrees. Join on the repo-relative tail starting at specs/. + p = path.replace("\\", "/") + i = p.find("specs/") + return p[i:] if i >= 0 else p + + +def dist(rows: list[dict]) -> dict[str, int]: + d = {c: 0 for c in CATEGORIES} + for r in rows: + c = r.get("category") + if c in d: + d[c] += 1 + else: + d.setdefault("(unrecognised category)", 0) + d["(unrecognised category)"] += 1 + return d + + +def split_unchanged(rows: list[dict]) -> tuple[int, int]: + """Split `unchanged` into (both parsed alike, both failed). + + Measured 2026-08-15, and the reason this split exists at all. Three files -- + tri/encoding/mime.t27, tri/search/aho_corasick.t27, tri/trees/quadtree.t27 -- + were `unchanged` before the mechanical repair and `field-loss` after it. The + repair did not introduce the loss; it made the loss observable. Before it, + BOTH binaries failed to parse the file, `diffbin` recorded reason `both + error`, and that landed in `unchanged`. + + So `unchanged` was carrying two different statements under one label: + + both parsed, field sets identical -> a measurement: no difference + both failed to parse -> no measurement was taken at all + + The second is the differential's own `not-evaluated`. Counting it as + `unchanged` is the same error R1 was written against: an absence of evidence + printed as evidence of absence. It also makes a repair look like a + regression, because repairing a file moves it out of the silent bucket and + any pre-existing divergence appears for the first time. + + This function does not change any category. It only reports how much of + `unchanged` is a measurement, so that a later tick comparing aggregates + across corpora is comparing like with like. + """ + parsed = failed = 0 + for r in rows: + if r.get("category") != "unchanged": + continue + if "both error" in str(r.get("reason", "")): + failed += 1 + else: + parsed += 1 + return parsed, failed + + +def print_dist(title: str, rows: list[dict], total_note: str) -> dict[str, int]: + d = dist(rows) + print(f"\n{title}") + print(f" {len(rows)} file(s) -- {total_note}") + for c in CATEGORIES: + print(f" {c:<28}{d.get(c, 0):>6}") + extra = {k: v for k, v in d.items() if k not in CATEGORIES} + for k, v in extra.items(): + print(f" {k:<28}{v:>6}") + parsed, failed = split_unchanged(rows) + if failed: + print(f" of which unchanged is:") + print(f" both parsed, identical {parsed:>6} (a measurement)") + print(f" both failed to parse {failed:>6} (NOT a measurement:" + f" no verdict was taken)") + return d + + +def main(argv: list[str]) -> int: + ap = argparse.ArgumentParser(prog="tri diffmodes", description=__doc__) + ap.add_argument("--jsonl", required=True, help="diffbin --jsonl output") + ap.add_argument("--status", required=True, help="corpus-status --out output") + ap.add_argument("--out", help="write the joined report as JSON") + args = ap.parse_args(argv) + + jp, sp = Path(args.jsonl), Path(args.status) + for p in (jp, sp): + if not p.exists(): + print(f"missing input: {p}") + return 1 + + rows = load_jsonl(jp) + st = json.loads(sp.read_text()) + status_of = {norm(r["path"]): r["status"] for r in st.get("rows", [])} + + joined, unjoined = [], [] + for r in rows: + key = norm(r.get("file", "")) + s = status_of.get(key) + if s is None: + unjoined.append(key) + else: + r = dict(r) + r["status"] = s + joined.append(r) + + print("=" * 72) + print("differential in two modes") + print("=" * 72) + print(f"diffbin rows : {len(rows)} ({jp})") + print(f"status rows : {len(status_of)} ({sp})") + print(f"joined : {len(joined)}") + print(f"in diff, no status : {len(unjoined)}") + if unjoined: + # This is not a rounding difference to be waved past. A file the + # differential measured but the status pass never classified has no + # definite status, which is exactly the condition tick C set out to + # remove. + print(" These files were measured but never classified. They are NOT") + print(" silently dropped from either mode; they are listed here and") + print(" counted in FULL only:") + for k in unjoined[:40]: + print(f" {k}") + if len(unjoined) > 40: + print(f" ... {len(unjoined) - 40} more") + + full = print_dist( + "MODE 1 -- FULL HISTORICAL CORPUS", + rows, + "every file the differential covered, nothing removed", + ) + + ev = [r for r in joined if r["status"] in EVALUABLE] + clean = print_dist( + "MODE 2 -- CLEAN / EVALUABLE CORPUS", + ev, + "status in " + ", ".join(EVALUABLE), + ) + + print("\nEXCLUDED FROM MODE 2 -- shown, not hidden") + by_status: dict[str, list[dict]] = {} + for r in joined: + if r["status"] not in EVALUABLE: + by_status.setdefault(r["status"], []).append(r) + if not by_status: + print(" (none)") + for s in EXCLUDED: + sel = by_status.get(s, []) + if not sel: + continue + d = dist(sel) + nonzero = ", ".join(f"{c}={d[c]}" for c in CATEGORIES if d[c]) + print(f" {s}: {len(sel)} file(s) [{nonzero or 'all zero'}]") + # Name every excluded file that the differential flagged as a change. + # A count alone lets an exclusion absorb a real regression. + flagged = [r for r in sel if r["category"] in ("field-loss", "unknown")] + for r in flagged: + print(f" {r['category']:<12}{norm(r['file'])}") + if not flagged: + print(" (no field-loss or unknown among them)") + for s, sel in sorted(by_status.items()): + if s in EXCLUDED: + continue + print(f" {s}: {len(sel)} file(s) (status not in the declared " + f"excluded set -- check the status vocabulary)") + + fl, unk = clean.get("field-loss", 0), clean.get("unknown", 0) + print("\n" + "-" * 72) + print("the three required figures") + print("-" * 72) + print(f" loss on the clean/evaluable corpus : {fl}") + excl_fl = sum(dist(sel).get("field-loss", 0) + for s, sel in by_status.items() if s not in EVALUABLE) + print(f" loss on excluded statuses (shown above) : {excl_fl}") + print(f" unexplained changes (unknown, mode 2) : {unk}") + print("\n Read this as a scoped claim, not a clearance. Mode 2 says: on the") + print(" files whose state is accounted for, the candidate lost no declared") + print(" field. It says nothing about the excluded files, and an exact") + print(" correlation on this corpus is not causation beyond this corpus.") + + if args.out: + Path(args.out).write_text(json.dumps({ + "jsonl": str(jp), "status": str(sp), + "mode_full": full, "mode_clean": clean, + "evaluable_statuses": list(EVALUABLE), + "excluded": {s: dist(sel) for s, sel in by_status.items()}, + "excluded_files": {s: [norm(r["file"]) for r in sel] + for s, sel in by_status.items()}, + "in_diff_without_status": unjoined, + "required": {"clean_field_loss": fl, + "excluded_field_loss": excl_fl, + "clean_unknown": unk}, + }, indent=2) + "\n") + print(f"\nwrote {args.out}") + + return 0 if (fl == 0 and unk == 0) else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/loop_rules.py b/scripts/tri_loop/loop_rules.py new file mode 100644 index 000000000..74a0b2237 --- /dev/null +++ b/scripts/tri_loop/loop_rules.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +"""tri loop-rules -- verify the loop-rule file against its recorded checksum. + +Why this exists. The durable rules of the hourly tick used to live inside one +long scheduled-task field. That field is silently truncated: 13277 characters +were submitted and 10069 stored, with no diagnostic. A rule that disappears +without a report is worse than a rule never written, because the tick keeps +behaving as if it were in force. + +So the rules live in docs/loop/LOOP-RULES.md under version control, and their +digest lives in docs/loop/LOOP-RULES.sha256. The scheduled task keeps only a +pointer plus the expected digest, so a truncated field still names where the +rules are and which version was expected. + +What this tool does and does not claim. It certifies that the rule text is +byte-identical to the text that was sealed. It says nothing about whether the +rules are good, nor whether the tick followed them. Identity, not correctness -- +the same distinction as the compiler seal (R6). + +Exit codes: + 0 digest matches + 1 digest differs, or the rule file / seal file is missing +""" +from __future__ import annotations + +import argparse +import hashlib +import subprocess +import sys +from pathlib import Path + +RULES = Path("docs/loop/LOOP-RULES.md") +SEAL = Path("docs/loop/LOOP-RULES.sha256") + + +def repo_root() -> Path: + try: + out = subprocess.run( + ["git", "rev-parse", "--show-toplevel"], + capture_output=True, text=True, check=True, + ).stdout.strip() + return Path(out) + except Exception: + return Path.cwd() + + +def digest(p: Path) -> str: + return hashlib.sha256(p.read_bytes()).hexdigest() + + +def read_seal(p: Path) -> str | None: + if not p.exists(): + return None + # Seal format is "<64-hex> ", the format FROZEN.md + # declares but the compiler seal file does not actually honour. Here the + # declared format and the written format agree, deliberately. + first = p.read_text().strip().split("\n")[0] + tok = first.split() + return tok[0] if tok else None + + +def main(argv: list[str]) -> int: + ap = argparse.ArgumentParser(prog="tri loop-rules", description=__doc__) + ap.add_argument("--reseal", action="store_true", + help="record the current digest as the sealed one; use only " + "after a deliberate edit to the rule file") + ap.add_argument("--print-digest", action="store_true", + help="print only the current digest and exit 0") + args = ap.parse_args(argv) + + root = repo_root() + rules, seal = root / RULES, root / SEAL + + if not rules.exists(): + print(f"MISSING: {RULES} does not exist.") + print("The rules are not recoverable from this tool. Recover them from " + "git history, not from memory: memory reproduces the defect with " + "the rule (R5).") + return 1 + + cur = digest(rules) + if args.print_digest: + print(cur) + return 0 + + n_lines = len(rules.read_text().splitlines()) + n_bytes = rules.stat().st_size + + if args.reseal: + seal.parent.mkdir(parents=True, exist_ok=True) + prev = read_seal(seal) + seal.write_text(f"{cur} {RULES.as_posix()}\n") + print(f"resealed {RULES}") + print(f" previous {prev or '(none)'}") + print(f" current {cur}") + print(f" size {n_bytes} bytes, {n_lines} lines") + print("\nA reseal records that the rules changed. It does not record WHY." + "\nThe reason belongs in the commit message and in the ledger.") + return 0 + + want = read_seal(seal) + print(f"rule file : {RULES} ({n_bytes} bytes, {n_lines} lines)") + print(f"current : {cur}") + print(f"sealed : {want or '(no seal recorded)'}") + + if want is None: + print("\nNOT SEALED. Nothing certifies which version of the rules is in " + "force. Run `tri loop-rules --reseal` once, in a tick that also " + "commits the rule file.") + return 1 + + if want == cur: + print("\nOK -- the rule text is byte-identical to the sealed text.") + print("This certifies identity only. It does not certify that the rules " + "are correct, nor that the tick obeyed them.") + return 0 + + print("\nMISMATCH -- the rule text is not the sealed text.") + print("Do not stop the tick over this, and do not reseal to make it quiet. " + "Record in the ledger WHICH version was in force, because a tick run " + "against unknown rules cannot be audited afterwards. Then either " + "restore the sealed text or reseal deliberately with a stated reason.") + return 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/tri_loop/triage.py b/scripts/tri_loop/triage.py new file mode 100644 index 000000000..12cf64c90 --- /dev/null +++ b/scripts/tri_loop/triage.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +"""tri triage -- classify every open issue into one of five ordered classes. + +Why this exists. + +The plan in #1697 measured "~47 of 102 open issues are autonomous wave-loop +journal noise". The ratio has moved since: of 237 open issues, roughly 80% +record work that was already done rather than work that is waiting. + +A record with no completion condition is not a task. Counting it as backlog is +the same composition error #2133 identified in the ratchet, applied to the +tracker instead of the corpus. This tool reports the split so that any statement +about "N open issues" can be read correctly. + +What changed, and why three classes were not enough. + +The earlier version had three classes -- journal, plan, actionable -- and +everything that did not look like a journal entry fell into `actionable` by +default. That default is the flaw. An issue waiting on hardware, an issue that +asks a research question with no defined answer, and an issue that duplicates +another are all things a loop cannot pick up and finish, yet all three were +counted as available work. The count of "~26 actionable" was therefore an upper +bound being read as an estimate. + +Five ordered classes, first match wins: + + actionable a defect or change with a checkable completion condition + research an open question; done is a matter of judgement, not a check + tracking a journal entry, plan or epic; records or aggregates, never closes + blocked cannot proceed without something outside the tool: hardware, an + upstream release, a human decision, an expired credential + duplicate superseded, obsolete, or the same subject as an earlier issue + +Ordering matters and is deliberate: `blocked` is tested before `actionable`, so +a blocked defect is never advertised as available. `duplicate` is tested last, +because a duplicate that is also blocked is more usefully seen as blocked. + +Autoclose is forbidden. This tool prints and exits; it never mutates the +tracker. The classification reads titles and labels, and reads bodies only for +the blocked and duplicate signals, where the evidence is not in the title. It +remains a composition estimate, never a verdict on an individual issue. + +Usage: + tri triage [owner/repo] [--json] [--class NAME] [--bodies] +""" +import json +import re +import subprocess +import sys +from collections import Counter + +CLASSES = ("actionable", "research", "tracking", "blocked", "duplicate") + +TRACKING = re.compile( + r"^(wave\s|wave\s*loop|feat\(igla\):\s*wave|formal:.*\(prop\.|tick\s" + r"|\[plan\]|epic\b|census\b|report:)", re.I +) +RESEARCH = re.compile( + r"(\?$|^(research|investigate|explore|study|survey|consider|should we|why\s)" + r"|\bopen question\b|\bhypothes)", re.I +) +BLOCKED = re.compile( + r"\b(blocked\s+(on|by)|waiting\s+(on|for)|needs\s+hardware|requires\s+the\s+board" + r"|routing[- ]pending|deferred\b|upstream\s+bug|awaiting\s+review" + r"|needs\s+a\s+human|cannot\s+proceed\s+until)\b", re.I +) +DUPLICATE = re.compile( + r"\b(duplicate\s+of|superseded\s+by|obsolete\b|closed\s+in\s+favour|see\s+instead)\b", + re.I +) +LABEL_MAP = { + "blocked": "blocked", "wontfix": "duplicate", "duplicate": "duplicate", + "question": "research", "research": "research", "journal": "tracking", + "epic": "tracking", "plan": "tracking", +} + + +def klass(row, use_bodies=True): + """Return (class, reason). First match in CLASSES order wins.""" + title = row.get("title") or "" + body = (row.get("body") or "") if use_bodies else "" + labels = {(l.get("name") or "").lower() for l in row.get("labels") or []} + + for name, cls in LABEL_MAP.items(): + if name in labels: + return cls, f"label:{name}" + + m = BLOCKED.search(title) or (BLOCKED.search(body[:2000]) if body else None) + if m: + return "blocked", f"phrase:{m.group(0).strip().lower()[:40]}" + + m = TRACKING.search(title) + if m: + return "tracking", f"title-form:{m.group(0).strip().lower()[:40]}" + + m = RESEARCH.search(title) + if m: + return "research", f"title-form:{m.group(0).strip().lower()[:40]}" + + m = DUPLICATE.search(title) or (DUPLICATE.search(body[:2000]) if body else None) + if m: + return "duplicate", f"phrase:{m.group(0).strip().lower()[:40]}" + + return "actionable", "default: no tracking, research, blocked or duplicate signal" + + +def main(argv): + repo = "gHashTag/t27" + as_json = "--json" in argv + use_bodies = "--bodies" in argv or True + want = None + if "--class" in argv: + i = argv.index("--class") + if i + 1 < len(argv): + want = argv[i + 1].lower() + if want not in CLASSES: + print(f"unknown class {want!r}; expected one of {', '.join(CLASSES)}", + file=sys.stderr) + return 2 + positional = [a for a in argv if not a.startswith("--")] + positional = [a for a in positional if a != want] + if positional: + repo = positional[0] + + fields = "number,title,labels,createdAt,comments" + if use_bodies: + fields += ",body" + out = subprocess.run( + ["gh", "issue", "list", "--repo", repo, "--state", "open", "--limit", "1000", + "--json", fields], + capture_output=True, text=True) + if out.returncode != 0: + print(out.stderr.strip(), file=sys.stderr) + return 2 + rows = json.loads(out.stdout) + + for r in rows: + r["class"], r["reason"] = klass(r, use_bodies) + r.pop("body", None) + counts = Counter(r["class"] for r in rows) + + if as_json: + payload = {"repo": repo, "open": len(rows), + "counts": {c: counts.get(c, 0) for c in CLASSES}, + "issues": sorted(rows, key=lambda r: r["number"])} + if want: + payload["issues"] = [r for r in payload["issues"] if r["class"] == want] + print(json.dumps(payload, indent=2)) + return 0 + + total = len(rows) or 1 + print(f"{repo}: {len(rows)} open") + for c in CLASSES: + v = counts.get(c, 0) + print(f" {v:5d} {c:<11s} ({100 * v / total:.0f}%)") + + for c in ([want] if want else ["actionable", "blocked"]): + sel = sorted((r for r in rows if r["class"] == c), key=lambda r: r["number"]) + print(f"\n{c} ({len(sel)}):") + for r in sel: + print(f' #{r["number"]:<6d} {r["createdAt"][:10]} {r["title"][:88]}') + print(f' {r["reason"]}') + + print("\nNOTE: a composition estimate from titles, labels and body prefixes -- not a") + print(" verdict on any single issue. This tool never closes anything, and") + print(" nothing here licenses a bulk action. `blocked` is tested before") + print(" `actionable` on purpose, so no blocked item is advertised as available.") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:]))