tri: restore cost and diffbin, and the same 634 specs now report 13 field losses where the old tool reported zero - #2159
Open
gHashTag wants to merge 1 commit into
Open
tri: restore cost and diffbin, and the same 634 specs now report 13 field losses where the old tool reported zero#2159gHashTag wants to merge 1 commit into
gHashTag wants to merge 1 commit into
Conversation
Closes #2158) Two measurement tools were lost and every number they had produced became unreproducible with them. cost.py and diffbin.py were written, quoted in #2151, and never committed; the working copy was later re-cloned. Six recovery routes came back empty -- dangling objects held only a git stash WIP with triage.py, the reflog records the clone rather than the content, shell history is absent, CI artifacts hold only FPGA outputs, no PR or issue comment carries the source, and the session snapshot preserved prose about the scripts instead of the scripts. So these are reimplementations from a written contract. Recalling what the old ones roughly did would have reproduced the old one's defect. That defect was the specification for the new one. It reported "0 regressions" over 634 specs while files were losing declared struct fields, because a per-file judgement had relabelled the loss as an acceptable trade and the aggregate then printed the judgement as if it were a measurement. No differential result may be called "0 regressions" unless the metric actually checks the claimed class of loss. diffbin now assigns five ordered categories -- unchanged, field-loss, strict-improvement, malformed-input-tradeoff, unknown -- with field-loss tested before strict-improvement, so removing a phantom while dropping a declared field is a loss and not an improvement. Phantom and declared are told apart by a stated rule: a removed field is a phantom only if its base type text was empty. Only an ExprIdentifier whose parent is a StructDecl counts, so identifiers in function bodies stay out of the totals. Re-measured on the same 634 specs and the same two binaries: 616 unchanged, 13 field-loss, 1 strict-improvement, 4 malformed-input-tradeoff, 0 unknown. handoff.t27 goes from 35 parsed fields to 12. All 17 files that moved are inside the damaged set and no well-formed spec changed at all, which is what 0 unknown is carrying. cost reports per stratum with n, median, p95, min-max ms/KB and coefficient of variation, alpha only at n >= 8 with its r2 and KB range, and no cross-family alpha at all: that number is a metric of corpus composition rather than of the parser (#2133), and a printed number gets quoted while its caveat does not travel with it. damage classifies the corrupt annotations by shape rather than repairing them (#2154): 125 lines, 65 files, 15 shapes, one fixture each. The first draft reported 429, of which 230 were the legitimate bound `target : < 5000ns`, so the fix was deleting two bad signals rather than tuning a threshold. loop-tools-tracked.sh fails when a loop tool is missing, untracked, or unrouted, and was verified to fail in exactly the pre-loss state. The dispatcher no longer looks for a built compiler before running helpers that never use one.
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This was referenced Aug 14, 2026
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2158
What this establishes
Two measurement tools were unrecoverable, and the recovery attempt is documented rather than asserted.
scripts/tri_loop/cost.pyanddiffbin.pywere written, used to produce the "634 specs, 0 regressions" figure quoted in #2151, and never committed to any branch; the working copy was later re-cloned. All six routes named in the order were checked:git stashWIP commit, holding onlytriage.pyb9287250-- that is the loss, not the content.bash_historyor.zsh_historyexistsfpga-synthesis,fpga-verilog,wp18-conformance-reportonlySo both are reimplemented from a written contract. That is the stronger outcome, because recalling what the old one roughly did would have reproduced its defect.
The old aggregate was wrong, and the same corpus now says so. Re-run over the identical 634 specs with the identical two binaries (
/tmp/t27c.baseand/tmp/t27c.fixed, both of which survived):Previously: "0 regressions".
specs/tri/agent/handoff.t27goes from 35 parsed fields to 12. Verified by hand onspecs/tri/crypto/sha256.t27, where base yieldsstateandcountand the candidate yields onlystate, having absorbedcount : u64into its type text.The finding that decides #2151, and it is a favourable one. All 17 files whose field sets moved are inside the damaged set from #2154 -- 13 of 13
field-lossand 4 of 4malformed-input-tradeoff. Not one well-formed spec changed behaviour, which is what0 unknownis carrying. The recovery change is therefore safe on clean input and its entire cost is confined to files that are themselves corrupt. That reorders the queue: repair #2154 first and the field-loss question largely dissolves, because the inputs producing it stop existing.The contract, and why each clause is there
tri diffbin-- five categories matched in order:unchanged,field-loss,strict-improvement,malformed-input-tradeoff,unknown.field-lossis tested beforestrict-improvement, so a change that removes a phantom and also drops a declared field is a loss, not an improvement. Exit status is driven byfield-lossandunknowncounts, never by a summary verdict.The discrimination the old tool lacked: a removed field is a phantom only if its type text in the base was EMPTY. That is the signature of an identifier lifted out of a type-argument list and promoted to a field. A removed field with a non-empty base type was declared by the author, and its removal is a loss. Counting fields cannot tell the two apart, which is how the old aggregate missed them.
Second correction: only an
ExprIdentifierwhose parent is aStructDeclcounts as a field.t27c parseprints that kind for every identifier in the tree including function bodies, so matching on kind alone would count expression operands as struct fields.tri cost-- per stratum: n, median, p95, min-max ms/KB, coefficient of variation. Alpha only at n >= 8, printed beside its r2 and the KB range it was fitted over. No cross-family alpha is printed at all. Not printed-with-a-caveat: not printed. A single exponent across these strata is a metric of corpus composition rather than of the parser (#2133), and a number gets quoted while its caveat does not travel with it. Measured onspecs/tri, alpha was withheld for 12 of 13 strata for insufficient n, and the one printed (utils, 0.14) carries r2 0.50 over a 2.2-3.0 KB range -- a slope through a cloud, which the output says.Build profile is labelled, so a debug millisecond cannot be quoted as a release one.
tri damage-- classifies the corrupt annotations of #2154 by shape and repairs nothing: 125 lines, 65 files, 15 distinct shapes, one emitted fixture per shape. Repair can then be a reviewable diff of a stated size per class instead of one sweep over 125 lines.Its first draft reported 429 damaged lines. 230 of those were
target : < 5000ns-- a legitimate less-than bound, not an unbalanced bracket; 17 were match arms, 10 were multi-line array literals, 6 were function signatures the field regex should never have matched. So the headline measured the regex, not the corpus. The fix was deleting the two bad signals, not tuning a threshold until the number looked right. Two survive:[[], which the language has no syntax for, and an odd".r#"is excluded after reading the three lines it flagged.This also revises the previous loop's figure upward from 115 lines / 63 files, because the older pattern missed the
[?[]and[std.StringHashMap(...)"shapes.scripts/ci/loop-tools-tracked.sh-- fails when a required tool is missing, when it exists but git does not track it, when anything underscripts/tri_loop/is untracked, or when the generic dispatch line is gone. Verified to fail in exactly the pre-loss state (FAIL, four untracked files) and to pass after the commit. A check only verified in its passing state is decoration.Its own check 4 was wrong in its first draft: it grepped
scripts/trifor each command name, andtri triagepassed only because the word appears in a comment. A check that detects its own documentation is worth nothing, so it now verifies the dispatch line and the resolved target file.Dispatcher defect fixed.
scripts/trilooked for a builtt27cand exited before dispatching loop helpers.tri triageandtri damageread the tracker and the spec text and never touch the compiler, yet a machine with no build could not run them. Loop dispatch now precedes the binary lookup, and the new workflow job runs with no compiler present to keep it that way.What this does NOT establish
tri damagedoes not bound the damage. It checks two textual signals on lines that look like field declarations. Damage that yields a balanced, quote-even, plausible-looking wrong type is invisible to it, and nothing in the output limits how much of that there is63 of 150 files underspecs/tri` failed to parse during the cost run and were excluded. That is a large exclusion and it is consistent with the NOPARSE population claimed in Corpus census W737-W739: the defect population is 168, not 490 — and what is still open #2152; it is recorded, not explained/tmp. The differential is reproducible today and not reproducible after a sandbox reset. Rebuilding both from source is the remaining step to make this number durableloop-tools-tracked.shdoes not make a tool correct, and does not prevent a tracked file being deleted in a later commit. It closes the one hole that swallowed these two files: work existing only outside version controlReproduce