loop: the differential tool counted unmeasured files as agreement - #2170
Open
gHashTag wants to merge 1 commit into
Open
loop: the differential tool counted unmeasured files as agreement#2170gHashTag wants to merge 1 commit into
gHashTag wants to merge 1 commit into
Conversation
`tri diffbin` reported five categories, and the first one carried two different
statements at once: "both binaries parsed and the field sets are identical", and
"neither binary parsed the file". Both incremented the same counter.
Same binaries, same 150-file slice, two versions of the same tool [measured]:
five categories: 150 unchanged, 0 field-loss, 0 unknown -- total agreement
six categories: 59 unchanged, 91 not-evaluated (both-error) -- coverage 39.3%
Nothing about the compilers changed between those two lines. The first line is the
shape that had been quoted in pull requests. Over the 634-spec library the split
was 330 measured against 286 unmeasured: a 52 % base printed as 100 %.
What changes:
* six mutually exclusive categories, with the partition ASSERTED at runtime
rather than claimed in a docstring
* every not-evaluated row carries a reason code -- both-error, base-timeout,
candidate-timeout, environment-failure, excluded-source-loss, or a named
other. "No verdict" for six reasons is six facts, and only some of them are
about the compiler. An uncoded not-evaluated is a hard error, exit 2.
* no PASS while any unknown remains, whatever the other counts say
* coverage printed on every run, and required in the same sentence as any "no
regressions" claim. A caveat in a neighbouring paragraph does not travel with
the number; readers quote numbers, not paragraphs.
* --exclude-status takes a corpus-status JSON and holds out
unrecoverable-source-loss files under their own code. It REFUSES to run if the
list selects nothing, because a silently empty exclusion produces a report
claiming files were held out when none were.
Full corpus, single completed run, uniform 12 s threshold, 1089 files [measured]:
524 unchanged/ok, 343 unchanged/fail, 0 regressions, 1 strict-improvement, 221
not-evaluated (195 both-timeout, 26 candidate-timeout). Coverage 868/1089 = 79.7 %.
The 26 ok->timeout files are the threshold, not a slowdown. Timed directly, three
runs each way: median candidate/base ratio 1.010, min 0.985, max 1.026, on files
taking 10.8-11.7 s against a 12 s wall [measured]. A 1-3 % jitter moves them
across it.
Also: R15 and R16 added to docs/loop/LOOP-RULES.md and resealed; corpus-parse,
corpus-status, diffmodes and loop-rules registered in loop-tools-tracked.sh, which
fails on an untracked tool -- the exact state that already destroyed two of these
scripts and every number they had produced.
Overlap disclosed: cost.py, diffbin.py, damage*.py, triage.py and
loop-tools-tracked.sh also appear in the branches behind #2159 and #2161. They are
included here because a measurement fix must not sit in a deferred branch while
its output is being quoted. Whichever lands first, the other needs a rebase; the
diffbin.py here supersedes both.
Not claimed: that coverage improved. It did not. Coverage rises only by repairing
files or excluding them with a status -- never by recategorising them.
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2166
The defect, in one table
Same binaries, same 150-file slice, two versions of the same tool
[measured]:150 unchanged, 0 field-loss, 0 unknown— total agreement59 unchanged, 91 not-evaluated (both-error), coverage 39.3 %Nothing about the compilers changed between those two lines. The first is the
shape that had been quoted in pull requests.
unchangedwas carrying two statements at once — "both binaries parsed the fileand the field sets are identical" and "neither binary parsed the file" — and both
incremented the same counter. No reviewer and no threshold could separate them.
Over the 634-spec library the split was 330 measured against 286 unmeasured: a
52 % base reported as 100 %.
The 39.3 % figure is for that slice, not for the corpus; the slice is the
alphabetically first 150 files and is unusually damaged.
What changes
runtime. Claiming exclusivity in a docstring is not a check.
not-evaluatedrow carries a reason code:both-error,base-timeout,candidate-timeout,environment-failure,excluded-source-loss, or a named other. "No verdict" for six reasons is sixdifferent facts, and only some of them are about the compiler. An uncoded
not-evaluatedis a hard error, exit 2.PASSwhile anyunknownremains, whatever the other counts say.regressions" claim. A caveat in a neighbouring paragraph does not travel with the
number — readers quote numbers, not paragraphs.
--exclude-statusholds outunrecoverable-source-lossfiles under their owncode, and refuses to run if the list selects nothing — a silently empty
exclusion yields a report claiming files were held out when none were.
Full corpus, one completed run, one threshold
[measured]1089 files, uniform 12 s,
/tmp/t27c.m2162base→/tmp/t27c.m2162fix:specs/tri/collections/array.t27)Coverage 868/1089 = 79.7 %. The zero above is a statement about those 868
files and about parse verdicts and field sets only. It says nothing about generated
code, type inference, diagnostics or timing.
Earlier 25 s runs are not aggregated with this one. A run at 25 s and a run at
12 s do not produce comparable
not-evaluatedsets, and joining them manufacturesa difference.
The 26 moved files are the threshold, not a slowdown
[measured]ok → timeoutin one direction and 0 back looks like a slowed candidate, so it wastimed rather than argued: three runs each way per file.
A 1–3 % jitter is enough to carry a file taking 11.5 s across a 12 s boundary. The
one-directionality is machine load during a 13-minute run, not a regression. This
is R14 predicting the pattern and then being confirmed on an independent run.
Disclosed overlap
cost.py,diffbin.py,damage*.py,triage.pyandloop-tools-tracked.shalsoappear in the branches behind #2159 and #2161. They are duplicated here on
purpose: a measurement fix must not sit in a deferred branch while its output is
being quoted in review. Whichever lands first, the other needs a rebase, and the
diffbin.pyin this PR supersedes both.Also here
unchangedmay never mean "we could not compare".corpus-parse,corpus-status,diffmodes,loop-rulesregistered inloop-tools-tracked.sh, which fails on an untracked tool — the exact state thatalready destroyed two of these scripts and every number they had produced.
Not claimed
Coverage did not improve. It was 52 % before and it is 52 % now on that
library; what changed is that the number is visible. Coverage rises only by
repairing files or excluding them with a status — never by recategorising them.