Skip to content

The two tools every parser claim rests on are lost, and the one that survived reported zero regressions while fields were being dropped #2158

Description

@gHashTag

Two measurement tools were lost, and every number they had produced became unreproducible in the same stroke.

scripts/tri_loop/cost.py and scripts/tri_loop/diffbin.py were written, used to produce a "634 specs, 0 regressions" figure and a parser scaling exponent, and quoted in #2151. Neither was ever committed to any branch. The working copy holding them was later re-cloned, and they went with it.

Six recovery routes were checked and all came back empty:

route result
dangling git objects one git stash WIP commit, holding only triage.py
reflog records the clone at b9287250 -- that is the loss, not the content
shell history no .bash_history or .zsh_history in the sandbox
CI artifacts fpga-synthesis, fpga-verilog, wp18-conformance-report only
PR and issue comments no comment carries the source of either script
session snapshot prose describing the scripts, not the scripts

So the source is gone and a reimplementation from a written contract is the only honest option. Recalling what the old one "roughly did" would reproduce its defect, which is the second half of this issue.

The old aggregate was wrong, and its wrongness is the specification for the new one

It reported "0 regressions" over 634 specs. It was simultaneously true that files lost struct fields: on input like a : Map<K, followed by b : u8, the candidate absorbs the following name : type pairs into the first field's type text, so three declared fields become one. Field loss was the tool's own stated regression criterion. Both statements stood because a per-file judgement had relabelled the loss as an acceptable trade, and the aggregate then reported the judgement as though it were a measurement.

The invariant that follows:

No differential result may be called "0 regressions" unless the metric actually checks the claimed class of loss.

Definition of done

  • tri diffbin assigns every file exactly one of five categories, matched in order: unchanged, field-loss, strict-improvement, malformed-input-tradeoff, unknown. field-loss is tested before strict-improvement, and is never folded into another count. Exit status is driven by field-loss and unknown, not by a summary verdict
  • Phantom and declared fields are distinguished by a stated rule, not by counting: a removed field is a phantom only if its type text in the base was empty
  • Only an ExprIdentifier whose parent is a StructDecl counts as a field, so identifiers in function bodies do not enter the totals
  • tri cost reports per stratum: n, median, p95, min-max ms/KB, coefficient of variation. Alpha only at n >= 8, printed with r2 and the KB range it was fitted over. No cross-family alpha is printed at all, because it is a metric of corpus composition rather than of the parser (wave 698: 36% of the parser backlog is Markdown, and the count could not say so #2133)
  • Absolute milliseconds are labelled with the build profile, so a debug figure cannot be quoted as a release one
  • Golden fixtures exist for: well-formed nested generic, tuple type, semicolon_phantom, field_swallow, EOF inside an unclosed generic, and one fixture per distinct shape of corpus damage
  • A CI step fails when a required loop tool is missing, untracked, or unrouted -- and is verified to fail in the pre-loss state, not merely to pass in the current one
  • Both commands run reproducibly from a clean worktree, and the report separates what changed from what was not proven

Blocks nothing, but every performance and correctness claim about the parser rests on these two tools, so their absence is the largest open risk to the loop's conclusions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions