fix(corpus-ratchet): remove the t27c test step; it was measured on the wrong tree - #2293
Merged
Conversation
…e wrong tree ce6ea62 added `cargo test -p t27c --release` on the strength of "1221 passed, 0 failed". That measurement was real and was taken in a working tree checked out at feat/wave-547/host-heapsort, not master. The two trees are not close: bootstrap/src/compiler.rs is 22,142 lines there against 36,970 on master. On master the same command is 1602 passed, 13 failed, 2 ignored, and the step went red on its first run. A gate that lands red gets disabled rather than obeyed, so it comes out rather than being papered over. 13 is a lower bound: cargo test stops at the first failing target, so the 21 binaries in bootstrap/tests/ never ran and are still unmeasured. It should return as a ratchet built from a `--no-fail-fast` run on master. The failing names and that plan are in #2292; #2288 and #2289 are reopened. Adding a failing step in front of the ratchet also made things worse for 34 minutes: corpus-ratchet was already failing at `Run the corpus ratchet`, and the short-circuit stopped that pre-existing failure from being reported at all. Kept: `Explain a failure` stays scoped to steps.ratchet.outcome. As a bare failure() it answers a failing `Build t27c` with "add an entry to suite_expectations.json", which is wrong independently of any test step. Refs #2292 Refs #2288 Refs #2289
gHashTag
enabled auto-merge (squash)
August 20, 2026 04:03
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This was referenced Aug 20, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2292
Refs #2288
Refs #2289
Removes the
cargo test -p t27c --releasestep added 34 minutes earlier ince6ea628b(PR #2291). I landed that step, it went red on its first run on master, and this takes it back out.Why it was wrong
ce6ea628bwas justified by "1221 passed, 0 failed, 0 ignored". That measurement was real. It was taken in the wrong working tree — the host checkout sits onfeat/wave-547/host-heapsort, while the gate runs onmaster, and the two are nowhere near each other:feat/wave-547/host-heapsortorigin/masterbootstrap/src/compiler.rs#[test]inbootstrap/srccargo test -p t27cmain binRun that went red: https://github.com/gHashTag/t27/actions/runs/32330080992
A gate that lands red gets disabled rather than obeyed, so it comes out rather than being silenced, baselined in place, or "fixed" by editing tests. The 13 failing test names are recorded in #2292 along with the plan to re-land this as a ratchet.
The failure routing did work
Worth keeping on the record, because it is the one part that behaved as designed:
So the
steps.ratchet.outcomescoping is kept. As a barefailure()it answers a failingBuild t27cwith "add an entry todocs/reports/suite_expectations.json", which is wrong independently of any test step. That is the only change fromce6ea628bthat survives.Diff against the pre-gate state
Against
eeb779e4c(master before #2291),corpus-ratchet.ymldiffers only by the scoping fix and a comment block recording why the step must not be re-added without a measurement taken on master. The red step and its explainer are gone.Honesty limits
cargo teststops at the first failing target, so the 21 test binaries inbootstrap/tests/never ran on master. Their state is unmeasured. A ratchet baseline needs--no-fail-fastfirst.corpus-ratchetwas already failing atRun the corpus ratchetoneeb779e4c,57a53005a,2255e4c32,bffd38982and400850702. A failing step in front of it short-circuited the job, so that pre-existing failure stopped being reported at all.corpus-ratchetis not a required check. This bullet originally read "masterhas no branch protection", citingGET /repos/gHashTag/t27/branches/master/protection→404 Branch not protected. That endpoint sees only legacy branch protection and 404s for a repo protected by rulesets; the claim was wrong. Master IS protected by the active rulesett27-master-protection— PRs required, deletion and non-fast-forward blocked, four required contexts:check-now-freshness,validate,check,check-linked-issue.corpus-ratchetis not among them (norcoverage, norwithdrawn-live), which is why all three sit red while PRs merge. ci(corpus-ratchet): run t27c's 1221 tests, which had never run on master #2291 still auto-merged at03:57:13Zwith its owncorpus-ratchetrun starting at03:57:16Z— three seconds before the gate it was adding began to run — because that gate was never required. This sharpens the finding: the step was added to a workflow outside the required set, so even green it could never have blocked a merge.corpus-ratchetrun will fail, atRun the corpus ratchet, for the pre-existing reason above. That is the state it is being returned to, not a new regression.math_compare.rscompiled by nothing, plateau test 2.10x tighter than its own goldens) stands untouched.