ci(corpus-ratchet): run t27c's 1221 tests, which had never run on master - #2291
Merged
Conversation
Eight workflows on origin/master run `cargo build --release -p t27c`, and `cargo build` does not compile `#[cfg(test)]` modules. Every test in the compiler crate was therefore unbuilt and unrun on the branch that gates merges. formal-yosys.yml does run them, but has never landed on master, so it gates nothing. Adds one step to corpus-ratchet.yml, which already builds the crate one step above, so the marginal cost is compiling the test targets. Measured before landing: 1221 passed, 0 failed, 0 ignored across 22 test binaries, in debug and release alike. Because nothing fails this lands as a plain gate rather than a ratchet. Also scopes the pre-existing `Explain a failure` step to steps.ratchet.outcome. As a bare failure() it would answer a failing Rust unit test with "add an entry to suite_expectations.json", pointing the reader at a corpus expectation for a defect not in the corpus. Coverage is stated openly rather than implied: 913 #[test] attributes live in bootstrap/src and 895 compile into this run. math_compare.rs (10 tests, no mod declaration anywhere) and proxy.rs (8 tests behind feature="server") are named in the workflow comment and the step summary. math_compare's failing plateau test is a wrong assertion, not a wrong implementation; it was left untouched and filed as #2290 rather than adjusted. Closes #2289 Closes #2288
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
gHashTag
enabled auto-merge (squash)
August 20, 2026 03:56
Contributor
Owner
Author
|
This went red on its first run on master and is being removed in #2293. The "1221 passed, 0 failed" measurement was real but was taken in a working tree checked out at The 13 failing test names and the plan to re-land this as a ratchet are in #2292. #2288 and #2289 are reopened. |
gHashTag
added a commit
that referenced
this pull request
Aug 20, 2026
#2294) The previous NOW entry asserts, in a BINDING honesty limit, that master has no branch protection, on the strength of `GET /repos/gHashTag/t27/branches/master/protection` returning `404 Branch not protected`. That endpoint reports only legacy branch protection and 404s for a repository protected by rulesets. Master IS protected, by the active ruleset t27-master-protection: PRs required, deletion and non-fast-forward blocked, and four required status contexts (check-now-freshness, validate, check, check-linked-issue). The observation behind the claim survives and is better explained. PR #2291 merged three seconds before its own corpus-ratchet run began not because nothing is required, but because corpus-ratchet is not one of the required four -- which is also why coverage and withdrawn-live sit red on master while PRs keep merging. This sharpens rather than softens the original finding: the cargo test step was added to a workflow outside the required set, so even green it could never have blocked a merge. It was wrong on its numbers and wrong in its placement, and the second error would have outlived fixing the first. Adding corpus-ratchet to the ruleset is deliberately not done here: it is currently failing on master, so requiring it would block every PR. That must follow the ratchet in #2292, not precede it. Refs #2292 Co-authored-by: Claude <claude@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2289
Closes #2288
The defect
Eight workflows on
origin/masterruncargo build --release -p t27c.cargo builddoes not compile#[cfg(test)]modules. So all 1221 tests in the compiler crate were unbuilt and unrun on the branch that gates merges..github/workflows/formal-yosys.yml:899does runcargo test -p t27c --release— but that file has never landed on master:A gate that exists only on a feature branch gates nothing. On
origin/masterthe onlycargo testinvocations arecargo test -p tri(cli-tri.yml) andrings-rust.yml'sring-*-rustmatrix, which never matchest27c.Measured before landing
All pass, so this lands as a plain gate, not a ratchet. A ratchet over an all-green baseline is strictly worse: it adds a file to maintain while ceasing to detect the first regression it exists for.
The change (2 files, +144 / -1)
1. One step in
corpus-ratchet.yml, placed directly after the existingBuild t27cso the crate is already compiled and the marginal cost is only the test targets. That workflow was chosen over the other seven because it already runs onpull_requestandpushto master, already caches cargo withtarget/in the key, and has a 45-minute ceiling against a ~314s job.2. Scopes the pre-existing
Explain a failurestep tosteps.ratchet.outcome == 'failure'. It was a barefailure(); with a test step above it, a failing Rust unit test would print "add an entry todocs/reports/suite_expectations.json" — sending the reader to bless a corpus expectation for a defect that is not in the corpus. A newExplain a unit-test failurestep covers the other case and says explicitly that it is not the ratchet.Verification
1221 passed, 0 failed, 0 ignored across 22 test binaries.grep -cmatches nothing. This is a shell-level proxy — no test was edited to force a real red run.if:conditions asserted); both newrun:bodies passbash -n.git diff --numstat origin/master -- docs/NOW.md→77 0(zero removed lines);^# NOWcount 133 → 134; no conflict markers.Coverage this does NOT provide
913
#[test]attributes live inbootstrap/src; 895 compile into this run. The missing 18 are named in the workflow comment and printed to the step summary, so a green tick is not misread as "every test inbootstrap/srcran":bootstrap/src/math_compare.rs— 10 tests, compiled by nothing. Nomod math_compare;exists in the crate, so rustc never opens the file (one of 8 orphan.rsfiles inbootstrap/src, and the only one with tests). Built standalone it is 9 pass / 1 fail:test_hybrid_v2_plateauasserts|n152 - n20| < 1e-9while the file's ownGOLDEN_V2records values 2.100e-9 apart. Independent recomputation reproduces the goldens exactly, so the test is wrong, not the implementation. It was not adjusted and not wired in — wiring it in today would land this gate red. Filed as bootstrap/src/math_compare.rs is compiled by nothing; its plateau test contradicts its own goldens #2290.bootstrap/src/proxy.rs:326— 8 tests behind#[cfg(all(test, feature = "server"))]. No workflow passes--features server, so they are stripped before type-checking.Not fixed here
bootstrap/Cargo.toml's[profile.release](opt-level=3, lto=true, codegen-units=1, strip=true) is silently discarded — the workspace rootCargo.tomlhas no[profile]section, so cargo printswarning: profiles for the non root package will be ignoredon every invocation. Everycargo build --release -p t27cacross 8+ workflows ships a non-LTO, non-stripped binary. Real and unfixed: changing it changes the binary every workflow produces, which is not a change to smuggle into a CI-gate PR. Recorded in #2289.