Skip to content

ci(corpus-ratchet): run t27c's 1221 tests, which had never run on master - #2291

Merged
gHashTag merged 1 commit into
masterfrom
feat/ci/t27c-unit-test-gate
Aug 20, 2026
Merged

ci(corpus-ratchet): run t27c's 1221 tests, which had never run on master#2291
gHashTag merged 1 commit into
masterfrom
feat/ci/t27c-unit-test-gate

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #2289
Closes #2288

The defect

Eight workflows on origin/master run cargo build --release -p t27c. cargo build does not compile #[cfg(test)] modules. So all 1221 tests in the compiler crate were unbuilt and unrun on the branch that gates merges.

.github/workflows/formal-yosys.yml:899 does run cargo test -p t27c --release — but that file has never landed on master:

$ git cat-file -e origin/master:.github/workflows/formal-yosys.yml
fatal: path '.github/workflows/formal-yosys.yml' exists on disk, but not in 'origin/master'

A gate that exists only on a feature branch gates nothing. On origin/master the only cargo test invocations are cargo test -p tri (cli-tri.yml) and rings-rust.yml's ring-*-rust matrix, which never matches t27c.

Measured before landing

cargo test -p t27c --no-run    exit 0   0 errors   51.15s   22 test executables
cargo test -p t27c             exit 0   1221 passed; 0 failed; 0 ignored
cargo test -p t27c --release   exit 0   1221 passed; 0 failed; 0 ignored

All pass, so this lands as a plain gate, not a ratchet. A ratchet over an all-green baseline is strictly worse: it adds a file to maintain while ceasing to detect the first regression it exists for.

The change (2 files, +144 / -1)

1. One step in corpus-ratchet.yml, placed directly after the existing Build t27c so the crate is already compiled and the marginal cost is only the test targets. That workflow was chosen over the other seven because it already runs on pull_request and push to master, already caches cargo with target/ in the key, and has a 45-minute ceiling against a ~314s job.

2. Scopes the pre-existing Explain a failure step to steps.ratchet.outcome == 'failure'. It was a bare failure(); with a test step above it, a failing Rust unit test would print "add an entry to docs/reports/suite_expectations.json" — sending the reader to bless a corpus expectation for a defect that is not in the corpus. A new Explain a unit-test failure step covers the other case and says explicitly that it is not the ratchet.

Verification

  • The step body was run verbatim, extracted from the parsed YAML: exit 0, 1221 passed, 0 failed, 0 ignored across 22 test binaries.
  • The gate bites. A shape-identical harness confirms non-zero exit when the test command fails (101), zero when it passes, and no false-fail when grep -c matches nothing. This is a shell-level proxy — no test was edited to force a real red run.
  • YAML parses under PyYAML (8 steps, ordering and if: conditions asserted); both new run: bodies pass bash -n.
  • git diff --numstat origin/master -- docs/NOW.md77 0 (zero removed lines); ^# NOW count 133 → 134; no conflict markers.

Coverage this does NOT provide

913 #[test] attributes live in bootstrap/src; 895 compile into this run. The missing 18 are named in the workflow comment and printed to the step summary, so a green tick is not misread as "every test in bootstrap/src ran":

  • bootstrap/src/math_compare.rs — 10 tests, compiled by nothing. No mod math_compare; exists in the crate, so rustc never opens the file (one of 8 orphan .rs files in bootstrap/src, and the only one with tests). Built standalone it is 9 pass / 1 fail: test_hybrid_v2_plateau asserts |n152 - n20| < 1e-9 while the file's own GOLDEN_V2 records values 2.100e-9 apart. Independent recomputation reproduces the goldens exactly, so the test is wrong, not the implementation. It was not adjusted and not wired in — wiring it in today would land this gate red. Filed as bootstrap/src/math_compare.rs is compiled by nothing; its plateau test contradicts its own goldens #2290.
  • bootstrap/src/proxy.rs:326 — 8 tests behind #[cfg(all(test, feature = "server"))]. No workflow passes --features server, so they are stripped before type-checking.

Not fixed here

bootstrap/Cargo.toml's [profile.release] (opt-level=3, lto=true, codegen-units=1, strip=true) is silently discarded — the workspace root Cargo.toml has no [profile] section, so cargo prints warning: profiles for the non root package will be ignored on every invocation. Every cargo build --release -p t27c across 8+ workflows ships a non-LTO, non-stripped binary. Real and unfixed: changing it changes the binary every workflow produces, which is not a change to smuggle into a CI-gate PR. Recorded in #2289.

Eight workflows on origin/master run `cargo build --release -p t27c`, and
`cargo build` does not compile `#[cfg(test)]` modules. Every test in the
compiler crate was therefore unbuilt and unrun on the branch that gates
merges. formal-yosys.yml does run them, but has never landed on master, so
it gates nothing.

Adds one step to corpus-ratchet.yml, which already builds the crate one
step above, so the marginal cost is compiling the test targets. Measured
before landing: 1221 passed, 0 failed, 0 ignored across 22 test binaries,
in debug and release alike. Because nothing fails this lands as a plain
gate rather than a ratchet.

Also scopes the pre-existing `Explain a failure` step to
steps.ratchet.outcome. As a bare failure() it would answer a failing Rust
unit test with "add an entry to suite_expectations.json", pointing the
reader at a corpus expectation for a defect not in the corpus.

Coverage is stated openly rather than implied: 913 #[test] attributes live
in bootstrap/src and 895 compile into this run. math_compare.rs (10 tests,
no mod declaration anywhere) and proxy.rs (8 tests behind feature="server")
are named in the workflow comment and the step summary. math_compare's
failing plateau test is a wrong assertion, not a wrong implementation; it
was left untouched and filed as #2290 rather than adjusted.

Closes #2289
Closes #2288
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag enabled auto-merge (squash) August 20, 2026 03:56
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-20 03:56:58 UTC

Summary

Status Count
Total Open PRs 19
PRs with Failing Checks 4
PRs with All Checks Green 15
READY 7
FAILING 4
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=cbbfac87dff3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit ce6ea62 into master Aug 20, 2026
16 of 19 checks passed
@gHashTag

Copy link
Copy Markdown
Owner Author

This went red on its first run on master and is being removed in #2293.

The "1221 passed, 0 failed" measurement was real but was taken in a working tree checked out at feat/wave-547/host-heapsort, not master. That branch's bootstrap/src/compiler.rs is 22,142 lines against master's 36,970. On master the same command is 1602 passed, 13 failed, 2 ignored: https://github.com/gHashTag/t27/actions/runs/32330080992

The 13 failing test names and the plan to re-land this as a ratchet are in #2292. #2288 and #2289 are reopened.

@gHashTag
gHashTag deleted the feat/ci/t27c-unit-test-gate branch August 20, 2026 04:08
gHashTag added a commit that referenced this pull request Aug 20, 2026
#2294)

The previous NOW entry asserts, in a BINDING honesty limit, that master
has no branch protection, on the strength of
`GET /repos/gHashTag/t27/branches/master/protection` returning
`404 Branch not protected`.

That endpoint reports only legacy branch protection and 404s for a
repository protected by rulesets. Master IS protected, by the active
ruleset t27-master-protection: PRs required, deletion and non-fast-forward
blocked, and four required status contexts (check-now-freshness, validate,
check, check-linked-issue).

The observation behind the claim survives and is better explained. PR
#2291 merged three seconds before its own corpus-ratchet run began not
because nothing is required, but because corpus-ratchet is not one of the
required four -- which is also why coverage and withdrawn-live sit red on
master while PRs keep merging.

This sharpens rather than softens the original finding: the cargo test
step was added to a workflow outside the required set, so even green it
could never have blocked a merge. It was wrong on its numbers and wrong
in its placement, and the second error would have outlived fixing the
first.

Adding corpus-ratchet to the ruleset is deliberately not done here: it is
currently failing on master, so requiring it would block every PR. That
must follow the ratchet in #2292, not precede it.

Refs #2292

Co-authored-by: Claude <claude@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants