From 0a43df26fc21f68e722b50ec7c8f63a7d2f3b24c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 20 Aug 2026 11:03:14 +0700 Subject: [PATCH] fix(corpus-ratchet): remove the t27c test step; it was measured on the wrong tree ce6ea628b added `cargo test -p t27c --release` on the strength of "1221 passed, 0 failed". That measurement was real and was taken in a working tree checked out at feat/wave-547/host-heapsort, not master. The two trees are not close: bootstrap/src/compiler.rs is 22,142 lines there against 36,970 on master. On master the same command is 1602 passed, 13 failed, 2 ignored, and the step went red on its first run. A gate that lands red gets disabled rather than obeyed, so it comes out rather than being papered over. 13 is a lower bound: cargo test stops at the first failing target, so the 21 binaries in bootstrap/tests/ never ran and are still unmeasured. It should return as a ratchet built from a `--no-fail-fast` run on master. The failing names and that plan are in #2292; #2288 and #2289 are reopened. Adding a failing step in front of the ratchet also made things worse for 34 minutes: corpus-ratchet was already failing at `Run the corpus ratchet`, and the short-circuit stopped that pre-existing failure from being reported at all. Kept: `Explain a failure` stays scoped to steps.ratchet.outcome. As a bare failure() it answers a failing `Build t27c` with "add an entry to suite_expectations.json", which is wrong independently of any test step. Refs #2292 Refs #2288 Refs #2289 --- .github/workflows/corpus-ratchet.yml | 69 ++++------------------------ docs/NOW.md | 60 ++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 59 deletions(-) diff --git a/.github/workflows/corpus-ratchet.yml b/.github/workflows/corpus-ratchet.yml index 8da7d8274..09c41095a 100644 --- a/.github/workflows/corpus-ratchet.yml +++ b/.github/workflows/corpus-ratchet.yml @@ -59,65 +59,17 @@ jobs: - name: Build t27c run: cargo build --release -p t27c - # Closes #2288. `cargo build` does not compile `#[cfg(test)]` modules, so - # eight workflows on master build t27c and not one of them ever compiled, - # let alone ran, a single test in it. Measured 2026-08-20: 1221 tests - # exist and 1221 of them had never executed in CI. They all pass -- so - # this lands as a plain gate, not a ratchet. + # A `cargo test -p t27c` step lived here for 34 minutes on 2026-08-20 + # (ce6ea628b) and is removed again by #2292. It was landed on a measurement + # of 1221 passing tests taken in a working tree checked out at + # feat/wave-547/host-heapsort, NOT at master -- that branch's + # bootstrap/src/compiler.rs is 22,142 lines against master's 36,970. On + # master the same command is 1602 passed / 13 failed / 2 ignored, so the + # step went red on its first run here. # - # It goes here because the crate is already built one step above, so the - # marginal cost is compiling the test targets and running them: measured - # 1m43s cold / 1.75s warm, against this job's 45-minute ceiling and its - # ~314s ratchet run. - # - # Scope, stated openly -- this runs what `cargo test -p t27c` REACHES, - # which is not every test in the tree. 913 `#[test]` attributes live in - # bootstrap/src; 895 compile into this run. The missing 18: - # * bootstrap/src/math_compare.rs -- 10 tests, and no `mod math_compare;` - # exists anywhere in the crate, so rustc never opens the file. Built - # standalone it is 9 pass / 1 fail (see #2288 follow-up); the failure - # is a bad assertion, not a bad implementation, so it is NOT silenced - # here and NOT wired in here -- wiring it in would land this gate red. - # * bootstrap/src/proxy.rs -- 8 tests behind - # `#[cfg(all(test, feature = "server"))]`. No workflow passes - # `--features server`, so they are stripped before type-checking. - # Neither hole is created by this step; both are named so the green tick - # is not read as "every test in bootstrap/src ran". - - name: Unit tests (cargo test -p t27c) - id: unit_tests - run: | - set -o pipefail - cargo test -p t27c --release 2>&1 | tee unit_tests.log - rc=$? - totals=$(grep -E '^test result:' unit_tests.log \ - | awk '{p+=$4; f+=$6; i+=$8} END {printf "%d passed, %d failed, %d ignored", p, f, i}') - bins=$(grep -cE '^test result:' unit_tests.log) - echo "" - echo "t27c unit tests: ${totals} across ${bins} test binaries." - { - echo "### t27c unit tests" - echo "" - echo "\`cargo test -p t27c --release\` -- **${totals}** across ${bins} test binaries." - echo "" - echo "Not covered: 10 tests in \`bootstrap/src/math_compare.rs\` (no \`mod\` declaration" - echo "anywhere, so rustc never compiles the file) and 8 in \`bootstrap/src/proxy.rs\`" - echo "(behind \`feature = \"server\"\`, which no workflow enables)." - } >> "$GITHUB_STEP_SUMMARY" - exit $rc - - - name: Explain a unit-test failure - if: failure() && steps.unit_tests.outcome == 'failure' - run: | - echo "::error title=t27c unit tests failed::A Rust test in the t27c crate failed." - echo "" - echo "This is NOT the corpus ratchet and has nothing to do with" - echo "docs/reports/suite_expectations.json. Do not bless an expectation for it." - echo "" - echo "Reproduce locally with:" - echo " cargo test -p t27c --release" - echo "" - echo "The failing test names are in the step log above and in the" - echo "corpus-ratchet-log artifact (unit_tests.log)." + # It should come back as a RATCHET over the 13 named in #2292, never as a + # plain gate: a gate that lands red gets disabled rather than obeyed. + # Do not re-add it without a measurement taken on master. - name: Run the corpus ratchet id: ratchet @@ -163,5 +115,4 @@ jobs: path: | ratchet.log suite_summary.json - unit_tests.log retention-days: 14 diff --git a/docs/NOW.md b/docs/NOW.md index 4fede1877..d02f9dd26 100644 --- a/docs/NOW.md +++ b/docs/NOW.md @@ -1,3 +1,63 @@ +# NOW -- the test gate landed red because the measurement was taken on the wrong tree (2026-08-20) + +Last updated: 2026-08-20 + +## fix(corpus-ratchet): remove the `cargo test -p t27c` step landed 34 minutes ago + +The entry directly below this one is **wrong**, and this is the correction. + +`ce6ea628b` added a `cargo test -p t27c --release` step to `corpus-ratchet.yml` on +the strength of "1221 passed, 0 failed, 0 ignored". That measurement was really +taken and really said that. **It was taken in the wrong working tree.** + +The host checkout sits on `feat/wave-547/host-heapsort`. The gate runs on `master`. +The two trees are nowhere near each other: + +| | `feat/wave-547/host-heapsort` | `origin/master` | +|---|---|---| +| `bootstrap/src/compiler.rs` | 22,142 lines | **36,970 lines** | +| `#[test]` in `bootstrap/src` | 1,781 | 1,622 | +| `cargo test -p t27c` main bin | 895 passed, 0 failed | **1602 passed, 13 failed** | + +The step went red on its first run on master. It is removed here. The 13 failing +test names, and the plan to re-land this as a ratchet rather than a gate, are in +#2292; #2288 and #2289 are reopened. + +Kept from `ce6ea628b`: the `Explain a failure` step stays scoped to +`steps.ratchet.outcome == 'failure'`. As a bare `failure()` it answers a failing +`Build t27c` with "add an entry to `docs/reports/suite_expectations.json`", which +is wrong independently of any test step. + +### Honesty limits (BINDING) + +- **The number in the entry below -- 1221 tests, 0 failures -- describes a tree + that is not master and never was.** On master the same command is **1602 passed, + 13 failed, 2 ignored** in the main binary. Both numbers are real measurements of + different things; only the second one is about the branch that gates merges. +- **13 is a lower bound, not the failing count.** `cargo test` stops at the first + failing target, so the 21 test binaries in `bootstrap/tests/` never ran on master. + Their state is **unmeasured**. A ratchet baseline needs `--no-fail-fast` first. +- **The local verification was real and proved nothing about master.** The step body + was run verbatim from the parsed YAML and exited 0; a shape-identical harness + confirmed it exits non-zero on failure. Both were true, on the wrong tree. Running + a command locally is not evidence about CI unless the checkout matches. +- **This made things worse for 34 minutes, not merely no better.** `corpus-ratchet` + was *already* failing on master at `Run the corpus ratchet` (`eeb779e4c`, + `57a53005a`, `2255e4c32`, `bffd38982`, `400850702`). Putting a failing step in + front of it short-circuited the job, so the pre-existing ratchet failure stopped + being reported at all. +- **`master` has no branch protection.** `GET /repos/gHashTag/t27/branches/master/protection` + returns `404 Branch not protected`. PR #2291 auto-merged at `03:57:13Z`; its own + `corpus-ratchet` run started at `03:57:16Z` -- it merged **three seconds before the + gate it was adding began to run**. Nothing in this repo that calls itself BLOCKING, + including `corpus-ratchet.yml`'s own `W632 (BLOCKING)` header, currently blocks + anything. That is not fixed here and is the larger finding. +- **#2290 stands unaffected** -- `bootstrap/src/math_compare.rs` is still compiled by + nothing, and its `test_hybrid_v2_plateau` still asserts a bound 2.10x tighter than + the file's own goldens. No test was edited in either direction. + +--- + # NOW -- 1221 tests that had never run in CI now run in CI (2026-08-20) Last updated: 2026-08-20