Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 10 additions & 59 deletions .github/workflows/corpus-ratchet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,65 +59,17 @@ jobs:
- name: Build t27c
run: cargo build --release -p t27c

# Closes #2288. `cargo build` does not compile `#[cfg(test)]` modules, so
# eight workflows on master build t27c and not one of them ever compiled,
# let alone ran, a single test in it. Measured 2026-08-20: 1221 tests
# exist and 1221 of them had never executed in CI. They all pass -- so
# this lands as a plain gate, not a ratchet.
# A `cargo test -p t27c` step lived here for 34 minutes on 2026-08-20
# (ce6ea628b) and is removed again by #2292. It was landed on a measurement
# of 1221 passing tests taken in a working tree checked out at
# feat/wave-547/host-heapsort, NOT at master -- that branch's
# bootstrap/src/compiler.rs is 22,142 lines against master's 36,970. On
# master the same command is 1602 passed / 13 failed / 2 ignored, so the
# step went red on its first run here.
#
# It goes here because the crate is already built one step above, so the
# marginal cost is compiling the test targets and running them: measured
# 1m43s cold / 1.75s warm, against this job's 45-minute ceiling and its
# ~314s ratchet run.
#
# Scope, stated openly -- this runs what `cargo test -p t27c` REACHES,
# which is not every test in the tree. 913 `#[test]` attributes live in
# bootstrap/src; 895 compile into this run. The missing 18:
# * bootstrap/src/math_compare.rs -- 10 tests, and no `mod math_compare;`
# exists anywhere in the crate, so rustc never opens the file. Built
# standalone it is 9 pass / 1 fail (see #2288 follow-up); the failure
# is a bad assertion, not a bad implementation, so it is NOT silenced
# here and NOT wired in here -- wiring it in would land this gate red.
# * bootstrap/src/proxy.rs -- 8 tests behind
# `#[cfg(all(test, feature = "server"))]`. No workflow passes
# `--features server`, so they are stripped before type-checking.
# Neither hole is created by this step; both are named so the green tick
# is not read as "every test in bootstrap/src ran".
- name: Unit tests (cargo test -p t27c)
id: unit_tests
run: |
set -o pipefail
cargo test -p t27c --release 2>&1 | tee unit_tests.log
rc=$?
totals=$(grep -E '^test result:' unit_tests.log \
| awk '{p+=$4; f+=$6; i+=$8} END {printf "%d passed, %d failed, %d ignored", p, f, i}')
bins=$(grep -cE '^test result:' unit_tests.log)
echo ""
echo "t27c unit tests: ${totals} across ${bins} test binaries."
{
echo "### t27c unit tests"
echo ""
echo "\`cargo test -p t27c --release\` -- **${totals}** across ${bins} test binaries."
echo ""
echo "Not covered: 10 tests in \`bootstrap/src/math_compare.rs\` (no \`mod\` declaration"
echo "anywhere, so rustc never compiles the file) and 8 in \`bootstrap/src/proxy.rs\`"
echo "(behind \`feature = \"server\"\`, which no workflow enables)."
} >> "$GITHUB_STEP_SUMMARY"
exit $rc

- name: Explain a unit-test failure
if: failure() && steps.unit_tests.outcome == 'failure'
run: |
echo "::error title=t27c unit tests failed::A Rust test in the t27c crate failed."
echo ""
echo "This is NOT the corpus ratchet and has nothing to do with"
echo "docs/reports/suite_expectations.json. Do not bless an expectation for it."
echo ""
echo "Reproduce locally with:"
echo " cargo test -p t27c --release"
echo ""
echo "The failing test names are in the step log above and in the"
echo "corpus-ratchet-log artifact (unit_tests.log)."
# It should come back as a RATCHET over the 13 named in #2292, never as a
# plain gate: a gate that lands red gets disabled rather than obeyed.
# Do not re-add it without a measurement taken on master.

- name: Run the corpus ratchet
id: ratchet
Expand Down Expand Up @@ -163,5 +115,4 @@ jobs:
path: |
ratchet.log
suite_summary.json
unit_tests.log
retention-days: 14
60 changes: 60 additions & 0 deletions docs/NOW.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,63 @@
# NOW -- the test gate landed red because the measurement was taken on the wrong tree (2026-08-20)

Last updated: 2026-08-20

## fix(corpus-ratchet): remove the `cargo test -p t27c` step landed 34 minutes ago

The entry directly below this one is **wrong**, and this is the correction.

`ce6ea628b` added a `cargo test -p t27c --release` step to `corpus-ratchet.yml` on
the strength of "1221 passed, 0 failed, 0 ignored". That measurement was really
taken and really said that. **It was taken in the wrong working tree.**

The host checkout sits on `feat/wave-547/host-heapsort`. The gate runs on `master`.
The two trees are nowhere near each other:

| | `feat/wave-547/host-heapsort` | `origin/master` |
|---|---|---|
| `bootstrap/src/compiler.rs` | 22,142 lines | **36,970 lines** |
| `#[test]` in `bootstrap/src` | 1,781 | 1,622 |
| `cargo test -p t27c` main bin | 895 passed, 0 failed | **1602 passed, 13 failed** |

The step went red on its first run on master. It is removed here. The 13 failing
test names, and the plan to re-land this as a ratchet rather than a gate, are in
#2292; #2288 and #2289 are reopened.

Kept from `ce6ea628b`: the `Explain a failure` step stays scoped to
`steps.ratchet.outcome == 'failure'`. As a bare `failure()` it answers a failing
`Build t27c` with "add an entry to `docs/reports/suite_expectations.json`", which
is wrong independently of any test step.

### Honesty limits (BINDING)

- **The number in the entry below -- 1221 tests, 0 failures -- describes a tree
that is not master and never was.** On master the same command is **1602 passed,
13 failed, 2 ignored** in the main binary. Both numbers are real measurements of
different things; only the second one is about the branch that gates merges.
- **13 is a lower bound, not the failing count.** `cargo test` stops at the first
failing target, so the 21 test binaries in `bootstrap/tests/` never ran on master.
Their state is **unmeasured**. A ratchet baseline needs `--no-fail-fast` first.
- **The local verification was real and proved nothing about master.** The step body
was run verbatim from the parsed YAML and exited 0; a shape-identical harness
confirmed it exits non-zero on failure. Both were true, on the wrong tree. Running
a command locally is not evidence about CI unless the checkout matches.
- **This made things worse for 34 minutes, not merely no better.** `corpus-ratchet`
was *already* failing on master at `Run the corpus ratchet` (`eeb779e4c`,
`57a53005a`, `2255e4c32`, `bffd38982`, `400850702`). Putting a failing step in
front of it short-circuited the job, so the pre-existing ratchet failure stopped
being reported at all.
- **`master` has no branch protection.** `GET /repos/gHashTag/t27/branches/master/protection`
returns `404 Branch not protected`. PR #2291 auto-merged at `03:57:13Z`; its own
`corpus-ratchet` run started at `03:57:16Z` -- it merged **three seconds before the
gate it was adding began to run**. Nothing in this repo that calls itself BLOCKING,
including `corpus-ratchet.yml`'s own `W632 (BLOCKING)` header, currently blocks
anything. That is not fixed here and is the larger finding.
- **#2290 stands unaffected** -- `bootstrap/src/math_compare.rs` is still compiled by
nothing, and its `test_hybrid_v2_plateau` still asserts a bound 2.10x tighter than
the file's own goldens. No test was edited in either direction.

---

# NOW -- 1221 tests that had never run in CI now run in CI (2026-08-20)

Last updated: 2026-08-20
Expand Down
Loading