Skip to content

ci(ratchet): let the corpus ratchet print its verdict instead of a bare exit code (Closes #2314) - #2315

Merged
gHashTag merged 1 commit into
masterfrom
fix/corpus-ratchet-verdict-2314
Aug 20, 2026
Merged

ci(ratchet): let the corpus ratchet print its verdict instead of a bare exit code (Closes #2314)#2315
gHashTag merged 1 commit into
masterfrom
fix/corpus-ratchet-verdict-2314

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #2314

The check is red, and the log is empty by construction

corpus-ratchet has failed on every master run since #82 (e7ef72bfb,
2026-08-19T21:43:12Z). Job 96381621406 at master head 9068cf49d, lines 6947 and
6948, consecutive, with nothing between them:

##[endgroup]
##[error]Process completed with exit code 1.

Explain a failure then annotates Corpus ratchet failed :: See the Ratchet section above. -- a section that has never once been printed.

Why

The step on master:

run: |
  set -o pipefail
  ./target/release/t27c suite --repo-root . --ratchet --corpus-only \
    --json suite_summary.json > ratchet.log 2>&1
  rc=$?
  sed -n '/--- Ratchet (W628) ---/,$p' ratchet.log
  exit $rc

A run: block with no shell: key executes under bash -e. The failing t27c
aborts the step where it stands, so rc=$?, the sed and the exit $rc execute
only when the ratchet passes -- only when there is nothing to read. On the single
path they were written for, they are dead code.

Verified off CI, both directions

The run: block was pulled out of the YAML by the parser and handed a stub
target/release/t27c. Master's block, against a stub that prints a Ratchet section
and exits 1:

=== MASTER's step, same failing t27c
stdout: ''
stderr: ''
exit: 1

Zero bytes on both streams, exit 1 -- the job log, reproduced. That is the negative
control: the defect is demonstrated, not inferred.

This PR's block, same harness, three stubs:

stub t27c printed step exit
verdict, exits 1 the Ratchet section through Error: RATCHET FAILED 1 (want 1)
verdict, exits 0 the Ratchet section 0 (want 0)
panics before the verdict, exits 101 "no Ratchet section", then tail -40 101 (want 101)

The exit code is carried through unchanged on all three paths, so this changes what
the job says and never what it concludes.

|| rc=$? was chosen over set +e or a shell: override because it moves exactly
one command off the -e abort path and leaves the rest of the step strict.

This does not make the check green, and must not

The ratchet is red for a real reason. Until now it lived only in ratchet.log inside
the corpus-ratchet-log artifact -- 287,391 bytes, 14-day retention:

--- Ratchet (W628) ---
  ledger:              221 / 221 cap
  observed (primary):  220
  UNEXPECTED FAILURES: 2
    + specs/fpga/power_analysis.t27 [parse-no-discard]
    + specs/fpga/vcd_conformance_compare.t27 [parse-no-discard]
  UNEXPECTED PASSES  : 3
    - specs/fpga/power_analysis.t27 [parse] (fixed -- remove from the ledger)
    - specs/fpga/vcd_conformance_compare.t27 [parse] (fixed -- remove from the ledger)
    - specs/tri/collections/array.t27 [parse] (fixed -- remove from the ledger)
  EXPIRED ENTRIES    : 0
RATCHET: FAIL

e7ef72bfb made two specs parse, which un-blocked the phases behind parse:

FAIL parse-no-discard (specs/fpga/power_analysis.t27): DISCARDED 3 top-level token(s)
FAIL parse-no-discard (specs/fpga/vcd_conformance_compare.t27): DISCARDED 120 top-level token(s)
FAIL no-vacuous-invariant (specs/fpga/power_analysis.t27): 1 invariant(s) declared but not lowered
FAIL no-vacuous-invariant (specs/fpga/vcd_conformance_compare.t27): 9 invariant(s) declared but not lowered

The mechanical route exists and needs no cap raise -- drop the 3 fixed rows, add the
2 new ones, land at 220 under the 221 cap, then lower max_entries. It is not
taken here
, because it would bless a spec that declares ten invariants and checks
none of them. Bless-versus-fix is a maintainer's decision, left open on #2314.

corpus-ratchet is not a required context, so this PR can merge while its own
ratchet job is red -- which is the point. Its job on this PR runs the new step
(pull_request workflows execute from the PR head), so this PR's own red ratchet run
is the live demonstration: it will print the verdict above instead of nothing.

Diff

file change
.github/workflows/corpus-ratchet.yml +22 / -3 -- the step, plus a comment recording the trap
docs/NOW.md +110 / -0 -- new top entry, pure prepend

No threshold moved, no assertion deleted, no baseline widened, no skip added.

…re exit code

The step's `rc=$?`, `sed` and `exit $rc` are dead code. A `run:` block with no
`shell:` key executes under `bash -e`, so the failing `t27c` aborts the step
where it stands and those three lines run only when the ratchet passes -- only
when there is nothing to read. The job log has therefore been `##[endgroup]`
followed immediately by `##[error]Process completed with exit code 1.` for the
step's entire history, while `Explain a failure` annotated "See the Ratchet
section above" for a section never printed. The verdict survived only in the
uploaded artifact, at 14-day retention.

`|| rc=$?` moves the failure off the `-e` abort path. A missing verdict now
prints the log tail rather than falling back to a bare exit code a second time.

Verified off CI by running the YAML-parsed block under `bash -e` against a stub
`target/release/t27c`: master's block emits zero bytes on both streams and exits
1, reproducing the job log; the new block prints the Ratchet section and exits 1,
prints it and exits 0 on success, and prints the tail and exits 101 on a panic
before the verdict. The outcome is carried through unchanged on all three paths.

This does not turn the check green and must not: the ratchet is red because two
specs now reach `parse-no-discard` and `no-vacuous-invariant`, which report 123
discarded top-level tokens and 10 declared-but-unlowered invariants. Whether to
bless that in the ledger or fix the specs is a maintainer's decision and is left
open.

Closes #2314
@gHashTag
gHashTag enabled auto-merge (squash) August 20, 2026 10:08
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-20 10:09:00 UTC

Summary

Status Count
Total Open PRs 21
PRs with Failing Checks 6
PRs with All Checks Green 15
READY 7
FAILING 6
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=cbbfac87dff3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit e09714d into master Aug 20, 2026
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Corpus ratchet step cannot print its verdict: rc/sed/exit are dead code under bash -e

1 participant