fix(suite): an absent or corrupt gate input must fail, not count as zero - #2287
Merged
Conversation
#2285 gave suite.rs a floor for the TARGET LIST. Two sites of the same shape on the INPUT side were left, and both reported a clean pass when what they read was missing or unreadable. The discipline was already in the file. load_expectations returns Ok(None) for a missing ledger and never an empty one -- "T31 is the bug where a gate treats 'no oracle' as 'pass'" -- and the ratchet turns that None into "RATCHET: FAIL -- Absence is not amnesty (T31)". Neither site followed it. A. Phase 7 catalog gate asked the working directory, not the tree. require_target_file closed the silent skip in #2285, but both operands stayed RELATIVE, so what the gate examined -- and whether the suite aborted -- depended on the cwd the binary was invoked from, and the failure named a relative path that cannot distinguish a wrong cwd from a lost file. Both are now joined onto the canonicalized repo root. The Err arm also still failed open: with the file proven present, reaching it meant the gate could not READ a file that exists, yet it printed one line, left gate_fail untouched, and the block below still printed "(lexer/parser conformance and the catalog gate are all clean)". It now bails. B. A corrupt baseline was byte-identical to a clean one. load_gen_verilog_smoke_baseline collapsed six conditions into HashSet::new() behind an eprintln!: missing file, unreadable file, invalid JSON, no expected_failures key, that key not an array, a non-string entry inside it. The caller set summary.baseline_failures = 0, which feeds summary.acceptable. The tracked baseline legitimately holds "expected_failures": [], so the fail-open path produced exactly the same output as the real file -- the right answer for the wrong reason on every run to date, which is why nothing caught it. It now returns anyhow::Result<Option<HashSet<String>>>: Ok(None) is absent, Err is present-but-unparseable, Ok(Some) is present-and-valid, and only the third may produce a count. That is load_expectations' signature and its reason. The caller decides in the open, and decides FAIL -- the file is tracked, so absence is a broken path, not a configuration. Tests: a_missing_baseline_is_none_not_an_empty_baseline is the counterpart, for the BASELINE, of a_missing_ledger_is_none_not_an_empty_ledger, which has kept absence out of the ORACLE since W628 while nothing kept it out of the baseline. a_corrupt_baseline_is_an_error_not_an_empty_baseline runs four corrupt payloads and asserts each is an Err naming the file. No criterion moved. CATALOG_ALLOWED, catalog_gate::run, the acceptable formula and the baseline file are untouched. With cwd == repo root -- the only configuration these gates have run in -- the resolved paths are identical and no finding changes. Effect on CI today: none. corpus-ratchet.yml runs from the repo root with both files present, so every guard passes and every printed number is identical. Honesty limits: this does not restore the 482 -> 27 coverage #2283 took from phase 3b, does not detect a phase that shrank without reaching zero, and phase 3b still cannot fail CI because it never calls record(...) -- a separate defect, unfixed. Not compiled: parse-checked with rustfmt only, so types and borrows are unproven until CI's cargo build. Closes #2286.
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
gHashTag
enabled auto-merge (squash)
August 20, 2026 03:21
This was referenced Aug 20, 2026
Open
gHashTag
added a commit
that referenced
this pull request
Aug 20, 2026
The cli-tri `build` job has never passed. Not regressed -- never passed: 11 runs on master, 11 failures, and 0 successes across all 56 runs on every branch the workflow has ever run on. ubuntu-latest ships no yosys, and the install sat as a line inside the run block of the final step, which is ordered after `cargo test -p tri`. So fpga::tests::test_smoke_gate_json_synthetic_verify_lean -- which calls the real smoke_gate(), whose verdict ANDs in yosys_ok -- hit "[smoke-gate] SKIP: yosys not on PATH" on every run. A failed step aborts the job, so the install step had never actually executed once. Hoisted into its own named `install yosys` step placed before the test step. The inline copy installed yosys and nothing else, so it is removed in full rather than trimmed; the comment on the final step now says yosys arrives from the step above instead of claiming to install it. Every other step keeps its order and content. cli/tri/src/fpga.rs is untouched. Relaxing the assertion to tolerate a missing binary would have produced a green check that verified no synthesis at all -- the absence-is-not-a-value defect already closed in GH-2285 and GH-2287. The YAML was wrong, not the test. Checked that yosys was the only gap: smoke_gate() spawns no lake, python3 or nextpnr, theorem_matrix_ok is a hardcoded true, and the verify-lean phase is a pure-Rust synthetic fixture. Closes #2302
This was referenced Aug 20, 2026
Closed
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2286.
#2285 gave
bootstrap/src/suite.rsa floor for the target list: a phase with nothing to check now fails, naming where the targets were supposed to come from. Two sites of the same shape on the input side were left, and both still reported a clean pass when what they read was missing or unreadable.The discipline was already in the file and neither site followed it.
load_expectationsreturnsOk(None)for a missing ledger, never an empty one, because "T31 is the bug where a gate treats 'no oracle' as 'pass'", and the ratchet turns thatNoneintoRATCHET: FAIL -- Absence is not amnesty (T31).A. The phase 7 catalog gate asked the working directory, not the tree
require_target_filefrom #2285 closed the silent skip -- the bareif cat.is_file()with noelsethat printedgate failures: 0for a gate that never ran. Two things behind it were left.Both operands were relative.
Path::new("specs/numeric/formats_catalog.t27")andPath::new("specs")resolve against the process cwd, so what this gate examined -- and, after #2285, whether the whole suite aborted -- depended on where the binary happened to be invoked. The failure named a relative path, which cannot tell a wrong cwd from a lost file. Both are now joined onto the already-canonicalizedrepo.The
Errarm still failed open.require_target_fileproves the file is there, so reachingErr(e) => println!(" catalog gate: could not run ({})", e)meant the gate could not read a file that exists. It printed one line, leftgate_failuntouched, and eleven lines later the block still printed(lexer/parser conformance and the catalog gate are all clean). A gate that could not run was announced as a gate that passed. It is now abail!.B. A corrupt baseline was byte-identical to a clean one
load_gen_verilog_smoke_baselinecollapsed six distinct conditions intoHashSet::new()behind aneprintln!: file missing, file unreadable, invalid JSON, noexpected_failureskey, that key not an array, and a non-string entry inside the array. The caller then didsummary.baseline_failures = baseline.len();-- zero -- andbaseline_failuresfeedssummary.acceptable.What made this one invisible:
docs/reports/gen_verilog_smoke_baseline.jsonlegitimately holds"expected_failures": []today. The fail-open path produced exactly the same output as the real file. It has been giving the right answer for the wrong reason on every run since, which is why nothing ever caught it.The loader now returns
anyhow::Result<Option<HashSet<String>>>--Ok(None)is absent,Erris present but unreadable or unparseable,Ok(Some(set))is present and valid, and only the third may produce a count. That isload_expectations' signature andload_expectations' reason. The caller decides what absence means, in the open, and decides FAIL: the file is tracked, so a missing one is a broken path, not a configuration.What changed
load_gen_verilog_smoke_baselineHashSet<String>; six conditions -> empty set +eprintln!anyhow::Result<Option<HashSet<String>>>; absent / corrupt / valid stay distinctsummary.baseline_failures = 0for an absent or corrupt file?on corrupt,bail!naming the absolute path on absentrepo.join(...), sorequire_target_filenames an absolute pathrepo.join("specs")Errarmprintln!and fall through,gate_failunchangedbail!naming the file and the errorThree tests.
a_missing_baseline_is_none_not_an_empty_baselineis the counterpart, for the baseline, ofa_missing_ledger_is_none_not_an_empty_ledger-- which has kept absence out of the oracle since W628 while nothing kept it out of the baseline.a_corrupt_baseline_is_an_error_not_an_empty_baselineruns four payloads and asserts each is anErrwhose message names the file.test_load_gen_verilog_smoke_baselineis updated for the new signature.No criterion moved
CATALOG_ALLOWEDis untouched, sogfternaryis still allowed by name.catalog_gate::runis untouched. Theacceptableformula is untouched. The baseline file is untouched. Under the only configuration in which these gates have ever run -- cwd == repo root --repo.join("specs")andPath::new("specs")resolve to the same directory and not one finding changes.Effect on CI today: none
.github/workflows/corpus-ratchet.ymlinvokest27c suite --repo-root . --ratchet --corpus-onlyfrom the repo root.docs/reports/gen_verilog_smoke_baseline.jsonandspecs/numeric/formats_catalog.t27are both tracked and present, so every new guard passes and every printed number is identical.Unlike the phase 3b failure count, these guards are CI-reachable: they
bail!out ofrun_comprehensivebefore the ratchet block, so they fail the job regardless of--ratchetbypassingtotal_failures.Honesty limits (BINDING)
record(...). Thirteenrecord(...)calls build the ledger the ratchet gates on;gen-verilog-yosys-smokeis not one of them, and with--ratchetthe exit code is the ratchet verdict alone. A separate defect, unfixed here. What is fixed is narrower: the baseline phase 3b reports against can no longer be absent or corrupt without saying so.require_targets("phase 6 integrity metrics + phase 7 catalog gate", ...)is left exactly as it is; phase 7 simply no longer depends on it. The five phase 6 metrics remain cwd-relative -- reporting-only and excluded fromtotal_fail, so out of scope here.suite.rswas parse-checked withrustfmt --edition 2021(no diagnostics), which proves valid Rust syntax and nothing about types or borrows. CI'scargo build --release -p t27cis the first real compile.Files
Two, and only two:
bootstrap/src/suite.rs(+179 / -41) anddocs/NOW.md(+150 / -0, append-only at the top).