fix(tri): rank baseline holes by failure history, not by trigger table (Closes #2359)

[gHashTag]

Closes #2359. Corrects #2309, merged yesterday as #2312.

That sweep's premise was: a gate with no default-branch baseline can turn a pull request red while no green state has ever existed. Measured, it does not hold for all four it reported.

workflow	runs	failures
emit-bitexact-gate	144	84
check-now-freshness	1442	2
seal-staleness-warn	169	0
loop-tools-gate	12	0

seal-staleness-warn is advisory by construction — its own header says so, and every path through its script ends in exit 0. It appears in the check list and cannot block anything. Reporting it beside emit-bitexact — 84 failures, no baseline, the gate that actually blocked two merges — presented four findings that are not the same size.

The trigger table was never the signal; whether the workflow has ever concluded failure is. The command now measures that per hole, prints it, sorts loudest-first, and --strict alarms only on gates observed to fail. A fourth kind joins the taxonomy: advisory by construction.

  of those, never run on master by any event: 2
     check-now-freshness.yml
       1442 run(s), 2 failure(s) — CAN and DOES fail — no green state has ever been observed
     seal-staleness-warn.yml
       169 run(s), 0 failure(s) — has never failed in its whole history; a missing baseline costs nothing yet

Two of yesterday's four have left the list on their own since: catalog-count-gate (dispatched on master, which gave it the baseline it never had) and loop-tools-gate (its push: paths finally changed there). The sweep is live, not a snapshot.

Third correction in this line, and they rhyme: triggers alone said 47, run counts alone said 46, the intersection said 4, and the intersection ranked by failure history says one alarm and one footnote. Every stage was confident and every stage was too coarse.

🤖 Generated with Claude Code

[github-actions]

📓 NotebookLM Notebook linked to this PR

• Notebook ID: b83263109fb055dc
• URL: https://notebooklm.google.com/notebook/b83263109fb055dc

This notebook contains session context, decisions, and artifacts for this work.

[github-actions]

PR Dashboard

Generated at: 2026-08-21 18:05:35 UTC

• fix(tri): rank baseline holes by failure history, not by trigger table (Closes #2359) #2360 fix(tri): rank baseline holes by failure history, not by trigger table (Closes #2359) | Created: 2026-08-21 | Updated: 2026-08-21 | 15 failing / 27 total
• corpus: freeze the damage, patch one class at a time, and say where one rule stops #2161 corpus: freeze the damage, patch one class at a time, and say where one rule stops | Created: 2026-08-14 | Updated: 2026-08-21 | 0 failing / 9 total
• Unbreak FPGA CI and correct two unbacked hardware claims #2081 Unbreak FPGA CI and correct two unbacked hardware claims | Created: 2026-08-11 | Updated: 2026-08-20 | 5 failing / 27 total

Summary

Status	Count
Total Open PRs	3
PRs with Failing Checks	2
PRs with All Checks Green	1
READY	1
FAILING	2
PENDING	0

Seal Status

• ⚠️ STALE -- sha256(compiler.rs)=65f033d04125 != manifest seal=87e5cbd3ad94.
  The committed NMSE numbers were certified against an older compiler.rs.
  Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

[gHashTag]

One inaccuracy in the second commit message, corrected here rather than by rewriting history: it says git add docs/NOW.md "matched nothing". It matched — docs/NOW.md still exists — but it is now a frozen archive whose first line says so, and new entries go to docs/now/*.md. So the add was a no-op because the file was unmodified, not because the path was missing. The substance stands: the freshness note was written and silently left out of the commit.

[gHashTag]

Review: not merging yet. The premise checks out; the gate relaxation that rides with it is untested and fails open twice.

Reviewed at head cafa94c3116f6245d2b55bf7f3b7f831f14edb3f. I built that exact file standalone (sha256 of the fetched cli/tri/src/cibase.rs = f2a944d5ba4e5151aa3938db73d801e3d3097b1cd0f333f2a5e5b4071c7874e2) and mutated it. Findings below are executed, not read.

What is right, and it is most of the PR

The central claim reproduces. Re-derived today against the live API, independently of the PR:

workflow	body claims	measured 2026-08-21
emit-bitexact-gate.yml	144 runs / 84 fail	145 / 86
check-now-freshness.yml	1442 / 2	1443 / 2
seal-staleness-warn.yml	169 / 0	169 / 0 exact
loop-tools-gate.yml	12 / 0	12 / 0 exact

Drift on the top two is ordinary elapsed-time drift and moves in the direction that strengthens the argument.

"Advisory by construction" verifies against the source, not just the run count. .github/workflows/seal-staleness-warn.yml says so in its own header at lines 9-10 — that it emits a warning annotation, always exits 0, never blocks merge, and is not in the required-check set — and every branch of the script ends exit 0 (lines 62, 67, 78). The correction to #2309 is real.

The reported list is not truncated. I looked hard for the failure mode where ranking quietly becomes filtering. There is no take, no truncate, no cap: for h in &holes walks all of them and holes.len() is printed above. No hole that #2309 reported is ranked out of view. Credit — that is the trap this change could easily have fallen into and did not.

Finding 1 — the new test does not touch the new code

a_gate_that_has_never_failed_ranks_below_one_that_has re-implements the two production lines inside its own body and asserts against its own local vector:

• production sort, line 194: holes.sort_by(|a, b| b.fails.cmp(&a.fails));
• test's private copy, line 293: holes.sort_by(|a, b| b.fails.cmp(&a.fails));
• production loud, line 220: let loud = holes.iter().filter(|h| h.fails > 0).count();
• test's private copy, line 302: let loud = holes.iter().filter(|h| h.fails > 0).count();

Mutant 1 — invert the production sort at line 194 to a.fails.cmp(&b.fails), so the command prints the quietest hole first, the exact inverse of the title:

running 3 tests
test tests::a_gate_that_has_never_failed_ranks_below_one_that_has ... ok
test result: ok. 3 passed; 0 failed

Survived. A survivor is a lead, not a verdict, so — Mutant 2, invert the test's own copy at line 293 with production untouched:

thread 'tests::a_gate_that_has_never_failed_ranks_below_one_that_has' panicked at src/lib.rs:294:9:
assertion `left == right` failed
  left: "seal-staleness-warn.yml"
 right: "emit-bitexact-gate.yml"
test result: FAILED. 2 passed; 1 failed

So the assertion is live and Mutant 1 is not an equivalent mutant — line 194 is observably load-bearing (on this repo the holes carry fails of 86, 2 and 0, all distinct, so inverting it genuinely reorders real output). The guard simply never reaches production. As written it tests that std::slice::sort_by sorts, which was not in doubt.

Finding 2 — --strict is wholly untested, and it is the part whose behaviour changed

--strict went from "bail if any hole exists" to if strict && loud > 0. Mutant 3 replaces line 244 with if false {, making --strict a permanent no-op:

test tests::pr_trigger_is_found_in_every_spelling ... ok
test tests::a_gate_that_has_never_failed_ranks_below_one_that_has ... ok
test tests::base64_round_trips_a_workflow_body ... ok
test result: ok. 3 passed; 0 failed

Survived. The only signal was warning: unused variable: strict from rustc — no test noticed.

Finding 3 — a failed API call is silently reported as "has never run anywhere at all", and exempts the hole from --strict

num() returns -1 on any unparseable input, and the call sites feed it gh(...).unwrap_or_default() — where gh() bails on a non-zero exit, so an API failure becomes "":

let fails = num(&gh(&[...]).unwrap_or_default());

Probing production num() directly:

test probe::what_num_returns_when_the_api_call_failed ... ok

asserting num("") == -1, and that -1 > 0 is false.

-1 is then consumed by two > 0 tests and loses both:

• line 220 filter(|h| h.fails > 0) — the hole is not counted in loud, so --strict exits 0 where the previous version exited non-zero. One transient 502 or rate-limit downgrades every hole to a silent pass.
• lines 200-206 — fails > 0 false, then runs > 0 false, so it prints has never run anywhere at all. That is a false statement of fact about a workflow whose history could not be read. A measurement failure is being rendered as a measured result, which is the same class of error tri ci baseline: which PR gates have never run on the branch they gate #2309 made.

Finding 4 — runs == 0, fails == 0 is exempted too

A gate that has never run anywhere is the strongest possible case of "no green state has ever been observed", and old --strict caught it. It is now below the alarm threshold. The code already knows these three states are different — the verdict text distinguishes them — but the strict decision collapses two of them into the exemption earned by only the third (169 runs, 0 failures is evidence; 0 runs is the absence of evidence).

Finding 5 — the --strict help text now states the old contract

Line 51, untouched by this diff (gh pr diff | grep -c returns 0):

/// Non-zero exit if any gate has no baseline. For use in CI.

That is no longer what it does.

Severity, stated honestly

Nothing in the repo invokes tri ci baseline --strict — the only match for ci baseline across the tree is cli/tri/src/cibase.rs itself. So this weakening is latent, not live: no CI job changes colour today. That is why this is "leave it open with specifics" and not "revert". But --strict says "For use in CI", and the moment it is wired up it will be wired up in the fail-open state.

Also, minor: when loud >= 2 the output reads 2 of them has gone red before.

What would change my mind

1. loud distinguishes the three states the verdict already distinguishes. Concretely, -1 must never rank or count as quieter than 0 — unknown history is not a clean history. Either keep unknown/never-run in the alarm, or give it its own non-zero exit; do not merge it into the exemption.
2. Don't print has never run anywhere at all when runs == -1. Say the history could not be read.
3. Update line 51 to the new contract.
4. Give the guard something that bites: lift the ranking and the loud decision into fn rank(&mut [Hole]) and fn loud(&[Hole]) -> usize, have the test call those, then re-run Mutant 1 and Mutant 3. They should die. Happy to hand back the exact mutants I used.

Findings 1 and 2 are the ones I would not merge without. Findings 3-5 are cheap once the guard is real, and 3 is the one that will bite in production.

Point 4 of the ask — the numbers — passes cleanly, and so does the no-truncation check. This is a good correction with a guard that has not caught up to it yet.