Skip to content

fix(tri): an unreadable history is not a zero, and the guard now reaches the command - #2367

Merged
gHashTag merged 1 commit into
masterfrom
fix/cibase-strict-unknown-history
Aug 21, 2026
Merged

fix(tri): an unreadable history is not a zero, and the guard now reaches the command#2367
gHashTag merged 1 commit into
masterfrom
fix/cibase-strict-unknown-history

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Repairs the three defects #2360 merged into master at cb446ea128c7b60cee6e100a49300d227455993d. Nothing is reverted. #2360's ranking idea is right and its other measurements reproduce exactly against the live API: check-now-freshness 1446/2, seal-staleness-warn 169/0, loop-tools-gate 12/0, catalog-count-gate 37/0.

Everything below was re-verified against origin/master at dfdc7ed642970365e91cd67f17ebfa2eb772eb4b, not taken from the review.


1. --strict failed open on unknown history, and said so out loud

On master, cli/tri/src/cibase.rs:

fn num(s: &str) -> i64 {
    s.trim().parse().unwrap_or(-1)          // line 112
}
...
let runs  = num(&gh(&[...]).unwrap_or_default());   // lines 168-173
let fails = num(&gh(&[...]).unwrap_or_default());   // lines 174-179

gh() bails on any non-zero exit, so a rate limit, a 502 or an expired token becomes "", and num("") == -1. That -1 then loses both > 0 tests it meets:

  • let loud = holes.iter().filter(|h| h.fails > 0).count(); (line 220) — the hole drops out, so --strict exits 0 where the pre-fix(tri): rank baseline holes by failure history, not by trigger table (Closes #2359) #2360 version exited non-zero. One transient API failure downgrades every hole in the sweep at once.
  • the verdict ladder (lines 200-206) falls through fails > 0, then runs > 0, and prints has never run anywhere at all — a false statement of fact about a workflow whose history could not be read.

An unreadable history is not a zero. Counts are Option<i64> now; None means "the API did not say" and propagates all the way to the exit code. The verdict for it is history could not be read — UNMEASURED, which is not the same as clean.

runs == 0, fails == 0 — a gate never run anywhere, the strongest form of "no green state has ever been observed" — was also exempt from --strict and no longer is. The one exemption that was actually earned survives: a gate that HAS run, often, and has never once concluded failure.

The same swallow existed one level up and is closed too. An unreadable default-branch run count used to mean "this workflow has a baseline", which deleted the finding outright. Unreadable workflows are now printed under workflows the sweep could not measure and counted toward --strict. Deliberate exception, measured: a 404 from the contents API is an answer — 12 of this repository's 54 registered active workflows have no file on master at all, and a workflow that is not on the default branch gates nothing there. Fetch::Absent keeps that apart from Fetch::Failed.

The doc comment was left untouched by #2360 and had been false since it merged:

/// Non-zero exit if any gate has no baseline. For use in CI.

Replaced with what --strict actually does.


2. The guard tested std::slice::sort_by, not tri ci baseline

a_gate_that_has_never_failed_ranks_below_one_that_has copied production line 194 into its own line 293 and production line 220 into its own line 302, then asserted against its own local vector.

BITING — two mutants planted in master's shipped file, both survived

Mutant A — invert the production sort at line 194, leave the test's copy at line 293 alone:

194:    holes.sort_by(|a, b| a.fails.cmp(&b.fails));
293:        holes.sort_by(|a, b| b.fails.cmp(&a.fails));
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 166 filtered out

Mutant Bif strict && loud > 0 becomes if false, making --strict a permanent no-op:

244:    if false {
warning: unused variable: `strict`
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 166 filtered out

ALIVE — the assertion is live, it is just aimed at its own copy

A surviving mutant is a lead, not a verdict, so it was composed with a genuine break: invert the test's own copy at line 293, production untouched.

thread 'cibase::tests::a_gate_that_has_never_failed_ranks_below_one_that_has' panicked at cli/tri/src/cibase.rs:294:9:
assertion `left == right` failed
  left: "seal-staleness-warn.yml"
 right: "emit-bitexact-gate.yml"
test result: FAILED. 2 passed; 1 failed

So Mutant A is not an equivalent mutant. The assertion responds to a sort inversion — just not the production one.

TRUE — and the decisive check

struct Hole plus the guard's verbatim body, lifted into a standalone file containing no production code at all, compiles and passes:

running 1 test
test a_gate_that_has_never_failed_ranks_below_one_that_has ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

A guard that cannot reach the code it claims to guard is not a guard.

The repair, and the same mutants dying

rank, severity, verdict, alarming, strict_verdict, count and is_not_found are lifted out of the command body, and the tests call them.

Mutant A′ (production rank inverted) now kills two tests:

thread 'cibase::tests::unknown_and_never_ran_outrank_a_measured_zero' panicked at cli/tri/src/cibase.rs:540:9:
assertion `left == right` failed
  left: ["measured-zero.yml", "never-ran.yml", "unreadable.yml", "goes-red.yml"]
 right: ["goes-red.yml", "unreadable.yml", "never-ran.yml", "measured-zero.yml"]

thread 'cibase::tests::a_gate_that_has_never_failed_ranks_below_one_that_has' panicked at cli/tri/src/cibase.rs:512:9:
assertion `left == right` failed
  left: "catalog-count-gate.yml"
 right: "check-now-freshness.yml"
test result: FAILED. 5 passed; 2 failed

Mutant B′ (strict_verdict forced to None, the exact analogue of if false):

thread 'cibase::tests::strict_fires_on_red_on_never_ran_and_on_unmeasured' panicked at cli/tri/src/cibase.rs:589:9:
assertion failed: strict_verdict(true, &red, &[], "master").is_some()
test result: FAILED. 6 passed; 1 failed

The same production-free extraction now yields 38 compile errors naming 10 distinct production items (rank, severity, verdict, alarming, strict_verdict, count, is_not_found, is_pr_gated, decode_b64, and the Severity type) and no binary.

Consumption proved separately

Unit tests calling a function do not prove the command calls it. Repaired build against the live API:

$ tri ci baseline --repo gHashTag/t27 --strict ; echo "EXIT=$?"
     check-now-freshness.yml
       1446 run(s), 2 failure(s) — CAN and DOES fail — no green state has ever been observed
     seal-staleness-warn.yml
       169 run(s), 0 failure(s) — has never failed in its whole history; a missing baseline costs nothing yet
Error: 1 PR gate(s) have no baseline on master and cannot be shown to be harmless (failed before, never ran anywhere, or history unreadable)
EXIT=1

Same build with production rank inverted and strict_verdict forced to None:

     seal-staleness-warn.yml     <- order flipped
     check-now-freshness.yml
EXIT=0                            <- no Error line at all

baseline() really does call both.


3. #2360's central motivating fact was false, and it is in master's source

The PR body's table puts emit-bitexact-gate at the head of "the four holes" with 144 runs, 84 failures, no baseline. The "no baseline" half was not true when it was written:

$ gh api repos/gHashTag/t27/actions/workflows/329310174/runs?branch=master
2026-08-20T01:05:13Z    workflow_dispatch    failure    99e9098153c3

That is total_count: 1, a workflow_dispatch on master 41 hours before #2360 opened (2026-08-20T01:05:13Z vs 2026-08-21T18:05:14Z). #2309 says so in its own body:

The fourth, catalog-count-gate.yml, declared workflow_dispatch; one dispatch on master gave it a baseline and it left the list — which is also how emit-bitexact acquired the baseline it had never had.

And #2360's own pasted output listed 2 holes and did not include it, contradicting its table three lines above. The repaired command reproduces that live: the two holes today are check-now-freshness and seal-staleness-warn.

#2309's real four were catalog-count-gate, check-now-freshness, loop-tools-gate, seal-staleness-warn, measuring 37/0, 1446/2, 12/0 and 169/0. Exactly one has ever concluded failure, and it did so twice. The ranking argument still holds — it just rests on 2 failures versus 0, not 84 versus 0.

The run and failure counts themselves are real; live now they are 147/88.

The falsehood was in three places, all corrected:

  1. the cibase.rs module doc,
  2. docs/now/2026-08-22-tri-ci-baseline-ranks-by-failure-history-not-by-trigger-tabl.md,
  3. the test's doc-comment plus its Hole { file: "emit-bitexact-gate.yml", … } fixture asserted to rank first.

The NOW entry is a dated record, so the correction is appended under its own heading and the original two bullets are left standing verbatim, with a line saying nothing above was edited. docs/NOW.md:1517 also mentions emit-bitexact with "0 master runs ever", but that entry is dated 2026-08-20 and was true when written, so it is left alone.


Severity: latent, not live

The only reference to ci baseline anywhere in the tree is cibase.rs itself — no workflow invokes it, so no job changes colour today. This is not a live CI failure and should not be read as one. It matters because the flag says "For use in CI" and would have been wired up in the fail-open state.

Verification

cargo test -p tri173 passed; 0 failed. cibase alone: 7 passed, up from 3. Cargo.lock is unchanged.

Closes #2361

…hes the command

Repairs three defects #2360 merged into master. Its ranking idea is right and
its other measurements reproduce exactly, so nothing is reverted.

1. --strict failed open on unknown history. gh() bails on any non-zero exit, so
   a rate limit or a 502 became "", num("") returned -1, and -1 lost both > 0
   tests it met: the hole dropped out of the strict count AND the verdict ladder
   printed "has never run anywhere at all" about a workflow nobody had measured.
   Counts are Option<i64> now and None propagates to the exit code. A gate that
   has never run anywhere (0 runs, 0 failures) is no longer exempt either. The
   one exemption that was earned survives: a gate that HAS run and has never
   once concluded failure.

   The same swallow one level up is closed too: an unreadable default-branch run
   count used to mean "has a baseline", which deleted the finding. A 404 from
   the contents API stays a real answer -- 12 of this repo's 54 registered
   active workflows have no file on master -- so Fetch::Absent is kept apart
   from Fetch::Failed.

2. The guard constrained std::slice::sort_by, not tri ci baseline. It copied the
   production sort and the loud predicate into its own body. Inverting the
   PRODUCTION sort survived (3 passed; 0 failed); so did replacing
   `if strict && loud > 0` with `if false`. rank, severity, verdict, alarming,
   strict_verdict, count and is_not_found are lifted out of the command body and
   the tests call them. Both mutants now die.

3. emit-bitexact was never one of #2309's four holes. It has had a master
   baseline since 2026-08-20T01:05:13Z, 41 hours before #2360 opened, and #2309
   says so in its own body. The claim is corrected in the module doc, the test
   doc-comment and its fixture; the NOW entry is a dated record, so a correction
   is appended rather than the claim edited away.

Latent, not live: cibase.rs is the only reference to `ci baseline` in the tree,
so no job changes colour today.

Closes #2361
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-21 20:33:04 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 1
PRs with All Checks Green 2
READY 1
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=65f033d04125 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag enabled auto-merge (squash) August 21, 2026 20:33
@gHashTag
gHashTag merged commit 6dcefd0 into master Aug 21, 2026
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tri ci baseline --strict fails open on unknown history; the guard from #2360 tests sort_by, not the command

2 participants