feat(pricing): monitor the rate table against two public sources

[c-1k]

What

PRICING_TABLE ships to customers in the published usertrust package and had no drift monitoring of any kind. PRICING_TABLE_VERSION is a hand-typed constant; nothing fetched, compared, or expired.

This matters beyond hygiene. receipt.pricing.appliedRates + tableVersion are what let a third party recompute a metered cost from the record alone — check out the stamped table version, recompute ceil(sum(counts × rates / 1000)), floored at 1. That the table is Apache-2.0 public source is what makes independent verifiability real rather than rhetorical. So a stale rate produces receipts that are internally consistent, cryptographically sound, and externally wrong — and the hash chain preserves them faithfully.

Measured against the shipped table

Re-measured live 2026-08-23T20:48:02Z against PRICING_TABLE_VERSION 2026-08-10:

Outcome	Count
Exact agreement, all four tiers	15
Expected deviation (ours higher)	1 — claude-sonnet-5
Sources conflict with each other	3 — deepseek-chat, deepseek-reasoner, gpt-5.6-sol
No upstream source at all	7
Total	26

Coverage 36/36 source→model mappings, across 19/19 mapped models. Exit status is 0.

Nothing in the table is understated. One entry is now known stale in the safe direction: gpt-5.6-sol carries 50/300 while OpenAI publishes 40/200 promotional through 2026-11-21. Report-only stays report-only, so that rate is adjudicated in #142 rather than changed here.

Five rules carry the weight

Vendor pinning. models.dev carries 190 providers, most of them resellers. An unpinned matcher answered claude-fable-5 at 30/185 against a true 100/500, and claude-opus-5 at 55/275 (a regional premium) — every one well-formed and wrong. Both sources are pinned to the vendor; a non-vendor row is treated as absent, never as an answer. Mapping is explicit per model, because command-a would otherwise fuzzy-match command-a-translate-08-2025.

Deviations only in the safe direction. Both sources publish claude-sonnet-5 at $2/$10 — the introductory rate through 2026-08-31 that pricing.ts deliberately does not carry. So "differs from upstream" must not mean "we are wrong", or the monitor's first act would be to undo a deliberate conservative choice. An allowlist entry suppresses failure only while our rate is ≥ upstream; if upstream ever rises above ours it fails despite the entry. Understatement is never allowlistable — which enforces the D1 invariant rather than restating it.

Exit 2 is never 0. "Could not check" and "checked, found nothing" are different states. A coverage floor derived from the map (not a transcribed constant, so it self-maintains) turns a silent upstream schema change into a failure instead of a clean sweep of nothing.

Only agree is silent. Every model gets exactly one outcome, the counts must sum to the table size, and every other outcome reports by construction — including the 7 models with no external check, which is the most useful thing the report says.

Catalogs vote once, not N times. LiteLLM and models.dev are not independent: both derive from the providers' published pages and both lag promotional changes, so N of them agreeing is one upstream fact with N mirrors. Measured 2026-08-22, both catalogs and this table priced gpt-5.6-sol at 50/300 while OpenAI's page said 40/200 — acting on "three-way agreement" would have overcharged output by 50% on the most-used model, with three sources apparently confirming it. Within hours LiteLLM moved to 40/200 while models.dev did not, so the agreement was visibly a timing artefact. The report now carries evidence, a count of distinct evidence tiers under an explicit billed > provider > catalog precedence, and reads "litellm, models.dev — 1 evidence tier" instead of implying two confirmations.

Testing

78 tests, no network. The suite carries a positive control (a rate mutated below upstream that must be reported) beside a negative control — an all-pass suite cannot detect an instrument that is disconnected. The exit-2 path is tested by stubbing fetch, not by blocking the network: Node's fetch ignores https_proxy, so a proxy-based probe succeeds and proves nothing.

The unit conversion is pinned to a real table entry (PRICING_TABLE["claude-opus-5"].inputPer1k) rather than a literal, because assert.equal(convert(5), 5) passes against a broken identity conversion and locks the error in.

Scope

Report-only: contents: read means GitHub refuses a push, so no upstream value can reach pricing.ts through this path. Does not touch the receipt shape, TrustReceipt, or shared/types.ts. No cross-repo token.

Pure decision-rule tests run in ci.yml; the live comparison stays weekly so no network fetch can flake a PR.

Deferred findings (not in this PR)

1. resolveRates prefix-matching understates silently — tracked in pricing: resolveRates prefix match understates silently and reports rateSource "table" #143. claude-opus-5-fast matches claude-opus-5, prices at standard rates (2× understatement — fast is $10/$50), and returns rateSource: "table", indistinguishable on the receipt from an exact match. Regional variants are safe only by luck: us.anthropic.… doesn't prefix-match, so it lands on FALLBACK_RATE with unknown: true — visible, and it fires unknownModelPolicy. The suffix case is the silent one.
2. No regional dimension. LiteLLM carries us./eu./au./jp. at a 10% premium.
3. 7 of 26 shipped rates have no external corroboration. Reported from day one; closing the gap is separate work.
4. gpt-5.6-sol is stale in the safe direction — tracked in pricing: adjudicate gpt-5.6-sol — table carries 50/300, OpenAI publishes 40/200 promotional through 2026-11-21 #142. Caught by this monitor on its own table; the promotional window through 2026-11-21 makes "correct" time-dependent, so it needs a decision rather than a drive-by edit.

[c-1k]

Codex review — round 1

codex review --base origin/master
model: gpt-5.6-sol (max effort)
REVIEWED_SHA: 378988e219b117012edce1d97282946f6870ee92
EXIT: 0

Verdict: REVISE — 4 P1, 1 P2.

Several comparison paths can report a clean result despite proven or possible understatement, including cache-write premiums and source conflicts. Cache schema loss and cache-gap reporting can also disappear silently, undermining the monitor's core contract.

The review named the changed files (103 references), so this is a real pass, not a no-op against an empty diff.

Findings

	Finding	File
P1	Fail omitted cache tiers that underprice premiums	scripts/pricing-drift/compare.mts:221-226
P1	Preserve understatement checks across source conflicts	scripts/pricing-drift/compare.mts:188
P1	Merge nonconflicting tiers from every source	scripts/pricing-drift/compare.mts:212
P1	Validate cache fields against upstream schema changes	scripts/pricing-drift/sources.mts:112-113
P2	Render cache gaps for otherwise agreeing models	scripts/pricing-drift/report.mts:125

All five accepted as real and fixed in 86929ec. Three of the four P1s were one underlying mistake — electing a single source as "the" consensus rather than resolving each tier across all sources — so the fix was structural rather than five patches.

The two worth calling out

The monitor had, in its own instrument, the defect it exists to detect. The "schema pin" test asserted LITELLM_FIELDS against a hardcoded copy of itself. A constant compared to itself says nothing about upstream. Had LiteLLM renamed cache_creation_input_token_cost, input and output would still have resolved, every row would still have counted as answered, coverage would have stayed at 19/19, and the run would have exited 0 with cache comparison silently disabled. Replaced with runtime sentinels that validate both fetched corpora by name at ingest and exit 2; the tautological test was deleted rather than left reading as coverage.

An assumption carried over from the design was wrong. I had written that an omitted cache tier is always conservative, because it meters at inputPer1k through the D1 fallback. That holds for a cache-read discount (~0.1x input). It is false for a cache-write premium (1.25x input), where the fallback understates by 20%. Understatement is now judged on the effective metered rate rather than on field presence. Probed the live table: no model currently omits a cache tier that upstream prices above its input rate, so this was latent — it would have passed silently the day one appeared.

Note packages/core/src/ledger/pricing.ts:58 makes the same overbroad claim in its own comment ("priced at inputPer1k — conservative overstatement"). Not touched here; flagged as a follow-up.

Live verdict unchanged after the fixes: 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. Round 2 running against 86929ec.

[c-1k]

Codex review — round 2

codex review --base origin/master
model: gpt-5.6-sol (max effort)
REVIEWED_SHA: 86929ec2466bd3e9d9708c33672339f14c332e9f
EXIT: 0

Verdict: REVISE — 3 P1, 4 P2. No repeats from round 1.

The checker has false-clean paths and workflow failures that bypass its tracking issue. It can also silently stop validating cache tiers after certain schema changes.

211 references to the changed files, so a real pass.

Findings

	Finding	File
P1	Evaluate definitive diffs before source conflicts	compare.mts:276
P1	File an issue when the checker tests fail	pricing-drift.yml:75
P1	Validate schema values, not just property names	sources.mts:125
P2	Handle pre-report process failures as unchecked runs	pricing-drift.yml:83
P2	Reuse the canonical cache-rate resolver	compare.mts:143
P2	Preserve the full source range for conflict reports	compare.mts:157
P2	Include orphan failures in JSON output	run.mts:101

All seven accepted and fixed in 2cd264b.

The one worth dwelling on

The first P1 is round 1's fix reappearing one abstraction higher. Round 1 stopped a source conflict from masking understatement. It did not stop a conflict from masking plain disagreement: ours 75/250 against sources 50/200 and 50/250 returned a non-failing source-conflict while input was a unanimous, unallowlisted mismatch at 50. Same defect, same file, one level up — the abstraction introduced to fix a defect class was where that class reappeared. Definitive diffs now classify before conflict; conflict wins only when every diff is conflicted.

Two others were the monitor failing to monitor itself. If drift:test failed, every later step was implicitly gated on success(), so the tracking-issue step was skipped and the run merely turned red — recording nothing, which is precisely the "fail the job only" option this design rejected up front. The reporting step is now always()-gated and names which stage broke. That is not the always() anti-pattern it resembles: that rule governs steps reporting success (a healthcheck ping gated always() reads healthy for failed runs). This step reports failure, so success() is what breaks it. Called out in a comment so it doesn't get "fixed" later.

The schema sentinel counted property names, so an upstream keeping a deprecated key holding null while moving the price elsewhere would have cleared the floor while the normalizer dropped every value. It now counts usable typed values.

effectiveRate also duplicated D1's fallback rather than calling resolveAppliedRates, and had already diverged — a negative finite cache rate meters at inputPer1k in the SDK but compared as negative here. AGENTS.md forbids a second resolution site, and a monitor that resolves rates differently from the thing it monitors is measuring the wrong quantity.

Live verdict unchanged across all three rounds: 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. 44 tests, no network. Round 3 running against 2cd264b.

[c-1k]

Codex review — rounds 3 and 4

round 3  REVIEWED_SHA: 2cd264b33d39e0cc3d94b98e2a7bc371c7f8c576  EXIT: 0  → 1 P1, 2 P2
round 4  REVIEWED_SHA: 23009a8cddc88f6a944abb2226e21e40292b9e4a  EXIT: 0  → 3 P1, 2 P2

Both passes named the changed files (113 and 213 references). All seven findings accepted and fixed — round 3 in 23009a8, round 4 in 4b36ce5.

Round 3 — 1 P1, 2 P2

The checker can silently lose an explicitly mapped source while retaining full coverage, and two conflict paths produce incomplete or inaccurate reports.

	Finding
P1	Enforce coverage for every mapped source
P2	Avoid claiming an in-range rate exceeds every source
P2	Preserve the range for conservative cache conflicts

The P1 was the coverage floor at the wrong granularity. It counted models with at least one answering source, so if LiteLLM renamed gpt-4o while models.dev still answered, the model stayed corroborated, coverage held at 19/19, and the run came back clean — with one of two independent checks silently gone. The entire value of two sources is independence; a gate that cannot notice one disappearing is not guarding that. Coverage is now counted over (source, model) pairs: 36/36 today, where the model-level view showed 19/19.

Round 4 — 3 P1, 2 P2

The monitor can emit a clean result for definite rate drift and can silently redefine or skip portions of its coverage.

	Finding
P1	Flag rates above every conflicting source
P1	Keep a fixed coverage floor outside MODEL_MAP
P1	Scope schema sentinels to mapped vendor rows
P2	Report which source-model mappings disappeared
P2	Set GH_REPO for failure-reporting commands

The second P1 inverts a claim I made in the round-3 commit. I described the derived coverage floor as self-maintaining and treated that as a virtue. It is the weakness: a gate computed from the thing it guards is not a gate. Setting one model's litellm to null lowers the actual and the expected count together, so 35/35 passes and the lost check closes the issue. Both now run — the derived expectation catches a source dropping a model, and checked-in MIN_MAPPINGS / MIN_CORROBORATED_MODELS catch the map itself being weakened. Lowering either is an explicit, reviewable edit.

The sentinel scoping is the same shape one level down: counting fields across the whole 3,040-row corpus is satisfied by rows this tool never reads, so a mapped provider could rename a cache field while a handful of reseller rows kept the old name. Now counted over the 17 mapped LiteLLM rows.

Where this stands

Five rounds, 20 findings, every one accepted as real. The live verdict has been byte-identical through all of them — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. Every fix closed a path by which a future run could report clean while something was wrong; none changed today's answer. That is the expected shape for this kind of tool, and it is why the review rounds were worth spending.

52 tests, no network in the suite. Round 5 running against 4b36ce5.

Recurring theme worth recording: three separate rounds found the same defect at successively higher levels of abstraction — conflict masking understatement (r1), conflict masking disagreement (r2), then a rate above every source still classified as conflict (r4). And twice the monitor carried, in its own instrument, the failure mode it exists to detect: a schema test that compared a constant to itself, and a coverage floor derived from the map it was guarding.

[c-1k]

Codex review — round 5

REVIEWED_SHA: 4b36ce5d8179077c1958d32e1b5cdf4adb91df68
EXIT: 0   →  1 P1, 3 P2   (163 references to the changed files)

The checker can silently lose cache-tier coverage and still exit successfully.

	Finding
P1	Pin cache-field coverage at the mapped-row baseline
P2	Classify conflicts using the effective cache rate
P2	Expose absolute-floor breaches in the report
P2	Keep non-default dispatches from rewriting the tracking issue

All four fixed in df416db.

The P1 is the schema sentinel's floor being shared across fields. One number is cleared by unaffected rows while a partial rename — one provider, or one field — strips the tier from every other mapped row. Those rows still answer, because input and output survive, so coverage stays full and a changed cache rate reads as agree. Worst on a one-source model like kimi-k3, where nothing else can contradict it.

Now per-source, per-field baselines measured against the mapped rows (litellm 17/17/17/17/8, models.dev 19/19/18/8), as a floor rather than an equality — upstream adding cache rates is fine, losing them is not.

Mutation-tested against the live feeds rather than only fixtures. Raising one baseline from 8 to 99 produces:

EXIT=2
LiteLLM: field coverage fell below the recorded baseline — schema likely changed:
  cache_creation_input_token_cost (8 of 99 mapped rows). Refusing to report a
  comparison that silently skipped these tiers.

That is a positive control on the real path, not just in the suite.

---

Stopping point

Six rounds, 24 findings, every one accepted as real: 5 → 7 → 3 → 5 → 4. Round 6 is running against df416db. My intent is to treat it as the last gate: fix anything P1, and ledger any remaining P2/P3 as residuals in this PR rather than continue indefinitely. This is a report-only tool — contents: read means it cannot write to pricing.ts under any failure mode — so the residual risk of a narrow edge case is a missed alarm, never a wrong rate.

The live verdict has been byte-identical through all six rounds — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0, 36/36 mappings. Not one fix changed today's answer; each closed a path by which a future run could report clean while something was wrong. For a monitor, that is the whole product, and it is why the rounds were worth spending.

Two patterns are worth recording beyond this PR:

The same defect kept reappearing one abstraction higher. Conflict masking understatement (r1) → conflict masking disagreement (r2) → a rate above every source still classified as conflict (r4) → a malformed rate compared raw instead of effective (r5). Four rounds, one idea: a comparison that cannot conclude something must not thereby conclude nothing is wrong.

Twice the monitor carried the exact failure mode it exists to detect. A schema test that compared a constant to itself, and a coverage floor derived from the map it was guarding. Both read as coverage while providing none — which is the definition of the thing this tool was built to catch.

[c-1k]

Codex review — round 6, and the gate is now blocked

REVIEWED_SHA: df416db40ad597d90043980dc1295ab448f5582e
EXIT: 0   →  1 P1, 2 P2

The checker can silently lose cache-tier coverage and can classify non-finite required rates as agreement. The workflow's default-branch state guard also admits an identically named tag.

All three fixed in 99afd36.

This round took three attempts. The first died on a malformed shell command Codex generated for itself; the second was cut off mid-investigation. Neither produced a findings block or a verdict sentence, and neither was read as clean — an absent verdict is a dead review, not a passing one. Only the third attempt concluded.

The P1 is the third narrowing of the same check, and the first two were both still too coarse

• A single shared occurrence floor is cleared by unaffected rows.
• Per-field totals are only a lower bound, so an upstream addition creates headroom: once one row gains cache_write, a later loss on a different row keeps the total at the baseline and passes — while that row still answers from input/output and the comparison silently skips its missing tier.

Now a per-(source, model, tier) expectation, 51 entries measured against the live feeds. Mutation-tested end to end:

EXIT=2
models.dev: schema/coverage regression — kimi-k3 lost cache_write.

A named model and a named tier, where the aggregate version returned a total that still summed correctly.

Also: a non-finite required rate reported agree. rawTier rejects Infinity/NaN, so inputPer1k: Infinity fell through the cache-absence path — required tiers are only excluded from cacheGaps — and emitted no diff at all, for a model whose metered cost is not a number. Now a malformed-rate outcome. And github.ref_name omits the ref namespace, so a dispatch on a tag sharing master's short name could rewrite master's tracking issue; both issue steps now require github.ref_type == 'branch'.

---

⚠️ The final SHA has no Codex verdict, and that is not a pass

Round 7 against 99afd36 terminated with:

ERROR: stream disconnected before completion: You have no credits remaining.
codex: Review was interrupted. Please re-run /review and wait for it to complete.

The Codex account is out of OpenAI API credits. That is also the likely cause of the two round-6 deaths. 99afd36's changes are therefore covered by 60 local tests and every local gate, but not by an independent review. Recording that plainly rather than letting six green rounds imply a seventh.

Re-run codex review --base origin/master once credits are restored.

Where this landed

Seven rounds attempted, six concluded, 28 findings, every one accepted as real: 5 → 7 → 3 → 5 → 4 → 3.

The live verdict never moved — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, 0 malformed, exit 0, 36/36 mappings. Not one of the 28 fixes changed today's answer; each closed a path by which a future run could report clean while something was wrong. For a monitor that is the entire product.

Local gates on 99afd36: 60 drift tests, 4367 repo tests, biome at the 39-warning master baseline, typecheck clean for both the repo and the new scripts.

Two patterns worth keeping

The same defect reappeared four times, each time one abstraction higher. Conflict masking understatement (r1) → conflict masking disagreement (r2) → a rate above every source still called conflict (r4) → a malformed rate compared raw instead of effective (r5). One idea underneath all of them: a comparison that cannot conclude something must not thereby conclude that nothing is wrong.

Three times the monitor carried the very failure mode it exists to detect. A schema test comparing a constant to itself; a coverage floor derived from the map it was guarding; and a cache-coverage baseline whose totals could stay satisfied while a specific model lost a specific tier. Each read as coverage while providing none.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99afd36efb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[c-1k]

Round 7 re-attempted — still blocked, and a live finding from the sweep

Gate: re-ran codex review --base origin/master on 99afd36 after a report that credits were restored. Still fails:

ERROR: stream disconnected before completion: You have no credits remaining.
warning: Falling back from WebSockets to HTTPS transport. stream disconnected...
Review was interrupted. Please re-run /review and wait for it to complete.
EXIT=1

It now dies in 1.3 KB within seconds, where earlier deaths produced 250–350 KB before dropping — the API rejects at connect rather than mid-stream. Diagnosed before reporting, since auth misrouting reads almost identically to a spend limit: codex is on API-key auth, this session's env also carries OPENAI_API_KEY, but the two fingerprints match — same key, no second org, balance simply at zero.

99afd36 still has no independent verdict. Six concluded rounds, 28 findings, all fixed; the final SHA is covered by 60 local tests and every local gate only.

---

🔴 Live billing exposure in resolveRates — measured, not theoretical

The prefix-match issue I deferred as a follow-up turns out to be live and considerably worse than the -fast case that surfaced it. Swept getModelRates against both sources:

20 uncataloged OpenAI/Anthropic models are silently underbilled, worst 16.7×.

model	meters at	actual	underbill
gpt-4o-mini-tts	1.5 / 6	25 / 100	16.7×
gpt-5.4-pro	25 / 150	300 / 1800	12×
o3-pro	20 / 80	200 / 800	10×
gpt-4o-mini-transcribe	1.5 / 6	12.5 / 50	8.3×
o3-deep-research	20 / 80	100 / 400	5×
gpt-4o-realtime-preview	25 / 100	50 / 200	2×
o4-mini-deep-research	11 / 44	20 / 80	1.8×

Longest-prefix-first means any suffix that raises the price collapses onto the cheaper base. Every model above returns rateSource: "table" — indistinguishable on a receipt from an exact match, so the receipt is internally consistent, chain-verifiable, and wrong.

Models landing on FALLBACK_RATE are excluded and are fine: that path sets unknown: true and warns. The bug is specifically that these do not.

20 is a floor. The sweep only sees models published upstream as distinct ids, so claude-opus-5-fast is not counted — models.dev carries fast mode under experimental.modes.fast.cost rather than as its own id. The suffix family that first surfaced the class is invisible to the sweep that confirms it.

Fix direction: an allowlist of suffixes known to preserve pricing, everything else routed to the loud unknown fallback. A denylist would enumerate -fast, -pro, -tts, -realtime-preview, … and fail open on whatever ships next — and this class is defined by "a suffix we have not seen yet", so failing open is the one behaviour it must not have.

Not fixed in this PR. It is packages/core money-path (tier 0/1) and belongs in its own change with its own gates, not folded into a CI script. Raised here because this PR is what found it.

[c-1k]

Codex rounds 7–9, connector threads, and the current gate state

Round 7 — 99afd36 · EXIT 0 · 2 P1, 3 P2 → fixed in 992fd20

The workflow can accept or publish stale clean results, undermining the monitor's alerting contract.

Both P1s were the workflow believing a clean result it never received. npm/tsx can exit 0 having written no report — the upload step only warns on a missing file, the failure predicate is false, and the clean step would then close a live tracking issue on the strength of a report that does not exist. A non-empty report is now part of the zero-exit contract. Separately, the default-branch predicates proved only that the event ref was named after the default branch, not that the checked-out SHA was still its head.

Also: unit conversion rejects overflow (a finite 1e308 becomes Infinity, the row stays non-null and satisfies coverage while resolveTier filters the value back out — a one-source model reaching agree with neither required tier compared); malformed-rate findings keep their answered sources rather than fabricating a coverage breach; and an allowlist can no longer absorb a conflicted tier.

Round 8 — 992fd20 · EXIT 0 · 2 P1, 2 P2 → fixed in ee92030

One P1 was a regression round 7's own fix introduced. The freshness step sat after checkout/setup-node/npm ci, so it implicitly required their success — an npm ci failure skipped it, the reporting step then found no fresh output and skipped too, leaving a red run and no could-not-run issue. Round 2 established that failures are always reported; round 7's guard quietly took it back.

The rule that resolves it, now stated at the site:

Filing a failure issue does not require verified freshness — if freshness cannot be established, reporting anyway is the safe error. Closing requires it positively, because closing on unknown state is how a live drift issue disappears. "We could not tell" must never resolve toward silence.

Round 9 — ee92030 · EXIT 0 · 1 P1 → fixed in 819a420

Round 8 left the third case open: clean and stale. Self-test passes, comparison passes, freshness false — close step skipped, failure predicate ignoring freshness, workflow finishing green having compared a tree master no longer serves. A stale head is now a could-not-check that fails the job, and a stale run comments rather than overwriting a live drift body.

Connector threads (a separate surface) → fixed in e11fd72, both resolved

chatgpt-codex-connector posted two P2s as review threads, which blocked the merge:

• Wholesale row disappearance was the quietest possible outcome. Round 8 established that an absent row is not a schema change. At the limit that was wrong: if LiteLLM returns {} or rewraps its rows, every mapped lookup hits that same continue, problems stays empty, and the sentinel passes. One absent row is a removal; every absent row is the shape having changed.
• Tier differences were attributed to every source that answered the model, so a cache-tier difference published by one source was rendered beside both — crediting a rate to a source that never stated it. TierDiff now carries publishedBy, resolved per tier.

---

State

e11fd72 — CI green (completed, non-skipped, all 6 jobs), MERGEABLE/CLEAN, zero unresolved threads, 71 tests.

Nine concluded CLI rounds plus the connector pass: 35 findings, every one accepted as real, trending 5 → 7 → 3 → 5 → 4 → 3 → 5 → 4 → 1. The live verdict has been identical through all of them — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, 0 malformed, exit 0, 36/36 mappings.

The gate is blocked again, with a different cause than before. Round 10 against 819a420:

ERROR: stream disconnected before completion:
Your organization has reached its configured enforced spend limit.
Update your limit at https://platform.openai.com/settings/organization/limits

Note this is not the earlier You have no credits remaining — that was a zero balance, this is a configured cap, and they are different settings pages. Reading the error text rather than assuming a repeat is the difference between topping up an account that has funds and raising a limit that is throttling it.

So e11fd72 carries CI, 71 tests and a mutation-verified guard suite, but no independent verdict. Recording that rather than letting nine green rounds imply a tenth.

[c-1k]

Exhaustiveness made mechanical — and the guard that did it was vacuous twice

HEADINGS was already a total Record, so a new outcome could not be added without a heading. That covered less than it looked: SECTION_ORDER, the array that actually drives rendering, was hand-listed, and the summary table was nine hand-written rows. Omitting an outcome from either compiles cleanly — so a new failing outcome could be counted, could fail the run, and could appear in no table and no section. Present in the verdict, absent from the report explaining it.

Both are now derived from one compile-checked list.

The guard was dead twice, and neither version was detectable by reading it

Attempt 1 — type _Uncovered = Exclude<Exclude<Outcome,"agree">, (typeof SECTION_ORDER)[number]>, with the array annotated Exclude<Outcome,"agree">[]. That annotation makes [number] the declared element type — every outcome — so _Uncovered is never whatever the array contains. Fixed with as const satisfies readonly ….

Attempt 2 — const _check: _Uncovered[] = []. An empty array is assignable to T[] for every T, so it passes whatever _Uncovered resolves to. Fixed with type _AssertNever<T extends never> = T.

Attempt 3 fails with Type '"unmapped"' does not satisfy the constraint 'never', naming the omission, and compiles clean when whole.

Both dead versions were found by deleting a member and checking the build actually breaks — neither by re-reading. A guard is not verified by being present, and this one looked correct twice while checking nothing, in the file whose subject is precisely that failure.

The procedural fix, now standard here: run the positive control first. Prove the mutated form errors, then prove the whole form is clean. Ordered the other way I would have accepted both dead versions, because absence of a compile error reads identically to a working guard.

---

State

f1f93a3 — CI green on the exact SHA (completed, non-skipped, six jobs), MERGEABLE/CLEAN, zero unresolved threads, 71 tests.

Live verdict, unchanged through every round: 16 agree · 1 expected deviation · 2 source conflicts · 7 uncorroborated · 0 malformed · exit 0 · 36/36 mappings.

Two commits now sit past the last independent verdict — the connector fixes (e11fd72) and this one (f1f93a3). The Codex gate is still capped; re-probed just now rather than assumed:

enforced spend limit   (terminated ~60s, 1272 bytes)

Nine concluded CLI rounds plus the connector pass: 35 findings, every one accepted as real. Recording the two ungated commits explicitly rather than letting nine green rounds cover them.

[c-1k]

Codex gate — CLEAN on the merge SHA

codex review --base origin/master     model: gpt-5.6-sol (max effort)
REVIEWED_SHA: c84e631fb1eda4480ad760726afd5caa44414a33
EXIT: 0        187 references to the changed files

Reviewed all changed implementation/workflow files, verified applicable guidance, and ran the checker tests, TypeScript check, Biome, and actionlint. No actionable correctness defect remained.

Four independent signals agree, which is the bar this PR has been holding itself to: the process terminated (EXIT=0), the tree did not move (REVIEWED_SHA, not UNATTRIBUTABLE), the review saw the diff (187 file references, so not a no-op against an empty base), and the reviewer concluded something (an explicit verdict sentence, not an absence).

The round before this one was discarded, not read

Round 15 came back EXIT=0 with a full transcript — and UNATTRIBUTABLE: 9c34031 -> c84e631. I had edited the worktree while it was running, which codex review reads directly, so its verdict describes a tree that never existed. Discarded and relaunched against a quiescent tree rather than mining it for findings. A confident review of a mixed state is worse than no review, because it looks like evidence.

Three instrument-reading failures of my own, all in the last few commits

Worth recording, because they are the same class the tool is built to catch and I committed them while writing it down:

• committed with a Biome error unread — the output tail was blank and I treated that as clean;
• committed with npm run typecheck failing — I watched for my own typecheck OK echo and did not notice its absence;
• ran a review over a tree I was actively editing.

The first two were caught by re-running and reading the summary line; the third by the REVIEWED_SHA guard. Found N errors is the verdict — not the shape of the output, and not the absence of something red.

Also worth noting: my own verdict extractor returned nothing here. It keys on Full review comments / Review comment:, which a clean run never prints — so a clean review and a dead review looked identical to it, and only reading the raw tail distinguished them. Exactly the "an extractor keying on one marker form has the same bug as the grep it replaced" problem, now with a third marker form.

---

Final state

c84e631 — 19 commits · Codex CLEAN · CI green on the exact SHA (6 jobs, completed, non-skipped) · MERGEABLE/CLEAN · zero unresolved threads · 76 tests.

Live verdict, identical through every one of the fifteen rounds: 16 agree · 1 expected deviation · 2 source conflicts · 7 uncorroborated · 0 malformed · exit 0 · 36/36 mappings.

Fourteen concluded CLI rounds plus a connector pass: 41 findings, every one accepted as real, trending 5 → 7 → 3 → 5 → 4 → 3 → 5 → 4 → 1 → [connector 2] → 2 → 3 → 1 → 1 → 0.

Not one of the 41 fixes changed today's answer. Each closed a path by which a future run could report clean while something was wrong — which, for a monitor, is the entire product.

What the rounds were actually about

One idea kept resurfacing one abstraction higher each time: a comparison that cannot conclude something must not thereby conclude that nothing is wrong. Conflict masking understatement → conflict masking disagreement → a rate above every source still called a conflict → a malformed rate compared raw instead of effective → a clean-but-stale run finishing green.

And five times the monitor carried, in its own instrument, the failure it exists to detect: a schema test comparing a constant to itself; a coverage floor derived from the map it guarded; a cache baseline whose totals stayed satisfied while one model lost one tier; an exhaustiveness guard that was vacuous twice; and a positive control that never exercised the exit code the workflow actually reads.

Every guard in this PR has been mutation-verified in both directions — including against the live feeds — because a guard is not verified by being present.