Skip to content

fix(ci): make the three document gates actually run - #542

Open
dmitrii-f-t27 wants to merge 1 commit into
gHashTag:mainfrom
dmitrii-f-t27:fix/gates
Open

fix(ci): make the three document gates actually run#542
dmitrii-f-t27 wants to merge 1 commit into
gHashTag:mainfrom
dmitrii-f-t27:fix/gates

Conversation

@dmitrii-f-t27

Copy link
Copy Markdown

All three of these gates currently pass or fail for reasons unrelated to what they check. Until they work, every check downstream of them means less than it appears to.

1. check_artefact_agreement.py verifies zero catalogue rows in CI. The script reads the SSOT from ROOT.parent / "t27" / ...; the workflow tries to place it with actions/checkout at path: ../t27, which checkout refuses because paths must be under GITHUB_WORKSPACE, and continue-on-error: true swallows the failure. The gate then takes the if CAT.exists(): false branch, cat = {}, and prints all nine known catalogue defects as [fixed]. All 36 runs have failed.

The workflow's own header says it exists because Every other check in this repository verifies an artefact against ITSELF and All of them passed while the oracle and the RTL implemented different formats under the name TNF16. The one cross-artefact gate was itself vacuous.

Fixed by checking the catalogue out in-workspace and passing its location via T27_ROOT, defaulting to the sibling path so local runs are unchanged; and by making a missing SSOT a loud failure instead of a silent pass.

The checker logic is untouched — with the sibling present it parses 92 rows and reproduces all 13 known divergences.

2. check_doc_refs.py crashes before printing anything. When a document quotes an absolute path, ROOT.parent / sib / "/root/..." collapses to that absolute path, and .exists() raises PermissionError on the runner, where /root is 0700. Triggered by docs/docs/deployment/runpod.md:95. Absolute paths are now skipped outright — they name a location on some machine, not a file in this tree — and the remaining .exists() calls are wrapped so an unreadable path can never abort the run.

The gate now reports a handful of genuine dangling references. Those are not fixed here; that is separate work.

3. check_paper_numbers.py absolves numbers using the document that indicts them. SOURCES globs research/**/*.md, and sourced() was a plain substring test over the concatenation of every source. research/frontier/DOCUMENT_TRACEABILITY_2026-08-10.md, whose entire purpose is to tabulate literals that have no source, therefore became a source for them — writing this number has no source made the number sourced.

Audit, withdrawal and rot inventories are now excluded from SOURCES, and sourced() requires digit boundaries so 0.145 stops matching inside 0.1456 and 10.1452.

The trailing unconditional sys.exit(0) is left alone: the script's own docstring says it cannot prove a number is right, so it is advisory by design.

All three gates were vacuous: one crashed before printing, one validated an
empty catalogue, one used the audit document as a source for the numbers that
audit indicts.

tools/check_doc_refs.py:62 -- widen the temp-path skip to every absolute path.
  It sat above both `.exists()` call sites, so a `/root/...` reference (from
  docs/docs/deployment/runpod.md:95) reached line 85 as `ROOT.parent/sib/p`,
  which collapses to `/root/...`; `Path.exists()` raises PermissionError rather
  than returning False when a parent is unreadable (0700 on the CI runner) and
  the unhandled exception killed the process before any output. An absolute path
  names a location on some machine, not a file in this tree, so resolving it
  here measured nothing anyway.
tools/check_doc_refs.py:35 -- add _exists(), an OSError-swallowing wrapper, and
  use it at the three resolution sites (lines 97, 98, 104), so no unreadable
  path can abort the run even if a new one gets past the skip.

.github/workflows/artefact-agreement.yml:31 -- check the SSOT catalogue out at
  `path: t27` instead of `path: ../t27`; actions/checkout rejects any path
  outside GITHUB_WORKSPACE, so the catalogue was never present.
.github/workflows/artefact-agreement.yml:31 -- drop continue-on-error, which
  swallowed that rejection.
.github/workflows/artefact-agreement.yml:39 -- pass T27_ROOT to the checker.
conformance/check_artefact_agreement.py:20 -- read T27_ROOT, defaulting to the
  existing sibling path so local runs are unchanged.
conformance/check_artefact_agreement.py:26 -- exit 1 when the catalogue is
  absent instead of proceeding with cat = {}. With an empty catalogue every one
  of the 13 baselined divergences stopped reproducing and the ratchet printed
  them all as [fixed]: a gate that compared nothing read as a gate that found
  everything fixed. Checker logic unchanged; with the sibling present it still
  parses 92 rows and reproduces all 13.

tools/check_paper_numbers.py:21 -- exclude DOCUMENT_TRACEABILITY*, WITHDRAWAL*
  and SCRIPT_ROT* from SOURCES. research/frontier/DOCUMENT_TRACEABILITY_
  2026-08-10.md exists to tabulate literals that have no source, so leaving it
  in SOURCES made writing "this number has no source" a source for the number.
tools/check_paper_numbers.py:79 -- replace the bare `v in blob` substring test
  with a digit-boundary regex, so 0.145 no longer matches inside 0.1456 or
  10.1452. The trailing sys.exit(0) is left alone: the docstring says this gate
  is advisory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag

Copy link
Copy Markdown
Owner

This sat for four days and I duplicated items 1 and 2 today without seeing it —
independently, from the same symptoms. That is corroboration for your diagnosis,
and a process failure on my side: I should have checked the open PRs first.

#564 is merged and covers your items 1 and 2. Same root causes, same
conclusions:

  • artefact-agreementROOT.parent / "t27" vs path: ../t27 which checkout
    refuses, with continue-on-error: true swallowing it. Worth adding to your
    write-up: the failure was not merely "verifies zero rows". The ratchet read
    the missing catalogue as twelve baseline entries that no longer reproduce
    and printed [fixed] next to each, including
    [catalogue] GF-T512: e=195 trits, m=316 needs 627 bits but declares 512.
    Following the job's own advice — --update-baseline — would have erased
    twelve recorded divergences. Reproduced both ways before touching it: without
    the catalogue, 12 false [fixed] and a failure; with it,
    catalogue rows parsed: 92 and OK: no new disagreements (13 known).
  • check_doc_refs.py — the PermissionError on /root/.... Fixed via a
    wrapper that treats unreadable as absent.

So those two hunks will conflict. Sorry.

Item 3 is still needed, and is untouched by anything I did. Both halves are
live on main today:

  • sourced() still opens with if v in blob: return True — a plain substring
    test, so 0.145 is absolved by 10.1452 before the digit-boundary regex on
    the next line is ever reached. The regex guards only the rounding branch.
  • SOURCES still globs research/**/*.md with no exclusion, so a withdrawal
    notice saying a number is wrong counts as a source for it.

That second one is the sharper bug of the two: it is a gate that clears a number
using the document that retracts it.

Two things from today that bear on your item 2, since the gate now runs and can
be argued with:

  • With the five public siblings materialised in CI, 63 references resolve
    cross-repo that previously read as dangling. trios-mesh is private and
    unreachable with the default token, so references resolving only there stay
    unverifiable — stated in the workflow rather than left silent.
  • The 16 that remain are real rot: check_no_star.sh named by three documents
    and present in none, paths written relative to research/ from inside
    research/frontier/, vocab.json, tnf_paper_v2.tex. Baselined at 266 so
    new rot fails from here, not deleted.

Rebase item 3 onto main and I will review it; the paper-numbers fix is the part
of this PR nobody has replicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants