fix(ci): make the three document gates actually run - #542
Conversation
All three gates were vacuous: one crashed before printing, one validated an
empty catalogue, one used the audit document as a source for the numbers that
audit indicts.
tools/check_doc_refs.py:62 -- widen the temp-path skip to every absolute path.
It sat above both `.exists()` call sites, so a `/root/...` reference (from
docs/docs/deployment/runpod.md:95) reached line 85 as `ROOT.parent/sib/p`,
which collapses to `/root/...`; `Path.exists()` raises PermissionError rather
than returning False when a parent is unreadable (0700 on the CI runner) and
the unhandled exception killed the process before any output. An absolute path
names a location on some machine, not a file in this tree, so resolving it
here measured nothing anyway.
tools/check_doc_refs.py:35 -- add _exists(), an OSError-swallowing wrapper, and
use it at the three resolution sites (lines 97, 98, 104), so no unreadable
path can abort the run even if a new one gets past the skip.
.github/workflows/artefact-agreement.yml:31 -- check the SSOT catalogue out at
`path: t27` instead of `path: ../t27`; actions/checkout rejects any path
outside GITHUB_WORKSPACE, so the catalogue was never present.
.github/workflows/artefact-agreement.yml:31 -- drop continue-on-error, which
swallowed that rejection.
.github/workflows/artefact-agreement.yml:39 -- pass T27_ROOT to the checker.
conformance/check_artefact_agreement.py:20 -- read T27_ROOT, defaulting to the
existing sibling path so local runs are unchanged.
conformance/check_artefact_agreement.py:26 -- exit 1 when the catalogue is
absent instead of proceeding with cat = {}. With an empty catalogue every one
of the 13 baselined divergences stopped reproducing and the ratchet printed
them all as [fixed]: a gate that compared nothing read as a gate that found
everything fixed. Checker logic unchanged; with the sibling present it still
parses 92 rows and reproduces all 13.
tools/check_paper_numbers.py:21 -- exclude DOCUMENT_TRACEABILITY*, WITHDRAWAL*
and SCRIPT_ROT* from SOURCES. research/frontier/DOCUMENT_TRACEABILITY_
2026-08-10.md exists to tabulate literals that have no source, so leaving it
in SOURCES made writing "this number has no source" a source for the number.
tools/check_paper_numbers.py:79 -- replace the bare `v in blob` substring test
with a digit-boundary regex, so 0.145 no longer matches inside 0.1456 or
10.1452. The trailing sys.exit(0) is left alone: the docstring says this gate
is advisory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fe40758 to
4db83e6
Compare
|
This sat for four days and I duplicated items 1 and 2 today without seeing it — #564 is merged and covers your items 1 and 2. Same root causes, same
So those two hunks will conflict. Sorry. Item 3 is still needed, and is untouched by anything I did. Both halves are
That second one is the sharper bug of the two: it is a gate that clears a number Two things from today that bear on your item 2, since the gate now runs and can
Rebase item 3 onto main and I will review it; the paper-numbers fix is the part |
All three of these gates currently pass or fail for reasons unrelated to what they check. Until they work, every check downstream of them means less than it appears to.
1.
check_artefact_agreement.pyverifies zero catalogue rows in CI. The script reads the SSOT fromROOT.parent / "t27" / ...; the workflow tries to place it withactions/checkoutatpath: ../t27, which checkout refuses because paths must be underGITHUB_WORKSPACE, andcontinue-on-error: trueswallows the failure. The gate then takes theif CAT.exists():false branch,cat = {}, and prints all nine known catalogue defects as[fixed]. All 36 runs have failed.The workflow's own header says it exists because Every other check in this repository verifies an artefact against ITSELF and All of them passed while the oracle and the RTL implemented different formats under the name TNF16. The one cross-artefact gate was itself vacuous.
Fixed by checking the catalogue out in-workspace and passing its location via
T27_ROOT, defaulting to the sibling path so local runs are unchanged; and by making a missing SSOT a loud failure instead of a silent pass.The checker logic is untouched — with the sibling present it parses 92 rows and reproduces all 13 known divergences.
2.
check_doc_refs.pycrashes before printing anything. When a document quotes an absolute path,ROOT.parent / sib / "/root/..."collapses to that absolute path, and.exists()raisesPermissionErroron the runner, where/rootis 0700. Triggered bydocs/docs/deployment/runpod.md:95. Absolute paths are now skipped outright — they name a location on some machine, not a file in this tree — and the remaining.exists()calls are wrapped so an unreadable path can never abort the run.The gate now reports a handful of genuine dangling references. Those are not fixed here; that is separate work.
3.
check_paper_numbers.pyabsolves numbers using the document that indicts them.SOURCESglobsresearch/**/*.md, andsourced()was a plain substring test over the concatenation of every source.research/frontier/DOCUMENT_TRACEABILITY_2026-08-10.md, whose entire purpose is to tabulate literals that have no source, therefore became a source for them — writing this number has no source made the number sourced.Audit, withdrawal and rot inventories are now excluded from
SOURCES, andsourced()requires digit boundaries so0.145stops matching inside0.1456and10.1452.The trailing unconditional
sys.exit(0)is left alone: the script's own docstring says it cannot prove a number is right, so it is advisory by design.