Three things that could not fail or could not be seen: duplicate agreement, seal coverage, and 348 specs that do not compile - #2208
Open
gHashTag wants to merge 8 commits into
Open
Three things that could not fail or could not be seen: duplicate agreement, seal coverage, and 348 specs that do not compile#2208gHashTag wants to merge 8 commits into
gHashTag wants to merge 8 commits into
Conversation
tmul is defined in 14 specs under specs/ternary/, dot27 in 9, quantize in 7. Nothing checked that the copies still compute the same thing, and copies drift. They agree. One behaviour each: dot27 one behaviour across 9 specs digest 8f8e7503 quantize one behaviour across 7 specs digest 1f7b9105 tmul one behaviour across 14 specs digest 91a68892 tmul's digest is the same value the exhaustive checker gets for the ripple adder's copy, so all fifteen definitions in the tree are one function. A negative result, and the point is that fifteen unchecked copies is a standing risk which is now a tripwire. Text comparison is the wrong instrument, and trying it first is why this file exists. My first pass hashed the normalised source and reported '2 variants of tmul, 3 of dot27, 3 of quantize'. Every one was an artefact: bitnet_mlp3.t27 writes its functions on a single line, so a regex ending at a newline-brace swallowed five following definitions into what it took to be tmul's body; and with balanced-brace extraction the count fell but did not reach one, because if(ta==1) and if (ta == 1) hash differently while computing the same thing. So the tool compares behaviour: each spec is compiled to C, the function is extracted from that spec's own output, and the same FNV-1a digest is folded over a fixed domain. Formatting is invisible to it; a semantic change is not. I nearly reported 'tmul has diverged across the BitNet family' as a finding about the repository. It was a finding about my regex -- the third time this session that an instrument, rather than the thing measured, produced the anomaly. Closes #2207 Refs #2205 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
The last of the decorative required checks. seal-coverage.yml was named required in docs/BRANCH-PROTECTION.md and its entire body was one echo. Establishing what it should assert took two attempts. In #2191 I matched seal FILENAMES against spec filenames and produced '1668 orphans of 1714, 1024 specs of 1070 uncovered' -- a finding about my assumption rather than the repository. I wrote neither a check nor a deletion then, and said so. This returns to it. Seals are keyed by MODULE name; the spec is named inside the file. A seal records spec_path, spec_hash, and the sha256 of each generated target at the moment of sealing. Its invariant is that the spec still exists and still hashes to what was recorded -- otherwise the four gen_hashes describe something the spec no longer produces, and the seal asserts something false. Measured across 1714 seals: 1507 hold 113 stale spec changed after sealing 89 dangling spec deleted, basename found nowhere in git 5 no spec_path 207 of 1714 (12%) do not hold, under a required check that said echo. Two seals named specs/vsa/core.t27, which had MOVED to specs/test_framework/ core.t27. They are repointed here and correctly become stale rather than fixed: the moved file's contents differ from what was sealed, so they need re-sealing. The 207 are recorded in tools/seal_baseline.txt as debt, one per line, so the gate holds the line without demanding they all be fixed at once. The job id stays 'coverage'. Renaming it would stop the required context reporting and send PRs to BLOCKED with every visible check green, as happened in #2191 -- verified here that the context still appears on an open PR before committing. Closes #2209 Refs #2207 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
changed the title
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 15:47:31 UTC
Summary
Seal Status
|
Split by whether the spec ever existed:
74 did exist in history -- deleted or moved. Sixteen went in one commit,
692ba52 'feat(clara): Complete DARPA CLARA PA-25-07-02 submission',
which removed specs/ml/ternary_{backprop,layer,loss,mlp,neuron}.t27 and
others. Ordinary orphans of a deletion.
15 found in NO commit under any path. Each records a spec_hash and all four
gen_hash_{c,rust,verilog,zig} -- reproducibility claims for a file nobody
can regenerate or check. All sealed 2026-06-17/18. Four of them are GF16
comparison and claims specs, in a repository whose GF16 claims have been
withdrawn twice. Stated as found and no further; the benign reading is that
they were sealed from an uncommitted working tree.
A correction to my own first pass. I tested existence with
'git log --diff-filter=D -- <exact path>', which only finds a deletion recorded at
that same path, and it reported 73 specs as never having existed. Re-tested by
basename across all history the number is 15. My instrument overstated by nearly
fivefold, and '73 seals reference specs that never existed' would have been a
serious and unsupported accusation.
That is the fourth time this session an anomaly came from the instrument rather
than the thing measured. When a scan reports something alarming about the
repository, re-derive it a second way before writing it down.
Refs #2209
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 16:16:35 UTC
Summary
Seal Status
|
…lesson
A seal whose spec was committed and later deleted, and a seal whose spec appears in
no commit at all, are two problems sharing one symptom. Their repairs differ, so
the gate no longer calls them the same thing:
74 dangling the spec was committed then deleted -- 16 by one commit,
692ba52 (DARPA CLARA submission). Remove the seal with the spec,
or restore both.
15 phantom the spec is in no commit and nowhere on disk. Its spec_hash and
four gen_hashes name a file nobody can fetch, so the record has no
checkable content. For four of them -- all GF16 claims/comparison
specs -- the seal file is the ONLY trace of the module in the tree.
_ever_existed asks git twice, by exact path and by basename across all history,
because the one-way version reported 73 phantoms where there are 15.
The negative control caught its own staleness: splitting the classification made
the planted missing-spec case phantom rather than dangling, and the control failed
until updated. That is what a control is for.
Skill ci-gates gains two sections. Section 7 records four occasions in one session
where an anomaly came from the instrument rather than the thing measured -- the
depth statistic ranking grids, a regex swallowing five definitions, a testbench
truncating i16 to a byte, and this git filter overstating fivefold. None was
caught by reasoning; three by a negative control and one by re-deriving the number
a different way. The corollary is that the more alarming a finding is, the more
likely it is yours: all four false alarms were accusations, and all four true
causes were typos and wrong regexes.
Section 8 records that a gate's vocabulary is part of its output.
Refs #2209
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 16:24:28 UTC
Summary
Seal Status
|
… not The plan was to re-seal the 113 stale seals. Testing one instead of batching stopped it and found something larger. t27c seal --save seals a spec that does not generate. specs/ml/optimizer/adamw.t27 fails on all four backends; seal --save wrote a seal anyway with gen_hash_rust=none, into a different filename (optimizer_AdamW.json, not AdamW.json), leaving the original stale seal untouched. Batch re-sealing would have created 113 duplicates under new names, blessed the non-generating ones with none hashes, and fixed nothing. 348 of 1114 specs (31.2%) do not generate with any backend, in a repository whose constitution makes specs the single source of truth. 1114 tracked 766 generate 68.8% 348 do not generate 31.2% The alternative explanation was checked and eliminated. A spec written for one target should not count as broken because another rejects it, so the gate accepts a spec if ANY backend takes it. On a 25-spec random sample of the 348, zero generated with any of gen-c/gen-rust/gen-verilog/gen-zig -- they fail in the parser, before a backend is reached. I checked because '31% of the source of truth does not compile' is an alarming claim, and by ci-gates SKILL.md section 7 -- four false alarms in this session, every one from the instrument -- an alarming claim is usually a fault in the measurement. specs/tri/ 70 parse error at module level 120 specs/scratch/ 58 parse error in fn 107 specs/fpga/ 35 Expected RBrace 45 specs/igla/ 15 Expected LBrace 36 specs/numeric/ 15 unknown cast target 34 specs/base/types.t27 is among them, failing at pack_trit line 172. tools/check_specs_generate.py records the 348 as debt with each compiler message, fails when a spec that used to generate stops, and reports when a baselined spec starts generating so the list cannot rot in the other direction either. The 113 stale seals are deliberately not re-sealed: 46 of the 95 specs behind them do not generate, and sealing those would record reproducibility for output that does not exist. Closes #2210 Refs #2209 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
changed the title
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 16:57:37 UTC
Summary
Seal Status
|
t27c seal --save sealed a spec no backend accepts, writing gen_hash_*: 'none' as though it were a hash. #2210 showed that batch re-sealing the 113 stale seals would have recorded 348 such claims. Two defects in one function, and the first is familiar: Err(_) => "none".to_string(), discards the compiler's own diagnosis -- the same swallowing pattern fixed across the Python tools in #2187/#2189/#2193, here in the Rust CLI -- and --save then wrote the 'none' out as a fact. Now the failure is kept and reported, and --save refuses: refusing to seal specs/ml/optimizer/adamw.t27: 4 of 4 backends rejected it gen-zig: Expected LBrace, got Colon (':') at line 93:65 ... A seal with gen_hash=none claims reproducibility for output that does not exist. Exit 1, no file written. A spec that generates still seals normally. --force keeps the deliberate case available and explicit rather than silent. Scope stated rather than implied: the identical Err(_) => 'none' pattern also exists in the HTTP seal_handler near line 2330. Fixing it there changes the JSON response shape, so it is left alone and named rather than half-done. My first patch hit that handler by accident -- the pattern appears twice and the naive replace found the wrong one first -- and the compiler caught it as an undefined variable. Closes #2211 Refs #2210 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 17:10:32 UTC
Summary
Seal Status
|
…ree backends
specs/base/types.t27 -- a base module -- did not parse:
result |= encoding << bit_pos; // pack_trit, line 172
Unexpected token in expression: Equals ('=') at line 172:13
Only += existed. The rest lexed as two tokens and died in parse_expr on the bare
'='; MinusEquals and the others were not in the token enum at all.
The part that mattered most: all three code generators did
if node.extra_op == "+=" { self.write(" += "); } else { self.write(" = "); }
so teaching the parser |= without touching them would have emitted x = rhs for
x |= rhs -- silently dropping the operator. A miscompilation is worse than the
parse error it replaces. This therefore changes four places together: the token
enum, the lexer, the parser, and all three backends through a shared
compound_binop().
Verified by reading the generated code rather than assuming it. C emits x |= 3;
Verilog expands to x = x | 3; and the same for all six operators.
Honest yield: 2 of 346, not the ~11 estimated. Six specs contain -=, four *= and
one |=, but only specs/base/types.t27 and compiler/runtime/runtime.t27 are fixed
by this alone; the rest carry further errors.
M5 freeze ceremony performed. bootstrap/build.rs refuses to build when
compiler.rs changes without a seal update -- a deliberate gate that makes a change
to the compiler core explicit rather than incidental. The seal is exactly
sha256(compiler.rs), and stage0/FROZEN_HASH is updated accordingly.
A dead arm in my own checker, found on the way: check_specs_generate.py ran
gen-zig, and there is no gen-zig subcommand -- the Zig backend is 'gen'. That arm
always returned non-zero and never contributed a verdict. It did not change the
headline, since a spec passing any other backend still passed, but the corrected
count is 768 generate / 346 not, against 766/348.
Closes #2212
Refs #2211
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-08-18 17:28:35 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
…sons 30 error instances across the non-generating specs are 'unknown cast target type f32/f64'. The cheap fix -- adding them to VALID_CAST_TYPES -- would have been wrong: var x: f32 parses today and the type tables know floats in four places, but the C generator emits the literal token f32, which is not a C type (five cc errors on a one-line probe, no typedef anywhere), and Verilog treats f32 as 32 plain bits with no float semantics. Float support is half-built end to end. Adding the cast targets would convert a visible parse error into uncompilable C and meaningless Verilog -- the compound-assignment lesson (#2212) at a scale where the missing half is 'implement floating point in three backends', which is an owner-level design decision rather than a gap. The parser now names the true state: cast to f32 is not supported: the language accepts float declarations, but no backend lowers float arithmetic (the C generator emits f32 verbatim, which is not a C type). This spec assumes a float-capable target that does not exist yet. Integer casts are untouched; a nonsense target still gets the old message. The 30 specs stay in the debt baseline with the truthful message attached. Also closes the last named instance of the swallowing pattern: seal_handler kept Err(_) => none after #2211 fixed the CLI path, because fixing it changes the response shape. Done additively -- existing fields unchanged, a new gen_failures array carries {backend, error} so a consumer can see why a hash is 'none' rather than treating the word as a digest. M5 ceremony performed for the compiler.rs change. Closes #2213 Refs #2212 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-18 17:37:05 UTC
Summary
Seal Status
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tmulis defined in 14 specs underspecs/ternary/.dot27in 9,quantizein 7. Nothing checked that the copies still compute the same thing, and copies drift.Result: they agree. One behaviour each, across every spec that defines them.
tmul's digest is91a68892— the same value the exhaustive checker gets for the ripple adder's copy, so all fifteen definitions in the tree are one function.A negative result, and the point is that something watches it now. Fifteen copies of a function with no check between them is a standing risk; this converts it into a tripwire.
Text comparison is the wrong instrument, and trying it first is why this exists
My first pass hashed the normalised source and reported "2 variants of
tmul, 3 ofdot27, 3 ofquantize". Every one was an artefact of the comparison:bitnet_mlp3.t27writes its functions on a single line, so a regex ending at\n}swallowed five following definitions into what it thought wastmul's body.if(ta==1)andif (ta == 1)hash differently while computing the same thing.So the tool compares behaviour: each spec is compiled to C, the function is extracted from that spec's own output, and the same FNV-1a digest is folded over a fixed domain. Formatting is invisible to it; a semantic change is not.
I nearly reported "tmul has diverged across the BitNet family" as a finding about the repository. It was a finding about my regex — the third time this session that an instrument, not the thing measured, produced the anomaly.
Refs #2205
The last of the decorative required checks.
seal-coverage.ymlis named required indocs/BRANCH-PROTECTION.mdand its entire body wasecho "Running SEAL coverage analysis...".Establishing what it should assert took two attempts, and the first was wrong in a way worth recording. In #2191 I scored coverage by matching seal filenames against spec filenames and got "1668 orphans of 1714, 1024 specs of 1070 uncovered" — a finding about my assumption, not the repository. I wrote neither a check nor a deletion and said so. This returns to it.
Seals are keyed by MODULE name; the spec is named inside the file. A seal records:
So a seal is a reproducibility record, and its invariant is: the spec it names still exists and still hashes to what was recorded. If the spec changed, the four
gen_hash_*no longer describe what it produces and the seal asserts something false.Measured across 1714 seals:
spec_path207 of 1714 (12%) do not hold, under a required check that says
echo.Of the dangling ones, 89 name a spec whose basename appears nowhere in git — genuinely deleted (
specs/numeric/binary16.t27,int4.t27,int8.t27, …). Two named a spec that had moved,specs/vsa/core.t27→specs/test_framework/core.t27; those are repointed in this PR and correctly becomestale, since the moved file's contents differ from what was sealed. They need re-sealing, not repointing.The 207 are recorded in
tools/seal_baseline.txtas debt, one per line, so the gate holds the line without demanding they all be fixed at once.The job id stays
coverage. Renaming it would make the required context stop reporting and send PRs toBLOCKEDwith every visible check green — learned in #2191, and verified here by checking the context still appears on an open PR before committing.Refs #2207, #2191
Plan for this iteration was to re-seal the 113 stale seals. Testing one instead of batching stopped it, and found something larger.
t27c seal --saveseals a spec that does not generatespecs/ml/optimizer/adamw.t27fails on all four backends.seal --savewrote a seal anyway, withgen_hash_rust=none, into a different filename (optimizer_AdamW.json, notAdamW.json) — so the original stale seal survived untouched.Batch re-sealing the 113 would have created 113 duplicates under new names, blessed the non-generating ones with
nonehashes, and left every original stale.348 of 1114 specs (31.2%) do not generate with any backend
The README says
.t27specs in → Zig, Verilog, C out; the constitution makes specs the single source of truth.The alternative explanation was checked and eliminated. A spec written for one target should not count as broken because another rejects it — so the gate accepts a spec if any backend takes it. On a 25-spec random sample of the 348, zero generated with any of
gen-c/gen-rust/gen-verilog/gen-zig. They fail in the parser, before a backend is reached.I checked that because "31% of the source of truth does not compile" is an alarming claim, and by the rule recorded in
.claude/skills/ci-gates/SKILL.md§7 — four false alarms in this session, all from the instrument — an alarming claim is usually a fault in the measurement.specs/tri/specs/scratch/specs/fpga/specs/igla/specs/numeric/specs/base/types.t27is among them — a base module, failing atpack_tritline 172.What this PR does
tools/check_specs_generate.pyrecords the 348 as debt intools/specs_generate_baseline.txt, one line each with the compiler's own first message, and fails when a spec that used to generate stops. It also reports when a baselined spec starts generating, so the debt list cannot silently rot in the other direction. Negative control plants a spec with a missing brace and proves it is reported while a valid one is not.The 113 stale seals are not re-sealed. 46 of the 95 specs behind them do not generate, and sealing those would record reproducibility for output that does not exist. That is now a measured reason rather than a hunch.
Refs #2209
Closes #2207
Closes #2209
Closes #2210
🤖 Generated with Claude Code