A file that documents the placeholder format is not a corrupted placeholder - #27
Merged
Conversation
…rrupted one
src/core/elision/regions.ts reduces 29.60% on the CLI and fell back to 0% on MCP.
Same file, same ratio, same engine. The trace named it:
fallbackReason: "Block hash corruption detected:
missing block hash [` + 64 hex + `] in token hasher."
That is not a hash. It is prose from regions.ts:17 -- "fixed width of
`<BLOCK_HASH:` + 64 hex + `>`" -- the line describing the format the placeholder
used to have.
detectCorruptedPlaceholders built its regex from two alternatives with different
strictness. ELISION_MARKER_PATTERN requires sha256:([a-f0-9]{12,64}); the legacy
alternative accepted <BLOCK_HASH:([^>]+)>, anything up to the next `>`. So it
matched from a backtick-quoted <BLOCK_HASH: through to a later >, captured
"` + 64 hex + `" as a hash, found it absent from the store, and failed the run.
createBlockPlaceholder emits <BLOCK_HASH:${hashContent(...)}> -- a sha256 digest
-- so requiring hex removes every prose match at no cost to real detection. The
bound now matches the pattern beside it, which had this right all along; two
alternatives in one regex with different strictness is what hid it.
Twenty-two files in this repository carry a <BLOCK_HASH:...>-shaped string,
including marker.ts, token-hasher.ts, token-hashing.ts, ARCHITECTURE.md and
CHANGELOG.md. Every one was unoptimizable over MCP, as is any user documentation
quoting the legacy format.
before after
regions.ts over MCP 0.00%, fallback 32.00%
token-hasher.ts over MCP fallback 35.28%
minimal file + one such comment 0.00%, fallback 81.6%
CLI corpus, 576 rows -- 576/576 byte-identical
Why no measurement could see it. The check opens `if (!hasher) return []`, and
the CLI supplies no TokenHasher -- deliberate and correct, since with no store
nothing claims to hold the content so nothing can be missing. MCP supplies one.
tools/corpus-harness drives the CLI, so every corpus number this project has ever
recorded was measured on the one route where this check is disabled. The 576/576
byte-identical result confirms it from the other side: the CLI never ran the gate
before or after. That is a new shape of §56's caution -- there byte-identical
meant the corpus lacks the shape, here it means the harness cannot reach the gate.
Both read as "no effect" and neither is.
This is §52's defect in a second gate. §52 stopped CONSTRAINT_DIRECTIVE_LOST
refusing a file for its own narrative comments; this is the block-hash integrity
gate refusing a file for describing the mechanism that processes it -- and the
file it refused was regions.ts, which defines region elision. Any check that
scans emitted content for the product's own markers is a candidate for the third.
test/unit/block-hash-false-positive.test.ts: 6 of 8 assertions fail against the
unfixed engine. The 2 that pass are the negative controls and they are the point
-- a genuine 64-hex placeholder absent from the store is still reported as
corruption, and one the hasher knows still resolves.
Found by adding `tokendamper mcp` to .mcp.json and pointing it at this
repository's own source, on the first non-trivial thing tried. M5a and M5b were
also MCP-adapter defects a full unit suite did not catch; the adapter is the
entry mode with the least end-to-end exercise, not the least tested one.
DECISIONS §57. 677 tests, typecheck, lint and build green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
src/core/elision/regions.tsreduces 29.60% on the CLI and fell back to 0% on MCP. Same file, same ratio, same engine. The trace named it:That isn't a hash. It's prose from
regions.ts:17— fixed width of`<BLOCK_HASH:`+ 64 hex +`>`— the line describing the format the placeholder used to have.The defect
detectCorruptedPlaceholdersbuilt its regex from two alternatives with different strictness:`${ELISION_MARKER_PATTERN.source}|<BLOCK_HASH:([^>]+)>`ELISION_MARKER_PATTERNrequiressha256:([a-f0-9]{12,64}). The legacy alternative accepted anything up to the next>, so it matched from a backtick-quoted<BLOCK_HASH:through to a later>, captured` + 64 hex + `as a hash, found it absent from the store, and failed the whole run.createBlockPlaceholderemits<BLOCK_HASH:${hashContent(...)}>— a sha256 digest — so requiring hex removes every prose match at no cost to real detection. The bound now matches the pattern beside it, which had it right all along.22 files in this repo carry a
<BLOCK_HASH:…>-shaped string, includingmarker.ts,token-hasher.ts,ARCHITECTURE.mdandCHANGELOG.md. Every one was unoptimizable over MCP, as is any user documentation quoting the legacy format.Measured
regions.tsover MCPtoken-hasher.tsover MCPImportant
The 576/576 needs its caveat, not just its number. It does not mean "safe because nothing moved" — it means the CLI route never executes this check, before or after, because it supplies no
TokenHasher.tools/corpus-harnessdrives the CLI, so every corpus number this project has ever recorded was measured on the one route where this gate is disabled. A trap entry now says so.This is a new shape of §56's caution: there byte-identical meant the corpus lacks the shape; here it means the harness cannot reach the gate. Both read as "no effect" and neither is.
§52's defect in a second gate
§52 stopped
CONSTRAINT_DIRECTIVE_LOSTrefusing a file for its own narrative comments. This is the block-hash integrity gate refusing a file for describing the mechanism that processes it — and the file it refused wasregions.ts, which defines region elision. Any check scanning emitted content for the product's own markers is a candidate for the third.Tests
test/unit/block-hash-false-positive.test.ts— 6 of 8 assertions fail against the unfixed engine. The 2 that pass are the negative controls and they are the point: a genuine 64-hex placeholder absent from the store is still reported as corruption, and one the hasher knows still resolves. Narrowing a detector must not cost the detection.Provenance
Found by adding
tokendamper mcpto a local.mcp.jsonand pointing it at this repository's own source — the first non-trivial thing tried. M5a and M5b were also MCP-adapter defects a full unit suite didn't catch. Three findings now share a cause: that adapter is the entry mode with the least end-to-end exercise, not the least tested one.DECISIONS §57. 677 tests, typecheck, lint and build green.
🤖 Generated with Claude Code