Skip to content

A file that documents the placeholder format is not a corrupted placeholder - #27

Merged
ojassug merged 1 commit into
mainfrom
fix-block-hash-false-positive
Aug 16, 2026
Merged

A file that documents the placeholder format is not a corrupted placeholder#27
ojassug merged 1 commit into
mainfrom
fix-block-hash-false-positive

Conversation

@ojassug

@ojassug ojassug commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

src/core/elision/regions.ts reduces 29.60% on the CLI and fell back to 0% on MCP. Same file, same ratio, same engine. The trace named it:

fallbackReason: "Block hash corruption detected: missing block hash [` + 64 hex + `] in token hasher."

That isn't a hash. It's prose from regions.ts:17fixed width of `<BLOCK_HASH:` + 64 hex + `>` — the line describing the format the placeholder used to have.

The defect

detectCorruptedPlaceholders built its regex from two alternatives with different strictness:

`${ELISION_MARKER_PATTERN.source}|<BLOCK_HASH:([^>]+)>`

ELISION_MARKER_PATTERN requires sha256:([a-f0-9]{12,64}). The legacy alternative accepted anything up to the next >, so it matched from a backtick-quoted <BLOCK_HASH: through to a later >, captured ` + 64 hex + ` as a hash, found it absent from the store, and failed the whole run.

createBlockPlaceholder emits <BLOCK_HASH:${hashContent(...)}> — a sha256 digest — so requiring hex removes every prose match at no cost to real detection. The bound now matches the pattern beside it, which had it right all along.

22 files in this repo carry a <BLOCK_HASH:…>-shaped string, including marker.ts, token-hasher.ts, ARCHITECTURE.md and CHANGELOG.md. Every one was unoptimizable over MCP, as is any user documentation quoting the legacy format.

Measured

before after
regions.ts over MCP 0.00%, fallback 32.00%
token-hasher.ts over MCP fallback 35.28%
minimal file + one such comment 0.00%, fallback 81.6%
CLI corpus, 576 rows 576/576 byte-identical

Important

The 576/576 needs its caveat, not just its number. It does not mean "safe because nothing moved" — it means the CLI route never executes this check, before or after, because it supplies no TokenHasher. tools/corpus-harness drives the CLI, so every corpus number this project has ever recorded was measured on the one route where this gate is disabled. A trap entry now says so.

This is a new shape of §56's caution: there byte-identical meant the corpus lacks the shape; here it means the harness cannot reach the gate. Both read as "no effect" and neither is.

§52's defect in a second gate

§52 stopped CONSTRAINT_DIRECTIVE_LOST refusing a file for its own narrative comments. This is the block-hash integrity gate refusing a file for describing the mechanism that processes it — and the file it refused was regions.ts, which defines region elision. Any check scanning emitted content for the product's own markers is a candidate for the third.

Tests

test/unit/block-hash-false-positive.test.ts6 of 8 assertions fail against the unfixed engine. The 2 that pass are the negative controls and they are the point: a genuine 64-hex placeholder absent from the store is still reported as corruption, and one the hasher knows still resolves. Narrowing a detector must not cost the detection.

Provenance

Found by adding tokendamper mcp to a local .mcp.json and pointing it at this repository's own source — the first non-trivial thing tried. M5a and M5b were also MCP-adapter defects a full unit suite didn't catch. Three findings now share a cause: that adapter is the entry mode with the least end-to-end exercise, not the least tested one.

DECISIONS §57. 677 tests, typecheck, lint and build green.

🤖 Generated with Claude Code

…rrupted one

src/core/elision/regions.ts reduces 29.60% on the CLI and fell back to 0% on MCP.
Same file, same ratio, same engine. The trace named it:

  fallbackReason: "Block hash corruption detected:
                   missing block hash [` + 64 hex + `] in token hasher."

That is not a hash. It is prose from regions.ts:17 -- "fixed width of
`<BLOCK_HASH:` + 64 hex + `>`" -- the line describing the format the placeholder
used to have.

detectCorruptedPlaceholders built its regex from two alternatives with different
strictness. ELISION_MARKER_PATTERN requires sha256:([a-f0-9]{12,64}); the legacy
alternative accepted <BLOCK_HASH:([^>]+)>, anything up to the next `>`. So it
matched from a backtick-quoted <BLOCK_HASH: through to a later >, captured
"` + 64 hex + `" as a hash, found it absent from the store, and failed the run.

createBlockPlaceholder emits <BLOCK_HASH:${hashContent(...)}> -- a sha256 digest
-- so requiring hex removes every prose match at no cost to real detection. The
bound now matches the pattern beside it, which had this right all along; two
alternatives in one regex with different strictness is what hid it.

Twenty-two files in this repository carry a <BLOCK_HASH:...>-shaped string,
including marker.ts, token-hasher.ts, token-hashing.ts, ARCHITECTURE.md and
CHANGELOG.md. Every one was unoptimizable over MCP, as is any user documentation
quoting the legacy format.

                                    before              after
  regions.ts over MCP               0.00%, fallback     32.00%
  token-hasher.ts over MCP          fallback            35.28%
  minimal file + one such comment   0.00%, fallback     81.6%
  CLI corpus, 576 rows              --                  576/576 byte-identical

Why no measurement could see it. The check opens `if (!hasher) return []`, and
the CLI supplies no TokenHasher -- deliberate and correct, since with no store
nothing claims to hold the content so nothing can be missing. MCP supplies one.
tools/corpus-harness drives the CLI, so every corpus number this project has ever
recorded was measured on the one route where this check is disabled. The 576/576
byte-identical result confirms it from the other side: the CLI never ran the gate
before or after. That is a new shape of §56's caution -- there byte-identical
meant the corpus lacks the shape, here it means the harness cannot reach the gate.
Both read as "no effect" and neither is.

This is §52's defect in a second gate. §52 stopped CONSTRAINT_DIRECTIVE_LOST
refusing a file for its own narrative comments; this is the block-hash integrity
gate refusing a file for describing the mechanism that processes it -- and the
file it refused was regions.ts, which defines region elision. Any check that
scans emitted content for the product's own markers is a candidate for the third.

test/unit/block-hash-false-positive.test.ts: 6 of 8 assertions fail against the
unfixed engine. The 2 that pass are the negative controls and they are the point
-- a genuine 64-hex placeholder absent from the store is still reported as
corruption, and one the hasher knows still resolves.

Found by adding `tokendamper mcp` to .mcp.json and pointing it at this
repository's own source, on the first non-trivial thing tried. M5a and M5b were
also MCP-adapter defects a full unit suite did not catch; the adapter is the
entry mode with the least end-to-end exercise, not the least tested one.

DECISIONS §57. 677 tests, typecheck, lint and build green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ojassug
ojassug merged commit 8cc30ec into main Aug 16, 2026
3 checks passed
@ojassug
ojassug deleted the fix-block-hash-false-positive branch August 16, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant