fix(compat): handle SQL lexical structure in normalizeSQL - #56
Merged
Conversation
Replace simplistic quote-flipping parser with proper SQL lexical analysis. Fixes two defects: 1. Apostrophes outside string literals (e.g., in comments) no longer flip parser into quote mode, preventing whitespace after the apostrophe from being incorrectly preserved. Handles line comments (--) and block comments (/* */) with proper boundary detection. 2. Double-quoted identifiers now preserve inner whitespace (e.g., "my table" remains distinct from "my table"). Handles "" escapes within identifiers. Also adds support for backtick and bracket identifier quoting per SQLite spec, with consistent whitespace preservation for all quoted contexts. Implements proper SQL lexer that tracks: - Line comments (-- until newline) - Block comments (/* ... */ non-nesting) - Single-quoted string literals (with '' escapes) - Double-quoted identifiers (with "" escapes) - Backtick-quoted identifiers - Bracket-quoted identifiers [...] All existing tests pass. Manifest signatures unchanged (backscroll's current DDL contains no apostrophes in comments, so the whitespace normalization behavior remains identical for the canonical schemas). Claude-Session: https://claude.ai/code/session_019LDXzStaKrArqJKvy4z3eF
…istic signatures When line comments end with CRLF (Windows/mixed line endings), CR was being included in the normalized output, producing non-deterministic signatures for identical schemas with different line ending styles. This violates the identity contract where `normalizeSQL` must produce the same result regardless of line ending style. Skip CR characters when copying line comment text — they are trailing whitespace that will be normalized to a single space anyway. This ensures deterministic signatures across Unix (LF) and Windows (CRLF) line endings. Adds test case for CRLF line ending in line comment. Claude-Session: https://claude.ai/code/session_019LDXzStaKrArqJKvy4z3eF
…c signatures Comments are cosmetic — internal whitespace differences should not create different schema signatures. This is the third latent defect in normalizeSQL: comment content was preserved verbatim, making comment formatting load-bearing. Example: Two CREATE statements differing only in comment spacing: "CREATE TABLE t ( -- note double\n a INTEGER )" "CREATE TABLE t ( -- note double\n a INTEGER )" were producing different signatures. Now both normalize to the same signature with single spaces in the comment. Implementation: - Line comments: collapse internal whitespace, trim trailing space before newline - Block comments: collapse internal whitespace, ensure single space before */ - Both: preserve comment delimiters and overall comment structure Adds comprehensive test suite covering comment whitespace collapsing for both line and block comments, including edge cases with multiple spaces and newlines. Manifest.json remains unchanged — defect was latent; this fix applies to future databases with comments inside CREATE statements. Claude-Session: https://claude.ai/code/session_019LDXzStaKrArqJKvy4z3eF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #52 (merged in #55). The
normalizeSQLrewrite shipped there fixes the reported lockout, but left two latent defects that reintroduce the same class of bug on future input. This lands the fix that missed the #55 merge window.Defect 1 — an apostrophe outside a literal disables normalization for the rest of the statement
The shipped scanner toggles quote state on any
', including one inside a SQL comment. Measured againstmain:Whitespace stays load-bearing for that DDL, which is exactly the condition issue #52 exists to eliminate. Not triggered today only because backscroll's current DDL comments happen to contain no apostrophes. A future migration comment reading
-- don'tor-- the row's idsilently reintroduces #52.Defect 2 — whitespace collapsed inside double-quoted identifiers
Two genuinely distinct tables produce one signature. This is the opposite and more dangerous failure mode: defect 1 causes false rejection, this one causes false acceptance.
Change
normalizeSQLnow scans real SQL lexical structure:--line comments and/* */block comments (quotes inside them are text, not delimiters), single-quoted literals with''escapes, and double-quoted identifiers with""escapes. Whitespace is collapsed only in code, preserved inside literals and quoted identifiers.Catalog signatures are unchanged —
manifest.jsonneeded no regeneration, confirming both defects were latent rather than active.Verification
Observed output of
just ci:Behavior after the change:
Relates to #52.
https://claude.ai/code/session_019LDXzStaKrArqJKvy4z3eF