Skip to content

fix(tokenizer): PRAGMA(...) mistyped as Function, corrupting completion scope - #393

Open
geircodes wants to merge 1 commit into
msarson:version-1.0.1from
geircodes:fix/pragma-directive-misclassified-as-function
Open

fix(tokenizer): PRAGMA(...) mistyped as Function, corrupting completion scope #393
geircodes wants to merge 1 commit into
msarson:version-1.0.1from
geircodes:fix/pragma-directive-misclassified-as-function

Conversation

@geircodes

Copy link
Copy Markdown

fix(tokenizer): PRAGMA(...) mistyped as Function, corrupting completion scope

What happened

Reported while investigating a different bug (missing INCLUDE keyword completion, see the
companion PR fix/word-completion-missing-directives): typing a few letters at the top of a real
member file (before any procedure) surfaced dozens of unrelated local variables belonging to
other, later procedures in that same file — not just the expected keyword/global noise.

Reproduced directly against a real ~1900-line member file: unfiltered word completion right after
a PRAGMA(...) line, before any real procedure, returned 132 variable-kind items pulled from
the local DATA sections of many unrelated procedures later in the file.

Root cause

server/src/tokenizer/TokenPatterns.ts's Directive token pattern recognized
COMPILE|OMIT|EMBED|SECTION|ENDSECTION|INCLUDE — but not PRAGMA. Since PRAGMA(...) doesn't
match any higher-priority pattern in orderedTokenTypes, it fell through to the generic catch-all:

[TokenType.Function]: /\b[A-Za-z_][A-Za-z0-9_]*(?=\()/i,

So every PRAGMA(...) line was tokenized as TokenType.Function — indistinguishable from a real
procedure/function declaration to TokenHelper.isProcedureOrFunction()
(type === Procedure || type === Function).

WordCompletionProvider's scope-analyzer fallback (findEnclosingToken, used when there is no
real containing procedure — e.g. global scope near the top of a file) picks the nearest
"procedure-like" token by type. A mistokenized PRAGMA token has no real end marker
(finishesAt/executionMarker both undefined, defaulting to Number.MAX_SAFE_INTEGER), so once
picked as the "containing procedure", collectProcLocals walks from that line all the way to
end-of-file — vacuuming in every top-level local variable declared in every procedure that
follows.

Fix

Add PRAGMA to the Directive pattern, and to its Label-exclusion list (for the same reason
COMPILE/OMIT/INCLUDE/etc. are already excluded there — defense in depth, since Directive is
checked before Label in orderedTokenTypes and would already win, but keeps the exclusion list
internally consistent).

Two-line diff:

[TokenType.Directive]: /\b(?:COMPILE|OMIT|EMBED|SECTION|ENDSECTION|INCLUDE|PRAGMA)\b/i,
...
[TokenType.Label]: /^(?!(?:COMPILE|OMIT|EMBED|SECTION|ENDSECTION|INCLUDE|PRAGMA|PROGRAM|MEMBER|END|CODE|DATA)(?![:\w]))[A-Za-z_][A-Za-z0-9_:]*/i,

Testing

New test in WordCompletionProvider.test.ts: a PRAGMA(...) line before any real procedure, with
two procedures declared after it (each with a local variable) — asserts neither later procedure's
locals appear in completion at the PRAGMA line.

Harness against the real repro file: leaked variable count went from 132 to 0 after the fix, with
no change to any other token's classification (only the 2 PRAGMA tokens in that file were
reclassified, from Function to Directive). Full suite in an isolated worktree off
origin/version-1.0.1: 2341 passing, 0 failing, 4 pending (pre-existing, unrelated).

Deployed and confirmed working live.

Scope

Two files: server/src/tokenizer/TokenPatterns.ts, server/src/test/WordCompletionProvider.test.ts
(new test only — no other test files touched). Independent of the companion
fix/word-completion-missing-directives PR — different files, different bug, safe to merge in
either order.

…on scope

PRAGMA was missing from the Directive token pattern, so it fell through to
the generic identifier-followed-by-paren Function pattern -- making it
indistinguishable from a real PROCEDURE/FUNCTION declaration to
TokenHelper.isProcedureOrFunction().

WordCompletionProvider's scope fallback (findEnclosingToken) then treated a
PRAGMA line with no visible containing procedure as an open-ended "enclosing
procedure" with no end marker, and collectProcLocals walked from that line to
end-of-file, surfacing every later procedure's local variables as if they
were in scope at the PRAGMA line. Reproduced live against a real multi-KLOC
member file: 132 unrelated locals leaked into a single completion request.

Add PRAGMA to the Directive pattern (and its Label-exclusion list, for the
same reason COMPILE/OMIT/INCLUDE/etc. are already excluded there).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant