fix(tokenizer): PRAGMA(...) mistyped as Function, corrupting completion scope - #393
Open
geircodes wants to merge 1 commit into
Open
Conversation
…on scope PRAGMA was missing from the Directive token pattern, so it fell through to the generic identifier-followed-by-paren Function pattern -- making it indistinguishable from a real PROCEDURE/FUNCTION declaration to TokenHelper.isProcedureOrFunction(). WordCompletionProvider's scope fallback (findEnclosingToken) then treated a PRAGMA line with no visible containing procedure as an open-ended "enclosing procedure" with no end marker, and collectProcLocals walked from that line to end-of-file, surfacing every later procedure's local variables as if they were in scope at the PRAGMA line. Reproduced live against a real multi-KLOC member file: 132 unrelated locals leaked into a single completion request. Add PRAGMA to the Directive pattern (and its Label-exclusion list, for the same reason COMPILE/OMIT/INCLUDE/etc. are already excluded there).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(tokenizer): PRAGMA(...) mistyped as Function, corrupting completion scope
What happened
Reported while investigating a different bug (missing
INCLUDEkeyword completion, see thecompanion PR
fix/word-completion-missing-directives): typing a few letters at the top of a realmember file (before any procedure) surfaced dozens of unrelated local variables belonging to
other, later procedures in that same file — not just the expected keyword/global noise.
Reproduced directly against a real ~1900-line member file: unfiltered word completion right after
a
PRAGMA(...)line, before any real procedure, returned 132 variable-kind items pulled fromthe local DATA sections of many unrelated procedures later in the file.
Root cause
server/src/tokenizer/TokenPatterns.ts'sDirectivetoken pattern recognizedCOMPILE|OMIT|EMBED|SECTION|ENDSECTION|INCLUDE— but notPRAGMA. SincePRAGMA(...)doesn'tmatch any higher-priority pattern in
orderedTokenTypes, it fell through to the generic catch-all:So every
PRAGMA(...)line was tokenized asTokenType.Function— indistinguishable from a realprocedure/function declaration to
TokenHelper.isProcedureOrFunction()(
type === Procedure || type === Function).WordCompletionProvider's scope-analyzer fallback (findEnclosingToken, used when there is noreal containing procedure — e.g. global scope near the top of a file) picks the nearest
"procedure-like" token by type. A mistokenized
PRAGMAtoken has no real end marker(
finishesAt/executionMarkerbothundefined, defaulting toNumber.MAX_SAFE_INTEGER), so oncepicked as the "containing procedure",
collectProcLocalswalks from that line all the way toend-of-file — vacuuming in every top-level local variable declared in every procedure that
follows.
Fix
Add
PRAGMAto theDirectivepattern, and to itsLabel-exclusion list (for the same reasonCOMPILE/OMIT/INCLUDE/etc. are already excluded there — defense in depth, sinceDirectiveischecked before
LabelinorderedTokenTypesand would already win, but keeps the exclusion listinternally consistent).
Two-line diff:
Testing
New test in
WordCompletionProvider.test.ts: aPRAGMA(...)line before any real procedure, withtwo procedures declared after it (each with a local variable) — asserts neither later procedure's
locals appear in completion at the
PRAGMAline.Harness against the real repro file: leaked variable count went from 132 to 0 after the fix, with
no change to any other token's classification (only the 2
PRAGMAtokens in that file werereclassified, from
FunctiontoDirective). Full suite in an isolated worktree offorigin/version-1.0.1: 2341 passing, 0 failing, 4 pending (pre-existing, unrelated).Deployed and confirmed working live.
Scope
Two files:
server/src/tokenizer/TokenPatterns.ts,server/src/test/WordCompletionProvider.test.ts(new test only — no other test files touched). Independent of the companion
fix/word-completion-missing-directivesPR — different files, different bug, safe to merge ineither order.