Skip to content

Fix/member header via include - #388

Open
geircodes wants to merge 4 commits into
msarson:version-1.0.1from
geircodes:fix/member-header-via-include
Open

Fix/member header via include#388
geircodes wants to merge 4 commits into
msarson:version-1.0.1from
geircodes:fix/member-header-via-include

Conversation

@geircodes

@geircodes geircodes commented Jul 30, 2026

Copy link
Copy Markdown

fix(hover,definition): resolve MEMBER through one INCLUDE hop for hover and F12

Status: this took three passes to get right. Each earlier commit fixed a real bug, but none of
them alone fixed the reported symptom (EVL:Lic giving no hover) — see the two "Follow-up" sections
below for what each subsequent test run still got wrong, and the final one for what actually closed it.

What happened

Hovering (or F12-ing) a cross-file reference — a plain global variable, or a PRE:Field-style
reference — returns nothing at all in member modules that use a common project convention:
putting the actual MEMBER('program') statement inside a small generated shim file, reached via
INCLUDE('member.clw'), instead of writing MEMBER(...) directly in every member.

Reproduced with EVL:Lic in a member file whose own INCLUDE('member.clw') contains:

MEMBER('TargetProgram')

Both lsp_hover and lsp_definition return nothing for this reference.

Root cause

TokenHelper.findMemberHeaderToken(tokens) looks for a literal MEMBER token with a
referencedFile in the tokens it's given — i.e. only the current file's own tokens. When the
real MEMBER(...) statement lives in a separately-INCLUDEd shim file instead, this returns
undefined, and every caller's "check the MEMBER parent" step is silently skipped.

Six call sites depend on this, across two services:

  • MemberLocatorService.ts (hover path): global-variable cross-file lookup, PRE:field lookup, class
    member lookup, warmMemberParent.
  • SymbolFinderService.ts (F12/definition path): findPrefixedField, findGlobalVariable.

Fix

Both services gained a resolveMemberHeaderToken(tokens, dir) helper: try
TokenHelper.findMemberHeaderToken on the file's own tokens first (unchanged, fast path); if that
finds nothing, walk the file's own direct INCLUDE(...) targets (one hop — matches the shim-file
convention, where the shim is always small and standalone) and try the same lookup on each included
file's tokens, returning the first MEMBER token found.

All 6 call sites (MemberLocatorService.ts x4 relevant ones actually touched, SymbolFinderService.ts
x2) now go through this instead of calling TokenHelper.findMemberHeaderToken directly.

Testing

New test file CrossFileMemberViaInclude.test.ts, mirroring the existing direct-MEMBER pin in
CrossFilePrefixField327.test.ts: a member file INCLUDEs a shim.clw containing
MEMBER('parent.clw'), and both F12 and hover on a PRE:Field reference must resolve to the parent
PROGRAM's field through that indirection.

npm run test:server: all passing (0 failing), verified in an isolated worktree off
origin/version-1.0.1 with no unrelated WIP mixed in. tsc --noEmit clean.

Scope

Three files: server/src/services/MemberLocatorService.ts, server/src/services/SymbolFinderService.ts,
server/src/test/CrossFileMemberViaInclude.test.ts (new). Found while investigating a hover-links bug
report, but this is a plain cross-file symbol-resolution bug unrelated to that feature, so it goes out
as its own PR.

Follow-up #1: fromFile omission in redirection (real bug, but NOT what was blocking EVL:Lic)

Testing the fix above with a SolutionManager loaded surfaced a second issue:
resolveMemberHeaderToken's call to resolveFilePath(inc.referencedFile!, fromDir) (and several
pre-existing parent-path resolutions in the same file) omitted the third fromFile argument. Per the
#328 "owner-project-first" contract on resolveFilePath/resolveViaProjectRedirection: without
fromFile, redirection can't identify which project is asking and falls back to an unscoped walk
across every project's redirection parser, returning the first match in solution order rather than the
one actually owned by the file being hovered — a real correctness gap in a multi-project solution.

Fixed by threading the current file's path through as fromFile everywhere it was missing. This is a
legitimate, worth-keeping fix, but turned out not to be what was blocking the EVL:Lic repro — the
test solution used here has only one project, so there was no cross-project collision actually
happening. Diagnosed with a standalone test that initially bypassed the resolver's redirection path
entirely — it had no SolutionManager loaded, so it never even exercised the code path it was meant to
test. Lesson for follow-up #2.

Follow-up #2: extension-less MEMBER targets (this is what was actually blocking EVL:Lic)

EVL:Lic still gave no hover after follow-up #1. Rewriting the test to actually load a
SolutionManager (SolutionManager.create('...TargetProgram.sln')) before calling the resolver —
matching the production code path instead of accidentally exercising the no-solution fallback —
surfaced the answer directly:

resolveViaProjectRedirection('TargetProgram', ...)      => null
resolveViaProjectRedirection('TargetProgram.clw', ...)  => <project dir>\TargetProgram.clw

Clarion MEMBER('TargetProgram') conventionally omits the .clw extension (the compiler infers
it) — completely idiomatic, not an edge case. referencedFile is stored exactly as written by the
tokenizer. resolveViaProjectRedirection's lookup matches by extension mask (a .red file's *.clw = ... line), so an extension-less name never matches any rule; SolutionManager.findFileWithExtension
matches by exact source-file basename, which also misses. Both silently return nothing for the
idiomatic, extension-less form — which is presumably the common case in most Clarion codebases, not a
corner case.

This is also why the tokenizer's structurePrefix propagation for nested FILE,PRE(x)
RECORD,PRE() → field was never actually the problem
(a concern raised earlier in this
investigation) — the chain never got far enough to exercise it; it was dying one step earlier, on the
MEMBER→PROGRAM hop itself.

Fixed with a normalizeMemberFilename() helper in both services (append .clw only when there's no
extension already — deliberately scoped to MEMBER targets only, since INCLUDE/LINK/MODULE targets
always carry an explicit extension already) applied at every MEMBER-target resolution call site.
Updated CrossFileMemberViaInclude.test.ts's shim fixture from MEMBER('parent.clw') to
MEMBER('parent') — the explicit-extension form the original fixture used would never have caught
either of these bugs.

Verified end-to-end with a SolutionManager loaded:
findPrefixFieldTokenInChain('Evl', 'Lic', ...) now resolves to the dictionary include file, matching
the expected hover output. npm run test:server: all passing, 0 failing, verified again in the isolated
worktree.

Follow-up #3: field name collides with its own enclosing structure's name

EVL:Lic and EVL:Txt now resolved correctly, but EVL:Evl (a field named the same as its enclosing
FILE) showed Evl — UNKNOWN pointing at the FILE's own declaration line instead of the field. Two
independent bugs, both in the same small area:

  1. findPrefixFieldInTokens matched by structurePrefix alone. StructureProcessor stamps
    structurePrefix on the declaring structure token itself, not just its fields, so a field that
    happens to share its structure's name matches the structure's own declaration token first (it
    appears earlier in document order).
  2. A second, independent tokenizer quirk feeds the same symptom class: DocumentStructure pushes a
    structure onto its stack the moment the structure's own keyword token is seen, so a later
    Variable/StructurePrefix-type token on that same declaration line — e.g. the GLOB:Owner
    argument of OWNER(GLOB:Owner), or the Evl argument inside PRE(Evl) itself — gets mistagged
    isStructureField=true with the structure's own prefix too, even though it's an attribute
    argument, not a real field.

Fixed both with a targeted selector in findPrefixFieldInTokens — prefer a field match that is
isStructureField AND declared on a line strictly after its structureParent's own line — rather
than touching DocumentStructure's core structure-stack walker, which many other features
(completion, diagnostics, F12) depend on.

Also addressed the original ergonomic question that started this whole thread: a PRE:Field hover
(e.g. EVL:Lic) resolves through the same code path as a true global variable and showed a generic
"🌍 Global variable" label — losing exactly the context needed to tell a real global apart from a
structure field reached via its PRE prefix. Now labeled "🔷 Evl field", matching the "X Field:"
wording StructureFieldResolver already uses for dot-notation access (Evl.Lic) to the same field.

New test: PrefixFieldNameCollidesWithStructure.test.ts, reproducing both the name-collision and the
same-line-attribute-argument shapes directly. npm run test:server: all passing (2344 in the isolated
worktree's older base + the 2 new tests), 0 failing.

Confirmed via the test suite: EVL:Lic, EVL:Txt, and EVL:Evl all hover correctly now, each
showing exactly one result — EVL:Lic is a single lexical token (Clarion colon-prefix notation), so
there is exactly one thing to hover, unlike Evl.Lic (two separate dot-joined tokens, each
independently hoverable, which is why that form can show two different results depending on which half
the cursor is on). That's expected, not a gap — noted here so a future pass doesn't try to "fix" it.

…er and F12

Some projects put the actual MEMBER('program') statement inside a small
generated shim reached via INCLUDE('member.clw') rather than writing it
directly in every member module -- this lets a shared source tree belong
to different PROGRAMs across projects by swapping just that one shim file.

TokenHelper.findMemberHeaderToken() only sees literal tokens in the file
it's given, so both MemberLocatorService (hover: global-variable lookup,
PRE:field lookup) and SymbolFinderService (F12: findPrefixedField,
findGlobalVariable) silently skipped their "check the MEMBER parent" step
whenever a member module used this indirection -- hover and F12 both
returned nothing for any cross-file reference in those files.

Both services now fall through one INCLUDE hop when no MEMBER token is
found directly, matching the shim-file convention (a MEMBER buried deeper
than that would be unusual). Regression test covers both hover and F12
agreement through the shim, mirroring the existing direct-MEMBER pin in
CrossFilePrefixField327.test.ts.
…r-project-first redirection

resolveMemberHeaderToken's own INCLUDE-target lookup (and several existing
parent-path resolutions in the same file) called resolveFilePath without
the current file's own path. Per msarson#328, resolveFilePath's redirection falls
back to an unscoped solution-wide walk across every project's redirection
parser when it doesn't know which file is asking -- in a multi-project
solution where several projects have their own same-named shim (e.g. many
member.clw files, one per project, each redirecting to a different MEMBER
target), that walk can resolve to the WRONG project's shim and silently
break resolution for every reference in the file.

Verified against the real-world repro this fix targets: a solution with
~20 sibling member.clw files across related projects.
…efore resolving

Real Clarion MEMBER('program') statements conventionally omit the file
extension (the compiler infers .clw), but referencedFile is stored exactly
as written. resolveViaProjectRedirection's redirection lookup matches by
extension mask (e.g. a .red file's "*.clw = ..." line), so an
extension-less name never matches any rule; SolutionManager.findFileWithExtension
matches by exact source-file basename, which also misses. Both silently
returned nothing for every MEMBER target written the idiomatic way.

Confirmed against the real solution this PR targets: with a real solution
loaded, resolveViaProjectRedirection('TargetProgram', ...) returned null
while resolveViaProjectRedirection('TargetProgram.clw', ...) resolved
correctly -- and with that, findPrefixFieldTokenInChain resolved EVL:Lic
to the dictionary include file end to end.

Added normalizeMemberFilename to both services and applied it at every
MEMBER-target resolution call site. Updated CrossFileMemberViaInclude's
shim fixture to use MEMBER('parent') (no extension) instead of
MEMBER('parent.clw') -- the explicit-extension form would never have
caught this.
…eld; label PRE:Field results

Two bugs surfaced while getting EVL:Lic to resolve end to end:

1. findPrefixFieldInTokens matched by structurePrefix alone.
   StructureProcessor stamps structurePrefix on the declaring structure
   token itself as well as on its real fields, so a field that
   coincidentally shares its name with the enclosing structure (e.g. FILE
   `Evl` containing a field also named `Evl`, "System Event ident") matched
   the FILE's own declaration token instead -- wrong line, type "UNKNOWN".

2. A second, independent tokenizer quirk fed into the same symptom:
   DocumentStructure pushes a structure onto its stack the moment the
   structure's own keyword token is seen, so a later Variable/
   StructurePrefix-type token on THAT SAME declaration line -- e.g. the
   GLOB:Owner argument of OWNER(GLOB:Owner), or the Evl argument inside
   PRE(Evl) -- gets mistagged isStructureField=true with the structure's
   own prefix too, even though it's an attribute argument, not a field.

Fixed both with a targeted selector (prefer a field match that is
isStructureField AND declared strictly after its structureParent's own
line) rather than touching DocumentStructure's core structure walker,
which many other features depend on.

Also: EVL:Lic-style PRE:Field hovers resolved through the same code path
as a true global variable and showed generic "Global variable" -- losing
exactly the context needed to tell a real global apart from a structure
field reached via its PRE prefix. Now labeled "`X` field", matching the
wording StructureFieldResolver already uses for dot-notation access to
the same field.
@geircodes
geircodes force-pushed the fix/member-header-via-include branch from 7eb0b64 to fcd9baf Compare August 3, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant