Skip to content

Content mappers - #4712

Open
andrewbranch wants to merge 38 commits into
microsoft:mainfrom
andrewbranch:content-mappers
Open

Content mappers#4712
andrewbranch wants to merge 38 commits into
microsoft:mainfrom
andrewbranch:content-mappers

Conversation

@andrewbranch

@andrewbranch andrewbranch commented Jul 23, 2026

Copy link
Copy Markdown
Member

Status: pending team discussion and community feedback. Design needs thorough validation by Volar and other interested ecosystems.

Implements #2824 (comment)

Overview

Content mappers are external integrations that allow TypeScript to include otherwise unsupported file types in a program. They transform a foreign file’s original text into valid TypeScript syntax and provide mappings between the original and transformed content.

Users specify a set of file extensions to be handled by a content mapper package in a tsconfig.json file:

{
  "compilerOptions": {
    // ...
  },
  "contentMappers": [
    {
      "package": "vue-content-mapper",
      "extensions": [".vue"]
    }
  ],
  "include": ["src"] // implicitly includes .vue as well as .ts
}

When contentMappers are specified, tsc must be run with --loadExternalPlugins. VS Code passes --loadExternalPlugins to tsc --lsp only in trusted workspaces; otherwise, contentMappers are ignored in the LSP server.

The package field will be resolved as a Node.js module name. The package.json of the content mapper package must specify a tsContentMapper top-level field describing how to spawn the content mapper process and what compiler options its transform requires:

{
  "name": "vue-content-mapper",
  "version": "1.0.0",
  "tsContentMapper": {
    "exec": ["node", "dist/server.js"],
    "compilerOptions": ["module", "jsx", "jsxImportSource"]
  }
}

Note that the content mapper process need not be run with Node.js or implemented in JavaScript; package resolution serves as a convenient way to associate a content mapper with a versioned identity that can be managed alongside other dependencies in a project, but the exec field can specify any command.

Protocol

When constructing a program for a config that specifies contentMappers, module resolution will recognize file lookups for the specified extensions, spawn any content mapper processes needed, and request transformed content from them over STDIO. Content mappers communicate with TypeScript over JSON-RPC. TypeScript sends all requests; mappers do not send requests or notifications. The protocol defines two messages that mappers must handle: initialize and transform.

type PositionEncoding = "utf-8" | "utf-16";

interface InitializeParams {
    protocolVersion: 1;
    /** The position encodings supported by TypeScript. The mapper must choose one of these encodings. */
    positionEncodings: PositionEncoding[];
    /** BCP 47 locale requested for diagnostics. */
    locale?: string;
}

interface InitializeResult {
    /** Must match the protocolVersion sent in InitializeParams. */
    protocolVersion: 1;
    /** The position encoding the mapper will use for all span mapping positions and diagnostic positions. */
    positionEncoding: PositionEncoding;
    /**
     * The source identifier displayed for mapper-produced diagnostics.
     * Must not be "ts", "tsc", "typescript", or any file extension TypeScript understands.
    */
    diagnosticSource: string;
}

interface TransformParams {
    fileName: string;
    /** Original content of the file to be transformed. */
    content: string;
    /** The subset of compiler options that the mapper requested in its package.json. */
    compilerOptions: Record<string, unknown>;
}

interface TransformResult {
    /** Valid JS, JSX, TS, TSX, or JSON text that TypeScript can parse, according to the specified `scriptKind`. */
    text: string;
    /** The kind of syntax returned in `text`. Defaults to `ScriptKind.TS` if not specified. */
    scriptKind?: ScriptKind;
    /** Mappings between the original and transformed content. */
    mappings?: SpanMapping[];
    /** Parse errors in the original content. */
    diagnostics?: MapperDiagnostic[];
}

/** Positions and lengths are in the specified `positionEncoding`. */
type SpanMapping = [
    generatedStart: number,
    generatedLength: number,
    originalStart: number,
    originalLength: number,
    kind: SpanMapKind,
    purpose?: SpanMapPurpose,
];

enum ScriptKind {
    JS = 1,
    JSX = 2,
    TS = 3,
    TSX = 4,
    JSON = 6,
}

enum SpanMapKind {
    /** Verbatim spans in generated output have the same length and content as their counterparts in original text. */
    Verbatim = 0,
    /** Atom spans in generated output may have different length and content than their counterparts in the original text. */
    Atom = 1,
    /** Alias spans in generated output may have different length and content than their counterparts in the original text, but diagnostics display their original text. */
    Alias = 2,
}

/**
 * SpanMapPurpose controls which TypeScript language service features will activate using a span
 * in the transformed content given a request position in the original content.
 */
enum SpanMapPurpose {
  /** Disables all language service features for the span. */
  None = 0,
  /** Used by features that inspect semantic information, such as hover, signature help, and completions. */
  Semantic = 1 << 0,
  /** Used by features that locate symbols, such as definitions, references, rename, and call hierarchy. */
  Navigation = 1 << 1,
  /** Enables both semantic and navigation features. This is the default when `purpose` is omitted. */
  All = Semantic | Navigation,
}

/** Start and length are in the specified `positionEncoding`. */
interface MapperDiagnostic {
    messageText: string;
    start: number;
    length: number;
    code?: number;
    source?: string;
}

Span maps

For a content mapper to be useful, it needs to provide a mapping between the transformed output and the original content. In the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. Take a simple example:

// original content:
(+ 1 2 "oops")

// transformed content:
add(1, 2, "oops");

// span mapping:
add(1, 2, "oops");
^^^                 [0, 3)    [1, 2) + atom
    ^               [4, 5)    [3, 4) 1 verbatim
       ^            [7, 8)    [5, 6) 2 verbatim
          ^^^^^^    [10, 16)  [7, 13) "oops" verbatim

TypeScript sees and checks the transformed content, in this case producing a diagnostic for the string literal "oops" because it is not a number. The span mapping allows TypeScript to report the diagnostic in the original content, at the correct location of the string literal ([7, 13), instead of [10, 16)).

In this example, add mapped to + with SpanMapKind.Atom, indicating a correspondence between the two spans, but with different lengths and content. If the name add failed to resolve, the displayed diagnostic range would cover +, but the message would still reference the identifier add:

add.lisp:1:2 - error TS2304: Cannot find name 'add'.

1 (+ 1 2 "oops")
   ~

The mapper can use SpanMapKind.Alias instead of SpanMapKind.Atom to indicate that the generated and original text name the same entity. When the diagnostic is rendered, the original text of the alias span (+) will be substituted for the generated text (add) in the diagnostic message:

add.lisp:1:2 - error TS2304: Cannot find name '+'.
1 (+ 1 2 "oops")
   ~

Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. Unlike in Volar, diagnostics in unmappable regions are not discarded. In the CLI, they cause a short snippet of the transformed content to be shown with the diagnostic. A common case may be a content mapper that synthesizes an import statement at the top of the file used in scaffolding. If that import fails to resolve, the user will see:

app.vue:1:26 - error TS2307: Cannot find module '@vue/content-mapper-utils' or its corresponding type declarations.
  This location is in code generated by the content mapper '@vue/content-mapper@1.0.0' and has no corresponding location in the original file.

1 import { scaffolding } from "@vue/content-mapper-utils";
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spans in the generated output must not overlap, but multiple may map to the same span in the original content. In other words, one range in the original content can map to multiple ranges in the transformed content. This can be useful in the language server when combined with SpanMapPurpose and SpanMapKind. Broadly speaking, when a language server request is received for a position in a content-mapped file, the handler maps it to one or more projections into the transformed content, chooses how and whether to handle multiple projections based on the kind and purpose of each, performs analysis on the transformed content, and maps results back into original content coordinates.

The language server currently supports the following features for content-mapped files:

  • Diagnostics - mapped to original content where possible; diagnostics in synthesized regions are collected and reported at the top of the file.
  • Hover - uses the first SpanMapPurpose.Semantic projection.
  • Signature help - uses the first SpanMapPurpose.Semantic projection.
  • Completions - uses the first SpanMapPurpose.Semantic projection, only if it is SpanMapKind.Verbatim.
  • Definitions - queries all SpanMapPurpose.Navigation projections and merges results.
  • References - queries all SpanMapPurpose.Navigation projections and merges results.
  • Type definitions - queries all SpanMapPurpose.Navigation projections and merges results.
  • Implementations - queries all SpanMapPurpose.Navigation projections and merges results.
  • Document highlights - queries all SpanMapPurpose.Navigation projections and merges results.
  • Rename - searches every SpanMapPurpose.Navigation projection, but only verbatim spans may initiate a rename or be edited.

Additional language service features may be added in the future. Aside from diagnostics, language service requests can be disabled for any span by explicitly setting its purpose to SpanMapPurpose.None. If purpose is omitted from the span mapping tuple, it defaults to SpanMapPurpose.All, enabling all supported language service features for that span.

Note

This feature mapping is not as granular as what Volar allows, and it must be statically determined by the content mapper during transformation. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. The current proposal goes beyond that, but is intentionally limited in scope. We expect that ecosystems implementing complex transforms will invariably want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features. Content mappers provide a baseline editing experience, but they also provide the API foundation for more specialized language servers to build on. Vue tooling, for example, may choose to implement a content mapper with language features enabled only for verbatim <script> blocks, while a separate language server handles the rest, accessing the AST, type, and symbol information of their transformed content through an API connection to TypeScript’s language server.

Failure handling

Mappers return diagnostics for unparseable content, and errors in the transformed text itself are handled by TypeScript like any other file. If the mapper fails in an unexpected way (e.g., crashes or doesn’t conform to the protocol), TypeScript will report a program diagnostic and treat the file as an empty TypeScript file. After five failures in a single project, TypeScript will stop attempting to transform files with that content mapper and issue a final diagnostic reporting the failure.

LSP activation

TypeScript’s language server can only know to care about the file extensions registered in contentMappers once the server is running and has discovered a tsconfig.json that specifies them. In the case where a user opens a directory in VS Code and opens a single .vue file, the TypeScript VS Code extension hasn’t even activated, much less spawned a server that knows about a contentMappers registration. To address this, third-party VS Code extensions need to explicitly activate the TypeScript extension and request that it search for content mappers handling open files:

const extension = vscode.extensions.getExtension("TypeScriptTeam.native-preview");
await extension?.activate();

await vscode.commands.executeCommand(
    "typescript.native-preview.discoverContentMappers",
    {
        uris: vscode.workspace.textDocuments
            .filter(document => document.languageId === "vue")
            .map(document => document.uri),
        extensions: [".vue"],
    },
);

Extension-provided content mappers are not yet supported, but a follow-up exploration is planned.

Emit

Content-mapped files are not emitted to JavaScript. When --declaration is enabled, however, declaration files are emitted from the transformed content. The declaration file name for App.svelte is App.d.svelte.ts. Declaration maps are currently not supported.

Incremental, build, watch, and process consolidation

Content mappers are supported in --incremental, --build, and --watch modes. A content mapper’s name and version are recorded in the .tsbuildinfo file and used as part of up-to-date checks. If a content mapper’s version changes, the next build will re-transform all files handled by that content mapper.

Note

Modifying a content mapper implementation during local development will not change its identity, so you’ll need to bump the local package.json version, or use --force or --clean to clear cached outputs if testing content mappers with --incremental or --build.

In --build mode with project references, and in some instances in the language server, it’s possible to have a project graph with many projects all defining the same content mapper. To avoid excessive spawning of child processes, TypeScript deduplicates content mapper processes by name and version. A single content mapper process cannot be assumed to be responsible for only one project; it must be able to handle requests for any file in any project (or outside of any loaded project, in LSP scenarios) that uses it in any order. Each transform request passes the compilerOptions subset of the project that requested the transform for that file, and must be handled idempotently.

API integration

Content-mapped SourceFiles can be inspected by the JavaScript API. For a content-mapped SourceFile, file.text is the transformed text, file.originalText is the original text, and file.spanMap exposes an API for mapping between the two. Regardless of the positionEncoding used by the content mapper, accessing the span map through the JavaScript API always yields UTF-16 positions.

const mapped = file.spanMap.generatedToOriginalPosition(10);
// { position, fidelity }
// See _packages/native-preview/src/ast/spanMap.ts for details.

To-do

  • Break out content mapper time in --extendedDiagnostics and LSP logs
  • Turn on additional language service features
  • Decide whether mapper parse diagnostics prevent text from being used
  • Allow content mappers to proactively return additional files when they know they’ll be needed, to avoid IPC overhead. (Transform request of root files can also be batched.)
  • Investigate timeout/cancellation in LSP
  • Investigate VS Code extension-provided content mappers, similar to the way TS Server plugins could be automatically provided by VS Code extensions
  • Investigate if declaration maps can work by double-mapping back to original text
  • In VS Code, provide a read-only view of the transformed content for debuggability
  • Provide a JavaScript library for implementing the content mapper protocol

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 266 out of 267 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

internal/contentmapper/hostimpl.go:83

  • The published protocol includes optional MapperDiagnostic.source, but the wire struct omits it, so JSON decoding silently discards every per-diagnostic source and always uses the initialize-time default. Add the field and use it as the diagnostic source when present (while retaining the initialized source as the fallback).

Comment thread internal/tsoptions/tsconfigparsing.go
Comment thread internal/tsoptions/parsinghelpers.go
Comment thread internal/contentmapper/hostimpl.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 267 out of 268 changed files in this pull request and generated no new comments.

@mikearnaldi

mikearnaldi commented Jul 23, 2026

Copy link
Copy Markdown

@mikearnaldi I’d like to see some real-world use cases for that before considering lifting the limitation; did you have something particular in mind? I think it’s possible with --rewriteRelativeImportExtensions (all imports of "./foo.x" would have to be rewritten to "./foo.js" in the output). But I know the TS content backing a Vue component file, for example, is very fake and wouldn’t produce useful JS output.

I have been toying in the past with the idea of a custom dsl for Effect using Volar, basically a new language compiling to TS behind the scenes to allow first class Effect constructs such as schemas, using Volar I had to then create my own compiler wrapping the typescript language server which worked but wasn't really fun, if TS was to support meta-languages natively I think it could do it all directly (as in for this we already have module resolution, custom-code => ts, etc), in my language I was compiling .ets files to .d.ts and .js (happy to consider alternative conventions ofc).

Anyway I am excited to see this initiative, regardless if we end up using it for something or not, I have been worried about the future of systems such as Volar/Vue/Astro since tsgo was announced

Comment on lines +7 to +10
-// let [|af|] = <|async () => {
+// let <|[|af|] = async () => {
// /*GOTO DEF*/await Promise.resolve(0);
// }|>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could extract this out into a separate PR, but this was an LSP spec violation, inherited from Strada (insofar as the TS Server protocol maps to LSP)—the targetSelectionRange ([|this range marker|]) most be contained in the targetRange (<|this range marker|>, historically called “context range” in Strada).

@andrewbranch

Copy link
Copy Markdown
Member Author

I shamelessly vibed a demo-quality-only Vue transform so I could test something semi-realistic in the editor, which provides a good example of different navigation/semantic projections. Hovering a count ref in the script block correctly shows its type as Ref<number, number>:

Screenshot 2026-07-23 at 8 40 55 AM

Hovering the template-unwrapped version of count in a template interpolation shows number:

Screenshot 2026-07-23 at 8 40 46 AM

But they’re interconnected via navigation projections, so both appear together in references/definitions:

Screenshot 2026-07-23 at 10 30 09 AM

Auto-imports works, as long as there’s already one real import written (otherwise, the import edit tries to write to the very top of the transformed content, which is synthesized—this is probably something we can tweak and improve):

Screen.Recording.2026-07-23.at.10.42.12.AM.mov

Errors in synthesized code appear in a group at the top of the file:

Screenshot 2026-07-23 at 8 39 56 AM

vs. in the CLI:

image

@mikearnaldi

Copy link
Copy Markdown

@andrewbranch I am shamelessly vibe coding a similar experiment I did with Volar with this, if you want to check it out at some point I am pushing it here: https://github.com/mikearnaldi/ets-go

One point of feedback (which I think is already mentioned in your PR) I needed to enable other language features such as semantic tokens

// gated on workspace trust by the client. It corresponds to the --loadExternalPlugins CLI flag.
LoadExternalPlugins bool
DebounceDelay time.Duration
Locale locale.Locale

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4660 is removing this because it's unused / the local can change due to user preference changes. Not sure what to do about that

@mikearnaldi

Copy link
Copy Markdown

Quick update, working on https://github.com/mikearnaldi/ets-go I ended up finding some issues, the patches are in https://github.com/mikearnaldi/ets-go/tree/main/patches , all but 0005 are bug fixes , 0005 is a new api I needed for a specific integration (which is not necessarily in scope for this PR)

@revskill10

This comment was marked as off-topic.

@jasonlyu123

Copy link
Copy Markdown

This is way more powerful than what I expected. It'll definitely save a lot of work for all the non-TS file tooling. Some of the features will just work without needing enhancement.

There are a few questions I want to ask:

  • Would it make sense for the content mapper to be able to specify which editor feature to disable in the targeted files?

    Svelte currently has a lot of custom handling in completion and code actions. Mostly to enhance the result from the TypeScript(6) language service and move unmappable entries, and also to reroute the request position to a different position. If there is an IPC API for the target feature, it might be better to take over the feature entirely in Svelte files rather than going through client-side middleware.

  • If I understand it correctly, the IPC API won’t automatically sourcemap the request and result position. Is it correct? Or is it just not done yet? Also, does that mean validations like this won’t be done in the IPC API?

  • Kind of out of the scope of this PR, but will there be a pure source file parsing api in the future?

    Svelte’s current transformation uses TypeScript 6’s SourceFile api to parse script tags. I tried it before, but it seems like getting a source file in the current TypeScript 7 api is slower than TypeScript 6’s SourceFile. Probably it also does semantic work and the serialisation overhead.

For mapping, Svelte currently uses sourcemap generated by magic-string. We're using the hires option to generate character-to-character mapping. Because it's mostly string manipulation, converting it to span mapping is probably the main problem we'll need to solve on the Svelte side. In the long term, it probably makes sense to rebuild our transformation with this “span-based” mapping in mind. We always have issues with source maps being 1 or 2 characters shorter because the end of the range is generated code. So having a custom mapping solution might not be a bad thing. I am currently experimenting with it in this branch if you’re interested.

@andrewbranch

Copy link
Copy Markdown
Member Author

@mikearnaldi good catches, thank you! I will push some of these to this PR. I think patch 0002 isn’t quite right; I started on a different fix today but didn’t have time to finish it. It got surprisingly complicated. When you map a position at the very end of a span, what to return is ambiguous, but it doesn’t depend on whether that position is part of the last segment in the file. The ambiguity is there at the end of any span, and it’s particularly problematic at the boundary between two original text spans that map to non-contiguous virtual text spans:

// original
pos:  0 1 2 3 4 5
text: f o o b a r
span: A - - B - -

// transformed
pos:  0 1 2 3 4 5 6 7 8
text: f o o b a z b a r
span: A - - X - - B - -

At position 3 in the original text, where do you map to? It looks like span B in the diagram, but if that were a completions request at position 3, that would put the insertion caret right between foo and bar, meaning you’re completing foo. Really, every mapping operation needs to know some context about what it’s doing and specify a left/right affinity. I think query inputs almost always prioritize left, because they often arise from the insertion caret, which renders to the left of the Nth character when it’s at position N. LSP results are almost always ranges, not individual positions, so there’s no ambiguity with those.

@andrewbranch

andrewbranch commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Wanted to give a quick update here before I’m out on vacation for a week and ask for feedback on a couple specific things from any implementers, especially anyone coming from Volar.

First, I’m reconsidering the SpanMapPurpose classification I mentioned in the PR description:

/**
 * SpanMapPurpose controls which TypeScript language service features will activate using a span
 * in the transformed content given a request position in the original content.
 */
enum SpanMapPurpose {
  /** Disables all language service features for the span. */
  None = 0,
  /** Used by features that inspect semantic information, such as hover, signature help, and completions. */
  Semantic = 1 << 0,
  /** Used by features that locate symbols, such as definitions, references, rename, and call hierarchy. */
  Navigation = 1 << 1,
  /** Enables both semantic and navigation features. This is the default when `purpose` is omitted. */
  All = Semantic | Navigation,
}

...

This feature mapping is not as granular as what Volar allows, and it must be statically determined by the content mapper during transformation. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. The current proposal goes beyond that, but is intentionally limited in scope. We expect that ecosystems implementing complex transforms will invariably want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features.

I stand by the motivation here, but I think SpanMapPurpose is kind of a leaky abstraction compared to a bit flag per language service feature—smaller is not necessarily simpler.

I also want to get feedback on the constraints I’ve put on overlapping segments. The current rules are

  • Mappings must be ordered by start position in the transformer output text, and spans in the transformer output text must not overlap.
  • Spans in the original text may be perfect duplicates (i.e., multiple spans in the transformer text may map to the same span in the original text) but otherwise must not overlap.

I’m not sure if Volar has either of these restrictions; I think it at least doesn’t have the latter. I couldn’t think of a realistic example that would rely on overlapping but non-duplicate spans in the original text, and had trouble reasoning about the semantic implications of that kind of mapping.

Another question that came up in the latest design meeting about this was whether content mapper processes would need to read their own configuration files, beyond the compilerOptions that we pass to each transform call. If so, that undermines our current logic for caching transform results in LSP/watch mode, invalidating declaration outputs in --incremental, and consolidating process spawns between different projects. Currently, we only spawn one process per unique name@version, and we cache transforms of individual files by that name@version identifier along with a hash of the compilerOptions that we passed to transform (the subset the mapper declared it relies on in its package.json). If mappers vary their outputs across projects with their own config, this breaks two important features of our current strategy:

  1. Currently, the cache key of a transform can be computed without spawning the mapper. This is most important for --build, where we need to check if a referenced project needs to be rebuilt or not. On the initial build, we write the hash of the mapper identity and the compiler options it depends on into the .tsbuildinfo, and then on a later build we can resolve the mapper’s package.json and see if the hash has changed. If the mapper actually depends on other configuration we’re unaware of, this no longer works.
  2. In the LSP or in --watch mode, if the mapper reads external config files, we won’t know to invalidate mapped file caches or reload the mapper process when the external config changes.

These are solvable problems, but at a performance and complexity penalty. How do existing transforms work? Are they consuming arbitrary config from the workspace?


@jasonlyu123 I just saw your comment as I was about to paste this—I’m on a plane with terrible WiFi so I’ll have to look closer at your links in a week. But your input about specific language feature overrides supports the first change I mentioned I was considering, so that’s helpful feedback! I’m not sure I fully understand the questions about the IPC API. The protocol in this PR is totally separate from the IPC API you can get by importing "typescript". These content mapper processes won’t automatically be able to use the TS API against the tsc process that spawns them. (At the time the transforms run, there is no program, no checker, and ASTs are in the process of being created.) You could theoretically implement a content mapper that imports "typescript" to use that API for whatever reason, but that would spawn a child process to do that, and be a separate communication channel from the compiler owning the program compilation. In the LSP plugin scenario, you’d have tsc --lsp both spawning content mappers and serving API requests to your API client (which could either be installed as an LSP middleware in our client or running its own LSP server for .svelte files). Those API requests would have access to the ASTs produced by content mappers, and can freely map positions between the original and virtual text. But the virtual text is basically treated as the canonical text in the API (it’s what we have an AST for; it’s what gets bound and type checked.) I’m not sure if this was helpful or more confusing, but I really appreciate you trying this out already! Happy to talk in depth when I return.

Also, yes, there will be a way to get a parsed SourceFile without setting up a whole program, so that would help with the script tag parsing. Coming soon.

@mikearnaldi

Copy link
Copy Markdown

@mikearnaldi good catches, thank you! I will push some of these to this PR. I think patch 0002 isn’t quite right; I started on a different fix today but didn’t have time to finish it. It got surprisingly complicated. When you map a position at the very end of a span, what to return is ambiguous, but it doesn’t depend on whether that position is part of the last segment in the file. The ambiguity is there at the end of any span, and it’s particularly problematic at the boundary between two original text spans that map to non-contiguous virtual text spans:

// original
pos:  0 1 2 3 4 5
text: f o o b a r
span: A - - B - -

// transformed
pos:  0 1 2 3 4 5 6 7 8
text: f o o b a z b a r
span: A - - X - - B - -

At position 3 in the original text, where do you map to? It looks like span B in the diagram, but if that were a completions request at position 3, that would put the insertion caret right between foo and bar, meaning you’re completing foo. Really, every mapping operation needs to know some context about what it’s doing and specify a left/right affinity. I think query inputs almost always prioritize left, because they often arise from the insertion caret, which renders to the left of the Nth character when it’s at position N. LSP results are almost always ranges, not individual positions, so there’s no ambiguity with those.

I am sure my patches are not perfect, I am mostly iterating with agents so please take everything in my repo as a "maybe", the reason for that patch is: when the file is empty or when the cursor is pointed at the end of a file I would not get LSP completions, with my patch I get LSP completions but I am not sure it's correct :)

@jasonlyu123

Copy link
Copy Markdown

I’m not sure I fully understand the questions about the IPC API. The protocol in this PR is totally separate from the IPC API you can get by importing "typescript".

Yeah. I should have worded it more clearly. The IPC API question is related to the first question. And in both cases, what I want to say is LSP-connected IPC API.

Those API requests would have access to the ASTs produced by content mappers, and can freely map positions between the original and virtual text. But the virtual text is basically treated as the canonical text in the API

I think this should answer my question. The scenario of my question is: when the Svelte language server handles a completions request, whether it's triggered in TS client-side middleware or a standard LSP request, the Svelte language server uses the LSP-connected IPC API to ask about the completion result. In that case, should the parameter sent to the IPC API be a generated position or the source position? Another thing is whether we can ask for positions that are purely generated? From the current implementation, it looks like the LSP-connected IPC API should all use generated positions and won't have mapping validations. So I am mostly asking for confirmation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants