Content mappers - #4712
Conversation
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 266 out of 267 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
internal/contentmapper/hostimpl.go:83
- The published protocol includes optional
MapperDiagnostic.source, but the wire struct omits it, so JSON decoding silently discards every per-diagnostic source and always uses the initialize-time default. Add the field and use it as the diagnostic source when present (while retaining the initialized source as the fallback).
I have been toying in the past with the idea of a custom dsl for Effect using Volar, basically a new language compiling to TS behind the scenes to allow first class Effect constructs such as schemas, using Volar I had to then create my own compiler wrapping the typescript language server which worked but wasn't really fun, if TS was to support meta-languages natively I think it could do it all directly (as in for this we already have module resolution, custom-code => ts, etc), in my language I was compiling Anyway I am excited to see this initiative, regardless if we end up using it for something or not, I have been worried about the future of systems such as Volar/Vue/Astro since tsgo was announced |
| -// let [|af|] = <|async () => { | ||
| +// let <|[|af|] = async () => { | ||
| // /*GOTO DEF*/await Promise.resolve(0); | ||
| // }|> |
There was a problem hiding this comment.
I could extract this out into a separate PR, but this was an LSP spec violation, inherited from Strada (insofar as the TS Server protocol maps to LSP)—the targetSelectionRange ([|this range marker|]) most be contained in the targetRange (<|this range marker|>, historically called “context range” in Strada).
|
@andrewbranch I am shamelessly vibe coding a similar experiment I did with Volar with this, if you want to check it out at some point I am pushing it here: https://github.com/mikearnaldi/ets-go One point of feedback (which I think is already mentioned in your PR) I needed to enable other language features such as semantic tokens |
| // gated on workspace trust by the client. It corresponds to the --loadExternalPlugins CLI flag. | ||
| LoadExternalPlugins bool | ||
| DebounceDelay time.Duration | ||
| Locale locale.Locale |
There was a problem hiding this comment.
#4660 is removing this because it's unused / the local can change due to user preference changes. Not sure what to do about that
|
Quick update, working on https://github.com/mikearnaldi/ets-go I ended up finding some issues, the patches are in https://github.com/mikearnaldi/ets-go/tree/main/patches , all but 0005 are bug fixes , 0005 is a new api I needed for a specific integration (which is not necessarily in scope for this PR) |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
This is way more powerful than what I expected. It'll definitely save a lot of work for all the non-TS file tooling. Some of the features will just work without needing enhancement. There are a few questions I want to ask:
For mapping, Svelte currently uses sourcemap generated by magic-string. We're using the |
|
@mikearnaldi good catches, thank you! I will push some of these to this PR. I think patch 0002 isn’t quite right; I started on a different fix today but didn’t have time to finish it. It got surprisingly complicated. When you map a position at the very end of a span, what to return is ambiguous, but it doesn’t depend on whether that position is part of the last segment in the file. The ambiguity is there at the end of any span, and it’s particularly problematic at the boundary between two original text spans that map to non-contiguous virtual text spans: At position 3 in the original text, where do you map to? It looks like span B in the diagram, but if that were a completions request at position 3, that would put the insertion caret right between |
|
Wanted to give a quick update here before I’m out on vacation for a week and ask for feedback on a couple specific things from any implementers, especially anyone coming from Volar. First, I’m reconsidering the
I stand by the motivation here, but I think I also want to get feedback on the constraints I’ve put on overlapping segments. The current rules are
I’m not sure if Volar has either of these restrictions; I think it at least doesn’t have the latter. I couldn’t think of a realistic example that would rely on overlapping but non-duplicate spans in the original text, and had trouble reasoning about the semantic implications of that kind of mapping. Another question that came up in the latest design meeting about this was whether content mapper processes would need to read their own configuration files, beyond the
These are solvable problems, but at a performance and complexity penalty. How do existing transforms work? Are they consuming arbitrary config from the workspace? @jasonlyu123 I just saw your comment as I was about to paste this—I’m on a plane with terrible WiFi so I’ll have to look closer at your links in a week. But your input about specific language feature overrides supports the first change I mentioned I was considering, so that’s helpful feedback! I’m not sure I fully understand the questions about the IPC API. The protocol in this PR is totally separate from the IPC API you can get by importing Also, yes, there will be a way to get a parsed SourceFile without setting up a whole program, so that would help with the script tag parsing. Coming soon. |
I am sure my patches are not perfect, I am mostly iterating with agents so please take everything in my repo as a "maybe", the reason for that patch is: when the file is empty or when the cursor is pointed at the end of a file I would not get LSP completions, with my patch I get LSP completions but I am not sure it's correct :) |
Yeah. I should have worded it more clearly. The IPC API question is related to the first question. And in both cases, what I want to say is LSP-connected IPC API.
I think this should answer my question. The scenario of my question is: when the Svelte language server handles a completions request, whether it's triggered in TS client-side middleware or a standard LSP request, the Svelte language server uses the LSP-connected IPC API to ask about the completion result. In that case, should the parameter sent to the IPC API be a generated position or the source position? Another thing is whether we can ask for positions that are purely generated? From the current implementation, it looks like the LSP-connected IPC API should all use generated positions and won't have mapping validations. So I am mostly asking for confirmation. |





Status: pending team discussion and community feedback. Design needs thorough validation by Volar and other interested ecosystems.
Implements #2824 (comment)
Overview
Content mappers are external integrations that allow TypeScript to include otherwise unsupported file types in a program. They transform a foreign file’s original text into valid TypeScript syntax and provide mappings between the original and transformed content.
Users specify a set of file extensions to be handled by a content mapper package in a
tsconfig.jsonfile:{ "compilerOptions": { // ... }, "contentMappers": [ { "package": "vue-content-mapper", "extensions": [".vue"] } ], "include": ["src"] // implicitly includes .vue as well as .ts }When
contentMappersare specified,tscmust be run with--loadExternalPlugins. VS Code passes--loadExternalPluginstotsc --lsponly in trusted workspaces; otherwise,contentMappersare ignored in the LSP server.The
packagefield will be resolved as a Node.js module name. The package.json of the content mapper package must specify atsContentMappertop-level field describing how to spawn the content mapper process and what compiler options its transform requires:{ "name": "vue-content-mapper", "version": "1.0.0", "tsContentMapper": { "exec": ["node", "dist/server.js"], "compilerOptions": ["module", "jsx", "jsxImportSource"] } }Note that the content mapper process need not be run with Node.js or implemented in JavaScript; package resolution serves as a convenient way to associate a content mapper with a versioned identity that can be managed alongside other dependencies in a project, but the
execfield can specify any command.Protocol
When constructing a program for a config that specifies
contentMappers, module resolution will recognize file lookups for the specified extensions, spawn any content mapper processes needed, and request transformed content from them over STDIO. Content mappers communicate with TypeScript over JSON-RPC. TypeScript sends all requests; mappers do not send requests or notifications. The protocol defines two messages that mappers must handle:initializeandtransform.Span maps
For a content mapper to be useful, it needs to provide a mapping between the transformed output and the original content. In the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. Take a simple example:
TypeScript sees and checks the transformed content, in this case producing a diagnostic for the string literal
"oops"because it is not a number. The span mapping allows TypeScript to report the diagnostic in the original content, at the correct location of the string literal ([7, 13), instead of[10, 16)).In this example,
addmapped to+withSpanMapKind.Atom, indicating a correspondence between the two spans, but with different lengths and content. If the nameaddfailed to resolve, the displayed diagnostic range would cover+, but the message would still reference the identifieradd:The mapper can use
SpanMapKind.Aliasinstead ofSpanMapKind.Atomto indicate that the generated and original text name the same entity. When the diagnostic is rendered, the original text of the alias span (+) will be substituted for the generated text (add) in the diagnostic message:Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. Unlike in Volar, diagnostics in unmappable regions are not discarded. In the CLI, they cause a short snippet of the transformed content to be shown with the diagnostic. A common case may be a content mapper that synthesizes an import statement at the top of the file used in scaffolding. If that import fails to resolve, the user will see:
Spans in the generated output must not overlap, but multiple may map to the same span in the original content. In other words, one range in the original content can map to multiple ranges in the transformed content. This can be useful in the language server when combined with
SpanMapPurposeandSpanMapKind. Broadly speaking, when a language server request is received for a position in a content-mapped file, the handler maps it to one or more projections into the transformed content, chooses how and whether to handle multiple projections based on the kind and purpose of each, performs analysis on the transformed content, and maps results back into original content coordinates.The language server currently supports the following features for content-mapped files:
SpanMapPurpose.Semanticprojection.SpanMapPurpose.Semanticprojection.SpanMapPurpose.Semanticprojection, only if it isSpanMapKind.Verbatim.SpanMapPurpose.Navigationprojections and merges results.SpanMapPurpose.Navigationprojections and merges results.SpanMapPurpose.Navigationprojections and merges results.SpanMapPurpose.Navigationprojections and merges results.SpanMapPurpose.Navigationprojections and merges results.SpanMapPurpose.Navigationprojection, but only verbatim spans may initiate a rename or be edited.Additional language service features may be added in the future. Aside from diagnostics, language service requests can be disabled for any span by explicitly setting its
purposetoSpanMapPurpose.None. Ifpurposeis omitted from the span mapping tuple, it defaults toSpanMapPurpose.All, enabling all supported language service features for that span.Note
This feature mapping is not as granular as what Volar allows, and it must be statically determined by the content mapper during transformation. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside
<script>blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. The current proposal goes beyond that, but is intentionally limited in scope. We expect that ecosystems implementing complex transforms will invariably want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features. Content mappers provide a baseline editing experience, but they also provide the API foundation for more specialized language servers to build on. Vue tooling, for example, may choose to implement a content mapper with language features enabled only for verbatim<script>blocks, while a separate language server handles the rest, accessing the AST, type, and symbol information of their transformed content through an API connection to TypeScript’s language server.Failure handling
Mappers return
diagnosticsfor unparseable content, and errors in the transformed text itself are handled by TypeScript like any other file. If the mapper fails in an unexpected way (e.g., crashes or doesn’t conform to the protocol), TypeScript will report a program diagnostic and treat the file as an empty TypeScript file. After five failures in a single project, TypeScript will stop attempting to transform files with that content mapper and issue a final diagnostic reporting the failure.LSP activation
TypeScript’s language server can only know to care about the file extensions registered in
contentMappersonce the server is running and has discovered atsconfig.jsonthat specifies them. In the case where a user opens a directory in VS Code and opens a single.vuefile, the TypeScript VS Code extension hasn’t even activated, much less spawned a server that knows about acontentMappersregistration. To address this, third-party VS Code extensions need to explicitly activate the TypeScript extension and request that it search for content mappers handling open files:Extension-provided content mappers are not yet supported, but a follow-up exploration is planned.
Emit
Content-mapped files are not emitted to JavaScript. When
--declarationis enabled, however, declaration files are emitted from the transformed content. The declaration file name forApp.svelteisApp.d.svelte.ts. Declaration maps are currently not supported.Incremental, build, watch, and process consolidation
Content mappers are supported in
--incremental,--build, and--watchmodes. A content mapper’s name and version are recorded in the.tsbuildinfofile and used as part of up-to-date checks. If a content mapper’s version changes, the next build will re-transform all files handled by that content mapper.Note
Modifying a content mapper implementation during local development will not change its identity, so you’ll need to bump the local package.json version, or use
--forceor--cleanto clear cached outputs if testing content mappers with--incrementalor--build.In
--buildmode with project references, and in some instances in the language server, it’s possible to have a project graph with many projects all defining the same content mapper. To avoid excessive spawning of child processes, TypeScript deduplicates content mapper processes by name and version. A single content mapper process cannot be assumed to be responsible for only one project; it must be able to handle requests for any file in any project (or outside of any loaded project, in LSP scenarios) that uses it in any order. Eachtransformrequest passes thecompilerOptionssubset of the project that requested the transform for that file, and must be handled idempotently.API integration
Content-mapped SourceFiles can be inspected by the JavaScript API. For a content-mapped SourceFile,
file.textis the transformed text,file.originalTextis the original text, andfile.spanMapexposes an API for mapping between the two. Regardless of thepositionEncodingused by the content mapper, accessing the span map through the JavaScript API always yields UTF-16 positions.To-do
--extendedDiagnosticsand LSP logs