unified: Add Swift parser based on swift-syntax#22195
Open
tausbn wants to merge 13 commits into
Open
Conversation
Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.
These were taking up roughly 20% of the JSON payload.
Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.
Introduces new yeast types for `Point`s and `Range`s (corresponding to the tree-sitter equivalents), since it seemed silly to have `swift-syntax-rs` pull in `tree-sitter` just to have those types available. This _does_ mean there is a slight overhead in the shared extractor when converting between these types, but I think this is negligible.
Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.
At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Swift-native swift-syntax parser exposed through Rust and prepares unified extraction to consume its JSON AST.
Changes:
- Adds Swift/Rust FFI parser and Bazel/Cargo builds.
- Adds JSON-to-yeast AST adaptation and minimal Swift rules.
- Extends yeast for externally constructed ASTs and owned locations.
Show a summary per file
| File | Description |
|---|---|
MODULE.bazel |
Registers Swift dependencies and toolchains. |
Cargo.toml |
Adds the parser crate to the workspace. |
Cargo.lock |
Locks the new crate. |
shared/yeast-schema/src/schema.rs |
Supports merging schema names. |
shared/yeast-macros/src/parse.rs |
Uses yeast-owned ranges. |
shared/yeast/src/build.rs |
Migrates build contexts to owned ranges. |
shared/yeast/src/lib.rs |
Adds external AST construction/desugaring APIs. |
shared/yeast/src/range.rs |
Defines owned point and range types. |
shared/yeast/tests/test.rs |
Tests desugaring hand-built ASTs. |
shared/tree-sitter-extractor/src/extractor/mod.rs |
Converts yeast points for extraction. |
unified/extractor/src/languages/mod.rs |
Exposes the Swift JSON adapter. |
unified/extractor/src/languages/swift/adapter.rs |
Converts JSON into a yeast AST. |
unified/extractor/src/languages/swift/swift.rs |
Adds minimal swift-syntax rules. |
unified/extractor/tests/fixtures/let_x.swiftsyntax.json |
Provides parser output fixture. |
unified/extractor/tests/swift_syntax_pipeline.rs |
Tests the adapter/desugarer pipeline. |
unified/swift-syntax-rs/.gitignore |
Ignores generated outputs. |
unified/swift-syntax-rs/.swift-version |
Pins Swift 6.3.2. |
unified/swift-syntax-rs/BUILD.bazel |
Defines Bazel Swift/Rust targets. |
unified/swift-syntax-rs/Cargo.toml |
Defines the Rust crate and CLI. |
unified/swift-syntax-rs/README.md |
Documents building and usage. |
unified/swift-syntax-rs/build.rs |
Builds and links the Swift shim for Cargo. |
unified/swift-syntax-rs/src/lib.rs |
Implements safe Rust FFI bindings. |
unified/swift-syntax-rs/src/main.rs |
Adds the parser CLI. |
unified/swift-syntax-rs/swift/Package.resolved |
Locks swift-syntax. |
unified/swift-syntax-rs/swift/Package.swift |
Defines the Swift FFI package. |
unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift |
Serializes SwiftSyntax ASTs as JSON. |
unified/swift-syntax-rs/xcode_transition.bzl |
Isolates macOS Xcode configuration. |
Review details
- Files reviewed: 26/27 changed files
- Comments generated: 9
- Review effort level: Medium
Comment on lines
+41
to
+43
| println!("cargo:rustc-link-search=native={}", build_dir.display()); | ||
| println!("cargo:rustc-link-lib=dylib=SwiftSyntaxFFI"); | ||
| println!("cargo:rustc-link-arg=-Wl,-rpath,{}", build_dir.display()); |
- BUILD.bazel: require x86_64 on the Linux branch of the compatibility select. The only registered standalone Linux Swift toolchain is x86_64-only, so without the CPU constraint Linux/aarch64 targets stayed "compatible" and then failed toolchain resolution instead of being skipped cleanly. - build.rs: emitting `rerun-if-changed` disables Cargo's default whole-package scan, so list every input to `swift build` — the package manifests, `Package.resolved`, `.swift-version`, and the whole Sources tree — so stale shim/toolchain output can't linger. (`.build/` is left unwatched, as it is this build's own output.) - src/lib.rs: a null result from the shim is not a parse failure — SwiftParser recovers from invalid syntax and always yields a tree — so it means the shim failed to serialize/allocate the JSON. Fix the error message and the variant doc accordingly. - README.md: `.swift-version` is not honored by the macOS Bazel build (which uses the host Xcode toolchain); document that it pins the Linux and local builds only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The doc claimed the schema passed to `Ast::with_schema` must already have every node kind and field name registered, contradicting the registration methods just below and the swift-syntax adapter, which starts from `Schema::new()` and registers names on demand during construction. Document that up-front registration is optional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.
Registers the `unified/swift-syntax-rs` workspace member (added in "unified: Add swift-syntax-rs") in the tree-sitter extractors crate universe. It has no external Rust dependencies, so its dependency entries are empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tausbn
force-pushed
the
tausbn/swift-syntax-rs
branch
from
4ba73bd to
da1bbb7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an alternative parser to
tree-sitter-swift, using the Swift-nativeswift-syntaxpackage instead.The current implementation exposes this parser through a Rust wrapper, using the C FFI to communicate with Swift. The Rust component passes the source file contents to the Swift component, which in turn responds with a JSON string containing the AST.
Note that at present, the AST is quite verbose (though some effort has been made to make it less so). In particular, it contains detailed location information about every single token in the input, which is almost certainly overkill for our purposes.
This PR does not attempt to replace the existing
yeastrules to be based on theswift-syntaxAST. That is left as follow-up work.Can be reviewed commit-by-commit.