Skip to content

Update logos to 0.16 - #133

Merged
adampetro merged 2 commits into
mainfrom
ap.logos-0.16
Aug 12, 2026
Merged

Update logos to 0.16#133
adampetro merged 2 commits into
mainfrom
ap.logos-0.16

Conversation

@adampetro

Copy link
Copy Markdown
Contributor

What

Update the logos dependency from 0.15 to 0.16.

Required source change

Logos 0.16 rejects greedy patterns that are equivalent to a dot repetition, unless the pattern opts in. The comment skip pattern needs this opt-in. The greedy match to the end of the line is the intended behavior for GraphQL comments:

#[logos(skip(r"#[^\n\r]*", allow_greedy = true))] // comments

Behavior changes

The lexer regression tests from #132 found one change in error recovery. Logos 0.16 reports one error for a failed match attempt, and the error span covers all consumed bytes. Logos 0.15 reported an error for only the first byte:

Input 0.15 0.16
.. two errors, spans 0..1 and 1..2 one error, span 0..2
"\u12" inner error span 1..2 inner error span 1..5
"\u{}" inner error span 1..2 inner error span 1..4

The new spans point at the full invalid sequence, so the error messages become more precise. Valid documents lex the same as before. No other test in the workspace changed behavior.

Stack usage changes

Logos 0.15 and 0.16 both generate matcher code that can recurse for each character in unoptimized builds. Optimized builds compile the recursion into loops and are not affected. The affected inputs changed in 0.16:

Input class (unoptimized builds) 0.15 0.16
Large numeric tokens stack overflow fixed
Large ASCII ignored runs stack overflow fixed
Skip runs with byte order marks stack overflow stack overflow
Strings with many multi-byte characters bounded stack overflow (new)

This PR restructures the stack usage tests for this:

  • The tests for large ASCII tokens are now enabled. They prevent a regression of the 0.16 fix.
  • The ignored tests now cover only the multi-byte inputs. Their #[ignore] reasons name logos 0.16. Run cargo test -p bluejay-parser -- --ignored in a debug build after the next logos upgrade. If the tests pass, remove the ignore attributes.

The multi-byte problem is a regression in logos 0.16. It only affects unoptimized builds, so it does not block this upgrade. A minimal reproduction is ready for an upstream report: a [^"]+ token overflows a 1 MiB stack at approximately 1,000 multi-byte characters, while 0.15 lexes one million multi-byte characters in the same stack.

Performance

The criterion benchmarks (cargo bench -p bluejay-parser --bench parse) show no change outside the measurement noise. A control run of 0.15 against its own baseline showed the same magnitude of variation (up to ±5%) as the 0.15 to 0.16 comparison.

How to verify

cargo test --all-features
cargo test -p bluejay-parser --release --features format-errors -- --ignored

The first command matches CI. The second command shows that the ignored tests pass in optimized builds.

Update the logos dependency from 0.15 to 0.16.

Add allow_greedy to the comment skip pattern. Logos 0.16 requires
this opt-in for greedy patterns that are equivalent to a dot
repetition. The greedy match to the end of the line is the intended
behavior for GraphQL comments.

Update three test expectations for a change in error recovery.
Logos 0.16 reports one error for a failed match attempt. The error
span covers all consumed bytes. Logos 0.15 reported an error for
only the first byte:
- ".." gives one error with span 0..2, not two errors.
- The inner error for "\u12" in a string covers the full escape
  sequence.
- The inner error for "\u{}" in a string covers the full escape
  sequence.

Restructure the stack usage tests for the changes in logos 0.16:
- Logos 0.16 lexes large ASCII tokens and large ASCII ignored runs
  with bounded stack usage. Logos 0.15 caused a stack overflow for
  these inputs in unoptimized builds. Enable the tests for these
  inputs to prevent a regression.
- Logos 0.16 uses stack space proportional to the match length for
  runs of multi-byte characters, for example emoji in a string.
  Logos 0.15 lexed these inputs with bounded stack usage. Optimized
  builds are not affected. Narrow the ignored tests to these inputs
  and update the documentation.

Benchmark results show no performance change outside the
measurement noise.

Assisted-By: devx/a4890344-dddf-4733-a94e-71230a3953d1
Add comment separators with carriage return and line feed, and with
carriage return alone, to the kitchen sink token stream test. The
test asserts exact spans for the tokens that follow each comment.
This makes sure that a comment ends at each line terminator and
does not consume the tokens on the next line.

Assisted-By: devx/a4890344-dddf-4733-a94e-71230a3953d1
@adampetro
adampetro marked this pull request as ready for review August 12, 2026 20:07
@adampetro
adampetro requested a review from swalkinshaw August 12, 2026 20:07

@swalkinshaw swalkinshaw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error span change is a nice improvement 👍

Thanks for clarying the unoptimized vs optimized build breakdown.

@adampetro
adampetro merged commit 53a5668 into main Aug 12, 2026
5 checks passed
@adampetro
adampetro deleted the ap.logos-0.16 branch August 12, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants