fix: parse heredocs with a single-character delimiter (#314) - #323
Merged
Conversation
The spec defines the heredoc delimiter as an Identifier -- ID_Start
(ID_Continue | '-')* -- whose trailing * permits one character. The
grammar used +, requiring a second, so `<<E` fell through to STRING_CHARS
and the parse failed while `<<EO` succeeded.
Long-standing rather than a v8 regression: the same pattern is present
pre-8.x.
The delimiter regex is duplicated in hcl2/utils.py, where HEREDOC_PATTERN
and HEREDOC_TRIM_PATTERN re-match a token the grammar has already
accepted. Relaxing only the grammar made `<<E` parse but raise
RuntimeError("Invalid Heredoc token") the moment preserve_heredocs=False
tried to flatten it -- caught by the new flatten test, not by the
existing suite. Both places now track each other, with a comment saying
why they must.
Tests cover the plain and trimmed forms, an empty body, flattening, an
attribute after the heredoc, and a body line ending in the delimiter --
that last one matters more with a one-character delimiter, since an
accidental match is likelier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rddimon
approved these changes
Aug 26, 2026
kkozik-amplify
added a commit
that referenced
this pull request
Aug 26, 2026
#323 landed the single-character delimiter fix. Only test_api.py conflicted -- both branches append a test class -- resolved by keeping both, main's first. CHANGELOG.md auto-merged this time thanks to the union driver from #319, so the usual two-file conflict was one. Verified the two heredoc changes compose: a single-character delimiter flattened to a value yields real newlines for the plain, trimmed and empty forms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kkozik-amplify
added a commit
that referenced
this pull request
Aug 26, 2026
#323 and #324 landed after this branch was opened. Added in the same style as the rest of the section -- PR link, no credit line, since both were fixed in-house. CHANGELOG.md conflicted because main carried the ungroomed form of every entry this branch had already rewritten; kept the groomed text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
caused by #314
Problem
<<Efails to parse while<<EOsucceeds. The spec defines the delimiter as anIdentifier:The trailing
*permits a single character. The grammar used+, requiring a second one, so<<Efell through toSTRING_CHARS.Long-standing rather than a v8 regression — the same pattern is present pre-8.x.
The fix is in two places, not one
Relaxing
+→*inhcl2.larkmakes<<Eparse, and the full suite stayed green. That was misleading:hcl2/utils.pycarries a duplicate of the delimiter regex inHEREDOC_PATTERN/HEREDOC_TRIM_PATTERN, which re-match a token the grammar has already accepted. With only the grammar relaxed:The grammar accepted it and the serializer then rejected it. Nothing in the existing suite flattens a single-character heredoc, so this only surfaced because the new tests cover the flatten path. Both sites now carry a comment noting they must track each other — this is the third time they have drifted (see also the CRLF work in #317).
Tests
TestSingleCharacterHeredocDelimiter— plain and trimmed forms, empty body, flattening, an attribute after the heredoc, and a body line ending in the delimiter (sayE/E), which matters more at one character since an accidental match is likelier. A multi-character control confirms nothing shifted.Without the grammar change, 6 of the 7 fail.
Test plan
pre-commit run: cleanSTRING_CHARSand the operator terminals are unaffectedScope: the character class is deliberately untouched
This PR changes the delimiter's length rule (
+→*), not its character class. Worth being explicit, because the spec argument above would also implicate the class, and it does not line up with the grammar's own identifier terminal in either direction:NAME)_EOFEO.FEO_F,EO-F,E11ENAMEis[a-zA-Z_][a-zA-Z0-9_-]*; the heredoc delimiter is[a-zA-Z][a-zA-Z0-9._-]*. So a leading underscore is a valid identifier but not a valid delimiter, and a dot is a valid delimiter but not a valid identifier.Left alone on purpose: #314 is specifically about length, nobody has reported the class, and changing it would widen what the lexer accepts on a path that has already needed three corrections this cycle. Filing it separately if we want it.
Test scope
Unit tests only, no integration fixture — the same call made for the empty-heredoc fix in #312. The behaviour is a lexer-level accept/reject plus one flatten path, both of which unit tests cover directly; a golden fixture would add four files without adding signal.