Skip to content

fix: parse heredocs with a single-character delimiter (#314) - #323

Merged
kkozik-amplify merged 1 commit into
mainfrom
fix/single-char-heredoc-delimiter
Aug 26, 2026
Merged

fix: parse heredocs with a single-character delimiter (#314)#323
kkozik-amplify merged 1 commit into
mainfrom
fix/single-char-heredoc-delimiter

Conversation

@kkozik-amplify

@kkozik-amplify kkozik-amplify commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

caused by #314

Problem

<<E fails to parse while <<EO succeeds. The spec defines the delimiter as an Identifier:

heredocTemplate = (("<<" | "<<-") Identifier Newline ... Identifier Newline);
Identifier      = ID_Start (ID_Continue | '-')*;

The trailing * permits a single character. The grammar used +, requiring a second one, so <<E fell through to STRING_CHARS.

Long-standing rather than a v8 regression — the same pattern is present pre-8.x.

The fix is in two places, not one

Relaxing +* in hcl2.lark makes <<E parse, and the full suite stayed green. That was misleading: hcl2/utils.py carries a duplicate of the delimiter regex in HEREDOC_PATTERN / HEREDOC_TRIM_PATTERN, which re-match a token the grammar has already accepted. With only the grammar relaxed:

hcl2.loads("a = <<E\nx\nE\n", serialization_options=SerializationOptions(preserve_heredocs=False))
# RuntimeError: Invalid Heredoc token: <<E\nx\nE

The grammar accepted it and the serializer then rejected it. Nothing in the existing suite flattens a single-character heredoc, so this only surfaced because the new tests cover the flatten path. Both sites now carry a comment noting they must track each other — this is the third time they have drifted (see also the CRLF work in #317).

Tests

TestSingleCharacterHeredocDelimiter — plain and trimmed forms, empty body, flattening, an attribute after the heredoc, and a body line ending in the delimiter (sayE / E), which matters more at one character since an accidental match is likelier. A multi-character control confirms nothing shifted.

Without the grammar change, 6 of the 7 fail.

Test plan

  • Full suite: 1511 passing (baseline 1504)
  • pre-commit run: clean
  • No lexing ambiguity observed; STRING_CHARS and the operator terminals are unaffected

Scope: the character class is deliberately untouched

This PR changes the delimiter's length rule (+*), not its character class. Worth being explicit, because the spec argument above would also implicate the class, and it does not line up with the grammar's own identifier terminal in either direction:

Delimiter Heredoc Valid identifier (NAME)
_EOF rejected accepted
EO.F accepted rejected
EO_F, EO-F, E1 accepted accepted
1E rejected rejected

NAME is [a-zA-Z_][a-zA-Z0-9_-]*; the heredoc delimiter is [a-zA-Z][a-zA-Z0-9._-]*. So a leading underscore is a valid identifier but not a valid delimiter, and a dot is a valid delimiter but not a valid identifier.

Left alone on purpose: #314 is specifically about length, nobody has reported the class, and changing it would widen what the lexer accepts on a path that has already needed three corrections this cycle. Filing it separately if we want it.

Test scope

Unit tests only, no integration fixture — the same call made for the empty-heredoc fix in #312. The behaviour is a lexer-level accept/reject plus one flatten path, both of which unit tests cover directly; a golden fixture would add four files without adding signal.

The spec defines the heredoc delimiter as an Identifier -- ID_Start
(ID_Continue | '-')* -- whose trailing * permits one character. The
grammar used +, requiring a second, so `<<E` fell through to STRING_CHARS
and the parse failed while `<<EO` succeeded.

Long-standing rather than a v8 regression: the same pattern is present
pre-8.x.

The delimiter regex is duplicated in hcl2/utils.py, where HEREDOC_PATTERN
and HEREDOC_TRIM_PATTERN re-match a token the grammar has already
accepted. Relaxing only the grammar made `<<E` parse but raise
RuntimeError("Invalid Heredoc token") the moment preserve_heredocs=False
tried to flatten it -- caught by the new flatten test, not by the
existing suite. Both places now track each other, with a comment saying
why they must.

Tests cover the plain and trimmed forms, an empty body, flattening, an
attribute after the heredoc, and a body line ending in the delimiter --
that last one matters more with a one-character delimiter, since an
accidental match is likelier.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kkozik-amplify
kkozik-amplify requested a review from a team as a code owner August 26, 2026 08:46
@kkozik-amplify
kkozik-amplify merged commit 37c0b8f into main Aug 26, 2026
8 checks passed
@kkozik-amplify
kkozik-amplify deleted the fix/single-char-heredoc-delimiter branch August 26, 2026 09:41
kkozik-amplify added a commit that referenced this pull request Aug 26, 2026
#323 landed the single-character delimiter fix. Only test_api.py
conflicted -- both branches append a test class -- resolved by keeping
both, main's first.

CHANGELOG.md auto-merged this time thanks to the union driver from #319,
so the usual two-file conflict was one.

Verified the two heredoc changes compose: a single-character delimiter
flattened to a value yields real newlines for the plain, trimmed and
empty forms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kkozik-amplify added a commit that referenced this pull request Aug 26, 2026
#323 and #324 landed after this branch was opened. Added in the same
style as the rest of the section -- PR link, no credit line, since both
were fixed in-house.

CHANGELOG.md conflicted because main carried the ungroomed form of every
entry this branch had already rewritten; kept the groomed text.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants