From ec91639a0dcd148fcd766ede4b45129b483caad9 Mon Sep 17 00:00:00 2001 From: Kamil Kozik Date: Wed, 26 Aug 2026 10:49:19 +0200 Subject: [PATCH 1/2] fix: return heredoc bodies as real multi-line values when unquoting (#303) The flatten path escapes the body to build a quoted-string source form (`'"a\nb"'`). That escaping ran before the strip_string_quotes early return, so a caller asking for the *value* got escaped *source* back: every line break arrived as a literal backslash-n. Reported from production, where a heredoc-defined PGP private key came out as a single line and was silently unusable. Before this fix no combination of SerializationOptions reproduced v7's plain multi-line string -- confirmed by brute-forcing all sixteen combinations of strip_string_quotes, preserve_heredocs, wrap_objects and explicit_blocks. This is the same defect #313 fixed for quoted strings -- strip_string_quotes should yield values, not source -- which touched StringRule and left both heredoc rules behind. Moving the escaping after the early return in each is the whole change; the quoted form is unaffected. The only two tests that failed were the ones asserting the reported behaviour (`"line1\\nline2"`). Both now assert real newlines, and each gains a sibling pinning that the quoted form still escapes, so the two paths cannot drift again. Docs: the option table described preserve_heredocs only as "keep heredocs in their original form", and the migration guide's V7_COMPAT recipe omitted it entirely -- which is how the reporter ended up without a working combination. Both now cover it, and the guide's example output is verified against the code. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + docs/01_getting_started.md | 2 +- docs/06_migrating_to_v8.md | 10 ++++++ hcl2/rules/strings.py | 12 +++++-- test/unit/rules/test_strings.py | 20 ++++++++++-- test/unit/test_api.py | 56 +++++++++++++++++++++++++++++++++ 6 files changed, 95 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54bb3b74..34881c2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ### Fixed - Restore `py.typed` marker so type checkers recognize `hcl2` (and `cli`) as typed packages. ([#298](https://github.com/amplify-education/python-hcl2/issues/298)) +- `preserve_heredocs=False` combined with `strip_string_quotes` now returns the heredoc body as a plain multi-line string instead of escaping every newline to a literal `\n`. The escaping is still applied to the quoted source form produced without `strip_string_quotes`. ([#303](https://github.com/amplify-education/python-hcl2/issues/303)) - Parse heredocs with an empty body again. A marker immediately followed by its closing delimiter failed to match, and the lexer then ran on to a later delimiter, silently absorbing the attributes in between. ([#309](https://github.com/amplify-education/python-hcl2/issues/309)) - Negative integer literals load as numbers again instead of `${-N}` expression strings, matching negative floats and the pre-8.x behaviour. ([#307](https://github.com/amplify-education/python-hcl2/issues/307)) - `strip_string_quotes` no longer unquotes string literals nested inside expressions, which produced invalid HCL such as `${upper(x)}` from `upper("x")`. ([#310](https://github.com/amplify-education/python-hcl2/issues/310)) diff --git a/docs/01_getting_started.md b/docs/01_getting_started.md index f34f1d0e..431fc06b 100644 --- a/docs/01_getting_started.md +++ b/docs/01_getting_started.md @@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions( | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings | | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings | | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.** | -| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form | +| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. | | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations | | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is | | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** | diff --git a/docs/06_migrating_to_v8.md b/docs/06_migrating_to_v8.md index d6383e0a..36f1be58 100644 --- a/docs/06_migrating_to_v8.md +++ b/docs/06_migrating_to_v8.md @@ -201,9 +201,19 @@ V7_COMPAT = SerializationOptions( strip_string_quotes=True, explicit_blocks=False, with_comments=False, + preserve_heredocs=False, ) data = hcl2.load(f, serialization_options=V7_COMPAT) ``` This restores the v7 dict shape but disables round-trip support and comment preservation. + +`preserve_heredocs=False` matters if your configuration uses heredocs. Left at its default, a heredoc value keeps its `<<-EOT` / `EOT` markers embedded in the string. Turned off alongside `strip_string_quotes`, you get the body as a plain multi-line string with real line breaks, which is what v7 returned: + +```python +hcl2.loads('x = <<-EOT\n line1\n line2\n EOT\n', serialization_options=V7_COMPAT) +# {'x': 'line1\nline2'} +``` + +Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable. diff --git a/hcl2/rules/strings.py b/hcl2/rules/strings.py index 5c4f50e9..76ce0660 100644 --- a/hcl2/rules/strings.py +++ b/hcl2/rules/strings.py @@ -170,9 +170,12 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext if not match: raise RuntimeError(f"Invalid Heredoc token: {heredoc}") heredoc = _strip_closing_marker_line(match.group(2)) - heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") if options.strip_string_quotes: + # The caller asked for the value, so hand back the body as-is: + # real newlines, no escaping. The escaping below exists only to + # build the quoted-string *source* form returned otherwise. return heredoc + heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") return f'"{heredoc}"' result = heredoc.rstrip(self._trim_chars) @@ -229,10 +232,13 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext if not options.preserve_heredocs: lines = [line.replace("\\", "\\\\").replace('"', '\\"') for line in lines] + if options.strip_string_quotes: + # Value, not source: join with real newlines regardless of + # preserve_heredocs, and skip the escaping done for the quoted form. + return "\n".join(lines) + sep = "\\n" if not options.preserve_heredocs else "\n" inner = sep.join(lines) - if options.strip_string_quotes: - return inner return '"' + inner + '"' diff --git a/test/unit/rules/test_strings.py b/test/unit/rules/test_strings.py index ce65aac3..ef800d91 100644 --- a/test/unit/rules/test_strings.py +++ b/test/unit/rules/test_strings.py @@ -280,10 +280,18 @@ def test_serialize_strip_string_quotes_preserve(self): self.assertEqual(rule.serialize(opts), "< Date: Wed, 26 Aug 2026 10:57:10 +0200 Subject: [PATCH 2/2] docs: note the two heredoc value gotchas in the migration guide Self-review turned up an asymmetry the guide did not mention: with strip_string_quotes set, a quoted string's `\n` resolves to a newline while a heredoc body's stays two literal characters. Both are correct -- HCL processes escapes in quoted templates only -- but a reader coming from the V7_COMPAT recipe has no way to predict it. Same for line endings: a heredoc in a CRLF file yields a body containing `\r\n`, since a carriage return inside the body is content rather than structure. Example output verified against the code rather than written from memory. Co-Authored-By: Claude Opus 5 (1M context) --- docs/06_migrating_to_v8.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/06_migrating_to_v8.md b/docs/06_migrating_to_v8.md index 36f1be58..0c4703bf 100644 --- a/docs/06_migrating_to_v8.md +++ b/docs/06_migrating_to_v8.md @@ -217,3 +217,13 @@ hcl2.loads('x = <<-EOT\n line1\n line2\n EOT\n', serialization_options=V7_COM ``` Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable. + +Two details of heredoc values are easy to trip over, and both match how HCL itself behaves: + +- **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates. +- **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`. + +```python +hcl2.loads('x = <