fix: return heredoc bodies as real multi-line values when unquoting (#303)

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Fixed
 
 - Restore `py.typed` marker so type checkers recognize `hcl2` (and `cli`) as typed packages. ([#298](https://github.com/amplify-education/python-hcl2/issues/298))
+- `preserve_heredocs=False` combined with `strip_string_quotes` now returns the heredoc body as a plain multi-line string instead of escaping every newline to a literal `\n`. The escaping is still applied to the quoted source form produced without `strip_string_quotes`. ([#303](https://github.com/amplify-education/python-hcl2/issues/303))
 - Parse heredocs whose delimiter is a single character, such as `<<E`. The spec defines the delimiter as an Identifier, which permits one character. ([#314](https://github.com/amplify-education/python-hcl2/issues/314))
 - Parse heredocs with an empty body again. A marker immediately followed by its closing delimiter failed to match, and the lexer then ran on to a later delimiter, silently absorbing the attributes in between. ([#309](https://github.com/amplify-education/python-hcl2/issues/309))
 - Negative integer literals load as numbers again instead of `${-N}` expression strings, matching negative floats and the pre-8.x behaviour. ([#307](https://github.com/amplify-education/python-hcl2/issues/307))

docs/01_getting_started.md

@@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings                                                                                                       |
 | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings                                                                                                        |
 | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.**                                    |
-| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form                                                                                                            |
+| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. |
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |

docs/06_migrating_to_v8.md

@@ -201,9 +201,29 @@ V7_COMPAT = SerializationOptions(
     strip_string_quotes=True,
     explicit_blocks=False,
     with_comments=False,
+    preserve_heredocs=False,
 )
 
 data = hcl2.load(f, serialization_options=V7_COMPAT)
 ```
 
 This restores the v7 dict shape but disables round-trip support and comment preservation.
+
+`preserve_heredocs=False` matters if your configuration uses heredocs. Left at its default, a heredoc value keeps its `<<-EOT` / `EOT` markers embedded in the string. Turned off alongside `strip_string_quotes`, you get the body as a plain multi-line string with real line breaks, which is what v7 returned:
+
+```python
+hcl2.loads('x = <<-EOT\n  line1\n  line2\n  EOT\n', serialization_options=V7_COMPAT)
+# {'x': 'line1\nline2'}
+```
+
+Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable.
+
+Two details of heredoc values are easy to trip over, and both match how HCL itself behaves:
+
+- **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates.
+- **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`.
+
+```python
+hcl2.loads('x = <<EOT\na\\nb\nEOT\n', serialization_options=V7_COMPAT)
+# {'x': 'a\\nb'}  — the backslash and the "n" are two literal characters
+```

hcl2/rules/strings.py

@@ -170,9 +170,12 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext
             if not match:
                 raise RuntimeError(f"Invalid Heredoc token: {heredoc}")
             heredoc = _strip_closing_marker_line(match.group(2))
-            heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
             if options.strip_string_quotes:
+                # The caller asked for the value, so hand back the body as-is:
+                # real newlines, no escaping. The escaping below exists only to
+                # build the quoted-string *source* form returned otherwise.
                 return heredoc
+            heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
             return f'"{heredoc}"'
 
         result = heredoc.rstrip(self._trim_chars)
@@ -229,10 +232,13 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext
         if not options.preserve_heredocs:
             lines = [line.replace("\\", "\\\\").replace('"', '\\"') for line in lines]
 
+        if options.strip_string_quotes:
+            # Value, not source: join with real newlines regardless of
+            # preserve_heredocs, and skip the escaping done for the quoted form.
+            return "\n".join(lines)
+
         sep = "\\n" if not options.preserve_heredocs else "\n"
         inner = sep.join(lines)
-        if options.strip_string_quotes:
-            return inner
         return '"' + inner + '"'
 
 

test/unit/rules/test_strings.py

@@ -280,10 +280,18 @@ def test_serialize_strip_string_quotes_preserve(self):
         self.assertEqual(rule.serialize(opts), "<<EOF\nhello world\nEOF")
 
     def test_serialize_strip_string_quotes_no_preserve(self):
+        """Asking for the value yields real newlines, not escaped ones."""
         token = HEREDOC_TEMPLATE("<<EOF\nline1\nline2\nEOF")
         rule = HeredocTemplateRule([token])
         opts = SerializationOptions(preserve_heredocs=False, strip_string_quotes=True)
-        self.assertEqual(rule.serialize(opts), "line1\\nline2")
+        self.assertEqual(rule.serialize(opts), "line1\nline2")
+
+    def test_serialize_no_preserve_keeps_escaping_for_the_quoted_form(self):
+        """Without strip_string_quotes the result is source, so escapes stay."""
+        token = HEREDOC_TEMPLATE("<<EOF\nline1\nline2\nEOF")
+        rule = HeredocTemplateRule([token])
+        opts = SerializationOptions(preserve_heredocs=False)
+        self.assertEqual(rule.serialize(opts), '"line1\\nline2"')
 
 
 # --- HeredocTrimTemplateRule tests ---
@@ -366,7 +374,15 @@ def test_serialize_strip_string_quotes_preserve(self):
         self.assertEqual(rule.serialize(opts), "<<-EOF\n    line1\n    line2\nEOF")
 
     def test_serialize_strip_string_quotes_no_preserve(self):
+        """Asking for the value yields real newlines, not escaped ones."""
         token = HEREDOC_TRIM_TEMPLATE("<<-EOF\n    line1\n    line2\nEOF")
         rule = HeredocTrimTemplateRule([token])
         opts = SerializationOptions(preserve_heredocs=False, strip_string_quotes=True)
-        self.assertEqual(rule.serialize(opts), "line1\\nline2")
+        self.assertEqual(rule.serialize(opts), "line1\nline2")
+
+    def test_serialize_no_preserve_keeps_escaping_for_the_quoted_form(self):
+        """Without strip_string_quotes the result is source, so escapes stay."""
+        token = HEREDOC_TRIM_TEMPLATE("<<-EOF\n    line1\n    line2\nEOF")
+        rule = HeredocTrimTemplateRule([token])
+        opts = SerializationOptions(preserve_heredocs=False)
+        self.assertEqual(rule.serialize(opts), '"line1\\nline2"')

test/unit/test_api.py

@@ -567,3 +567,59 @@ def test_body_line_ending_in_the_delimiter_still_safe(self):
 
     def test_multi_character_delimiter_unaffected(self):
         self.assertEqual(loads("a = <<EOF\nx\nEOF\n"), {"a": '"<<EOF\nx\nEOF"'})
+
+
+class TestHeredocFlattenedToValue(TestCase):
+    """`strip_string_quotes` + `preserve_heredocs=False` yields real newlines.
+
+    The flatten path escapes the body to build a quoted-string *source* form
+    (`'"a\\nb"'`). That escaping ran before the `strip_string_quotes` early
+    return, so asking for the value handed back escaped source instead: every
+    line break arrived as a literal backslash-n. Before this fix no combination
+    of options reproduced v7's plain multi-line string.
+    """
+
+    _VALUE = SerializationOptions(strip_string_quotes=True, preserve_heredocs=False)
+    _SOURCE = SerializationOptions(preserve_heredocs=False)
+
+    def test_value_has_real_newlines(self):
+        result = loads("a = <<EOT\nline1\nline2\nEOT\n", serialization_options=self._VALUE)
+        self.assertEqual(result, {"a": "line1\nline2"})
+
+    def test_trimmed_value_has_real_newlines(self):
+        source = "a = <<-EOT\n  line1\n  line2\n  EOT\n"
+        result = loads(source, serialization_options=self._VALUE)
+        self.assertEqual(result, {"a": "line1\nline2"})
+
+    def test_multiline_secret_survives_intact(self):
+        """The reported case: a heredoc-defined key block must stay multi-line."""
+        source = (
+            "keys = {\n"
+            "  private = <<-EOT\n"
+            "  -----BEGIN PGP PRIVATE KEY BLOCK-----\n"
+            "  line1\n"
+            "  -----END PGP PRIVATE KEY BLOCK-----\n"
+            "  EOT\n"
+            "}\n"
+        )
+        value = loads(source, serialization_options=self._VALUE)["keys"]["private"]
+        self.assertEqual(value.count("\n"), 2)
+        self.assertNotIn("\\n", value)
+        self.assertTrue(value.startswith("-----BEGIN"))
+        self.assertTrue(value.endswith("KEY BLOCK-----"))
+
+    def test_quoted_form_still_escapes(self):
+        """Without strip_string_quotes the result is source and must escape."""
+        result = loads("a = <<EOT\nline1\nline2\nEOT\n", serialization_options=self._SOURCE)
+        self.assertEqual(result, {"a": '"line1\\nline2"'})
+
+    def test_embedded_quote_and_backslash_only_escaped_in_source_form(self):
+        source = 'a = <<EOT\nsay "hi"\nback\\slash\nEOT\n'
+        self.assertEqual(
+            loads(source, serialization_options=self._VALUE),
+            {"a": 'say "hi"\nback\\slash'},
+        )
+        self.assertEqual(
+            loads(source, serialization_options=self._SOURCE),
+            {"a": '"say \\"hi\\"\\nback\\\\slash"'},
+        )