Skip decorative-bullet normalization inside fenced code blocks

[jurgenwerk]

There was a case where ai assistant code patches were failing over and over even though it re-read the file multiple times. Here's an example:

Claude's summary:

The AI assistant's code patches could fail forever with "search pattern not found" even though the search block matched the target file exactly. The cause is in markdownToHtml: it prefixes emoji/star-led lines with a * list marker so marked renders them as lists, and it applied that rewrite to the entire message — including the content of fenced code blocks. The host extracts search/replace patches from the rendered HTML, so a patch containing a line that starts with an emoji was mutated before matching:

bot message (markdown)                 file on disk
  ```gts                                 🚧 SITE UNDER ...
  ╔═══ SEARCH ════╗
  🚧 SITE UNDER ...

    │ markdownToHtml inserts "* " before 🚧
    ▼

• 🚧 SITE UNDER ... ──match──✗──► 🚧 SITE UNDER ...

The bot re-reads the file and resends a correct patch, the renderer corrupts it again, and the loop never converges. The same rewrite also corrupted patches that did apply: REPLACE content with emoji-led lines was written to the file with stray `* ` markers injected.

The normalization now walks the markdown line by line with code-fence tracking (``` and ~~~ fences) and leaves fenced content verbatim; lines outside fences are normalized exactly as before.

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files  ±0      1 suites  ±0   2h 25m 46s ⏱️ + 2m 20s
4 202 tests +8  4 188 ✅ +8  14 💤 ±0  0 ❌ ±0 
4 221 runs  +8  4 207 ✅ +8  14 💤 ±0  0 ❌ ±0 

Results for commit 8341f99. ± Comparison against earlier commit 52318dd.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   18m 42s ⏱️ + 2m 13s
2 174 tests +1  2 174 ✅ +1  0 💤 ±0  0 ❌ ±0 
2 254 runs  +1  2 254 ✅ +1  0 💤 ±0  0 ❌ ±0 

Results for commit 8341f99. ± Comparison against earlier commit 52318dd.

[Copilot]

Pull request overview

This PR fixes a markdown rendering edge case where markdownToHtml normalized “decorative bullet” lines (emoji/star-led) inside fenced code blocks, which could corrupt search/replace patch text extracted from rendered HTML and cause patch application to fail or loop indefinitely.

Changes:

• Reworked decorative-bullet normalization to be line-based with fenced-code-block tracking, leaving fenced content unchanged.
• Updated the decorative bullet regex to operate on single lines (start-of-line) to support the new normalization approach.
• Added unit tests to confirm normalization still produces list items outside fences and does not mutate fenced code content.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
packages/runtime-common/marked-sync.ts	Adds fence-aware normalization so decorative bullets are only prefixed outside fenced code blocks.
packages/host/tests/unit/marked-sync-test.ts	Adds coverage for decorative bullet normalization behavior both outside and inside fenced code blocks.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52318dda79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Handle fenced code blocks nested in list items

When Markdown nests a fenced block directly in a list item, such as - ```gts followed by 🚧 ..., Marked treats it as fenced code, but this pattern only recognizes fences preceded by whitespace. The code line is therefore rewritten to * 🚧 ..., and the closing fence is mistaken for a new opener, which both corrupts the fenced content and disables decorative-bullet normalization for the remainder of the message.

Useful? React with 👍 / 👎.

[jurgenwerk]

[Claude Code 🤖] Fixed in 8341f99: openers now also match with a leading list marker (- ```gts, 1. ```gts), while closers stay whitespace-only, so the closing fence can no longer be mistaken for an opener. Covered by the new list-nested-fence test, which also asserts normalization still applies after the fence closes.

[habdelra]

[Claude Code 🤖] I reviewed this as a fence-detection correctness problem rather than a markdown-rendering one, because that's what the failure mode demands: the rewrite is cosmetic when it's wrong about prose and destructive when it's wrong about code, so the only question that matters is whether the tracker's idea of "inside a fence" can ever disagree with marked's. I ran every shape below against marked 12.0.2 (inside the workspace catalog's ^12.0.1) with the same code renderer this module registers, comparing normalized output and rendered HTML on main and on this branch.

Bottom line: the diagnosis is right and the approach is the right one, but the fence tracker has two holes that still let a corrupted patch through — one of which reproduces the original bug byte-for-byte. Both are fixable with a single edit to CODE_FENCE_PATTERN; I verified the replacement.

What lands right. Moving from a global regex to line-wise scanning with fence state is the correct shape — the rewrite has to happen before block lexing (turning a line into a list item is a block-level decision), so a pre-pass is unavoidable and it has to carry its own fence model. The closer test is more careful than it looks: the marker.length >= fenceLength clause is exactly what keeps a model patching a .md file from having its inner ``` close the outer ```` block, and the empty-info-string clause matches CommonMark's rule so a content line like ``` trailing doesn't end the block early. I verified both against marked and left a confirmation thread so a later simplification pass doesn't collapse them. Streaming also improves: a message cut mid-fence now protects everything after the opener, where main corrupted it.

On the Codex bot's finding — confirmed, and it understates the consequence. It's right that a fence opened on a list-marker line (- ```gts) is invisible to ^(\s*) and its content gets rewritten. What it missed is what the closing fence then does: it matches, inFence is still false, so it registers as an opener and the tracker is inverted against marked from there on. While inverted, the next real fence opener with no info string satisfies the closer test and switches normalization back on inside a genuine code block. I reproduced a message where that puts * into a SEARCH pattern in the rendered <pre>, identical to main's output. Detail and repro in the thread on normalizeDecorativeBullets.

Recommendations, most severe first:

1. Blocking — allow a leading list marker on the fence pattern so opener and closer pair up; without it a list-nested patch still corrupts and the tracker's parity can invert. See the thread on normalizeDecorativeBullets (lines 91–106) for the repro and the verified replacement pattern.
2. Blocking-adjacent, one character — the $ anchor makes the guard a total no-op on CRLF input, because JS treats \r as a line terminator that . won't match and $ won't skip. See the thread on CODE_FENCE_PATTERN. Bot messages are LF, so the patch path is safe today; markdown files rendered through the same helper are not.
3. Add a round-trip test — render a real SEARCH/REPLACE block and assert extractCodeData returns it unchanged. Both findings above pass the current assertions. Thread on the new test in marked-sync-test.ts.
4. Narrow the doc comment above normalizeDecorativeBullets: only fenced blocks are protected, and 4-space indented code blocks still get a marker injected. Thread on lines 79–83.
5. Consider pinning the (improved) behavior change where a decorative bullet alone on a line is no longer promoted to a list item. Thread on DECORATIVE_BULLET_PATTERN.

If you'd rather close the whole class than keep patching the pattern, the durable version is to let marked locate the code: lex once over markdown.replace(/\r\n|\r/g, '\n') — matching what Lexer.lex does, so raw lengths map to source offsets — collect the ranges of code tokens, and rewrite only lines outside them. That is a bigger change than this PR needs; the pattern edit is a reasonable place to stop, and the round-trip test is what makes stopping there safe.

Adjacent, out of scope. escapeHtmlOutsideCodeBlocks in packages/runtime-common/helpers/html.ts runs immediately before this normalization on bot messages and carries a third, much cruder notion of "code block" — /`[\s\S]*?`/g, i.e. any backtick to the next backtick, with no fence awareness at all. It happens not to interact badly with this change (it only rewrites < and >), but that's now three independent models of where code starts and ends in one pipeline. Worth consolidating for whoever touches it next.

---

Generated by Claude Code

[habdelra]

[Claude Code 🤖] The $ anchor makes this guard a complete no-op on CRLF input — every fence line fails to match, inFence never becomes true, and the corruption this PR fixes comes back verbatim.

Mechanism. JS classifies \r (U+000D) as a line terminator, so . never matches it, and $ without the m flag anchors only to true end-of-string (JS has no Perl-style "also before a final newline"). markdown.split('\n') leaves the \r sitting at the end of every line, so on ```gts\r the (.*) group stops at gts and $ then fails with \r unconsumed. No backtracking rescues it.

Verified.

> /^(\s*)(`{3,}|~{3,})(.*)$/.test('```gts\r')   // false
> /^(\s*)(`{3,}|~{3,})(.*)$/.test('```gts')     // true

End to end, with ['```gts','file.gts','🚧 SITE UNDER CONSTRUCTION 🚧','```'].join('\r\n'):

<pre data-code-language="gts">file.gts
* 🚧 SITE UNDER CONSTRUCTION 🚧</pre>

versus the same content joined with \n:

<pre data-code-language="gts">file.gts
🚧 SITE UNDER CONSTRUCTION 🚧</pre>

Why it's silent. Lexer.lex in marked does src.replace(/\r\n|\r/g, '\n') before tokenizing, so marked parses CRLF markdown perfectly well — only this pre-pass is blind to it. Nothing downstream complains; the marker just lands in the <pre>.

The fix — tolerate the CR, and drop the .-vs-line-terminator dependency while you're in here:

const CODE_FENCE_PATTERN = /^(\s*)(`{3,}|~{3,})([^\r\n]*)\r?$/;

(See the thread on normalizeDecorativeBullets below — one pattern edit can carry this and the list-marker case together.)

Scope: regression against the PR's own contract, non-blocking on the reported path but a one-character fix. Bot messages arriving over Matrix are LF in practice, so the patch-corruption path is unlikely to hit this. markdownToHtml is also reached from packages/base/markdown-file-def.gts and packages/base/default-templates/markdown.gts with file contents, where a Windows-authored .md is entirely ordinary — and there the guard is simply off.

---

Generated by Claude Code

[jurgenwerk]

[Claude Code 🤖] Fixed in 8341f99: the fence patterns now end in (.*)\r?$ — the greedy group still stops before the CR (. cannot match it) and the explicit \r? lets $ anchor. Added a CRLF round-trip test that renders and re-extracts a patch from \r\n input.

[habdelra]

[Claude Code 🤖] inFence can invert relative to marked's own parse, and once inverted a genuine fenced patch gets normalized again — the exact corruption this PR exists to stop reproduces on this branch. The Codex bot flagged the first step of this; I verified it and it goes one step further than the bot described.

The mechanism, in three moves.

1. CODE_FENCE_PATTERN anchors at ^(\s*), so a fence opened on the same line as a list marker — - ```gts or 1. ```gts — never matches. inFence stays false and the fenced content is rewritten. marked does treat that as a fenced code block, so parseHtmlContent hands it to extractCodeData as a patch.
2. The closing fence of that same block (``` indented to the list content column) does match ^(\s*), and because inFence is still false it is read as an opener. The tracker's state is now inverted against marked's.
3. While inverted, the next real fence opener carrying no info string (a bare ```) satisfies the closer test — same char, length ≥, empty info — and flips inFence back to false. Everything inside that code block is now normalized.

Verified. Move 1 alone, ['Steps:','','- ```gts',' 🚧 SITE UNDER CONSTRUCTION 🚧',' ```','','🌟 after'].join('\n') renders:

<ul>
<li><pre data-code-language="gts">* 🚧 SITE UNDER CONSTRUCTION 🚧</pre></li>
</ul>
<p>🌟 after</p>

The marker is inside the <pre> (patch corrupted), and 🌟 after is no longer a list item (move 2's side effect — normalization is off for the rest of the message).

Moves 1–3 together, with a bare-fence patch block after a list-nested one:

- ```gts
  const x = 1;

file.gts
🚧 SEARCH LINE

renders the patch block as:

```html
<pre data-code-language="">file.gts
* 🚧 SEARCH LINE</pre>

Byte-identical to what main produces. extractCodeData reads preElement.innerHTML verbatim into parseSearchReplace, so that * goes straight into the search pattern.

How much of this is new. The nested-fence content corruption (move 1) is pre-existing — main corrupts it too — so this is a hole in the fix rather than a fresh bug. What is new is move 2's side effect: after a phantom opener, decorative bullets silently stop rendering as lists for the remainder of the message.

The fix. Let the opener and its closer be seen as a pair by allowing a leading list marker; combined with the CR fix from the CODE_FENCE_PATTERN thread:

const CODE_FENCE_PATTERN =
  /^(\s*(?:(?:[-*+]|\d{1,9}[.)])\s+)?)(`{3,}|~{3,})([^\r\n]*)\r?$/;

I ran that against eight shapes: it turns all four failing ones green (list-nested fence, numbered-list-nested fence, the parity inversion above, CRLF) and leaves both new tests plus the nested-4-backtick and trailing-info-string cases byte-identical.

Residual, and the durable alternative. Hand-rolled fence detection will keep leaking: a line beginning with an inline triple-backtick span (```code``` is inline) still registers as a fence, CommonMark's 3-space indentation ceiling for fences is unmodelled, and so is "backtick fences may not carry a backtick in the info string". None of those is likely from a model writing patches, so the pattern edit is a reasonable place to stop. If you'd rather close the class, let marked decide instead of re-deriving it: lex once with bfmMarked.lexer(markdown.replace(/\r\n|\r/g, '\n')) — that's the same CR normalization Lexer.lex performs, so token raw lengths line up with source offsets — walk the block tokens recording the ranges of code tokens, and rewrite only lines outside them. Costs one extra lex pass and can never disagree with the parser that renders the result.

Scope: blocking. The current shape can still emit a corrupted patch, which is the failure mode this PR is meant to eliminate.

---

Generated by Claude Code

[jurgenwerk]

[Claude Code 🤖] Fixed in 8341f99. Openers and closers are now matched separately: openers additionally accept a leading list marker, closers require a same-char, at-least-as-long marker with nothing after it. The inversion case is covered by the new test asserting that content after a list-nested fence still normalizes.

[habdelra]

[Claude Code 🤖] Confirmation — this three-part closer test is correct and each clause is load-bearing. Recording why, because all three read like defensive noise and none of them is.

• marker[0] === fenceChar stops a ~~~ line from closing a ``` block. CommonMark requires the closing fence to use the same character.
• marker.length >= fenceLength is what makes nested fences work, and this is the case that matters most here: when the model patches a markdown file, it wraps the SEARCH/REPLACE block in ```` and the file content itself contains ```. Without the length comparison the inner fence would close the outer block and every subsequent line would be treated as prose.
• fenceMatch[3].trim() === '' matches CommonMark's rule that a closing fence carries no info string. Without it, a content line like ``` trailing would end the block early.

Verified against marked 12.0.2 (inside the workspace catalog's ^12.0.1). The nested case — outer ````md, inner ```js around an emoji-led line — comes back with the inner fences intact and no marker injected:

<pre data-code-language="md">file.md
&lt;&lt;&lt;&lt;&lt;&lt;&lt; SEARCH
```js
🚧 x

...

And `` ``` trailing `` stays inside the block in both this code's tracking and marked's own parse, so the two agree.

Scope: confirmation, nothing to change. Worth keeping as-is if a later pass is tempted to collapse this into a plain `marker === fence` equality.

---
_Generated by [Claude Code](https://claude.ai/code)_

[habdelra]

[Claude Code 🤖] Unflagged behavior change: a decorative bullet alone on its own line is no longer turned into a list item.

Mechanism. The old pattern was (^|\n)(\s*)(bullet)(\s+) run globally over the whole string, so \s+ could match the line's own trailing newline — a bare 🌟 line satisfied it. Now the subject is a single line with the newline already stripped by split('\n'), so (\s+) has nothing to match and the line falls through untouched.

Verified, '🌟\n\ntext':

	rendered
before	<ul>\n<li>🌟</li>\n</ul>\n<p>text</p>
after	<p>🌟</p>\n<p>text</p>

The difference is bigger when a bare bullet leads into a real one — 'intro\n\n🌟\n🌟 second' used to collapse both lines into a single <li> via lazy continuation (<li>🌟\n🌟 second</li>), and now produces a paragraph followed by a one-item list.

Verdict: the new behavior is better — a lone emoji with no text after it isn't a bullet, and the lazy-continuation merge was surprising. Flagging it only because it looks incidental rather than intended, and nothing pins it: the next edit to this regex could restore the old behavior without a test noticing. Worth an assertion alongside the two new tests.

Scope: non-blocking; a test, not a code change.

---

Generated by Claude Code

[jurgenwerk]

[Claude Code 🤖] Restored in 8341f99: the bullet pattern now accepts end-of-line after the bullet ((\s+|\r?$)), so a decorative bullet alone on its line becomes a list item again. Test added.

[habdelra]

[Claude Code 🤖] The comment promises a broader invariant than the code delivers. "Code-block content must survive rendering verbatim" reads as covering all code blocks; only fenced ones are protected. A 4-space indented code block is still rewritten.

Verified, 'Example:\n\n 🚧 SITE UNDER CONSTRUCTION 🚧\n\n🌟 tail':

<p>Example:</p>
<pre data-code-language="">* 🚧 SITE UNDER CONSTRUCTION 🚧</pre>

That matters because the extraction side doesn't distinguish the two: parseHtmlContent in packages/host/app/lib/formatted-message/utils.ts walks top-level children and routes every <pre> through extractCodeData, which reads innerHTML straight into parseSearchReplace. An indented code block is a <pre> like any other, so a patch expressed that way corrupts exactly as before.

How much this matters: little, in practice. The source-code-editing skill teaches fenced blocks with a language on every example, and extractCodeData keys the file URL off the first line, so an indented patch is off the taught path. Not asking for it in this PR.

The ask is the wording — so the next reader doesn't build on an invariant that isn't there. Something like:

// ...but never inside fenced code blocks. Fenced content must survive
// rendering verbatim: search/replace patches are extracted back out of the
// rendered HTML, and an inserted marker makes the patch text no longer match
// the file it targets. Indented (4-space) code blocks are not tracked; the
// patch format is always fenced.

Scope: follow-up for the indented-block gap, comment accuracy for this PR. Non-blocking.

---

Generated by Claude Code

[jurgenwerk]

[Claude Code 🤖] Fixed in 8341f99 by making the code deliver what the comment promised: lines with 4-space (or tab) indentation outside a fence are now left verbatim, since extraction treats an indented code block's

 exactly like a fenced one. The asymmetry you describe (cosmetic when skipped wrongly, destructive when applied wrongly) is now recorded next to the pattern.

[habdelra]

[Claude Code 🤖] This test asserts the shape of the rendered HTML, one step away from the contract that actually broke — and both blocking findings in this review pass it.

The contract is a round trip: a patch the model puts inside a fence has to come back out of extractCodeData byte-identical. Asserting substrings of markdownToHtml's output can hold while the round trip fails, and here it does — neither the CRLF case nor the list-nested-fence case is visible to result.includes(' 🚧 …') / assert.false(result.includes('* 🚧')), because both change the input shape, not the assertions' subject.

The test that would have caught them. Render a real SEARCH/REPLACE block through markdownToHtml, push the result through parseHtmlContent (packages/host/app/lib/formatted-message/utils.ts), and assert codeData.searchReplaceBlock equals the source text — then parameterize it over the shapes: LF and CRLF, fence at top level and nested in a list item, bare ``` and ```gts openers. Use the real markers from @cardstack/runtime-common (SEARCH_MARKER / SEPARATOR_MARKER / REPLACE_MARKER) rather than emoji-only prose, so the test reads as the scenario it defends.

packages/host/tests/integration/components/formatted-aibot-message-test.gts already has the rendering harness for this, and packages/host/tests/unit/code-patching-test.ts covers parseSearchReplace in isolation. Nothing covers the seam between them, which is precisely where this bug lives — a renderer-side rewrite that only manifests as a parser-side mismatch.

On the first new test (prefixes decorative bullets with a list marker): the same behavior is already asserted at the integration level in packages/host/tests/integration/components/ai-assistant-panel/general-test.gts ("it converts decorative star bullets into markdown lists" and the multi-bullet variant). Keeping a unit-level pin is fine and cheaper to run — just noting it isn't new ground, so the round-trip test is where the marginal coverage is.

Scope: test coverage. Non-blocking on its own, but it is what turns the two blocking threads into something a future edit can't silently undo.

---

Generated by Claude Code

[jurgenwerk]

[Claude Code 🤖] Added in 8341f99: a round-trip test now renders a fenced search/replace block through the same pipeline as message bodyHTML and re-extracts it with parseHtmlContent, asserting byte-identity with what was authored — plus a CRLF variant. Both blocking findings fail this test on the previous commit.