Skip to content

feat: implement scoped prefixes/suffixes - #282

Open
fzipi wants to merge 13 commits into
mainfrom
feat/add-scoped-prefixes-suffixes
Open

feat: implement scoped prefixes/suffixes#282
fzipi wants to merge 13 commits into
mainfrom
feat/add-scoped-prefixes-suffixes

Conversation

@fzipi

@fzipi fzipi commented Feb 8, 2026

Copy link
Copy Markdown
Member

what

  • implement scoped prefixes/suffixes

why

  • simplify regex generation

refs

Summary by CodeRabbit

  • Bug Fixes

    • Prefix and suffix directives now apply only within their respective assemble blocks, including nested and multiple blocks.
    • Included content preserves directive order and scope without unintended global application.
    • Stored expressions inherit active block directives without duplicate application.
  • Tests

    • Added coverage for block scoping, nested blocks, stored expressions, and included-file scenarios.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements block-scoped handling of ##!^ (prefix) and ##!$ (suffix) directives so prefixes/suffixes are applied by each Assemble processor instance rather than globally, simplifying regex assembly behavior.

Changes:

  • Added prefix/suffix directive handling to the Assemble processor and applied them during Complete().
  • Updated the parser to pass prefix/suffix directives through as raw lines (instead of collecting them on the parser).
  • Removed operator-level prefix/suffix application and updated/added tests for the new scoping behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
regex/processors/assemble.go Adds block-scoped prefix/suffix collection and applies them on completion of an assemble block.
regex/parser/parser.go Changes prefix/suffix parsing to pass directives through verbatim rather than populating Parser.Prefixes/Suffixes.
regex/parser/include_test.go Updates include parsing expectations to reflect pass-through directive behavior.
regex/operators/assembler.go Removes global prefix/suffix application and leaves a note about new behavior.
regex/operators/assembler_test.go Adds tests validating block-scoped prefix/suffix behavior across top-level and multiple blocks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread regex/processors/assemble.go
Comment thread regex/parser/parser.go
Comment thread regex/operators/assembler.go Outdated
Comment thread regex/operators/assembler_test.go
@fzipi

fzipi commented Feb 8, 2026

Copy link
Copy Markdown
Member Author

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot AI commented Feb 8, 2026

Copy link
Copy Markdown
Contributor

@fzipi I've opened a new pull request, #283, to work on those changes. Once the pull request is ready, I'll request review from you.

Comment thread regex/operators/assembler_test.go
Comment thread regex/processors/assemble.go Outdated
Comment thread regex/processors/assemble.go Outdated
@fzipi
fzipi force-pushed the feat/add-scoped-prefixes-suffixes branch from f0a39e2 to afb8fd0 Compare April 25, 2026 10:07
@fzipi
fzipi requested a review from Copilot April 25, 2026 10:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread regex/parser/parser.go Outdated
Comment thread regex/parser/parser.go Outdated
@coderabbitai

coderabbitai Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0deec3e9-280f-4214-ad53-8e90d0415ed0

📥 Commits

Reviewing files that changed from the base of the PR and between eb3944a and cb88d62.

📒 Files selected for processing (4)
  • regex/operators/assembler.go
  • regex/operators/assembler_test.go
  • regex/parser/include_test.go
  • regex/parser/parser.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • regex/operators/assembler.go

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Parser now emits prefix/suffix directive lines inline. Assemble applies them per block, including to stored expressions and included content. Operator-level wrapping is removed. Tests cover scoping, nesting, inclusion, and stashing.

Changes

Block-scoped prefix/suffix directive handling

Layer / File(s) Summary
Parser directive pass-through and include scoping
regex/parser/parser.go, regex/parser/include_test.go
Parser output preserves prefix/suffix directives. Included content remains scoped within an assemble block. Parser tests expect raw directives and separators.
Assemble processor block-scoped prefix/suffix support
regex/processors/assemble.go
Assemble captures directives, applies them to completed output, and applies and clears them when storing expressions.
Operator simplification and block-scoped test coverage
regex/operators/assembler.go, regex/operators/assembler_test.go
Operator.complete removes parser-level wrapping. Tests cover top-level, nested, multi-block, included-file, and stashed-expression behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to cb88d

The scoped prefix and suffix changes are merge-ready after normal checks and review; no actionable merge-blocking risk remains.

Suggested reviewers: theseion

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: implementing scoped prefixes and suffixes.
Description check ✅ Passed The description includes what, why, and reference information, although it uses "refs" instead of the template heading "references".
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@regex/parser/parser.go`:
- Around line 297-301: The code currently returns out verbatim which leaks
included file prefix/suffix directives (##!^ / ##!$) into the caller's top-level
assemble scope; change mergePrefixesSuffixes (and the include handling path that
currently returns out) to detect if the included content has Parser.Prefixes or
Parser.Suffixes and, when present, inject an explicit local assemble boundary
(or equivalent scope delimiter) before and after the included content so those
directives apply only to the included block; update the return path so it
returns the wrapped/merged content (not raw out) and keep existing
Prefixes/Suffixes collection for compatibility, ensuring assemble.go's
store()/Complete() behavior no longer extends the caller scope.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6f17f55f-9ba7-4827-a08d-bbffb1897063

📥 Commits

Reviewing files that changed from the base of the PR and between 2a50d8a and 1404ad4.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (5)
  • regex/operators/assembler.go
  • regex/operators/assembler_test.go
  • regex/parser/include_test.go
  • regex/parser/parser.go
  • regex/processors/assemble.go

Comment thread regex/parser/parser.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
regex/parser/parser.go (1)

279-285: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Re-scope included prefix/suffix directives before returning the parsed buffer.

Line 285 still returns the included output verbatim, and validateIncludedFileDirectives only rejects flags. That means ##!^ / ##!$ lines from the child file stay in the caller's current assemble block; regex/processors/assemble.go keeps them active until store() or Complete(), so caller lines after the include get wrapped too. Please restore a local assemble boundary here, or an equivalent scoping wrapper, whenever the included output contains prefix/suffix directives.

Also applies to: 288-295

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@regex/parser/parser.go` around lines 279 - 285, The included buffer is
returned verbatim so any child-file prefix/suffix assemble directives (e.g.,
##!^ / ##!$) leak into the caller; modify the return path in the Parse/validate
block to detect those directives on newP's output and wrap the parsed buffer in
a local assemble-scoping boundary before returning. Concretely: after calling
newP.Parse(false) and validateIncludedFileDirectives(newP), check newP (or
out.String()) for prefix/suffix directive markers and, if present, create a
short-lived assemble scope (e.g., call a helper like
reScopeIncludedAssemble(newP) or newP.WrapWithLocalAssembleBoundary()) that
inserts the equivalent begin/end assemble tokens around out so the caller's
assembler (assemble.go store()/Complete()) does not inherit the child's active
prefix/suffix directives; then return that wrapped buffer and newP.variables.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@regex/parser/parser.go`:
- Around line 279-285: The included buffer is returned verbatim so any
child-file prefix/suffix assemble directives (e.g., ##!^ / ##!$) leak into the
caller; modify the return path in the Parse/validate block to detect those
directives on newP's output and wrap the parsed buffer in a local
assemble-scoping boundary before returning. Concretely: after calling
newP.Parse(false) and validateIncludedFileDirectives(newP), check newP (or
out.String()) for prefix/suffix directive markers and, if present, create a
short-lived assemble scope (e.g., call a helper like
reScopeIncludedAssemble(newP) or newP.WrapWithLocalAssembleBoundary()) that
inserts the equivalent begin/end assemble tokens around out so the caller's
assembler (assemble.go store()/Complete()) does not inherit the child's active
prefix/suffix directives; then return that wrapped buffer and newP.variables.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e8d9ac11-6d71-469c-b83e-86678124dc1c

📥 Commits

Reviewing files that changed from the base of the PR and between 1404ad4 and 9a7d8e9.

📒 Files selected for processing (1)
  • regex/parser/parser.go

fzipi and others added 9 commits June 7, 2026 14:22
Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
…er handling

Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
Address review feedback from theseion:
- Remove redundant else-if branch in Complete() since the outer guard
  already ensures at least one of prefixes/suffixes is non-empty
- Add test for multiple nesting levels to verify block-scoped
  prefix/suffix isolation across nested assemble blocks

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@fzipi
fzipi force-pushed the feat/add-scoped-prefixes-suffixes branch from 9a7d8e9 to eb3944a Compare June 7, 2026 17:22
@fzipi
fzipi requested a review from theseion August 4, 2026 22:51
@fzipi
fzipi requested a lite review from Copilot August 6, 2026 15:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new prefix/suffix handling in Assemble can emit incorrect output for directive-only/stash-only blocks and currently clears directive state on stash, which can silently break later content/stashes in the same block.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

regex/processors/assemble.go:140

  • store() clears a.prefixes/a.suffixes after stashing. That mutates the directive scope for the rest of the assemble block, so any subsequent ##!=< stores (or remaining emitted output in the same block) will silently lose the active prefix/suffix. Keeping the directives in-scope avoids surprising behavior; the Complete() guard can prevent stash-only blocks from emitting stray prefix/suffix.
		// Clear prefixes/suffixes after applying to stored expression
		// so they won't be applied again in Complete()
		a.prefixes = nil
		a.suffixes = nil
	}

regex/processors/assemble.go:95

  • Complete() applies prefixes/suffixes even when result is empty (e.g., an assemble block that only contains directives, or a block that only stashes content). That can incorrectly emit prefix+suffix as output. Guard prefix/suffix application on result != "" so directive-only/stash-only blocks stay silent.
	// Apply block-scoped prefixes and suffixes
	if len(a.prefixes) > 0 || len(a.suffixes) > 0 {
		result = strings.Join(a.prefixes, "") + result + strings.Join(a.suffixes, "")
		logger.Trace().Msgf("Applied block-scoped prefixes/suffixes: %s", result)
	}

regex/operators/assembler.go:135

  • This comment claims Parser.Prefixes/Parser.Suffixes are still populated, but the parser now passes prefix/suffix directives through as raw lines and no longer appends to those fields. Updating this avoids leaving misleading documentation in the operator path.
	// Note: Prefix/suffix application is now handled by individual Assemble processors (block-scoped).
	// While Parser.Prefixes and Parser.Suffixes are still populated by the parser for include file
	// merging compatibility, they are not used in the operator path as prefix/suffix directives are
	// passed through as raw lines and processed by each Assemble instance.
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Passing `##!^` and `##!$` through as raw lines left them active for the rest
of the including file. The Assemble processor holds a directive until store()
or Complete(), so an include carrying a prefix wrapped the caller's own
content as well, regardless of where the include appeared:

    include/inc.ra:  ##!^ PRE_
                     alpha
    932100.ra:       ##!> include inc
                     beta

    before: PRE_(?:alph|bet)a
    after:  (?:PRE_alph|bet)a

Wrap included content in its own assemble block when the included file
contains prefix or suffix directives, which bounds them to that content. As
before, the block is omitted when there are no such directives, because
wrapping changes the semantics of content processed by the cmdline processor
in the including file.

This restores the local scope that mergePrefixesSuffixes used to provide.
Parser.Prefixes and Parser.Suffixes are populated again, now only so
parseFile can tell whether an included file needs the block, and the
assembler comment describing them is corrected.

The three include tests asserted the leaking output, so they could not catch
this. Restore their expectations and add end-to-end tests covering an include
before, after, and on both sides of the caller's own content.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fzipi

fzipi commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

Pushed cb88d62, which fixes the include scoping problem @coderabbitai raised in this comment. The finding was correct and had not been addressed — the 2026-06-07 commits renamed the helper but left the behaviour unchanged.

The regression

Prefix and suffix directives from an included file were passed through as raw lines. Assemble keeps a directive active until store() or Complete(), so an include carrying ##!^ wrapped the including file's content too:

include/inc.ra:  ##!^ PRE_
                 alpha

932100.ra:       ##!> include inc
                 beta

Built both branches and compared against main:

caller layout main this PR before cb88d62
line after include (?:PRE_alph|bet)a PRE_(?:alph|bet)a
line before include (?:bet|PRE_alph)a PRE_(?:bet|alph)a
lines both sides (?:gamm|PRE_alph|bet)a PRE_(?:gamm|alph|bet)a
suffix instead of prefix alpha_SUF|beta (?:alph|bet)a_SUF

The leak was position-independent: the include's prefix applied to the entire caller block no matter where the include sat. That is the opposite of what #41 asks for, since it makes include-supplied prefixes maximally global.

Root cause

mergePrefixesSuffixes used to wrap included content in ##!> assemble / ##!<. That wrapper was the local scope boundary. Removing it in favour of pass-through dropped the boundary along with the old merging logic.

The fix

parseFile wraps included content in its own assemble block when the included file contains prefix or suffix directives. The pass-through stays, so the block-scoping mechanism this PR introduces is what applies them — the block just gives them a scope to be applied in.

The block is still omitted when there are no such directives, preserving the reason for the original IMPORTANT comment: wrapping changes the semantics of content processed by the cmdline processor in the including file.

Parser.Prefixes / Parser.Suffixes are populated again, used only for that "does this include need a block" decision. This also resolves @copilot's point that the fields were dead state while assembler.go claimed they were still populated; that comment is corrected.

Why CI stayed green

TestParserInclude_Prefixes, _Suffixes, and _FlagsPrefixesSuffixes had their expected output changed to drop the ##!> assemble / ##!< wrapper, with a comment saying directives are passed through as-is. The tests were updated to match the regression, so nothing was left to catch it.

Their expectations are restored, and TestAssemble_IncludedPrefixIsScopedToIncludedContent / _IncludedSuffixIsScopedToIncludedContent now cover this end to end, at the level the regression actually showed up. Verified they fail without the fix and pass with it.

go test ./..., go vet, and gofmt are clean.

@theseion your three review comments look addressed by the earlier commits (field rename, the redundant branch, and the multi-level nesting test). This should be ready for another look when you have time.

@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@fzipi Thanks for the detailed regression analysis and fix.

The conditional assemble block restores the required scope boundary for directives from included files. Keeping the block absent when the included file has no prefix or suffix directives preserves the existing cmdline behavior.

The restored parser expectations and the new end-to-end include-scoping tests cover the failure mode.


✏️ Learnings added
Learnt from: fzipi
URL: https://github.com/coreruleset/crs-toolchain/pull/282

Timestamp: 2026-08-17T23:01:20.544Z
Learning: In `regex/parser/parser.go`, when an included regex file contains `##!^` prefix or `##!$` suffix directives, `parseFile` must wrap the included output in its own `##!> assemble` / `##!<` block. `regex/processors/Assemble` retains directives until `store()` or `Complete()`, so omitting this block leaks included-file directives into the including file's assemble scope. Do not add this wrapper when the included file has no prefix or suffix directives, because that changes `cmdline` processor semantics.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

You are interacting with an AI system.

@fzipi
fzipi requested review from theseion and removed request for theseion August 17, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants