feat: add regex perf benchmarking command - #641
Open
fzipi wants to merge 19 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the preflightAssembly stub with a real implementation that detects `include`/`include-except` directives and returns an actionable error when the crsRoot has no regex-assembly/ directory, preventing the crs-toolchain assembler from calling logger.Fatal (os.Exit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Params struct, Run orchestrator, resolveRegex, runCorpus, newCorpus, and timeMatch helpers to internal/regexperf; all 19 package tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 29 minutes and 13 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds a new ftw ChangesRegex Performance Benchmarking Feature
Sequence Diagram(s) sequenceDiagram
participant CLI as CLI
participant PerfCmd as cmd/regex/perf.runPerfE
participant Assembler as crs-toolchain
participant Compiler as regexp.Compile
participant Corpus as CorpusIterator
participant Stats as regexperf.Stats
participant Output as output.Output
CLI->>PerfCmd: invoke ftw regex perf (--file/--pattern, --subject/--corpus)
PerfCmd->>Assembler: Assemble .ra (if --file)
PerfCmd->>Compiler: Compile regex (assembled or raw)
PerfCmd->>Corpus: create iterator (corpus or raw)
Corpus->>PerfCmd: provide subject strings
PerfCmd->>Compiler: Match subject N times (timeMatch)
PerfCmd->>Stats: Add(subject, minNs, matched)
Stats->>Output: printSummary (json/plain)
PerfCmd->>Output: write to stdout or --out-file
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
CLAUDE.md (1)
84-84: ⚡ Quick winAdd languages to fenced code blocks to satisfy markdown linting.
These fences are unlabeled and trigger MD040; add
text(or another appropriate language) after the opening backticks.Suggested patch
-``` +```text CLI Layer (cmd/) ↓ orchestrates Runner Layer (runner/) @@ -``` +```text YAML Test File ↓ Unmarshal (test package) schema.Test with stagesAlso applies to: 140-140
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@CLAUDE.md` at line 84, The unlabeled fenced code blocks in CLAUDE.md are triggering MD040; update each opening triple-backtick to include a language (e.g., change ``` to ```text) for the code examples shown (the block containing "CLI Layer (cmd/)"→"Runner Layer (runner/)" and the block containing "YAML Test File"→"schema.Test with stages") so both fences are labeled (use `text` or another appropriate language).Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CLAUDE.md`:
- Line 100: Update the documented CLI subcommand list to include the new
regex-related commands by adding `regex` (and `regex perf` if applicable)
alongside the existing `run`, `check`, `quantitative`, and `self_update` entries
so the README reflects the PR changes and contributor guidance stays current;
reference the new subcommands `regex` and `regex perf` in the same list format
where the CLI subcommands are enumerated.
- Line 254: Update the platform name to the official capitalization "GitHub" in
the sentence containing "CI runs tests on Ubuntu and Windows" in CLAUDE.md;
locate the sentence "CI runs tests on Ubuntu and Windows (see
`.github/workflows/test.yml`)" and replace any occurrence of "Github" or
"github" with "GitHub" so the docs use the correct branding.
In `@cmd/regex/perf_test.go`:
- Line 4: The test file declares the wrong package; change the package
declaration in perf_test.go from "package cmd" to "package regex" so the tests
live in and can access the regex package (ensure any package-level test
utilities or imports still compile after changing the package name, e.g.,
references in perf_test.go to functions/types from regex).
In `@cmd/regex/perf.go`:
- Line 4: The file cmd/regex/perf.go currently declares the wrong package;
change its package declaration from "cmd" to "regex" so the file belongs to the
regex package and can be imported by cmd/root.go; update the top-level package
line in perf.go (the file containing perf-related code) to "package regex".
In `@cmd/regex/regex.go`:
- Line 4: The package declaration at the top of this file is incorrect — change
the package line from `package cmd` to `package regex` so it matches the import
alias and usage elsewhere (see import `regex
"github.com/coreruleset/go-ftw/v2/cmd/regex"` and the call to
`regex.New(cmdContext)` in root.go); update only the package name in this file
to `regex` to resolve the compilation error.
In `@docs/superpowers/plans/2026-06-08-regex-perf.md`:
- Line 13: The heading "Resolved spec open-items (verified against crs-toolchain
`main`)" is using H3 (###) and creates a level jump; change that heading to the
correct level (e.g., ##) to continue the H1/H2 flow so it satisfies
MD001/markdown-lint and remove the level jump introduced by the "### Resolved
spec open-items (verified against crs-toolchain `main`)" line.
In `@docs/superpowers/specs/2026-06-08-regex-perf-design.md`:
- Around line 65-69: The doc currently uses two different flag names
(`--file-out` in the table and `--out-file` elsewhere), causing confusion;
update all occurrences to use the single canonical flag `--out-file` (the same
long flag used by the implemented command and referenced in the `quantitative`
command) — change the table entry label, the note text that mentions
`--file-out`, and any other mentions in the spec so every reference consistently
uses `--out-file`.
- Line 39: Several fenced code blocks in the spec file are unlabeled (triggering
MD040); locate the unlabeled triple-backtick fences (examples around the blocks
referenced) and add appropriate language identifiers (e.g., text, bash, go) to
each opening fence so markdownlint stops flagging them—ensure you update the
fences at the mentioned locations (around lines 39, 73, 81, 134, 165) by
replacing ``` with ```text, ```bash, or ```go as appropriate for the snippet
contents.
---
Nitpick comments:
In `@CLAUDE.md`:
- Line 84: The unlabeled fenced code blocks in CLAUDE.md are triggering MD040;
update each opening triple-backtick to include a language (e.g., change ``` to
```text) for the code examples shown (the block containing "CLI Layer
(cmd/)"→"Runner Layer (runner/)" and the block containing "YAML Test
File"→"schema.Test with stages") so both fences are labeled (use `text` or
another appropriate language).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 2af42fde-ddf2-48db-ad8f-a938229d07fe
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (16)
CLAUDE.mdREADME.mdcmd/regex/perf.gocmd/regex/perf_test.gocmd/regex/regex.gocmd/regex_wiring_test.gocmd/root.godocs/superpowers/plans/2026-06-08-regex-perf.mddocs/superpowers/specs/2026-06-08-regex-perf-design.mdgo.modinternal/regexperf/benchmark.gointernal/regexperf/benchmark_test.gointernal/regexperf/compile.gointernal/regexperf/compile_test.gointernal/regexperf/stats.gointernal/regexperf/stats_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
ftw regex perfcommand that benchmarks the runtime performance of a regular expression against a set of input subjects..ra) file via the crs-toolchain assembler, or takes a raw--patterndirectly.--corpusleipzig/raw and related flags) or a single inline--subject.regexpengine (RE2), keeping the minimum of--repeatmatches, and reports total, mean, median, p99, max, throughput, and the top-N slowest subjects innormalorjsonoutput.Because it uses RE2 (linear time), it measures Coraza-realistic performance and is not a ReDoS detector; this is stated in the command help and README.
Implementation
internal/regexperf:compile.go(assembler integration + Go compile),stats.go(aggregation, percentiles, bounded top-N heap, normal/JSON output),benchmark.go(orchestration and min-of-K timing).cmd/regexpackage with theregexparent group andperfsubcommand, registered on the root command.internal/corpusloaders by calling the leipzig/raw constructors directly, so the Coraza engine is not pulled into this package.github.com/coreruleset/crs-toolchain/v2 v2.9.0.go mod verifypasses;govulncheckreports no new vulnerabilities introduced by this change.Notes
os.Exitwhen anincludefragment cannot be opened. A preflight step validates that each top-level include referenced by the.rafile exists under<crs-path>/regex-assembly/include(or/exclude) and returns an actionable error instead. Includes referenced transitively by other fragments are not validated and remain a documented limitation.os.Exiton a missing file.Test Plan
go test -race ./...passesgo build ./...andgo vet ./...cleanftw regex perf --pattern '(?i)union\s+select' --subject "' UNION SELECT 1,2,3" -o jsonreturns a report withsubjectCount:1, matchCount:1ftw regex perf --file <some>.ra -C <coreruleset> -s 10Kcompiles and benchmarks against the corpus🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests
Chores