Judge the benchmark ratchet against a window of main runs

[leynos]

Summary

Fixes the false-positive mode recorded in #219, in the specific form seen on
#289: a main run measured medium-single-nocb at 0.760 against a baseline of
1.013, passed as a 25% improvement — improvements are not gated — and published
that outlier as the next baseline. Every pull request after it compared an
ordinary 1.110 against 0.760 and reported a 46% regression, three re-runs
included, while the other three scenarios agreed with the baseline to within
0.14.

Two independent properties produced that, and neither fix works without the
other.

One sample was the whole estimate, so its noise was the bar's noise. The
bar is now the median of the last seven main samples, held in
main-baseline-history.json inside the existing baseline artefact. The flat
30% threshold is joined by a noise band measured from those same samples, and a
candidate must exceed both: the flat threshold alone cannot tell a slow runner
from a slow change, and the observed spread alone would let a consistent
regression through. Spread comes from the median absolute deviation rather than
a standard deviation, which the outlier being tolerated would inflate in
proportion to itself. The band is capped at 1.00 — past that the benchmark is
saying it cannot measure what it gates on, and an uncapped band would disable
the ratchet silently instead.

The candidate is measured once, on whichever runner CI gave it. The window
makes the bar robust to one noisy run; it cannot make the candidate robust, and
the only recourse for an unlucky measurement was a human pressing re-run. A
flagged scenario is now measured again in the same job and fails only if it
regressed both times, so a flake has to land on the same scenario twice to
survive. The second benchmark is spent only on a run that was about to fail.
Confirmation may only turn a failure into a pass; a confirmation that could not
compare leaves the first verdict standing; and the re-measurement writes under
its own prefix so the sample recorded into the window stays the primary
measurement.

The sample was only published if its own run passed, and a run passes when
it is faster than the bar. A fast anomaly was therefore always accepted while
the ordinary measurements that would have corrected it were the ones rejected —
a bar biased towards the low tail of the noise, and sticky once there. Every
completed main run now records its sample and publishes the artefact
(!cancelled(), not success(), and not always(): an interrupted run records
nothing), and the fetch asks GitHub for completed runs rather than
successful ones, since publishing from a failing run achieves nothing while
the fetch still filters it out.

Review walkthrough

• benchmarks/ratchet_history.py — the window, the statistics, and the reasons
  for each choice.
• benchmarks/ratchet_rust_performance.py — _baseline_window prefers the
  window and falls back to the single-sample baseline; _compare_scenario
  applies max(flat, noise).
• .github/workflows/ci.yml — the recording step, the publication conditions,
  --run-status completed, and the candidate now staged before the
  comparison can fail the step.
• benchmarks/confirm_regression.py — intersects the two verdicts, and the
  three asymmetries that keep the re-measurement from being a second chance to
  fail.
• benchmarks/update_baseline_history.py — always writes an output file, so a
  run that measured nothing carries the window forward instead of publishing an
  artefact the next run reads as a fresh start.

Compatibility

• A window emptied by a benchmark_profile_version or worker_iterations
  change, absent on a first run, or lost to an expired artefact, falls back to
  the single-sample baseline this replaces. ratchet-report.json reports
  baseline_sample_count so a surprising verdict can be read against the
  evidence behind it.
• The first run after this merges has no history and uses that fallback; the
  window fills over the following merges.

Validation

• make check-fmt, make lint, make typecheck, make test,
  make markdownlint, make nixie: all green.
• actionlint .github/workflows/ci.yml: clean.
• Tests: cuprum/unittests/test_benchmark_ratchet_history.py (the incident's
  numbers as a regression test, plus Hypothesis properties — a candidate at the
  median always passes, observed noise may only widen the bar, one arbitrary
  sample cannot outvote a majority), test_benchmark_baseline_history_cli.py,
  test_benchmark_baseline_publication_contract.py (the workflow declarations
  that would silently reinstate the bias), and
  tests/features/benchmark_ratchet_noise.feature.

Notes

• Based on Skip benchmark-ratchet on non-performance pull requests (4.4.3.) #289; review that first. The two touch different regions of
  ci.yml.
• This does not address benchmark-ratchet: false-positive regressions from overhead-bound, noise-sensitive rust/python ratio #219's root finding that the scenarios are
  overhead-bound. It makes the gate honest about its own noise; it cannot
  create signal that is not there.

References

• Issue: benchmark-ratchet: false-positive regressions from overhead-bound, noise-sensitive rust/python ratio #219
• Evidence from the incident: benchmark-ratchet: false-positive regressions from overhead-bound, noise-sensitive rust/python ratio #219 (comment)
• Lody session: https://lody.ai/leynos/sessions/8f317346-7ef7-42fe-82d5-14eccf68991d

Summary by Sourcery

Make the benchmark ratchet resilient to noisy measurements by using a rolling main-branch baseline, recording every completed sample, and confirming failures before rejecting changes.

New Features:

• Judge benchmark regressions against a rolling seven-run main-branch baseline using median ratios and observed noise tolerance.
• Confirm flagged regressions with a second benchmark measurement before failing the job.
• Persist benchmark samples and their provenance in the main baseline artifact across completed main runs.

Bug Fixes:

• Prevent anomalously fast or noisy main runs from poisoning future benchmark baselines and causing persistent false-positive regressions.
• Ensure failing main runs contribute their measurements to the baseline window while interrupted runs do not publish incomplete samples.

Enhancements:

• Fall back to the existing single-sample baseline when history is unavailable or incompatible, and report the supporting sample count.
• Separate benchmark ratio extraction from ratchet policy evaluation and preserve compatibility for existing consumers.
• Use completed workflow runs when retrieving baseline artifacts so recorded samples are not silently omitted.

CI:

• Update the CI workflow to stage benchmark samples before comparison, publish rolling history, and perform conditional regression confirmation.

Documentation:

• Document the rolling baseline, noise-aware threshold, confirmation behavior, fallback rules, and artifact contents.

Tests:

• Add unit, property-based, contract, and behavioral coverage for baseline history, noise handling, publication behavior, and regression confirmation.

[sourcery-ai]

Sorry @leynos, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3da06792-367e-41f6-a7fc-597006e4a1a0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Introduce a rolling window of main-branch benchmark samples with noise-aware thresholds for the Rust performance ratchet, and update the CI workflow plus helpers so every completed main run records, publishes, and later reuses that window rather than a single, pass-filtered baseline sample.

Sequence diagram for recording and using the rolling benchmark baseline window

sequenceDiagram
    actor Developer
    participant GitHubActions
    participant fetch_main_benchmark_baseline_py as fetch_main_benchmark_baseline.py
    participant ratchet_rust_performance_py as ratchet_rust_performance.py
    participant update_baseline_history_py as update_baseline_history.py

    Developer->>GitHubActions: push main
    GitHubActions->>fetch_main_benchmark_baseline_py: run with --run-status completed
    fetch_main_benchmark_baseline_py-->>GitHubActions: download main baseline artifact
    GitHubActions->>ratchet_rust_performance_py: compare_rust_regressions
        activate ratchet_rust_performance_py
        ratchet_rust_performance_py->>ratchet_rust_performance_py: load_history(baseline_history)
        ratchet_rust_performance_py->>ratchet_rust_performance_py: _baseline_window(baseline, candidate, history)
        ratchet_rust_performance_py->>ratchet_rust_performance_py: compare_scenario with median_ratio and noise_tolerance
        ratchet_rust_performance_py-->>GitHubActions: ratchet-report.json
        deactivate ratchet_rust_performance_py
    GitHubActions->>update_baseline_history_py: record this run's sample
        activate update_baseline_history_py
        update_baseline_history_py->>update_baseline_history_py: load_history(--history)
        update_baseline_history_py->>update_baseline_history_py: run_ratios(candidate)
        update_baseline_history_py->>update_baseline_history_py: BaselineHistory.appended(sample, window_size)
        update_baseline_history_py-->>GitHubActions: main-baseline-history.json
        deactivate update_baseline_history_py
    GitHubActions->>GitHubActions: upload benchmark-ratchet-main-baseline artifact

    Developer->>GitHubActions: open pull_request
    GitHubActions->>fetch_main_benchmark_baseline_py: fetch window artifact
    GitHubActions->>ratchet_rust_performance_py: compare_rust_regressions using window
    ratchet_rust_performance_py-->>GitHubActions: pass/fail based on max(flat_threshold, noise_tolerance)

Loading

File-Level Changes

Change	Details	Files
Add a rolling baseline history window with median-based bar and noise band, and wire it into the ratchet comparison.	Implement BaselineHistory, HistorySample, RatchetPolicy, and statistical helpers for median and MAD-based noise tolerance. Load and write history JSON with schema versioning and safe fallbacks for missing, unreadable, or incompatible files. Change compare_rust_regressions to prefer the history window, falling back to a single baseline sample, and to compute per-scenario thresholds as max(flat, noise). Extend ScenarioComparison and ComparisonReport to carry baseline_sample_count, noise_tolerance, and effective_threshold, and propagate them into report JSON.	benchmarks/ratchet_history.py benchmarks/ratchet_rust_performance.py benchmarks/ratchet_types.py
Factor out ratio extraction and profile metadata into a dedicated module used by both the ratchet and the history recorder.	Move plan/throughput loading, backend validation, scenario pairing, and ratio computation into a standalone module. Expose run_ratios and profile_metadata so both comparison and recording paths share one implementation. Update consumers to import from the new module instead of the old location.	benchmarks/ratchet_ratios.py benchmarks/comparison_analysis.py benchmarks/ratchet_rust_performance.py
Ensure CI records and publishes baseline history from every completed main run and that the ratchet reads that window.	Fetch baseline artifacts using a configurable run_status filter and pass --run-status completed from the workflow. Stage candidate plan/throughput into main-* files before running the ratchet so failing comparisons still leave a publishable sample. Run a history recorder script on completed main pushes to append the candidate sample (or carry the window forward on missing/unusable data) and always write the history file. Upload the updated history file as part of the baseline artifact and fail if it is missing. Pass the baseline history path into the ratchet CLI so comparisons use the window when available.	.github/workflows/ci.yml benchmarks/fetch_main_benchmark_baseline.py benchmarks/update_baseline_history.py benchmarks/ratchet_rust_performance.py
Add tests that lock in history window behaviour, noise-threshold semantics, CI publication/fetch contracts, and the new CLI/ratio plumbing.	Add unit tests for the history CLI to ensure samples are recorded with provenance, windows are pruned, unusable runs carry the window forward, and first runs still produce valid history. Add workflow contract tests that assert CI publishes/records on !cancelled() main pushes, includes the history file, uses --run-status completed, and passes --baseline-history with candidate staging before comparison. Extend existing ratchet tests to use RatchetPolicy and verify error conditions still hold with the refactored API. Add comprehensive tests for baseline history and noise handling, including incident reproduction, threshold composition, window mechanics, persistence, and Hypothesis properties. Add behaviour and feature tests describing noise-tolerance scenarios in terms of real incident measurements. Verify fetch_main_benchmark_baseline honours the run_status argument when querying workflow runs.	cuprum/unittests/test_benchmark_baseline_history_cli.py cuprum/unittests/test_benchmark_baseline_publication_contract.py cuprum/unittests/test_benchmark_ci_ratchet.py cuprum/unittests/test_benchmark_ratchet_history.py cuprum/unittests/test_fetch_main_benchmark_baseline.py tests/behaviour/test_benchmark_ratchet_noise_behaviour.py tests/features/benchmark_ratchet_noise.feature
Update documentation to describe the new window-based baseline and noise-aware ratchet semantics.	Document the ratchet rule as median-of-window baseline and combined flat/noise thresholds, including MAD-based spread estimation and cap. Add a developers-guide section explaining the failure mode of single-sample, pass-filtered baselines and how the window plus unconditional recording fixes it, including profile-compatibility and fallback behaviour. Update users-guide CI description to mention window-based comparison, history artefact publication, profile pruning, and the dual-threshold failure condition. Adjust snapshot tests as needed for maturin build output changes (not detailed in diff).	docs/cuprum-design.md docs/developers-guide.md docs/users-guide.md cuprum/unittests/__snapshots__/test_maturin_build.ambr

Possibly linked issues

• benchmark-ratchet: false-positive regressions from overhead-bound, noise-sensitive rust/python ratio #219: The PR addresses issue 219 by adding a rolling main history window, noise-aware thresholds, and unbiased baseline recording to eliminate false-positive ratchet failures.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77d131a91b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Defer loading the fallback until the history needs it

When the inspected benchmark-ratchet job fails before run_smoke_benchmarks creates the candidate files, the new recording step deliberately carries the existing history forward, and the upload step can consequently publish a valid history-only baseline artifact. On the next run, that history could perform the comparison, but _load_baseline eagerly opens the absent main-plan.json and main-throughput.json first, causing exit code 2 before _baseline_window examines the history; subsequent PR benchmark checks can therefore remain broken after a transient main-branch benchmark failure. Load these fallback files only when no compatible history exists, or carry the previous fallback files into every published artifact.

Useful? React with 👍 / 👎.

[leynos]

CI status on this branch:

• CodeScene Code Health, changes, lint-test, extension-gated tests, wheel builds: green.
• Typecheck and test (3.13, 3.14, 3.15a): failing on the four test_non_positive_timeout_at_public_boundary cases that Preserve partial capture when a timed-out run's readers have not reached EOF (#292) #293 fixes. That failure is main's, not this branch's — main's own run 31733260892 already fails them on 3.14 and 3.15a, and 3.13 has since joined as runner Python patch releases landed. Nothing here touches the subprocess path.
• benchmark-ratchet: skipped, because it needs: typecheck-test and that job failed. That is the dependency-failure behaviour Skip benchmark-ratchet on non-performance pull requests (4.4.3.) #289 pins deliberately — a failed dependency skips the paid job rather than running it ungated — but it does mean the live path this PR changes (fetch → compare → record → publish) will not exercise in CI until Preserve partial capture when a timed-out run's readers have not reached EOF (#292) #293 lands.

Until then the coverage is local: unit and property tests over the window and the threshold, behavioural scenarios stating the incident's own numbers, and contract tests over the four workflow declarations that would silently reinstate the bias. All six make gates are green, and actionlint is clean.

Worth noting for whoever merges: the first main run after this lands has no history and uses the single-sample fallback, so the window fills over the following few merges. Until it does, ratchet-report.json will report baseline_sample_count: 1.