ci(benchmarks): add percentage-based regression gate for microbenchmarks - #20057
ci(benchmarks): add percentage-based regression gate for microbenchmarks#20057igoragoli wants to merge 8 commits into
Conversation
Adds a check-big-regressions job in the gate stage, before check-slo-breaches, failing when a metric regressed more than 10% against the baseline commit. The step script drops FLAKY_BENCHMARKS_REGEX matches before comparing, since benchmark_analyzer compare pairwise has no exclusion flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
igoragoli
added
the
changelog/no-changelog
Codeowners resolved asResolved from the full PR diff against |
Circular import analysis
|
Dependency direction analysis
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e6d9d7ae5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
BenchmarksBenchmark execution time: 2026-09-04 15:20:47 Comparing candidate commit 23cbffd in PR branch Found 0 performance improvements and 9 performance regressions! Performance is the same for 577 metrics, 10 unstable metrics, 3 known flaky benchmarks, 15 flaky benchmarks without significant changes.
|
Replaces the repo-local step script with a bp-runner experiment, matching how check-slo-breaches already works. The platform's fail_on_regression step covers the baseline lookup, the bypass label and the missing-baseline case, so none of that needs a local implementation. Drops the jq flaky filtering with it. benchmark_analyzer convert reads FLAKY_BENCHMARKS_REGEX and stamps flaky: "true" on matching benchmarks, and both --fail-on-regression and --fail-on-breach already skip flagged benchmarks, so the filtering was reimplementing an existing feature. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b89c2ebba
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
fail-on-regression.sh reads ARTIFACTS_DIR/baseline_branch.txt to decide whether a baseline exists. run-benchmarks.sh only writes it per-scenario (reports/<scenario>/), and the loop only copied *.converted.json to the flat reports/ dir the gate actually reads, so the gate always saw it missing and warned+passed without comparing anything. Confirmed live: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/2013171889#L281
check-big-regressions and check-slo-breaches run concurrently in the same stage, not sequentially. The threshold is set via bp-runner's regression_threshold key, not a FAIL_ON_REGRESSION_THRESHOLD variable. Also reframe the no-baseline warning around the actual trigger (no baseline wheel built) rather than PR existence.
2 participants
Description
Adds a
check-big-regressionsjob in thegatestage, beforecheck-slo-breaches. Fails when a metric regressed more than 10% against baseline.Small regressions accumulate and lead to PRs with unrelated changes breaching performance SLOs. Percentage-based checks should reduce this.
Known flaky benchmarks cannot fail the gate.
benchmark_analyzer convertreadsFLAKY_BENCHMARKS_REGEXduring themicrobenchmarksjob and marks matching benchmarks, and the regression check skips marked benchmarks.Testing
https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/2016179612
Failed because of flaky benches varying more than 10%. #20050 marks those as flaky and should be merged before.
Risks
Any flaky benchmark that varies more than 10% will block PRs.
The fix is either to:
FLAKY_BENCHMARKS_REGEXinmicrobenchmarks.yml, then re-runmicrobenchmarksand the gate. Re-running the gate alone is not enough, since the flaky mark is applied whenmicrobenchmarksconverts its results.performance/ignore-performance-regressionlabel, as described in Performance Quality Gates.Additional Notes