perf(bench-ratchet): seed the baseline from a window, reduced in ratio space

[mparrett]

Follow-up to #564. seed-baseline currently takes the newest snapshot per machine key. This takes a window of five and medians it, reduces in ratio space rather than raw ns/op, gates on whether a snapshot agrees with its window instead of on how far its anchor drifted, and lands the re-seeded docs/perf/baseline.json alongside the method, since check gates on timing only where baseline.Machines[key] exists and a baseline whose provenance no longer matches the tool degrades to no gate rather than to a failure.

The re-seed

Regenerated as the last step after rebasing, so the carried arm64/Apple M3 profile is byte-identical to the one on main and the amd64 windows are current:

go run ./cmd/bench-ratchet -perf-data-dir <perf-data>/timeline seed-baseline

Corpus: perf-data at f8ed4298, 426 timeline files, window 5, as of 2026-09-07. Five amd64 tiers seeded from full 5/5 windows, newest contributing SHAs 477a5d36, b0397f63, 928c2170, 87579506, and f3ca5f9b. The 8370C tier reports 16 benchmarks below quorum and skips them. The new AMD EPYC 9V45 key has one snapshot and is skipped by -seed-min-window (see below). All 768 seeded ratio identities hold.

A fresh seed writes only the gated fields, so the per-sample arrays and best_since provenance go with it; both are omitempty reporting data that update and aggregate repopulate as runs accumulate. check gates on ns_per_op, ratio_to_anchor, allocs_per_op, and bytes_per_op, all present.

Correction to my #564 review

I said there to skip a candidate whose anchor is an outlier against its own tier. I built that, ran it against perf-data, and the corpus refutes it.

Snapshot a588a69d2759 (EPYC 9V74) sits 22.4% off its window's anchor. It is also uniformly 22.4% fast in raw ns/op across all 162 of its benchmarks, and agrees with its window on every ratio_to_anchor to within 0.1%. The host was fast that day and the anchor divided it back out, which is what the anchor is for. Anchor-deviation gating would have discarded it and two more like it.

Over the 24 most recent amd64 snapshots as of 2026-08-05:

	range / worst
anchor deviation from tier median	−22.4% … +3.1%
ratio_to_anchor offset from tier median	median 0.03%, worst 1.75%

So the gate is on ratio coherence: a snapshot is rejected when its ratios sit more than -seed-coherence-tolerance (default 5%) off the rest of the window. That catches the case that damages a baseline, the mixed capture, where the anchor caught the slow tail and the benchmarks did not, so every ratio is uniformly wrong while the raw numbers look ordinary. Nothing in the current corpus trips it, which is the expected result and not evidence the check is idle.

Why a window

One snapshot is one CI run, and one CI run is one sample. Seeding from the newest versus a median of five, same corpus, same day (2026-08-05):

	share of 758 (tier, benchmark) floors
differ by more than 5% (the gate budget)	22.6%
differ by more than 10%	11.3%
differ by more than 25%	5.8%

Part of that spread is real code movement across the window and part is sampling; I can't separate them without a null control. Either way, seeding from one snapshot sets a fifth of the gate's thresholds from a single observation of it.

Median rather than min: update already takes a min over history when it ratchets, and seeding with a second minimum stacks two of them into a floor no clean run reaches.

Why ratio space

Raw ns_per_op carries host speed; ratio_to_anchor does not (see the table above). Reducing the quantity that carries host speed and then dividing imports that speed into the stored floor. The two approaches agree whenever a window comes from one host and diverge exactly when it does not, which a shared runner pool guarantees. ns_per_op is derived back from the reduced ratio and the window's anchor, so ratio_to_anchor == ns_per_op / anchor.ns_per_op holds by construction.

Additional fixes

• Candidates are grouped by the machine key read from each snapshot's content, not its filename slug. Under filename grouping a mis-named file forms a group of its own and is stored under the key its content names, so a one-file reduction could overwrite the five-run window the correctly named files just produced. The filename is still reported when it disagrees, since it is what the log prints and what a human greps for. No file in perf-data has this shape as of 2026-09-02; this guards a silent, order-dependent failure rather than repairing live data.
• A machine key with fewer than -seed-min-window snapshots (default 3) is skipped and reported. The coherence check cannot vote on fewer than three, and the benchmark quorum passes everything at (1+1)/2 = 1, so a one-snapshot tier would be seeded with every floor a single observation. A new runner CPU landed one on 2026-09-06. Skipping leaves check ungated on that machine until the runs accrue, the same call the benchmark-level quorum makes.
• M3 preservation requires Machine.Arch != -seed-arch as well as the model string, mirroring the skip in the candidate scan. Without it an amd64 model string containing "M3" would overwrite its own fresh seed.
• The arch filter applies to the file's content, not only its name, and warns when the two disagree.
• Snapshot names parse through an anchored regexp. A positional split on - yields a plausible-looking wrong SHA, and the machine slug is full of dashes. Unparseable names are skipped and reported: 120 in the current corpus, all predating the machine-slug naming.
• A benchmark present in under half the window is skipped rather than seeded from one observation. check reports it as NEW, which is honest.
• b.N movement across the window is reported, not acted on. b.N is an output of the timing loop, so it moves when per-op cost moves, and ns/op is N-dependent wherever iterations share state. Excluding on it would shrink the gate silently.
• Every matching exclusion prefix is marked, not just the first. BenchmarkClojureTestSuite is a prefix of BenchmarkClojureTestSuiteCompileAndRun, so the longer entry was reported as stale on every run while it was in fact filtering.
• filterUnstableBenchmarks (from build: bench-ratchet infrastructure — deterministic rebaseline, Go 1.26.5 baseline, pre-push gate #780) sources its prefixes from unstableBenchmarks in seed.go rather than a second copy of the list.

Verification

• go test ./cmd/bench-ratchet: 14 seed tests as of 2026-09-07, including TestSeedBaselineKeepsUniformlyFastSnapshot (the negative result above), TestSeedBaselineRejectsMixedCapture, TestSeedBaselineGroupsMisnamedFileIntoContentWindow (a full EPYC window plus one Intel-named EPYC file), TestSeedBaselinePreservesOnlyOffArchM3, and TestSeedBaselineSkipsMachineBelowMinWindow. Each guard test fails with its guard removed.
• The seed run above against the real perf-data timeline, output in the commit message.
• -seed-window 1 -seed-min-window 1 reproduces the current behaviour, which is how the comparison table above was produced.

Not addressed here

The 5% budget still has no measured floor under it. Nothing in this PR tells you whether 5% is above or below the gap between two builds that cannot differ on these tiers; a comment-only-edit control run through the existing dispatch would. Worth doing before the budget is tightened.

The perf-release-baseline workflow from #740 has not run yet, so there is no tighten-merge for this seed to clobber. Dispatch the v1.12.2 backfill after this merges and it tightens on top of the fresh seed.

[nnunley]

#564 landed on main as c9bf9335d100aa00b548b70eae6215d4486828b5 (squash merge, 2026-08-07).

This PR still lists the pre-squash #564 commits (6a825ba8c4a7 … 7f8d298bd120) plus its own tip 5601f1587de6, so it needs the rebase you described in the body before the four-file window/ratio-space delta is reviewable on its own.

No rush and no action needed from my side — flagging it since the blocker you were waiting on is cleared.

[nnunley]

Thanks for the rebase. The argument for ratio-space reduction convinced me, and the
negative result — keeping the uniformly-fast host that anchor-deviation gating would have
discarded — is the part that makes the coherence gate credible rather than decorative.
Deriving ns_per_op back from the reduced ratio so ratio == ns/anchor holds by
construction is the right call given what check relies on. Marking every matching
exclusion prefix rather than the first is a real fix: …TestSuite is a prefix of
…TestSuiteCompileAndRun, so first-match would permanently mis-report a live exclusion.

The open question: this lands the method but not the result. docs/perf/baseline.json
on main still comes from the old newest-snapshot, raw-ns seeding. By your own table 22.6% of
the 758 floors move by more than the 5% gate budget under the new method, so until someone
re-runs the seed, the gate keeps enforcing the old floors. Is the re-seed a follow-up, or
should it ride along here?

One nit: M3 preservation now keys only on CPUModel containing "M3" and overwrites a freshly
seeded entry. The old code also required the machine key. Nothing in the current baseline
trips it, but an amd64 model string containing "M3" would silently override its own fresh
seed; also requiring Arch != archPrefix closes it.

I ran the unit tests, not a seed against the real perf-data timeline, so the corpus figures
in the body are yours, not independently reproduced by me.

[mparrett]

Is the re-seed a follow-up, or should it ride along here?

Rides along. I will re-seed and commit baseline.json here, with the seed-baseline invocation pasted into the body the way nooga asked for on #564.

The technical reason not to split it: check gates on timing only when baseline.Machines[key] exists (main.go:452-459), so a baseline whose provenance no longer matches the tool degrades to no gate rather than to a failure. Landing the method without the data leaves that gap open on whichever tier drifted.

Mechanism

Rebase (23 commits behind), fix the M3 guard, then re-derive as the last step before merge so the window is current:

git worktree add ../lg-perf-data origin/perf-data     # 364 timeline snapshots

go run ./cmd/bench-ratchet \
  -perf-data-dir ../lg-perf-data/timeline \
  -baseline docs/perf/baseline.json \
  seed-baseline

go run ./cmd/bench-ratchet check                      # exit 0 on current main

No measurement runs: it derives over snapshots already on perf-data, so it is deterministic against a fixed corpus and re-running at merge costs one command.

M3 nit

Taking it. seed.go:162 becomes:

if mb.Machine.Arch != opt.archPrefix && strings.Contains(mb.Machine.CPUModel, "M3") {

Machine.Arch is already the first half of perfdata.MachineKey, so the guard is free.

Ordering against #740

#740 writes baseline.json too, but tighten-only (aggregate, main.go:519), so it adopts floors in the tightening direction only — a floor the window method says should be looser survives it. It has not run yet either: docs/perf/historical/ still holds only v1.8.0.json.

seed-baseline is a full rewrite of the seeded tiers, so it clobbers any tighten-merge that lands before it. Either I re-seed before you dispatch the v1.12.2 backfill, or you re-dispatch after this merges. Your call — the re-seed is my last step either way, so tell me which and I will time it accordingly.

Staying out

The 5% budget still has no measured floor under it, which is a measurement question rather than a seeding one. Out of scope here.

The corpus figures in the body are still only mine. The seed output will put the tier counts and window sizes where you can check them against perf-data directly.

[nooga]

The method discussion looks settled, but the diff doesn't yet do what the PR is for: docs/perf/baseline.json isn't in it, and the M3 guard at cmd/bench-ratchet/seed.go:162 is still a bare strings.Contains(CPUModel, "M3") without the Arch check you agreed to. Please push the re-seed and the guard fix, then rebase — #780 is landing first and touches the same region of cmd/bench-ratchet/main.go, so expect a real (small) conflict there.

[mparrett]

Both done and pushed; the branch is rebased onto current main and 0 behind.

The guard. seed.go now reads mb.Machine.Arch != opt.archPrefix && strings.Contains(mb.Machine.CPUModel, "M3"), which is the line I agreed to on 08-21, and it mirrors the existing skip a few lines down at seed.go:200. There is a regression test for it: against the bare guard the seeded amd64 tier comes back as the carried-over 999 instead of the freshly seeded 100, so it fails for the reason it exists.

The re-seed. docs/perf/baseline.json is in the diff now, seeded from the 381-snapshot timeline with the default window of 5:

go run ./cmd/bench-ratchet -perf-data-dir <perf-data>/timeline seed-baseline

Five amd64 tiers rebuilt from full 5/5 windows, newest contributing SHAs dadb8e0b, 69d88da3, e344ff23, 98ad2bfc and f3ca5f9b. Two tiers reported benchmarks below quorum and skipped them, and the seed log lists the b.N movement per tier.

Two things to know before you read the diff:

• It deletes far more than it adds. A fresh seed writes only the gated fields, so the per-sample arrays and the best_since provenance go with it. Both are omitempty reporting data that update and aggregate repopulate as runs accumulate; check gates on ns_per_op, ratio_to_anchor, allocs_per_op and bytes_per_op, all present.
• This is what closes the gap perf(bytecode): compact lazy var metadata #781 reports, where the committed baseline predates go1.26.5 and InitFromLGB reads +274% allocs against it on main. That tier's allocs move 7391 to 26294, which is what main does today.

What it does not fix. The arm64/Apple M3 tier is still the 2026-07-18 capture, carried over rather than seeded, because there is no CI counterpart to derive it from. Your point on #780 about that profile having a single owner is unchanged by this PR.

I will rebase again over #780 once it lands, per your ordering. One thing still open from my 08-21 note: if the v1.12.2 backfill for #740 has already been dispatched, it wants a re-dispatch after this merges, since seed-baseline is a full rewrite of the seeded tiers and clobbers a tighten-merge that lands before it.

[nnunley]

Requesting changes on exact head 139b05e5. The ratio-space reduction and committed medians check out independently, but content/filename mismatch handling can silently overwrite a correct machine window.

seedBaseline groups by filename slug (seed.go:127–137), seedOneMachine warns and returns the content key (:201–216, :264), then the outer loop assigns merged.Machines[key] = mb (:159). An Intel-named file carrying EPYC content is therefore reduced separately and can overwrite the proper five-run EPYC reduction with a one-file result. The current mismatch test has no proper EPYC window, so it misses this collision. Please group candidates by the machine key read from content (while retaining the filename warning), and add a test with both the normal EPYC window and an Intel-named/EPYC-content file.

Also rebase over merged #780 and regenerate baseline.json last. GitHub reports a real main.go conflict; both this PR’s seed flags and #780’s rebaselineForce must survive. The carried M3 is pre-#780 (f154c7… versus current-main dadb8e…), so taking this branch’s JSON would restore stale bars.

Independent validation: 25 current five-window snapshots reproduce all five amd64 profiles with zero discrepancies in anchors, ratios, derived ns/op, bytes, allocs, or quorum membership; all 906 committed ratio identities hold.

[mparrett]

@nnunley — fixed, rebased, and re-seeded, in that order. Head is 7a0059f2.

The grouping collision. You were right about both the mechanism and the test gap. gatherCandidates now reads every snapshot up front and groups the profiles by the machine key derived from their content, so a mis-named file becomes one more candidate in the window it belongs to rather than a group of its own stored under someone else's key. Window truncation and the coherence check both run over the content-keyed group. seedOneMachine no longer returns a key, because there is only one it could return.

The filename warning stays. It no longer decides anything, but the filename is what the seed log prints and what someone greps for, so a divergence still has to be visible. The architecture filter runs on content as before; a file named for the seeded architecture whose profile is a different one is now reported, while a file named for another architecture is skipped quietly.

TestSeedBaselineGroupsMisnamedFileIntoContentWindow is the test you asked for: a full five-snapshot EPYC window plus one Intel-named/EPYC-content file. The stray file is uniformly slow — anchor and benchmark scaled together, ratio unchanged — so the coherence check keeps it and grouping is the only thing under test. Against the old code it stores 200 ns/op, the stray file's own number, instead of the window's 100. I ran it both ways to confirm it fails for the reason it exists.

The perf-data claim was wrong. The code carried a claim that perf-data holds Intel-slugged files with EPYC content. It does not. All 382 timeline files agree with their names as of 2026-09-02, and the branch is append-only with no deletions or renames in its history, so there was never a file of that shape to observe. I removed the claim from the comment and said so in the commit message: this is a guard on a silent, order-dependent failure, not a repair of live data. The bug is real; my evidence for it having already bitten was not.

The rebase. Onto current main, so #780 is underneath. Two things needed hand-resolution:

• The flag block in main.go. Both sides survive — -force keeps your reworded description ("replace current-machine timing and rebase global deterministic metrics"), and the five -seed-* flags sit below it.
• filterUnstableBenchmarks. This branch had deleted it, because the only caller was the old seed path it replaced; build: bench-ratchet infrastructure — deterministic rebaseline, Go 1.26.5 baseline, pre-push gate #780 then added a new caller in update plus a unit test. It is restored, sourcing its prefixes from unstableBenchmarks in seed.go rather than keeping a second copy of the list — the two paths disagreeing about which benchmarks are ratchetable is what a duplicated list invites. forceRebaseline came through untouched.

The re-seed. Five amd64 tiers, all from full 5/5 windows, newest contributing SHAs dadb8e0b, 33fb9e3a, e344ff23, 98ad2bfc, and f3ca5f9b. Three tiers reported benchmarks below quorum and skipped them. All 766 committed ratio identities hold.

Running it after the rebase is what settles the stale-bars problem you flagged: the carried arm64/Apple M3 profile is now byte-identical to the one on main at dadb8e0b, not the pre-#780 capture at f154c7. I diffed the two profiles to confirm rather than reading the SHA field.

gofmt clean, go vet clean, cmd/bench-ratchet tests pass, and go build succeeds under linux/amd64, js/wasm, and plan9/amd64. Ready for another look.

One thing still open from my 08-21 note, unchanged by any of this: if the v1.12.2 backfill for #740 has already been dispatched, it wants a re-dispatch after this merges. seed-baseline rewrites the seeded tiers wholesale, so it would clobber a tighten-merge that lands before it.

[mparrett]

@nnunley rebased onto current main, one more fix, and re-seeded last. Head is c0fbea9d.

The #740 ordering question is closed. perf-release-baseline.yml has never run and docs/perf/historical/ still holds only v1.8.0.json, so the v1.12.2 backfill was never dispatched and there is no tighten-merge for this seed to clobber. Dispatch it after this merges and it tightens on top of the fresh seed. The run list answered what I had been asking you to decide.

One new fix. A dry re-seed against the current corpus wrote a tier I had not seen before:

amd64/AMD EPYC 9V45 96-Core Processor: 157 benchmarks from 1/1 snapshots

A new runner CPU landed its first timeline snapshot on 2026-09-06. The coherence check cannot vote on fewer than three snapshots and the benchmark quorum passes everything at (1+1)/2 = 1, so every floor for that tier would have been a single observation, which is the case the window exists to prevent. -seed-min-window (default 3, the coherence check's own floor) now skips such a key and says so; check runs ungated on that machine until the runs accrue, the same call the benchmark-level quorum already makes. TestSeedBaselineSkipsMachineBelowMinWindow covers it and fails with the gate removed. The -seed-window 1 comparison mode now needs -seed-min-window 1 alongside it.

The re-seed. perf-data at f8ed4298, 426 files. Five amd64 tiers from full 5/5 windows, newest contributing SHAs 477a5d36, b0397f63, 928c2170, 87579506, and f3ca5f9b; the 8370C tier skips 16 benchmarks below quorum; 9V45 skipped as above. All 768 seeded ratio identities hold, and the carried M3 profile is byte-identical to main's. Against the 09-02 seed, 73 of 906 floors moved by more than 5%, which is the corpus growing, and the reason the seed has to be the last commit.

Everything from your 09-02 review is otherwise unchanged at this head: content-keyed grouping, the EPYC-window-plus-misnamed-file test, and filterUnstableBenchmarks sourcing its list from seed.go. The PR body is rewritten to describe the diff as it now stands, with the seed invocation and tier summary in it.

Ready for another look. The min-window gate is the only logic new since your review. If the corpus moves enough between review and merge to matter, I will re-seed once more at the end; the command and the numbers are in the commit message either way.

[mparrett]

Rebased onto 638b4a6a; head is now 257a58c7. All six commits replayed with no conflicts, and the branch's own diff against its base is byte-identical to what it was before the rebase.

All 15 required checks green.