Skip to content

tooling: add bench-baton, a machine-quiescence lease for benchmarks - #795

Open
nnunley wants to merge 1 commit into
nooga:mainfrom
nnunley:tooling/bench-baton
Open

tooling: add bench-baton, a machine-quiescence lease for benchmarks#795
nnunley wants to merge 1 commit into
nooga:mainfrom
nnunley:tooling/bench-baton

Conversation

@nnunley

@nnunley nnunley commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Benchmarks and the pre-push ratchet need a quiet machine; a test suite or a second benchmark running alongside produces numbers the ratchet cannot tell from a regression (part of what happened in #791). cmd/bench-baton arbitrates one pool per machine with two lanes:

  • bench holds the pool exclusively (benchmarks, make bench-ratchet).
  • build runs concurrently up to --max-shared slots (builds, suites, make generate).
  • A waiting bench closes the gate, so a stream of builds cannot starve it.

Leases are flock(2)s, so a dead holder releases by itself; state.json, ledger.jsonl, and per-run logs live under $BENCH_BATON_HOME (default the user cache dir) per pool, in the same shape as the Python tool this replaces so an existing pool stays readable. A !unix stub keeps go build ./... clean elsewhere.

go run ./cmd/bench-baton bench --owner pr641 --cwd WT -- make bench-ratchet
go run ./cmd/bench-baton build --owner pr641 --cwd WT -- go test ./...
go run ./cmd/bench-baton status | ledger -n 10 | reap

The skill card at .agents/skills/bench-baton/SKILL.md (linked from .claude/skills, like docs-status) carries the rules: timing gates go through bench, never build; stable --owner per worker; cheap commands take no lease.

Tests pin the lane semantics: shared leases overlap up to the cap, an exclusive waits for shared holders, a waiting exclusive defers new shared work, reap drops dead holders, and run logs, ledgers, and returns the wrapped exit code.

Pushed past the pre-push ratchet, which is red on current main (#791) until the fix lands.

Benchmarks and the pre-push ratchet need a quiet machine; a test suite or a
second benchmark running alongside produces numbers the ratchet cannot
tell from a regression (the 2026-09-03 gate failures in nooga#791 were partly
that). cmd/bench-baton arbitrates one pool per machine with two lanes:
`bench` holds the pool exclusively, `build` runs concurrently up to
--max-shared slots, and a waiting bench closes the gate so builds cannot
starve it. Leases are flock(2)s, so a dead holder releases by itself; a
JSON state file, a ledger, and per-run logs live under
$BENCH_BATON_HOME (default the user cache dir) per pool, in the same
shape as the Python tool this replaces so an existing pool stays readable.

The skill card at .agent/skills/bench-baton (linked from .agents/skills)
carries the rules: timing gates go through bench, never build; stable
--owner per worker; cheap commands take no lease. Registered in the skill
index and manifest. A !unix stub keeps `go build ./...` clean elsewhere.

Tests pin the lane semantics: shared leases overlap up to the cap, an
exclusive waits for shared holders, a waiting exclusive defers new shared
work, reap drops dead holders, and run logs, ledgers, and returns the
wrapped exit code.
@mparrett
mparrett self-requested a review September 4, 2026 13:24
@mparrett

mparrett commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Nice one! Review incoming.

@mparrett mparrett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blocking findings on 94c182d:

  1. cmd/bench-baton/main.go:469 kills only the immediate child on timeout, then releases the lease. I reproduced this with a wrapped shell that started sleep in the background: run returned rc=127, status reported no holder, and the descendant PID was still alive. A timed-out make or test process can therefore keep consuming CPU while the next benchmark runs. Please terminate and wait for the full process group before releasing; the same lifecycle needs consideration when the baton itself is terminated so a workload cannot outlive its lease.

  2. cmd/bench-baton/main.go:315-345 implements max-shared as a per-invocation slot search rather than a pool capacity. With one live holder started using --max-shared 1, a second default invocation immediately acquired slot1 and ran concurrently. Different workers can therefore silently defeat the requested cap, while status always reports the hard-coded value 4. Please make the capacity a pool invariant and reject mismatches, or otherwise enforce one consistent value and report the actual capacity.

The package build, vet, unit tests, and 10 repeated race-enabled test runs otherwise pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants