tooling: add bench-baton, a machine-quiescence lease for benchmarks - #795
tooling: add bench-baton, a machine-quiescence lease for benchmarks#795nnunley wants to merge 1 commit into
Conversation
Benchmarks and the pre-push ratchet need a quiet machine; a test suite or a second benchmark running alongside produces numbers the ratchet cannot tell from a regression (the 2026-09-03 gate failures in nooga#791 were partly that). cmd/bench-baton arbitrates one pool per machine with two lanes: `bench` holds the pool exclusively, `build` runs concurrently up to --max-shared slots, and a waiting bench closes the gate so builds cannot starve it. Leases are flock(2)s, so a dead holder releases by itself; a JSON state file, a ledger, and per-run logs live under $BENCH_BATON_HOME (default the user cache dir) per pool, in the same shape as the Python tool this replaces so an existing pool stays readable. The skill card at .agent/skills/bench-baton (linked from .agents/skills) carries the rules: timing gates go through bench, never build; stable --owner per worker; cheap commands take no lease. Registered in the skill index and manifest. A !unix stub keeps `go build ./...` clean elsewhere. Tests pin the lane semantics: shared leases overlap up to the cap, an exclusive waits for shared holders, a waiting exclusive defers new shared work, reap drops dead holders, and run logs, ledgers, and returns the wrapped exit code.
|
Nice one! Review incoming. |
mparrett
left a comment
There was a problem hiding this comment.
Two blocking findings on 94c182d:
-
cmd/bench-baton/main.go:469 kills only the immediate child on timeout, then releases the lease. I reproduced this with a wrapped shell that started sleep in the background: run returned rc=127, status reported no holder, and the descendant PID was still alive. A timed-out make or test process can therefore keep consuming CPU while the next benchmark runs. Please terminate and wait for the full process group before releasing; the same lifecycle needs consideration when the baton itself is terminated so a workload cannot outlive its lease.
-
cmd/bench-baton/main.go:315-345 implements max-shared as a per-invocation slot search rather than a pool capacity. With one live holder started using --max-shared 1, a second default invocation immediately acquired slot1 and ran concurrently. Different workers can therefore silently defeat the requested cap, while status always reports the hard-coded value 4. Please make the capacity a pool invariant and reject mismatches, or otherwise enforce one consistent value and report the actual capacity.
The package build, vet, unit tests, and 10 repeated race-enabled test runs otherwise pass.
Benchmarks and the pre-push ratchet need a quiet machine; a test suite or a second benchmark running alongside produces numbers the ratchet cannot tell from a regression (part of what happened in #791).
cmd/bench-batonarbitrates one pool per machine with two lanes:benchholds the pool exclusively (benchmarks,make bench-ratchet).buildruns concurrently up to--max-sharedslots (builds, suites,make generate).benchcloses the gate, so a stream of builds cannot starve it.Leases are
flock(2)s, so a dead holder releases by itself;state.json,ledger.jsonl, and per-run logs live under$BENCH_BATON_HOME(default the user cache dir) per pool, in the same shape as the Python tool this replaces so an existing pool stays readable. A!unixstub keepsgo build ./...clean elsewhere.The skill card at
.agents/skills/bench-baton/SKILL.md(linked from.claude/skills, likedocs-status) carries the rules: timing gates go throughbench, neverbuild; stable--ownerper worker; cheap commands take no lease.Tests pin the lane semantics: shared leases overlap up to the cap, an exclusive waits for shared holders, a waiting exclusive defers new shared work,
reapdrops dead holders, andrunlogs, ledgers, and returns the wrapped exit code.Pushed past the pre-push ratchet, which is red on current
main(#791) until the fix lands.