test: add direct-call baseline and threaded-contention benchmarks - #129
test: add direct-call baseline and threaded-contention benchmarks#129IIIIbntttt wants to merge 3 commits into
Conversation
Close the two gaps left open in bagowix#84 after the CodSpeed suite landed: - test_baseline_direct_call and its async twin measure the unwrapped callable, so any call-path result divides into an overhead ratio — the number a prospective user actually wants. - benchmarks/test_contention.py drives one closed breaker from four worker threads to track the work under the shared lock as a trend. The docstring and CONTRIBUTING.md spell out that instruction counting runs threads one at a time: the number is lock-path work, not wall-clock contention.
…-contention # Conflicts: # CHANGELOG.md
bagowix
left a comment
bagowix
left a comment
There was a problem hiding this comment.
Thanks for picking this up — this closes out #84 exactly as scoped in the status check, and the execution is careful: the pool is reused across iterations so thread startup stays out of the measurement, the warm-up assert both validates correctness and fills the window to steady state before measuring, and the Valgrind caveat is spelled out right where the number will be read. I ran the full gate locally on the branch (ruff, mypy, pyright, pyrefly, full suite at 100% coverage, pytest benchmarks --codspeed -n 0) — all green, and the workflow picks up the new module with no changes, as advertised.
One non-blocking nit, take it or leave it: the async baseline is slightly asymmetric with test_call_async. The breaker path runs a wrapper coroutine (guarded) that awaits breaker.call(...), while the baseline runs the bare _async_work coroutine with no wrapper — so the overhead ratio includes one extra coroutine frame that real usage doesn't pay (in real code the user's own coroutine exists in both cases: await fn(...) vs await breaker.call(fn, ...)). Wrapping the baseline the same way (async def bare(): return await _async_work(1, 2)) would make the ratio exactly apples-to-apples. For a trend metric it's noise either way, so happy to merge as is.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe benchmark suite adds synchronous and asynchronous direct-call baselines and a four-worker threaded contention benchmark. Documentation explains baseline comparisons and the limits of serialized instruction-count measurements. ChangesBenchmark coverage
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Comment |
The breaker path drives a wrapper coroutine that awaits breaker.call, while the baseline ran the bare coroutine — so the overhead ratio charged the breaker for one coroutine frame real usage pays on both sides. Wrapping the baseline the same way makes the ratio isolate the breaker itself.
2 participants
Summary
Closes the two gaps left open in #84 after the CodSpeed suite landed (per the status check in the issue comments):
test_baseline_direct_callmeasures the unwrapped callable, so any call-path result divides into an overhead ratio — the number a prospective user actually wants. Its async twin runs the bare coroutine on the same pre-builtasyncio.Runnerthe async benchmarks use, so the loop-dispatch cost the async paths pay is present in their baseline too and the ratio stays honest.benchmarks/test_contention.pydrives one closed breaker from four worker threads (25 calls each, pool reused across iterations so thread startup is excluded), tracking the work under the shared lock as a trend. The module docstring andCONTRIBUTING.mdcarry the caveat: CodSpeed's simulation mode counts instructions under Valgrind, which runs threads one at a time — the number is lock-path work, not wall-clock contention. Correctness under real contention stays withtests/test_concurrency.py/ Run the test suite on free-threaded CPython (3.14t) #102.No runtime code is touched; the CodSpeed workflow needs no change (it collects
benchmarks/wholesale).docs/is intentionally untouched — thecorrectness.mdCodSpeed paragraph stays accurate as written; the reading guidance is contributor-facing and lives inCONTRIBUTING.md.Checklist
uv run ruff format --checkanduv run ruff checkpassuv run mypy,uv run pyrightanduv run pyrefly checkpassdocs/) for user-facing changesCHANGELOG.md[Unreleased]updatedRelated issues
Closes #84
Summary by CodeRabbit
Documentation
Tests