test: Benchmarks for lock free apply_block

[sergerad]

Summary

These are benchmarks which will be used to compare results from #2345 work.

Preliminary results:

Mixed read/write benchmark: lock-free store vs next

Workload: benchmark-mixed -n 50000 -p 50 --storage-map-entries 64 --account-update-blocks 20 -r 16
(238 blocks applied while 16 concurrent readers loop get_block_header with MMR proof; release build, identical parameters on both branches.)

Metric	lock-free	next (locks)	Δ
Reads/sec	83,770	73,834	+13%
Total reads	33.7M	22.4M	+50%
P50 read latency	90µs	83µs	~equal
P95	687µs	664µs	~equal
P99	1.26ms	1.26ms	equal
P99.9	4.1ms	9.8ms	2.4× better
Avg block insertion	426ms	355ms	20% slower (reader-contention artifact, see -r 0 below)
Total seeding time	405s	304s	33% slower (same artifact)
Avg get-batch-inputs	2.0ms	1.8ms	~equal

Read side: confirms the thesis

P50–P99 are identical — uncontended reads cost the same on both branches. The entire
difference is in the extreme tail, exactly where lock waits live. On next, a read
arriving during a commit window waits for the writer's lock; with ~16 readers × 238
blocks those blocked samples are only ~0.02% of 22M reads, so they surface at P99.9
(9.8ms vs 4.1ms) and the true worst case is hidden — blocked reads on next can wait
up to a full insertion (~350ms), which the summary doesn't print (no max/P99.99).
Aggregate read throughput is also +13% despite the lock-free readers doing 50% more
total work.

Write side: confirmed as a benchmark artifact

On next, the lock throttles the readers: during every commit, all 16 readers park
on the RwLock and donate their cores to the writer and the rayon block-building
pipeline. On the lock-free branch, wait-free readers never park — they run straight
through every commit, competing with the writer for CPU. next is not writing faster
because locks are cheaper; it is writing faster because its readers are forcibly
idled. A closed-loop, unthrottled reader workload maximizes this effect; production
readers are request-driven and would not saturate cores this way.

This was verified by re-running the identical workload with zero readers
(seed-store, same parameters), isolating the pure write path:

Pure write path benchmark (-r 0): write perf is at parity

Metric	lock-free	next (locks)	Δ
Avg block insertion	262ms	252ms	+4% (within run-to-run noise)
Total seeding time	158.8s	153.9s	+3%
Per-block insert spread	228–345ms	234–335ms	overlapping
Avg get-batch-inputs	1.36ms	1.11ms	+23% (µs-scale, ~once per block ≈ 60ms over the whole run)

With no readers, the lock-free branch inserts blocks within 4% of next, and the
per-block distributions overlap almost entirely — so ≤4% (likely ~0) is the true cost
of the new commit path (per-block RocksDB snapshot creation and ArcSwap snapshot
publication). Putting all four data points together:

• next: 252ms → 355ms under 16 readers (+41% from reader CPU contention)
• lock-free: 262ms → 426ms under 16 readers (+63% from reader CPU contention)

Both writers slow down heavily when unthrottled readers saturate the machine; the
difference is that next partially shields its writer by stalling every reader
during every commit — which is precisely the read-tail cost the lock-free refactor
removes. The only measurable real regression is get-batch-inputs (+250µs per call,
once per block), which now reads through RocksDB snapshot views instead of the live
trees — a footnote, not a concern.

Bottom line: reads no longer stall behind block commits — the blocking tail is
gone (2.4× better P99.9, more at the unreported max) and read throughput is higher —
while the pure write path is unchanged (≤4%, within noise). The apparent write
slowdown in the mixed run is the writer no longer being subsidized by stalled
readers.

Changelog

changelog = "none"
reason    = "Internal benchmark tooling for the stress-test binary only."

[Mirko-von-Leipzig]

Oh that's interesting. Good to see our primary hypothesis being validated.

I didn't foresee the concurrency effects of the blocked readers 😁 Effectively at high traffic we should expect to only have a single core working on apply block? And perhaps not even that because we are using async so it will be multiplexed with the reader work..

We can probably improve this, with some other tools, e.g. a dedicated thread pool for writes (not good for RPC traffic though), or blocking new readers while we are writing, or some sort of task priority.

Regardless though, this looks like a massive win.

[Mirko-von-Leipzig]

Actually I wonder if we can't use a rayon thread pool with a high priority, which is only used for apply block.. That should just work iiuc.

And all our actual apply block work is sync, we've just wrapped each part in async. Concept:

Tokio / general Rayon compute
             │
             │ normal priority
             ▼
        all available CPUs
             ▲
             │ high weight while runnable
             │
Dedicated apply_block Rayon pool

The SMT backend uses rayon afaiu - is that correct? And if yes, does it only use the primitives, or does it spawn its own rayon pool?