perf(ingest): narrow ingest_lock to the manifest commit, calibrated by a new load-test harness

[explise]

Summary

Ingest saturated at ~43 MiB/s (~205k records/s) while the node used 0.84 of the 8 cores it was given. The limit was serialization, not hardware. This PR builds the harness that found that, fixes the bottleneck, and corrects a simulator constant the harness proved wrong.

Three commits, each independently reviewable:

Commit	What
f5685ad	Narrow ingest_lock to the manifest commit only
c77841f	The load-test harness + results that motivated it
9bda6f9	Recalibrate modeled_mibps_per_core 250 → 100

The bottleneck

ingest_lock was held across the whole of Ingestor::ingest — encode, compress, PUT, and manifest commit — so every request waited out the encode of the one ahead of it. Encode is the expensive half (~13.3 ms/MiB versus ~9 ms for a commit).

The signature was unmistakable: throughput flat across a 32× concurrency range while p50 latency scaled linearly with it.

concurrency	MiB/s	p50 ms
1	~43	107
32	~43	3023

The fix

Ingestor::ingest splits into write_batches (route → batch → encode → PUT) and commit_files (publish to manifest). Only the commit takes the lock.

Encoding outside the lock is safe because data files are content-addressed — two writers encoding the same records produce the same path and the same bytes, and writers encoding different records cannot collide. The manifest commit is a read-modify-write of a single object and genuinely does need mutual exclusion.

ingest calls both halves, so its behaviour is unchanged and no existing caller moves. Concurrency stays bounded by the ingest_sem permit the handler already holds, so in-flight bodies still can't pile up in memory.

Pre-existing failure mode, not introduced here: files land in the store before the manifest references them, so a commit that never happens leaves orphaned objects. That was already true of ingest; the split doesn't change it. Called out in the rustdoc.

Calibration

modeled_mibps_per_core = 250.0 was a placeholder, and ModeledExecutor divides ScanPlan::total_bytes() by it — every simulated query duration inherited the error. Measured: 892 MiB/s on a 4-column scan, 635 MiB/s on a full-projection aggregate at 8 cores, i.e. 79–111 MiB/s/core. Only a metadata-only COUNT(*), which reads no column data, ever reached 250.

Still optimistic in one direction — measured against local MinIO, so it carries no real-S3 first-byte latency. It's the CPU-side decode ceiling, documented as such.

Notes for review

• bench/loadgen is excluded from the workspace on purpose. A load generator must read the real clock, and crates/vdg/tests/seams.rs fails the build on wall-clock calls under crates/. Excluding it keeps that DST gate strict rather than adding an exemption for benchmark code, and keeps reqwest's TLS stack out of the workspace lockfile and CI.
• New [ingest] knobs (max_batch_rows / max_batch_bytes). Without them the client's request size silently decides the PUT rate, since ingest flushes leftover buffers every call — a batch-size sweep would only have been measuring the load generator. Defaults reproduce BatchPolicy::default() exactly, with a test asserting it.
• bench/results/ is committed as the evidence trail (~152K of JSON). Happy to drop it or trim to just the artifacts RESULTS.md cites if you'd rather keep it out of the tree.

Testing

• scripts/verify.sh --all rust — green.
• New crates/ingest/tests/concurrent_ingest.rs pins the two properties the narrow lock depends on: concurrent writes are lossless, and content-addressing prevents path collisions. A refactor that widens or drops the remaining lock fails there rather than in production.
• The throughput claims above come from the committed run; docs/load-test.md has the reproduction steps.

🤖 Generated with Claude Code