Support concurrent Rocks-backed indices with bounded shared resource budgets

[kylebernhardy]

Outcome

Allow multiple full-text indexes to make useful concurrent progress without overwhelming a shared RocksDB instance.

Acceptance criteria

• Each logical index owns an independent Tantivy writer, reader, ingestion queue, and generation.
• A process-wide budget bounds total indexing workers, search workers, merge threads, memory, queued bytes, and open generations.
• Database-wide write and durability admission coordinates indexes without imposing a wrapper-owned transaction commit lane.
• One hot index cannot starve search or indexing for other indexes indefinitely.
• Write stalls and background errors apply backpressure and health transitions consistently.
• Benchmarks cover one, several, and many concurrently active indexes under foreground Harper-style traffic.

Comment generated by kAIle (Codex GPT-5)