Implement the shared transaction-log runtime for derived indexes

[kriszyp]

Goal

Provide one Harper-owned delivery and recovery protocol for non-transactional derived indexes such as full-text and a future asynchronous HNSW plane. Primary records remain authoritative. A derived index is either proven current through its durable cursor or explicitly unavailable pending rebuild.

This issue covers the shared Harper runtime. Tantivy storage, schema declarations, query APIs, blob extraction, generation activation, and migration of the current synchronous HNSW index are separate work.

Architecture

Each registered backend has an independent runner, transaction-log iterator, cursor vector, and backpressure state.

• Any Harper worker may write records and append to the shared physical transaction logs.
• RocksDatabase.tryLock() elects one temporary runner owner per backend id across workers. There is no permanent worker-0 writer.
• A root-store committed event is only a wake-up. The elected runner reads committed transaction-log entries; the write path never waits for derived-index work.
• Different indexes use different lock keys, so their runners and native writers can make progress in parallel. One deferred or failed backend does not block another.
• The owner is sticky while work or non-durable accepted progress remains, then releases after a short idle grace period.
• Schema activation installs the same registration on every worker before that worker can accept table work or evict. Registration is worker-local; only runner ownership is elected.

This deliberately does not use the formerly proposed same-thread aftercommit delivery. That path retains audit objects on every write, makes every worker a producer into backend state, and does not provide one serial replay path after worker loss.

Cursor and replay contract

The backend durably owns this versioned cursor:

type DerivedIndexCursor = {
	format: 1;
	logs: Record<string, number>; // physical log name -> completed transaction timestamp
};

The current rocksdb-js transaction-log iterator exposes { timestamp, data, endTxn }, not physical file offsets. Harper therefore uses its existing transaction timestamp identity rather than inventing or requiring a new rocksdb-js cursor primitive.

On startup, owner handoff, or accepted-work loss, the aggregate reader exact-seeks every saved log boundary. It validates and consumes exactly one complete anchor transaction from each same iterator before returning subsequent physical entries. This closes a validation/resume race and still returns a later physical entry whose timestamp is lower than the anchor. Missing, incomplete, or duplicate boundaries require rebuild.

oldestSequenceNumber is used for diagnostics and to prove that a newly discovered log still retains its beginning. It cannot be compared with a timestamp cursor. A missing exact timestamp can require a full-log scan during rare owner reconstruction; sticky ownership keeps that cost off the steady-state path.

Harper retains every complete cursor vector offered to a backend until its durability barrier advances. A reported durable cursor must equal one whole offered vector; per-log positions from different batch boundaries cannot be combined. The backend must publish the cursor atomically with the index state covered by that barrier.

Accepted-but-not-durable progress is bounded. At the configured cap, the runner stops reading and waits for a backend state-change notification rather than repeatedly probing the backend on unrelated database commits.

Delivery contract

const DERIVED_INDEX_ACCEPTED = 1;
const DERIVED_INDEX_DEFERRED = 0;
const DERIVED_INDEX_FAILED = -1;

interface DerivedIndexBackend {
	readonly id: string;
	getDurableCursor(): DerivedIndexCursor | undefined;
	deliver(batch: DerivedIndexBatch): 1 | 0 | -1;
	onStateChange(
		wake: (change?: 'changed' | 'accepted-work-lost' | 'failed') => void
	): () => void;
}

• deliver() is synchronous and non-blocking. It may enqueue into native work but may not wait for a writer mutex, merge, or durability barrier.
• accepted means the backend owns the batch. It does not mean the cursor is durable.
• deferred retains the exact batch and stops only that runner until a backend wake.
• accepted-work-lost discards process-local offered progress and reconstructs from the durable cursor.
• failed makes that backend unavailable pending rebuild.

Drain limits apply between complete source transactions. One backend call receives each bounded batch, including cursor-only progress through unrelated transactions.

Authoritative state and projection

The log identifies records that may need revisiting. Before delivery, Harper reads current committed state directly from the primary store, never through Resource.get() and never through a caching source. Repeated references to one (tableId, recordId) in a batch share one primary read and one schema-compiled projection.

The backend receives only declared derived-index attributes. A present local record produces its current version and projection; a durable removal fact produces absence. This latest-state protocol is idempotent by primary key and safe under replay overlap.

put, patch, delete, invalidate, relocate, and local-only evict are eligible record actions. Whole-table reload, decode failures, transaction framing failures, primary read failures, and projection failures are fail-closed.

Cache eviction

Cache eviction is local residency state, not a canonical delete. For a table with a derived-index registration, direct and batched RocksDB eviction stage a bodyless, LOCAL_ONLY eviction marker in the same native transaction as the fresh-version-guarded removal. An aborted or conflicting eviction commits neither the removal nor the marker.

The derived runtime consumes this marker and resolves the authoritative row as absent. Boot replay treats it as a successful control entry without replaying or counting a record. Live subscriptions ignore it, subscription replay and customer history filter it, and the LOCAL_ONLY bit prevents replication. Raw transaction-log readers retain it for the derived runtime; the customer read_audit_log and read_transaction_log operations route through the filtered Table history methods.

Registration counts are scoped by audit-store identity and table id. Tables without a derived-index registration pay only an O(1) guard during eviction and add no transaction-log writes. LMDB derived indexes are not part of this release.

Failure behavior

The backend transitions to needs-rebuild without cursor advancement when Harper cannot prove continuity or authoritative state, including:

• missing saved log or exact boundary;
• newly discovered log whose first sequence was already purged;
• incomplete or duplicate transaction boundary;
• corrupt or undecodable log entry;
• unexpected per-log iterator failure;
• removed physical log;
• malformed, regressed, or unoffered durable cursor vector;
• primary-read or projection failure; or
• permanent backend failure.

Transaction-log retention is not pinned in this phase. A backend that falls behind retained history rebuilds rather than silently skipping work.

Current implementation unit

The feature branch now contains:

• opt-in physical log names and exact-resume failure metadata on RocksTransactionLogStore.getRange();
• the lock-elected DerivedIndexRuntime with bounded batches, backpressure, durable/offered cursor reconciliation, owner epochs, and isolated failure states;
• reference-counted table registration used only by the RocksDB eviction paths;
• atomic local-only markers for direct and batched cache eviction, including customer-history and boot-replay isolation;
• focused fake-backend coverage; and
• an end-to-end audited RocksDB table test for put, patch, delete, eviction, current-state projection, and commit wake-up.

Database/schema lifecycle registration, activation/rebuild generation management, metrics, and a real backend remain follow-on units.

Acceptance

• No derived-index work is awaited from a primary record commit.
• At most one runner owns one backend id across Harper workers; independent indexes can run in parallel.
• Restart and worker handoff resume from an exact durable boundary with no silent cursor leap.
• Deferred and accepted-but-not-durable work is retried without blocking other indexes.
• Every relevant committed mutation converges to current primary state, including replicated/source-version writes and local eviction.
• Any unprovable gap makes only the affected backend unavailable pending rebuild.
• Focused resource tests, cross-worker ownership/crash tests, replay/reopen tests, and write-path performance benchmarks pass before promotion.

Detailed design: docs/derived-index-runtime-stage-1.md on the implementation branch.

Comment generated by kAIle (GPT-5)