Non-CDC index mutations silently burn CDC event ids

[pathscale]

Context

From the 2026-07-27 concurrency work (root-cause analysis behind patches 0006/0010). indexset's CDC maps assign event ids with fetch_add inside every mutation, including the non-CDC convenience methods: insert()/remove() call put_cdc/remove_cdc internally and then discard the events (see indexset src/concurrent/map.rs, e.g. insert → put_cdc(..).0; try_insert → .ok().map(|_| ())).

Consequence: any non-CDC mutation on an index that also feeds a persistence stream consumes ids without queueing events → a permanent gap in the event-id sequence. Since the strict-deferral fix (patch 0006), BatchOperation::validate never force-applies a gapped stream, so such a gap stalls persistence forever, loudly (tracing::error after 8 attempts). That is deliberate: a loud stall instead of silent positional corruption of the on-disk index. Vacuum was the one reachable case on persisted tables and was fixed in patch 0010 (VacuumPersistence sink).

What remains

• In-memory tables still use non-CDC mutations (e.g. the non-CDC reinsert in src/table/mod.rs, vacuum's sink-less path in src/table/vacuum/vacuum.rs). Harmless today — no persistence consumer exists there — but it is a trap: attaching any future consumer (replication, S3 sync, CDC subscribers) to a path that mixes in even one non-CDC mutation produces an unfixable stall by design.
• Nothing enforces the invariant "an index with a persistence consumer is only mutated through *_cdc".

Suggested direction

One or more of:

1. Documentation: state the invariant on LockMap-adjacent index types and in the persistence module docs.
2. Debug guard: wrap indexes destined for persistence in a newtype that only exposes *_cdc methods (compile-time enforcement, the strongest option).
3. Runtime tripwire: a debug_assert!/counter comparing ids consumed vs events queued per index (catches regressions in tests immediately, given the suite already runs everything through wait_for_ops with timeouts).

Verification hints

The stall symptom is deterministic: wait_for_ops hangs, tracing::error names the last applied and next available ids. tests/persistence/vacuum.rs shows the pattern for turning a stall into a 30s-timeout failure. Reverting patch 0010's one-line .with_persistence(..) wiring reproduces a genuine gap stall on demand.

---

Migrated from pathscale/wt-dead#167. Originally opened by @pathscale on 2026-07-26T21:23:37Z.

[pathscale]

Valid hardening issue rather than a currently reproduced persistent-path failure. Persisted generated mutations use CDC today, but the invariant is not encoded in the type system and non-CDC methods consume event IDs internally. Before 1.0, at minimum document and test the invariant; preferably expose a persistence-only wrapper that cannot call non-CDC mutation methods or add a deterministic event-gap tripwire.

[pathscale]

Resolved by the dependency chain consumed in PR #46. WorkTablesIndex 0.0.5 const-specializes ordinary set writes with EMIT_CDC = false; its map regression passes, and an additional multimap probe performing ordinary insert/replace/remove followed by insert_cdc observed first event id 0. The silent ID-burning mechanism described here is gone for both map and multimap paths. Recommend closing when PR #46 lands.

[pathscale]

Resolved by the WorkTablesIndex 0.0.5 dependency chain now on master through PRs 46 and 47. Ordinary map and multimap mutations no longer allocate/discard CDC event IDs, so mixing non-CDC operations no longer creates the permanent event-sequence gaps described here.