Skip to content

feat: move WTI structural persistence off mutation path - #50

Merged
pathscale merged 2 commits into
fix/overwrite-inplace-v2from
feat/logical-index-persistence
Aug 4, 2026
Merged

feat: move WTI structural persistence off mutation path#50
pathscale merged 2 commits into
fix/overwrite-inplace-v2from
feat/logical-index-persistence

Conversation

@pathscale

Copy link
Copy Markdown
Owner

Summary

  • Add the opt-in logical-index-persistence feature for persisted primary and unique-secondary WorkTablesIndex indexes.
  • Foreground WTI mutations emit one logical set/remove event; the existing background persistence worker translates it through a private shadow WTI into the current structural CDC page writes.
  • Preserve the existing DataBucket index format and direct WTI point-read path. Feature-off behavior is unchanged.
  • Keep the DSL contract explicit: omitted using means WorkTablesIndex; using congee and using arctic select the forked ART backends. Those backends already have native topology checkpoint/WAL support, so their crates need no change here.
  • Keep non-unique WTI secondary indexes on structural multimap CDC for this first stage.

Correctness and compatibility

  • Added concurrent same-row mutation, drain, reload, primary lookup, and unique-secondary audit coverage.
  • Added logical-event sentinel validation and event-ID gap regression coverage.
  • Existing WTI/IndexSet on-disk switching and ART reload tests pass.
  • Existing best-effort crash-durability boundary is unchanged; this does not claim a new durable WAL.

Local performance probe

Ten interleaved ARM trials, 2,000,000 existing-key mutations and 8,000,000 point reads per trial:

  • structural CDC update median: 80.916 ns/op
  • logical foreground update median: 71.440 ns/op (11.71% faster)
  • raw WTI read median: 105.096 ns/op
  • wrapped WTI read median: 103.068 ns/op (no observed regression; the apparent gain is treated as noise)

These are engineering probes, not publication numbers. Generated-table select measurements still need a quiet-window campaign before making a sub-percent claim.

Verification

  • cargo fmt --all -- --check
  • cargo check --lib
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all-features --lib --tests: 152 library and 356 integration tests passed; 4 pre-existing tests ignored

Stacking note

This branch is based on current master after PR 47 and PR 49. Against PR 48 head b76a60a, the combined tree compiles with all features and the concurrent logical-WTI reload test passes. The current cherry-pick has one mechanical src/lib.rs export-list conflict because PR 48 has not yet rebased onto current master; there is no implementation conflict with its in-place update path.

@pathscale
pathscale force-pushed the feat/logical-index-persistence branch from 18d9f52 to 1908f6b Compare August 4, 2026 23:53
@pathscale
pathscale changed the base branch from master to fix/overwrite-inplace-v2 August 4, 2026 23:53
@pathscale
pathscale force-pushed the feat/logical-index-persistence branch from 1908f6b to 788c8ee Compare August 4, 2026 23:54
@pathscale
pathscale merged commit 788c8ee into fix/overwrite-inplace-v2 Aug 4, 2026
@pathscale
pathscale deleted the feat/logical-index-persistence branch August 4, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant