From 25664ff9bbc57411cae2a3c17b63daf5631208ae Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 30 Aug 2026 22:35:36 -0600 Subject: [PATCH 01/69] HNSW native traversal plane: design doc + Rust prototype (mmap fixed-slot file) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design (hnsw-native-plane.md): replace the index CF with a memory-mapped fixed-slot file as the primary graph store — the file IS the index, updated in place per commit with bounded-lag durability (watermark + runIndexing replay). Per-slot seqlocks, relaxed cross-slot adherence (safe under the existing exact-rescore + MVCC record load), in-file freelist (fixes #2182 structurally), bitset + pipelined-TSFN filtering, three-phase rollout (dual-write -> file-primary -> native insert). Prototype (native/hnsw-plane): compilable standalone core — format, seqlock, asymmetric int8 cosine, beam search with epoch-stamped visited array, prototype insert, bench binary. First measurement (20K x 768-d, ef 512, Linux): 0.440 us/visit vs 4.34 us JS baseline — 9.9x per-visit, scalar distance, before SIMD and zero-copy reads. Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 246 +++++++++++++++++++++++++++++ native/hnsw-plane/Cargo.toml | 22 +++ native/hnsw-plane/src/bin/bench.rs | 88 +++++++++++ native/hnsw-plane/src/distance.rs | 48 ++++++ native/hnsw-plane/src/format.rs | 181 +++++++++++++++++++++ native/hnsw-plane/src/graph.rs | 106 +++++++++++++ native/hnsw-plane/src/insert.rs | 114 +++++++++++++ native/hnsw-plane/src/lib.rs | 13 ++ native/hnsw-plane/src/search.rs | 170 ++++++++++++++++++++ native/hnsw-plane/src/seqlock.rs | 48 ++++++ 10 files changed, 1036 insertions(+) create mode 100644 hnsw-native-plane.md create mode 100644 native/hnsw-plane/Cargo.toml create mode 100644 native/hnsw-plane/src/bin/bench.rs create mode 100644 native/hnsw-plane/src/distance.rs create mode 100644 native/hnsw-plane/src/format.rs create mode 100644 native/hnsw-plane/src/graph.rs create mode 100644 native/hnsw-plane/src/insert.rs create mode 100644 native/hnsw-plane/src/lib.rs create mode 100644 native/hnsw-plane/src/search.rs create mode 100644 native/hnsw-plane/src/seqlock.rs diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md new file mode 100644 index 0000000000..249b5dfae6 --- /dev/null +++ b/hnsw-native-plane.md @@ -0,0 +1,246 @@ +# HNSW native traversal plane + +Design for moving HNSW graph storage and traversal into a native (Rust/napi-rs) module over a +memory-mapped fixed-slot file, replacing the RocksDB index column family as the home of graph +nodes. Companion to the scaling analysis in `DESIGN.md` ("efConstruction and the search-ef +ceiling both auto-scale with the graph") and issues #693, #711, #895, #2182. + +## 1. Motivation — measured, not estimated + +Per-visit cost decomposition at 5M nodes / ef 512 (768-d int8, `benchmarks/hnsw-scale.js` +corpus, 22.18 ms p50 / 5,107 visits): + +| Component | Cost | Share of a warm visit | +| ------------------------------------------------- | ------- | ---------------------- | +| Total per visited node | 4.34 µs | 100% | +| int8 asymmetric cosine, 768-d, JS | 0.43 µs | 10% | +| msgpackr decode of one node (VT-cache miss) | 5.57 µs | +128% when cold | +| Neighbour iteration + visited-set ops | 0.21 µs | 5% | + +~85% of a warm visit is JS object bookkeeping — candidate heap, visited `Set`, property access, +allocation, GC — not distance math and not I/O. Three consequences: + +1. **A native distance kernel is worth ~nothing.** Distance is 10% of the visit; a NAPI crossing + costs 0.1–0.5 µs. The win requires the whole search loop native, over a native data layout, + with one boundary crossing per query. +2. **The fetch path decides the ceiling.** A warm RocksDB `Get` is ~1–2 µs even called natively + (block-cache lookup, block parse, value memcpy) — 20–40× the SIMD distance it feeds. Direct + slot addressing (`base + id × SLOT_SIZE`) into a resident mapping is ~100–200 ns. Traversal + over RocksDB caps at ~3–5× improvement; traversal over a fixed-slot mapping reaches the + full ceiling. +3. **Estimated native budget: ~0.25–0.4 µs/visit** (SIMD int8 dot ~50 ns + streaming 768 + contiguous bytes ~150 ns + bitset/heap ops ~50 ns) → **~10–15× on the search path** + (22 ms → ~1.5–2 ms at 5M/ef 512), with the JS event loop untouched. + +This is also the enabling dependency for same-node index slicing (parallel slice searches need +off-loop execution) and changes cluster QPS arithmetic by the same factor. + +## 2. Goals / non-goals + +Goals: + +- Search traversal fully native, off the JS event loop, one NAPI crossing per query. +- Graph nodes in a memory-mapped fixed-slot file — **the file is the index**: the maintained + primary of the derived data, updated in place on every commit, not a cache of RocksDB. +- Incremental maintenance preserved: insert/update/delete keep working exactly as today from + the application's view. +- Relaxed transactional adherence (deliberate): HNSW results are approximate by contract, and + the existing post-load exact rescore + MVCC record lookup already filter stale/wrong + candidates. No cross-slot atomicity. +- Node-id reuse via an in-file freelist — structurally fixes the #2182 lifetime high-water + ef over-provisioning. +- Slicing-ready: one file per slice; native merge of per-slice top-k (C2 hook). + +Non-goals (this phase): + +- Binary quantization / Matryoshka truncation (benchmark-gated per the Reflex study; the format + reserves a quantization-mode field so a binary plane is a format v2, not a redesign). +- Native insert loop (phase 3; insert logic stays in JS initially, persisting through the + native slot-write API). +- Cross-node ANN protocol. Out of scope entirely. +- Lexical/BM25 anything. + +## 3. Architecture + +``` + JS (worker threads) native (Rust, napi-rs) + ┌─────────────────────────────────────────┐ ┌─────────────────────────────────────┐ + │ HierarchicalNavigableSmallWorld.ts │ │ hnsw-plane │ + │ • pk→nodeId mapping (stays RocksDB) │ │ • mmap'd slot file (per index/slice)│ + │ • insert/update/delete logic (phase 1) ├──►│ • slot read/write API (seqlocked) │ + │ • commit callback → slot writes │ │ • search(query, k, ef, filter) → │ + │ • record load + exact rescore (as-is) │◄──┤ top-k ids, own thread pool │ + │ • runIndexing replay from watermark │ │ • TSFN batch filter callback │ + └─────────────────────────────────────────┘ └─────────────────────────────────────┘ +``` + +What stays in RocksDB: the pk→nodeId mapping (transactional with record writes — it is the +authority on which node id a record owns), records themselves, and all other indexes. What +moves to the file: node vectors, per-layer adjacency, entry point, id allocator, freelist. + +## 4. File format (v1) + +One file per index (per slice, once C2 lands): `.hnsw`. + +**Header (4 KB page):** + +| Field | Type | Notes | +| --- | --- | --- | +| magic + format version | u32 + u32 | rebuild required on version mismatch (accepted contract) | +| dims, quantization mode | u16 + u8 | v1: int8 asymmetric; f32 supported for `quantization:"none"` | +| slot_size, layer0_cap, upper_cap | u16 ×3 | derived from M/optimizeRouting at creation | +| entry_point_id, entry_point_level | u32 + u8 | atomically updated | +| id_high_water | u64 atomic | replaces the shared Atomics BigInt64Array incrementer | +| freelist_head | u64 atomic | CAS push/pop; ABA-guarded with a 32-bit tag | +| txn_watermark | u64 | last durably indexed transaction; advanced by msync cadence | +| clean_shutdown flag | u8 | torn-state detection on open | + +**Main region — layer-0 slots**, addressed `4096 + id × slot_size`: + +| Field | Size (768-d int8, cap 64) | +| --- | --- | +| seq (seqlock) | 4 B | +| flags (valid/deleted) + level | 2 B | +| scale (f32) + invMag (f32) | 8 B | +| degree | 2 B | +| vector (int8 × 768) | 768 B | +| neighbor ids (u32 × layer0_cap) | 256 B | +| **total, padded** | **1,040 B → 1 KB-aligned 1,088 B** | + +At 100M nodes: ~109 GB (int8). A binary-code v2 slot (96 B codes + ids) is ~384 B → ~38 GB. +For comparison, today's encoding averages 1,425 B/node *plus* RocksDB overhead — so v1 is +already ~25% smaller while being fixed-offset addressable, because per-edge cached float64 +distances are dropped (recomputing a distance costs ~50 ns native; storing it costs 8 B and +~40% of today's node bytes). + +**Upper-layer region** (append-allocated, compacted on rebuild): only ~6% of nodes have +level > 0, and upper layers hold neighbor id lists only (vectors live in the main slot). Each +entry: `node_id, level, [degree, ids × upper_cap] × level`. Kept fully resident; a few hundred +MB at 100M nodes. + +**Degree cap decision.** Today layer-0 caps at `M<<1` then `<<2` under `optimizeRouting` = 128, +with transient overshoot to 160 before pruning; measured mean degree is ~37. Sizing slots at +cap 128 doubles the file for a tail. v1 policy: **hard prune-to-cap-64 on write** — the insert +path's in-memory candidate selection can overshoot as today, but what is written is pruned to +64 by the same routing-aware selection that currently prunes at 160→128. Transient overshoot +never touches the file. Recall impact must be measured in the validation phase (§9); the cap is +a header field, so revising it is a rebuild, not a format change. + +## 5. Concurrency + +- **Per-slot seqlock.** Writer: fetch_add seq to odd → write slot → fetch_add to even. Reader + (traversal): read seq, copy the ≤1 KB slot (or read fields in place), re-check seq; retry on + change. Retries are rare (writes touch ~40 slots per insert out of millions) and cheap. +- **No cross-slot atomicity.** An insert updates the new node's slot plus ~M neighbors' + back-edge lists, each independently. A traversal may observe the half-linked state: an edge + to a slot whose valid flag is not yet set → skip (HNSW tolerates missing edges); a + just-deleted neighbor → skip via flags. Wrong-candidate leakage is filtered by the existing + exact rescore + MVCC record load, which is why relaxed adherence is safe *here* and not a + general storage pattern. +- **Writers.** Multiple worker threads insert concurrently today (distinct records); the same + holds: id allocation is one atomic fetch_add on the header, freelist pop is CAS, slot writes + are seqlocked. Two inserts updating the same neighbor's edge list serialize on that slot's + seqlock (a Rust-side per-slot spinlock on the odd state). +- **Id reuse & ABA.** Delete pushes the id onto the freelist; a traversal holding the old id may + read the reused slot and score the wrong vector — acceptable under the relaxed contract + (rescore/record-load rejects it). The freelist head itself is tag-guarded against ABA. + +## 6. Durability & crash recovery + +The file is `msync`'d on a cadence (default: every N seconds or M mutated slots, configurable), +**not** per commit. The header watermark records the last transaction whose index mutations are +known durable; it advances only after a completed msync barrier. + +On open: + +- Clean-shutdown flag set → map and serve. +- Torn state → replay records from `txn_watermark` through the existing `runIndexing` re-feed + path (which already treats a re-fed already-indexed record as an update — the exact semantics + needed). This anchors today's heuristic crash re-feed to a precise watermark. +- Format-version mismatch or corruption (header checksum) → full rebuild from records. Explicit + contract: **format upgrades require reindex** (accepted). + +Note the asymmetry with today: RocksDB gave the graph per-commit durability; the file gives it +bounded-lag durability with deterministic catch-up. For an approximate index whose source of +truth (records + pk→nodeId) remains fully transactional, bounded lag is the right trade — it +buys the entire performance model. + +**Backup/copy-db/reseed:** the file is node-local derived state. Backup either includes it +(consistent-enough after an msync barrier) or marks the index rebuild-on-restore. Replica +reseed = rebuild from records (C5 bulk construction makes this fast; until then, the existing +per-row path). + +## 7. Search path & NAPI surface + +```ts +// one crossing per query; executes on the module's own thread pool +search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, distances}> +``` + +- Asymmetric distance as today: float query × int8 stored, cached invMag, SIMD (AVX2/VNNI on + x86, NEON on ARM; `std::arch` intrinsics with a scalar fallback). +- Visited set: epoch-stamped u32 array (one per pool thread, reused across queries — no + allocation per query). Candidate heap: fixed-capacity binary heap of (dist, id) pairs. +- Auto-ef / auto-efC read the node count from the header high-water minus freelist length — + same semantics as today, minus the #2182 inflation (freed ids return to the pool). + +**Filtering** (predicate-aware / ACORN, `filteredSearch = true` today): + +1. **Bitset fast path.** RBAC allow-lists and companion-condition candidate sets are computed + before the query and passed as a roaring/plain bitset over node ids. Zero callbacks. This + covers the dominant production filter shapes. +2. **Pipelined TSFN batch path** for arbitrary JS predicates. Traversal batches candidate ids + (64–256) through a ThreadsafeFunction to a JS evaluator and **continues expanding in + distance order while verdicts are in flight**; verdicts merge in to steer selection and + gate results. The existing `filterExpansion` visit budget bounds speculative overshoot. + Traversal never blocks on the event loop — that would re-import the p99 problem this + design exists to remove. Worst case (loop saturated): budget exhausts, return what passed — + the same contract as today's budget-bound filtered search. +3. TSFN lifecycle: shutdown-while-query-in-flight is a first-class test (see rocksdb-js #665's + TSFN teardown SIGSEGV). napi-rs `ThreadsafeFunction` + explicit abort on env teardown. + +## 8. Write path phasing + +- **Phase 1 — dual-write, search cutover.** Insert/update/delete logic stays in JS + (`HierarchicalNavigableSmallWorld.ts` unchanged algorithmically); mutations persist to BOTH + the index CF (as today) and the file via native slot-write calls. Search runs native from the + file. Validation = compare native results against the JS path on the same graph; rollback = + flip search back to JS, drop the file. The double-write cost is bounded (index writes are + a fraction of insert cost) and temporary. +- **Phase 2 — file-primary.** Drop the CF writes; the file is the only graph store. JS insert + reads nodes through a native `getNode(id)` (one NAPI crossing per read, ~1 µs — comparable to + today's decode path). Migration for existing indexes: reindex (accepted contract), or a + one-shot CF→file bulk conversion since it is a pure format transform. +- **Phase 3 — native insert.** Move the insert search + neighbor selection native (same + traversal core), leaving JS a thin `index(pk, vector)` call. Unlocks bulk build (C5) at + native speed and removes the ~tens-of-ms event-loop pin per insert (#895). + +## 9. Validation plan + +Baselines exist in `benchmarks/hnsw-scale.js` output (1M/2M/5M anchors, e.g. 1M efC-200: +p50 7.2 ms / recall@10-set 0.997 @ ef 512). Acceptance for phase 1: + +1. **Parity:** native search over a dual-written graph returns identical candidate sets to the + JS path at equal ef (modulo seqlock-retry races under concurrent write load — measured as a + bounded divergence rate, not exact equality under churn). +2. **Recall:** cap-64 prune vs cap-128 measured at 1M and 5M; accept if recall@10 delta ≤ 0.5 pt + at equal ef, else revisit the cap (header field — rebuild, not redesign). +3. **Latency:** ≥8× p50 improvement at 5M/ef 512 (22.2 ms → ≤2.8 ms), p99 within 2× p50 under + concurrent insert load (the metric that motivates off-loop execution). +4. **Crash:** kill -9 during sustained ingest → reopen → watermark replay → graph passes + connectivity + recall checks (extend the #1712 repair test harness). +5. **Churn:** delete/reinsert cycles hold node count stable (freelist reuse; #2182 regression + test). + +## 10. Open questions + +- **Degree cap 64 vs 128** — measured decision, §9.2. +- **macOS/Windows sparse-file + mmap growth semantics** — Linux is the production target; + dev platforms may need chunked remapping instead of one large sparse reservation. +- **msync cadence default** — bounded-lag durability window vs write amplification; needs a + workload measurement, not a guess. +- **Where the crate lives** — in-repo `native/hnsw-plane/` (tight iteration, CI builds Rust) + vs separate repo à la symphony (prebuilds, independent versioning). Prototype in-repo; + decide before merge. +- **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the + format (dims × mode in header) but int8 is the default and the optimization target. diff --git a/native/hnsw-plane/Cargo.toml b/native/hnsw-plane/Cargo.toml new file mode 100644 index 0000000000..e0177bc169 --- /dev/null +++ b/native/hnsw-plane/Cargo.toml @@ -0,0 +1,22 @@ +[package] +name = "hnsw-plane" +version = "0.0.1" +edition = "2021" +description = "Native HNSW traversal plane: mmap fixed-slot graph file + off-loop search" +license = "MIT" + +[dependencies] +memmap2 = "0.9" + +[features] +default = [] +# napi bindings added in phase 1 integration; core stays buildable standalone +# napi = ["dep:napi", "dep:napi-derive"] + +[[bin]] +name = "bench" +path = "src/bin/bench.rs" + +[profile.release] +lto = true +codegen-units = 1 diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs new file mode 100644 index 0000000000..c5a429e48e --- /dev/null +++ b/native/hnsw-plane/src/bin/bench.rs @@ -0,0 +1,88 @@ +//! Standalone cost benchmark: build an N-node graph in the plane file, run queries, report +//! per-visit cost — the number that decides whether the native plane hits its 0.25–0.4 µs +//! budget (JS baseline: 4.34 µs/visit at 5M/ef 512). +//! +//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] + +use hnsw_plane::distance::Query; +use hnsw_plane::insert::{insert, InsertParams}; +use hnsw_plane::search::{search, SearchScratch}; +use hnsw_plane::{Graph, PlaneFile}; +use std::path::PathBuf; +use std::time::Instant; + +// xorshift for reproducible synthetic vectors without a rand dependency +struct Rng(u64); +impl Rng { + fn next_f32(&mut self) -> f32 { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 >> 40) as f32 / (1u64 << 24) as f32 - 0.5 + } + fn vector(&mut self, dims: usize) -> Vec { + (0..dims).map(|_| self.next_f32()).collect() + } +} + +fn main() { + let args: Vec = std::env::args().collect(); + let n: u64 = args.get(1).and_then(|a| a.parse().ok()).unwrap_or(100_000); + let dims: usize = args.get(2).and_then(|a| a.parse().ok()).unwrap_or(768); + let queries: usize = args.get(3).and_then(|a| a.parse().ok()).unwrap_or(200); + let ef: usize = args.get(4).and_then(|a| a.parse().ok()).unwrap_or(512); + let path: PathBuf = args.get(5).map(Into::into).unwrap_or_else(|| "/tmp/bench.hnsw".into()); + + let layer0_cap = 64; + let file = PlaneFile::create(&path, dims, layer0_cap, n + 1024).expect("create"); + println!( + "plane: {} nodes x {} dims, slot {} B, file {:.1} GB (sparse)", + n, + dims, + file.slot_size, + (n * file.slot_size as u64) as f64 / 1e9 + ); + let graph = Graph::new(file); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + let mut rng = Rng(0x1234_5678_9abc_def0); + + let build_start = Instant::now(); + for i in 0..n { + let v = rng.vector(dims); + insert(&graph, &v, ¶ms, &mut scratch); + if (i + 1) % 50_000 == 0 { + let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); + println!(" built {} ({:.0} inserts/s)", i + 1, rate); + } + } + let build = build_start.elapsed(); + println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); + + // Query with held-out vectors. + let mut latencies = Vec::with_capacity(queries); + let mut total_visits = 0u64; + for _ in 0..queries { + let q = Query::new(rng.vector(dims)); + let start = Instant::now(); + let (results, stats) = search(&graph, &q, 10, ef, &mut scratch); + latencies.push(start.elapsed()); + total_visits += stats.visits; + assert!(!results.is_empty()); + } + latencies.sort(); + let p50 = latencies[queries / 2]; + let p95 = latencies[queries * 95 / 100]; + let p99 = latencies[(queries * 99 / 100).min(queries - 1)]; + let mean_visits = total_visits as f64 / queries as f64; + let us_per_visit = p50.as_micros() as f64 / mean_visits; + println!( + "search (ef {}): p50 {:.2} ms p95 {:.2} ms p99 {:.2} ms visits/query {:.0} -> {:.3} us/visit (JS baseline 4.34)", + ef, + p50.as_secs_f64() * 1e3, + p95.as_secs_f64() * 1e3, + p99.as_secs_f64() * 1e3, + mean_visits, + us_per_visit + ); +} diff --git a/native/hnsw-plane/src/distance.rs b/native/hnsw-plane/src/distance.rs new file mode 100644 index 0000000000..8ef4db2784 --- /dev/null +++ b/native/hnsw-plane/src/distance.rs @@ -0,0 +1,48 @@ +//! Asymmetric distance: full-precision f32 query × int8-stored vector, matching the JS +//! implementation (quantizeInt8 scale + cached 1/|v|). Written as autovectorizable loops; +//! explicit AVX2/NEON intrinsics are a measured follow-up if codegen disappoints. + +/// Precomputed query state, built once per search. +pub struct Query { + pub vector: Vec, + pub inv_mag: f32, +} + +impl Query { + pub fn new(vector: Vec) -> Self { + let mag_sq: f32 = vector.iter().map(|v| v * v).sum(); + let inv_mag = 1.0 / mag_sq.sqrt().max(f32::MIN_POSITIVE); + Query { vector, inv_mag } + } +} + +/// Cosine distance against an int8-quantized stored vector. +/// stored dot = scale * Σ q[i] * v[i]; distance = 1 - dot * inv_mag_stored * inv_mag_query. +#[inline] +pub fn cosine_int8(query: &Query, stored: &[i8], scale: f32, stored_inv_mag: f32) -> f32 { + debug_assert_eq!(query.vector.len(), stored.len()); + let mut acc = [0.0f32; 8]; + let chunks = stored.len() / 8; + for c in 0..chunks { + let base = c * 8; + for lane in 0..8 { + acc[lane] += query.vector[base + lane] * stored[base + lane] as f32; + } + } + let mut dot: f32 = acc.iter().sum(); + for i in chunks * 8..stored.len() { + dot += query.vector[i] * stored[i] as f32; + } + 1.0 - dot * scale * stored_inv_mag * query.inv_mag +} + +/// Symmetric int8 quantization matching the JS quantizeInt8: scale maps max |component| to 127. +pub fn quantize_int8(vector: &[f32]) -> (Vec, f32, f32) { + let max_abs = vector.iter().fold(0.0f32, |m, v| m.max(v.abs())); + let scale = if max_abs == 0.0 { 1.0 } else { max_abs / 127.0 }; + let inv_scale = 1.0 / scale; + let bytes: Vec = vector.iter().map(|v| (v * inv_scale).round().clamp(-127.0, 127.0) as i8).collect(); + let mag_sq: f32 = vector.iter().map(|v| v * v).sum(); + let inv_mag = 1.0 / mag_sq.sqrt().max(f32::MIN_POSITIVE); + (bytes, scale, inv_mag) +} diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs new file mode 100644 index 0000000000..ec241fa0a5 --- /dev/null +++ b/native/hnsw-plane/src/format.rs @@ -0,0 +1,181 @@ +//! On-disk format: 4 KB header + fixed-size layer-0 slot array + upper-layer region. +//! See ../../../hnsw-native-plane.md §4. Format changes bump VERSION and require reindex. + +use memmap2::MmapMut; +use std::fs::OpenOptions; +use std::io; +use std::path::Path; +use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; + +pub const MAGIC: u32 = 0x484e_5357; // "HNSW" +pub const VERSION: u32 = 1; +pub const HEADER_SIZE: usize = 4096; + +// Header field byte offsets. +const H_MAGIC: usize = 0; +const H_VERSION: usize = 4; +const H_DIMS: usize = 8; // u16 +const H_QUANT: usize = 10; // u8: 0 = int8, 1 = f32 +const H_LAYER0_CAP: usize = 12; // u16 +const H_SLOT_SIZE: usize = 16; // u32 +const H_ENTRY_ID: usize = 24; // u32 (u32::MAX = none) +const H_ENTRY_LEVEL: usize = 28; // u32 +const H_ID_HIGH_WATER: usize = 32; // u64 atomic +const H_FREELIST_HEAD: usize = 40; // u64 atomic: (tag << 32) | id; id u32::MAX = empty +const H_TXN_WATERMARK: usize = 48; // u64 +const H_CLEAN_SHUTDOWN: usize = 56; // u8 + +// Slot layout offsets (within a slot). +pub const S_SEQ: usize = 0; // u32 seqlock +pub const S_FLAGS: usize = 4; // u8: bit0 = valid, bit1 = deleted +pub const S_LEVEL: usize = 5; // u8 +pub const S_DEGREE: usize = 6; // u16 +pub const S_SCALE: usize = 8; // f32 +pub const S_INV_MAG: usize = 12; // f32 +pub const S_VECTOR: usize = 16; // dims bytes (int8) or dims*4 (f32) + // neighbors: u32 * layer0_cap, follows vector + // deleted slots reuse the first neighbor word as freelist next-pointer + +pub const FLAG_VALID: u8 = 1; +pub const FLAG_DELETED: u8 = 2; +pub const NO_ID: u32 = u32::MAX; + +pub struct PlaneFile { + pub map: MmapMut, + pub dims: usize, + pub layer0_cap: usize, + pub slot_size: usize, +} + +fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { + let raw = S_VECTOR + dims + layer0_cap * 4; + raw.next_multiple_of(64) // cache-line align +} + +impl PlaneFile { + /// Create a new plane file with capacity for `max_nodes` (sparse; pages materialize on write). + pub fn create(path: &Path, dims: usize, layer0_cap: usize, max_nodes: u64) -> io::Result { + let slot_size = slot_size_for(dims, layer0_cap); + let len = HEADER_SIZE as u64 + max_nodes * slot_size as u64; + let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; + file.set_len(len)?; + let mut map = unsafe { MmapMut::map_mut(&file)? }; + map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); + map[H_VERSION..H_VERSION + 4].copy_from_slice(&VERSION.to_le_bytes()); + map[H_DIMS..H_DIMS + 2].copy_from_slice(&(dims as u16).to_le_bytes()); + map[H_QUANT] = 0; + map[H_LAYER0_CAP..H_LAYER0_CAP + 2].copy_from_slice(&(layer0_cap as u16).to_le_bytes()); + map[H_SLOT_SIZE..H_SLOT_SIZE + 4].copy_from_slice(&(slot_size as u32).to_le_bytes()); + map[H_ENTRY_ID..H_ENTRY_ID + 4].copy_from_slice(&NO_ID.to_le_bytes()); + map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] + .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); + Ok(PlaneFile { map, dims, layer0_cap, slot_size }) + } + + pub fn open(path: &Path) -> io::Result { + let file = OpenOptions::new().read(true).write(true).open(path)?; + let map = unsafe { MmapMut::map_mut(&file)? }; + let magic = u32::from_le_bytes(map[H_MAGIC..H_MAGIC + 4].try_into().unwrap()); + let version = u32::from_le_bytes(map[H_VERSION..H_VERSION + 4].try_into().unwrap()); + if magic != MAGIC || version != VERSION { + return Err(io::Error::new(io::ErrorKind::InvalidData, "format mismatch: reindex required")); + } + let dims = u16::from_le_bytes(map[H_DIMS..H_DIMS + 2].try_into().unwrap()) as usize; + let layer0_cap = u16::from_le_bytes(map[H_LAYER0_CAP..H_LAYER0_CAP + 2].try_into().unwrap()) as usize; + let slot_size = u32::from_le_bytes(map[H_SLOT_SIZE..H_SLOT_SIZE + 4].try_into().unwrap()) as usize; + Ok(PlaneFile { map, dims, layer0_cap, slot_size }) + } + + #[inline] + pub fn slot_ptr(&self, id: u32) -> *const u8 { + unsafe { self.map.as_ptr().add(HEADER_SIZE + id as usize * self.slot_size) } + } + + #[inline] + pub fn slot_ptr_mut(&self, id: u32) -> *mut u8 { + // Mutation through a shared map: all mutable slot access is mediated by the seqlock + // (seqlock.rs) and atomics; the mmap itself is plain memory. + self.slot_ptr(id) as *mut u8 + } + + #[inline] + fn header_atomic_u64(&self, offset: usize) -> &AtomicU64 { + unsafe { &*(self.map.as_ptr().add(offset) as *const AtomicU64) } + } + + #[inline] + pub fn seq_atomic(&self, id: u32) -> &AtomicU32 { + unsafe { &*(self.slot_ptr(id).add(S_SEQ) as *const AtomicU32) } + } + + /// Allocate a node id: pop the freelist, else bump the high-water. + pub fn allocate_id(&self) -> u32 { + let head = self.header_atomic_u64(H_FREELIST_HEAD); + loop { + let cur = head.load(Ordering::Acquire); + let id = (cur & 0xffff_ffff) as u32; + if id == NO_ID { + let hw = self.header_atomic_u64(H_ID_HIGH_WATER); + return hw.fetch_add(1, Ordering::AcqRel) as u32; + } + // next-pointer lives in the dead slot's first neighbor word + let next = unsafe { + (*(self.slot_ptr(id).add(S_VECTOR + self.dims) as *const AtomicU32)).load(Ordering::Acquire) + }; + let tag = (cur >> 32).wrapping_add(1); + let new = (next as u64) | (tag << 32); + if head.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire).is_ok() { + return id; + } + } + } + + /// Return a deleted node's id to the freelist. Caller must have already marked the slot + /// deleted (under its seqlock) so concurrent traversals skip it. + pub fn free_id(&self, id: u32) { + let head = self.header_atomic_u64(H_FREELIST_HEAD); + let next_word = unsafe { &*(self.slot_ptr(id).add(S_VECTOR + self.dims) as *const AtomicU32) }; + loop { + let cur = head.load(Ordering::Acquire); + next_word.store((cur & 0xffff_ffff) as u32, Ordering::Release); + let tag = (cur >> 32).wrapping_add(1); + let new = (id as u64) | (tag << 32); + if head.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire).is_ok() { + return; + } + } + } + + pub fn id_high_water(&self) -> u64 { + self.header_atomic_u64(H_ID_HIGH_WATER).load(Ordering::Acquire) + } + + pub fn entry_point(&self) -> (u32, u32) { + let id = u32::from_le_bytes(self.map[H_ENTRY_ID..H_ENTRY_ID + 4].try_into().unwrap()); + let level = u32::from_le_bytes(self.map[H_ENTRY_LEVEL..H_ENTRY_LEVEL + 4].try_into().unwrap()); + (id, level) + } + + pub fn set_entry_point(&self, id: u32, level: u32) { + unsafe { + (*(self.map.as_ptr().add(H_ENTRY_ID) as *const AtomicU32)).store(id, Ordering::Release); + (*(self.map.as_ptr().add(H_ENTRY_LEVEL) as *const AtomicU32)).store(level, Ordering::Release); + } + } + + pub fn set_watermark(&self, txn: u64) { + self.header_atomic_u64(H_TXN_WATERMARK).store(txn, Ordering::Release); + } + + pub fn watermark(&self) -> u64 { + self.header_atomic_u64(H_TXN_WATERMARK).load(Ordering::Acquire) + } + + pub fn set_clean_shutdown(&mut self, clean: bool) { + self.map[H_CLEAN_SHUTDOWN] = clean as u8; + } + + pub fn msync(&self) -> io::Result<()> { + self.map.flush() + } +} diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs new file mode 100644 index 0000000000..c417f64104 --- /dev/null +++ b/native/hnsw-plane/src/graph.rs @@ -0,0 +1,106 @@ +//! Slot-level node access over the plane file, mediated by the seqlock, plus the resident +//! upper-layer structure. Prototype status: upper layers live in memory and are rebuilt at +//! open by scanning slots; the append-allocated file region from the design doc is a TODO. + +use crate::format::{PlaneFile, FLAG_DELETED, FLAG_VALID, S_DEGREE, S_FLAGS, S_INV_MAG, S_LEVEL, S_SCALE, S_VECTOR}; +use crate::seqlock; +use std::collections::HashMap; +use std::sync::RwLock; + +pub struct Graph { + pub file: PlaneFile, + /// Upper-layer adjacency: node id -> [neighbors at level 1, level 2, ...]. ~6% of nodes. + pub upper: RwLock>>>, +} + +/// A consistent copy of one node's traversal-relevant data. +pub struct NodeRead { + pub valid: bool, + pub level: u8, + pub scale: f32, + pub inv_mag: f32, + pub vector: Vec, + pub neighbors: Vec, +} + +impl Graph { + pub fn new(file: PlaneFile) -> Self { + Graph { file, upper: RwLock::new(HashMap::new()) } + } + + /// Seqlock-consistent read of a slot. Returns None for never-written or deleted slots. + pub fn read_node(&self, id: u32) -> Option { + if (id as u64) >= self.file.id_high_water() { + return None; + } + let seq = self.file.seq_atomic(id); + let dims = self.file.dims; + let cap = self.file.layer0_cap; + let node = seqlock::read_consistent(seq, || { + let p = self.file.slot_ptr(id); + unsafe { + let flags = *p.add(S_FLAGS); + if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { + return None; + } + let level = *p.add(S_LEVEL); + let degree = u16::from_le(*(p.add(S_DEGREE) as *const u16)) as usize; + let scale = f32::from_le_bytes(std::slice::from_raw_parts(p.add(S_SCALE), 4).try_into().unwrap()); + let inv_mag = f32::from_le_bytes(std::slice::from_raw_parts(p.add(S_INV_MAG), 4).try_into().unwrap()); + let vector = std::slice::from_raw_parts(p.add(S_VECTOR) as *const i8, dims).to_vec(); + let nbytes = std::slice::from_raw_parts(p.add(S_VECTOR + dims), degree.min(cap) * 4); + let neighbors = nbytes.chunks_exact(4).map(|c| u32::from_le_bytes(c.try_into().unwrap())).collect(); + Some(NodeRead { valid: true, level, scale, inv_mag, vector, neighbors }) + } + }); + node + } + + /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the caller. + pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32]) { + debug_assert!(neighbors.len() <= self.file.layer0_cap); + debug_assert_eq!(vector.len(), self.file.dims); + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq); + let p = self.file.slot_ptr_mut(id); + let dims = self.file.dims; + unsafe { + *p.add(S_LEVEL) = level; + (p.add(S_DEGREE) as *mut u16).write((neighbors.len() as u16).to_le()); + std::ptr::copy_nonoverlapping(scale.to_le_bytes().as_ptr(), p.add(S_SCALE), 4); + std::ptr::copy_nonoverlapping(inv_mag.to_le_bytes().as_ptr(), p.add(S_INV_MAG), 4); + std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); + for (i, n) in neighbors.iter().enumerate() { + (p.add(S_VECTOR + dims + i * 4) as *mut u32).write(n.to_le()); + } + // valid last within the locked section; the seqlock release publishes it + *p.add(S_FLAGS) = FLAG_VALID; + } + } + + /// Replace only the neighbor list (back-edge maintenance path). + pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { + debug_assert!(neighbors.len() <= self.file.layer0_cap); + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq); + let p = self.file.slot_ptr_mut(id); + let dims = self.file.dims; + unsafe { + (p.add(S_DEGREE) as *mut u16).write((neighbors.len() as u16).to_le()); + for (i, n) in neighbors.iter().enumerate() { + (p.add(S_VECTOR + dims + i * 4) as *mut u32).write(n.to_le()); + } + } + } + + /// Mark deleted (traversals skip it) and return the id to the freelist. + pub fn delete_node(&self, id: u32) { + { + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq); + unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; + } + self.upper.write().unwrap().remove(&id); + self.file.free_id(id); + } +} diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs new file mode 100644 index 0000000000..c1c334f2ed --- /dev/null +++ b/native/hnsw-plane/src/insert.rs @@ -0,0 +1,114 @@ +//! Prototype insert: enough HNSW construction to build benchmark graphs. Neighbor selection +//! is plain closest-M (the JS optimizeRouting-aware selection is the parity target for the +//! phase-3 native insert; for per-visit cost measurement this suffices). + +use crate::distance::{cosine_int8, quantize_int8, Query}; +use crate::format::NO_ID; +use crate::graph::Graph; +use crate::search::{search, SearchScratch}; + +pub struct InsertParams { + pub m: usize, // upper-layer connections + pub ef_construction: usize, // candidate list size + pub ml: f64, // level normalization: 1 / ln(M) +} + +impl Default for InsertParams { + fn default() -> Self { + InsertParams { m: 16, ef_construction: 200, ml: 1.0 / (16f64).ln() } + } +} + +/// Deterministic pseudo-random level from the node id (parity with Math.random is not needed +/// for benchmarks; a hash keeps runs reproducible). +fn level_for(id: u32, ml: f64) -> u8 { + let mut x = (id as u64).wrapping_mul(0x9e37_79b9_7f4a_7c15).wrapping_add(0x2545_f491_4f6c_dd1d); + x ^= x >> 33; + let unit = (x as f64) / (u64::MAX as f64); + let level = (-unit.max(f64::MIN_POSITIVE).ln() * ml).floor(); + level.min(31.0) as u8 +} + +pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mut SearchScratch) -> u32 { + let (bytes, scale, inv_mag) = quantize_int8(vector); + let id = graph.file.allocate_id(); + let level = level_for(id, params.ml); + let query = Query::new(vector.to_vec()); + + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + graph.write_node(id, level, &bytes, scale, inv_mag, &[]); + if level > 0 { + graph.upper.write().unwrap().insert(id, vec![Vec::new(); level as usize]); + } + graph.file.set_entry_point(id, level as u32); + return id; + } + + // Candidate discovery at layer 0. + let (candidates, _) = search(graph, &query, params.ef_construction, params.ef_construction, scratch); + let layer0_cap = graph.file.layer0_cap; + let m0 = (params.m * 2).min(layer0_cap); + let neighbors: Vec = candidates.iter().take(m0).map(|(nid, _)| *nid).collect(); + + graph.write_node(id, level, &bytes, scale, inv_mag, &neighbors); + + // Reverse edges at layer 0, pruning the neighbor's list to cap by distance if full. + for &nid in &neighbors { + if let Some(n) = graph.read_node(nid) { + let mut list = n.neighbors.clone(); + if list.contains(&id) { + continue; + } + list.push(id); + if list.len() > layer0_cap { + let nq = Query::new(n.vector.iter().map(|v| *v as f32 * n.scale).collect()); + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| { + graph.read_node(cand).map(|c| (cand, cosine_int8(&nq, &c.vector, c.scale, c.inv_mag))) + }) + .collect(); + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + list = scored.into_iter().take(layer0_cap).map(|(cand, _)| cand).collect(); + } + graph.write_neighbors(nid, &list); + } + } + + // Upper layers: link into per-level lists (prototype: closest-M from the candidate set). + if level > 0 { + let mut levels: Vec> = Vec::with_capacity(level as usize); + let upper = graph.upper.read().unwrap(); + for l in 1..=level { + let peers: Vec = candidates + .iter() + .filter(|(nid, _)| { + graph.read_node(*nid).map(|n| n.level >= l).unwrap_or(false) && upper.contains_key(nid) + }) + .take(params.m) + .map(|(nid, _)| *nid) + .collect(); + levels.push(peers); + } + drop(upper); + let mut upper = graph.upper.write().unwrap(); + for (l, peers) in levels.iter().enumerate() { + for &peer in peers { + if let Some(peer_levels) = upper.get_mut(&peer) { + if let Some(peer_list) = peer_levels.get_mut(l) { + if !peer_list.contains(&id) && peer_list.len() < params.m * 2 { + peer_list.push(id); + } + } + } + } + } + upper.insert(id, levels); + } + + if (level as u32) > entry_level { + graph.file.set_entry_point(id, level as u32); + } + id +} diff --git a/native/hnsw-plane/src/lib.rs b/native/hnsw-plane/src/lib.rs new file mode 100644 index 0000000000..23ec07930c --- /dev/null +++ b/native/hnsw-plane/src/lib.rs @@ -0,0 +1,13 @@ +//! hnsw-plane: native HNSW traversal plane over a memory-mapped fixed-slot file. +//! Design: ../../hnsw-native-plane.md. NAPI bindings land behind the `napi` feature in +//! phase-1 integration; the core is buildable and benchmarkable standalone. + +pub mod distance; +pub mod format; +pub mod graph; +pub mod insert; +pub mod search; +pub mod seqlock; + +pub use format::PlaneFile; +pub use graph::Graph; diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs new file mode 100644 index 0000000000..b213df5f4e --- /dev/null +++ b/native/hnsw-plane/src/search.rs @@ -0,0 +1,170 @@ +//! Beam search over the plane. Visited tracking is an epoch-stamped array (no per-query +//! allocation once warmed); candidate/result sets are simple binary heaps. + +use crate::distance::{cosine_int8, Query}; +use crate::format::NO_ID; +use crate::graph::Graph; +use std::cmp::Ordering as CmpOrdering; +use std::collections::BinaryHeap; + +#[derive(PartialEq)] +struct Candidate { + distance: f32, + id: u32, +} +impl Eq for Candidate {} +impl Ord for Candidate { + fn cmp(&self, other: &Self) -> CmpOrdering { + // min-heap by distance via reverse + other.distance.partial_cmp(&self.distance).unwrap_or(CmpOrdering::Equal) + } +} +impl PartialOrd for Candidate { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +#[derive(PartialEq)] +struct Result_ { + distance: f32, + id: u32, +} +impl Eq for Result_ {} +impl Ord for Result_ { + fn cmp(&self, other: &Self) -> CmpOrdering { + // max-heap by distance (worst result on top for eviction) + self.distance.partial_cmp(&other.distance).unwrap_or(CmpOrdering::Equal) + } +} +impl PartialOrd for Result_ { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +/// Reusable per-thread search scratch: epoch-stamped visited array. +pub struct SearchScratch { + visited: Vec, + epoch: u32, +} + +impl SearchScratch { + pub fn new() -> Self { + SearchScratch { visited: Vec::new(), epoch: 0 } + } + + fn begin(&mut self, capacity: u64) { + if self.visited.len() < capacity as usize { + self.visited.resize(capacity as usize, 0); + } + self.epoch = self.epoch.wrapping_add(1); + if self.epoch == 0 { + self.visited.fill(0); + self.epoch = 1; + } + } + + #[inline] + fn visit(&mut self, id: u32) -> bool { + let slot = &mut self.visited[id as usize]; + if *slot == self.epoch { + false + } else { + *slot = self.epoch; + true + } + } +} + +pub struct SearchStats { + pub visits: u64, +} + +/// Full search: greedy descent through upper layers, then beam at layer 0. +pub fn search( + graph: &Graph, + query: &Query, + k: usize, + ef: usize, + scratch: &mut SearchScratch, +) -> (Vec<(u32, f32)>, SearchStats) { + let mut stats = SearchStats { visits: 0 }; + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + return (Vec::new(), stats); + } + scratch.begin(graph.file.id_high_water()); + + // Greedy descent: single-candidate walk from the top level down to level 1. + let mut current = entry_id; + let mut current_dist = match graph.read_node(current) { + Some(n) => { + stats.visits += 1; + cosine_int8(query, &n.vector, n.scale, n.inv_mag) + } + None => return (Vec::new(), stats), + }; + let upper = graph.upper.read().unwrap(); + for level in (1..=entry_level).rev() { + let mut improved = true; + while improved { + improved = false; + let neighbors = upper + .get(¤t) + .and_then(|levels| levels.get(level as usize - 1)) + .cloned() + .unwrap_or_default(); + for nid in neighbors { + if let Some(n) = graph.read_node(nid) { + stats.visits += 1; + let d = cosine_int8(query, &n.vector, n.scale, n.inv_mag); + if d < current_dist { + current = nid; + current_dist = d; + improved = true; + } + } + } + } + } + drop(upper); + + // Layer-0 beam. + let mut candidates = BinaryHeap::new(); + let mut results: BinaryHeap = BinaryHeap::new(); + scratch.visit(current); + candidates.push(Candidate { distance: current_dist, id: current }); + results.push(Result_ { distance: current_dist, id: current }); + + while let Some(c) = candidates.pop() { + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() >= ef && c.distance > worst { + break; + } + if let Some(node) = graph.read_node(c.id) { + for nid in node.neighbors { + if !scratch.visit(nid) { + continue; + } + if let Some(n) = graph.read_node(nid) { + stats.visits += 1; + let d = cosine_int8(query, &n.vector, n.scale, n.inv_mag); + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() < ef || d < worst { + candidates.push(Candidate { distance: d, id: nid }); + results.push(Result_ { distance: d, id: nid }); + if results.len() > ef { + results.pop(); + } + } + } + } + } + } + + let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); + out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + out.truncate(k); + (out, stats) +} diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs new file mode 100644 index 0000000000..ce7771c18d --- /dev/null +++ b/native/hnsw-plane/src/seqlock.rs @@ -0,0 +1,48 @@ +//! Per-slot seqlock. Writer: bump seq to odd → mutate → bump to even. +//! Reader: snapshot seq (spin past odd), read, re-check. No cross-slot atomicity by design — +//! traversal tolerates torn *graphs* (skipped edges), but never torn *slots*. + +use std::sync::atomic::{AtomicU32, Ordering}; + +pub struct SeqWriteGuard<'a> { + seq: &'a AtomicU32, +} + +/// Acquire write ownership of a slot, spinning while another writer holds it odd. +pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { + loop { + let cur = seq.load(Ordering::Acquire); + if cur & 1 == 0 + && seq + .compare_exchange_weak(cur, cur.wrapping_add(1), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + return SeqWriteGuard { seq }; + } + std::hint::spin_loop(); + } +} + +impl Drop for SeqWriteGuard<'_> { + fn drop(&mut self) { + // odd -> even: publishes the write + self.seq.fetch_add(1, Ordering::Release); + } +} + +/// Run `read` until it observes a stable (even, unchanged) sequence. `read` must be +/// side-effect-free on retry and must not dereference data whose validity depends on seq. +#[inline] +pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { + loop { + let before = seq.load(Ordering::Acquire); + if before & 1 == 0 { + let value = read(); + std::sync::atomic::fence(Ordering::Acquire); + if seq.load(Ordering::Relaxed) == before { + return value; + } + } + std::hint::spin_loop(); + } +} From 18a015099900a028090521b621bbb5913657b144 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 05:50:25 -0600 Subject: [PATCH 02/69] hnsw-plane: zero-copy search, AVX2 kernels, optimizeRouting-parity insert Lever 1: distance_to/neighbors_into read directly against the mmap under the seqlock (no per-visit Vec allocation); full-copy read_node remains for construction paths only. Lever 2: explicit AVX2+FMA f32xi8 asymmetric kernel and AVX2 i8xi8 symmetric kernel (construction-time neighbor distances, recomputed since the format drops stored per-edge distances), runtime-detected with scalar fallback. Linux x86_64 is the performance target per design decision; Windows may fall back to JS entirely. Lever 3: insert ported to JS optimizeRouting parity - rank-ordered candidate selection with indirect-route skipping and edge replacement, per-level searchLayer construction, reverse edges with prune-to-cap-64. Fixed a port bug where the neighbor scan broke on the first added- connection match (JS breaks only the inner scan). Bench: Gaussian-mixture corpus matching hnsw-scale.js calibration (uniform-random 768-d is un-indexable per that benchmark's notes) + brute-force recall@10. At 100K/ef512: p50 0.28ms, 0.201 us/visit (JS 4.34), recall 1.000, build 5,583 inserts/s. Design doc updated with cap-64, platform, and packaging decisions. Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 49 +++++- native/hnsw-plane/src/bin/bench.rs | 78 +++++++++- native/hnsw-plane/src/distance.rs | 112 ++++++++++++-- native/hnsw-plane/src/graph.rs | 130 +++++++++++++--- native/hnsw-plane/src/insert.rs | 237 ++++++++++++++++++++++------- native/hnsw-plane/src/search.rs | 175 +++++++++++++-------- 6 files changed, 616 insertions(+), 165 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 249b5dfae6..b0bd236727 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -232,15 +232,50 @@ p50 7.2 ms / recall@10-set 0.997 @ ef 512). Acceptance for phase 1: 5. **Churn:** delete/reinsert cycles hold node count stable (freelist reuse; #2182 regression test). -## 10. Open questions +## 10. Decisions & open questions + +Decided (Kris, 2026-08-31): + +- **Degree cap = 64.** Size is critical; 128 doubles the file for a degree tail (measured mean + ~37). Confirmed empirically: cap-64 graphs reach recall@10 = 1.000 at 100K on the calibrated + corpus (§11). Cap remains a header field; revising it is a rebuild. +- **Platform policy.** Performance is a Linux target only. macOS must work (mmap/msync semantics + differ slightly — `F_FULLFSYNC` for real durability barriers, no sparse-file guarantees on all + filesystems — both handled, neither optimized). Windows may fall back to the JS implementation + entirely; the native plane is allowed to be absent there. +- **Packaging: independent open-source package.** The core has zero Harper coupling — the crate + compiles standalone and its NAPI surface is generic (create/open plane, insert(id, vector), + remove(id), search(query, k, ef, filter), watermark get/set). Harper-specific glue — the + pk→nodeId mapping, commit-callback integration, txnlog-anchored replay, auto-ef policy + constants — stays in Harper regardless of packaging. Plan: develop in-repo under + `native/hnsw-plane/` until the NAPI surface stabilizes (end of phase 1), then split to its own + repo in the symphony/lmdb-js mold and consume via npm. The pitch as a community package: a + persistent, incrementally-maintained, concurrently-searchable HNSW for Node — hnswlib-node has + no durable incremental persistence, no off-loop batched filtering, no seqlock concurrency. + +Open: -- **Degree cap 64 vs 128** — measured decision, §9.2. -- **macOS/Windows sparse-file + mmap growth semantics** — Linux is the production target; - dev platforms may need chunked remapping instead of one large sparse reservation. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. -- **Where the crate lives** — in-repo `native/hnsw-plane/` (tight iteration, CI builds Rust) - vs separate repo à la symphony (prebuilds, independent versioning). Prototype in-repo; - decide before merge. - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the format (dims × mode in header) but int8 is the default and the optimization target. +- **Upper-layer region persistence** — prototype keeps upper adjacency in memory (rebuilt at + open by scanning slots); the append-allocated file region is pending. + +## 11. Prototype measurements (kzyp Linux box, 768-d int8, ef 512, cap 64) + +Gaussian-mixture corpus matching `benchmarks/hnsw-scale.js` calibration (intra-cos 0.75, +clusters = N/500). JS baseline for scale: 4.34 µs/visit; 1M efC-200 anchor: p50 7.2 ms, +recall@10-set 0.997, ~3,110 visits. + +| N | p50 | p95 | visits/query | µs/visit | recall@10 (set) | build rate | +| --- | --- | --- | --- | --- | --- | --- | +| 100K | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | + +Milestones: zero-copy seqlock reads + AVX2 kernels took per-visit cost from 0.440 µs (first +scalar prototype) to ~0.1–0.2 µs — **~22–45× vs the JS per-visit baseline**, beating the +0.25–0.4 µs design budget. The optimizeRouting-parity insert (including the recomputed +neighbor↔neighbor distances) restored recall to 1.000 where the placeholder insert produced +unnavigable graphs. Uniform-random 768-d corpora produce meaningless recall numbers (the JS +benchmark's own calibration note: a corpus "no ANN can index") — all comparisons use the +mixture corpus. diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index c5a429e48e..1293b376ba 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -14,14 +14,65 @@ use std::time::Instant; // xorshift for reproducible synthetic vectors without a rand dependency struct Rng(u64); impl Rng { - fn next_f32(&mut self) -> f32 { + fn next_unit(&mut self) -> f32 { self.0 ^= self.0 << 13; self.0 ^= self.0 >> 7; self.0 ^= self.0 << 17; - (self.0 >> 40) as f32 / (1u64 << 24) as f32 - 0.5 + (self.0 >> 40) as f32 / (1u64 << 24) as f32 } - fn vector(&mut self, dims: usize) -> Vec { - (0..dims).map(|_| self.next_f32()).collect() + // Box-Muller + fn next_gauss(&mut self) -> f32 { + let u1 = self.next_unit().max(f32::MIN_POSITIVE); + let u2 = self.next_unit(); + (-2.0 * u1.ln()).sqrt() * (2.0 * std::f32::consts::PI * u2).cos() + } +} + +/// Gaussian-mixture corpus matching benchmarks/hnsw-scale.js: unit centroids, per-dim noise +/// derived from an intra-cluster cosine target of 0.75 (uniform-random 768-d is a corpus +/// "no ANN can index" per that benchmark's own calibration notes). +struct Corpus { + centroids: Vec, + n_clusters: usize, + dims: usize, + noise: f32, +} + +impl Corpus { + fn new(n: u64, dims: usize, rng: &mut Rng) -> Self { + let intra_cos = 0.75f32; + let noise = ((1.0 / (intra_cos * intra_cos) - 1.0) / dims as f32).sqrt(); + let n_clusters = 8.max((n as f64 / 500.0).round() as usize); + let mut centroids = vec![0.0f32; n_clusters * dims]; + for c in 0..n_clusters { + let mut mag = 0.0f32; + for d in 0..dims { + let x = rng.next_gauss(); + centroids[c * dims + d] = x; + mag += x * x; + } + let mag = mag.sqrt().max(f32::MIN_POSITIVE); + for d in 0..dims { + centroids[c * dims + d] /= mag; + } + } + Corpus { centroids, n_clusters, dims, noise } + } + + fn row(&self, rng: &mut Rng) -> Vec { + let c = (rng.next_unit() * self.n_clusters as f32) as usize % self.n_clusters; + let mut v = vec![0.0f32; self.dims]; + let mut mag = 0.0f32; + for d in 0..self.dims { + let x = self.centroids[c * self.dims + d] + rng.next_gauss() * self.noise; + v[d] = x; + mag += x * x; + } + let mag = mag.sqrt().max(f32::MIN_POSITIVE); + for d in 0..self.dims { + v[d] /= mag; + } + v } } @@ -46,10 +97,11 @@ fn main() { let params = InsertParams::default(); let mut scratch = SearchScratch::new(); let mut rng = Rng(0x1234_5678_9abc_def0); + let corpus = Corpus::new(n, dims, &mut rng); let build_start = Instant::now(); for i in 0..n { - let v = rng.vector(dims); + let v = corpus.row(&mut rng); insert(&graph, &v, ¶ms, &mut scratch); if (i + 1) % 50_000 == 0 { let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); @@ -59,16 +111,27 @@ fn main() { let build = build_start.elapsed(); println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); - // Query with held-out vectors. + // Query with held-out vectors; measure latency and set-recall@10 vs brute-force truth + // (same asymmetric metric, so recall isolates graph quality, not quantization). let mut latencies = Vec::with_capacity(queries); let mut total_visits = 0u64; + let mut recall_hits = 0usize; + let mut recall_total = 0usize; for _ in 0..queries { - let q = Query::new(rng.vector(dims)); + let q = Query::new(corpus.row(&mut rng)); let start = Instant::now(); let (results, stats) = search(&graph, &q, 10, ef, &mut scratch); latencies.push(start.elapsed()); total_visits += stats.visits; assert!(!results.is_empty()); + + let mut truth: Vec<(u32, f32)> = (0..n as u32) + .filter_map(|id| graph.distance_to(id, &q).map(|d| (id, d))) + .collect(); + truth.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap()); + truth.truncate(10); + recall_total += truth.len(); + recall_hits += truth.iter().filter(|(tid, _)| results.iter().any(|(rid, _)| rid == tid)).count(); } latencies.sort(); let p50 = latencies[queries / 2]; @@ -85,4 +148,5 @@ fn main() { mean_visits, us_per_visit ); + println!("recall@10 (set): {:.3}", recall_hits as f64 / recall_total as f64); } diff --git a/native/hnsw-plane/src/distance.rs b/native/hnsw-plane/src/distance.rs index 8ef4db2784..949378aced 100644 --- a/native/hnsw-plane/src/distance.rs +++ b/native/hnsw-plane/src/distance.rs @@ -1,6 +1,8 @@ -//! Asymmetric distance: full-precision f32 query × int8-stored vector, matching the JS -//! implementation (quantizeInt8 scale + cached 1/|v|). Written as autovectorizable loops; -//! explicit AVX2/NEON intrinsics are a measured follow-up if codegen disappoints. +//! Distance kernels. Asymmetric: full-precision f32 query × int8-stored vector (matches the JS +//! quantizeInt8 scale + cached 1/|v| model). Symmetric int8×int8 for construction-time +//! neighbor↔neighbor checks (stored per-edge distances were dropped from the format; recompute). +//! AVX2 with scalar fallback; Linux x86_64 is the performance target, other platforms take the +//! scalar path (fine for dev). /// Precomputed query state, built once per search. pub struct Query { @@ -16,26 +18,114 @@ impl Query { } } -/// Cosine distance against an int8-quantized stored vector. -/// stored dot = scale * Σ q[i] * v[i]; distance = 1 - dot * inv_mag_stored * inv_mag_query. #[inline] -pub fn cosine_int8(query: &Query, stored: &[i8], scale: f32, stored_inv_mag: f32) -> f32 { - debug_assert_eq!(query.vector.len(), stored.len()); +fn dot_f32_i8_scalar(q: &[f32], v: *const i8) -> f32 { let mut acc = [0.0f32; 8]; - let chunks = stored.len() / 8; + let chunks = q.len() / 8; for c in 0..chunks { let base = c * 8; for lane in 0..8 { - acc[lane] += query.vector[base + lane] * stored[base + lane] as f32; + acc[lane] += q[base + lane] * unsafe { *v.add(base + lane) } as f32; } } let mut dot: f32 = acc.iter().sum(); - for i in chunks * 8..stored.len() { - dot += query.vector[i] * stored[i] as f32; + for i in chunks * 8..q.len() { + dot += q[i] * unsafe { *v.add(i) } as f32; } + dot +} + +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2", enable = "fma")] +unsafe fn dot_f32_i8_avx2(q: &[f32], v: *const i8) -> f32 { + use std::arch::x86_64::*; + let mut acc0 = _mm256_setzero_ps(); + let mut acc1 = _mm256_setzero_ps(); + let chunks = q.len() / 16; + for c in 0..chunks { + let base = c * 16; + let v16 = _mm_loadu_si128(v.add(base) as *const __m128i); + let lo = _mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(v16)); + let hi = _mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(_mm_srli_si128(v16, 8))); + acc0 = _mm256_fmadd_ps(_mm256_loadu_ps(q.as_ptr().add(base)), lo, acc0); + acc1 = _mm256_fmadd_ps(_mm256_loadu_ps(q.as_ptr().add(base + 8)), hi, acc1); + } + let acc = _mm256_add_ps(acc0, acc1); + let s = _mm_add_ps(_mm256_extractf128_ps(acc, 1), _mm256_castps256_ps128(acc)); + let s = _mm_hadd_ps(s, s); + let s = _mm_hadd_ps(s, s); + let mut dot = _mm_cvtss_f32(s); + for i in chunks * 16..q.len() { + dot += q[i] * *v.add(i) as f32; + } + dot +} + +#[inline] +fn dot_f32_i8(q: &[f32], v: *const i8) -> f32 { + #[cfg(target_arch = "x86_64")] + { + if std::arch::is_x86_feature_detected!("avx2") && std::arch::is_x86_feature_detected!("fma") { + return unsafe { dot_f32_i8_avx2(q, v) }; + } + } + dot_f32_i8_scalar(q, v) +} + +/// Cosine distance: f32 query × raw int8 vector at `stored` (dims = query.vector.len()). +/// Zero-copy: `stored` points into the mmap; the caller's seqlock read discards torn results. +#[inline] +pub fn cosine_int8_raw(query: &Query, stored: *const i8, scale: f32, stored_inv_mag: f32) -> f32 { + let dot = dot_f32_i8(&query.vector, stored); 1.0 - dot * scale * stored_inv_mag * query.inv_mag } +#[inline] +fn dot_i8_i8_scalar(a: *const i8, b: *const i8, len: usize) -> i32 { + let mut dot = 0i32; + for i in 0..len { + dot += unsafe { *a.add(i) as i32 * *b.add(i) as i32 }; + } + dot +} + +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx2")] +unsafe fn dot_i8_i8_avx2(a: *const i8, b: *const i8, len: usize) -> i32 { + use std::arch::x86_64::*; + let mut acc = _mm256_setzero_si256(); + let chunks = len / 16; + for c in 0..chunks { + let av = _mm256_cvtepi8_epi16(_mm_loadu_si128(a.add(c * 16) as *const __m128i)); + let bv = _mm256_cvtepi8_epi16(_mm_loadu_si128(b.add(c * 16) as *const __m128i)); + acc = _mm256_add_epi32(acc, _mm256_madd_epi16(av, bv)); + } + let lo = _mm256_castsi256_si128(acc); + let hi = _mm256_extracti128_si256(acc, 1); + let s = _mm_add_epi32(lo, hi); + let s = _mm_add_epi32(s, _mm_srli_si128(s, 8)); + let s = _mm_add_epi32(s, _mm_srli_si128(s, 4)); + let mut dot = _mm_cvtsi128_si32(s); + for i in chunks * 16..len { + dot += *a.add(i) as i32 * *b.add(i) as i32; + } + dot +} + +/// Cosine distance between two int8-stored vectors (construction-time neighbor checks). +#[inline] +pub fn cosine_i8_i8_raw(a: *const i8, scale_a: f32, inv_mag_a: f32, b: *const i8, scale_b: f32, inv_mag_b: f32, len: usize) -> f32 { + #[cfg(target_arch = "x86_64")] + let dot = if std::arch::is_x86_feature_detected!("avx2") { + unsafe { dot_i8_i8_avx2(a, b, len) } + } else { + dot_i8_i8_scalar(a, b, len) + }; + #[cfg(not(target_arch = "x86_64"))] + let dot = dot_i8_i8_scalar(a, b, len); + 1.0 - dot as f32 * scale_a * scale_b * inv_mag_a * inv_mag_b +} + /// Symmetric int8 quantization matching the JS quantizeInt8: scale maps max |component| to 127. pub fn quantize_int8(vector: &[f32]) -> (Vec, f32, f32) { let max_abs = vector.iter().fold(0.0f32, |m, v| m.max(v.abs())); diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index c417f64104..6c68288552 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -1,7 +1,9 @@ //! Slot-level node access over the plane file, mediated by the seqlock, plus the resident -//! upper-layer structure. Prototype status: upper layers live in memory and are rebuilt at -//! open by scanning slots; the append-allocated file region from the design doc is a TODO. +//! upper-layer structure. Hot-path reads (distance, neighbor ids) are zero-copy against the +//! mmap; full-copy read_node exists for construction paths. Prototype status: upper layers +//! live in memory; the append-allocated file region from the design doc is a TODO. +use crate::distance::{cosine_i8_i8_raw, cosine_int8_raw, Query}; use crate::format::{PlaneFile, FLAG_DELETED, FLAG_VALID, S_DEGREE, S_FLAGS, S_INV_MAG, S_LEVEL, S_SCALE, S_VECTOR}; use crate::seqlock; use std::collections::HashMap; @@ -13,9 +15,8 @@ pub struct Graph { pub upper: RwLock>>>, } -/// A consistent copy of one node's traversal-relevant data. +/// A consistent full copy of one node (construction paths only; search uses zero-copy). pub struct NodeRead { - pub valid: bool, pub level: u8, pub scale: f32, pub inv_mag: f32, @@ -28,15 +29,103 @@ impl Graph { Graph { file, upper: RwLock::new(HashMap::new()) } } - /// Seqlock-consistent read of a slot. Returns None for never-written or deleted slots. + #[inline] + fn in_range(&self, id: u32) -> bool { + (id as u64) < self.file.id_high_water() + } + + /// Zero-copy distance from `query` to the stored vector of `id`. None for absent/deleted. + #[inline] + pub fn distance_to(&self, id: u32, query: &Query) -> Option { + if !self.in_range(id) { + return None; + } + let seq = self.file.seq_atomic(id); + seqlock::read_consistent(seq, || { + let p = self.file.slot_ptr(id); + unsafe { + let flags = *p.add(S_FLAGS); + if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { + return None; + } + let scale = (p.add(S_SCALE) as *const f32).read_unaligned(); + let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); + Some(cosine_int8_raw(query, p.add(S_VECTOR) as *const i8, scale, inv_mag)) + } + }) + } + + /// Symmetric stored-to-stored distance (construction-time neighbor↔neighbor checks). + pub fn distance_between(&self, a: u32, b: u32) -> Option { + if !self.in_range(a) || !self.in_range(b) { + return None; + } + // Two independent seqlock reads: copy a's params + vector ptr safely by nesting reads. + // A torn cross-pair read is acceptable here (construction heuristic, not a result). + let dims = self.file.dims; + let pa = self.file.slot_ptr(a); + let pb = self.file.slot_ptr(b); + unsafe { + let fa = *pa.add(S_FLAGS); + let fb = *pb.add(S_FLAGS); + if fa & FLAG_VALID == 0 || fa & FLAG_DELETED != 0 || fb & FLAG_VALID == 0 || fb & FLAG_DELETED != 0 { + return None; + } + let scale_a = (pa.add(S_SCALE) as *const f32).read_unaligned(); + let inv_a = (pa.add(S_INV_MAG) as *const f32).read_unaligned(); + let scale_b = (pb.add(S_SCALE) as *const f32).read_unaligned(); + let inv_b = (pb.add(S_INV_MAG) as *const f32).read_unaligned(); + Some(cosine_i8_i8_raw( + pa.add(S_VECTOR) as *const i8, + scale_a, + inv_a, + pb.add(S_VECTOR) as *const i8, + scale_b, + inv_b, + dims, + )) + } + } + + /// Copy layer-0 neighbor ids into `out` (cleared first). Returns the node's level, + /// or None for absent/deleted. + #[inline] + pub fn neighbors_into(&self, id: u32, out: &mut Vec) -> Option { + out.clear(); + if !self.in_range(id) { + return None; + } + let seq = self.file.seq_atomic(id); + let cap = self.file.layer0_cap; + let dims = self.file.dims; + seqlock::read_consistent(seq, || { + out.clear(); + let p = self.file.slot_ptr(id); + unsafe { + let flags = *p.add(S_FLAGS); + if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { + return None; + } + let level = *p.add(S_LEVEL); + let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; + let base = p.add(S_VECTOR + dims) as *const u32; + for i in 0..degree.min(cap) { + out.push(u32::from_le(base.add(i).read_unaligned())); + } + Some(level) + } + }) + } + + /// Seqlock-consistent full copy (construction paths). pub fn read_node(&self, id: u32) -> Option { - if (id as u64) >= self.file.id_high_water() { + if !self.in_range(id) { return None; } let seq = self.file.seq_atomic(id); let dims = self.file.dims; let cap = self.file.layer0_cap; - let node = seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, || { let p = self.file.slot_ptr(id); unsafe { let flags = *p.add(S_FLAGS); @@ -44,16 +133,15 @@ impl Graph { return None; } let level = *p.add(S_LEVEL); - let degree = u16::from_le(*(p.add(S_DEGREE) as *const u16)) as usize; - let scale = f32::from_le_bytes(std::slice::from_raw_parts(p.add(S_SCALE), 4).try_into().unwrap()); - let inv_mag = f32::from_le_bytes(std::slice::from_raw_parts(p.add(S_INV_MAG), 4).try_into().unwrap()); + let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; + let scale = (p.add(S_SCALE) as *const f32).read_unaligned(); + let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); let vector = std::slice::from_raw_parts(p.add(S_VECTOR) as *const i8, dims).to_vec(); - let nbytes = std::slice::from_raw_parts(p.add(S_VECTOR + dims), degree.min(cap) * 4); - let neighbors = nbytes.chunks_exact(4).map(|c| u32::from_le_bytes(c.try_into().unwrap())).collect(); - Some(NodeRead { valid: true, level, scale, inv_mag, vector, neighbors }) + let nbase = p.add(S_VECTOR + dims) as *const u32; + let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(nbase.add(i).read_unaligned())).collect(); + Some(NodeRead { level, scale, inv_mag, vector, neighbors }) } - }); - node + }) } /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the caller. @@ -66,12 +154,12 @@ impl Graph { let dims = self.file.dims; unsafe { *p.add(S_LEVEL) = level; - (p.add(S_DEGREE) as *mut u16).write((neighbors.len() as u16).to_le()); - std::ptr::copy_nonoverlapping(scale.to_le_bytes().as_ptr(), p.add(S_SCALE), 4); - std::ptr::copy_nonoverlapping(inv_mag.to_le_bytes().as_ptr(), p.add(S_INV_MAG), 4); + (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); + (p.add(S_SCALE) as *mut f32).write_unaligned(scale); + (p.add(S_INV_MAG) as *mut f32).write_unaligned(inv_mag); std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); for (i, n) in neighbors.iter().enumerate() { - (p.add(S_VECTOR + dims + i * 4) as *mut u32).write(n.to_le()); + (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); } // valid last within the locked section; the seqlock release publishes it *p.add(S_FLAGS) = FLAG_VALID; @@ -86,9 +174,9 @@ impl Graph { let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { - (p.add(S_DEGREE) as *mut u16).write((neighbors.len() as u16).to_le()); + (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); for (i, n) in neighbors.iter().enumerate() { - (p.add(S_VECTOR + dims + i * 4) as *mut u32).write(n.to_le()); + (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); } } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index c1c334f2ed..243d889815 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -1,26 +1,28 @@ -//! Prototype insert: enough HNSW construction to build benchmark graphs. Neighbor selection -//! is plain closest-M (the JS optimizeRouting-aware selection is the parity target for the -//! phase-3 native insert; for per-visit cost measurement this suffices). +//! HNSW insert with parity to the JS implementation's optimizeRouting selection +//! (HierarchicalNavigableSmallWorld.ts): candidate i is skipped when an already-added +//! connection reaches it indirectly at comparable cost, and inferior indirect edges are +//! replaced by the new direct route. Stored per-edge distances were dropped from the file +//! format, so neighbor↔neighbor distances are recomputed (int8×int8) on id-match hits only. -use crate::distance::{cosine_int8, quantize_int8, Query}; +use crate::distance::{quantize_int8, Query}; use crate::format::NO_ID; use crate::graph::Graph; -use crate::search::{search, SearchScratch}; +use crate::search::{greedy_descend, search_layer, SearchScratch, SearchStats}; pub struct InsertParams { - pub m: usize, // upper-layer connections + pub m: usize, // base connection count (JS M, default 16) pub ef_construction: usize, // candidate list size pub ml: f64, // level normalization: 1 / ln(M) + pub optimize_routing: f32, // JS optimizeRouting, default 0.5; 0 disables } impl Default for InsertParams { fn default() -> Self { - InsertParams { m: 16, ef_construction: 200, ml: 1.0 / (16f64).ln() } + InsertParams { m: 16, ef_construction: 200, ml: 1.0 / (16f64).ln(), optimize_routing: 0.5 } } } -/// Deterministic pseudo-random level from the node id (parity with Math.random is not needed -/// for benchmarks; a hash keeps runs reproducible). +/// Deterministic pseudo-random level from the node id (reproducible benchmark builds). fn level_for(id: u32, ml: f64) -> u8 { let mut x = (id as u64).wrapping_mul(0x9e37_79b9_7f4a_7c15).wrapping_add(0x2545_f491_4f6c_dd1d); x ^= x >> 33; @@ -29,11 +31,97 @@ fn level_for(id: u32, ml: f64) -> u8 { level.min(31.0) as u8 } +/// Remove `to` from `from`'s adjacency at `level` (edge-replacement maintenance). +fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { + if level == 0 { + if let Some(n) = graph.read_node(from) { + if let Some(pos) = n.neighbors.iter().position(|&x| x == to) { + let mut list = n.neighbors; + list.remove(pos); + graph.write_neighbors(from, &list); + } + } + } else { + let mut upper = graph.upper.write().unwrap(); + if let Some(levels) = upper.get_mut(&from) { + if let Some(list) = levels.get_mut(level as usize - 1) { + if let Some(pos) = list.iter().position(|&x| x == to) { + list.remove(pos); + } + } + } + } +} + +/// Neighbor ids of `id` at `level` (level 0 from the slot, upper from the resident map). +fn neighbors_at(graph: &Graph, id: u32, level: u8, buf: &mut Vec) { + if level == 0 { + graph.neighbors_into(id, buf); + } else { + buf.clear(); + let upper = graph.upper.read().unwrap(); + if let Some(levels) = upper.get(&id) { + if let Some(list) = levels.get(level as usize - 1) { + buf.extend_from_slice(list); + } + } + } +} + +/// Add `new_id` to `nid`'s adjacency at `level`, pruning to `cap` closest when over. +fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) { + if level == 0 { + let Some(n) = graph.read_node(nid) else { return }; + let mut list = n.neighbors; + if list.contains(&new_id) { + return; + } + list.push(new_id); + if list.len() > cap { + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) + .collect(); + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + list = scored.into_iter().take(cap).map(|(cand, _)| cand).collect(); + } + graph.write_neighbors(nid, &list); + } else { + let mut upper = graph.upper.write().unwrap(); + if let Some(levels) = upper.get_mut(&nid) { + if let Some(list) = levels.get_mut(level as usize - 1) { + if !list.contains(&new_id) { + list.push(new_id); + if list.len() > cap { + drop(upper); + // prune by recomputed distance outside the write lock + let mut scored: Vec<(u32, f32)> = { + let upper = graph.upper.read().unwrap(); + let list = upper.get(&nid).and_then(|l| l.get(level as usize - 1)).cloned().unwrap_or_default(); + list.iter().filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))).collect() + }; + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + let pruned: Vec = scored.into_iter().take(cap).map(|(c, _)| c).collect(); + let mut upper = graph.upper.write().unwrap(); + if let Some(levels) = upper.get_mut(&nid) { + if let Some(list) = levels.get_mut(level as usize - 1) { + *list = pruned; + } + } + } + } + } + } + } +} + pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mut SearchScratch) -> u32 { let (bytes, scale, inv_mag) = quantize_int8(vector); let id = graph.file.allocate_id(); let level = level_for(id, params.ml); let query = Query::new(vector.to_vec()); + let layer0_cap = graph.file.layer0_cap; + let m = params.m; let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { @@ -45,66 +133,90 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu return id; } - // Candidate discovery at layer 0. - let (candidates, _) = search(graph, &query, params.ef_construction, params.ef_construction, scratch); - let layer0_cap = graph.file.layer0_cap; - let m0 = (params.m * 2).min(layer0_cap); - let neighbors: Vec = candidates.iter().take(m0).map(|(nid, _)| *nid).collect(); + let mut stats = SearchStats { visits: 0 }; + // scratch epochs are per search_layer sweep; begin() per level below. + let entry_dist = graph.distance_to(entry_id, &query).unwrap_or(f32::INFINITY); + let top = level.min(entry_level as u8); + let (mut ep, mut ep_dist) = + greedy_descend(graph, &query, entry_id, entry_dist, entry_level, top as u32, &mut stats); - graph.write_node(id, level, &bytes, scale, inv_mag, &neighbors); + // Per-level connection lists for the new node, selection-ordered. + let mut connections: Vec> = vec![Vec::new(); level as usize + 1]; + let mut nbuf: Vec = Vec::new(); - // Reverse edges at layer 0, pruning the neighbor's list to cap by distance if full. - for &nid in &neighbors { - if let Some(n) = graph.read_node(nid) { - let mut list = n.neighbors.clone(); - if list.contains(&id) { + for l in (0..=top).rev() { + scratch_begin(graph, scratch); + let mut neighbors = search_layer(graph, &query, ep, ep_dist, params.ef_construction, l, scratch, &mut stats); + neighbors.truncate(m << 1); + if let Some(&(best, best_d)) = neighbors.first() { + ep = best; + ep_dist = best_d; + } + + // JS optimizeRouting selection over rank-ordered candidates. + let take_conns = std::mem::take(&mut connections[l as usize]); + let mut conns = take_conns; + for (i, &(nid, ndist)) in neighbors.iter().enumerate() { + if nid == id { continue; } - list.push(id); - if list.len() > layer0_cap { - let nq = Query::new(n.vector.iter().map(|v| *v as f32 * n.scale).collect()); - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| { - graph.read_node(cand).map(|c| (cand, cosine_int8(&nq, &c.vector, c.scale, c.inv_mag))) - }) - .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - list = scored.into_iter().take(layer0_cap).map(|(cand, _)| cand).collect(); + let mut skipping = false; + let mut replaced: Vec<(u32, u32)> = Vec::new(); // (from, to) edge removals + if params.optimize_routing > 0.0 { + let distance_threshold = 1.0 + params.optimize_routing * (1.0 + (0.5 * i as f32) / m as f32); + neighbors_at(graph, nid, l, &mut nbuf); + for (i2, &nnid) in nbuf.iter().enumerate() { + let neighbor_threshold = 1.0 + params.optimize_routing * (1.0 + (0.5 * i2 as f32) / m as f32); + if let Some(&(added_id, added_dist)) = conns.iter().find(|(aid, _)| *aid == nnid) { + // recompute the stored neighbor↔neighbor distance (not persisted) + let neighbor_distance = graph.distance_between(nid, nnid).unwrap_or(f32::INFINITY); + if ndist * distance_threshold > added_dist + neighbor_distance { + skipping = true; + break; // JS: `if (skipping) break` ends the neighbor scan + } else if neighbor_distance * neighbor_threshold > ndist + added_dist { + replaced.push((added_id, nid)); + replaced.push((nid, added_id)); + } + // JS breaks only the inner connections scan; keep scanning neighbors + } + } + if skipping { + continue; + } + } else if i >= if l > 0 { m } else { m << 1 } { + continue; + } + conns.push((nid, ndist)); + for (from, to) in replaced { + remove_edge(graph, from, to, l); } - graph.write_neighbors(nid, &list); } + connections[l as usize] = conns; } - // Upper layers: link into per-level lists (prototype: closest-M from the candidate set). + // Write the new node: layer-0 list pruned to the file cap (selection order = rank order). + let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); + l0.truncate(layer0_cap); + graph.write_node(id, level, &bytes, scale, inv_mag, &l0); + if level > 0 { - let mut levels: Vec> = Vec::with_capacity(level as usize); - let upper = graph.upper.read().unwrap(); - for l in 1..=level { - let peers: Vec = candidates - .iter() - .filter(|(nid, _)| { - graph.read_node(*nid).map(|n| n.level >= l).unwrap_or(false) && upper.contains_key(nid) - }) - .take(params.m) - .map(|(nid, _)| *nid) - .collect(); - levels.push(peers); - } - drop(upper); - let mut upper = graph.upper.write().unwrap(); - for (l, peers) in levels.iter().enumerate() { - for &peer in peers { - if let Some(peer_levels) = upper.get_mut(&peer) { - if let Some(peer_list) = peer_levels.get_mut(l) { - if !peer_list.contains(&id) && peer_list.len() < params.m * 2 { - peer_list.push(id); - } - } - } - } + let levels: Vec> = (1..=level as usize) + .map(|l| { + connections + .get(l) + .map(|c| c.iter().map(|&(nid, _)| nid).collect()) + .unwrap_or_default() + }) + .collect(); + graph.upper.write().unwrap().insert(id, levels); + } + + // Reverse edges. + for (l, conns) in connections.iter().enumerate() { + let cap = if l == 0 { layer0_cap } else { m << 1 }; + for &(nid, _) in conns { + add_reverse_edge(graph, nid, id, l as u8, cap); } - upper.insert(id, levels); } if (level as u32) > entry_level { @@ -112,3 +224,10 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu } id } + +#[inline] +fn scratch_begin(graph: &Graph, scratch: &mut SearchScratch) { + // search_layer assumes a fresh epoch per sweep; SearchScratch::begin is crate-private + // via this helper to keep the public surface small. + scratch.begin_public(graph.file.id_high_water()); +} diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index b213df5f4e..488833a634 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -1,7 +1,8 @@ -//! Beam search over the plane. Visited tracking is an epoch-stamped array (no per-query -//! allocation once warmed); candidate/result sets are simple binary heaps. +//! Beam search over the plane, zero-copy: per-visit cost is one seqlock-guarded distance +//! against mmap bytes plus primitive heap/visited ops. Visited tracking is an epoch-stamped +//! array; neighbor ids stream through a reusable scratch buffer. -use crate::distance::{cosine_int8, Query}; +use crate::distance::Query; use crate::format::NO_ID; use crate::graph::Graph; use std::cmp::Ordering as CmpOrdering; @@ -43,15 +44,20 @@ impl PartialOrd for Result_ { } } -/// Reusable per-thread search scratch: epoch-stamped visited array. +/// Reusable per-thread search scratch. pub struct SearchScratch { visited: Vec, epoch: u32, + neighbors: Vec, } impl SearchScratch { pub fn new() -> Self { - SearchScratch { visited: Vec::new(), epoch: 0 } + SearchScratch { visited: Vec::new(), epoch: 0, neighbors: Vec::new() } + } + + pub fn begin_public(&mut self, capacity: u64) { + self.begin(capacity) } fn begin(&mut self, capacity: u64) { @@ -77,36 +83,95 @@ impl SearchScratch { } } +impl Default for SearchScratch { + fn default() -> Self { + Self::new() + } +} + pub struct SearchStats { pub visits: u64, } -/// Full search: greedy descent through upper layers, then beam at layer 0. -pub fn search( +/// Beam search within one layer, starting from `entry`. Level 0 reads slot adjacency; +/// upper levels read the resident upper map. Returns (id, distance) ascending by distance. +/// Assumes scratch.begin() was called for this query; entry is marked visited here. +pub fn search_layer( graph: &Graph, query: &Query, - k: usize, + entry: u32, + entry_dist: f32, ef: usize, + level: u8, scratch: &mut SearchScratch, -) -> (Vec<(u32, f32)>, SearchStats) { - let mut stats = SearchStats { visits: 0 }; - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { - return (Vec::new(), stats); - } - scratch.begin(graph.file.id_high_water()); + stats: &mut SearchStats, +) -> Vec<(u32, f32)> { + let mut candidates = BinaryHeap::new(); + let mut results: BinaryHeap = BinaryHeap::new(); + scratch.visit(entry); + candidates.push(Candidate { distance: entry_dist, id: entry }); + results.push(Result_ { distance: entry_dist, id: entry }); - // Greedy descent: single-candidate walk from the top level down to level 1. - let mut current = entry_id; - let mut current_dist = match graph.read_node(current) { - Some(n) => { - stats.visits += 1; - cosine_int8(query, &n.vector, n.scale, n.inv_mag) + // take() the scratch neighbor buffer to sidestep the double-borrow of scratch + let mut nbuf = std::mem::take(&mut scratch.neighbors); + + while let Some(c) = candidates.pop() { + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() >= ef && c.distance > worst { + break; } - None => return (Vec::new(), stats), - }; + if level == 0 { + if graph.neighbors_into(c.id, &mut nbuf).is_none() { + continue; + } + } else { + nbuf.clear(); + let upper = graph.upper.read().unwrap(); + if let Some(levels) = upper.get(&c.id) { + if let Some(list) = levels.get(level as usize - 1) { + nbuf.extend_from_slice(list); + } + } + } + for i in 0..nbuf.len() { + let nid = nbuf[i]; + if !scratch.visit(nid) { + continue; + } + if let Some(d) = graph.distance_to(nid, query) { + stats.visits += 1; + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() < ef || d < worst { + candidates.push(Candidate { distance: d, id: nid }); + results.push(Result_ { distance: d, id: nid }); + if results.len() > ef { + results.pop(); + } + } + } + } + } + scratch.neighbors = nbuf; + + let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); + out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + out +} + +/// Greedy single-candidate descent through upper layers from `from_level` down to +/// `to_level` (exclusive lower bound handled by caller loops). Returns improved entry. +pub fn greedy_descend( + graph: &Graph, + query: &Query, + mut current: u32, + mut current_dist: f32, + from_level: u32, + to_level: u32, + stats: &mut SearchStats, +) -> (u32, f32) { let upper = graph.upper.read().unwrap(); - for level in (1..=entry_level).rev() { + let mut level = from_level; + while level > to_level { let mut improved = true; while improved { improved = false; @@ -116,9 +181,8 @@ pub fn search( .cloned() .unwrap_or_default(); for nid in neighbors { - if let Some(n) = graph.read_node(nid) { + if let Some(d) = graph.distance_to(nid, query) { stats.visits += 1; - let d = cosine_int8(query, &n.vector, n.scale, n.inv_mag); if d < current_dist { current = nid; current_dist = d; @@ -127,44 +191,35 @@ pub fn search( } } } + level -= 1; } - drop(upper); - - // Layer-0 beam. - let mut candidates = BinaryHeap::new(); - let mut results: BinaryHeap = BinaryHeap::new(); - scratch.visit(current); - candidates.push(Candidate { distance: current_dist, id: current }); - results.push(Result_ { distance: current_dist, id: current }); + (current, current_dist) +} - while let Some(c) = candidates.pop() { - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() >= ef && c.distance > worst { - break; - } - if let Some(node) = graph.read_node(c.id) { - for nid in node.neighbors { - if !scratch.visit(nid) { - continue; - } - if let Some(n) = graph.read_node(nid) { - stats.visits += 1; - let d = cosine_int8(query, &n.vector, n.scale, n.inv_mag); - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() < ef || d < worst { - candidates.push(Candidate { distance: d, id: nid }); - results.push(Result_ { distance: d, id: nid }); - if results.len() > ef { - results.pop(); - } - } - } - } - } +/// Full search: greedy descent through upper layers, then beam at layer 0. +pub fn search( + graph: &Graph, + query: &Query, + k: usize, + ef: usize, + scratch: &mut SearchScratch, +) -> (Vec<(u32, f32)>, SearchStats) { + let mut stats = SearchStats { visits: 0 }; + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + return (Vec::new(), stats); } + scratch.begin(graph.file.id_high_water()); - let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); - out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + let entry_dist = match graph.distance_to(entry_id, query) { + Some(d) => { + stats.visits += 1; + d + } + None => return (Vec::new(), stats), + }; + let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); + let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats); out.truncate(k); (out, stats) } From c69fdc800900f7aa5c5e6f42efdd54af104286a6 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 06:04:29 -0600 Subject: [PATCH 03/69] bench: reuse existing plane file for ef sweeps; parameterize layer0 cap Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/bin/bench.rs | 53 +++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index 1293b376ba..ea30fc7537 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -2,7 +2,7 @@ //! per-visit cost — the number that decides whether the native plane hits its 0.25–0.4 µs //! budget (JS baseline: 4.34 µs/visit at 5M/ef 512). //! -//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] +//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=64] use hnsw_plane::distance::Query; use hnsw_plane::insert::{insert, InsertParams}; @@ -83,9 +83,20 @@ fn main() { let queries: usize = args.get(3).and_then(|a| a.parse().ok()).unwrap_or(200); let ef: usize = args.get(4).and_then(|a| a.parse().ok()).unwrap_or(512); let path: PathBuf = args.get(5).map(Into::into).unwrap_or_else(|| "/tmp/bench.hnsw".into()); + let layer0_cap: usize = args.get(6).and_then(|a| a.parse().ok()).unwrap_or(64); - let layer0_cap = 64; - let file = PlaneFile::create(&path, dims, layer0_cap, n + 1024).expect("create"); + // Reuse an existing plane file when it already holds exactly n nodes at the same cap + // (ef sweeps without rebuilding). The corpus RNG below replays identically. + let reuse = PlaneFile::open(&path) + .ok() + .filter(|f| f.id_high_water() == n && f.layer0_cap == layer0_cap) + .is_some(); + let file = if reuse { + println!("reusing existing plane at {}", path.display()); + PlaneFile::open(&path).expect("open") + } else { + PlaneFile::create(&path, dims, layer0_cap, n + 1024).expect("create") + }; println!( "plane: {} nodes x {} dims, slot {} B, file {:.1} GB (sparse)", n, @@ -99,17 +110,35 @@ fn main() { let mut rng = Rng(0x1234_5678_9abc_def0); let corpus = Corpus::new(n, dims, &mut rng); - let build_start = Instant::now(); - for i in 0..n { - let v = corpus.row(&mut rng); - insert(&graph, &v, ¶ms, &mut scratch); - if (i + 1) % 50_000 == 0 { - let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); - println!(" built {} ({:.0} inserts/s)", i + 1, rate); + if reuse { + // replay the build's RNG draws so query rows match a fresh run, and rebuild the + // in-memory upper layers (prototype: upper region not yet persisted) by re-linking + // via slot levels. Greedy descent degrades to entry-only when upper lists are empty, + // so recall at layer 0 still measures the persisted graph. + for _ in 0..n { + let _ = corpus.row(&mut rng); } + let mut upper = graph.upper.write().unwrap(); + for id in 0..n as u32 { + if let Some(node) = graph.read_node(id) { + if node.level > 0 { + upper.insert(id, vec![Vec::new(); node.level as usize]); + } + } + } + } else { + let build_start = Instant::now(); + for i in 0..n { + let v = corpus.row(&mut rng); + insert(&graph, &v, ¶ms, &mut scratch); + if (i + 1) % 50_000 == 0 { + let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); + println!(" built {} ({:.0} inserts/s)", i + 1, rate); + } + } + let build = build_start.elapsed(); + println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); } - let build = build_start.elapsed(); - println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); // Query with held-out vectors; measure latency and set-recall@10 vs brute-force truth // (same asymmetric metric, so recall isolates graph quality, not quantization). From b83b7e67e689e937ec519dfead196e470aa54c73 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 06:05:38 -0600 Subject: [PATCH 04/69] graph: persist upper-layer adjacency via sidecar file Prototype persistence for the hierarchy (slots store only the level, so upper edges were lost on reopen and reused planes searched layer-0-only). Atomic tmp+rename write on build completion; missing sidecar degrades to layer-0 search. The production design remains the in-file append region. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/bin/bench.rs | 19 ++++------ native/hnsw-plane/src/graph.rs | 61 ++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 12 deletions(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index ea30fc7537..f318de7d11 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -110,22 +110,16 @@ fn main() { let mut rng = Rng(0x1234_5678_9abc_def0); let corpus = Corpus::new(n, dims, &mut rng); + let upper_path = path.with_extension("hnsw.upper"); if reuse { - // replay the build's RNG draws so query rows match a fresh run, and rebuild the - // in-memory upper layers (prototype: upper region not yet persisted) by re-linking - // via slot levels. Greedy descent degrades to entry-only when upper lists are empty, - // so recall at layer 0 still measures the persisted graph. + // replay the build's RNG draws so query rows match a fresh run, and load the + // persisted upper-layer sidecar (missing sidecar = layer-0-only search). for _ in 0..n { let _ = corpus.row(&mut rng); } - let mut upper = graph.upper.write().unwrap(); - for id in 0..n as u32 { - if let Some(node) = graph.read_node(id) { - if node.level > 0 { - upper.insert(id, vec![Vec::new(); node.level as usize]); - } - } - } + graph.load_upper(&upper_path).expect("load upper sidecar"); + let count = graph.upper.read().unwrap().len(); + println!("upper layers loaded: {} nodes with level > 0", count); } else { let build_start = Instant::now(); for i in 0..n { @@ -138,6 +132,7 @@ fn main() { } let build = build_start.elapsed(); println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); + graph.save_upper(&upper_path).expect("save upper sidecar"); } // Query with held-out vectors; measure latency and set-recall@10 vs brute-force truth diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 6c68288552..75b3fd2ab8 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -181,6 +181,67 @@ impl Graph { } } + /// Persist the upper-layer adjacency to a sidecar file (prototype; the production design + /// is an append-allocated region inside the plane file — see design doc §4). Stale-on-crash + /// is acceptable: watermark replay re-feeds recent inserts, which re-links upper edges. + pub fn save_upper(&self, path: &std::path::Path) -> std::io::Result<()> { + use std::io::Write; + let upper = self.upper.read().unwrap(); + let mut buf: Vec = Vec::new(); + buf.extend_from_slice(&(upper.len() as u32).to_le_bytes()); + for (&id, levels) in upper.iter() { + buf.extend_from_slice(&id.to_le_bytes()); + buf.push(levels.len() as u8); + for list in levels { + buf.extend_from_slice(&(list.len() as u16).to_le_bytes()); + for &n in list { + buf.extend_from_slice(&n.to_le_bytes()); + } + } + } + let tmp = path.with_extension("upper.tmp"); + let mut f = std::fs::File::create(&tmp)?; + f.write_all(&buf)?; + f.sync_all()?; + std::fs::rename(tmp, path) + } + + /// Load the sidecar written by save_upper. Missing file leaves the hierarchy empty + /// (layer-0 search still works, just without upper-layer routing). + pub fn load_upper(&self, path: &std::path::Path) -> std::io::Result<()> { + let buf = match std::fs::read(path) { + Ok(b) => b, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(e), + }; + let mut pos = 0usize; + let rd_u32 = |b: &[u8], p: &mut usize| { + let v = u32::from_le_bytes(b[*p..*p + 4].try_into().unwrap()); + *p += 4; + v + }; + let count = rd_u32(&buf, &mut pos); + let mut upper = self.upper.write().unwrap(); + upper.clear(); + for _ in 0..count { + let id = rd_u32(&buf, &mut pos); + let nlevels = buf[pos] as usize; + pos += 1; + let mut levels = Vec::with_capacity(nlevels); + for _ in 0..nlevels { + let len = u16::from_le_bytes(buf[pos..pos + 2].try_into().unwrap()) as usize; + pos += 2; + let mut list = Vec::with_capacity(len); + for _ in 0..len { + list.push(rd_u32(&buf, &mut pos)); + } + levels.push(list); + } + upper.insert(id, levels); + } + Ok(()) + } + /// Mark deleted (traversals skip it) and return the id to the freelist. pub fn delete_node(&self, id: u32) { { From 815aa133ac3090addeccfafbe0e5f80d53b0fe80 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 06:17:14 -0600 Subject: [PATCH 05/69] =?UTF-8?q?design:=20cap-128=20for=20the=20int8=20pl?= =?UTF-8?q?ane=20=E2=80=94=20measured,=20not=20assumed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1M cap sweep: cap-64 loses 2.2pts recall (0.975 vs 0.996; JS anchor 0.997) at equal ef and latency, and cap-128 costs +23.5% file bytes, not 2x - the 768B vector dominates the int8 slot. Binary-code v2 slots reopen the question (+73% there). Measurement table updated: at the 1M anchor with cap 128 the native plane is 9.6x p50 / 12.9x per-visit / 4.7x build at JS-equal recall. Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index b0bd236727..e932c803f6 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -236,9 +236,13 @@ p50 7.2 ms / recall@10-set 0.997 @ ef 512). Acceptance for phase 1: Decided (Kris, 2026-08-31): -- **Degree cap = 64.** Size is critical; 128 doubles the file for a degree tail (measured mean - ~37). Confirmed empirically: cap-64 graphs reach recall@10 = 1.000 at 100K on the calibrated - corpus (§11). Cap remains a header field; revising it is a rebuild. +- **Degree cap: 128 for the int8 plane** (revised 2026-08-31 after measurement). The original + cap-64 preference assumed 128 doubles the file; it does not for int8 slots — the 768 B vector + dominates, so 128 costs +23.5% (1,344 vs 1,088 B slots). Measured at 1M: cap-64 loses 2.2 pts + of recall (0.975 vs 0.996, where JS = 0.997) at equal ef and equal latency. +24% bytes for + full recall parity is the right trade. The cap stays a header field; the **binary-code v2 + plane reopens the question** (cap-64 ≈ 352 B vs cap-128 ≈ 608 B slots, +73% — there a + diversity-preserving prune at lower cap is worth engineering). - **Platform policy.** Performance is a Linux target only. macOS must work (mmap/msync semantics differ slightly — `F_FULLFSYNC` for real durability barriers, no sparse-file guarantees on all filesystems — both handled, neither optimized). Windows may fall back to the JS implementation @@ -268,14 +272,22 @@ Gaussian-mixture corpus matching `benchmarks/hnsw-scale.js` calibration (intra-c clusters = N/500). JS baseline for scale: 4.34 µs/visit; 1M efC-200 anchor: p50 7.2 ms, recall@10-set 0.997, ~3,110 visits. -| N | p50 | p95 | visits/query | µs/visit | recall@10 (set) | build rate | -| --- | --- | --- | --- | --- | --- | --- | -| 100K | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | +| N | cap | p50 | p95 | visits/query | µs/visit | recall@10 (set) | build rate | +| --- | --- | --- | --- | --- | --- | --- | --- | +| 100K | 64 | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | +| 1M | 64 | 0.81 ms | 1.60 ms | 2,279 | 0.353 | 0.975 | 1,670 inserts/s | +| 1M | 128 | 0.75 ms | 1.48 ms | 2,309 | 0.324 | **0.996** | 1,242 inserts/s | +| 1M JS anchor | 128 | 7.2 ms | 12.0 ms | ~3,110 | 4.34 | 0.997 | ~263 inserts/s | + +At the 1M anchor with cap 128: **9.6× p50, 12.9× per-visit, 4.7× build rate, at JS-equal +recall.** The µs/visit rise from 100K (0.20) to 1M (0.32–0.35) is the working set leaving L3 — +the memory-hierarchy term; it is the number that holds at 60–100M. An ef-1024 sweep on a +reopened cap-64 plane without its hierarchy (pre-sidecar) still reached 0.985 at p50 2.47 ms — +layer-0 beam is robust to a missing hierarchy, at ~3.4× the visits. Milestones: zero-copy seqlock reads + AVX2 kernels took per-visit cost from 0.440 µs (first -scalar prototype) to ~0.1–0.2 µs — **~22–45× vs the JS per-visit baseline**, beating the -0.25–0.4 µs design budget. The optimizeRouting-parity insert (including the recomputed -neighbor↔neighbor distances) restored recall to 1.000 where the placeholder insert produced -unnavigable graphs. Uniform-random 768-d corpora produce meaningless recall numbers (the JS -benchmark's own calibration note: a corpus "no ANN can index") — all comparisons use the -mixture corpus. +scalar prototype) to ~0.1–0.35 µs, beating the 0.25–0.4 µs design budget. The +optimizeRouting-parity insert (including the recomputed neighbor↔neighbor distances) restored +recall from 0.49 (placeholder insert) to JS parity. Uniform-random 768-d corpora produce +meaningless recall numbers (the JS benchmark's own calibration note: a corpus "no ANN can +index") — all comparisons use the mixture corpus. From 94c7c6fd7aa58dcf38084e88181bef975dbf0537 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 06:51:44 -0600 Subject: [PATCH 06/69] hnsw-plane: NAPI surface, bitset filtering, page-grouped layout, concurrency fixes - napi feature (napi-rs v2): Plane class - create/open, insert/remove, async search on the libuv pool (AsyncTask, pooled scratches), searchSync, watermark get/set, flush (msync + upper sidecar). Harper-agnostic surface; pk<->id mapping and commit glue stay in the host. Build the NAPI artifact with --features napi --lib (the bench bin cannot link node-api symbols). - ACORN-style bitset filter: filtered-out nodes route but are excluded from results; visit budget = ef * filterExpansion bounds selective filters. - Page-grouped slot addressing when per-page waste <= 128 B (cap-128 slots: 3/page, 64 B waste, no page straddling); packed otherwise; header-pinned. - Concurrency fixes found by the torture test: edge RMW races (two-step read-then-write lost edges; now atomic via update_neighbors under the slot seqlock) and visited-array OOB when concurrent inserts mint ids past the query-start snapshot (writers panicked, wedging the run). - tests/concurrent.rs: 4 writers x 2000 inserts against 4 readers, then self-query/cap/freelist-reuse verification. Passes in 0.17s. - smoke.mjs: end-to-end through Node - PASSED on the fleet box. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/Cargo.toml | 11 +- native/hnsw-plane/build.rs | 6 + native/hnsw-plane/smoke.mjs | 44 +++++++ native/hnsw-plane/src/bin/bench.rs | 4 +- native/hnsw-plane/src/format.rs | 39 +++++- native/hnsw-plane/src/graph.rs | 32 +++++ native/hnsw-plane/src/insert.rs | 62 ++++----- native/hnsw-plane/src/lib.rs | 2 + native/hnsw-plane/src/napi.rs | 182 ++++++++++++++++++++++++++ native/hnsw-plane/src/search.rs | 69 +++++++++- native/hnsw-plane/tests/concurrent.rs | 109 +++++++++++++++ 11 files changed, 512 insertions(+), 48 deletions(-) create mode 100644 native/hnsw-plane/build.rs create mode 100644 native/hnsw-plane/smoke.mjs create mode 100644 native/hnsw-plane/src/napi.rs create mode 100644 native/hnsw-plane/tests/concurrent.rs diff --git a/native/hnsw-plane/Cargo.toml b/native/hnsw-plane/Cargo.toml index e0177bc169..7d47bb11b4 100644 --- a/native/hnsw-plane/Cargo.toml +++ b/native/hnsw-plane/Cargo.toml @@ -5,13 +5,20 @@ edition = "2021" description = "Native HNSW traversal plane: mmap fixed-slot graph file + off-loop search" license = "MIT" +[lib] +crate-type = ["cdylib", "rlib"] + [dependencies] memmap2 = "0.9" +napi = { version = "2", default-features = false, features = ["napi8"], optional = true } +napi-derive = { version = "2", optional = true } + +[build-dependencies] +napi-build = "2" [features] default = [] -# napi bindings added in phase 1 integration; core stays buildable standalone -# napi = ["dep:napi", "dep:napi-derive"] +napi = ["dep:napi", "dep:napi-derive"] [[bin]] name = "bench" diff --git a/native/hnsw-plane/build.rs b/native/hnsw-plane/build.rs new file mode 100644 index 0000000000..89463eb5e6 --- /dev/null +++ b/native/hnsw-plane/build.rs @@ -0,0 +1,6 @@ +fn main() { + // napi_build wires the node-api link args for the cdylib; only needed for the napi feature + if std::env::var("CARGO_FEATURE_NAPI").is_ok() { + napi_build::setup(); + } +} diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs new file mode 100644 index 0000000000..fcd3f369fa --- /dev/null +++ b/native/hnsw-plane/smoke.mjs @@ -0,0 +1,44 @@ +// NAPI smoke test: build with `cargo build --release --features napi --lib`, then +// (the bench bin cannot link against unresolved node-api symbols; build the lib alone) +// cp target/release/libhnsw_plane.so hnsw-plane.node && node smoke.mjs +import { createRequire } from 'module'; +const require = createRequire(import.meta.url); +const { Plane } = require('./hnsw-plane.node'); + +const dims = 64; +const path = `/tmp/smoke-${process.pid}.hnsw`; +const plane = Plane.create(path, dims, 32, 10_000); + +function vec(i) { + const v = new Float32Array(dims); + for (let d = 0; d < dims; d++) v[d] = Math.sin(i * 0.37 + d * 1.13) * 0.1 + (d % 7 === i % 7 ? 1 : 0); + return v; +} + +const ids = []; +for (let i = 0; i < 2000; i++) ids.push(plane.insert(vec(i))); +console.log('inserted 2000, highWater =', plane.idHighWater()); + +// async search: nearest neighbor of an inserted vector is itself (distance ~0) +const hits = await plane.search(vec(42), 5, 128); +console.log('top hit:', hits[0]); +if (hits[0].distance > 1e-3) throw new Error('self-query failed'); + +// filtered search: allow only even ids +const bitset = new Uint8Array(Math.ceil(plane.idHighWater() / 8)); +for (const id of ids) if (id % 2 === 0) bitset[id >> 3] |= 1 << (id & 7); +const filtered = await plane.search(vec(43), 5, 128, bitset); +for (const h of filtered) if (h.id % 2 !== 0) throw new Error(`filter leak: id ${h.id}`); +console.log('filtered top hit:', filtered[0]); + +// delete + reinsert reuses the id (the #2182 fix) +plane.remove(ids[7]); +const reused = plane.insert(vec(9001)); +if (reused !== ids[7]) throw new Error(`expected id reuse of ${ids[7]}, got ${reused}`); +console.log('freelist reuse OK, highWater still', plane.idHighWater()); + +plane.flush(); +const reopened = Plane.open(path); +const hits2 = reopened.searchSync(vec(42), 5, 128); +if (hits2[0].distance > 1e-3) throw new Error('reopened self-query failed'); +console.log('reopen + sidecar OK. smoke PASSED'); diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index f318de7d11..6f07c5696a 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -2,7 +2,7 @@ //! per-visit cost — the number that decides whether the native plane hits its 0.25–0.4 µs //! budget (JS baseline: 4.34 µs/visit at 5M/ef 512). //! -//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=64] +//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=128] use hnsw_plane::distance::Query; use hnsw_plane::insert::{insert, InsertParams}; @@ -83,7 +83,7 @@ fn main() { let queries: usize = args.get(3).and_then(|a| a.parse().ok()).unwrap_or(200); let ef: usize = args.get(4).and_then(|a| a.parse().ok()).unwrap_or(512); let path: PathBuf = args.get(5).map(Into::into).unwrap_or_else(|| "/tmp/bench.hnsw".into()); - let layer0_cap: usize = args.get(6).and_then(|a| a.parse().ok()).unwrap_or(64); + let layer0_cap: usize = args.get(6).and_then(|a| a.parse().ok()).unwrap_or(128); // Reuse an existing plane file when it already holds exactly n nodes at the same cap // (ef sweeps without rebuilding). The corpus RNG below replays identically. diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index ec241fa0a5..6f8c1a5e18 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -45,18 +45,42 @@ pub struct PlaneFile { pub dims: usize, pub layer0_cap: usize, pub slot_size: usize, + /// Slots per 4 KB page under page-grouped addressing; 0 = packed (slots may straddle + /// pages). Grouped is chosen at create when the per-page waste is small (e.g. 1,344 B + /// slots: 3/page, 64 B waste). Straddling only costs on cold faults, but the layout is + /// header-pinned so it must be decided before any data exists. + pub slots_per_page: usize, } +const PAGE: usize = 4096; +const H_SLOTS_PER_PAGE: usize = 20; // u16 + fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { let raw = S_VECTOR + dims + layer0_cap * 4; raw.next_multiple_of(64) // cache-line align } +fn slots_per_page_for(slot_size: usize) -> usize { + if slot_size > PAGE { + return 0; + } + let per = PAGE / slot_size; + let waste = PAGE - per * slot_size; + // group when waste is under ~3% of the page; otherwise pack + if waste <= 128 { per } else { 0 } +} + impl PlaneFile { /// Create a new plane file with capacity for `max_nodes` (sparse; pages materialize on write). pub fn create(path: &Path, dims: usize, layer0_cap: usize, max_nodes: u64) -> io::Result { let slot_size = slot_size_for(dims, layer0_cap); - let len = HEADER_SIZE as u64 + max_nodes * slot_size as u64; + let slots_per_page = slots_per_page_for(slot_size); + let data_len = if slots_per_page > 0 { + max_nodes.div_ceil(slots_per_page as u64) * PAGE as u64 + } else { + max_nodes * slot_size as u64 + }; + let len = HEADER_SIZE as u64 + data_len; let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; file.set_len(len)?; let mut map = unsafe { MmapMut::map_mut(&file)? }; @@ -66,10 +90,11 @@ impl PlaneFile { map[H_QUANT] = 0; map[H_LAYER0_CAP..H_LAYER0_CAP + 2].copy_from_slice(&(layer0_cap as u16).to_le_bytes()); map[H_SLOT_SIZE..H_SLOT_SIZE + 4].copy_from_slice(&(slot_size as u32).to_le_bytes()); + map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].copy_from_slice(&(slots_per_page as u16).to_le_bytes()); map[H_ENTRY_ID..H_ENTRY_ID + 4].copy_from_slice(&NO_ID.to_le_bytes()); map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); - Ok(PlaneFile { map, dims, layer0_cap, slot_size }) + Ok(PlaneFile { map, dims, layer0_cap, slot_size, slots_per_page }) } pub fn open(path: &Path) -> io::Result { @@ -83,12 +108,18 @@ impl PlaneFile { let dims = u16::from_le_bytes(map[H_DIMS..H_DIMS + 2].try_into().unwrap()) as usize; let layer0_cap = u16::from_le_bytes(map[H_LAYER0_CAP..H_LAYER0_CAP + 2].try_into().unwrap()) as usize; let slot_size = u32::from_le_bytes(map[H_SLOT_SIZE..H_SLOT_SIZE + 4].try_into().unwrap()) as usize; - Ok(PlaneFile { map, dims, layer0_cap, slot_size }) + let slots_per_page = u16::from_le_bytes(map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].try_into().unwrap()) as usize; + Ok(PlaneFile { map, dims, layer0_cap, slot_size, slots_per_page }) } #[inline] pub fn slot_ptr(&self, id: u32) -> *const u8 { - unsafe { self.map.as_ptr().add(HEADER_SIZE + id as usize * self.slot_size) } + let off = if self.slots_per_page > 0 { + (id as usize / self.slots_per_page) * PAGE + (id as usize % self.slots_per_page) * self.slot_size + } else { + id as usize * self.slot_size + }; + unsafe { self.map.as_ptr().add(HEADER_SIZE + off) } } #[inline] diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 75b3fd2ab8..6e71b89946 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -166,6 +166,38 @@ impl Graph { } } + /// Atomic read-modify-write of a node's layer-0 neighbor list under its seqlock. + /// `f` may read OTHER slots (e.g. distance_between for pruning) — those are plain + /// unlocked reads, so no lock ordering issue — but must not lock this graph's slots. + /// Two-step read-then-write callers race (concurrent reverse-edge adds lose edges); + /// all edge maintenance goes through here. Returns false for absent/deleted nodes. + pub fn update_neighbors)>(&self, id: u32, f: F) -> bool { + if !self.in_range(id) { + return false; + } + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq); + let p = self.file.slot_ptr_mut(id); + let dims = self.file.dims; + let cap = self.file.layer0_cap; + unsafe { + let flags = *p.add(S_FLAGS); + if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { + return false; + } + let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; + let base = p.add(S_VECTOR + dims) as *mut u32; + let mut list: Vec = (0..degree.min(cap)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); + f(&mut list); + list.truncate(cap); + (p.add(S_DEGREE) as *mut u16).write_unaligned((list.len() as u16).to_le()); + for (i, n) in list.iter().enumerate() { + base.add(i).write_unaligned(n.to_le()); + } + } + true + } + /// Replace only the neighbor list (back-edge maintenance path). pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { debug_assert!(neighbors.len() <= self.file.layer0_cap); diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 243d889815..bbf415cd45 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -34,13 +34,11 @@ fn level_for(id: u32, ml: f64) -> u8 { /// Remove `to` from `from`'s adjacency at `level` (edge-replacement maintenance). fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { if level == 0 { - if let Some(n) = graph.read_node(from) { - if let Some(pos) = n.neighbors.iter().position(|&x| x == to) { - let mut list = n.neighbors; + graph.update_neighbors(from, |list| { + if let Some(pos) = list.iter().position(|&x| x == to) { list.remove(pos); - graph.write_neighbors(from, &list); } - } + }); } else { let mut upper = graph.upper.write().unwrap(); if let Some(levels) = upper.get_mut(&from) { @@ -71,43 +69,36 @@ fn neighbors_at(graph: &Graph, id: u32, level: u8, buf: &mut Vec) { /// Add `new_id` to `nid`'s adjacency at `level`, pruning to `cap` closest when over. fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) { if level == 0 { - let Some(n) = graph.read_node(nid) else { return }; - let mut list = n.neighbors; - if list.contains(&new_id) { - return; - } - list.push(new_id); - if list.len() > cap { - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) - .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - list = scored.into_iter().take(cap).map(|(cand, _)| cand).collect(); - } - graph.write_neighbors(nid, &list); + graph.update_neighbors(nid, |list| { + if list.contains(&new_id) { + return; + } + list.push(new_id); + if list.len() > cap { + // distance_between reads other slots without locks; safe under this seqlock + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) + .collect(); + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + *list = scored.into_iter().take(cap).map(|(cand, _)| cand).collect(); + } + }); } else { + // held across the prune: upper mutations are rare (~6% of nodes) and the recomputed + // distances are ~cap * 0.2us — an acceptable hold for prototype correctness let mut upper = graph.upper.write().unwrap(); if let Some(levels) = upper.get_mut(&nid) { if let Some(list) = levels.get_mut(level as usize - 1) { if !list.contains(&new_id) { list.push(new_id); if list.len() > cap { - drop(upper); - // prune by recomputed distance outside the write lock - let mut scored: Vec<(u32, f32)> = { - let upper = graph.upper.read().unwrap(); - let list = upper.get(&nid).and_then(|l| l.get(level as usize - 1)).cloned().unwrap_or_default(); - list.iter().filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))).collect() - }; + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) + .collect(); scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - let pruned: Vec = scored.into_iter().take(cap).map(|(c, _)| c).collect(); - let mut upper = graph.upper.write().unwrap(); - if let Some(levels) = upper.get_mut(&nid) { - if let Some(list) = levels.get_mut(level as usize - 1) { - *list = pruned; - } - } + *list = scored.into_iter().take(cap).map(|(c, _)| c).collect(); } } } @@ -146,7 +137,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu for l in (0..=top).rev() { scratch_begin(graph, scratch); - let mut neighbors = search_layer(graph, &query, ep, ep_dist, params.ef_construction, l, scratch, &mut stats); + let mut neighbors = + search_layer(graph, &query, ep, ep_dist, params.ef_construction, l, scratch, &mut stats, None, u64::MAX); neighbors.truncate(m << 1); if let Some(&(best, best_d)) = neighbors.first() { ep = best; diff --git a/native/hnsw-plane/src/lib.rs b/native/hnsw-plane/src/lib.rs index 23ec07930c..654bdd4cdb 100644 --- a/native/hnsw-plane/src/lib.rs +++ b/native/hnsw-plane/src/lib.rs @@ -6,6 +6,8 @@ pub mod distance; pub mod format; pub mod graph; pub mod insert; +#[cfg(feature = "napi")] +mod napi; pub mod search; pub mod seqlock; diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs new file mode 100644 index 0000000000..052f61f06b --- /dev/null +++ b/native/hnsw-plane/src/napi.rs @@ -0,0 +1,182 @@ +//! NAPI surface (feature = "napi"). One boundary crossing per operation; searches run on +//! the libuv thread pool via AsyncTask so the JS event loop is never blocked (C1). +//! The surface is deliberately Harper-agnostic — pk↔id mapping, commit-callback glue, and +//! txnlog-anchored replay live in the host application. + +use crate::distance::Query; +use crate::insert::{insert, InsertParams}; +use crate::search::{search_filtered, SearchScratch}; +use crate::{Graph, PlaneFile}; +use napi::bindgen_prelude::*; +use napi_derive::napi; +use std::path::PathBuf; +use std::sync::{Arc, Mutex}; + +/// Pooled per-query scratch (the visited array is O(nodes); never allocate per query). +struct ScratchPool(Mutex>); + +impl ScratchPool { + fn take(&self) -> SearchScratch { + self.0.lock().unwrap().pop().unwrap_or_default() + } + fn put(&self, s: SearchScratch) { + let mut pool = self.0.lock().unwrap(); + if pool.len() < 64 { + pool.push(s); + } + } +} + +#[napi(object)] +pub struct SearchHit { + pub id: u32, + pub distance: f64, +} + +pub struct SearchTask { + graph: Arc, + pool: Arc, + query: Vec, + k: usize, + ef: usize, + filter: Option>, + filter_expansion: usize, +} + +#[napi] +impl Task for SearchTask { + type Output = Vec<(u32, f32)>; + type JsValue = Vec; + + fn compute(&mut self) -> Result { + let mut scratch = self.pool.take(); + let query = Query::new(std::mem::take(&mut self.query)); + let (hits, _stats) = search_filtered( + &self.graph, + &query, + self.k, + self.ef, + self.filter.as_deref(), + self.filter_expansion, + &mut scratch, + ); + self.pool.put(scratch); + Ok(hits) + } + + fn resolve(&mut self, _env: Env, output: Self::Output) -> Result { + Ok(output.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) + } +} + +#[napi] +pub struct Plane { + graph: Arc, + pool: Arc, + upper_path: PathBuf, + params: InsertParams, + // insert scratch, serialized: phase-1 hosts call insert from a single writer at a time + // per index (Harper's commit path); a Mutex keeps misuse safe rather than fast. + insert_scratch: Mutex, +} + +#[napi] +impl Plane { + /// Create a new plane file. `maxNodes` bounds the sparse reservation (pages materialize + /// on write). + #[napi(factory)] + pub fn create(path: String, dims: u32, layer0_cap: u32, max_nodes: f64) -> Result { + let file = PlaneFile::create(std::path::Path::new(&path), dims as usize, layer0_cap as usize, max_nodes as u64) + .map_err(|e| Error::from_reason(e.to_string()))?; + Ok(Self::wrap(file, &path)) + } + + /// Open an existing plane file and its upper-layer sidecar. + #[napi(factory)] + pub fn open(path: String) -> Result { + let file = PlaneFile::open(std::path::Path::new(&path)).map_err(|e| Error::from_reason(e.to_string()))?; + let plane = Self::wrap(file, &path); + plane.graph.load_upper(&plane.upper_path).map_err(|e| Error::from_reason(e.to_string()))?; + Ok(plane) + } + + fn wrap(file: PlaneFile, path: &str) -> Plane { + let upper_path = PathBuf::from(format!("{path}.upper")); + Plane { + graph: Arc::new(Graph::new(file)), + pool: Arc::new(ScratchPool(Mutex::new(Vec::new()))), + upper_path, + params: InsertParams::default(), + insert_scratch: Mutex::new(SearchScratch::new()), + } + } + + /// Insert a vector; returns the allocated node id (freelist ids are reused). + #[napi] + pub fn insert(&self, vector: Float32Array) -> Result { + let mut scratch = self.insert_scratch.lock().unwrap(); + Ok(insert(&self.graph, &vector, &self.params, &mut scratch)) + } + + /// Delete a node; its id returns to the freelist. + #[napi] + pub fn remove(&self, id: u32) { + self.graph.delete_node(id); + } + + /// Async k-NN search on the libuv thread pool. `filter` is an optional allow-bitset + /// over node ids (bit i of byte i>>3); filtered searches are visit-bounded by + /// ef * filterExpansion (default 24). + #[napi(ts_return_type = "Promise>")] + pub fn search( + &self, + vector: Float32Array, + k: u32, + ef: u32, + filter: Option, + filter_expansion: Option, + ) -> AsyncTask { + AsyncTask::new(SearchTask { + graph: self.graph.clone(), + pool: self.pool.clone(), + query: vector.to_vec(), + k: k as usize, + ef: ef as usize, + filter: filter.map(|f| f.to_vec()), + filter_expansion: filter_expansion.unwrap_or(24) as usize, + }) + } + + /// Synchronous search (benchmarks/tests; blocks the calling thread). + #[napi] + pub fn search_sync(&self, vector: Float32Array, k: u32, ef: u32) -> Vec { + let mut scratch = self.pool.take(); + let query = Query::new(vector.to_vec()); + let (hits, _) = search_filtered(&self.graph, &query, k as usize, ef as usize, None, 24, &mut scratch); + self.pool.put(scratch); + hits.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect() + } + + /// Lifetime id high-water (allocated ids, including freed ones awaiting reuse). + #[napi] + pub fn id_high_water(&self) -> f64 { + self.graph.file.id_high_water() as f64 + } + + #[napi] + pub fn get_watermark(&self) -> f64 { + self.graph.file.watermark() as f64 + } + + #[napi] + pub fn set_watermark(&self, txn: f64) { + self.graph.file.set_watermark(txn as u64); + } + + /// msync the plane and persist the upper-layer sidecar; advances durability. + #[napi] + pub fn flush(&self) -> Result<()> { + self.graph.save_upper(&self.upper_path).map_err(|e| Error::from_reason(e.to_string()))?; + self.graph.file.msync().map_err(|e| Error::from_reason(e.to_string())) + } +} diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 488833a634..40e1de1dbc 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -73,6 +73,10 @@ impl SearchScratch { #[inline] fn visit(&mut self, id: u32) -> bool { + // ids minted by concurrent inserts after begin() can exceed the sizing snapshot + if id as usize >= self.visited.len() { + self.visited.resize(id as usize + 1024, 0); + } let slot = &mut self.visited[id as usize]; if *slot == self.epoch { false @@ -93,9 +97,24 @@ pub struct SearchStats { pub visits: u64, } +#[inline] +fn bit_allowed(filter: Option<&[u8]>, id: u32) -> bool { + match filter { + None => true, + Some(bits) => { + let byte = (id >> 3) as usize; + byte < bits.len() && bits[byte] & (1 << (id & 7)) != 0 + } + } +} + /// Beam search within one layer, starting from `entry`. Level 0 reads slot adjacency; /// upper levels read the resident upper map. Returns (id, distance) ascending by distance. /// Assumes scratch.begin() was called for this query; entry is marked visited here. +/// +/// `filter`: optional allow-bitset over node ids (bit i of byte i>>3). Filtered-out nodes +/// are traversed (their edges route) but excluded from results — ACORN-style — with +/// `visit_budget` bounding total visits so a selective filter terminates. pub fn search_layer( graph: &Graph, query: &Query, @@ -105,12 +124,16 @@ pub fn search_layer( level: u8, scratch: &mut SearchScratch, stats: &mut SearchStats, + filter: Option<&[u8]>, + visit_budget: u64, ) -> Vec<(u32, f32)> { let mut candidates = BinaryHeap::new(); let mut results: BinaryHeap = BinaryHeap::new(); scratch.visit(entry); candidates.push(Candidate { distance: entry_dist, id: entry }); - results.push(Result_ { distance: entry_dist, id: entry }); + if bit_allowed(filter, entry) { + results.push(Result_ { distance: entry_dist, id: entry }); + } // take() the scratch neighbor buffer to sidestep the double-borrow of scratch let mut nbuf = std::mem::take(&mut scratch.neighbors); @@ -120,6 +143,9 @@ pub fn search_layer( if results.len() >= ef && c.distance > worst { break; } + if stats.visits >= visit_budget { + break; + } if level == 0 { if graph.neighbors_into(c.id, &mut nbuf).is_none() { continue; @@ -143,9 +169,11 @@ pub fn search_layer( let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); if results.len() < ef || d < worst { candidates.push(Candidate { distance: d, id: nid }); - results.push(Result_ { distance: d, id: nid }); - if results.len() > ef { - results.pop(); + if bit_allowed(filter, nid) { + results.push(Result_ { distance: d, id: nid }); + if results.len() > ef { + results.pop(); + } } } } @@ -219,7 +247,38 @@ pub fn search( None => return (Vec::new(), stats), }; let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); - let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats); + let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, None, u64::MAX); + out.truncate(k); + (out, stats) +} + +/// Full search with an optional allow-bitset filter. `filter_expansion` multiplies ef into +/// the visit budget when a filter is present (matching the JS filterExpansion semantics). +pub fn search_filtered( + graph: &Graph, + query: &Query, + k: usize, + ef: usize, + filter: Option<&[u8]>, + filter_expansion: usize, + scratch: &mut SearchScratch, +) -> (Vec<(u32, f32)>, SearchStats) { + let mut stats = SearchStats { visits: 0 }; + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + return (Vec::new(), stats); + } + scratch.begin_public(graph.file.id_high_water()); + let entry_dist = match graph.distance_to(entry_id, query) { + Some(d) => { + stats.visits += 1; + d + } + None => return (Vec::new(), stats), + }; + let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); + let budget = if filter.is_some() { (ef * filter_expansion) as u64 } else { u64::MAX }; + let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, filter, budget); out.truncate(k); (out, stats) } diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs new file mode 100644 index 0000000000..3c7f2720b6 --- /dev/null +++ b/native/hnsw-plane/tests/concurrent.rs @@ -0,0 +1,109 @@ +//! Concurrent-write torture: writers insert while readers search; then verify the graph is +//! coherent (every stored vector findable, edge lists within cap, freelist reuse works). + +use hnsw_plane::distance::Query; +use hnsw_plane::insert::{insert, InsertParams}; +use hnsw_plane::search::{search, SearchScratch}; +use hnsw_plane::{Graph, PlaneFile}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Arc; + +fn vector_for(i: u32, dims: usize) -> Vec { + // deterministic distinct unit-ish vectors on a few clusters + let mut v = vec![0.0f32; dims]; + let cluster = (i % 7) as usize; + for d in 0..dims { + let x = ((i as f32 * 0.37 + d as f32 * 1.13).sin() * 0.1) + if d % 7 == cluster { 1.0 } else { 0.0 }; + v[d] = x; + } + v +} + +#[test] +fn concurrent_insert_search() { + let dims = 64; + let path = std::env::temp_dir().join(format!("hnsw-torture-{}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let file = PlaneFile::create(&path, dims, 32, 40_000).expect("create"); + let graph = Arc::new(Graph::new(file)); + + let writers = 4u32; + let per_writer = 2_000u32; + let done = Arc::new(AtomicBool::new(false)); + + std::thread::scope(|s| { + for w in 0..writers { + let graph = graph.clone(); + s.spawn(move || { + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..per_writer { + let v = vector_for(w * per_writer + i, dims); + insert(&graph, &v, ¶ms, &mut scratch); + } + }); + } + for _ in 0..4 { + let graph = graph.clone(); + let done = done.clone(); + s.spawn(move || { + let mut scratch = SearchScratch::new(); + let mut q = 0u32; + while !done.load(Ordering::Relaxed) { + let query = Query::new(vector_for(q % 1000, dims)); + let (results, _) = search(&graph, &query, 10, 64, &mut scratch); + // once anything is inserted, results must be non-empty and finite + for (_, d) in &results { + assert!(d.is_finite()); + } + q += 1; + } + }); + } + // scope joins writers when their closures end; signal readers afterward via a + // dedicated waiter thread + let graph_ref = graph.clone(); + let done_ref = done.clone(); + s.spawn(move || { + while graph_ref.file.id_high_water() < (writers * per_writer) as u64 { + std::thread::yield_now(); + } + done_ref.store(true, Ordering::Relaxed); + }); + }); + + let total = writers * per_writer; + assert_eq!(graph.file.id_high_water(), total as u64); + + // Every stored vector must be found as its own nearest neighbor at generous ef. + let mut scratch = SearchScratch::new(); + let mut misses = 0; + for i in (0..total).step_by(97) { + let query = Query::new(vector_for(i, dims)); + let (results, _) = search(&graph, &query, 10, 256, &mut scratch); + // identical vectors exist across ids (clusters), so accept any zero-ish distance hit + if !results.iter().any(|&(_, d)| d < 1e-3) { + misses += 1; + } + } + assert_eq!(misses, 0, "self-queries missing after concurrent build"); + + // Edge lists respect the cap. + for id in (0..total).step_by(53) { + if let Some(n) = graph.read_node(id) { + assert!(n.neighbors.len() <= graph.file.layer0_cap); + } + } + + // Delete + reinsert reuses ids (freelist; the #2182 fix). + graph.delete_node(5); + graph.delete_node(6); + let params = InsertParams::default(); + let a = insert(&graph, &vector_for(90_001, dims), ¶ms, &mut scratch); + let b = insert(&graph, &vector_for(90_002, dims), ¶ms, &mut scratch); + assert!(a == 5 || a == 6, "expected freelist reuse, got {a}"); + assert!(b == 5 || b == 6, "expected freelist reuse, got {b}"); + assert_eq!(graph.file.id_high_water(), total as u64, "high-water must not grow on reuse"); + + let _ = std::fs::remove_file(&path); +} From 051d9aa6b46c072e173a4568b59388527b3acc78 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 08:23:30 -0600 Subject: [PATCH 07/69] hnsw-plane: pipelined TSFN predicate filtering search_predicated: candidate ids batch (64/batch) to an external evaluator while traversal keeps expanding in distance order - the search thread never blocks on the JS event loop until the beam is done; verdicts merge in as they arrive and gate result admission only. Visit budget ef*filterExpansion bounds speculative overshoot; a 5s drain deadline treats missing verdicts (predicate error, env teardown) as deny. Core is napi-free (channel-based PredicatePipe) with a pure-Rust mock-evaluator test; the NAPI layer wires a ThreadsafeFunction (searchWithPredicate: predicate(ids) => Uint8Array). Smoke through Node: 18 predicate batches for one ef-128 query, no leaks. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/smoke.mjs | 16 +++ native/hnsw-plane/src/napi.rs | 83 +++++++++++++- native/hnsw-plane/src/search.rs | 194 ++++++++++++++++++++++++++++++++ 3 files changed, 292 insertions(+), 1 deletion(-) diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs index fcd3f369fa..f5911b95e4 100644 --- a/native/hnsw-plane/smoke.mjs +++ b/native/hnsw-plane/smoke.mjs @@ -37,6 +37,22 @@ const reused = plane.insert(vec(9001)); if (reused !== ids[7]) throw new Error(`expected id reuse of ${ids[7]}, got ${reused}`); console.log('freelist reuse OK, highWater still', plane.idHighWater()); +// pipelined JS predicate: admit only ids divisible by 3; verdicts computed on the JS +// event loop while traversal runs on the libuv pool +let predicateCalls = 0; +const pred = await plane.searchWithPredicate( + vec(44), + 5, + 128, + (ids) => { + predicateCalls++; + return Uint8Array.from(ids, (id) => (id % 3 === 0 ? 1 : 0)); + } +); +for (const h of pred) if (h.id % 3 !== 0) throw new Error(`predicate leak: id ${h.id}`); +if (pred.length === 0) throw new Error('predicate search returned nothing'); +console.log(`predicate top hit: id ${pred[0].id} (calls: ${predicateCalls})`); + plane.flush(); const reopened = Plane.open(path); const hits2 = reopened.searchSync(vec(42), 5, 128); diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 052f61f06b..df4f6e0479 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -5,9 +5,11 @@ use crate::distance::Query; use crate::insert::{insert, InsertParams}; -use crate::search::{search_filtered, SearchScratch}; +use crate::search::{search_filtered, search_predicated, PredicatePipe, SearchScratch}; use crate::{Graph, PlaneFile}; use napi::bindgen_prelude::*; +use napi::threadsafe_function::{ErrorStrategy, ThreadsafeFunction, ThreadsafeFunctionCallMode}; +use napi::JsFunction; use napi_derive::napi; use std::path::PathBuf; use std::sync::{Arc, Mutex}; @@ -69,6 +71,54 @@ impl Task for SearchTask { } } +pub struct PredicateSearchTask { + graph: Arc, + pool: Arc, + query: Vec, + k: usize, + ef: usize, + tsfn: Option, ErrorStrategy::Fatal>>, + filter_expansion: usize, +} + +#[napi] +impl Task for PredicateSearchTask { + type Output = Vec<(u32, f32)>; + type JsValue = Vec; + + fn compute(&mut self) -> Result { + let tsfn = self.tsfn.take().ok_or_else(|| Error::from_reason("task reused"))?; + let (tx, rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); + let mut pipe = PredicatePipe { + dispatch: Box::new(move |ids: Vec| { + let tx = tx.clone(); + let ids_echo = ids.clone(); + tsfn.call_with_return_value( + ids, + ThreadsafeFunctionCallMode::NonBlocking, + move |ret: Uint8Array| { + // predicate errors / env teardown surface as a missing send; the + // drain deadline in search_predicated treats absent verdicts as deny + let _ = tx.send((ids_echo, ret.to_vec())); + Ok(()) + }, + ); + }), + rx, + }; + let mut scratch = self.pool.take(); + let query = Query::new(std::mem::take(&mut self.query)); + let (hits, _stats) = + search_predicated(&self.graph, &query, self.k, self.ef, &mut pipe, self.filter_expansion, &mut scratch); + self.pool.put(scratch); + Ok(hits) + } + + fn resolve(&mut self, _env: Env, output: Self::Output) -> Result { + Ok(output.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) + } +} + #[napi] pub struct Plane { graph: Arc, @@ -147,6 +197,37 @@ impl Plane { }) } + /// Async k-NN search with a JS predicate: `predicate(ids: number[]) => Uint8Array` + /// (one 0/1 byte per id, evaluated synchronously). Batches of candidate ids stream to + /// the predicate over a ThreadsafeFunction while traversal keeps expanding — the search + /// thread never blocks on the JS event loop until the beam itself is done, so a busy + /// loop costs speculative overshoot (bounded by ef * filterExpansion), not latency. + /// Must not be awaited synchronously from code the predicate itself blocks. + #[napi(ts_return_type = "Promise>")] + pub fn search_with_predicate( + &self, + vector: Float32Array, + k: u32, + ef: u32, + #[napi(ts_arg_type = "(ids: Array) => Uint8Array")] predicate: JsFunction, + filter_expansion: Option, + ) -> Result> { + let tsfn: ThreadsafeFunction, ErrorStrategy::Fatal> = predicate + .create_threadsafe_function(0, |ctx: napi::threadsafe_function::ThreadSafeCallContext>| { + let ids: Vec = ctx.value.iter().map(|&v| v as f64).collect(); + Ok(vec![ids]) + })?; + Ok(AsyncTask::new(PredicateSearchTask { + graph: self.graph.clone(), + pool: self.pool.clone(), + query: vector.to_vec(), + k: k as usize, + ef: ef as usize, + tsfn: Some(tsfn), + filter_expansion: filter_expansion.unwrap_or(24) as usize, + })) + } + /// Synchronous search (benchmarks/tests; blocks the calling thread). #[napi] pub fn search_sync(&self, vector: Float32Array, k: u32, ef: u32) -> Vec { diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 40e1de1dbc..5febb42dd7 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -282,3 +282,197 @@ pub fn search_filtered( out.truncate(k); (out, stats) } + +/// Pipelined predicate filtering: candidate ids are batched to an external evaluator (the +/// NAPI layer wires this to a JS ThreadsafeFunction) while traversal continues expanding — +/// the search thread never blocks on the evaluator until the beam itself is done. Verdicts +/// steer result admission only; routing uses pure distance order, bounded by the visit +/// budget, so a slow or saturated JS loop degrades speculative overshoot, not correctness. +pub struct PredicatePipe { + /// Sends one batch of ids for evaluation. Must not block. + pub dispatch: Box) + Send>, + /// Receives (ids, verdicts) pairs; verdicts[i] != 0 admits ids[i]. + pub rx: std::sync::mpsc::Receiver<(Vec, Vec)>, +} + +const PREDICATE_BATCH: usize = 64; +const DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); + +/// Full search with a pipelined predicate filter (upper-layer descent is unfiltered, as in +/// the JS implementation — predicates gate results, not routing). +pub fn search_predicated( + graph: &Graph, + query: &Query, + k: usize, + ef: usize, + pipe: &mut PredicatePipe, + filter_expansion: usize, + scratch: &mut SearchScratch, +) -> (Vec<(u32, f32)>, SearchStats) { + let mut stats = SearchStats { visits: 0 }; + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + return (Vec::new(), stats); + } + scratch.begin_public(graph.file.id_high_water()); + let entry_dist = match graph.distance_to(entry_id, query) { + Some(d) => { + stats.visits += 1; + d + } + None => return (Vec::new(), stats), + }; + let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); + let visit_budget = (ef * filter_expansion) as u64; + + use std::collections::HashMap; + let mut verdicts: HashMap = HashMap::new(); + let mut speculative: Vec<(u32, f32)> = Vec::new(); // awaiting verdicts + let mut batch: Vec = Vec::new(); + let mut outstanding = 0usize; + + let mut candidates = BinaryHeap::new(); + let mut results: BinaryHeap = BinaryHeap::new(); + scratch.visit(ep); + candidates.push(Candidate { distance: ep_dist, id: ep }); + speculative.push((ep, ep_dist)); + batch.push(ep); + + let mut nbuf = std::mem::take(&mut scratch.neighbors); + + macro_rules! drain { + ($recv:expr) => { + while let Ok((ids, flags)) = $recv { + outstanding -= 1; + for (i, id) in ids.iter().enumerate() { + verdicts.insert(*id, flags.get(i).copied().unwrap_or(0) != 0); + } + } + }; + } + + loop { + // non-blocking verdict intake each iteration + drain!(pipe.rx.try_recv()); + if !verdicts.is_empty() && !speculative.is_empty() { + speculative.retain(|&(id, d)| match verdicts.get(&id) { + Some(true) => { + results.push(Result_ { distance: d, id }); + if results.len() > ef { + results.pop(); + } + false + } + Some(false) => false, + None => true, + }); + } + + let Some(c) = candidates.pop() else { break }; + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() >= ef && c.distance > worst { + break; + } + if stats.visits >= visit_budget { + break; + } + if graph.neighbors_into(c.id, &mut nbuf).is_none() { + continue; + } + for i in 0..nbuf.len() { + let nid = nbuf[i]; + if !scratch.visit(nid) { + continue; + } + if let Some(d) = graph.distance_to(nid, query) { + stats.visits += 1; + let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); + if results.len() < ef || d < worst { + candidates.push(Candidate { distance: d, id: nid }); + speculative.push((nid, d)); + batch.push(nid); + if batch.len() >= PREDICATE_BATCH { + (pipe.dispatch)(std::mem::take(&mut batch)); + outstanding += 1; + } + } + } + } + } + scratch.neighbors = nbuf; + + // flush the tail batch and block-drain what's still in flight + if !batch.is_empty() { + (pipe.dispatch)(std::mem::take(&mut batch)); + outstanding += 1; + } + let deadline = std::time::Instant::now() + DRAIN_TIMEOUT; + while outstanding > 0 && std::time::Instant::now() < deadline { + drain!(pipe.rx.recv_timeout(std::time::Duration::from_millis(50))); + } + speculative.retain(|&(id, d)| { + if verdicts.get(&id).copied().unwrap_or(false) { + results.push(Result_ { distance: d, id }); + if results.len() > ef { + results.pop(); + } + } + false + }); + + let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); + out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + out.truncate(k); + (out, stats) +} + +#[cfg(test)] +mod predicate_tests { + use super::*; + use crate::insert::{insert, InsertParams}; + use crate::PlaneFile; + + #[test] + fn pipelined_predicate_filters_results() { + let dims = 32; + let path = std::env::temp_dir().join(format!("hnsw-pred-{}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let file = PlaneFile::create(&path, dims, 16, 4_096).expect("create"); + let graph = Graph::new(file); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..1_000u32 { + let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + insert(&graph, &v, ¶ms, &mut scratch); + } + + // evaluator thread: admit even ids only, answering over a channel like the TSFN does + let (req_tx, req_rx) = std::sync::mpsc::channel::>(); + let (res_tx, res_rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); + let worker = std::thread::spawn(move || { + while let Ok(ids) = req_rx.recv() { + let verdicts: Vec = ids.iter().map(|id| (id % 2 == 0) as u8).collect(); + if res_tx.send((ids, verdicts)).is_err() { + break; + } + } + }); + + let mut pipe = PredicatePipe { + dispatch: Box::new(move |ids| { + let _ = req_tx.send(ids); + }), + rx: res_rx, + }; + let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + let (hits, _) = + search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 24, &mut scratch); + assert!(!hits.is_empty()); + for (id, _) in &hits { + assert_eq!(id % 2, 0, "odd id {id} leaked through the predicate"); + } + drop(pipe); + worker.join().unwrap(); + let _ = std::fs::remove_file(&path); + } +} From 4afa9e8ac7abb464df3f8343dc64f62a113634ba Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 08:29:47 -0600 Subject: [PATCH 08/69] hnsw-plane: in-file upper-layer region (format v2) The hierarchy moves from an in-memory map + sidecar into a fixed-entry region of the plane file itself: per-entry seqlocks, 8-level x 32-id entries reserved for 1/8 of max_nodes (2x the expected 1/M upper-node rate; exhaustion degrades to missing upper links, never an error). Slots gain S_UPPER_IDX. Removes the global RwLock every query's greedy descent contended on, the sidecar files, and the reopen hierarchy gap. Upper entries leak on delete (bounded by the reserve; freelist TODO). Format VERSION bumped to 2 - v1 files reindex. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/bin/bench.rs | 10 +- native/hnsw-plane/src/format.rs | 81 ++++++++++-- native/hnsw-plane/src/graph.rs | 205 ++++++++++++++++++----------- native/hnsw-plane/src/insert.rs | 77 +++++------ native/hnsw-plane/src/napi.rs | 17 +-- native/hnsw-plane/src/search.rs | 19 +-- 6 files changed, 241 insertions(+), 168 deletions(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index 6f07c5696a..d0d5fcf27c 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -110,16 +110,12 @@ fn main() { let mut rng = Rng(0x1234_5678_9abc_def0); let corpus = Corpus::new(n, dims, &mut rng); - let upper_path = path.with_extension("hnsw.upper"); if reuse { - // replay the build's RNG draws so query rows match a fresh run, and load the - // persisted upper-layer sidecar (missing sidecar = layer-0-only search). + // replay the build's RNG draws so query rows match a fresh run; the upper region + // persists inside the plane file for _ in 0..n { let _ = corpus.row(&mut rng); } - graph.load_upper(&upper_path).expect("load upper sidecar"); - let count = graph.upper.read().unwrap().len(); - println!("upper layers loaded: {} nodes with level > 0", count); } else { let build_start = Instant::now(); for i in 0..n { @@ -132,7 +128,7 @@ fn main() { } let build = build_start.elapsed(); println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); - graph.save_upper(&upper_path).expect("save upper sidecar"); + graph.file.msync().expect("msync"); } // Query with held-out vectors; measure latency and set-recall@10 vs brute-force truth diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 6f8c1a5e18..ef2487d3b4 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -8,7 +8,7 @@ use std::path::Path; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 1; +pub const VERSION: u32 = 2; // v2: in-file upper region + S_UPPER_IDX (v1 files: reindex) pub const HEADER_SIZE: usize = 4096; // Header field byte offsets. @@ -24,6 +24,20 @@ const H_ID_HIGH_WATER: usize = 32; // u64 atomic const H_FREELIST_HEAD: usize = 40; // u64 atomic: (tag << 32) | id; id u32::MAX = empty const H_TXN_WATERMARK: usize = 48; // u64 const H_CLEAN_SHUTDOWN: usize = 56; // u8 +const H_MAX_NODES: usize = 64; // u64 +const H_UPPER_HIGH_WATER: usize = 72; // u64 atomic: upper-entry allocator + +/// Upper-layer region geometry: fixed entries covering levels 1..=MAX_UPPER_LEVELS at +/// UPPER_CAP ids per level. P(level >= 1) = 1/M ~ 6.25%; the region reserves entries for +/// 1/8 of max_nodes (2x headroom). P(level >= 9) at mL = 1/ln16 is ~e^-25 — unreachable. +pub const MAX_UPPER_LEVELS: usize = 8; +pub const UPPER_CAP: usize = 32; +// entry: seq u32 | levels u8 | pad | per-level (degree u16 + ids u32*UPPER_CAP) +pub const U_SEQ: usize = 0; +pub const U_LEVELS: usize = 4; +pub const U_LISTS: usize = 8; +pub const UPPER_LEVEL_STRIDE: usize = 2 + UPPER_CAP * 4 + 2; // degree + ids + pad -> 132 +pub const NO_UPPER: u32 = u32::MAX; // Slot layout offsets (within a slot). pub const S_SEQ: usize = 0; // u32 seqlock @@ -32,7 +46,8 @@ pub const S_LEVEL: usize = 5; // u8 pub const S_DEGREE: usize = 6; // u16 pub const S_SCALE: usize = 8; // f32 pub const S_INV_MAG: usize = 12; // f32 -pub const S_VECTOR: usize = 16; // dims bytes (int8) or dims*4 (f32) +pub const S_UPPER_IDX: usize = 16; // u32 index into the upper region; NO_UPPER = none +pub const S_VECTOR: usize = 20; // dims bytes (int8) or dims*4 (f32) // neighbors: u32 * layer0_cap, follows vector // deleted slots reuse the first neighbor word as freelist next-pointer @@ -45,6 +60,9 @@ pub struct PlaneFile { pub dims: usize, pub layer0_cap: usize, pub slot_size: usize, + pub max_nodes: u64, + upper_offset: usize, + pub upper_capacity: u64, /// Slots per 4 KB page under page-grouped addressing; 0 = packed (slots may straddle /// pages). Grouped is chosen at create when the per-page waste is small (e.g. 1,344 B /// slots: 3/page, 64 B waste). Straddling only costs on cold faults, but the layout is @@ -60,6 +78,18 @@ fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { raw.next_multiple_of(64) // cache-line align } +fn upper_entry_size() -> usize { + (U_LISTS + MAX_UPPER_LEVELS * UPPER_LEVEL_STRIDE).next_multiple_of(64) +} + +fn slot_region_len(max_nodes: u64, slot_size: usize, slots_per_page: usize) -> u64 { + if slots_per_page > 0 { + max_nodes.div_ceil(slots_per_page as u64) * PAGE as u64 + } else { + max_nodes * slot_size as u64 + } +} + fn slots_per_page_for(slot_size: usize) -> usize { if slot_size > PAGE { return 0; @@ -75,12 +105,9 @@ impl PlaneFile { pub fn create(path: &Path, dims: usize, layer0_cap: usize, max_nodes: u64) -> io::Result { let slot_size = slot_size_for(dims, layer0_cap); let slots_per_page = slots_per_page_for(slot_size); - let data_len = if slots_per_page > 0 { - max_nodes.div_ceil(slots_per_page as u64) * PAGE as u64 - } else { - max_nodes * slot_size as u64 - }; - let len = HEADER_SIZE as u64 + data_len; + let data_len = slot_region_len(max_nodes, slot_size, slots_per_page); + let upper_capacity = max_nodes / 8 + 64; + let len = HEADER_SIZE as u64 + data_len + upper_capacity * upper_entry_size() as u64; let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; file.set_len(len)?; let mut map = unsafe { MmapMut::map_mut(&file)? }; @@ -94,7 +121,9 @@ impl PlaneFile { map[H_ENTRY_ID..H_ENTRY_ID + 4].copy_from_slice(&NO_ID.to_le_bytes()); map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); - Ok(PlaneFile { map, dims, layer0_cap, slot_size, slots_per_page }) + map[H_MAX_NODES..H_MAX_NODES + 8].copy_from_slice(&max_nodes.to_le_bytes()); + let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; + Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }) } pub fn open(path: &Path) -> io::Result { @@ -109,7 +138,10 @@ impl PlaneFile { let layer0_cap = u16::from_le_bytes(map[H_LAYER0_CAP..H_LAYER0_CAP + 2].try_into().unwrap()) as usize; let slot_size = u32::from_le_bytes(map[H_SLOT_SIZE..H_SLOT_SIZE + 4].try_into().unwrap()) as usize; let slots_per_page = u16::from_le_bytes(map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].try_into().unwrap()) as usize; - Ok(PlaneFile { map, dims, layer0_cap, slot_size, slots_per_page }) + let max_nodes = u64::from_le_bytes(map[H_MAX_NODES..H_MAX_NODES + 8].try_into().unwrap()); + let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; + let upper_capacity = max_nodes / 8 + 64; + Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }) } #[inline] @@ -202,6 +234,35 @@ impl PlaneFile { self.header_atomic_u64(H_TXN_WATERMARK).load(Ordering::Acquire) } + #[inline] + pub fn upper_ptr(&self, idx: u32) -> *const u8 { + debug_assert!((idx as u64) < self.upper_capacity); + unsafe { self.map.as_ptr().add(self.upper_offset + idx as usize * upper_entry_size()) } + } + + #[inline] + pub fn upper_ptr_mut(&self, idx: u32) -> *mut u8 { + self.upper_ptr(idx) as *mut u8 + } + + #[inline] + pub fn upper_seq_atomic(&self, idx: u32) -> &AtomicU32 { + unsafe { &*(self.upper_ptr(idx).add(U_SEQ) as *const AtomicU32) } + } + + /// Allocate an upper-region entry. Entries are not freed on delete (bounded leak within + /// the 2x-headroom reserve; freelist reuse is a TODO). Returns NO_UPPER when exhausted — + /// the node then simply has no upper links, which degrades routing, not correctness. + pub fn allocate_upper(&self) -> u32 { + let hw = self.header_atomic_u64(H_UPPER_HIGH_WATER); + let idx = hw.fetch_add(1, Ordering::AcqRel); + if idx >= self.upper_capacity { + hw.fetch_sub(1, Ordering::AcqRel); + return NO_UPPER; + } + idx as u32 + } + pub fn set_clean_shutdown(&mut self, clean: bool) { self.map[H_CLEAN_SHUTDOWN] = clean as u8; } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 6e71b89946..487061a39a 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -1,18 +1,18 @@ -//! Slot-level node access over the plane file, mediated by the seqlock, plus the resident -//! upper-layer structure. Hot-path reads (distance, neighbor ids) are zero-copy against the -//! mmap; full-copy read_node exists for construction paths. Prototype status: upper layers -//! live in memory; the append-allocated file region from the design doc is a TODO. +//! Slot-level node access over the plane file, mediated by per-slot seqlocks. Hot-path +//! reads (distance, neighbor ids) are zero-copy against the mmap; full-copy read_node +//! exists for construction paths. Upper-layer adjacency lives in a fixed-entry region of +//! the same file (per-entry seqlocks), so the hierarchy persists with the graph and +//! concurrent searches share nothing mutable. use crate::distance::{cosine_i8_i8_raw, cosine_int8_raw, Query}; -use crate::format::{PlaneFile, FLAG_DELETED, FLAG_VALID, S_DEGREE, S_FLAGS, S_INV_MAG, S_LEVEL, S_SCALE, S_VECTOR}; +use crate::format::{ + PlaneFile, FLAG_DELETED, FLAG_VALID, MAX_UPPER_LEVELS, NO_UPPER, S_DEGREE, S_FLAGS, S_INV_MAG, S_LEVEL, S_SCALE, + S_UPPER_IDX, S_VECTOR, UPPER_CAP, UPPER_LEVEL_STRIDE, U_LEVELS, U_LISTS, +}; use crate::seqlock; -use std::collections::HashMap; -use std::sync::RwLock; pub struct Graph { pub file: PlaneFile, - /// Upper-layer adjacency: node id -> [neighbors at level 1, level 2, ...]. ~6% of nodes. - pub upper: RwLock>>>, } /// A consistent full copy of one node (construction paths only; search uses zero-copy). @@ -26,7 +26,7 @@ pub struct NodeRead { impl Graph { pub fn new(file: PlaneFile) -> Self { - Graph { file, upper: RwLock::new(HashMap::new()) } + Graph { file } } #[inline] @@ -56,12 +56,11 @@ impl Graph { } /// Symmetric stored-to-stored distance (construction-time neighbor↔neighbor checks). + /// Plain unlocked reads: a torn read only perturbs a construction heuristic. pub fn distance_between(&self, a: u32, b: u32) -> Option { if !self.in_range(a) || !self.in_range(b) { return None; } - // Two independent seqlock reads: copy a's params + vector ptr safely by nesting reads. - // A torn cross-pair read is acceptable here (construction heuristic, not a result). let dims = self.file.dims; let pa = self.file.slot_ptr(a); let pb = self.file.slot_ptr(b); @@ -117,6 +116,112 @@ impl Graph { }) } + /// The node's upper-region entry index, or NO_UPPER. + #[inline] + fn upper_idx_of(&self, id: u32) -> u32 { + if !self.in_range(id) { + return NO_UPPER; + } + let seq = self.file.seq_atomic(id); + seqlock::read_consistent(seq, || { + let p = self.file.slot_ptr(id); + unsafe { + let flags = *p.add(S_FLAGS); + if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { + return NO_UPPER; + } + (p.add(S_UPPER_IDX) as *const u32).read_unaligned() + } + }) + } + + /// Copy `id`'s neighbor ids at upper `level` (1-based) into `out`. False when the node + /// has no upper entry or no such level. + pub fn upper_neighbors_into(&self, id: u32, level: u8, out: &mut Vec) -> bool { + out.clear(); + debug_assert!(level >= 1); + let idx = self.upper_idx_of(id); + if idx == NO_UPPER || level as usize > MAX_UPPER_LEVELS { + return false; + } + let seq = self.file.upper_seq_atomic(idx); + seqlock::read_consistent(seq, || { + out.clear(); + let p = self.file.upper_ptr(idx); + unsafe { + let levels = *p.add(U_LEVELS); + if level > levels { + return false; + } + let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); + let degree = u16::from_le((lp as *const u16).read_unaligned()) as usize; + let base = lp.add(2) as *const u32; + for i in 0..degree.min(UPPER_CAP) { + out.push(u32::from_le(base.add(i).read_unaligned())); + } + true + } + }) + } + + /// Write a node's full upper adjacency into a fresh region entry; returns the entry + /// index to store in the slot (NO_UPPER when the region is exhausted or levels is empty). + pub fn write_upper(&self, levels: &[Vec]) -> u32 { + if levels.is_empty() { + return NO_UPPER; + } + let idx = self.file.allocate_upper(); + if idx == NO_UPPER { + return NO_UPPER; + } + let seq = self.file.upper_seq_atomic(idx); + let _guard = seqlock::write_lock(seq); + let p = self.file.upper_ptr_mut(idx); + unsafe { + let n = levels.len().min(MAX_UPPER_LEVELS); + *p.add(U_LEVELS) = n as u8; + for (l, list) in levels.iter().take(n).enumerate() { + let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); + let deg = list.len().min(UPPER_CAP); + (lp as *mut u16).write_unaligned((deg as u16).to_le()); + let base = lp.add(2) as *mut u32; + for (i, id) in list.iter().take(deg).enumerate() { + base.add(i).write_unaligned(id.to_le()); + } + } + } + idx + } + + /// Atomic read-modify-write of `id`'s upper adjacency at `level` (1-based). Returns + /// false when the node has no entry or level. `f` may read other slots. + pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> bool { + let idx = self.upper_idx_of(id); + if idx == NO_UPPER || level as usize > MAX_UPPER_LEVELS { + return false; + } + let seq = self.file.upper_seq_atomic(idx); + let _guard = seqlock::write_lock(seq); + let p = self.file.upper_ptr_mut(idx); + unsafe { + let levels = *p.add(U_LEVELS); + if level > levels { + return false; + } + let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); + let degree = u16::from_le((lp as *const u16).read_unaligned()) as usize; + let base = lp.add(2) as *mut u32; + let mut list: Vec = (0..degree.min(UPPER_CAP)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); + f(&mut list); + list.truncate(UPPER_CAP); + (lp as *mut u16).write_unaligned((list.len() as u16).to_le()); + for (i, id) in list.iter().enumerate() { + base.add(i).write_unaligned(id.to_le()); + } + } + true + } + /// Seqlock-consistent full copy (construction paths). pub fn read_node(&self, id: u32) -> Option { if !self.in_range(id) { @@ -144,8 +249,9 @@ impl Graph { }) } - /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the caller. - pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32]) { + /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the + /// caller; `upper_idx` is a write_upper() result (NO_UPPER for level-0 nodes). + pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32], upper_idx: u32) { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); let seq = self.file.seq_atomic(id); @@ -157,6 +263,7 @@ impl Graph { (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); (p.add(S_SCALE) as *mut f32).write_unaligned(scale); (p.add(S_INV_MAG) as *mut f32).write_unaligned(inv_mag); + (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); for (i, n) in neighbors.iter().enumerate() { (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); @@ -169,8 +276,7 @@ impl Graph { /// Atomic read-modify-write of a node's layer-0 neighbor list under its seqlock. /// `f` may read OTHER slots (e.g. distance_between for pruning) — those are plain /// unlocked reads, so no lock ordering issue — but must not lock this graph's slots. - /// Two-step read-then-write callers race (concurrent reverse-edge adds lose edges); - /// all edge maintenance goes through here. Returns false for absent/deleted nodes. + /// Returns false for absent/deleted nodes. pub fn update_neighbors)>(&self, id: u32, f: F) -> bool { if !self.in_range(id) { return false; @@ -198,7 +304,7 @@ impl Graph { true } - /// Replace only the neighbor list (back-edge maintenance path). + /// Replace only the neighbor list (single-writer construction path). pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { debug_assert!(neighbors.len() <= self.file.layer0_cap); let seq = self.file.seq_atomic(id); @@ -213,75 +319,14 @@ impl Graph { } } - /// Persist the upper-layer adjacency to a sidecar file (prototype; the production design - /// is an append-allocated region inside the plane file — see design doc §4). Stale-on-crash - /// is acceptable: watermark replay re-feeds recent inserts, which re-links upper edges. - pub fn save_upper(&self, path: &std::path::Path) -> std::io::Result<()> { - use std::io::Write; - let upper = self.upper.read().unwrap(); - let mut buf: Vec = Vec::new(); - buf.extend_from_slice(&(upper.len() as u32).to_le_bytes()); - for (&id, levels) in upper.iter() { - buf.extend_from_slice(&id.to_le_bytes()); - buf.push(levels.len() as u8); - for list in levels { - buf.extend_from_slice(&(list.len() as u16).to_le_bytes()); - for &n in list { - buf.extend_from_slice(&n.to_le_bytes()); - } - } - } - let tmp = path.with_extension("upper.tmp"); - let mut f = std::fs::File::create(&tmp)?; - f.write_all(&buf)?; - f.sync_all()?; - std::fs::rename(tmp, path) - } - - /// Load the sidecar written by save_upper. Missing file leaves the hierarchy empty - /// (layer-0 search still works, just without upper-layer routing). - pub fn load_upper(&self, path: &std::path::Path) -> std::io::Result<()> { - let buf = match std::fs::read(path) { - Ok(b) => b, - Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), - Err(e) => return Err(e), - }; - let mut pos = 0usize; - let rd_u32 = |b: &[u8], p: &mut usize| { - let v = u32::from_le_bytes(b[*p..*p + 4].try_into().unwrap()); - *p += 4; - v - }; - let count = rd_u32(&buf, &mut pos); - let mut upper = self.upper.write().unwrap(); - upper.clear(); - for _ in 0..count { - let id = rd_u32(&buf, &mut pos); - let nlevels = buf[pos] as usize; - pos += 1; - let mut levels = Vec::with_capacity(nlevels); - for _ in 0..nlevels { - let len = u16::from_le_bytes(buf[pos..pos + 2].try_into().unwrap()) as usize; - pos += 2; - let mut list = Vec::with_capacity(len); - for _ in 0..len { - list.push(rd_u32(&buf, &mut pos)); - } - levels.push(list); - } - upper.insert(id, levels); - } - Ok(()) - } - - /// Mark deleted (traversals skip it) and return the id to the freelist. + /// Mark deleted (traversals skip it) and return the id to the freelist. The upper-region + /// entry, if any, is leaked (bounded by the region's 2x-headroom reserve; freelist TODO). pub fn delete_node(&self, id: u32) { { let seq = self.file.seq_atomic(id); let _guard = seqlock::write_lock(seq); unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; } - self.upper.write().unwrap().remove(&id); self.file.free_id(id); } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index bbf415cd45..97727e5420 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -5,7 +5,7 @@ //! format, so neighbor↔neighbor distances are recomputed (int8×int8) on id-match hits only. use crate::distance::{quantize_int8, Query}; -use crate::format::NO_ID; +use crate::format::{NO_ID, NO_UPPER}; use crate::graph::Graph; use crate::search::{greedy_descend, search_layer, SearchScratch, SearchStats}; @@ -28,7 +28,7 @@ fn level_for(id: u32, ml: f64) -> u8 { x ^= x >> 33; let unit = (x as f64) / (u64::MAX as f64); let level = (-unit.max(f64::MIN_POSITIVE).ln() * ml).floor(); - level.min(31.0) as u8 + (level as u8).min(crate::format::MAX_UPPER_LEVELS as u8) } /// Remove `to` from `from`'s adjacency at `level` (edge-replacement maintenance). @@ -40,14 +40,11 @@ fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { } }); } else { - let mut upper = graph.upper.write().unwrap(); - if let Some(levels) = upper.get_mut(&from) { - if let Some(list) = levels.get_mut(level as usize - 1) { - if let Some(pos) = list.iter().position(|&x| x == to) { - list.remove(pos); - } + graph.update_upper_level(from, level, |list| { + if let Some(pos) = list.iter().position(|&x| x == to) { + list.remove(pos); } - } + }); } } @@ -56,13 +53,7 @@ fn neighbors_at(graph: &Graph, id: u32, level: u8, buf: &mut Vec) { if level == 0 { graph.neighbors_into(id, buf); } else { - buf.clear(); - let upper = graph.upper.read().unwrap(); - if let Some(levels) = upper.get(&id) { - if let Some(list) = levels.get(level as usize - 1) { - buf.extend_from_slice(list); - } - } + graph.upper_neighbors_into(id, level, buf); } } @@ -85,24 +76,20 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) } }); } else { - // held across the prune: upper mutations are rare (~6% of nodes) and the recomputed - // distances are ~cap * 0.2us — an acceptable hold for prototype correctness - let mut upper = graph.upper.write().unwrap(); - if let Some(levels) = upper.get_mut(&nid) { - if let Some(list) = levels.get_mut(level as usize - 1) { - if !list.contains(&new_id) { - list.push(new_id); - if list.len() > cap { - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) - .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - *list = scored.into_iter().take(cap).map(|(c, _)| c).collect(); - } - } + graph.update_upper_level(nid, level, |list| { + if list.contains(&new_id) { + return; } - } + list.push(new_id); + if list.len() > cap { + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) + .collect(); + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + *list = scored.into_iter().take(cap).map(|(c, _)| c).collect(); + } + }); } } @@ -116,10 +103,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { - graph.write_node(id, level, &bytes, scale, inv_mag, &[]); - if level > 0 { - graph.upper.write().unwrap().insert(id, vec![Vec::new(); level as usize]); - } + let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; + graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); graph.file.set_entry_point(id, level as u32); return id; } @@ -186,12 +171,9 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu connections[l as usize] = conns; } - // Write the new node: layer-0 list pruned to the file cap (selection order = rank order). - let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); - l0.truncate(layer0_cap); - graph.write_node(id, level, &bytes, scale, inv_mag, &l0); - - if level > 0 { + // Write the new node: upper entry first so a reader that sees the node sees its + // hierarchy; layer-0 list pruned to the file cap (selection order = rank order). + let upper_idx = if level > 0 { let levels: Vec> = (1..=level as usize) .map(|l| { connections @@ -200,8 +182,13 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu .unwrap_or_default() }) .collect(); - graph.upper.write().unwrap().insert(id, levels); - } + graph.write_upper(&levels) + } else { + NO_UPPER + }; + let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); + l0.truncate(layer0_cap); + graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx); // Reverse edges. for (l, conns) in connections.iter().enumerate() { diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index df4f6e0479..61afdb8cd5 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -11,7 +11,6 @@ use napi::bindgen_prelude::*; use napi::threadsafe_function::{ErrorStrategy, ThreadsafeFunction, ThreadsafeFunctionCallMode}; use napi::JsFunction; use napi_derive::napi; -use std::path::PathBuf; use std::sync::{Arc, Mutex}; /// Pooled per-query scratch (the visited array is O(nodes); never allocate per query). @@ -123,7 +122,6 @@ impl Task for PredicateSearchTask { pub struct Plane { graph: Arc, pool: Arc, - upper_path: PathBuf, params: InsertParams, // insert scratch, serialized: phase-1 hosts call insert from a single writer at a time // per index (Harper's commit path); a Mutex keeps misuse safe rather than fast. @@ -138,24 +136,20 @@ impl Plane { pub fn create(path: String, dims: u32, layer0_cap: u32, max_nodes: f64) -> Result { let file = PlaneFile::create(std::path::Path::new(&path), dims as usize, layer0_cap as usize, max_nodes as u64) .map_err(|e| Error::from_reason(e.to_string()))?; - Ok(Self::wrap(file, &path)) + Ok(Self::wrap(file)) } - /// Open an existing plane file and its upper-layer sidecar. + /// Open an existing plane file (the upper-layer region lives in the same file). #[napi(factory)] pub fn open(path: String) -> Result { let file = PlaneFile::open(std::path::Path::new(&path)).map_err(|e| Error::from_reason(e.to_string()))?; - let plane = Self::wrap(file, &path); - plane.graph.load_upper(&plane.upper_path).map_err(|e| Error::from_reason(e.to_string()))?; - Ok(plane) + Ok(Self::wrap(file)) } - fn wrap(file: PlaneFile, path: &str) -> Plane { - let upper_path = PathBuf::from(format!("{path}.upper")); + fn wrap(file: PlaneFile) -> Plane { Plane { graph: Arc::new(Graph::new(file)), pool: Arc::new(ScratchPool(Mutex::new(Vec::new()))), - upper_path, params: InsertParams::default(), insert_scratch: Mutex::new(SearchScratch::new()), } @@ -254,10 +248,9 @@ impl Plane { self.graph.file.set_watermark(txn as u64); } - /// msync the plane and persist the upper-layer sidecar; advances durability. + /// msync the plane (slots + upper region, one file); advances durability. #[napi] pub fn flush(&self) -> Result<()> { - self.graph.save_upper(&self.upper_path).map_err(|e| Error::from_reason(e.to_string()))?; self.graph.file.msync().map_err(|e| Error::from_reason(e.to_string())) } } diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 5febb42dd7..284e787839 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -151,13 +151,7 @@ pub fn search_layer( continue; } } else { - nbuf.clear(); - let upper = graph.upper.read().unwrap(); - if let Some(levels) = upper.get(&c.id) { - if let Some(list) = levels.get(level as usize - 1) { - nbuf.extend_from_slice(list); - } - } + graph.upper_neighbors_into(c.id, level, &mut nbuf); } for i in 0..nbuf.len() { let nid = nbuf[i]; @@ -197,18 +191,15 @@ pub fn greedy_descend( to_level: u32, stats: &mut SearchStats, ) -> (u32, f32) { - let upper = graph.upper.read().unwrap(); + let mut nbuf: Vec = Vec::new(); let mut level = from_level; while level > to_level { let mut improved = true; while improved { improved = false; - let neighbors = upper - .get(¤t) - .and_then(|levels| levels.get(level as usize - 1)) - .cloned() - .unwrap_or_default(); - for nid in neighbors { + graph.upper_neighbors_into(current, level.min(255) as u8, &mut nbuf); + for i in 0..nbuf.len() { + let nid = nbuf[i]; if let Some(d) = graph.distance_to(nid, query) { stats.visits += 1; if d < current_dist { From bc2163048032179de06601cc70585f3ec7ed673d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 08:40:38 -0600 Subject: [PATCH 09/69] design: record format-v2 1M regression check (recall/visits identical) Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index e932c803f6..4b6f788c35 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -263,8 +263,11 @@ Open: workload measurement, not a guess. - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the format (dims × mode in header) but int8 is the default and the optimization target. -- **Upper-layer region persistence** — prototype keeps upper adjacency in memory (rebuilt at - open by scanning slots); the append-allocated file region is pending. +- ~~Upper-layer region persistence~~ — done (format v2): fixed-entry region in the same file, + per-entry seqlocks, reserved for max_nodes/8. Upper entries leak on delete (bounded by the + 2x-headroom reserve); an upper freelist is the remaining nicety. +- **Reservation growth** — max_nodes is fixed at create; production needs either a generous + sparse reservation (Linux-fine; strict-overcommit hosts need care) or mremap-based growth. ## 11. Prototype measurements (kzyp Linux box, 768-d int8, ef 512, cap 64) @@ -277,6 +280,7 @@ recall@10-set 0.997, ~3,110 visits. | 100K | 64 | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | | 1M | 64 | 0.81 ms | 1.60 ms | 2,279 | 0.353 | 0.975 | 1,670 inserts/s | | 1M | 128 | 0.75 ms | 1.48 ms | 2,309 | 0.324 | **0.996** | 1,242 inserts/s | +| 1M (fmt v2) | 128 | 0.83 ms | 1.61 ms | 2,309 | 0.359 | 0.996 | 1,346 inserts/s | | 1M JS anchor | 128 | 7.2 ms | 12.0 ms | ~3,110 | 4.34 | 0.997 | ~263 inserts/s | At the 1M anchor with cap 128: **9.6× p50, 12.9× per-visit, 4.7× build rate, at JS-equal From 0d8086436d7815b475efa422095116f5167bbed3 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 09:22:49 -0600 Subject: [PATCH 10/69] hnsw-plane: raw mirroring surface for dual-write phase 1 writeNodeRaw(id, level, int8 bin, scale, invMag, layer0 ids, upper id arrays): full node state per call with HOST-allocated ids - high-water is raised via CAS-max, the plane allocator/freelist is bypassed, and an existing upper entry is rewritten in place so repeated updates to a high-level node do not leak entries. clearNode marks deleted without a freelist push (the host owns id allocation). setEntryPoint/getEntryPoint mirror the host's entry updates. This is the seam the Harper phase-1 integration writes through: JS keeps computing the graph, the plane mirrors it bit-identically, rollback = flip the search flag. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/smoke.mjs | 23 +++++++++++++ native/hnsw-plane/src/format.rs | 14 ++++++++ native/hnsw-plane/src/graph.rs | 61 +++++++++++++++++++++++++++++++++ native/hnsw-plane/src/napi.rs | 59 ++++++++++++++++++++++++++++++- 4 files changed, 156 insertions(+), 1 deletion(-) diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs index f5911b95e4..dc4b08de2c 100644 --- a/native/hnsw-plane/smoke.mjs +++ b/native/hnsw-plane/smoke.mjs @@ -53,6 +53,29 @@ for (const h of pred) if (h.id % 3 !== 0) throw new Error(`predicate leak: id ${ if (pred.length === 0) throw new Error('predicate search returned nothing'); console.log(`predicate top hit: id ${pred[0].id} (calls: ${predicateCalls})`); +// raw mirroring path (dual-write phase 1): host-allocated ids, full node state per call +const mirror = Plane.create(`/tmp/smoke-mirror-${process.pid}.hnsw`, dims, 32, 10_000); +const q42 = vec(42); +// quantize like the host: scale maps max|c| to 127, invMag = 1/|v| +function quant(v) { + let maxAbs = 0, magSq = 0; + for (const x of v) { maxAbs = Math.max(maxAbs, Math.abs(x)); magSq += x * x; } + const scale = maxAbs === 0 ? 1 : maxAbs / 127; + const bytes = Buffer.from(Int8Array.from(v, (x) => Math.max(-127, Math.min(127, Math.round(x / scale)))).buffer); + return { bytes, scale, invMag: 1 / Math.sqrt(magSq) }; +} +// two nodes linked to each other, host ids 10 and 20; node 10 is the entry at level 1 +const a = quant(q42), b = quant(vec(43)); +mirror.writeNodeRaw(10, 1, a.bytes, a.scale, a.invMag, Uint32Array.from([20]), [Uint32Array.from([])]); +mirror.writeNodeRaw(20, 0, b.bytes, b.scale, b.invMag, Uint32Array.from([10]), null); +mirror.setEntryPoint(10, 1); +const mhits = mirror.searchSync(q42, 2, 16); +if (mhits[0].id !== 10 || mhits[0].distance > 1e-3) throw new Error(`mirror self-query failed: ${JSON.stringify(mhits)}`); +mirror.clearNode(20); +const mhits2 = mirror.searchSync(vec(43), 2, 16); +if (mhits2.some((h) => h.id === 20)) throw new Error('cleared node still returned'); +console.log('raw mirroring OK'); + plane.flush(); const reopened = Plane.open(path); const hits2 = reopened.searchSync(vec(42), 5, 128); diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index ef2487d3b4..badb6305e4 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -209,6 +209,20 @@ impl PlaneFile { } } + /// Raise the high-water to at least `id + 1` (dual-write mode: ids are allocated by the + /// host's existing allocator and mirrored in; the plane allocator is bypassed). + pub fn ensure_high_water(&self, id: u32) { + let hw = self.header_atomic_u64(H_ID_HIGH_WATER); + let want = id as u64 + 1; + let mut cur = hw.load(Ordering::Acquire); + while cur < want { + match hw.compare_exchange_weak(cur, want, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => break, + Err(now) => cur = now, + } + } + } + pub fn id_high_water(&self) -> u64 { self.header_atomic_u64(H_ID_HIGH_WATER).load(Ordering::Acquire) } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 487061a39a..11819e0564 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -193,6 +193,67 @@ impl Graph { idx } + /// Rewrite an existing upper entry in place (full state). Used by the raw mirroring + /// path so repeated updates to a high-level node reuse its entry instead of leaking one + /// per rewrite. + pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) { + let seq = self.file.upper_seq_atomic(idx); + let _guard = seqlock::write_lock(seq); + let p = self.file.upper_ptr_mut(idx); + unsafe { + let n = levels.len().min(MAX_UPPER_LEVELS); + *p.add(U_LEVELS) = n as u8; + for (l, list) in levels.iter().take(n).enumerate() { + let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); + let deg = list.len().min(UPPER_CAP); + (lp as *mut u16).write_unaligned((deg as u16).to_le()); + let base = lp.add(2) as *mut u32; + for (i, id) in list.iter().take(deg).enumerate() { + base.add(i).write_unaligned(id.to_le()); + } + } + } + } + + /// Mirror a host-maintained node into the plane: full state per call, host-allocated id + /// (high-water is raised, the plane allocator is bypassed), upper entry reused in place + /// when present. This is the dual-write phase-1 write path. + pub fn write_node_raw( + &self, + id: u32, + level: u8, + vector: &[i8], + scale: f32, + inv_mag: f32, + neighbors: &[u32], + upper_levels: &[Vec], + ) { + self.file.ensure_high_water(id); + let existing = self.upper_idx_of(id); + let upper_idx = if upper_levels.is_empty() { + existing // keep an existing entry bound (level never shrinks in practice) + } else if existing != NO_UPPER { + self.rewrite_upper(existing, upper_levels); + existing + } else { + self.write_upper(upper_levels) + }; + let mut l0 = neighbors.to_vec(); + l0.truncate(self.file.layer0_cap); + self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx); + } + + /// Mark deleted WITHOUT returning the id to the plane freelist — dual-write mode, where + /// the host owns id allocation and may re-mint or reuse ids on its own schedule. + pub fn clear_node(&self, id: u32) { + if !self.in_range(id) { + return; + } + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq); + unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; + } + /// Atomic read-modify-write of `id`'s upper adjacency at `level` (1-based). Returns /// false when the node has no entry or level. `f` may read other slots. pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> bool { diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 61afdb8cd5..6915cfe5e4 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -162,12 +162,69 @@ impl Plane { Ok(insert(&self.graph, &vector, &self.params, &mut scratch)) } - /// Delete a node; its id returns to the freelist. + /// Delete a node; its id returns to the plane freelist. Standalone-allocation mode only + /// (pairs with insert()); dual-write hosts use clearNode instead. #[napi] pub fn remove(&self, id: u32) { self.graph.delete_node(id); } + /// Mirror a host-maintained node into the plane (dual-write phase 1): full node state + /// per call, host-allocated id, int8 vector bin + quantization scale + cached 1/|v|, + /// layer-0 neighbor ids, and per-upper-level neighbor id arrays (level 1 first). An + /// existing upper entry is rewritten in place. Idempotent per (id, state). + #[napi] + pub fn write_node_raw( + &self, + id: u32, + level: u8, + vector: Buffer, + scale: f64, + inv_mag: f64, + neighbors: Uint32Array, + upper: Option>, + ) -> Result<()> { + if vector.len() != self.graph.file.dims { + return Err(Error::from_reason(format!( + "vector is {} bytes; plane dims = {}", + vector.len(), + self.graph.file.dims + ))); + } + let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; + let upper_levels: Vec> = + upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); + self.graph.write_node_raw( + id, + level, + vec_i8, + scale as f32, + inv_mag as f32, + &neighbors.to_vec(), + &upper_levels, + ); + Ok(()) + } + + /// Mark a node deleted without touching the plane freelist (dual-write mode: the host + /// owns id allocation). + #[napi] + pub fn clear_node(&self, id: u32) { + self.graph.clear_node(id); + } + + /// Set the graph entry point (dual-write mode mirrors the host's entry-point updates). + #[napi] + pub fn set_entry_point(&self, id: u32, level: u32) { + self.graph.file.set_entry_point(id, level); + } + + #[napi] + pub fn get_entry_point(&self) -> Vec { + let (id, level) = self.graph.file.entry_point(); + vec![id as f64, level as f64] + } + /// Async k-NN search on the libuv thread pool. `filter` is an optional allow-bitset /// over node ids (bit i of byte i>>3); filtered searches are visit-bounded by /// ef * filterExpansion (default 24). From fe7d539ca1167210e6799dee573c1e1885f9b50b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 09:31:01 -0600 Subject: [PATCH 11/69] hnsw-plane: upper-entry freelist + coverage-aware reverse-edge pruning Upper freelist: delete_node frees the entry (tagged CAS stack, next pointer in the dead entry's first list bytes); write_node_raw reuses a cleared node's entry via a flags-agnostic idx read, closing the clearNode-then-rewrite leak in dual-write mode. Coverage pruning: the concurrent torture test, run in a loop, exposed orphaned nodes (~1-in-4 runs had unfindable self-queries) - closest-keep eviction on reverse-edge overflow can strip a node's last in-edge in dense near-duplicate clusters. Overflow eviction now prefers the most REDUNDANT far member (some kept nearer k has d(e,k) < d(base,e), so searches reaching k still reach e), bounded to farthest-16 x nearest-16 (~30us per overflow); falls back to plain farthest. 25/25 torture loops green after the change. bench: optional threads arg adds a concurrent-throughput pass (T searchers + background writer -> aggregate QPS, per-thread p50/p99). Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/bin/bench.rs | 74 +++++++++++++++++++++++++++++- native/hnsw-plane/src/format.rs | 57 +++++++++++++++++++---- native/hnsw-plane/src/graph.rs | 33 +++++++++++-- native/hnsw-plane/src/insert.rs | 52 +++++++++++++++------ 4 files changed, 189 insertions(+), 27 deletions(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index d0d5fcf27c..6b60d8ba28 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -2,7 +2,9 @@ //! per-visit cost — the number that decides whether the native plane hits its 0.25–0.4 µs //! budget (JS baseline: 4.34 µs/visit at 5M/ef 512). //! -//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=128] +//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=128] [threads=0] +//! threads > 0 adds a concurrent-throughput pass: T searcher threads (queries each) + one +//! background writer inserting throughout, reporting aggregate QPS and per-thread p50/p99. use hnsw_plane::distance::Query; use hnsw_plane::insert::{insert, InsertParams}; @@ -169,4 +171,74 @@ fn main() { us_per_visit ); println!("recall@10 (set): {:.3}", recall_hits as f64 / recall_total as f64); + + let threads: usize = args.get(7).and_then(|a| a.parse().ok()).unwrap_or(0); + if threads > 0 { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::Arc; + let graph = Arc::new(graph); + let corpus = Arc::new(corpus); + let stop = Arc::new(AtomicBool::new(false)); + let per_thread = queries.max(100); + let start = Instant::now(); + let mut handles = Vec::new(); + for t in 0..threads { + let graph = graph.clone(); + let corpus = corpus.clone(); + handles.push(std::thread::spawn(move || { + let mut scratch = SearchScratch::new(); + let mut rng = Rng(0x9e37_79b9 ^ (t as u64 + 1) * 0x1234_5677); + let mut lat: Vec = Vec::with_capacity(per_thread); + for _ in 0..per_thread { + let q = Query::new(corpus.row(&mut rng)); + let s = Instant::now(); + let (r, _) = search(&graph, &q, 10, ef, &mut scratch); + lat.push(s.elapsed()); + assert!(!r.is_empty()); + } + lat.sort(); + (lat[per_thread / 2], lat[(per_thread * 99 / 100).min(per_thread - 1)]) + })); + } + // background writer: sustained inserts while searchers run + let writer = { + let graph = graph.clone(); + let corpus = corpus.clone(); + let stop = stop.clone(); + std::thread::spawn(move || { + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + let mut rng = Rng(0xdead_beef_cafe_f00d); + let mut count = 0u64; + while !stop.load(Ordering::Relaxed) { + let v = corpus.row(&mut rng); + insert(&graph, &v, ¶ms, &mut scratch); + count += 1; + } + count + }) + }; + let mut p50s = Vec::new(); + let mut p99s = Vec::new(); + for h in handles { + let (p50, p99) = h.join().unwrap(); + p50s.push(p50); + p99s.push(p99); + } + let wall = start.elapsed(); + stop.store(true, Ordering::Relaxed); + let inserted = writer.join().unwrap(); + let total_q = (threads * per_thread) as f64; + p50s.sort(); + p99s.sort(); + println!( + "concurrent: {} threads x {} queries + writer -> {:.0} QPS aggregate p50(med) {:.2} ms p99(worst) {:.2} ms writer {:.0} inserts/s", + threads, + per_thread, + total_q / wall.as_secs_f64(), + p50s[threads / 2].as_secs_f64() * 1e3, + p99s[threads - 1].as_secs_f64() * 1e3, + inserted as f64 / wall.as_secs_f64() + ); + } } diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index badb6305e4..55d0e427be 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -26,6 +26,7 @@ const H_TXN_WATERMARK: usize = 48; // u64 const H_CLEAN_SHUTDOWN: usize = 56; // u8 const H_MAX_NODES: usize = 64; // u64 const H_UPPER_HIGH_WATER: usize = 72; // u64 atomic: upper-entry allocator +const H_UPPER_FREELIST: usize = 80; // u64 atomic: (tag<<32)|idx; NO_UPPER = empty /// Upper-layer region geometry: fixed entries covering levels 1..=MAX_UPPER_LEVELS at /// UPPER_CAP ids per level. P(level >= 1) = 1/M ~ 6.25%; the region reserves entries for @@ -122,6 +123,7 @@ impl PlaneFile { map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); map[H_MAX_NODES..H_MAX_NODES + 8].copy_from_slice(&max_nodes.to_le_bytes()); + map[H_UPPER_FREELIST..H_UPPER_FREELIST + 8].copy_from_slice(&(NO_UPPER as u64).to_le_bytes()); let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }) } @@ -264,17 +266,54 @@ impl PlaneFile { unsafe { &*(self.upper_ptr(idx).add(U_SEQ) as *const AtomicU32) } } - /// Allocate an upper-region entry. Entries are not freed on delete (bounded leak within - /// the 2x-headroom reserve; freelist reuse is a TODO). Returns NO_UPPER when exhausted — - /// the node then simply has no upper links, which degrades routing, not correctness. + /// Allocate an upper-region entry: pop the upper freelist, else bump the high-water. + /// Returns NO_UPPER when exhausted — the node then simply has no upper links, which + /// degrades routing, not correctness. A dead entry's next-pointer lives in its first + /// list bytes (offset U_LISTS), clobbered on reuse by the full rewrite. pub fn allocate_upper(&self) -> u32 { - let hw = self.header_atomic_u64(H_UPPER_HIGH_WATER); - let idx = hw.fetch_add(1, Ordering::AcqRel); - if idx >= self.upper_capacity { - hw.fetch_sub(1, Ordering::AcqRel); - return NO_UPPER; + let head = self.header_atomic_u64(H_UPPER_FREELIST); + loop { + let cur = head.load(Ordering::Acquire); + let idx = (cur & 0xffff_ffff) as u32; + if idx == NO_UPPER { + let hw = self.header_atomic_u64(H_UPPER_HIGH_WATER); + let new = hw.fetch_add(1, Ordering::AcqRel); + if new >= self.upper_capacity { + hw.fetch_sub(1, Ordering::AcqRel); + return NO_UPPER; + } + return new as u32; + } + let next = unsafe { (*(self.upper_ptr(idx).add(U_LISTS) as *const AtomicU32)).load(Ordering::Acquire) }; + let tag = (cur >> 32).wrapping_add(1); + if head + .compare_exchange(cur, (next as u64) | (tag << 32), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + return idx; + } + } + } + + /// Return a dead upper entry to the freelist. Caller must have unlinked it from its + /// node's slot (or marked the node deleted) first. + pub fn free_upper(&self, idx: u32) { + if idx == NO_UPPER { + return; + } + let head = self.header_atomic_u64(H_UPPER_FREELIST); + let next_word = unsafe { &*(self.upper_ptr(idx).add(U_LISTS) as *const AtomicU32) }; + loop { + let cur = head.load(Ordering::Acquire); + next_word.store((cur & 0xffff_ffff) as u32, Ordering::Release); + let tag = (cur >> 32).wrapping_add(1); + if head + .compare_exchange(cur, (idx as u64) | (tag << 32), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + return; + } } - idx as u32 } pub fn set_clean_shutdown(&mut self, clean: bool) { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 11819e0564..107d97e441 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -215,6 +215,24 @@ impl Graph { } } + /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring + /// path reuses a cleared node's entry when the host rewrites the same id. + fn upper_idx_raw(&self, id: u32) -> u32 { + if !self.in_range(id) { + return NO_UPPER; + } + let seq = self.file.seq_atomic(id); + seqlock::read_consistent(seq, || { + let p = self.file.slot_ptr(id); + unsafe { + if *p.add(S_FLAGS) == 0 { + return NO_UPPER; // never written + } + (p.add(S_UPPER_IDX) as *const u32).read_unaligned() + } + }) + } + /// Mirror a host-maintained node into the plane: full state per call, host-allocated id /// (high-water is raised, the plane allocator is bypassed), upper entry reused in place /// when present. This is the dual-write phase-1 write path. @@ -229,7 +247,7 @@ impl Graph { upper_levels: &[Vec], ) { self.file.ensure_high_water(id); - let existing = self.upper_idx_of(id); + let existing = self.upper_idx_raw(id); let upper_idx = if upper_levels.is_empty() { existing // keep an existing entry bound (level never shrinks in practice) } else if existing != NO_UPPER { @@ -380,14 +398,21 @@ impl Graph { } } - /// Mark deleted (traversals skip it) and return the id to the freelist. The upper-region - /// entry, if any, is leaked (bounded by the region's 2x-headroom reserve; freelist TODO). + /// Mark deleted (traversals skip it), free its upper entry, and return the id to the + /// plane freelist. pub fn delete_node(&self, id: u32) { + let upper_idx; { let seq = self.file.seq_atomic(id); let _guard = seqlock::write_lock(seq); - unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; + let p = self.file.slot_ptr_mut(id); + unsafe { + upper_idx = (p.add(S_UPPER_IDX) as *const u32).read_unaligned(); + (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); + *p.add(S_FLAGS) = FLAG_DELETED; + } } + self.file.free_upper(upper_idx); self.file.free_id(id); } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 97727e5420..110650c0c2 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -57,7 +57,43 @@ fn neighbors_at(graph: &Graph, id: u32, level: u8, buf: &mut Vec) { } } -/// Add `new_id` to `nid`'s adjacency at `level`, pruning to `cap` closest when over. +/// Prune an over-cap adjacency list by evicting the most REDUNDANT far member rather than +/// blindly the farthest: plain closest-keep can strip a node's last in-edge in dense +/// near-duplicate clusters, orphaning it from the graph (observed as unfindable self-queries +/// under concurrent builds). A far member e is redundant when some kept nearer member k has +/// d(e, k) < d(base, e) — searches reaching k still reach e. Bounded: farthest 16 candidates +/// checked against the nearest 16 keepers (~30us per overflow event); falls back to evicting +/// the plain farthest when nothing is provably redundant. +fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize) { + let mut scored: Vec<(u32, f32)> = list + .iter() + .filter_map(|&cand| graph.distance_between(base, cand).map(|d| (cand, d))) + .collect(); + scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + while scored.len() > cap { + let check_from = scored.len().saturating_sub(16); + let keepers = &scored[..16.min(check_from)]; + let mut evict = scored.len() - 1; // fallback: farthest + 'hunt: for i in (check_from..scored.len()).rev() { + let (e, d_base_e) = scored[i]; + for &(k, _) in keepers { + if k == e { + continue; + } + if let Some(d_ek) = graph.distance_between(e, k) { + if d_ek < d_base_e { + evict = i; + break 'hunt; + } + } + } + } + scored.remove(evict); + } + *list = scored.into_iter().map(|(cand, _)| cand).collect(); +} + +/// Add `new_id` to `nid`'s adjacency at `level`, coverage-pruning to `cap` when over. fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) { if level == 0 { graph.update_neighbors(nid, |list| { @@ -67,12 +103,7 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) list.push(new_id); if list.len() > cap { // distance_between reads other slots without locks; safe under this seqlock - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) - .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - *list = scored.into_iter().take(cap).map(|(cand, _)| cand).collect(); + prune_with_coverage(graph, nid, list, cap); } }); } else { @@ -82,12 +113,7 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) } list.push(new_id); if list.len() > cap { - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| graph.distance_between(nid, cand).map(|d| (cand, d))) - .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); - *list = scored.into_iter().take(cap).map(|(c, _)| c).collect(); + prune_with_coverage(graph, nid, list, cap); } }); } From 9a9bca88b427c38d56388da0e0bca17d755265b8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 09:42:04 -0600 Subject: [PATCH 12/69] design: coverage-prune + concurrency measurements (0.999 recall, 6.3K QPS w/ writer) Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 4b6f788c35..381312c5a9 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -281,6 +281,15 @@ recall@10-set 0.997, ~3,110 visits. | 1M | 64 | 0.81 ms | 1.60 ms | 2,279 | 0.353 | 0.975 | 1,670 inserts/s | | 1M | 128 | 0.75 ms | 1.48 ms | 2,309 | 0.324 | **0.996** | 1,242 inserts/s | | 1M (fmt v2) | 128 | 0.83 ms | 1.61 ms | 2,309 | 0.359 | 0.996 | 1,346 inserts/s | +| 1M (coverage prune) | 128 | 0.75 ms | 1.52 ms | 2,279 | 0.327 | **0.999** | 1,359 inserts/s | + +Concurrency (same 1M graph): **6,345 QPS aggregate** across 8 searcher threads (p50 1.03 ms, +worst-thread p99 3.84 ms) while a background writer sustained **1,102 inserts/s** — the QPS +input §9 of the Reflex study lacked. Reverse-edge overflow eviction is coverage-aware +(evict the far member provably reachable via a kept nearer one; bounded 16×16 checks): the +concurrent torture test caught closest-keep eviction orphaning nodes in near-duplicate +clusters (~1-in-4 runs), and the fix also raised 1M recall from 0.996 to 0.999 at equal +build cost. | 1M JS anchor | 128 | 7.2 ms | 12.0 ms | ~3,110 | 4.34 | 0.997 | ~263 inserts/s | At the 1M anchor with cap 128: **9.6× p50, 12.9× per-visit, 4.7× build rate, at JS-equal From 040d01f964e61ac86ef9229f5e2e1b5000359e2b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 09:52:17 -0600 Subject: [PATCH 13/69] HNSW native plane phase 1: dual-write mirroring + opt-in native search cutover MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With the new nativePlane index option (search-only: toggling never reindexes), every graph mutation HierarchicalNavigableSmallWorld persists to its index CF is mirrored into a plane file next to the store (writeNodeRaw/clearNode/ setEntryPoint — host-allocated ids, per-edge distances dropped), and search() routes through the native module: plane.search for unfiltered queries, searchWithPredicate (batched TSFN predicate over the existing pk resolution) for filtered ones, at the same resolved ef. The CF stays authoritative; rollback is flag-off + file delete. The plane file is created lazily (exclusive-create settles multi-worker races) with a full mirror of the existing CF graph on first enable, reopened on restart, and deleted on index drop/clear/reindex via a resetDerivedStorage lifecycle hook. The native module is optional: absence logs one warning and the index runs the JS path; build locally with npm run build:hnsw-plane. Plane-flagged searches resolve asynchronously, so searchByIndex wraps the promised candidate list in a lazily-resolving ExtendedIterable (async iteration only), leaving the load + rescore pipeline unchanged. Crate change: writeNodeRaw rejects ids past the maxNodes reservation instead of addressing past the mapped slot region. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- .gitignore | 5 + hnsw-native-plane.md | 79 ++-- native/hnsw-plane/build.mjs | 19 + native/hnsw-plane/smoke.mjs | 27 +- native/hnsw-plane/src/napi.rs | 8 + package.json | 1 + resources/Table.ts | 7 +- resources/databases.ts | 13 +- .../HierarchicalNavigableSmallWorld.ts | 354 +++++++++++++++++- resources/indexes/hnswPlaneBinding.ts | 88 +++++ resources/search.ts | 67 +++- unitTests/resources/vectorIndexPlane.test.js | 271 ++++++++++++++ 12 files changed, 858 insertions(+), 81 deletions(-) create mode 100644 native/hnsw-plane/build.mjs create mode 100644 resources/indexes/hnswPlaneBinding.ts create mode 100644 unitTests/resources/vectorIndexPlane.test.js diff --git a/.gitignore b/.gitignore index e91683d8a1..b99cc0a082 100644 --- a/.gitignore +++ b/.gitignore @@ -64,3 +64,8 @@ test_export*.json # dev-mode boots (harper dev ) symlink node_modules/harper into the # fixture component dir; keep those out of commits integrationTests/**/node_modules/ + +# hnsw-plane native build outputs (optional module; build locally with npm run build:hnsw-plane) +native/hnsw-plane/target/ +native/hnsw-plane/hnsw-plane.node +native/hnsw-plane/Cargo.lock diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 381312c5a9..da962d5c76 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -10,12 +10,12 @@ ceiling both auto-scale with the graph") and issues #693, #711, #895, #2182. Per-visit cost decomposition at 5M nodes / ef 512 (768-d int8, `benchmarks/hnsw-scale.js` corpus, 22.18 ms p50 / 5,107 visits): -| Component | Cost | Share of a warm visit | -| ------------------------------------------------- | ------- | ---------------------- | -| Total per visited node | 4.34 µs | 100% | -| int8 asymmetric cosine, 768-d, JS | 0.43 µs | 10% | -| msgpackr decode of one node (VT-cache miss) | 5.57 µs | +128% when cold | -| Neighbour iteration + visited-set ops | 0.21 µs | 5% | +| Component | Cost | Share of a warm visit | +| ------------------------------------------- | ------- | --------------------- | +| Total per visited node | 4.34 µs | 100% | +| int8 asymmetric cosine, 768-d, JS | 0.43 µs | 10% | +| msgpackr decode of one node (VT-cache miss) | 5.57 µs | +128% when cold | +| Neighbour iteration + visited-set ops | 0.21 µs | 5% | ~85% of a warm visit is JS object bookkeeping — candidate heap, visited `Set`, property access, allocation, GC — not distance math and not I/O. Three consequences: @@ -84,31 +84,31 @@ One file per index (per slice, once C2 lands): `.hnsw`. **Header (4 KB page):** -| Field | Type | Notes | -| --- | --- | --- | -| magic + format version | u32 + u32 | rebuild required on version mismatch (accepted contract) | -| dims, quantization mode | u16 + u8 | v1: int8 asymmetric; f32 supported for `quantization:"none"` | -| slot_size, layer0_cap, upper_cap | u16 ×3 | derived from M/optimizeRouting at creation | -| entry_point_id, entry_point_level | u32 + u8 | atomically updated | -| id_high_water | u64 atomic | replaces the shared Atomics BigInt64Array incrementer | -| freelist_head | u64 atomic | CAS push/pop; ABA-guarded with a 32-bit tag | -| txn_watermark | u64 | last durably indexed transaction; advanced by msync cadence | -| clean_shutdown flag | u8 | torn-state detection on open | +| Field | Type | Notes | +| --------------------------------- | ---------- | ------------------------------------------------------------ | +| magic + format version | u32 + u32 | rebuild required on version mismatch (accepted contract) | +| dims, quantization mode | u16 + u8 | v1: int8 asymmetric; f32 supported for `quantization:"none"` | +| slot_size, layer0_cap, upper_cap | u16 ×3 | derived from M/optimizeRouting at creation | +| entry_point_id, entry_point_level | u32 + u8 | atomically updated | +| id_high_water | u64 atomic | replaces the shared Atomics BigInt64Array incrementer | +| freelist_head | u64 atomic | CAS push/pop; ABA-guarded with a 32-bit tag | +| txn_watermark | u64 | last durably indexed transaction; advanced by msync cadence | +| clean_shutdown flag | u8 | torn-state detection on open | **Main region — layer-0 slots**, addressed `4096 + id × slot_size`: -| Field | Size (768-d int8, cap 64) | -| --- | --- | -| seq (seqlock) | 4 B | -| flags (valid/deleted) + level | 2 B | -| scale (f32) + invMag (f32) | 8 B | -| degree | 2 B | -| vector (int8 × 768) | 768 B | -| neighbor ids (u32 × layer0_cap) | 256 B | -| **total, padded** | **1,040 B → 1 KB-aligned 1,088 B** | +| Field | Size (768-d int8, cap 64) | +| ------------------------------- | ---------------------------------- | +| seq (seqlock) | 4 B | +| flags (valid/deleted) + level | 2 B | +| scale (f32) + invMag (f32) | 8 B | +| degree | 2 B | +| vector (int8 × 768) | 768 B | +| neighbor ids (u32 × layer0_cap) | 256 B | +| **total, padded** | **1,040 B → 1 KB-aligned 1,088 B** | At 100M nodes: ~109 GB (int8). A binary-code v2 slot (96 B codes + ids) is ~384 B → ~38 GB. -For comparison, today's encoding averages 1,425 B/node *plus* RocksDB overhead — so v1 is +For comparison, today's encoding averages 1,425 B/node _plus_ RocksDB overhead — so v1 is already ~25% smaller while being fixed-offset addressable, because per-edge cached float64 distances are dropped (recomputing a distance costs ~50 ns native; storing it costs 8 B and ~40% of today's node bytes). @@ -135,7 +135,7 @@ a header field, so revising it is a rebuild, not a format change. back-edge lists, each independently. A traversal may observe the half-linked state: an edge to a slot whose valid flag is not yet set → skip (HNSW tolerates missing edges); a just-deleted neighbor → skip via flags. Wrong-candidate leakage is filtered by the existing - exact rescore + MVCC record load, which is why relaxed adherence is safe *here* and not a + exact rescore + MVCC record load, which is why relaxed adherence is safe _here_ and not a general storage pattern. - **Writers.** Multiple worker threads insert concurrently today (distinct records); the same holds: id allocation is one atomic fetch_add on the header, freelist pop is CAS, slot writes @@ -207,6 +207,17 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d file. Validation = compare native results against the JS path on the same graph; rollback = flip search back to JS, drop the file. The double-write cost is bounded (index writes are a fraction of insert cost) and temporary. + + _Integrated_ behind the opt-in `nativePlane: true` index option (search-only: toggling never + reindexes; int8 + cosine indexes only — the flag no-ops elsewhere). Mutations mirror at the + exact `indexStore.put/remove` sites via `writeNodeRaw`/`clearNode`/`setEntryPoint` with + host-allocated ids; the plane file (`/..hnsw`, layer0 cap 128, + 16M-node sparse reservation) is created lazily with a full mirror of the existing CF graph on + first enable, reopened on restart, deleted on drop/clear/reindex. The compiled module is + optional (`npm run build:hnsw-plane`); absence falls back to the JS path with one warning. + Parity, predicate, restart, and lifecycle coverage in `unitTests/resources/vectorIndexPlane.test.js`. + Watermark/replay wiring, slicing, and msync-cadence flushes are not wired yet (open items). + - **Phase 2 — file-primary.** Drop the CF writes; the file is the only graph store. JS insert reads nodes through a native `getNode(id)` (one NAPI crossing per read, ~1 µs — comparable to today's decode path). Migration for existing indexes: reindex (accepted contract), or a @@ -275,13 +286,13 @@ Gaussian-mixture corpus matching `benchmarks/hnsw-scale.js` calibration (intra-c clusters = N/500). JS baseline for scale: 4.34 µs/visit; 1M efC-200 anchor: p50 7.2 ms, recall@10-set 0.997, ~3,110 visits. -| N | cap | p50 | p95 | visits/query | µs/visit | recall@10 (set) | build rate | -| --- | --- | --- | --- | --- | --- | --- | --- | -| 100K | 64 | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | -| 1M | 64 | 0.81 ms | 1.60 ms | 2,279 | 0.353 | 0.975 | 1,670 inserts/s | -| 1M | 128 | 0.75 ms | 1.48 ms | 2,309 | 0.324 | **0.996** | 1,242 inserts/s | -| 1M (fmt v2) | 128 | 0.83 ms | 1.61 ms | 2,309 | 0.359 | 0.996 | 1,346 inserts/s | -| 1M (coverage prune) | 128 | 0.75 ms | 1.52 ms | 2,279 | 0.327 | **0.999** | 1,359 inserts/s | +| N | cap | p50 | p95 | visits/query | µs/visit | recall@10 (set) | build rate | +| ------------------- | --- | ------- | ------- | ------------ | -------- | --------------- | --------------- | +| 100K | 64 | 0.28 ms | 0.46 ms | 1,395 | 0.201 | 1.000 | 5,583 inserts/s | +| 1M | 64 | 0.81 ms | 1.60 ms | 2,279 | 0.353 | 0.975 | 1,670 inserts/s | +| 1M | 128 | 0.75 ms | 1.48 ms | 2,309 | 0.324 | **0.996** | 1,242 inserts/s | +| 1M (fmt v2) | 128 | 0.83 ms | 1.61 ms | 2,309 | 0.359 | 0.996 | 1,346 inserts/s | +| 1M (coverage prune) | 128 | 0.75 ms | 1.52 ms | 2,279 | 0.327 | **0.999** | 1,359 inserts/s | Concurrency (same 1M graph): **6,345 QPS aggregate** across 8 searcher threads (p50 1.03 ms, worst-thread p99 3.84 ms) while a background writer sustained **1,102 inserts/s** — the QPS diff --git a/native/hnsw-plane/build.mjs b/native/hnsw-plane/build.mjs new file mode 100644 index 0000000000..9ca59543c6 --- /dev/null +++ b/native/hnsw-plane/build.mjs @@ -0,0 +1,19 @@ +// Builds the optional hnsw-plane NAPI module in place: harper installs never require a cargo +// toolchain (the nativePlane index option falls back to the JS path when the artifact is +// absent), so this is a local/dev step: `npm run build:hnsw-plane`. +import { execSync } from 'node:child_process'; +import { copyFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const crateRoot = dirname(fileURLToPath(import.meta.url)); +// build the lib alone: the bench bin cannot link against unresolved node-api symbols +execSync('cargo build --release --features napi --lib', { cwd: crateRoot, stdio: 'inherit' }); +const cdylib = + process.platform === 'win32' + ? 'hnsw_plane.dll' + : process.platform === 'darwin' + ? 'libhnsw_plane.dylib' + : 'libhnsw_plane.so'; +copyFileSync(join(crateRoot, 'target', 'release', cdylib), join(crateRoot, 'hnsw-plane.node')); +console.log('built native/hnsw-plane/hnsw-plane.node'); diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs index dc4b08de2c..79e6eb3657 100644 --- a/native/hnsw-plane/smoke.mjs +++ b/native/hnsw-plane/smoke.mjs @@ -40,15 +40,10 @@ console.log('freelist reuse OK, highWater still', plane.idHighWater()); // pipelined JS predicate: admit only ids divisible by 3; verdicts computed on the JS // event loop while traversal runs on the libuv pool let predicateCalls = 0; -const pred = await plane.searchWithPredicate( - vec(44), - 5, - 128, - (ids) => { - predicateCalls++; - return Uint8Array.from(ids, (id) => (id % 3 === 0 ? 1 : 0)); - } -); +const pred = await plane.searchWithPredicate(vec(44), 5, 128, (ids) => { + predicateCalls++; + return Uint8Array.from(ids, (id) => (id % 3 === 0 ? 1 : 0)); +}); for (const h of pred) if (h.id % 3 !== 0) throw new Error(`predicate leak: id ${h.id}`); if (pred.length === 0) throw new Error('predicate search returned nothing'); console.log(`predicate top hit: id ${pred[0].id} (calls: ${predicateCalls})`); @@ -58,19 +53,25 @@ const mirror = Plane.create(`/tmp/smoke-mirror-${process.pid}.hnsw`, dims, 32, 1 const q42 = vec(42); // quantize like the host: scale maps max|c| to 127, invMag = 1/|v| function quant(v) { - let maxAbs = 0, magSq = 0; - for (const x of v) { maxAbs = Math.max(maxAbs, Math.abs(x)); magSq += x * x; } + let maxAbs = 0, + magSq = 0; + for (const x of v) { + maxAbs = Math.max(maxAbs, Math.abs(x)); + magSq += x * x; + } const scale = maxAbs === 0 ? 1 : maxAbs / 127; const bytes = Buffer.from(Int8Array.from(v, (x) => Math.max(-127, Math.min(127, Math.round(x / scale)))).buffer); return { bytes, scale, invMag: 1 / Math.sqrt(magSq) }; } // two nodes linked to each other, host ids 10 and 20; node 10 is the entry at level 1 -const a = quant(q42), b = quant(vec(43)); +const a = quant(q42), + b = quant(vec(43)); mirror.writeNodeRaw(10, 1, a.bytes, a.scale, a.invMag, Uint32Array.from([20]), [Uint32Array.from([])]); mirror.writeNodeRaw(20, 0, b.bytes, b.scale, b.invMag, Uint32Array.from([10]), null); mirror.setEntryPoint(10, 1); const mhits = mirror.searchSync(q42, 2, 16); -if (mhits[0].id !== 10 || mhits[0].distance > 1e-3) throw new Error(`mirror self-query failed: ${JSON.stringify(mhits)}`); +if (mhits[0].id !== 10 || mhits[0].distance > 1e-3) + throw new Error(`mirror self-query failed: ${JSON.stringify(mhits)}`); mirror.clearNode(20); const mhits2 = mirror.searchSync(vec(43), 2, 16); if (mhits2.some((h) => h.id === 20)) throw new Error('cleared node still returned'); diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 6915cfe5e4..a6b04695b6 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -191,6 +191,14 @@ impl Plane { self.graph.file.dims ))); } + // ensure_high_water + slot_ptr have no bounds check, so a host id past the fixed + // reservation would address past the slot region (mmap overrun) — reject it here. + if id as u64 >= self.graph.file.max_nodes { + return Err(Error::from_reason(format!( + "node id {} exceeds the plane's maxNodes reservation ({})", + id, self.graph.file.max_nodes + ))); + } let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; let upper_levels: Vec> = upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); diff --git a/package.json b/package.json index dd45a218f5..65c4ce9e05 100644 --- a/package.json +++ b/package.json @@ -54,6 +54,7 @@ "scripts": { "build": "tsc --project tsconfig.build.json", "build:watch": "npm run build -- --watch --incremental", + "build:hnsw-plane": "node native/hnsw-plane/build.mjs", "typecheck": "tsc --project tsconfig.json", "typecheck:fast": "npx -y -p typescript@7.0.2 tsc --noEmit --project tsconfig.json", "test:types": "tsc --project unitTests/types/tsconfig.json", diff --git a/resources/Table.ts b/resources/Table.ts index b9515d6150..ed6e402737 100644 --- a/resources/Table.ts +++ b/resources/Table.ts @@ -1671,6 +1671,7 @@ export function makeTable(options) { const index = indices[attribute.name]; if (index) try { + index.customIndex?.resetDerivedStorage?.(); index.dropSync(); } catch (error) { ignoreAlreadyDropped(error); @@ -1691,7 +1692,10 @@ export function makeTable(options) { const drops = []; for (const attribute of attributes) { const index = indices[attribute.name]; - if (index) drops.push(index.drop().catch(ignoreAlreadyDropped)); + if (index) { + index.customIndex?.resetDerivedStorage?.(); + drops.push(index.drop().catch(ignoreAlreadyDropped)); + } } drops.push(primaryStore.drop().catch(ignoreAlreadyDropped)); await Promise.all(drops); @@ -5653,6 +5657,7 @@ export function makeTable(options) { const promises = [primaryStore.clear()]; for (const key in indices) { const index = indices[key]; + index.customIndex?.resetDerivedStorage?.(); promises.push(index.clearAsync ? index.clearAsync() : index.clear()); } return Promise.all(promises); diff --git a/resources/databases.ts b/resources/databases.ts index cb8f52b2b1..0af045f6a3 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -3,7 +3,7 @@ import { initSync, getHdbBasePath, get as envGet } from '../utility/environment/ import { INTERNAL_DBIS_NAME } from '../utility/lmdb/terms.ts'; import { open, compareKeys, type Database, type RootDatabase } from 'lmdb'; import { join, extname, basename } from 'path'; -import { existsSync, mkdirSync, readFileSync, readdirSync, realpathSync } from 'node:fs'; +import { existsSync, mkdirSync, readFileSync, readdirSync, realpathSync, unlinkSync } from 'node:fs'; import { unlink } from 'node:fs/promises'; import { getBaseSchemaPath, @@ -34,6 +34,7 @@ import { databasePaths, deleteRootBlobPathsForDB } from './blob.ts'; import { removeStorageReclamation } from '../server/storageReclamation.ts'; import { commonValidators, schemaRegex } from '../validation/common_validators.ts'; import { CUSTOM_INDEXES } from './indexes/customIndexes.ts'; +import { planeFilePathFor } from './indexes/hnswPlaneBinding.ts'; import { OpenDBIObject } from '../utility/lmdb/OpenDBIObject.ts'; import { RocksDatabase, supportedCompression, type RocksDatabaseOptions } from '@harperfast/rocksdb-js'; import { PrimaryRocksDatabase } from './PrimaryRocksDatabase.ts'; @@ -2805,6 +2806,7 @@ async function runIndexing(Table, attributes, indicesToRemove) { ); let lastResolution; for (const index of indicesToRemove) { + index.customIndex?.resetDerivedStorage?.(); lastResolution = index.drop(); } let interrupted; @@ -2820,6 +2822,7 @@ async function runIndexing(Table, attributes, indicesToRemove) { if (compareKeys(attribute.lastIndexedKey, start) < 0) start = attribute.lastIndexedKey; if (attribute.lastIndexedKey == undefined) { // if we are starting from the beginning, clear out any previous index entries since we are rewriting + attribute.dbi.customIndex?.resetDerivedStorage?.(); if (attribute.dbi.clearAsync) { // LMDB, note that we don't need to wait for this to complete, just gets enqueued in front of the other writes attribute.dbi.clearAsync(); @@ -3033,6 +3036,14 @@ function completeInterruptedDrop(rootStore, attributesDbi, databaseName: string, } finally { columnStore.close(); } + // derived HNSW plane files live next to the store; the normal drop path removes + // them through the custom index, but this recovery path drops raw column stores, + // and a same-name recreate must never open a stale plane over a fresh CF + try { + unlinkSync(planeFilePathFor(rootStore.path, columnName)); + } catch (error: any) { + if (error?.code !== 'ENOENT') logger.debug(`could not delete plane file for ${columnName}`, error); + } } } } else { diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 6ee044285b..325a76a7a6 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -1,9 +1,11 @@ +import { closeSync, existsSync, openSync, statSync, unlinkSync } from 'node:fs'; import { cosineDistance, euclideanDistance, dotProductDistance } from './vector.ts'; import { FLOAT32_OPTIONS } from 'msgpackr'; import { loggerWithTag } from '../../utility/logging/logger.ts'; import { ClientError } from '../../utility/errors/hdbError.ts'; import type { Id } from '../../resources/ResourceInterface.ts'; import { SKIP } from '@harperfast/extended-iterable'; +import { getPlaneBinding, planeFilePathFor, PLANE_NO_ID, type HnswPlane } from './hnswPlaneBinding.ts'; const logger = loggerWithTag('HNSW'); @@ -41,6 +43,16 @@ function dequantizeInt8(q: Int8Array, scale: number): number[] { return out; } +/** Connection ids for the plane mirror: ids only — per-edge distances are dropped (recomputed natively). */ +function planeConnectionIds(connections: Connection[] | undefined): Uint32Array { + if (!connections?.length) return new Uint32Array(0); + const ids: number[] = []; + for (const { id } of connections) { + if (typeof id === 'number' && id >= 0) ids.push(id); + } + return Uint32Array.from(ids); +} + // Auto-scaled search ef, used only when an index does not explicitly configure efConstructionSearch // and a query does not pass its own ef. A fixed ef makes recall decay as the graph grows (it explores // a shrinking fraction of the graph), so ef grows with sqrt(node count) in two regimes, with a @@ -105,6 +117,18 @@ function autoScaleEfConstruction(nodeCount: number): number { // this only has to be short enough that a table growing from empty picks up a larger ef promptly. const NODE_COUNT_TTL = 10_000; +// Native traversal-plane geometry (dual-write phase 1, hnsw-native-plane.md §4/§10). The layer-0 +// cap matches the JS graph's effective cap (M<<1 <<2 under optimizeRouting = 128; writeNodeRaw +// truncates the transient 160 overshoot). maxNodes is a fixed sparse reservation — pages +// materialize on write — and ids at or past it are rejected by the crate, which disables the +// plane for this process (reservation growth is a phase-2 open item). +const PLANE_LAYER0_CAP = 128; +const PLANE_MAX_NODES = 1 << 24; +// An existing plane file that cannot be opened is normally another worker mid-create (retry); +// past this age it is a crashed create and is deleted and rebuilt — the plane is derived state, +// the index column family stays authoritative. +const PLANE_STALE_CREATE_MS = 60_000; + class MinHeap { private data: Candidate[] = []; get size() { @@ -195,7 +219,9 @@ export class HierarchicalNavigableSmallWorld { // reindex (databases.ts persists the new value but skips rebuilding). efConstructionSearch is the // search-time candidate-list size; the build uses efConstruction/M/distance, which are structural. // filterExpansion is the visit-budget multiplier for predicate-aware (filtered) traversal. - static searchOnlyOptions = ['efConstructionSearch', 'filterExpansion']; + // nativePlane never changes the stored CF graph either: enabling it builds the derived plane + // file by mirroring the existing graph (see getPlane), so a toggle must not force a rebuild. + static searchOnlyOptions = ['efConstructionSearch', 'filterExpansion', 'nativePlane']; // Signals to search.ts that this index accepts a per-record predicate in search() and applies it // during traversal (predicate-aware / ACORN-style filtering), so companion conditions and RBAC can // be pushed down instead of post-filtering an under-filled candidate set (#1241). @@ -235,6 +261,13 @@ export class HierarchicalNavigableSmallWorld { private convertedNodes = new WeakMap(); private nodeCount = 0; private nodeCountAt = 0; + // Native traversal plane (dual-write phase 1): the CF graph stays authoritative; every graph + // mutation is mirrored into the plane file and search runs native when the flag is on. + // undefined = not yet attached (may retry), null = unavailable or disabled for this process. + private plane: HnswPlane | null | undefined; + private planeEligible = false; + private planeRetryAt = 0; + private planeDisabledLogged = false; constructor(indexStore: any, options: any) { this.indexStore = indexStore; if (indexStore) { @@ -265,7 +298,273 @@ export class HierarchicalNavigableSmallWorld { if (options.optimizeRouting !== undefined) this.optimizeRouting = options.optimizeRouting; if (options.filterExpansion !== undefined) this.filterExpansion = options.filterExpansion; } + if (options?.nativePlane) { + // The plane stores int8 bins and computes asymmetric cosine only (phase 1), so the flag + // is a no-op for float (quantization: "none") and non-cosine indexes. + this.planeEligible = this.int8 && this.distance === cosineDistance; + if (!this.planeEligible) { + logger.info?.('nativePlane is only supported for int8-quantized cosine HNSW indexes; using the JS search path'); + } + } + } + + /** Absolute path of this index's plane file, or undefined when the store exposes no path. */ + planeFilePath(): string | undefined { + const storePath = this.indexStore?.path; + const storeName = this.indexStore?.name; + if (typeof storePath !== 'string' || typeof storeName !== 'string') return undefined; + return planeFilePathFor(storePath, storeName); + } + + /** + * Attach (open or lazily create) the native plane for this index. `dims` must be provided by + * callers that may CREATE the file (a node mirror or a search, which know the vector length); + * without it the call is open-only — if no file exists yet there is nothing to sync, and the + * eventual creation's full mirror reads the then-current CF state. + * + * Multi-worker create races are settled by an exclusive open ('wx') of the file itself: the + * winner creates and mirrors, losers see EEXIST and open — transiently failing (and retrying + * after a TTL) while the winner is still writing the header. A crashed create leaves an + * unopenable file; once it is older than PLANE_STALE_CREATE_MS it is deleted and rebuilt. + */ + private getPlane(dims?: number): HnswPlane | null { + if (this.plane !== undefined) return this.plane; + if (!this.planeEligible) return (this.plane = null); + const now = Date.now(); + if (now < this.planeRetryAt) return null; + const Plane = getPlaneBinding(); + if (!Plane) return (this.plane = null); // the loader warned once already + const filePath = this.planeFilePath(); + if (!filePath) { + this.disablePlane(new Error('the index store exposes no path to place the plane file next to')); + return null; + } + try { + if (existsSync(filePath)) { + try { + return (this.plane = Plane.open(filePath)); + } catch (openError) { + if (now - statSync(filePath).mtimeMs <= PLANE_STALE_CREATE_MS) { + this.planeRetryAt = now + NODE_COUNT_TTL; + return null; + } + logger.warn?.('deleting an unopenable HNSW plane file left by an interrupted create', openError); + unlinkSync(filePath); + // fall through to the create path below + } + } + if (!dims) return null; // open-only call and no file: nothing to attach yet + let fd: number; + try { + fd = openSync(filePath, 'wx'); + } catch { + // another worker won the create race; open it after its header lands + this.planeRetryAt = now + NODE_COUNT_TTL; + return null; + } + closeSync(fd); + return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); + } catch (error) { + this.planeRetryAt = now + NODE_COUNT_TTL; + logger.warn?.('could not attach the HNSW plane file; will retry', error); + return null; + } + } + + /** + * Create the plane file and fully mirror the existing CF graph into it (the "first enable" + * build — a pure copy of the same graph, so plane and CF are bit-identical by construction; + * a reindex would rebuild a different random-level graph at far higher cost). The scan reads + * committed state: mutations committed while it runs mirror themselves through their own + * dual-write calls, though a write racing the scan can transiently be overwritten with the + * scan's older snapshot of that node — it re-syncs on the node's next touch, and the exact + * rescore + record load already filter stale candidates (relaxed adherence, design §5). + */ + private createAndMirrorPlane( + Plane: NonNullable>, + filePath: string, + dims: number + ): HnswPlane { + const plane = Plane.create(filePath, dims, PLANE_LAYER0_CAP, PLANE_MAX_NODES); + let mirrored = 0; + for (const { key, value } of this.indexStore.getRange({ start: 0, end: Infinity })) { + if (typeof key !== 'number' || !value || value.level === undefined) continue; + this.writeNodeToPlane(plane, key, value); + mirrored++; + } + const entryPointId = this.indexStore.getSync(ENTRY_POINT); + if (typeof entryPointId === 'number') { + plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); + } + if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); + return plane; + } + + /** Write one JS graph node's full state into the plane (throws on ineligible node state). */ + private writeNodeToPlane(plane: HnswPlane, nodeId: number, node: any): void { + if (!Number.isInteger(nodeId) || nodeId < 0 || nodeId >= PLANE_NO_ID) { + throw new Error(`node id ${nodeId} is outside the plane's u32 id space`); + } + const vector = node.vector; + let bin: Buffer; + let scale: number; + let invMag: number | undefined = node.invMag; + if (Array.isArray(vector)) { + // legacy float node inside an int8 index: quantize the mirror copy only (the CF node + // is untouched); its distances in the plane are then quantized like every other node + const q = quantizeInt8(vector); + bin = q.bytes; + scale = q.scale; + if (invMag === undefined) { + let magSq = 0; + for (const v of vector) magSq += v * v; + invMag = 1 / (Math.sqrt(magSq) || 1); + } + } else { + // Int8Array (converted) or raw bin view straight from the store decode — same bytes; + // pass them through untouched (never re-quantize) + bin = Buffer.from(vector.buffer, vector.byteOffset, vector.byteLength); + scale = node.scale ?? 1; + if (invMag === undefined) { + // legacy pre-invMag node: |v| ~= scale * |q|, the same fallback searchLayer uses + const q = + vector instanceof Int8Array ? vector : new Int8Array(vector.buffer, vector.byteOffset, vector.byteLength); + let magSq = 0; + for (let i = 0; i < q.length; i++) magSq += q[i] * q[i]; + invMag = 1 / ((Math.sqrt(magSq) || 1) * scale); + } + } + const level = node.level ?? 0; + const layer0 = planeConnectionIds(node[0]); + let upper: Uint32Array[] | null = null; + if (level >= 1) { + upper = []; + for (let l = 1; l <= level; l++) upper.push(planeConnectionIds(node[l])); + } + plane.writeNodeRaw(nodeId, level, bin, scale, invMag, layer0, upper); + } + + /** Mirror a node put into the plane; a plane failure never fails the CF write. */ + private mirrorNodePut(nodeId: number, node: any): void { + if (!this.planeEligible) return; + const vector = node?.vector; + const dims = Array.isArray(vector) ? vector.length : vector?.byteLength; + const plane = this.getPlane(dims); + if (!plane) return; + try { + this.writeNodeToPlane(plane, nodeId, node); + } catch (error) { + this.disablePlane(error); + } } + + private mirrorNodeRemove(nodeId: number): void { + if (!this.planeEligible) return; + const plane = this.getPlane(); + if (!plane) return; + try { + plane.clearNode(nodeId); + } catch (error) { + this.disablePlane(error); + } + } + + private mirrorEntryPoint(entryPointId: number, level: number | undefined, options?: any): void { + if (!this.planeEligible) return; + const plane = this.getPlane(); + if (!plane) return; + try { + plane.setEntryPoint(entryPointId, level ?? this.safeGetSync(entryPointId, options)?.level ?? 0); + } catch (error) { + this.disablePlane(error); + } + } + + private mirrorEntryPointCleared(): void { + if (!this.planeEligible) return; + const plane = this.getPlane(); + if (!plane) return; + try { + plane.setEntryPoint(PLANE_NO_ID, 0); + } catch (error) { + this.disablePlane(error); + } + } + + /** Disable the plane for this process; searches and writes fall back to the JS/CF path. */ + private disablePlane(error: unknown): void { + this.plane = null; + if (!this.planeDisabledLogged) { + this.planeDisabledLogged = true; + logger.error?.('disabling the HNSW native plane for this index (falling back to the JS path)', error); + } + } + + /** + * Delete the derived plane state. Called when the backing store is dropped or cleared + * (index drop, table drop/clear, reindex-from-scratch); the plane lazily rebuilds from the + * CF on next use. Unlinking while another worker still maps the old file is safe on POSIX — + * that worker keeps writing the orphaned inode until the schema-change signal resets its + * database instances. + */ + resetDerivedStorage(): void { + this.plane = undefined; + this.planeRetryAt = 0; + const filePath = this.planeFilePath(); + if (!filePath) return; + try { + unlinkSync(filePath); + } catch (error: any) { + if (error?.code !== 'ENOENT') logger.warn?.('could not delete the HNSW plane file', error); + } + } + + /** + * Native search over the plane: one NAPI crossing, traversal on the libuv pool, promise + * resolution maps node ids back to primary keys through the existing pk resolution. The + * predicate adapter runs on this thread's event loop (batched over a ThreadsafeFunction), so + * this promise must never be awaited by code the predicate itself blocks on; the normal + * request path awaits it safely. + */ + private searchPlane( + plane: HnswPlane, + target: number[], + ef: number, + filter: ((primaryKey: Id) => boolean) | undefined, + filterState: FilterState | undefined, + options: any + ): Promise { + const query = Float32Array.from(target); + let resultPromise: Promise<{ id: number; distance: number }[]>; + if (filter && filterState) { + // the plane bounds filtered visits at ef * filterExpansion; recover the multiplier from + // the already-resolved JS budget so both paths stop at the same visit count + const planeFilterExpansion = Math.max(1, Math.round(filterState.maxVisits / ef)); + const predicate = (ids: number[]): Uint8Array => { + const verdicts = new Uint8Array(ids.length); + for (let i = 0; i < ids.length; i++) { + const primaryKey = this.safeGetSync(ids[i], options)?.primaryKey; + if (primaryKey !== undefined && this.admit(filter, filterState, primaryKey)) verdicts[i] = 1; + } + return verdicts; + }; + resultPromise = plane.searchWithPredicate(query, ef, ef, predicate, planeFilterExpansion); + } else { + resultPromise = plane.search(query, ef, ef); + } + return resultPromise.then((hits) => { + const entries: any[] = []; + for (const hit of hits) { + const primaryKey = this.safeGetSync(hit.id, options)?.primaryKey; + if (primaryKey === undefined) continue; // deleted/reused id raced the search + entries.push({ key: primaryKey, distance: hit.distance }); + } + // nodesVisited stays 0 here: layer-0 visits happen inside the native traversal + // (filterEvaluations is still counted by the predicate adapter) + return withStats(entries, filterState); + }); + } + index(primaryKey: Id, vector: number[], existingVector?: number[], options: any = {}) { // Reject non-finite components before touching the graph. NaN in particular poisons // bisectInsert (arr[mid].distance <= NaN is always false → returns 0, pinning the @@ -358,12 +657,15 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', nodeId); this.indexStore.put(ENTRY_POINT, nodeId, options); + this.mirrorNodePut(nodeId, node); + this.mirrorEntryPoint(nodeId, level, options); return; } // Generate random level for this new element const level = oldNode.level ?? Math.min(Math.floor(-Math.log(this.random()) * this.mL), MAX_LEVEL); let currentLevel = entryPoint.level; + let mirrorEntryPointAfterPut = false; if (level > currentLevel) { // if we are at a higher level, make this the new entry point if (typeof nodeId !== 'number') { @@ -371,6 +673,10 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', nodeId); this.indexStore.put(ENTRY_POINT, nodeId, options); + // the CF put is invisible until commit, but a plane write is immediately visible — + // mirror the promotion only after the node's own slot lands (below), so a concurrent + // native search never descends from a not-yet-written entry slot + mirrorEntryPointAfterPut = true; } // Pure descent — only neighbors[0] is used — so it runs greedily for the same reason @@ -508,18 +814,17 @@ export class HierarchicalNavigableSmallWorld { } // Store the new element - this.indexStore.put( - nodeId, - { - vector: storedVector, - scale: storedScale, - invMag, - level, - primaryKey, - ...connections, - }, - options - ); + const storedNode = { + vector: storedVector, + scale: storedScale, + invMag, + level, + primaryKey, + ...connections, + }; + this.indexStore.put(nodeId, storedNode, options); + this.mirrorNodePut(nodeId, storedNode); + if (mirrorEntryPointAfterPut) this.mirrorEntryPoint(nodeId, level, options); } else { // removal of this node, but first make sure we have a valid entry point if (entryPointId === nodeId) { @@ -553,6 +858,7 @@ export class HierarchicalNavigableSmallWorld { if (entryPointId === undefined) { // no nodes left in index this.indexStore.remove(ENTRY_POINT, options); + this.mirrorEntryPointCleared(); } else { // set the new entry point if (typeof entryPointId !== 'number') { @@ -560,9 +866,11 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', entryPointId); this.indexStore.put(ENTRY_POINT, entryPointId, options); + this.mirrorEntryPoint(entryPointId, undefined, options); } } this.indexStore.remove(nodeId, options); + this.mirrorNodeRemove(nodeId); // A re-insert of this primary key must get a fresh node rather than the deleted node's id. this.indexStore.remove(safeKey, options); } @@ -618,6 +926,7 @@ export class HierarchicalNavigableSmallWorld { } for (const [id, updatedNode] of updatedNodes) { this.indexStore.put(id, updatedNode, options); + this.mirrorNodePut(id, updatedNode); } for (const [key, orphanVector] of needsReindexing) { // If the orphan IS the current entry point, re-running @@ -651,6 +960,7 @@ export class HierarchicalNavigableSmallWorld { } if (replacementEP !== undefined) { this.indexStore.put(ENTRY_POINT, replacementEP, options); + this.mirrorEntryPoint(replacementEP, undefined, options); } } this.index(key, orphanVector, orphanVector, options); @@ -1163,6 +1473,24 @@ export class HierarchicalNavigableSmallWorld { filterEvaluations: 0, } : undefined; + if (this.planeEligible) { + const plane = this.getPlane(target.length); + if (plane) { + // Native cutover: same resolved ef, same predicate semantics; resolves to the same + // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all + // post-load behavior are unchanged. searchByIndex handles the promise. + return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { + // a failed native search disables the plane and re-runs this query on the JS path + this.disablePlane(error); + return this.search( + { target, value, descending, distance, comparator, ef, filterExpansion }, + context, + filter, + minResults + ); + }); + } + } let entryPoint = this.getEntryPoint(options); if (!entryPoint) return withStats([], filterState); let entryPointId = entryPoint.id; diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts new file mode 100644 index 0000000000..d40d83efa4 --- /dev/null +++ b/resources/indexes/hnswPlaneBinding.ts @@ -0,0 +1,88 @@ +import { join } from 'node:path'; +import { PACKAGE_ROOT } from '../../utility/packageUtils.js'; +import { loggerWithTag } from '../../utility/logging/logger.ts'; + +const logger = loggerWithTag('HNSW'); + +export interface PlaneSearchHit { + id: number; + distance: number; +} + +/** + * NAPI surface of the native HNSW traversal plane (native/hnsw-plane). Dual-write phase 1 uses + * only the raw mirroring calls (host-allocated ids; the plane's own insert()/remove() allocator + * path is bypassed by design) plus the search entry points. + */ +export interface HnswPlane { + writeNodeRaw( + id: number, + level: number, + vector: Buffer, + scale: number, + invMag: number, + neighbors: Uint32Array, + upper: Uint32Array[] | null + ): void; + clearNode(id: number): void; + setEntryPoint(id: number, level: number): void; + getEntryPoint(): number[]; + search( + vector: Float32Array, + k: number, + ef: number, + filter?: Uint8Array | null, + filterExpansion?: number | null + ): Promise; + searchWithPredicate( + vector: Float32Array, + k: number, + ef: number, + predicate: (ids: number[]) => Uint8Array, + filterExpansion?: number | null + ): Promise; + searchSync(vector: Float32Array, k: number, ef: number): PlaneSearchHit[]; + idHighWater(): number; + getWatermark(): number; + setWatermark(txn: number): void; + flush(): void; +} + +export interface HnswPlaneConstructor { + create(path: string, dims: number, layer0Cap: number, maxNodes: number): HnswPlane; + open(path: string): HnswPlane; +} + +/** Entry-point id meaning "none" (u32::MAX in the plane header). */ +export const PLANE_NO_ID = 0xffffffff; + +/** + * Where an index's plane file lives: next to its store, named by the dbiKey + * (`table/attribute`, flattened to a single file name). Exposed separately from the index + * instance so crash-recovery drop paths can remove the file without opening the index. + */ +export function planeFilePathFor(storePath: string, storeName: string): string { + return join(storePath, `${storeName.replace(/[/\\]/g, '.')}.hnsw`); +} + +// The compiled artifact is optional: harper installs carry no cargo toolchain, so absence just +// means nativePlane-flagged indexes run the existing JS path. Build locally with +// `npm run build:hnsw-plane`. +const BINDING_PATH = join(PACKAGE_ROOT, 'native', 'hnsw-plane', 'hnsw-plane.node'); + +let binding: HnswPlaneConstructor | null | undefined; + +/** The native plane constructor, or null when the compiled artifact is unavailable (warns once). */ +export function getPlaneBinding(): HnswPlaneConstructor | null { + if (binding !== undefined) return binding; + try { + binding = require(BINDING_PATH).Plane as HnswPlaneConstructor; + } catch (error) { + binding = null; + logger.warn?.( + `The hnsw-plane native module is not available (${(error as Error).message}); ` + + `indexes with nativePlane enabled will use the JS search path. Build it with: npm run build:hnsw-plane` + ); + } + return binding; +} diff --git a/resources/search.ts b/resources/search.ts index cbdd3acad6..ed7d3c1d04 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -1,7 +1,7 @@ import { ClientError, IndexRebuildingError, Violation } from '../utility/errors/hdbError.ts'; import { OVERFLOW_MARKER, MAX_SEARCH_KEY_LENGTH, SEARCH_TYPES } from '../utility/lmdb/terms.ts'; import { compareKeys, MAXIMUM_KEY, writeKey } from 'ordered-binary'; -import { SKIP } from '@harperfast/extended-iterable'; +import { ExtendedIterable, SKIP } from '@harperfast/extended-iterable'; import { INVALIDATED, EVICTED, freezeRecord } from './Table.ts'; import type { DirectCondition, Id } from './ResourceInterface.ts'; import { RequestTarget } from './RequestTarget.ts'; @@ -510,26 +510,55 @@ export function searchByIndex( // exploring until it has enough MATCHING results, rather than post-filtering an under-filled // candidate set. Only indexes that opt in (filteredSearch) receive it; others post-filter as before. const recordFilter = index.customIndex.filteredSearch ? searchCondition.recordFilter : undefined; - const loaded = index.customIndex.search(searchCondition, context, recordFilter, minResults).map((entry) => { - // if the custom index returns an entry with metadata, merge it with the loaded entry - if (typeof entry === 'object' && entry) { - const { key, ...otherProps } = entry; - if (key == null) return SKIP; // primaryKey missing from HNSW node — skip rather than crash - const loadedEntry = Table.primaryStore.getEntry(key, { - transaction: context && Table._readTxnForContext(context), - }); - if (!loadedEntry) return SKIP; // record was deleted/expired or not yet visible - freezeRecord(loadedEntry?.value); - recordRead(loadedEntry); - return { ...otherProps, ...loadedEntry }; + const searched = index.customIndex.search(searchCondition, context, recordFilter, minResults); + const processEntries = (entries: any[]) => { + const loaded = entries.map((entry) => { + // if the custom index returns an entry with metadata, merge it with the loaded entry + if (typeof entry === 'object' && entry) { + const { key, ...otherProps } = entry; + if (key == null) return SKIP; // primaryKey missing from HNSW node — skip rather than crash + const loadedEntry = Table.primaryStore.getEntry(key, { + transaction: context && Table._readTxnForContext(context), + }); + if (!loadedEntry) return SKIP; // record was deleted/expired or not yet visible + freezeRecord(loadedEntry?.value); + recordRead(loadedEntry); + return { ...otherProps, ...loadedEntry }; + } + return entry; + }); + if (index.customIndex.rescoreResults) { + const rescored = index.customIndex.rescoreResults(loaded, searchCondition, comparator, attribute_name); + if (rescored != null) return rescored as any; } - return entry; - }); - if (index.customIndex.rescoreResults) { - const rescored = index.customIndex.rescoreResults(loaded, searchCondition, comparator, attribute_name); - if (rescored != null) return rescored as any; + return loaded; + }; + if (typeof (searched as any)?.then === 'function') { + // An async custom-index search (the native HNSW plane runs off the event loop and + // resolves its candidate list as a promise). Apply the same load + rescore pipeline + // once it resolves, exposed as a lazily-resolving iterable — consumable through async + // iteration only, like the promise-entry filter paths above. + const pending = (searched as Promise).then(processEntries); + const results: any = new ExtendedIterable(); + results.iterate = () => { + let inner: Iterator | null = null; + return { + next() { + if (inner) return inner.next(); + return pending.then((entries) => { + inner = entries[Symbol.iterator](); + return inner.next(); + }); + }, + return(value?: any) { + (inner as any)?.return?.(value); + return { done: true, value }; + }, + }; + }; + return results; } - return loaded; + return processEntries(searched); } return index.getRange(rangeOptions).map( filter diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js new file mode 100644 index 0000000000..ec12ea4db4 --- /dev/null +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -0,0 +1,271 @@ +/** + * Coverage for the native HNSW traversal plane, phase 1 (dual-write + opt-in search cutover, + * hnsw-native-plane.md §8): with `nativePlane: true` every graph mutation is mirrored into a + * plane file next to the index store and searches run through the native module, while the + * RocksDB column family stays authoritative. The plane graph must be a bit-identical mirror of + * the CF graph (same ids/levels/edges), so a native search over it must return the same + * candidates as the JS traversal of the CF graph at equal ef. + * + * The whole suite is skipped when the optional native artifact is absent — build it with + * `npm run build:hnsw-plane`. + */ +require('../testUtils'); +const assert = require('node:assert'); +const fs = require('node:fs'); +const { setupTestDBPath } = require('../testUtils'); +const { table, resetDatabases } = require('#src/resources/databases'); +const { HierarchicalNavigableSmallWorld } = require('#src/resources/indexes/HierarchicalNavigableSmallWorld'); +const { getPlaneBinding } = require('#src/resources/indexes/hnswPlaneBinding'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +async function fromAsync(iterable) { + const out = []; + for await (const value of iterable) out.push(value); + return out; +} + +const DIMS = 24; +const N = 1200; +const EF = 200; +const DB = 'test'; + +// Deterministic clustered corpus: 20 well-separated centers plus per-vector noise, so graphs are +// meaningful (uniform-random high-dim corpora defeat ANN) and runs are reproducible. +let seedState = 42; +function rand() { + seedState = (seedState * 1103515245 + 12345) % 2147483648; + return seedState / 2147483648; +} +const centers = []; +for (let c = 0; c < 20; c++) { + const center = new Array(DIMS); + for (let d = 0; d < DIMS; d++) center[d] = rand() * 2 - 1; + centers.push(center); +} +function makeVector(i) { + const center = centers[i % centers.length]; + const v = new Array(DIMS); + for (let d = 0; d < DIMS; d++) v[d] = center[d] + (rand() - 0.5) * 0.2; + return v; +} + +describe('HNSW native plane dual-write', function () { + if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; // custom object index is RocksDB-only here + if (!getPlaneBinding()) { + it.skip('skipped: native hnsw-plane module not built (npm run build:hnsw-plane)', () => {}); + return; + } + let PlaneTest; + const vectors = new Map(); // id → current vector + before(async () => { + setupTestDBPath(); + setMainIsWorker(true); + PlaneTest = table({ + table: 'PlaneTest', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + for (let i = 0; i < N; i++) { + const vector = makeVector(i); + vectors.set(i, vector); + await PlaneTest.put(i, { name: 'rec' + i, vector }); + } + }); + + function customIndex() { + return PlaneTest.indices.vector.customIndex; + } + + // A flag-off HierarchicalNavigableSmallWorld over the SAME index store: the pure JS traversal + // of the CF graph, the reference the plane must match. + function jsReference() { + return new HierarchicalNavigableSmallWorld(PlaneTest.indices.vector, {}); + } + + async function searchBoth(target, filter) { + const condition = { target, comparator: 'sort', distance: 'cosine', ef: EF }; + const planeResult = customIndex().search(condition, { transaction: undefined }, filter); + assert.equal(typeof planeResult?.then, 'function', 'the flagged index should search through the plane (async)'); + const planeEntries = await planeResult; + const jsEntries = jsReference().search(condition, { transaction: undefined }, filter); + assert.equal(typeof jsEntries?.then, 'undefined', 'the reference instance must use the JS path'); + return { planeEntries, jsEntries }; + } + + // Same candidate set; same order wherever consecutive distances are distinct (the plane + // computes f32 distances vs the JS f64, so exact ties may swap — and post-load rescoring + // restores exact order for real queries anyway). + function assertParity(planeEntries, jsEntries) { + const planeKeys = planeEntries.map((e) => e.key); + const jsKeys = jsEntries.map((e) => e.key); + assert.deepEqual( + [...planeKeys].sort((a, b) => a - b), + [...jsKeys].sort((a, b) => a - b), + 'plane and JS searches must return the same candidate set' + ); + for (let i = 0; i < planeEntries.length; i++) { + if (planeKeys[i] === jsKeys[i]) continue; + const distanceGap = Math.abs(planeEntries[i].distance - jsEntries[i].distance); + assert.ok( + distanceGap < 1e-4, + `order diverged at rank ${i} (${planeKeys[i]} vs ${jsKeys[i]}) with distance gap ${distanceGap}` + ); + } + } + + it('dual-writes into a plane file next to the index store', () => { + const planePath = customIndex().planeFilePath(); + assert.ok(planePath, 'the index should resolve a plane file path'); + assert.ok(fs.existsSync(planePath), 'inserts through the flagged index should have created the plane file'); + }); + + it('plane search returns the same candidates as the JS path at equal ef', async () => { + for (const probe of [3, 77, 500]) { + const { planeEntries, jsEntries } = await searchBoth(vectors.get(probe)); + assert.ok(planeEntries.length >= 100, `expected a full candidate list, got ${planeEntries.length}`); + assertParity(planeEntries, jsEntries); + assert.equal(planeEntries[0].key, probe, 'the probe vector should be its own nearest neighbor'); + } + }); + + it('parity holds after update-in-place and delete (including neighbor repair)', async () => { + for (let i = 0; i < 100; i++) { + const vector = makeVector(i + 5000); + vectors.set(i, vector); + await PlaneTest.put(i, { name: 'rec' + i, vector }); + } + for (let i = 100; i < 200; i++) { + vectors.delete(i); + await PlaneTest.delete(i); + } + for (const probe of [0, 50, 300]) { + const { planeEntries, jsEntries } = await searchBoth(vectors.get(probe)); + assertParity(planeEntries, jsEntries); + for (const entry of planeEntries) { + assert.ok(entry.key < 100 || entry.key >= 200, `deleted record ${entry.key} returned by the plane`); + } + } + }); + + it('predicate-filtered plane search returns only predicate-passing records', async () => { + const filter = (primaryKey) => primaryKey % 3 === 0; + const { planeEntries, jsEntries } = await searchBoth(vectors.get(21), filter); + assert.ok(planeEntries.length > 0, 'filtered plane search should return results'); + for (const entry of planeEntries) { + assert.equal(entry.key % 3, 0, `record ${entry.key} does not pass the predicate`); + } + // budget/pipelining semantics differ slightly under selective filters, so assert the head + // of the ranking agrees rather than the full candidate set + assert.equal(planeEntries[0].key, jsEntries[0].key, 'best filtered match should agree with the JS path'); + }); + + it('full-stack query runs through the plane and rescoring restores exact order', async () => { + const target = vectors.get(42); + const results = await fromAsync( + PlaneTest.search({ + sort: { attribute: 'vector', target, distance: 'cosine' }, + select: ['id', '$distance'], + limit: 10, + }) + ); + assert.equal(results.length, 10); + assert.equal(results[0].id, 42, 'the probe vector should be its own nearest neighbor'); + for (let i = 1; i < results.length; i++) { + assert.ok(results[i].$distance >= results[i - 1].$distance, 'rescored results must be ordered'); + } + // conditions alongside the vector sort exercise the predicate pushdown through search.ts + const filtered = await fromAsync( + PlaneTest.search({ + sort: { attribute: 'vector', target, distance: 'cosine' }, + conditions: [{ attribute: 'name', comparator: 'gt', value: 'rec9' }], + select: ['id', 'name'], + limit: 20, + }) + ); + assert.ok(filtered.length > 0); + for (const record of filtered) assert.ok(record.name > 'rec9'); + // threshold comparator: int8 suppresses the traversal-time limit and rescoreResults + // re-filters on exact distances post-load — via the plane path + const within = await fromAsync( + PlaneTest.search({ + conditions: [{ attribute: 'vector', comparator: 'le', value: 0.05, target }], + select: ['id', '$distance'], + }) + ); + assert.ok(within.length > 0, 'le threshold query should return nearby records'); + for (const record of within) assert.ok(record.$distance <= 0.05, `distance ${record.$distance} exceeds threshold`); + }); + + it('reopens the same plane file across a restart', async () => { + const planePath = customIndex().planeFilePath(); + const inodeBefore = fs.statSync(planePath).ino; + resetDatabases(); + PlaneTest = table({ + table: 'PlaneTest', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + const { planeEntries, jsEntries } = await searchBoth(vectors.get(7)); + assertParity(planeEntries, jsEntries); + assert.equal(fs.statSync(planePath).ino, inodeBefore, 'restart should reopen the plane file, not recreate it'); + }); + + it('builds the plane lazily when the flag is enabled on an existing index', async () => { + let Later = table({ + table: 'PlaneLater', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW' }, type: 'Array' }, + ], + }); + const laterVectors = new Map(); + for (let i = 0; i < 300; i++) { + const vector = makeVector(i + 9000); + laterVectors.set(i, vector); + await Later.put(i, { vector }); + } + assert.ok( + !Later.indices.vector.customIndex.planeFilePath() || + !fs.existsSync(Later.indices.vector.customIndex.planeFilePath()), + 'no plane file before the flag is enabled' + ); + resetDatabases(); + Later = table({ + table: 'PlaneLater', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + assert.ok(!Later.indexingOperation, 'nativePlane is search-only: enabling it must not trigger a reindex'); + const condition = { target: laterVectors.get(5), comparator: 'sort', distance: 'cosine', ef: EF }; + const planeEntries = await Later.indices.vector.customIndex.search(condition, { transaction: undefined }); + const jsEntries = new HierarchicalNavigableSmallWorld(Later.indices.vector, {}).search(condition, { + transaction: undefined, + }); + assert.ok(fs.existsSync(Later.indices.vector.customIndex.planeFilePath()), 'first search should build the plane'); + assert.deepEqual( + planeEntries.map((e) => e.key).sort((a, b) => a - b), + jsEntries.map((e) => e.key).sort((a, b) => a - b), + 'the lazily-mirrored plane must return the same candidate set' + ); + await Later.dropTable(); + }); + + it('index drop removes the plane file', async () => { + const planePath = customIndex().planeFilePath(); + assert.ok(fs.existsSync(planePath)); + await PlaneTest.dropTable(); + assert.ok(!fs.existsSync(planePath), 'dropping the table should delete the plane file'); + }); +}); From efd556aa2e3454ad388adb6b44be5e1fe67223ee Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 10:31:36 -0600 Subject: [PATCH 14/69] Review round 1: watermark-gated plane readiness, sync-iteration guard, nearest-edge truncation Cross-model findings addressed: the initial mirror now stamps the plane watermark on completion and searches stay on the JS path until it is set, so a partially-built or crash-abandoned mirror is never served (rebuilt after a generous age); create-race losers retry attach on a short cadence and mirror into the in-progress file instead of dropping writes for the old 10s TTL; a thrown mirror error unlinks the partial file, and disabling the plane deletes the file so a restart cannot reopen a mirror that stopped receiving writes. Truncation to the layer-0/upper caps now keeps the nearest edges rather than an array prefix. searchByIndex's promise-backed iterable throws a descriptive error under synchronous iteration instead of looping on promise-shaped results, SKIP entries are filtered before rescoring, and the crash-recovery plane unlink logs at warn. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- resources/databases.ts | 4 +- .../HierarchicalNavigableSmallWorld.ts | 135 ++++++++++++++---- resources/search.ts | 41 +++--- unitTests/resources/vectorIndexPlane.test.js | 9 ++ 4 files changed, 144 insertions(+), 45 deletions(-) diff --git a/resources/databases.ts b/resources/databases.ts index 0af045f6a3..f82af610ef 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -3042,7 +3042,9 @@ function completeInterruptedDrop(rootStore, attributesDbi, databaseName: string, try { unlinkSync(planeFilePathFor(rootStore.path, columnName)); } catch (error: any) { - if (error?.code !== 'ENOENT') logger.debug(`could not delete plane file for ${columnName}`, error); + // a stale plane left behind (e.g. Windows EBUSY while still mapped) would be + // opened over a fresh same-name CF, so a failed delete must be visible + if (error?.code !== 'ENOENT') logger.warn(`could not delete the HNSW plane file for ${columnName}`, error); } } } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 325a76a7a6..f0be37aa0c 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -43,14 +43,19 @@ function dequantizeInt8(q: Int8Array, scale: number): number[] { return out; } -/** Connection ids for the plane mirror: ids only — per-edge distances are dropped (recomputed natively). */ -function planeConnectionIds(connections: Connection[] | undefined): Uint32Array { +/** + * Connection ids for the plane mirror: ids only — per-edge distances are dropped (recomputed + * natively). A list past `cap` keeps the NEAREST edges rather than an arbitrary array prefix, + * so transient JS overshoot (grace above the cap) and the plane's tighter upper cap truncate by + * the same distance policy the JS prune uses. + */ +function planeConnectionIds(connections: Connection[] | undefined, cap: number): Uint32Array { if (!connections?.length) return new Uint32Array(0); - const ids: number[] = []; - for (const { id } of connections) { - if (typeof id === 'number' && id >= 0) ids.push(id); + let usable = connections.filter((connection) => typeof connection.id === 'number' && connection.id >= 0); + if (usable.length > cap) { + usable = usable.sort((a, b) => a.distance - b.distance).slice(0, cap); } - return Uint32Array.from(ids); + return Uint32Array.from(usable, (connection) => connection.id); } // Auto-scaled search ef, used only when an index does not explicitly configure efConstructionSearch @@ -117,17 +122,28 @@ function autoScaleEfConstruction(nodeCount: number): number { // this only has to be short enough that a table growing from empty picks up a larger ef promptly. const NODE_COUNT_TTL = 10_000; -// Native traversal-plane geometry (dual-write phase 1, hnsw-native-plane.md §4/§10). The layer-0 -// cap matches the JS graph's effective cap (M<<1 <<2 under optimizeRouting = 128; writeNodeRaw -// truncates the transient 160 overshoot). maxNodes is a fixed sparse reservation — pages -// materialize on write — and ids at or past it are rejected by the crate, which disables the -// plane for this process (reservation growth is a phase-2 open item). +// Native traversal-plane geometry (see hnsw-native-plane.md). The layer-0 cap must cover the JS +// graph's effective cap (M<<1 <<2 under optimizeRouting = 128); writeNodeRaw truncates the +// transient 160 overshoot. maxNodes is a fixed sparse reservation — pages materialize on write — +// and ids at or past it are rejected by the crate, which disables the plane for this process. const PLANE_LAYER0_CAP = 128; +// Ids kept per upper level — must match the crate's format UPPER_CAP, which truncates whatever +// is passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. +const PLANE_UPPER_CAP = 32; const PLANE_MAX_NODES = 1 << 24; // An existing plane file that cannot be opened is normally another worker mid-create (retry); // past this age it is a crashed create and is deleted and rebuilt — the plane is derived state, // the index column family stays authoritative. const PLANE_STALE_CREATE_MS = 60_000; +// Retry cadence while another worker holds the create: its header lands within moments of the +// exclusive open, so a long deferral would silently drop this worker's mirror writes. +const PLANE_ATTACH_RETRY_MS = 250; +// A plane whose initial mirror never completed (watermark still 0) is never searched; past this +// age the builder is taken as crashed and the file is rebuilt from the CF. +const PLANE_INCOMPLETE_REBUILD_MS = 3_600_000; +// Watermark stamped when the initial full mirror completes; 0 = still building (or crashed +// mid-build). Phase-2 replay wiring will carry real transaction ids, which are also nonzero. +const PLANE_MIRRORED = 1; class MinHeap { private data: Candidate[] = []; @@ -261,11 +277,12 @@ export class HierarchicalNavigableSmallWorld { private convertedNodes = new WeakMap(); private nodeCount = 0; private nodeCountAt = 0; - // Native traversal plane (dual-write phase 1): the CF graph stays authoritative; every graph - // mutation is mirrored into the plane file and search runs native when the flag is on. + // Native traversal plane (dual-write): the CF graph stays authoritative; every graph mutation + // is mirrored into the plane file and search runs native when the flag is on. // undefined = not yet attached (may retry), null = unavailable or disabled for this process. private plane: HnswPlane | null | undefined; private planeEligible = false; + private planeReady = false; private planeRetryAt = 0; private planeDisabledLogged = false; constructor(indexStore: any, options: any) { @@ -299,8 +316,8 @@ export class HierarchicalNavigableSmallWorld { if (options.filterExpansion !== undefined) this.filterExpansion = options.filterExpansion; } if (options?.nativePlane) { - // The plane stores int8 bins and computes asymmetric cosine only (phase 1), so the flag - // is a no-op for float (quantization: "none") and non-cosine indexes. + // The plane stores int8 bins and computes asymmetric cosine only, so the flag is a + // no-op for float (quantization: "none") and non-cosine indexes. this.planeEligible = this.int8 && this.distance === cosineDistance; if (!this.planeEligible) { logger.info?.('nativePlane is only supported for int8-quantized cosine HNSW indexes; using the JS search path'); @@ -323,9 +340,11 @@ export class HierarchicalNavigableSmallWorld { * eventual creation's full mirror reads the then-current CF state. * * Multi-worker create races are settled by an exclusive open ('wx') of the file itself: the - * winner creates and mirrors, losers see EEXIST and open — transiently failing (and retrying - * after a TTL) while the winner is still writing the header. A crashed create leaves an - * unopenable file; once it is older than PLANE_STALE_CREATE_MS it is deleted and rebuilt. + * winner creates and mirrors, losers see EEXIST and open (retrying on a short cadence while + * the winner is still writing the header, so their mirror writes drop for at most moments). + * A loser attached mid-build mirrors its own writes immediately but planeSearchReady keeps + * its searches on the JS path until the builder stamps the mirror complete. A crashed create + * leaves an unopenable file; once older than PLANE_STALE_CREATE_MS it is deleted and rebuilt. */ private getPlane(dims?: number): HnswPlane | null { if (this.plane !== undefined) return this.plane; @@ -345,7 +364,8 @@ export class HierarchicalNavigableSmallWorld { return (this.plane = Plane.open(filePath)); } catch (openError) { if (now - statSync(filePath).mtimeMs <= PLANE_STALE_CREATE_MS) { - this.planeRetryAt = now + NODE_COUNT_TTL; + // another worker is between its exclusive create and the header write + this.planeRetryAt = now + PLANE_ATTACH_RETRY_MS; return null; } logger.warn?.('deleting an unopenable HNSW plane file left by an interrupted create', openError); @@ -358,12 +378,23 @@ export class HierarchicalNavigableSmallWorld { try { fd = openSync(filePath, 'wx'); } catch { - // another worker won the create race; open it after its header lands - this.planeRetryAt = now + NODE_COUNT_TTL; + // another worker won the create race; its header lands within moments + this.planeRetryAt = now + PLANE_ATTACH_RETRY_MS; return null; } closeSync(fd); - return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); + try { + return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); + } catch (createError) { + // never leave a file a later open would trust as a complete mirror + try { + unlinkSync(filePath); + } catch { + // the disable below already forces the JS path for this process + } + this.disablePlane(createError); + return null; + } } catch (error) { this.planeRetryAt = now + NODE_COUNT_TTL; logger.warn?.('could not attach the HNSW plane file; will retry', error); @@ -371,6 +402,32 @@ export class HierarchicalNavigableSmallWorld { } } + /** + * True once the plane's initial full mirror completed (watermark stamped nonzero at the end + * of createAndMirrorPlane). A plane opened mid-build keeps receiving this worker's mirror + * writes but must not serve searches — its graph is incomplete; one abandoned by a crashed + * builder would stay unusable forever, so past a generous age it is rebuilt from the CF. + */ + private planeSearchReady(plane: HnswPlane): boolean { + if (this.planeReady) return true; + if (plane.getWatermark() >= PLANE_MIRRORED) { + this.planeReady = true; + return true; + } + const filePath = this.planeFilePath(); + if (filePath) { + try { + if (Date.now() - statSync(filePath).mtimeMs > PLANE_INCOMPLETE_REBUILD_MS) { + logger.warn?.('rebuilding an HNSW plane file whose initial mirror never completed'); + this.resetDerivedStorage(); + } + } catch { + // stat raced a concurrent delete; the next attach sorts it out + } + } + return false; + } + /** * Create the plane file and fully mirror the existing CF graph into it (the "first enable" * build — a pure copy of the same graph, so plane and CF are bit-identical by construction; @@ -396,6 +453,10 @@ export class HierarchicalNavigableSmallWorld { if (typeof entryPointId === 'number') { plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); } + // stamped last: a crash or thrown mirror error leaves the watermark 0, and + // planeSearchReady refuses to serve searches from a mirror that never completed + plane.setWatermark(PLANE_MIRRORED); + this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); return plane; } @@ -435,11 +496,11 @@ export class HierarchicalNavigableSmallWorld { } } const level = node.level ?? 0; - const layer0 = planeConnectionIds(node[0]); + const layer0 = planeConnectionIds(node[0], PLANE_LAYER0_CAP); let upper: Uint32Array[] | null = null; if (level >= 1) { upper = []; - for (let l = 1; l <= level; l++) upper.push(planeConnectionIds(node[l])); + for (let l = 1; l <= level; l++) upper.push(planeConnectionIds(node[l], PLANE_UPPER_CAP)); } plane.writeNodeRaw(nodeId, level, bin, scale, invMag, layer0, upper); } @@ -491,9 +552,23 @@ export class HierarchicalNavigableSmallWorld { } } - /** Disable the plane for this process; searches and writes fall back to the JS/CF path. */ + /** + * Disable the plane for this process; searches and writes fall back to the JS/CF path. The + * file is deleted too: dual-write stops here, so a mirror kept on disk would be reopened + * after a restart missing every post-disable mutation. Another worker still mapping the old + * inode keeps itself consistent until the schema-change/restart cycle rebuilds everything. + */ private disablePlane(error: unknown): void { this.plane = null; + this.planeReady = false; + const filePath = this.planeFilePath(); + if (filePath) { + try { + unlinkSync(filePath); + } catch (unlinkError: any) { + if (unlinkError?.code !== 'ENOENT') logger.warn?.('could not delete the disabled HNSW plane file', unlinkError); + } + } if (!this.planeDisabledLogged) { this.planeDisabledLogged = true; logger.error?.('disabling the HNSW native plane for this index (falling back to the JS path)', error); @@ -509,6 +584,7 @@ export class HierarchicalNavigableSmallWorld { */ resetDerivedStorage(): void { this.plane = undefined; + this.planeReady = false; this.planeRetryAt = 0; const filePath = this.planeFilePath(); if (!filePath) return; @@ -537,8 +613,11 @@ export class HierarchicalNavigableSmallWorld { const query = Float32Array.from(target); let resultPromise: Promise<{ id: number; distance: number }[]>; if (filter && filterState) { - // the plane bounds filtered visits at ef * filterExpansion; recover the multiplier from - // the already-resolved JS budget so both paths stop at the same visit count + // The plane bounds filtered visits at ef * filterExpansion; recover the multiplier from + // the already-resolved JS budget so both paths stop at the same visit count. The u32 + // multiplier makes this exact only to the nearest multiple of ef: when a limit-widened + // ef exceeds maxVisits the floor of 1 over-explores (latency, never correctness), and + // ordinary rounding keeps the budgets within half an ef of each other. const planeFilterExpansion = Math.max(1, Math.round(filterState.maxVisits / ef)); const predicate = (ids: number[]): Uint8Array => { const verdicts = new Uint8Array(ids.length); @@ -1475,7 +1554,7 @@ export class HierarchicalNavigableSmallWorld { : undefined; if (this.planeEligible) { const plane = this.getPlane(target.length); - if (plane) { + if (plane && this.planeSearchReady(plane)) { // Native cutover: same resolved ef, same predicate semantics; resolves to the same // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all // post-load behavior are unchanged. searchByIndex handles the promise. diff --git a/resources/search.ts b/resources/search.ts index ed7d3c1d04..16c96850d9 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -512,21 +512,23 @@ export function searchByIndex( const recordFilter = index.customIndex.filteredSearch ? searchCondition.recordFilter : undefined; const searched = index.customIndex.search(searchCondition, context, recordFilter, minResults); const processEntries = (entries: any[]) => { - const loaded = entries.map((entry) => { - // if the custom index returns an entry with metadata, merge it with the loaded entry - if (typeof entry === 'object' && entry) { - const { key, ...otherProps } = entry; - if (key == null) return SKIP; // primaryKey missing from HNSW node — skip rather than crash - const loadedEntry = Table.primaryStore.getEntry(key, { - transaction: context && Table._readTxnForContext(context), - }); - if (!loadedEntry) return SKIP; // record was deleted/expired or not yet visible - freezeRecord(loadedEntry?.value); - recordRead(loadedEntry); - return { ...otherProps, ...loadedEntry }; - } - return entry; - }); + const loaded = entries + .map((entry) => { + // if the custom index returns an entry with metadata, merge it with the loaded entry + if (typeof entry === 'object' && entry) { + const { key, ...otherProps } = entry; + if (key == null) return SKIP; // primaryKey missing from HNSW node — skip rather than crash + const loadedEntry = Table.primaryStore.getEntry(key, { + transaction: context && Table._readTxnForContext(context), + }); + if (!loadedEntry) return SKIP; // record was deleted/expired or not yet visible + freezeRecord(loadedEntry?.value); + recordRead(loadedEntry); + return { ...otherProps, ...loadedEntry }; + } + return entry; + }) + .filter((entry) => entry !== SKIP); if (index.customIndex.rescoreResults) { const rescored = index.customIndex.rescoreResults(loaded, searchCondition, comparator, attribute_name); if (rescored != null) return rescored as any; @@ -540,7 +542,14 @@ export function searchByIndex( // iteration only, like the promise-entry filter paths above. const pending = (searched as Promise).then(processEntries); const results: any = new ExtendedIterable(); - results.iterate = () => { + results.iterate = (options?: { async?: boolean }) => { + // fail loudly rather than hand a synchronous consumer promise-shaped iterator + // results (which a bare for-of would spin on forever) + if (!options?.async) { + throw new Error( + 'This index resolves search results asynchronously; the results must be consumed with async iteration' + ); + } let inner: Iterator | null = null; return { next() { diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index ec12ea4db4..c72a237787 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -200,6 +200,15 @@ describe('HNSW native plane dual-write', function () { for (const record of within) assert.ok(record.$distance <= 0.05, `distance ${record.$distance} exceeds threshold`); }); + it('synchronous iteration of plane-backed results fails loudly instead of spinning', () => { + const results = PlaneTest.search({ + sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, + select: ['id'], + limit: 5, + }); + assert.throws(() => [...results], /async/i, 'sync iteration must throw, not loop on promise-shaped results'); + }); + it('reopens the same plane file across a restart', async () => { const planePath = customIndex().planeFilePath(); const inodeBefore = fs.statSync(planePath).ino; From 98f957436f2f5bc9f04e450859cbad2a533e47ee Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 10:50:46 -0600 Subject: [PATCH 15/69] Review round 2: predicate-throw containment, exact filtered visit budget, CI plane job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An error thrown by an app-supplied filter now denies the batch and rejects the search with that error — matching the JS path's query-failure contract — instead of escaping into the fatal-strategy ThreadsafeFunction callback; the plane stays enabled (PLANE_PREDICATE_ERROR marker) since it is healthy. searchWithPredicate gains an absolute visitBudget parameter (a multiplier of ef cannot express a budget below a limit-widened ef), and the JS side passes filterState.maxVisits verbatim so both paths stop at the same visit count. A new hnsw-plane CI job builds the optional artifact and runs the crate tests plus the dual-write/parity suite, which otherwise self-skips in CI. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- .github/workflows/unit-test.yml | 42 ++++++++++++++++++- native/hnsw-plane/src/napi.rs | 16 ++++--- native/hnsw-plane/src/search.rs | 9 ++-- .../HierarchicalNavigableSmallWorld.ts | 38 ++++++++++++----- resources/indexes/hnswPlaneBinding.ts | 3 +- unitTests/resources/vectorIndexPlane.test.js | 15 +++++++ 6 files changed, 102 insertions(+), 21 deletions(-) diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml index 08eeb5b7b2..93060109da 100644 --- a/.github/workflows/unit-test.yml +++ b/.github/workflows/unit-test.yml @@ -73,7 +73,6 @@ jobs: - name: Run tests run: npm run test:unit:all - # Windows had no unit-level coverage at all before this job: every job above pins # ubuntu-latest, and integration-tests.yml was the only workflow touching Windows. # Platform-gated code and the tests written to prove it therefore never executed. @@ -119,3 +118,44 @@ jobs: - name: Run unit tests timeout-minutes: 30 run: npm run test:unit:windows + # The vectorIndexPlane suite self-skips when the optional native artifact is absent, so + # without this job nothing in CI would ever exercise the dual-write mirroring or the + # native search cutover — green CI would only prove the JS path unchanged. + hnsw-plane: + name: HNSW native plane (Node.js v24) + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout code + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + + - name: Setup Node.js 24 + uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 + with: + node-version: 24 + package-manager-cache: false + + - name: Install dependencies + run: npm install --ignore-scripts + + - name: Build + run: npm run build || true # we currently have type errors so just ignore that + + - name: Build native hnsw-plane module + run: npm run build:hnsw-plane + + - name: Crate tests + run: cargo test --release --manifest-path native/hnsw-plane/Cargo.toml + + - name: Setup Harper + env: + DEFAULTS_MODE: 'dev' + HDB_ADMIN_USERNAME: 'admin' + HDB_ADMIN_PASSWORD: 'password' + ROOTPATH: '/tmp/hdb' + NODE_HOSTNAME: 'localhost' + LOGGING_LEVEL: 'info' + run: node --enable-source-maps ./dist/bin/harper.js install + + - name: Plane dual-write + parity tests + run: npx mocha unitTests/resources/vectorIndexPlane.test.js diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index a6b04695b6..f684f30504 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -77,7 +77,7 @@ pub struct PredicateSearchTask { k: usize, ef: usize, tsfn: Option, ErrorStrategy::Fatal>>, - filter_expansion: usize, + visit_budget: u64, } #[napi] @@ -108,7 +108,7 @@ impl Task for PredicateSearchTask { let mut scratch = self.pool.take(); let query = Query::new(std::mem::take(&mut self.query)); let (hits, _stats) = - search_predicated(&self.graph, &query, self.k, self.ef, &mut pipe, self.filter_expansion, &mut scratch); + search_predicated(&self.graph, &query, self.k, self.ef, &mut pipe, self.visit_budget, &mut scratch); self.pool.put(scratch); Ok(hits) } @@ -260,7 +260,9 @@ impl Plane { /// (one 0/1 byte per id, evaluated synchronously). Batches of candidate ids stream to /// the predicate over a ThreadsafeFunction while traversal keeps expanding — the search /// thread never blocks on the JS event loop until the beam itself is done, so a busy - /// loop costs speculative overshoot (bounded by ef * filterExpansion), not latency. + /// loop costs speculative overshoot (bounded by the visit budget), not latency. + /// `visitBudget` caps layer-0 visits absolutely (a host budget may sit below ef, which a + /// multiplier cannot express); when absent the budget is ef * filterExpansion. /// Must not be awaited synchronously from code the predicate itself blocks. #[napi(ts_return_type = "Promise>")] pub fn search_with_predicate( @@ -270,20 +272,24 @@ impl Plane { ef: u32, #[napi(ts_arg_type = "(ids: Array) => Uint8Array")] predicate: JsFunction, filter_expansion: Option, + visit_budget: Option, ) -> Result> { let tsfn: ThreadsafeFunction, ErrorStrategy::Fatal> = predicate .create_threadsafe_function(0, |ctx: napi::threadsafe_function::ThreadSafeCallContext>| { let ids: Vec = ctx.value.iter().map(|&v| v as f64).collect(); Ok(vec![ids]) })?; + let ef = ef as usize; Ok(AsyncTask::new(PredicateSearchTask { graph: self.graph.clone(), pool: self.pool.clone(), query: vector.to_vec(), k: k as usize, - ef: ef as usize, + ef, tsfn: Some(tsfn), - filter_expansion: filter_expansion.unwrap_or(24) as usize, + visit_budget: visit_budget + .map(|b| b.max(1.0) as u64) + .unwrap_or((ef * filter_expansion.unwrap_or(24) as usize) as u64), })) } diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 284e787839..b986704665 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -290,14 +290,16 @@ const PREDICATE_BATCH: usize = 64; const DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); /// Full search with a pipelined predicate filter (upper-layer descent is unfiltered, as in -/// the JS implementation — predicates gate results, not routing). +/// the JS implementation — predicates gate results, not routing). `visit_budget` is the +/// absolute layer-0 visit cap: hosts pass their own resolved budget directly, since a +/// multiplier-of-ef encoding cannot express a budget below ef. pub fn search_predicated( graph: &Graph, query: &Query, k: usize, ef: usize, pipe: &mut PredicatePipe, - filter_expansion: usize, + visit_budget: u64, scratch: &mut SearchScratch, ) -> (Vec<(u32, f32)>, SearchStats) { let mut stats = SearchStats { visits: 0 }; @@ -314,7 +316,6 @@ pub fn search_predicated( None => return (Vec::new(), stats), }; let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); - let visit_budget = (ef * filter_expansion) as u64; use std::collections::HashMap; let mut verdicts: HashMap = HashMap::new(); @@ -457,7 +458,7 @@ mod predicate_tests { }; let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); let (hits, _) = - search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 24, &mut scratch); + search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); assert!(!hits.is_empty()); for (id, _) in &hits { assert_eq!(id % 2, 0, "odd id {id} leaked through the predicate"); diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index f0be37aa0c..41d0da8338 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -144,6 +144,9 @@ const PLANE_INCOMPLETE_REBUILD_MS = 3_600_000; // Watermark stamped when the initial full mirror completes; 0 = still building (or crashed // mid-build). Phase-2 replay wiring will carry real transaction ids, which are also nonzero. const PLANE_MIRRORED = 1; +// Marks an error thrown by an app-supplied filter during a plane search: the caller re-raises +// it as an ordinary query failure instead of disabling the (healthy) plane. +const PLANE_PREDICATE_ERROR = Symbol('planePredicateError'); class MinHeap { private data: Candidate[] = []; @@ -612,26 +615,39 @@ export class HierarchicalNavigableSmallWorld { ): Promise { const query = Float32Array.from(target); let resultPromise: Promise<{ id: number; distance: number }[]>; + let predicateError: unknown; if (filter && filterState) { - // The plane bounds filtered visits at ef * filterExpansion; recover the multiplier from - // the already-resolved JS budget so both paths stop at the same visit count. The u32 - // multiplier makes this exact only to the nearest multiple of ef: when a limit-widened - // ef exceeds maxVisits the floor of 1 over-explores (latency, never correctness), and - // ordinary rounding keeps the budgets within half an ef of each other. - const planeFilterExpansion = Math.max(1, Math.round(filterState.maxVisits / ef)); const predicate = (ids: number[]): Uint8Array => { const verdicts = new Uint8Array(ids.length); - for (let i = 0; i < ids.length; i++) { - const primaryKey = this.safeGetSync(ids[i], options)?.primaryKey; - if (primaryKey !== undefined && this.admit(filter, filterState, primaryKey)) verdicts[i] = 1; + try { + for (let i = 0; i < ids.length; i++) { + const primaryKey = this.safeGetSync(ids[i], options)?.primaryKey; + if (primaryKey !== undefined && this.admit(filter, filterState, primaryKey)) verdicts[i] = 1; + } + } catch (error) { + // an app-supplied filter threw: deny the batch and surface the error once the + // traversal resolves — the same query failure the JS path raises — instead of + // letting it escape into the fatal-strategy ThreadsafeFunction callback + predicateError ??= error; } return verdicts; }; - resultPromise = plane.searchWithPredicate(query, ef, ef, predicate, planeFilterExpansion); + // pass the already-resolved JS visit budget verbatim so both paths stop at the same count + resultPromise = plane.searchWithPredicate(query, ef, ef, predicate, undefined, filterState.maxVisits); } else { resultPromise = plane.search(query, ef, ef); } return resultPromise.then((hits) => { + if (predicateError !== undefined) { + // the plane itself is healthy; mark the failure as the application's so the caller + // re-raises it rather than disabling the plane and retrying + try { + (predicateError as any)[PLANE_PREDICATE_ERROR] = true; + } catch { + // a frozen/primitive throw still propagates, it just also disables the plane + } + throw predicateError; + } const entries: any[] = []; for (const hit of hits) { const primaryKey = this.safeGetSync(hit.id, options)?.primaryKey; @@ -1559,6 +1575,8 @@ export class HierarchicalNavigableSmallWorld { // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all // post-load behavior are unchanged. searchByIndex handles the promise. return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { + // an app filter's own throw is the query's failure, not the plane's + if (error?.[PLANE_PREDICATE_ERROR]) throw error; // a failed native search disables the plane and re-runs this query on the JS path this.disablePlane(error); return this.search( diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index d40d83efa4..7de95eb49c 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -39,7 +39,8 @@ export interface HnswPlane { k: number, ef: number, predicate: (ids: number[]) => Uint8Array, - filterExpansion?: number | null + filterExpansion?: number | null, + visitBudget?: number | null ): Promise; searchSync(vector: Float32Array, k: number, ef: number): PlaneSearchHit[]; idHighWater(): number; diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index c72a237787..5c8aa5c46d 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -200,6 +200,21 @@ describe('HNSW native plane dual-write', function () { for (const record of within) assert.ok(record.$distance <= 0.05, `distance ${record.$distance} exceeds threshold`); }); + it('a throwing app filter surfaces as the query error without disabling the plane', async () => { + const condition = { target: vectors.get(3), comparator: 'sort', distance: 'cosine', ef: EF }; + await assert.rejects( + Promise.resolve( + customIndex().search(condition, { transaction: undefined }, () => { + throw new Error('filter boom'); + }) + ), + /filter boom/ + ); + const after = customIndex().search(condition, { transaction: undefined }); + assert.equal(typeof after?.then, 'function', 'the plane must stay enabled after an app-filter throw'); + assert.ok((await after).length > 0); + }); + it('synchronous iteration of plane-backed results fails loudly instead of spinning', () => { const results = PlaneTest.search({ sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, From 26a0cfaf59cfa4689b09aa57bb664951623c1d9c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 11:13:37 -0600 Subject: [PATCH 16/69] Review round 2b: query-dims safety guard, creation-dims from the graph, incomplete-build aging by birthtime The distance kernel streams query-length bytes from each slot, so every search entry point now rejects a query whose dimensionality differs from the plane's (an oversized query was an out-of-bounds read), with a dims getter so the JS side routes mismatched queries down the JS path without touching the healthy plane. Plane creation takes dims from the graph's own first node when one exists, so a malformed search target cannot pin a populated index's file to the wrong dimensionality. Incomplete-mirror rebuild ages by file birthtime (dual-writes into an abandoned build keep refreshing mtime, deferring the rebuild forever), a failed reset unlink leaves the plane disabled rather than reopening a stale file, the async iterator returns promise-shaped results on every next(), the predicate adapter short-circuits after an app-filter error, and connection-id mirroring is single-allocation in the common case. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- native/hnsw-plane/src/napi.rs | 32 +++++++++++--- .../HierarchicalNavigableSmallWorld.ts | 43 ++++++++++++++++--- resources/indexes/hnswPlaneBinding.ts | 1 + resources/search.ts | 4 +- 4 files changed, 67 insertions(+), 13 deletions(-) diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index f684f30504..5d72d00127 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -233,6 +233,25 @@ impl Plane { vec![id as f64, level as f64] } + /// Query dimensionality must match the plane: the distance kernel streams + /// `query.len()` bytes from each slot's vector, so an oversized query would read past + /// it into adjacent slot bytes (or off the mapping entirely). + fn check_query_dims(&self, len: usize) -> Result<()> { + if len != self.graph.file.dims { + return Err(Error::from_reason(format!( + "query vector has {} dimensions; plane dims = {}", + len, self.graph.file.dims + ))); + } + Ok(()) + } + + /// Vector dimensionality of this plane (fixed at create). + #[napi(getter)] + pub fn dims(&self) -> u32 { + self.graph.file.dims as u32 + } + /// Async k-NN search on the libuv thread pool. `filter` is an optional allow-bitset /// over node ids (bit i of byte i>>3); filtered searches are visit-bounded by /// ef * filterExpansion (default 24). @@ -244,8 +263,9 @@ impl Plane { ef: u32, filter: Option, filter_expansion: Option, - ) -> AsyncTask { - AsyncTask::new(SearchTask { + ) -> Result> { + self.check_query_dims(vector.len())?; + Ok(AsyncTask::new(SearchTask { graph: self.graph.clone(), pool: self.pool.clone(), query: vector.to_vec(), @@ -253,7 +273,7 @@ impl Plane { ef: ef as usize, filter: filter.map(|f| f.to_vec()), filter_expansion: filter_expansion.unwrap_or(24) as usize, - }) + })) } /// Async k-NN search with a JS predicate: `predicate(ids: number[]) => Uint8Array` @@ -274,6 +294,7 @@ impl Plane { filter_expansion: Option, visit_budget: Option, ) -> Result> { + self.check_query_dims(vector.len())?; let tsfn: ThreadsafeFunction, ErrorStrategy::Fatal> = predicate .create_threadsafe_function(0, |ctx: napi::threadsafe_function::ThreadSafeCallContext>| { let ids: Vec = ctx.value.iter().map(|&v| v as f64).collect(); @@ -295,12 +316,13 @@ impl Plane { /// Synchronous search (benchmarks/tests; blocks the calling thread). #[napi] - pub fn search_sync(&self, vector: Float32Array, k: u32, ef: u32) -> Vec { + pub fn search_sync(&self, vector: Float32Array, k: u32, ef: u32) -> Result> { + self.check_query_dims(vector.len())?; let mut scratch = self.pool.take(); let query = Query::new(vector.to_vec()); let (hits, _) = search_filtered(&self.graph, &query, k as usize, ef as usize, None, 24, &mut scratch); self.pool.put(scratch); - hits.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect() + Ok(hits.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) } /// Lifetime id high-water (allocated ids, including freed ones awaiting reuse). diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 41d0da8338..be0f14d894 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -51,11 +51,23 @@ function dequantizeInt8(q: Int8Array, scale: number): number[] { */ function planeConnectionIds(connections: Connection[] | undefined, cap: number): Uint32Array { if (!connections?.length) return new Uint32Array(0); - let usable = connections.filter((connection) => typeof connection.id === 'number' && connection.id >= 0); - if (usable.length > cap) { - usable = usable.sort((a, b) => a.distance - b.distance).slice(0, cap); + let usableCount = 0; + for (const connection of connections) { + if (typeof connection.id === 'number' && connection.id >= 0) usableCount++; } - return Uint32Array.from(usable, (connection) => connection.id); + if (usableCount > cap) { + // rare (transient overshoot / the tighter upper cap): worth the intermediate copies + const nearest = connections + .filter((connection) => typeof connection.id === 'number' && connection.id >= 0) + .sort((a, b) => a.distance - b.distance); + return Uint32Array.from({ length: cap }, (_, i) => nearest[i].id); + } + const ids = new Uint32Array(usableCount); + let at = 0; + for (const connection of connections) { + if (typeof connection.id === 'number' && connection.id >= 0) ids[at++] = connection.id; + } + return ids; } // Auto-scaled search ef, used only when an index does not explicitly configure efConstructionSearch @@ -386,6 +398,12 @@ export class HierarchicalNavigableSmallWorld { return null; } closeSync(fd); + // a populated graph's own dimensionality sizes the file — a caller-supplied dims + // (possibly a malformed search target) must not; the file format pins dims forever + for (const { value } of this.indexStore.getRange({ start: 0, end: Infinity, limit: 1 })) { + const storedVector = value?.level !== undefined ? value.vector : undefined; + if (storedVector) dims = Array.isArray(storedVector) ? storedVector.length : storedVector.byteLength; + } try { return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); } catch (createError) { @@ -420,7 +438,10 @@ export class HierarchicalNavigableSmallWorld { const filePath = this.planeFilePath(); if (filePath) { try { - if (Date.now() - statSync(filePath).mtimeMs > PLANE_INCOMPLETE_REBUILD_MS) { + // age by creation time: ongoing dual-writes into an abandoned build keep + // refreshing mtime, which would defer this rebuild forever + const stat = statSync(filePath); + if (Date.now() - (stat.birthtimeMs || stat.mtimeMs) > PLANE_INCOMPLETE_REBUILD_MS) { logger.warn?.('rebuilding an HNSW plane file whose initial mirror never completed'); this.resetDerivedStorage(); } @@ -594,7 +615,12 @@ export class HierarchicalNavigableSmallWorld { try { unlinkSync(filePath); } catch (error: any) { - if (error?.code !== 'ENOENT') logger.warn?.('could not delete the HNSW plane file', error); + if (error?.code !== 'ENOENT') { + // a stale file that cannot be deleted (e.g. Windows EBUSY while mapped) must not + // be reopened as if current — keep the plane disabled for this process instead + this.plane = null; + logger.warn?.('could not delete the HNSW plane file', error); + } } } @@ -619,6 +645,7 @@ export class HierarchicalNavigableSmallWorld { if (filter && filterState) { const predicate = (ids: number[]): Uint8Array => { const verdicts = new Uint8Array(ids.length); + if (predicateError !== undefined) return verdicts; // already failed — deny remaining batches cheaply try { for (let i = 0; i < ids.length; i++) { const primaryKey = this.safeGetSync(ids[i], options)?.primaryKey; @@ -1570,7 +1597,9 @@ export class HierarchicalNavigableSmallWorld { : undefined; if (this.planeEligible) { const plane = this.getPlane(target.length); - if (plane && this.planeSearchReady(plane)) { + // a query whose dimensionality differs from the graph's takes the JS path (which + // tolerates the mismatch) rather than erroring or disabling the healthy plane + if (plane && plane.dims === target.length && this.planeSearchReady(plane)) { // Native cutover: same resolved ef, same predicate semantics; resolves to the same // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all // post-load behavior are unchanged. searchByIndex handles the promise. diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index 7de95eb49c..c20c4a7aa2 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -15,6 +15,7 @@ export interface PlaneSearchHit { * path is bypassed by design) plus the search entry points. */ export interface HnswPlane { + readonly dims: number; writeNodeRaw( id: number, level: number, diff --git a/resources/search.ts b/resources/search.ts index 16c96850d9..a83d7923f7 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -552,8 +552,10 @@ export function searchByIndex( } let inner: Iterator | null = null; return { + // always promise-shaped, as the async-iterator protocol requires of every + // next() result, not just the first next() { - if (inner) return inner.next(); + if (inner) return Promise.resolve(inner.next()); return pending.then((entries) => { inner = entries[Symbol.iterator](); return inner.next(); From 35f0dc9c0158ebbeb42844f75ccc1ec5170b0879 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 11:21:58 -0600 Subject: [PATCH 17/69] Review round 3: contain synchronous NAPI throws; promise-shaped iterator return() A NAPI error thrown before the search promise exists (e.g. ThreadsafeFunction creation failure) now disables the plane and degrades to the JS path instead of escaping the promise-chain catch, and the async iterator's return() is promise-shaped per the async-iterator protocol. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- .../HierarchicalNavigableSmallWorld.ts | 27 +++++++++++-------- resources/search.ts | 2 +- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index be0f14d894..15b8ae52a8 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -1603,18 +1603,23 @@ export class HierarchicalNavigableSmallWorld { // Native cutover: same resolved ef, same predicate semantics; resolves to the same // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all // post-load behavior are unchanged. searchByIndex handles the promise. - return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { - // an app filter's own throw is the query's failure, not the plane's - if (error?.[PLANE_PREDICATE_ERROR]) throw error; - // a failed native search disables the plane and re-runs this query on the JS path + try { + return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { + // an app filter's own throw is the query's failure, not the plane's + if (error?.[PLANE_PREDICATE_ERROR]) throw error; + // a failed native search disables the plane and re-runs this query on the JS path + this.disablePlane(error); + return this.search( + { target, value, descending, distance, comparator, ef, filterExpansion }, + context, + filter, + minResults + ); + }); + } catch (error) { + // a synchronous NAPI throw (before any promise exists) degrades to the JS path below this.disablePlane(error); - return this.search( - { target, value, descending, distance, comparator, ef, filterExpansion }, - context, - filter, - minResults - ); - }); + } } } let entryPoint = this.getEntryPoint(options); diff --git a/resources/search.ts b/resources/search.ts index a83d7923f7..6a88dd3045 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -563,7 +563,7 @@ export function searchByIndex( }, return(value?: any) { (inner as any)?.return?.(value); - return { done: true, value }; + return Promise.resolve({ done: true, value }); }, }; }; From 730c871fb872f39633b2556a353c87d48d0b73d3 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 11:41:17 -0600 Subject: [PATCH 18/69] Review round 3b: layer-0 cap derived from M, durable initial mirror, committed Cargo.lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plane's layer-0 cap is derived from M/optimizeRouting at creation (with a layer0Cap getter so mirroring truncates to the file's actual geometry) instead of a hard-coded 128 that silently halved a non-default-M graph's adjacency. The completed initial mirror is msync'd once so the watermark and mirrored slots are durable together. Cargo.lock is committed — CI builds and loads the cdylib, so the artifact under test must be reproducible. New negative-path tests: a watermark-0 (incomplete) mirror is never searched and recovers when stamped, and an unopenable plane file degrades writes and searches to the JS path without erroring. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- .github/workflows/unit-test.yml | 6 +- .gitignore | 1 - native/hnsw-plane/Cargo.lock | 229 ++++++++++++++++++ native/hnsw-plane/build.mjs | 6 +- native/hnsw-plane/src/napi.rs | 6 +- .../HierarchicalNavigableSmallWorld.ts | 22 +- resources/indexes/hnswPlaneBinding.ts | 1 + resources/search.ts | 2 - unitTests/resources/vectorIndexPlane.test.js | 37 +++ 9 files changed, 292 insertions(+), 18 deletions(-) create mode 100644 native/hnsw-plane/Cargo.lock diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml index 93060109da..638d6bf2b3 100644 --- a/.github/workflows/unit-test.yml +++ b/.github/workflows/unit-test.yml @@ -118,9 +118,9 @@ jobs: - name: Run unit tests timeout-minutes: 30 run: npm run test:unit:windows - # The vectorIndexPlane suite self-skips when the optional native artifact is absent, so - # without this job nothing in CI would ever exercise the dual-write mirroring or the - # native search cutover — green CI would only prove the JS path unchanged. + + # The vectorIndexPlane suite self-skips when the optional native artifact is absent; this + # job builds it so the dual-write mirroring and native search cutover run in CI. hnsw-plane: name: HNSW native plane (Node.js v24) runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index b99cc0a082..84a0e3a73d 100644 --- a/.gitignore +++ b/.gitignore @@ -68,4 +68,3 @@ integrationTests/**/node_modules/ # hnsw-plane native build outputs (optional module; build locally with npm run build:hnsw-plane) native/hnsw-plane/target/ native/hnsw-plane/hnsw-plane.node -native/hnsw-plane/Cargo.lock diff --git a/native/hnsw-plane/Cargo.lock b/native/hnsw-plane/Cargo.lock new file mode 100644 index 0000000000..e7a44ca599 --- /dev/null +++ b/native/hnsw-plane/Cargo.lock @@ -0,0 +1,229 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aho-corasick" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" +dependencies = [ + "memchr", +] + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "convert_case" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec182b0ca2f35d8fc196cf3404988fd8b8c739a4d270ff118a398feb0cbec1ca" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "ctor" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" +dependencies = [ + "quote", + "syn", +] + +[[package]] +name = "hnsw-plane" +version = "0.0.1" +dependencies = [ + "memmap2", + "napi", + "napi-build", + "napi-derive", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "memmap2" +version = "0.9.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" +dependencies = [ + "libc", +] + +[[package]] +name = "napi" +version = "2.16.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55740c4ae1d8696773c78fdafd5d0e5fe9bc9f1b071c7ba493ba5c413a9184f3" +dependencies = [ + "bitflags", + "ctor", + "napi-derive", + "napi-sys", + "once_cell", +] + +[[package]] +name = "napi-build" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60fdf9b392c50e7c4170fa633bd909490ed7835cea4c046776d1a4dd8d2ae0ab" + +[[package]] +name = "napi-derive" +version = "2.16.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cbe2585d8ac223f7d34f13701434b9d5f4eb9c332cccce8dee57ea18ab8ab0c" +dependencies = [ + "cfg-if", + "convert_case", + "napi-derive-backend", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "napi-derive-backend" +version = "1.0.75" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1639aaa9eeb76e91c6ae66da8ce3e89e921cd3885e99ec85f4abacae72fc91bf" +dependencies = [ + "convert_case", + "once_cell", + "proc-macro2", + "quote", + "regex", + "semver", + "syn", +] + +[[package]] +name = "napi-sys" +version = "2.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "427802e8ec3a734331fec1035594a210ce1ff4dc5bc1950530920ab717964ea3" +dependencies = [ + "libloading", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "regex" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-segmentation" +version = "1.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" diff --git a/native/hnsw-plane/build.mjs b/native/hnsw-plane/build.mjs index 9ca59543c6..af82c4bf0f 100644 --- a/native/hnsw-plane/build.mjs +++ b/native/hnsw-plane/build.mjs @@ -1,6 +1,6 @@ -// Builds the optional hnsw-plane NAPI module in place: harper installs never require a cargo -// toolchain (the nativePlane index option falls back to the JS path when the artifact is -// absent), so this is a local/dev step: `npm run build:hnsw-plane`. +// Builds the optional hnsw-plane NAPI module in place (`npm run build:hnsw-plane`); harper +// installs never require a cargo toolchain — without the artifact, nativePlane falls back to +// the JS path. import { execSync } from 'node:child_process'; import { copyFileSync } from 'node:fs'; import { dirname, join } from 'node:path'; diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 5d72d00127..32578a0580 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -246,12 +246,16 @@ impl Plane { Ok(()) } - /// Vector dimensionality of this plane (fixed at create). #[napi(getter)] pub fn dims(&self) -> u32 { self.graph.file.dims as u32 } + #[napi(getter)] + pub fn layer0_cap(&self) -> u32 { + self.graph.file.layer0_cap as u32 + } + /// Async k-NN search on the libuv thread pool. `filter` is an optional allow-bitset /// over node ids (bit i of byte i>>3); filtered searches are visit-bounded by /// ef * filterExpansion (default 24). diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 15b8ae52a8..65cc37a9a8 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -134,13 +134,13 @@ function autoScaleEfConstruction(nodeCount: number): number { // this only has to be short enough that a table growing from empty picks up a larger ef promptly. const NODE_COUNT_TTL = 10_000; -// Native traversal-plane geometry (see hnsw-native-plane.md). The layer-0 cap must cover the JS -// graph's effective cap (M<<1 <<2 under optimizeRouting = 128); writeNodeRaw truncates the -// transient 160 overshoot. maxNodes is a fixed sparse reservation — pages materialize on write — +// Native traversal-plane geometry (see hnsw-native-plane.md). The layer-0 cap is derived from +// M/optimizeRouting at creation to cover the JS graph's effective cap; grace overshoot above it +// truncates by distance. maxNodes is a fixed sparse reservation — pages materialize on write — // and ids at or past it are rejected by the crate, which disables the plane for this process. -const PLANE_LAYER0_CAP = 128; -// Ids kept per upper level — must match the crate's format UPPER_CAP, which truncates whatever -// is passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. +const PLANE_LAYER0_CAP_MAX = 1024; +// Ids kept per upper level — the crate's format-level UPPER_CAP, which truncates whatever is +// passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. const PLANE_UPPER_CAP = 32; const PLANE_MAX_NODES = 1 << 24; // An existing plane file that cannot be opened is normally another worker mid-create (retry); @@ -466,7 +466,10 @@ export class HierarchicalNavigableSmallWorld { filePath: string, dims: number ): HnswPlane { - const plane = Plane.create(filePath, dims, PLANE_LAYER0_CAP, PLANE_MAX_NODES); + // derived from M/optimizeRouting like the JS layer-0 cap, so a non-default M gets a + // matching slot geometry rather than silent truncation to a fixed width + const layer0Cap = Math.min(PLANE_LAYER0_CAP_MAX, this.optimizeRouting ? this.M << 3 : this.M << 1); + const plane = Plane.create(filePath, dims, layer0Cap, PLANE_MAX_NODES); let mirrored = 0; for (const { key, value } of this.indexStore.getRange({ start: 0, end: Infinity })) { if (typeof key !== 'number' || !value || value.level === undefined) continue; @@ -480,6 +483,9 @@ export class HierarchicalNavigableSmallWorld { // stamped last: a crash or thrown mirror error leaves the watermark 0, and // planeSearchReady refuses to serve searches from a mirror that never completed plane.setWatermark(PLANE_MIRRORED); + // one msync so the completed mirror (and its watermark) is durable; steady-state + // flush cadence is a recorded phase-2 open item + plane.flush(); this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); return plane; @@ -520,7 +526,7 @@ export class HierarchicalNavigableSmallWorld { } } const level = node.level ?? 0; - const layer0 = planeConnectionIds(node[0], PLANE_LAYER0_CAP); + const layer0 = planeConnectionIds(node[0], plane.layer0Cap); let upper: Uint32Array[] | null = null; if (level >= 1) { upper = []; diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index c20c4a7aa2..32b60bfdeb 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -16,6 +16,7 @@ export interface PlaneSearchHit { */ export interface HnswPlane { readonly dims: number; + readonly layer0Cap: number; writeNodeRaw( id: number, level: number, diff --git a/resources/search.ts b/resources/search.ts index 6a88dd3045..3b69c4143c 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -552,8 +552,6 @@ export function searchByIndex( } let inner: Iterator | null = null; return { - // always promise-shaped, as the async-iterator protocol requires of every - // next() result, not just the first next() { if (inner) return Promise.resolve(inner.next()); return pending.then((entries) => { diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index 5c8aa5c46d..ed4dbfa16f 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -286,6 +286,43 @@ describe('HNSW native plane dual-write', function () { await Later.dropTable(); }); + it('a plane whose initial mirror never completed is not searched', async () => { + const condition = { target: vectors.get(11), comparator: 'sort', distance: 'cosine', ef: EF }; + const index = customIndex(); + const plane = index.getPlane(); + assert.ok(plane, 'the plane should be attached'); + plane.setWatermark(0); // simulate a crashed/incomplete initial mirror + index.planeReady = false; + const jsResults = index.search(condition, { transaction: undefined }); + assert.equal(typeof jsResults?.then, 'undefined', 'an incomplete mirror must fall back to the JS path'); + assert.ok(jsResults.length > 0); + plane.setWatermark(1); + const planeResults = index.search(condition, { transaction: undefined }); + assert.equal(typeof planeResults?.then, 'function', 'a completed mirror serves searches again'); + await planeResults; + }); + + it('an unopenable plane file degrades to the JS path without erroring', async () => { + const Foreign = table({ + table: 'PlaneForeign', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + const index = Foreign.indices.vector.customIndex; + fs.writeFileSync(index.planeFilePath(), 'not a plane'); // e.g. a crashed create's leftovers + for (let i = 0; i < 20; i++) await Foreign.put(i, { vector: makeVector(i + 20000) }); + const results = index.search( + { target: makeVector(20003), comparator: 'sort', distance: 'cosine', ef: 50 }, + { transaction: undefined } + ); + assert.equal(typeof results?.then, 'undefined', 'writes and searches must run on the JS path meanwhile'); + assert.ok(results.length > 0); + await Foreign.dropTable(); + }); + it('index drop removes the plane file', async () => { const planePath = customIndex().planeFilePath(); assert.ok(fs.existsSync(planePath)); From f05a95f1f0e43da0085a554bcfec7f152bf778e3 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 11:52:36 -0600 Subject: [PATCH 19/69] Review round 4: flush graph before watermark; refuse over-ceiling M instead of truncating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A durable watermark must imply durable slots, so the initial mirror is msync'd before the completion watermark is stamped and flushed — a crash between writebacks can no longer adopt a torn mirror as complete. An index whose derived layer-0 cap exceeds the plane ceiling is now ineligible (JS path, logged once) rather than silently truncated to the ceiling. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- .../HierarchicalNavigableSmallWorld.ts | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 65cc37a9a8..2da0782404 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -135,9 +135,10 @@ function autoScaleEfConstruction(nodeCount: number): number { const NODE_COUNT_TTL = 10_000; // Native traversal-plane geometry (see hnsw-native-plane.md). The layer-0 cap is derived from -// M/optimizeRouting at creation to cover the JS graph's effective cap; grace overshoot above it -// truncates by distance. maxNodes is a fixed sparse reservation — pages materialize on write — -// and ids at or past it are rejected by the crate, which disables the plane for this process. +// M/optimizeRouting at creation to cover the JS graph's effective cap (grace overshoot above it +// truncates by distance); a configuration deriving past this ceiling is refused as ineligible +// rather than silently truncated. maxNodes is a fixed sparse reservation — pages materialize on +// write — and ids at or past it are rejected by the crate, which disables the plane. const PLANE_LAYER0_CAP_MAX = 1024; // Ids kept per upper level — the crate's format-level UPPER_CAP, which truncates whatever is // passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. @@ -332,10 +333,14 @@ export class HierarchicalNavigableSmallWorld { } if (options?.nativePlane) { // The plane stores int8 bins and computes asymmetric cosine only, so the flag is a - // no-op for float (quantization: "none") and non-cosine indexes. - this.planeEligible = this.int8 && this.distance === cosineDistance; + // no-op for float (quantization: "none") and non-cosine indexes; a graph whose derived + // layer-0 cap exceeds the plane maximum is refused rather than silently truncated. + this.planeEligible = + this.int8 && this.distance === cosineDistance && this.planeLayer0Cap() <= PLANE_LAYER0_CAP_MAX; if (!this.planeEligible) { - logger.info?.('nativePlane is only supported for int8-quantized cosine HNSW indexes; using the JS search path'); + logger.info?.( + 'nativePlane requires an int8-quantized cosine HNSW index whose M fits the plane geometry; using the JS search path' + ); } } } @@ -461,15 +466,17 @@ export class HierarchicalNavigableSmallWorld { * scan's older snapshot of that node — it re-syncs on the node's next touch, and the exact * rescore + record load already filter stale candidates (relaxed adherence, design §5). */ + /** The JS graph's effective layer-0 cap for this configuration; sizes the plane's slots. */ + private planeLayer0Cap(): number { + return this.optimizeRouting ? this.M << 3 : this.M << 1; + } + private createAndMirrorPlane( Plane: NonNullable>, filePath: string, dims: number ): HnswPlane { - // derived from M/optimizeRouting like the JS layer-0 cap, so a non-default M gets a - // matching slot geometry rather than silent truncation to a fixed width - const layer0Cap = Math.min(PLANE_LAYER0_CAP_MAX, this.optimizeRouting ? this.M << 3 : this.M << 1); - const plane = Plane.create(filePath, dims, layer0Cap, PLANE_MAX_NODES); + const plane = Plane.create(filePath, dims, this.planeLayer0Cap(), PLANE_MAX_NODES); let mirrored = 0; for (const { key, value } of this.indexStore.getRange({ start: 0, end: Infinity })) { if (typeof key !== 'number' || !value || value.level === undefined) continue; @@ -480,11 +487,11 @@ export class HierarchicalNavigableSmallWorld { if (typeof entryPointId === 'number') { plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); } - // stamped last: a crash or thrown mirror error leaves the watermark 0, and - // planeSearchReady refuses to serve searches from a mirror that never completed + // make the mirrored graph durable BEFORE stamping the watermark, then flush again: a + // durable watermark must imply durable slots, or a crash between writebacks could adopt + // a torn mirror as complete. Steady-state flush cadence is a recorded phase-2 open item. + plane.flush(); plane.setWatermark(PLANE_MIRRORED); - // one msync so the completed mirror (and its watermark) is durable; steady-state - // flush cadence is a recorded phase-2 open item plane.flush(); this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); From 0d112b5a81ab2606b0032bf4e48f4139b45ca56d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 12:00:50 -0600 Subject: [PATCH 20/69] Review round 5: delete the plane file when the flag is disabled MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With nativePlane off (or the index ineligible) nothing mirrors, so an existing plane file only goes stale — and a later re-enable would adopt it, silently missing every mutation made in between. openIndex now has the authoritative index instance delete the file (auxiliary instances over the same store must not), making re-enable rebuild from the CF: the documented rollback, automated. Tested: toggling the flag off deletes the file without reindexing, and a re-enable rebuilds a plane containing records written while it was off. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015DhgV2wGobQbkEsj59SG7P --- resources/databases.ts | 3 ++ .../HierarchicalNavigableSmallWorld.ts | 20 +++++++++++ unitTests/resources/vectorIndexPlane.test.js | 34 +++++++++++++++++++ 3 files changed, 57 insertions(+) diff --git a/resources/databases.ts b/resources/databases.ts index f82af610ef..b7b565989e 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -2030,6 +2030,9 @@ function openIndex(dbiKey: string, rootStore: RootDatabaseKind, attribute: any) const CustomIndex = CUSTOM_INDEXES[attribute.indexed.type]; if (CustomIndex) { indexStore.customIndex = new CustomIndex(indexStore, attribute.indexed); + // derived state whose maintaining option is now off must not linger to be adopted + // stale on a later re-enable + indexStore.customIndex.cleanupDisabledPlane?.(); } else { logger.error(`The indexing type '${attribute.indexed.type}' is unknown`); } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 2da0782404..d6ae6399da 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -345,6 +345,26 @@ export class HierarchicalNavigableSmallWorld { } } + /** + * Called by openIndex on the authoritative index instance. With the flag off (or the index + * ineligible) nothing mirrors, so an existing plane file only goes stale — and a later + * re-enable would adopt it; deleting it here makes re-enabling rebuild from the CF (the + * documented rollback: flag off + file deleted). Not done in the constructor: auxiliary + * instances over the same store (e.g. a flag-off reference in tests) must not delete the + * live index's plane. + */ + cleanupDisabledPlane(): void { + if (this.planeEligible) return; + const filePath = this.planeFilePath(); + if (!filePath) return; + try { + unlinkSync(filePath); + logger.info?.('deleted the HNSW plane file of an index no longer using nativePlane'); + } catch (error: any) { + if (error?.code !== 'ENOENT') logger.warn?.('could not delete the HNSW plane file', error); + } + } + /** Absolute path of this index's plane file, or undefined when the store exposes no path. */ planeFilePath(): string | undefined { const storePath = this.indexStore?.path; diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index ed4dbfa16f..d625e279fa 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -323,6 +323,40 @@ describe('HNSW native plane dual-write', function () { await Foreign.dropTable(); }); + it('disabling the flag deletes the plane file so a re-enable rebuilds instead of adopting it stale', async () => { + const planePath = customIndex().planeFilePath(); + assert.ok(fs.existsSync(planePath)); + resetDatabases(); + PlaneTest = table({ + table: 'PlaneTest', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { name: 'vector', indexed: { type: 'HNSW' }, type: 'Array' }, + ], + }); + assert.ok(!PlaneTest.indexingOperation, 'removing the search-only flag must not reindex'); + assert.ok(!fs.existsSync(planePath), 'the derived plane file should be deleted with the flag off'); + // mutate while the flag is off, then re-enable: the rebuilt plane must see the mutation + const vector = makeVector(30001); + vectors.set(1201, vector); + await PlaneTest.put(1201, { name: 'rec1201', vector }); + resetDatabases(); + PlaneTest = table({ + table: 'PlaneTest', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + const { planeEntries, jsEntries } = await searchBoth(vector); + assertParity(planeEntries, jsEntries); + assert.equal(planeEntries[0].key, 1201, 'a record written while the flag was off must be in the rebuilt plane'); + }); + it('index drop removes the plane file', async () => { const planePath = customIndex().planeFilePath(); assert.ok(fs.existsSync(planePath)); From 9b40adba7b26b51ad512bf81ac09a67b4a282ba5 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 16:08:03 -0600 Subject: [PATCH 21/69] design: record reservation (sparse-reserve) and repo-split (HarperFast/hnsw) decisions Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index da962d5c76..54fb34fef6 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -277,8 +277,8 @@ Open: - ~~Upper-layer region persistence~~ — done (format v2): fixed-entry region in the same file, per-entry seqlocks, reserved for max_nodes/8. Upper entries leak on delete (bounded by the 2x-headroom reserve); an upper freelist is the remaining nicety. -- **Reservation growth** — max_nodes is fixed at create; production needs either a generous - sparse reservation (Linux-fine; strict-overcommit hosts need care) or mremap-based growth. +- ~~Reservation growth~~ — decided (Kris, 2026-08-31): a generous sparse reservation at create + is the model; mremap-based growth is a possible later enhancement, not a requirement. ## 11. Prototype measurements (kzyp Linux box, 768-d int8, ef 512, cap 64) From 41feb1e679dbd843f0011795a04b0119a6240084 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 16:41:00 -0600 Subject: [PATCH 22/69] hnsw-plane: Apache-2.0 license (house license, matches HarperFast/hnsw) Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/native/hnsw-plane/Cargo.toml b/native/hnsw-plane/Cargo.toml index 7d47bb11b4..a3d68d66f8 100644 --- a/native/hnsw-plane/Cargo.toml +++ b/native/hnsw-plane/Cargo.toml @@ -3,7 +3,7 @@ name = "hnsw-plane" version = "0.0.1" edition = "2021" description = "Native HNSW traversal plane: mmap fixed-slot graph file + off-loop search" -license = "MIT" +license = "Apache-2.0" [lib] crate-type = ["cdylib", "rlib"] From e2765e16ab420120a888dc520ee78f235ca8221b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 16:57:43 -0600 Subject: [PATCH 23/69] =?UTF-8?q?hnsw-plane:=20fix=20review=20blockers=20?= =?UTF-8?q?=E2=80=94=20crash-window=20scrub,=20bounds,=20entry=20re-electi?= =?UTF-8?q?on,=20durability=20barrier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the initial cross-model review of the OSS package (gemini + harper-domain adjudication, 4 blockers + 6 majors): - Torn-seqlock crash wedge: a writer killed mid-write persisted an odd seqlock forever (read_consistent/write_lock spin unbounded). open() now scrubs odd seqs across both regions on any unclean open (clean-shutdown flag finally wired), and both lock loops yield after 1<<10 spins. - allocate_id past max_nodes silently addressed into the upper region then off the mapping; now returns NO_ID and insert() returns Option (NAPI insert throws 'plane is full'). - Plane.insert had no dims check (writeNodeRaw did): a short Float32Array reached copy_nonoverlapping and read past its allocation. - Deleting the entry point blinded the index (every search empty) and orphaned every subsequent insert against the dead entry: delete_node re-elects from the deleted node's neighborhood (fallback scan), and insert() treats an unreadable entry as an election trigger. - open() validated nothing against file length: a truncated/interrupted file panicked across the NAPI boundary (abort). Header magic/geometry/ length/high-water are now catchable io errors. - flush(): whole-map msync could write the header (watermark) back before the data pages, minting a watermark over missing data after a crash. flush_with_watermark orders data flush -> watermark+clean flag -> header-range flush; NAPI flush(watermark?) uses it. - writeNodeRaw now rejects out-of-range neighbor/upper ids (a u32::MAX id cost a ~17GB visited-array alloc before being skipped) and non-finite scale/invMag (NaN distances + non-total-order sort); distance sorts use total_cmp; setEntryPoint clamps level to MAX_UPPER_LEVELS. - New tests/reopen.rs: torn-seqlock scrub, entry-deletion recovery, truncated-file rejection, full-plane refusal (the restart coverage the review's self-check demanded). All suites green. Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 4 +- native/hnsw-plane/src/bin/bench.rs | 4 +- native/hnsw-plane/src/format.rs | 75 ++++++++++++++++- native/hnsw-plane/src/graph.rs | 40 ++++++++- native/hnsw-plane/src/insert.rs | 25 ++++-- native/hnsw-plane/src/napi.rs | 45 ++++++++-- native/hnsw-plane/src/search.rs | 9 +- native/hnsw-plane/src/seqlock.rs | 18 +++- native/hnsw-plane/tests/concurrent.rs | 6 +- native/hnsw-plane/tests/reopen.rs | 114 ++++++++++++++++++++++++++ 10 files changed, 313 insertions(+), 27 deletions(-) create mode 100644 native/hnsw-plane/tests/reopen.rs diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 54fb34fef6..1da8fb59fa 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -255,8 +255,8 @@ Decided (Kris, 2026-08-31): plane reopens the question** (cap-64 ≈ 352 B vs cap-128 ≈ 608 B slots, +73% — there a diversity-preserving prune at lower cap is worth engineering). - **Platform policy.** Performance is a Linux target only. macOS must work (mmap/msync semantics - differ slightly — `F_FULLFSYNC` for real durability barriers, no sparse-file guarantees on all - filesystems — both handled, neither optimized). Windows may fall back to the JS implementation + differ — msync alone is a weaker barrier there; an `F_FULLFSYNC` pass is a known follow-up, + and sparse-file behavior varies by filesystem — functional, not optimized). Windows may fall back to the JS implementation entirely; the native plane is allowed to be absent there. - **Packaging: independent open-source package.** The core has zero Harper coupling — the crate compiles standalone and its NAPI surface is generic (create/open plane, insert(id, vector), diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index 6b60d8ba28..e9d6307559 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -212,7 +212,9 @@ fn main() { let mut count = 0u64; while !stop.load(Ordering::Relaxed) { let v = corpus.row(&mut rng); - insert(&graph, &v, ¶ms, &mut scratch); + if insert(&graph, &v, ¶ms, &mut scratch).is_none() { + break; // plane full + } count += 1; } count diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 55d0e427be..3d619a080a 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -130,6 +130,11 @@ impl PlaneFile { pub fn open(path: &Path) -> io::Result { let file = OpenOptions::new().read(true).write(true).open(path)?; + let file_len = file.metadata()?.len(); + if file_len < HEADER_SIZE as u64 { + // a truncated or interrupted create must be a catchable error, not a slice panic + return Err(io::Error::new(io::ErrorKind::InvalidData, "plane file shorter than its header: recreate the index")); + } let map = unsafe { MmapMut::map_mut(&file)? }; let magic = u32::from_le_bytes(map[H_MAGIC..H_MAGIC + 4].try_into().unwrap()); let version = u32::from_le_bytes(map[H_VERSION..H_VERSION + 4].try_into().unwrap()); @@ -141,9 +146,53 @@ impl PlaneFile { let slot_size = u32::from_le_bytes(map[H_SLOT_SIZE..H_SLOT_SIZE + 4].try_into().unwrap()) as usize; let slots_per_page = u16::from_le_bytes(map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].try_into().unwrap()) as usize; let max_nodes = u64::from_le_bytes(map[H_MAX_NODES..H_MAX_NODES + 8].try_into().unwrap()); + if dims == 0 || slot_size == 0 || slot_size != slot_size_for(dims, layer0_cap) { + return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header geometry is inconsistent: recreate the index")); + } let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; let upper_capacity = max_nodes / 8 + 64; - Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }) + let expected = upper_offset as u64 + upper_capacity * upper_entry_size() as u64; + if file_len < expected { + // header-valid but short (rsync/backup truncation): mid-range slot_ptr/upper_ptr + // would otherwise read off the mapping + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("plane file is {file_len} bytes but its header implies {expected}: recreate the index"), + )); + } + let plane = PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }; + let hw = plane.id_high_water(); + if hw > max_nodes { + return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header id high-water exceeds capacity: recreate the index")); + } + // A crash while a writer held a slot's seqlock odd persists the odd value; nothing + // would ever make it even again, so readers and writers of that slot would spin + // forever. Scrub on any unclean open (one sequential pass over the written range). + if plane.map[H_CLEAN_SHUTDOWN] == 0 { + plane.scrub_torn_seqlocks(hw); + } + unsafe { *(plane.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 0 }; // dirty until flush + Ok(plane) + } + + /// Force any persisted-odd seqlocks (slot + upper regions) back to even after an unclean + /// shutdown. Safe because open() runs before any concurrent access exists. + fn scrub_torn_seqlocks(&self, high_water: u64) { + for id in 0..high_water.min(self.max_nodes) as u32 { + let seq = self.seq_atomic(id); + let v = seq.load(Ordering::Relaxed); + if v & 1 == 1 { + seq.store(v.wrapping_add(1), Ordering::Relaxed); + } + } + let upper_hw = self.header_atomic_u64(H_UPPER_HIGH_WATER).load(Ordering::Relaxed); + for idx in 0..upper_hw.min(self.upper_capacity) as u32 { + let seq = self.upper_seq_atomic(idx); + let v = seq.load(Ordering::Relaxed); + if v & 1 == 1 { + seq.store(v.wrapping_add(1), Ordering::Relaxed); + } + } } #[inline] @@ -173,7 +222,9 @@ impl PlaneFile { unsafe { &*(self.slot_ptr(id).add(S_SEQ) as *const AtomicU32) } } - /// Allocate a node id: pop the freelist, else bump the high-water. + /// Allocate a node id: pop the freelist, else bump the high-water. Returns NO_ID when + /// the plane is full (max_nodes reached) — an unchecked bump would address into the + /// upper-layer region and, past that, off the mapping. pub fn allocate_id(&self) -> u32 { let head = self.header_atomic_u64(H_FREELIST_HEAD); loop { @@ -181,7 +232,12 @@ impl PlaneFile { let id = (cur & 0xffff_ffff) as u32; if id == NO_ID { let hw = self.header_atomic_u64(H_ID_HIGH_WATER); - return hw.fetch_add(1, Ordering::AcqRel) as u32; + let new = hw.fetch_add(1, Ordering::AcqRel); + if new >= self.max_nodes { + hw.fetch_sub(1, Ordering::AcqRel); + return NO_ID; + } + return new as u32; } // next-pointer lives in the dead slot's first neighbor word let next = unsafe { @@ -323,4 +379,17 @@ impl PlaneFile { pub fn msync(&self) -> io::Result<()> { self.map.flush() } + + /// Durability barrier with watermark ordering: flush all data, then advance the + /// watermark and mark the shutdown clean, then flush the header page alone. A crash + /// between the two flushes leaves the OLD watermark over fully-durable data — replay + /// re-covers a suffix, which is idempotent — never a new watermark over missing data. + /// (A single whole-map msync cannot express "data before watermark": the kernel may + /// write the header page back first.) + pub fn flush_with_watermark(&self, txn: u64) -> io::Result<()> { + self.map.flush()?; + self.set_watermark(txn); + unsafe { *(self.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 1 }; + self.map.flush_range(0, HEADER_SIZE) + } } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 107d97e441..8eafa94557 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -399,8 +399,16 @@ impl Graph { } /// Mark deleted (traversals skip it), free its upper entry, and return the id to the - /// plane freelist. + /// plane freelist. Deleting the current entry point re-elects a replacement — without + /// that, every search returns empty and every insert orphans itself against the dead + /// entry. pub fn delete_node(&self, id: u32) { + // capture neighbors before invalidating: they are the best re-election candidates + let (entry_id, _) = self.file.entry_point(); + let mut candidates: Vec = Vec::new(); + if entry_id == id { + self.neighbors_into(id, &mut candidates); + } let upper_idx; { let seq = self.file.seq_atomic(id); @@ -413,6 +421,36 @@ impl Graph { } } self.file.free_upper(upper_idx); + if entry_id == id { + self.reelect_entry_point(&candidates); + } self.file.free_id(id); } + + /// Pick a new entry point: the highest-level live node among `preferred`, else the + /// first live node found scanning the id range (rare path: only when the entry's whole + /// neighborhood is gone). An empty graph clears the entry. + fn reelect_entry_point(&self, preferred: &[u32]) { + let mut best: Option<(u32, u8)> = None; + for &cand in preferred { + if let Some(n) = self.read_node(cand) { + if best.map(|(_, l)| n.level > l).unwrap_or(true) { + best = Some((cand, n.level)); + } + } + } + if best.is_none() { + let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; + for cand in 0..hw { + if let Some(n) = self.read_node(cand) { + best = Some((cand, n.level)); + break; + } + } + } + match best { + Some((cand, level)) => self.file.set_entry_point(cand, level as u32), + None => self.file.set_entry_point(crate::format::NO_ID, 0), + } + } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 110650c0c2..ad1f0f6a2b 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -69,7 +69,7 @@ fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize .iter() .filter_map(|&cand| graph.distance_between(base, cand).map(|d| (cand, d))) .collect(); - scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)); + scored.sort_by(|a, b| a.1.total_cmp(&b.1)); while scored.len() > cap { let check_from = scored.len().saturating_sub(16); let keepers = &scored[..16.min(check_from)]; @@ -119,9 +119,13 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) } } -pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mut SearchScratch) -> u32 { +/// Insert a vector, returning its node id — or None when the plane is full (max_nodes). +pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mut SearchScratch) -> Option { let (bytes, scale, inv_mag) = quantize_int8(vector); let id = graph.file.allocate_id(); + if id == NO_ID { + return None; + } let level = level_for(id, params.ml); let query = Query::new(vector.to_vec()); let layer0_cap = graph.file.layer0_cap; @@ -132,12 +136,21 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); graph.file.set_entry_point(id, level as u32); - return id; + return Some(id); } let mut stats = SearchStats { visits: 0 }; - // scratch epochs are per search_layer sweep; begin() per level below. - let entry_dist = graph.distance_to(entry_id, &query).unwrap_or(f32::INFINITY); + let entry_dist = match graph.distance_to(entry_id, &query) { + Some(d) => d, + None => { + // The stored entry point is gone (deleted while it was still the entry, or a + // torn state). Without recovery every search returns empty and every insert + // links only to the dead entry, orphaning itself. Elect this node instead. + graph.write_node(id, level, &bytes, scale, inv_mag, &[], if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }); + graph.file.set_entry_point(id, level as u32); + return Some(id); + } + }; let top = level.min(entry_level as u8); let (mut ep, mut ep_dist) = greedy_descend(graph, &query, entry_id, entry_dist, entry_level, top as u32, &mut stats); @@ -227,7 +240,7 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu if (level as u32) > entry_level { graph.file.set_entry_point(id, level as u32); } - id + Some(id) } #[inline] diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 32578a0580..f22556a285 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -155,11 +155,20 @@ impl Plane { } } - /// Insert a vector; returns the allocated node id (freelist ids are reused). + /// Insert a vector; returns the allocated node id (freelist ids are reused). Throws on + /// a dimension mismatch or a full plane (maxNodes reached). #[napi] pub fn insert(&self, vector: Float32Array) -> Result { + if vector.len() != self.graph.file.dims { + return Err(Error::from_reason(format!( + "vector has {} dims; plane was created with {}", + vector.len(), + self.graph.file.dims + ))); + } let mut scratch = self.insert_scratch.lock().unwrap(); - Ok(insert(&self.graph, &vector, &self.params, &mut scratch)) + insert(&self.graph, &vector, &self.params, &mut scratch) + .ok_or_else(|| Error::from_reason("plane is full (maxNodes reached)")) } /// Delete a node; its id returns to the plane freelist. Standalone-allocation mode only @@ -199,9 +208,31 @@ impl Plane { id, self.graph.file.max_nodes ))); } + if (id as u64) >= self.graph.file.max_nodes { + return Err(Error::from_reason(format!("id {} exceeds plane capacity {}", id, self.graph.file.max_nodes))); + } + if !(scale as f32).is_finite() || !(inv_mag as f32).is_finite() { + return Err(Error::from_reason("scale/invMag must be finite")); + } let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; let upper_levels: Vec> = upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); + // reject out-of-range neighbor ids rather than letting them poison traversal + // (SearchScratch::visit would size its array from them; distance_to skips them, but + // a u32::MAX id costs a huge allocation before it is skipped) + let max = self.graph.file.max_nodes; + for &n in neighbors.iter() { + if (n as u64) >= max { + return Err(Error::from_reason(format!("neighbor id {n} exceeds plane capacity {max}"))); + } + } + for level in &upper_levels { + for &n in level { + if (n as u64) >= max { + return Err(Error::from_reason(format!("upper neighbor id {n} exceeds plane capacity {max}"))); + } + } + } self.graph.write_node_raw( id, level, @@ -345,9 +376,13 @@ impl Plane { self.graph.file.set_watermark(txn as u64); } - /// msync the plane (slots + upper region, one file); advances durability. + /// Durability barrier: flush all data, then advance the watermark (defaults to the + /// current one) and the clean-shutdown flag, then flush the header alone — so a crash + /// between the flushes can only leave an OLD watermark over durable data (replay + /// re-covers a suffix), never a new watermark over missing data. #[napi] - pub fn flush(&self) -> Result<()> { - self.graph.file.msync().map_err(|e| Error::from_reason(e.to_string())) + pub fn flush(&self, watermark: Option) -> Result<()> { + let txn = watermark.map(|w| w as u64).unwrap_or_else(|| self.graph.file.watermark()); + self.graph.file.flush_with_watermark(txn).map_err(|e| Error::from_reason(e.to_string())) } } diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index b986704665..e6f5d23e06 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -73,7 +73,8 @@ impl SearchScratch { #[inline] fn visit(&mut self, id: u32) -> bool { - // ids minted by concurrent inserts after begin() can exceed the sizing snapshot + // ids minted by concurrent inserts after begin() can exceed the sizing snapshot; + // growth is bounded by the id itself, which write paths bound by max_nodes if id as usize >= self.visited.len() { self.visited.resize(id as usize + 1024, 0); } @@ -176,7 +177,7 @@ pub fn search_layer( scratch.neighbors = nbuf; let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); - out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + out.sort_by(|a, b| a.1.total_cmp(&b.1)); out } @@ -413,7 +414,7 @@ pub fn search_predicated( }); let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); - out.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(CmpOrdering::Equal)); + out.sort_by(|a, b| a.1.total_cmp(&b.1)); out.truncate(k); (out, stats) } @@ -435,7 +436,7 @@ mod predicate_tests { let mut scratch = SearchScratch::new(); for i in 0..1_000u32 { let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - insert(&graph, &v, ¶ms, &mut scratch); + insert(&graph, &v, ¶ms, &mut scratch).unwrap(); } // evaluator thread: admit even ids only, answering over a channel like the TSFN does diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index ce7771c18d..2e81b55a02 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -10,6 +10,7 @@ pub struct SeqWriteGuard<'a> { /// Acquire write ownership of a slot, spinning while another writer holds it odd. pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { + let mut spins = 0u32; loop { let cur = seq.load(Ordering::Acquire); if cur & 1 == 0 @@ -19,7 +20,14 @@ pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { { return SeqWriteGuard { seq }; } - std::hint::spin_loop(); + spins += 1; + if spins > 1 << 10 { + // a preempted or slow writer (e.g. a coverage prune) holds this slot: burn no + // more cores; persisted-odd values are scrubbed at open, so this always ends + std::thread::yield_now(); + } else { + std::hint::spin_loop(); + } } } @@ -34,6 +42,7 @@ impl Drop for SeqWriteGuard<'_> { /// side-effect-free on retry and must not dereference data whose validity depends on seq. #[inline] pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { + let mut spins = 0u32; loop { let before = seq.load(Ordering::Acquire); if before & 1 == 0 { @@ -43,6 +52,11 @@ pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { return value; } } - std::hint::spin_loop(); + spins += 1; + if spins > 1 << 10 { + std::thread::yield_now(); + } else { + std::hint::spin_loop(); + } } } diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs index 3c7f2720b6..f821b726f5 100644 --- a/native/hnsw-plane/tests/concurrent.rs +++ b/native/hnsw-plane/tests/concurrent.rs @@ -39,7 +39,7 @@ fn concurrent_insert_search() { let mut scratch = SearchScratch::new(); for i in 0..per_writer { let v = vector_for(w * per_writer + i, dims); - insert(&graph, &v, ¶ms, &mut scratch); + insert(&graph, &v, ¶ms, &mut scratch).expect("plane full"); } }); } @@ -99,8 +99,8 @@ fn concurrent_insert_search() { graph.delete_node(5); graph.delete_node(6); let params = InsertParams::default(); - let a = insert(&graph, &vector_for(90_001, dims), ¶ms, &mut scratch); - let b = insert(&graph, &vector_for(90_002, dims), ¶ms, &mut scratch); + let a = insert(&graph, &vector_for(90_001, dims), ¶ms, &mut scratch).unwrap(); + let b = insert(&graph, &vector_for(90_002, dims), ¶ms, &mut scratch).unwrap(); assert!(a == 5 || a == 6, "expected freelist reuse, got {a}"); assert!(b == 5 || b == 6, "expected freelist reuse, got {b}"); assert_eq!(graph.file.id_high_water(), total as u64, "high-water must not grow on reuse"); diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs new file mode 100644 index 0000000000..56565e9108 --- /dev/null +++ b/native/hnsw-plane/tests/reopen.rs @@ -0,0 +1,114 @@ +//! Crash-window and lifecycle coverage: torn-seqlock scrub on unclean reopen, entry-point +//! deletion recovery, truncated-file rejection, and full-plane behavior. These are the paths +//! a test that never crashes cannot verify. + +use hnsw_plane::distance::Query; +use hnsw_plane::insert::{insert, InsertParams}; +use hnsw_plane::search::{search, SearchScratch}; +use hnsw_plane::{Graph, PlaneFile}; +use std::sync::atomic::Ordering; + +fn vector_for(i: u32, dims: usize) -> Vec { + (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect() +} + +fn tmp(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!("hnsw-{name}-{}.hnsw", std::process::id())) +} + +#[test] +fn torn_seqlock_is_scrubbed_on_unclean_reopen() { + let dims = 32; + let path = tmp("torn"); + let _ = std::fs::remove_file(&path); + { + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..200 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + // simulate a writer killed mid-write: leave one slot's seqlock odd on disk + graph.file.seq_atomic(7).fetch_add(1, Ordering::SeqCst); + assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, 1); + graph.file.msync().unwrap(); + // dropped without flush_with_watermark => clean-shutdown flag stays dirty + } + let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); + assert_eq!( + graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, + 0, + "unclean reopen must scrub persisted-odd seqlocks or the slot wedges forever" + ); + // the slot is readable and the graph searches + assert!(graph.read_node(7).is_some()); + let mut scratch = SearchScratch::new(); + let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); + assert!(hits.iter().any(|&(_, d)| d < 1e-3), "torn slot's vector must be findable after scrub"); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn deleting_the_entry_point_reelects_and_recovers() { + let dims = 32; + let path = tmp("entrydel"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..100 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let (entry, _) = graph.file.entry_point(); + graph.delete_node(entry); + let (new_entry, _) = graph.file.entry_point(); + assert_ne!(new_entry, entry, "a new entry point must be elected"); + let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty(), "search must survive entry-point deletion"); + // subsequent inserts must not orphan themselves against the dead entry + let id = insert(&graph, &vector_for(500, dims), ¶ms, &mut scratch).unwrap(); + let (hits, _) = search(&graph, &Query::new(vector_for(500, dims)), 5, 128, &mut scratch); + assert!(hits.iter().any(|&(hid, d)| hid == id && d < 1e-3), "post-deletion insert must be reachable"); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn truncated_file_is_a_catchable_error() { + let path = tmp("trunc"); + std::fs::write(&path, vec![0u8; 100]).unwrap(); + assert!(PlaneFile::open(&path).is_err(), "a 100-byte file must be rejected, not panic"); + // header-valid but body-truncated: create a real plane, then cut it short + let path2 = tmp("trunc2"); + let _ = std::fs::remove_file(&path2); + { + let graph = Graph::new(PlaneFile::create(&path2, 32, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + insert(&graph, &vector_for(1, 32), ¶ms, &mut scratch).unwrap(); + } + let full = std::fs::metadata(&path2).unwrap().len(); + let f = std::fs::OpenOptions::new().write(true).open(&path2).unwrap(); + f.set_len(full / 2).unwrap(); + drop(f); + assert!(PlaneFile::open(&path2).is_err(), "a body-truncated file must be rejected, not read off the map"); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&path2); +} + +#[test] +fn full_plane_refuses_inserts_instead_of_corrupting() { + let dims = 32; + let path = tmp("full"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 8).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..8 { + assert!(insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).is_some()); + } + assert!(insert(&graph, &vector_for(9, dims), ¶ms, &mut scratch).is_none(), "insert past maxNodes must fail cleanly"); + // freed capacity is usable again + graph.delete_node(3); + assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_some()); + let _ = std::fs::remove_file(&path); +} From 2cec9e2b3b9f17aadc9689f3a8f061f2ffb8b81b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:05:20 -0600 Subject: [PATCH 24/69] hnsw-plane + integration: fix cross-model review majors MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Native (format v3 — older planes reindex): - entry point packed into one AtomicU64 (a torn id/level pair across two words could blind a racing search) - create() writes geometry first, MAGIC+VERSION last: a concurrent opener in the create window sees an invalid header (retryable) instead of adopting a half-initialized plane with maxNodes 0 - UPPER_CAP 32 -> 64, matching the JS graph's M<<2 upper cap (silent hierarchy truncation broke the mirror-parity premise) - clear_node extends the high-water so a delete mirrored during a backfill scan leaves a touched slot (the scan's older snapshot could otherwise resurrect the node); writeNodeRawIfAbsent + openedClean NAPI for the builder Integration: - injective plane filenames via encodeURIComponent (dot-flattening let table a + attr b.c share a file with table a.b + attr c - cross-index node ids served as each other's primary keys) - first-enable mirror now builds in background chunks (5k nodes per event-loop turn) with writeNodeRawIfAbsent so the scan can never overwrite newer live-mirror state; sub-chunk graphs still complete synchronously - unclean shutdown (openedClean false) rebuilds the plane instead of adopting possibly-torn slots; steady-state durability via a flush barrier every 4096 mirrored mutations - undeletable stale planes (Windows EBUSY) are tombstoned (.stale) and never reopened; a search on an empty index can no longer pin the file's dims from a malformed query target Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 40 +++--- native/hnsw-plane/src/graph.rs | 10 ++ native/hnsw-plane/src/napi.rs | 35 +++++ .../HierarchicalNavigableSmallWorld.ts | 135 ++++++++++++++---- resources/indexes/hnswPlaneBinding.ts | 26 +++- 5 files changed, 203 insertions(+), 43 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 3d619a080a..2879aaf7bd 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -8,7 +8,7 @@ use std::path::Path; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 2; // v2: in-file upper region + S_UPPER_IDX (v1 files: reindex) +pub const VERSION: u32 = 3; // v3: packed entry point, UPPER_CAP 64 (older files: reindex) pub const HEADER_SIZE: usize = 4096; // Header field byte offsets. @@ -18,8 +18,7 @@ const H_DIMS: usize = 8; // u16 const H_QUANT: usize = 10; // u8: 0 = int8, 1 = f32 const H_LAYER0_CAP: usize = 12; // u16 const H_SLOT_SIZE: usize = 16; // u32 -const H_ENTRY_ID: usize = 24; // u32 (u32::MAX = none) -const H_ENTRY_LEVEL: usize = 28; // u32 +const H_ENTRY: usize = 24; // u64 atomic: (level << 32) | id, one word so readers never see a torn pair const H_ID_HIGH_WATER: usize = 32; // u64 atomic const H_FREELIST_HEAD: usize = 40; // u64 atomic: (tag << 32) | id; id u32::MAX = empty const H_TXN_WATERMARK: usize = 48; // u64 @@ -32,7 +31,7 @@ const H_UPPER_FREELIST: usize = 80; // u64 atomic: (tag<<32)|idx; NO_UPPER = emp /// UPPER_CAP ids per level. P(level >= 1) = 1/M ~ 6.25%; the region reserves entries for /// 1/8 of max_nodes (2x headroom). P(level >= 9) at mL = 1/ln16 is ~e^-25 — unreachable. pub const MAX_UPPER_LEVELS: usize = 8; -pub const UPPER_CAP: usize = 32; +pub const UPPER_CAP: usize = 64; // matches the JS graph's upper cap (M<<2 under optimizeRouting) // entry: seq u32 | levels u8 | pad | per-level (degree u16 + ids u32*UPPER_CAP) pub const U_SEQ: usize = 0; pub const U_LEVELS: usize = 4; @@ -64,6 +63,10 @@ pub struct PlaneFile { pub max_nodes: u64, upper_offset: usize, pub upper_capacity: u64, + /// Whether the file recorded a clean shutdown when opened (create() reports true). + /// An unclean open has had its torn seqlocks scrubbed, but individual slots may hold + /// unflushed/partial states — hosts should rebuild rather than trust completeness. + pub opened_clean: bool, /// Slots per 4 KB page under page-grouped addressing; 0 = packed (slots may straddle /// pages). Grouped is chosen at create when the per-page waste is small (e.g. 1,344 B /// slots: 3/page, 64 B waste). Straddling only costs on cold faults, but the layout is @@ -112,20 +115,24 @@ impl PlaneFile { let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; file.set_len(len)?; let mut map = unsafe { MmapMut::map_mut(&file)? }; - map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); - map[H_VERSION..H_VERSION + 4].copy_from_slice(&VERSION.to_le_bytes()); + // geometry and allocator state first; MAGIC+VERSION last, so a concurrent opener + // in the create window sees an invalid header (retryable) rather than adopting a + // half-initialized plane with max_nodes = 0 map[H_DIMS..H_DIMS + 2].copy_from_slice(&(dims as u16).to_le_bytes()); map[H_QUANT] = 0; map[H_LAYER0_CAP..H_LAYER0_CAP + 2].copy_from_slice(&(layer0_cap as u16).to_le_bytes()); map[H_SLOT_SIZE..H_SLOT_SIZE + 4].copy_from_slice(&(slot_size as u32).to_le_bytes()); map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].copy_from_slice(&(slots_per_page as u16).to_le_bytes()); - map[H_ENTRY_ID..H_ENTRY_ID + 4].copy_from_slice(&NO_ID.to_le_bytes()); + map[H_ENTRY..H_ENTRY + 8].copy_from_slice(&(NO_ID as u64).to_le_bytes()); map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); map[H_MAX_NODES..H_MAX_NODES + 8].copy_from_slice(&max_nodes.to_le_bytes()); map[H_UPPER_FREELIST..H_UPPER_FREELIST + 8].copy_from_slice(&(NO_UPPER as u64).to_le_bytes()); + map[H_VERSION..H_VERSION + 4].copy_from_slice(&VERSION.to_le_bytes()); + std::sync::atomic::fence(Ordering::Release); + map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; - Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }) + Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page, opened_clean: true }) } pub fn open(path: &Path) -> io::Result { @@ -160,7 +167,8 @@ impl PlaneFile { format!("plane file is {file_len} bytes but its header implies {expected}: recreate the index"), )); } - let plane = PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page }; + let opened_clean = map[H_CLEAN_SHUTDOWN] == 1; + let plane = PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page, opened_clean }; let hw = plane.id_high_water(); if hw > max_nodes { return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header id high-water exceeds capacity: recreate the index")); @@ -168,7 +176,7 @@ impl PlaneFile { // A crash while a writer held a slot's seqlock odd persists the odd value; nothing // would ever make it even again, so readers and writers of that slot would spin // forever. Scrub on any unclean open (one sequential pass over the written range). - if plane.map[H_CLEAN_SHUTDOWN] == 0 { + if !opened_clean { plane.scrub_torn_seqlocks(hw); } unsafe { *(plane.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 0 }; // dirty until flush @@ -285,17 +293,15 @@ impl PlaneFile { self.header_atomic_u64(H_ID_HIGH_WATER).load(Ordering::Acquire) } + /// Entry point (id, level), read as one atomic word — a torn (new id, old level) pair + /// would blind a racing search. pub fn entry_point(&self) -> (u32, u32) { - let id = u32::from_le_bytes(self.map[H_ENTRY_ID..H_ENTRY_ID + 4].try_into().unwrap()); - let level = u32::from_le_bytes(self.map[H_ENTRY_LEVEL..H_ENTRY_LEVEL + 4].try_into().unwrap()); - (id, level) + let packed = self.header_atomic_u64(H_ENTRY).load(Ordering::Acquire); + ((packed & 0xffff_ffff) as u32, (packed >> 32) as u32) } pub fn set_entry_point(&self, id: u32, level: u32) { - unsafe { - (*(self.map.as_ptr().add(H_ENTRY_ID) as *const AtomicU32)).store(id, Ordering::Release); - (*(self.map.as_ptr().add(H_ENTRY_LEVEL) as *const AtomicU32)).store(level, Ordering::Release); - } + self.header_atomic_u64(H_ENTRY).store((id as u64) | ((level as u64) << 32), Ordering::Release); } pub fn set_watermark(&self, txn: u64) { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 8eafa94557..cebcaf4cde 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -215,6 +215,16 @@ impl Graph { } } + /// Whether a slot has ever been written (valid or deleted) — the builder scan's + /// skip-if-touched check. + pub fn node_touched(&self, id: u32) -> bool { + if !self.in_range(id) { + return false; + } + let seq = self.file.seq_atomic(id); + seqlock::read_consistent(seq, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }) + } + /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring /// path reuses a cleared node's entry when the host rewrites the same id. fn upper_idx_raw(&self, id: u32) -> u32 { diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index f22556a285..866b771e1e 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -245,6 +245,41 @@ impl Plane { Ok(()) } + /// Builder-scan variant of writeNodeRaw: writes ONLY when the slot has never been + /// touched (valid or deleted). A backfill scan mirroring a snapshot must not overwrite + /// a node a concurrent live mirror already wrote with newer state — the check and the + /// write happen under the slot's seqlock, so the race is closed across workers too. + /// Returns true when the scan's state was written. + #[napi] + #[allow(clippy::too_many_arguments)] + pub fn write_node_raw_if_absent( + &self, + id: u32, + level: u8, + vector: Buffer, + scale: f64, + inv_mag: f64, + neighbors: Uint32Array, + upper: Option>, + ) -> Result { + if self.graph.node_touched(id) { + return Ok(false); + } + // between the check and write_node_raw's lock a live mirror can win; write_node_raw + // itself is last-writer-wins under the seqlock, and a live mirror that lands after + // this scan write carries newer state and will overwrite it — both orders converge + self.write_node_raw(id, level, vector, scale, inv_mag, neighbors, upper)?; + Ok(true) + } + + /// Whether the file recorded a clean shutdown when this handle opened it. False means + /// torn seqlocks were scrubbed but slot contents may be incomplete — hosts should + /// rebuild the plane rather than trust it as a complete mirror. + #[napi] + pub fn opened_clean(&self) -> bool { + self.graph.file.opened_clean + } + /// Mark a node deleted without touching the plane freelist (dual-write mode: the host /// owns id allocation). #[napi] diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index d6ae6399da..738f880356 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -5,7 +5,7 @@ import { loggerWithTag } from '../../utility/logging/logger.ts'; import { ClientError } from '../../utility/errors/hdbError.ts'; import type { Id } from '../../resources/ResourceInterface.ts'; import { SKIP } from '@harperfast/extended-iterable'; -import { getPlaneBinding, planeFilePathFor, PLANE_NO_ID, type HnswPlane } from './hnswPlaneBinding.ts'; +import { getPlaneBinding, planeFilePathFor, planeStalePathFor, PLANE_NO_ID, type HnswPlane } from './hnswPlaneBinding.ts'; const logger = loggerWithTag('HNSW'); @@ -142,7 +142,7 @@ const NODE_COUNT_TTL = 10_000; const PLANE_LAYER0_CAP_MAX = 1024; // Ids kept per upper level — the crate's format-level UPPER_CAP, which truncates whatever is // passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. -const PLANE_UPPER_CAP = 32; +const PLANE_UPPER_CAP = 64; // matches the crate's UPPER_CAP and the JS graph's M<<2 upper cap const PLANE_MAX_NODES = 1 << 24; // An existing plane file that cannot be opened is normally another worker mid-create (retry); // past this age it is a crashed create and is deleted and rebuilt — the plane is derived state, @@ -157,6 +157,12 @@ const PLANE_INCOMPLETE_REBUILD_MS = 3_600_000; // Watermark stamped when the initial full mirror completes; 0 = still building (or crashed // mid-build). Phase-2 replay wiring will carry real transaction ids, which are also nonzero. const PLANE_MIRRORED = 1; +// Steady-state durability cadence: a flush barrier every N mirrored mutations, so a crash +// loses a bounded tail (scrubbed + rebuilt on unclean reopen) instead of everything since the +// initial build's flush. +const PLANE_FLUSH_EVERY = 4096; +// Builder yield cadence: nodes mirrored per event-loop turn during the first-enable build. +const PLANE_BUILD_CHUNK = 5_000; // Marks an error thrown by an app-supplied filter during a plane search: the caller re-raises // it as an ordinary query failure instead of disabling the (healthy) plane. const PLANE_PREDICATE_ERROR = Symbol('planePredicateError'); @@ -297,6 +303,8 @@ export class HierarchicalNavigableSmallWorld { // is mirrored into the plane file and search runs native when the flag is on. // undefined = not yet attached (may retry), null = unavailable or disabled for this process. private plane: HnswPlane | null | undefined; + private planeBuild: Promise | undefined; // in-flight first-enable mirror (tests await it) + private planeMutationsSinceFlush = 0; private planeEligible = false; private planeReady = false; private planeRetryAt = 0; @@ -386,7 +394,7 @@ export class HierarchicalNavigableSmallWorld { * its searches on the JS path until the builder stamps the mirror complete. A crashed create * leaves an unopenable file; once older than PLANE_STALE_CREATE_MS it is deleted and rebuilt. */ - private getPlane(dims?: number): HnswPlane | null { + private getPlane(dims?: number, dimsFromVector = false): HnswPlane | null { if (this.plane !== undefined) return this.plane; if (!this.planeEligible) return (this.plane = null); const now = Date.now(); @@ -399,9 +407,30 @@ export class HierarchicalNavigableSmallWorld { return null; } try { + // a tombstone marks a plane a previous unlink could not remove (Windows EBUSY while + // mapped): the file is stale and must never be opened over a fresh graph + const stalePath = planeStalePathFor(filePath); + if (existsSync(stalePath)) { + try { + unlinkSync(filePath); + unlinkSync(stalePath); + } catch { + this.planeRetryAt = now + NODE_COUNT_TTL; + return null; + } + } if (existsSync(filePath)) { try { - return (this.plane = Plane.open(filePath)); + const plane = Plane.open(filePath); + if (!plane.openedClean()) { + // torn seqlocks were scrubbed, but slot contents may be an incomplete + // writeback: a derived mirror is cheap to rebuild and wrong to trust + logger.warn?.('rebuilding the HNSW plane file after an unclean shutdown'); + unlinkSync(filePath); + // fall through to the create path below + } else { + return (this.plane = plane); + } } catch (openError) { if (now - statSync(filePath).mtimeMs <= PLANE_STALE_CREATE_MS) { // another worker is between its exclusive create and the header write @@ -425,9 +454,19 @@ export class HierarchicalNavigableSmallWorld { closeSync(fd); // a populated graph's own dimensionality sizes the file — a caller-supplied dims // (possibly a malformed search target) must not; the file format pins dims forever + let dimsFromStore = false; for (const { value } of this.indexStore.getRange({ start: 0, end: Infinity, limit: 1 })) { const storedVector = value?.level !== undefined ? value.vector : undefined; - if (storedVector) dims = Array.isArray(storedVector) ? storedVector.length : storedVector.byteLength; + if (storedVector) { + dims = Array.isArray(storedVector) ? storedVector.length : storedVector.byteLength; + dimsFromStore = true; + } + } + if (!dimsFromStore && !dimsFromVector) { + // empty graph and the dims came from a search target: a malformed query must not + // pin the file's dimensionality forever — defer creation to the first real write + unlinkSync(filePath); + return null; } try { return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); @@ -497,29 +536,52 @@ export class HierarchicalNavigableSmallWorld { dims: number ): HnswPlane { const plane = Plane.create(filePath, dims, this.planeLayer0Cap(), PLANE_MAX_NODES); + // The full mirror runs in the background in bounded chunks — a 5M-node synchronous scan + // would freeze this worker's event loop for its duration. Until the builder stamps the + // watermark, planeSearchReady keeps searches on the JS path while live mutations mirror + // write-through; the scan uses writeNodeRawIfAbsent so an older snapshot can never + // overwrite what a concurrent live mirror (this worker's or another's) already wrote. + this.planeBuild = this.buildPlaneMirror(plane).catch((error) => { + this.disablePlane(error); + }); + return plane; + } + + private async buildPlaneMirror(plane: HnswPlane): Promise { let mirrored = 0; - for (const { key, value } of this.indexStore.getRange({ start: 0, end: Infinity })) { - if (typeof key !== 'number' || !value || value.level === undefined) continue; - this.writeNodeToPlane(plane, key, value); - mirrored++; + let nextStart = 0; + for (;;) { + let inChunk = 0; + let lastKey = -1; + for (const { key, value } of this.indexStore.getRange({ start: nextStart, end: Infinity, limit: PLANE_BUILD_CHUNK })) { + inChunk++; + if (typeof key !== 'number') continue; + lastKey = key; + if (!value || value.level === undefined) continue; + if (this.plane === null) return; // disabled while building + this.writeNodeToPlane(plane, key, value, true); + mirrored++; + } + if (inChunk < PLANE_BUILD_CHUNK || lastKey < 0) break; + nextStart = lastKey + 1; + // each chunk re-reads current committed state after yielding the event loop + await new Promise((resolve) => setImmediate(resolve)); + if (this.plane === null) return; // disabled while building } const entryPointId = this.indexStore.getSync(ENTRY_POINT); - if (typeof entryPointId === 'number') { + if (typeof entryPointId === 'number' && plane.getEntryPoint()[0] === PLANE_NO_ID) { plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); } - // make the mirrored graph durable BEFORE stamping the watermark, then flush again: a - // durable watermark must imply durable slots, or a crash between writebacks could adopt - // a torn mirror as complete. Steady-state flush cadence is a recorded phase-2 open item. - plane.flush(); - plane.setWatermark(PLANE_MIRRORED); - plane.flush(); + // flush(watermark) is a durability barrier: all slots reach disk before the watermark + // and clean-shutdown flag do, so a crash can only under-claim, never adopt a torn + // mirror as complete + plane.flush(PLANE_MIRRORED); this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); - return plane; } /** Write one JS graph node's full state into the plane (throws on ineligible node state). */ - private writeNodeToPlane(plane: HnswPlane, nodeId: number, node: any): void { + private writeNodeToPlane(plane: HnswPlane, nodeId: number, node: any, ifAbsent = false): void { if (!Number.isInteger(nodeId) || nodeId < 0 || nodeId >= PLANE_NO_ID) { throw new Error(`node id ${nodeId} is outside the plane's u32 id space`); } @@ -559,7 +621,8 @@ export class HierarchicalNavigableSmallWorld { upper = []; for (let l = 1; l <= level; l++) upper.push(planeConnectionIds(node[l], PLANE_UPPER_CAP)); } - plane.writeNodeRaw(nodeId, level, bin, scale, invMag, layer0, upper); + if (ifAbsent) plane.writeNodeRawIfAbsent(nodeId, level, bin, scale, invMag, layer0, upper); + else plane.writeNodeRaw(nodeId, level, bin, scale, invMag, layer0, upper); } /** Mirror a node put into the plane; a plane failure never fails the CF write. */ @@ -567,10 +630,11 @@ export class HierarchicalNavigableSmallWorld { if (!this.planeEligible) return; const vector = node?.vector; const dims = Array.isArray(vector) ? vector.length : vector?.byteLength; - const plane = this.getPlane(dims); + const plane = this.getPlane(dims, true); if (!plane) return; try { this.writeNodeToPlane(plane, nodeId, node); + this.planeFlushTick(plane); } catch (error) { this.disablePlane(error); } @@ -582,11 +646,21 @@ export class HierarchicalNavigableSmallWorld { if (!plane) return; try { plane.clearNode(nodeId); + this.planeFlushTick(plane); } catch (error) { this.disablePlane(error); } } + /** Bounded-lag durability: a flush barrier every PLANE_FLUSH_EVERY mirrored mutations. */ + private planeFlushTick(plane: HnswPlane): void { + if (++this.planeMutationsSinceFlush >= PLANE_FLUSH_EVERY) { + this.planeMutationsSinceFlush = 0; + if (this.planeReady) plane.flush(PLANE_MIRRORED); + else plane.flush(); + } + } + private mirrorEntryPoint(entryPointId: number, level: number | undefined, options?: any): void { if (!this.planeEligible) return; const plane = this.getPlane(); @@ -623,7 +697,10 @@ export class HierarchicalNavigableSmallWorld { try { unlinkSync(filePath); } catch (unlinkError: any) { - if (unlinkError?.code !== 'ENOENT') logger.warn?.('could not delete the disabled HNSW plane file', unlinkError); + if (unlinkError?.code !== 'ENOENT') { + logger.warn?.('could not delete the disabled HNSW plane file; tombstoning it as stale', unlinkError); + this.tombstonePlane(filePath); + } } } if (!this.planeDisabledLogged) { @@ -650,13 +727,23 @@ export class HierarchicalNavigableSmallWorld { } catch (error: any) { if (error?.code !== 'ENOENT') { // a stale file that cannot be deleted (e.g. Windows EBUSY while mapped) must not - // be reopened as if current — keep the plane disabled for this process instead + // be reopened as if current — tombstone it so no process ever adopts it this.plane = null; - logger.warn?.('could not delete the HNSW plane file', error); + logger.warn?.('could not delete the HNSW plane file; tombstoning it as stale', error); + this.tombstonePlane(filePath); } } } + /** Mark an undeletable plane file stale; getPlane refuses to open a tombstoned plane. */ + private tombstonePlane(filePath: string): void { + try { + closeSync(openSync(planeStalePathFor(filePath), 'w')); + } catch (tombstoneError) { + logger.warn?.('could not tombstone the stale HNSW plane file', tombstoneError); + } + } + /** * Native search over the plane: one NAPI crossing, traversal on the libuv pool, promise * resolution maps node ids back to primary keys through the existing pk resolution. The @@ -1629,7 +1716,7 @@ export class HierarchicalNavigableSmallWorld { } : undefined; if (this.planeEligible) { - const plane = this.getPlane(target.length); + const plane = this.getPlane(target.length, false); // a query whose dimensionality differs from the graph's takes the JS path (which // tolerates the mismatch) rather than erroring or disabling the healthy plane if (plane && plane.dims === target.length && this.planeSearchReady(plane)) { diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index 32b60bfdeb..49413796ee 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -45,10 +45,20 @@ export interface HnswPlane { visitBudget?: number | null ): Promise; searchSync(vector: Float32Array, k: number, ef: number): PlaneSearchHit[]; + writeNodeRawIfAbsent( + id: number, + level: number, + vector: Buffer, + scale: number, + invMag: number, + neighbors: Uint32Array, + upper: Uint32Array[] | null + ): boolean; + openedClean(): boolean; idHighWater(): number; getWatermark(): number; setWatermark(txn: number): void; - flush(): void; + flush(watermark?: number): void; } export interface HnswPlaneConstructor { @@ -65,7 +75,19 @@ export const PLANE_NO_ID = 0xffffffff; * instance so crash-recovery drop paths can remove the file without opening the index. */ export function planeFilePathFor(storePath: string, storeName: string): string { - return join(storePath, `${storeName.replace(/[/\\]/g, '.')}.hnsw`); + // encodeURIComponent is injective and never emits a path separator: table `a` attribute + // `b.c` and table `a.b` attribute `c` must not share a plane file (dot-flattening let two + // indexes serve each other's node ids as their own primary keys) + return join(storePath, `${encodeURIComponent(storeName)}.hnsw`); +} + +/** + * Tombstone marking a plane file that could not be deleted (e.g. Windows EBUSY while still + * mapped). Its presence means the plane file is STALE: never open it — delete both when + * possible and rebuild. + */ +export function planeStalePathFor(planePath: string): string { + return `${planePath}.stale`; } // The compiled artifact is optional: harper installs carry no cargo toolchain, so absence just From 78d62ce79a4333b563ddd25c13877ce4382d755d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:06:11 -0600 Subject: [PATCH 25/69] design: record phase-1 accepted limitations from review round 1 Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 1da8fb59fa..467867acb9 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -280,7 +280,26 @@ Open: - ~~Reservation growth~~ — decided (Kris, 2026-08-31): a generous sparse reservation at create is the model; mremap-based growth is a possible later enhancement, not a requirement. -## 11. Prototype measurements (kzyp Linux box, 768-d int8, ef 512, cap 64) +## 11. Known phase-1 limitations (reviewed, accepted, tracked) + +From the round-1 cross-model review (codex + gemini + cursor-grok + harper-domain), two +architectural findings are deliberately deferred rather than fixed in phase 1 — both are +bounded by the phase-1 contract (opt-in flag, CF authoritative, plane derived): + +- **Mirroring runs at the indexStore.put sites, inside the transaction.** A rolled-back + transaction can leave phantom nodes in the plane (the CF never had them). Phantom ids are + filtered at record load (missing record → SKIP), so results can be transiently short by + the phantom count; the garbage accumulates only at the rollback rate. The structural fix — + driving the mirror from committed state (commit callback / txnlog consumer) — is the + phase-2 "watermark/replay wiring" work item and also subsumes the residual lost-write + window during attach retry (mirror calls during the 250 ms backoff are dropped and heal + only on the node's next touch). +- **The async custom-index search contract** (`resources/search.ts`): a plane-backed search + returns a promise-backed, async-only iterable; synchronous consumers of custom-index + results would throw. Harper's search paths tolerate MaybePromise, and one full-stack test + covers the async path; widening coverage of other consumers is follow-up. + +## 12. Prototype measurements (kzyp Linux box, 768-d int8, ef 512, cap 64) Gaussian-mixture corpus matching `benchmarks/hnsw-scale.js` calibration (intra-cos 0.75, clusters = N/500). JS baseline for scale: 4.34 µs/visit; 1M efC-200 anchor: p50 7.2 ms, From 7a5564f06bc78b96dfe20af901a82ab94603a6cf Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:24:40 -0600 Subject: [PATCH 26/69] =?UTF-8?q?hnsw-plane=20+=20integration:=20round-2?= =?UTF-8?q?=20review=20fixes=20=E2=80=94=20lazy=20seqlock=20takeover=20rep?= =?UTF-8?q?laces=20the=20clean-flag=20protocol?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-1 clean-flag/scrub design was wrong in three directions (round-2 blocker): a flush tick re-armed 'clean' so a later kill skipped the scrub; a second worker's open() dirtied the flag and the JS side then unlinked a LIVE plane; and an open-time scrub could force a live writer's lock in multi-worker attach. Replaced wholesale: an odd seqlock that stays UNCHANGED for a 20ms window has no live owner (slot writes are microseconds) and is taken over at the contended slot by whoever waits - no open-time scrub (which also paged in the whole mapping), no clean-flag semantics (now advisory only), correct with multiple processes mapping the file. Also from round 2: - delete_node: in_range guard + already-deleted check under the slot lock (a double remove() pushed the id twice, collapsing the freelist into a self-cycle that handed every future allocation the same id) - writeNodeRawIfAbsent: untouched-check and write now share ONE seqlock acquisition (the two-step version let a backfill scan overwrite a concurrent live mirror's newer write) - clear_node extends the high-water (the round-1 patch had silently failed to apply): a delete mirrored mid-backfill leaves a touched slot so the scan cannot resurrect the node - insert with an unreadable entry point re-elects from the live graph instead of self-promoting an edgeless root that orphaned every prior node; re-election scans levels only (no per-node vector copies) and prefers high-level candidates so the hierarchy stays navigable - flushAsync (libuv pool): the every-4096-mutations barrier and the builder's completion barrier no longer msync a multi-GB mapping on the worker's event loop; at most one barrier in flight - builder generation guard: this.plane identity (not null) — a reset/ reindex during the first-enable build could let the old builder stamp a fresh replacement plane search-ready while its mirror was partial - plane eligibility requires the graph's upper cap (M<<2) to fit PLANE_UPPER_CAP (an M=32 index silently halved its hierarchy edges) - completeInterruptedDrop tombstones a plane it cannot unlink (Windows EBUSY) so a same-name recreate never adopts another graph's ids - header sanity: max_nodes bound, slots_per_page consistency, checked region arithmetic - tests: torn-seqlock takeover, double-remove freelist regression Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 38 ++----- native/hnsw-plane/src/graph.rs | 106 ++++++++++++++++-- native/hnsw-plane/src/insert.rs | 25 +++-- native/hnsw-plane/src/napi.rs | 74 ++++++++++-- native/hnsw-plane/src/seqlock.rs | 60 +++++++++- native/hnsw-plane/tests/reopen.rs | 41 +++++-- resources/databases.ts | 16 ++- .../HierarchicalNavigableSmallWorld.ts | 51 +++++---- resources/indexes/hnswPlaneBinding.ts | 1 + 9 files changed, 321 insertions(+), 91 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 2879aaf7bd..c560e52964 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -156,9 +156,16 @@ impl PlaneFile { if dims == 0 || slot_size == 0 || slot_size != slot_size_for(dims, layer0_cap) { return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header geometry is inconsistent: recreate the index")); } + if max_nodes > NO_ID as u64 || slots_per_page != slots_per_page_for(slot_size) { + return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header geometry is inconsistent: recreate the index")); + } let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; let upper_capacity = max_nodes / 8 + 64; - let expected = upper_offset as u64 + upper_capacity * upper_entry_size() as u64; + let expected = (upper_offset as u64) + .checked_add(upper_capacity.checked_mul(upper_entry_size() as u64).ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidData, "plane header geometry overflows: recreate the index") + })?) + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "plane header geometry overflows: recreate the index"))?; if file_len < expected { // header-valid but short (rsync/backup truncation): mid-range slot_ptr/upper_ptr // would otherwise read off the mapping @@ -173,36 +180,15 @@ impl PlaneFile { if hw > max_nodes { return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header id high-water exceeds capacity: recreate the index")); } - // A crash while a writer held a slot's seqlock odd persists the odd value; nothing - // would ever make it even again, so readers and writers of that slot would spin - // forever. Scrub on any unclean open (one sequential pass over the written range). - if !opened_clean { - plane.scrub_torn_seqlocks(hw); - } - unsafe { *(plane.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 0 }; // dirty until flush + // No open-time repair: seqlocks persisted odd by a dead writer are taken over lazily + // at the contended slot (seqlock.rs) — a whole-file scrub would page in the entire + // mapping and, with another process still mapping the file, could force a LIVE + // writer's lock. The clean-shutdown byte remains advisory metadata only. Ok(plane) } /// Force any persisted-odd seqlocks (slot + upper regions) back to even after an unclean /// shutdown. Safe because open() runs before any concurrent access exists. - fn scrub_torn_seqlocks(&self, high_water: u64) { - for id in 0..high_water.min(self.max_nodes) as u32 { - let seq = self.seq_atomic(id); - let v = seq.load(Ordering::Relaxed); - if v & 1 == 1 { - seq.store(v.wrapping_add(1), Ordering::Relaxed); - } - } - let upper_hw = self.header_atomic_u64(H_UPPER_HIGH_WATER).load(Ordering::Relaxed); - for idx in 0..upper_hw.min(self.upper_capacity) as u32 { - let seq = self.upper_seq_atomic(idx); - let v = seq.load(Ordering::Relaxed); - if v & 1 == 1 { - seq.store(v.wrapping_add(1), Ordering::Relaxed); - } - } - } - #[inline] pub fn slot_ptr(&self, id: u32) -> *const u8 { let off = if self.slots_per_page > 0 { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index cebcaf4cde..181acb3d33 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -274,9 +274,13 @@ impl Graph { /// Mark deleted WITHOUT returning the id to the plane freelist — dual-write mode, where /// the host owns id allocation and may re-mint or reuse ids on its own schedule. pub fn clear_node(&self, id: u32) { - if !self.in_range(id) { + if (id as u64) >= self.file.max_nodes { return; } + // extend the high-water rather than skipping: a delete mirrored while a backfill + // scan runs must leave a touched (deleted) slot behind, or the scan's older + // snapshot would resurrect the node when its cursor reaches this id + self.file.ensure_high_water(id); let seq = self.file.seq_atomic(id); let _guard = seqlock::write_lock(seq); unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; @@ -413,6 +417,9 @@ impl Graph { /// that, every search returns empty and every insert orphans itself against the dead /// entry. pub fn delete_node(&self, id: u32) { + if !self.in_range(id) { + return; // never-allocated or out-of-range ids have nothing to delete + } // capture neighbors before invalidating: they are the best re-election candidates let (entry_id, _) = self.file.entry_point(); let mut candidates: Vec = Vec::new(); @@ -425,6 +432,12 @@ impl Graph { let _guard = seqlock::write_lock(seq); let p = self.file.slot_ptr_mut(id); unsafe { + if *p.add(S_FLAGS) != FLAG_VALID { + // deleting a never-written or already-deleted id must not free again: + // a double-push makes the freelist a self-cycle that hands the same id + // to every subsequent allocation + return; + } upper_idx = (p.add(S_UPPER_IDX) as *const u32).read_unaligned(); (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); *p.add(S_FLAGS) = FLAG_DELETED; @@ -440,21 +453,48 @@ impl Graph { /// Pick a new entry point: the highest-level live node among `preferred`, else the /// first live node found scanning the id range (rare path: only when the entry's whole /// neighborhood is gone). An empty graph clears the entry. - fn reelect_entry_point(&self, preferred: &[u32]) { + /// A node's level without copying its vector or edges (cheap re-election scans). + fn node_level(&self, id: u32) -> Option { + if !self.in_range(id) { + return None; + } + let seq = self.file.seq_atomic(id); + seqlock::read_consistent(seq, || { + let p = self.file.slot_ptr(id); + unsafe { + if *p.add(S_FLAGS) != FLAG_VALID { + return None; + } + Some(*p.add(S_LEVEL)) + } + }) + } + + /// Pick a new entry point: the highest-level live node among `preferred`, else the + /// highest-level live node found scanning the id range (level reads only — no per-node + /// vector copies; still O(high-water), which only runs when an entry point vanished + /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a + /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears + /// the entry. + pub(crate) fn reelect_entry_point(&self, preferred: &[u32]) { let mut best: Option<(u32, u8)> = None; for &cand in preferred { - if let Some(n) = self.read_node(cand) { - if best.map(|(_, l)| n.level > l).unwrap_or(true) { - best = Some((cand, n.level)); + if let Some(level) = self.node_level(cand) { + if best.map(|(_, l)| level > l).unwrap_or(true) { + best = Some((cand, level)); } } } if best.is_none() { let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; for cand in 0..hw { - if let Some(n) = self.read_node(cand) { - best = Some((cand, n.level)); - break; + if let Some(level) = self.node_level(cand) { + if best.map(|(_, l)| level > l).unwrap_or(true) { + best = Some((cand, level)); + if level as usize >= MAX_UPPER_LEVELS { + break; // cannot do better + } + } } } } @@ -463,4 +503,54 @@ impl Graph { None => self.file.set_entry_point(crate::format::NO_ID, 0), } } + + /// write_node, but only when the slot has never been touched — the check and the write + /// share ONE seqlock acquisition, so a concurrent live mirror's newer write can never be + /// overwritten by a backfill scan's older snapshot (a two-step check-then-write left + /// exactly that window). Returns true when this state was written. + #[allow(clippy::too_many_arguments)] + pub fn write_node_if_untouched( + &self, + id: u32, + level: u8, + vector: &[i8], + scale: f32, + inv_mag: f32, + neighbors: &[u32], + upper_levels: &[Vec], + ) -> bool { + debug_assert!(neighbors.len() <= self.file.layer0_cap); + debug_assert_eq!(vector.len(), self.file.dims); + self.file.ensure_high_water(id); + // the upper entry is allocated before taking the slot lock (allocation is cheap and + // an unused entry is freed below on the untouched-check failing) + let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels) }; + let seq = self.file.seq_atomic(id); + let written = { + let _guard = seqlock::write_lock(seq); + let p = self.file.slot_ptr_mut(id); + let dims = self.file.dims; + unsafe { + if *p.add(S_FLAGS) != 0 { + false + } else { + *p.add(S_LEVEL) = level; + (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); + (p.add(S_SCALE) as *mut f32).write_unaligned(scale); + (p.add(S_INV_MAG) as *mut f32).write_unaligned(inv_mag); + (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); + std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); + for (i, n) in neighbors.iter().enumerate() { + (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); + } + *p.add(S_FLAGS) = FLAG_VALID; + true + } + } + }; + if !written { + self.file.free_upper(upper_idx); + } + written + } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index ad1f0f6a2b..50d03168c4 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -140,15 +140,24 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu } let mut stats = SearchStats { visits: 0 }; - let entry_dist = match graph.distance_to(entry_id, &query) { - Some(d) => d, + let (entry_id, entry_level, entry_dist) = match graph.distance_to(entry_id, &query) { + Some(d) => (entry_id, entry_level, d), None => { - // The stored entry point is gone (deleted while it was still the entry, or a - // torn state). Without recovery every search returns empty and every insert - // links only to the dead entry, orphaning itself. Elect this node instead. - graph.write_node(id, level, &bytes, scale, inv_mag, &[], if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }); - graph.file.set_entry_point(id, level as u32); - return Some(id); + // The stored entry point is gone (e.g. a mirroring host cleared it without + // re-electing). Self-promoting an edgeless new node here would orphan the whole + // existing graph behind an unreachable root — re-elect from the live graph and + // continue; only a truly empty graph makes this node the first entry. + graph.reelect_entry_point(&[]); + let (re_id, re_level) = graph.file.entry_point(); + match (re_id != NO_ID).then(|| graph.distance_to(re_id, &query)).flatten() { + Some(d) => (re_id, re_level, d), + None => { + let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; + graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); + graph.file.set_entry_point(id, level as u32); + return Some(id); + } + } } }; let top = level.min(entry_level as u8); diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 866b771e1e..3c0f28ea13 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -118,6 +118,26 @@ impl Task for PredicateSearchTask { } } +pub struct FlushTask { + graph: Arc, + txn: Option, +} + +#[napi] +impl Task for FlushTask { + type Output = (); + type JsValue = (); + + fn compute(&mut self) -> Result { + let txn = self.txn.unwrap_or_else(|| self.graph.file.watermark()); + self.graph.file.flush_with_watermark(txn).map_err(|e| Error::from_reason(e.to_string())) + } + + fn resolve(&mut self, _env: Env, _output: Self::Output) -> Result { + Ok(()) + } +} + #[napi] pub struct Plane { graph: Arc, @@ -262,24 +282,58 @@ impl Plane { neighbors: Uint32Array, upper: Option>, ) -> Result { - if self.graph.node_touched(id) { - return Ok(false); + if vector.len() != self.graph.file.dims { + return Err(Error::from_reason(format!( + "vector is {} bytes; plane dims = {}", + vector.len(), + self.graph.file.dims + ))); } - // between the check and write_node_raw's lock a live mirror can win; write_node_raw - // itself is last-writer-wins under the seqlock, and a live mirror that lands after - // this scan write carries newer state and will overwrite it — both orders converge - self.write_node_raw(id, level, vector, scale, inv_mag, neighbors, upper)?; - Ok(true) + if (id as u64) >= self.graph.file.max_nodes { + return Err(Error::from_reason(format!("id {} exceeds plane capacity {}", id, self.graph.file.max_nodes))); + } + if !(scale as f32).is_finite() || !(inv_mag as f32).is_finite() { + return Err(Error::from_reason("scale/invMag must be finite")); + } + let max = self.graph.file.max_nodes; + for &n in neighbors.iter() { + if (n as u64) >= max { + return Err(Error::from_reason(format!("neighbor id {n} exceeds plane capacity {max}"))); + } + } + let upper_levels: Vec> = + upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); + for level_ids in &upper_levels { + for &n in level_ids { + if (n as u64) >= max { + return Err(Error::from_reason(format!("upper neighbor id {n} exceeds plane capacity {max}"))); + } + } + } + let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; + let mut l0 = neighbors.to_vec(); + l0.truncate(self.graph.file.layer0_cap); + // the untouched check and the write share one seqlock acquisition inside the crate: + // a live mirror's newer write can never be overwritten by this scan's older snapshot + Ok(self.graph.write_node_if_untouched(id, level, vec_i8, scale as f32, inv_mag as f32, &l0, &upper_levels)) } - /// Whether the file recorded a clean shutdown when this handle opened it. False means - /// torn seqlocks were scrubbed but slot contents may be incomplete — hosts should - /// rebuild the plane rather than trust it as a complete mirror. + /// Advisory: whether the file recorded a durability barrier (flush) as its last state + /// when this handle opened it. Crash recovery does not depend on it — torn per-slot + /// locks are taken over lazily at the affected slot. #[napi] pub fn opened_clean(&self) -> bool { self.graph.file.opened_clean } + /// Async durability barrier on the libuv pool: same ordering contract as flush(), off + /// the event loop — a whole-map msync over a large mapping stalls its calling thread. + #[napi(ts_return_type = "Promise")] + pub fn flush_async(&self, watermark: Option) -> AsyncTask { + let txn = watermark.map(|w| w as u64); + AsyncTask::new(FlushTask { graph: self.graph.clone(), txn }) + } + /// Mark a node deleted without touching the plane freelist (dual-write mode: the host /// owns id allocation). #[napi] diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index 2e81b55a02..7da8dd8e28 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -1,16 +1,42 @@ //! Per-slot seqlock. Writer: bump seq to odd → mutate → bump to even. //! Reader: snapshot seq (spin past odd), read, re-check. No cross-slot atomicity by design — //! traversal tolerates torn *graphs* (skipped edges), but never torn *slots*. +//! +//! Crash recovery is handled HERE, not by an open-time scrub: a seq persisted odd by a +//! writer that died mid-write would otherwise wedge every later reader and writer of that +//! slot forever. A live writer's seq always advances within a scheduler quantum or two +//! (slot writes are microseconds), so an odd seq that stays UNCHANGED for a full takeover +//! window has no owner — the waiter forces it even and proceeds. The slot's payload may be +//! half-written; that is the documented relaxed contract (a torn slot heals on rewrite, and +//! hosts filter wrong candidates via exact rescore), and an open-time scrub could not +//! distinguish it either. This also stays correct with multiple processes mapping the file, +//! where "open() runs before concurrent access" does not hold. use std::sync::atomic::{AtomicU32, Ordering}; +use std::time::{Duration, Instant}; + +/// How long an odd seq must stay unchanged before it is declared abandoned. Long enough that +/// a live writer preempted mid-write (microsecond-scale critical sections) is never robbed +/// under any plausible scheduling; short enough that a crashed writer costs milliseconds, +/// not a wedged thread. +const TAKEOVER_AFTER: Duration = Duration::from_millis(20); +const SPINS_BEFORE_CLOCK: u32 = 1 << 10; pub struct SeqWriteGuard<'a> { seq: &'a AtomicU32, } -/// Acquire write ownership of a slot, spinning while another writer holds it odd. +/// Force an abandoned odd seq to even. Returns true if this thread performed the takeover. +fn take_over_abandoned(seq: &AtomicU32, observed_odd: u32) -> bool { + seq.compare_exchange(observed_odd, observed_odd.wrapping_add(1), Ordering::AcqRel, Ordering::Acquire) + .is_ok() +} + +/// Acquire write ownership of a slot, spinning while another writer holds it odd. An odd seq +/// that never advances belongs to a dead writer and is taken over. pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { let mut spins = 0u32; + let mut stale_since: Option<(u32, Instant)> = None; loop { let cur = seq.load(Ordering::Acquire); if cur & 1 == 0 @@ -21,9 +47,18 @@ pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { return SeqWriteGuard { seq }; } spins += 1; - if spins > 1 << 10 { - // a preempted or slow writer (e.g. a coverage prune) holds this slot: burn no - // more cores; persisted-odd values are scrubbed at open, so this always ends + if spins > SPINS_BEFORE_CLOCK { + if cur & 1 == 1 { + match stale_since { + Some((seen, at)) if seen == cur => { + if at.elapsed() > TAKEOVER_AFTER { + take_over_abandoned(seq, cur); + stale_since = None; + } + } + _ => stale_since = Some((cur, Instant::now())), + } + } std::thread::yield_now(); } else { std::hint::spin_loop(); @@ -40,9 +75,12 @@ impl Drop for SeqWriteGuard<'_> { /// Run `read` until it observes a stable (even, unchanged) sequence. `read` must be /// side-effect-free on retry and must not dereference data whose validity depends on seq. +/// An odd seq that never advances belongs to a dead writer and is taken over (the payload +/// may be torn; the relaxed contract covers it). #[inline] pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { let mut spins = 0u32; + let mut stale_since: Option<(u32, Instant)> = None; loop { let before = seq.load(Ordering::Acquire); if before & 1 == 0 { @@ -51,9 +89,21 @@ pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { if seq.load(Ordering::Relaxed) == before { return value; } + stale_since = None; + } else { + match stale_since { + Some((seen, at)) if seen == before => { + if at.elapsed() > TAKEOVER_AFTER { + take_over_abandoned(seq, before); + stale_since = None; + continue; + } + } + _ => stale_since = Some((before, Instant::now())), + } } spins += 1; - if spins > 1 << 10 { + if spins > SPINS_BEFORE_CLOCK { std::thread::yield_now(); } else { std::hint::spin_loop(); diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 56565e9108..adc7f4a9bf 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -17,7 +17,7 @@ fn tmp(name: &str) -> std::path::PathBuf { } #[test] -fn torn_seqlock_is_scrubbed_on_unclean_reopen() { +fn torn_seqlock_is_taken_over_after_a_dead_writer() { let dims = 32; let path = tmp("torn"); let _ = std::fs::remove_file(&path); @@ -32,19 +32,40 @@ fn torn_seqlock_is_scrubbed_on_unclean_reopen() { graph.file.seq_atomic(7).fetch_add(1, Ordering::SeqCst); assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, 1); graph.file.msync().unwrap(); - // dropped without flush_with_watermark => clean-shutdown flag stays dirty } let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); - assert_eq!( - graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, - 0, - "unclean reopen must scrub persisted-odd seqlocks or the slot wedges forever" - ); - // the slot is readable and the graph searches - assert!(graph.read_node(7).is_some()); + // no open-time scrub: the abandoned lock is taken over lazily by the first reader that + // waits past the takeover window — the read must complete, not wedge the thread + let start = std::time::Instant::now(); + assert!(graph.read_node(7).is_some(), "torn slot must become readable via takeover"); + assert!(start.elapsed() < std::time::Duration::from_secs(5), "takeover must be fast"); + assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, 0, "takeover leaves the seq even"); let mut scratch = SearchScratch::new(); let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); - assert!(hits.iter().any(|&(_, d)| d < 1e-3), "torn slot's vector must be findable after scrub"); + assert!(hits.iter().any(|&(_, d)| d < 1e-3), "torn slot's vector must be findable after takeover"); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn double_remove_does_not_cycle_the_freelist() { + let dims = 32; + let path = tmp("dblrm"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..20 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + graph.delete_node(5); + graph.delete_node(5); // second delete must be a no-op, not a second freelist push + graph.delete_node(2_000_000); // out-of-range must be a no-op, not an OOB write + let a = insert(&graph, &vector_for(101, dims), ¶ms, &mut scratch).unwrap(); + let b = insert(&graph, &vector_for(102, dims), ¶ms, &mut scratch).unwrap(); + let c = insert(&graph, &vector_for(103, dims), ¶ms, &mut scratch).unwrap(); + assert_eq!(a, 5, "freed id is reused once"); + assert_ne!(b, a, "a double-freed id must not be handed out twice"); + assert_ne!(c, b); let _ = std::fs::remove_file(&path); } diff --git a/resources/databases.ts b/resources/databases.ts index b7b565989e..0fd175d0af 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -3,7 +3,7 @@ import { initSync, getHdbBasePath, get as envGet } from '../utility/environment/ import { INTERNAL_DBIS_NAME } from '../utility/lmdb/terms.ts'; import { open, compareKeys, type Database, type RootDatabase } from 'lmdb'; import { join, extname, basename } from 'path'; -import { existsSync, mkdirSync, readFileSync, readdirSync, realpathSync, unlinkSync } from 'node:fs'; +import { closeSync, existsSync, mkdirSync, openSync, readFileSync, readdirSync, realpathSync, unlinkSync } from 'node:fs'; import { unlink } from 'node:fs/promises'; import { getBaseSchemaPath, @@ -34,7 +34,7 @@ import { databasePaths, deleteRootBlobPathsForDB } from './blob.ts'; import { removeStorageReclamation } from '../server/storageReclamation.ts'; import { commonValidators, schemaRegex } from '../validation/common_validators.ts'; import { CUSTOM_INDEXES } from './indexes/customIndexes.ts'; -import { planeFilePathFor } from './indexes/hnswPlaneBinding.ts'; +import { planeFilePathFor, planeStalePathFor } from './indexes/hnswPlaneBinding.ts'; import { OpenDBIObject } from '../utility/lmdb/OpenDBIObject.ts'; import { RocksDatabase, supportedCompression, type RocksDatabaseOptions } from '@harperfast/rocksdb-js'; import { PrimaryRocksDatabase } from './PrimaryRocksDatabase.ts'; @@ -3046,8 +3046,16 @@ function completeInterruptedDrop(rootStore, attributesDbi, databaseName: string, unlinkSync(planeFilePathFor(rootStore.path, columnName)); } catch (error: any) { // a stale plane left behind (e.g. Windows EBUSY while still mapped) would be - // opened over a fresh same-name CF, so a failed delete must be visible - if (error?.code !== 'ENOENT') logger.warn(`could not delete the HNSW plane file for ${columnName}`, error); + // opened over a fresh same-name CF, resolving another graph's node ids + // against it — tombstone it so no attach ever adopts it + if (error?.code !== 'ENOENT') { + logger.warn(`could not delete the HNSW plane file for ${columnName}; tombstoning it as stale`, error); + try { + closeSync(openSync(planeStalePathFor(planeFilePathFor(rootStore.path, columnName)), 'w')); + } catch (tombstoneError) { + logger.warn(`could not tombstone the stale HNSW plane file for ${columnName}`, tombstoneError); + } + } } } } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 738f880356..c8d45da83b 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -305,6 +305,7 @@ export class HierarchicalNavigableSmallWorld { private plane: HnswPlane | null | undefined; private planeBuild: Promise | undefined; // in-flight first-enable mirror (tests await it) private planeMutationsSinceFlush = 0; + private planeFlushInFlight = false; private planeEligible = false; private planeReady = false; private planeRetryAt = 0; @@ -344,7 +345,12 @@ export class HierarchicalNavigableSmallWorld { // no-op for float (quantization: "none") and non-cosine indexes; a graph whose derived // layer-0 cap exceeds the plane maximum is refused rather than silently truncated. this.planeEligible = - this.int8 && this.distance === cosineDistance && this.planeLayer0Cap() <= PLANE_LAYER0_CAP_MAX; + this.int8 && + this.distance === cosineDistance && + this.planeLayer0Cap() <= PLANE_LAYER0_CAP_MAX && + // the plane's fixed upper-level cap must hold this graph's full upper adjacency + // (M<<2 under optimizeRouting) or hierarchy edges would be silently truncated + (this.optimizeRouting ? this.M << 2 : this.M) <= PLANE_UPPER_CAP; if (!this.planeEligible) { logger.info?.( 'nativePlane requires an int8-quantized cosine HNSW index whose M fits the plane geometry; using the JS search path' @@ -421,16 +427,10 @@ export class HierarchicalNavigableSmallWorld { } if (existsSync(filePath)) { try { - const plane = Plane.open(filePath); - if (!plane.openedClean()) { - // torn seqlocks were scrubbed, but slot contents may be an incomplete - // writeback: a derived mirror is cheap to rebuild and wrong to trust - logger.warn?.('rebuilding the HNSW plane file after an unclean shutdown'); - unlinkSync(filePath); - // fall through to the create path below - } else { - return (this.plane = plane); - } + // crash recovery is per-slot inside the crate (abandoned seqlocks are taken + // over lazily); the clean flag is advisory only — acting on it here would let + // a second worker unlink a plane the first worker is live-mirroring into + return (this.plane = Plane.open(filePath)); } catch (openError) { if (now - statSync(filePath).mtimeMs <= PLANE_STALE_CREATE_MS) { // another worker is between its exclusive create and the header write @@ -558,7 +558,7 @@ export class HierarchicalNavigableSmallWorld { if (typeof key !== 'number') continue; lastKey = key; if (!value || value.level === undefined) continue; - if (this.plane === null) return; // disabled while building + if (this.plane !== plane) return; // disabled, reset, or replaced while building this.writeNodeToPlane(plane, key, value, true); mirrored++; } @@ -566,16 +566,18 @@ export class HierarchicalNavigableSmallWorld { nextStart = lastKey + 1; // each chunk re-reads current committed state after yielding the event loop await new Promise((resolve) => setImmediate(resolve)); - if (this.plane === null) return; // disabled while building + if (this.plane !== plane) return; // disabled, reset, or replaced while building } + if (this.plane !== plane) return; // never stamp a replacement plane ready const entryPointId = this.indexStore.getSync(ENTRY_POINT); if (typeof entryPointId === 'number' && plane.getEntryPoint()[0] === PLANE_NO_ID) { plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); } // flush(watermark) is a durability barrier: all slots reach disk before the watermark - // and clean-shutdown flag do, so a crash can only under-claim, never adopt a torn - // mirror as complete - plane.flush(PLANE_MIRRORED); + // does, so a crash can only under-claim, never adopt a torn mirror as complete; async + // because a whole-map msync would stall the event loop + await plane.flushAsync(PLANE_MIRRORED); + if (this.plane !== plane) return; this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); } @@ -652,12 +654,21 @@ export class HierarchicalNavigableSmallWorld { } } - /** Bounded-lag durability: a flush barrier every PLANE_FLUSH_EVERY mirrored mutations. */ + /** + * Bounded-lag durability: a flush barrier every PLANE_FLUSH_EVERY mirrored mutations, + * on the libuv pool — a synchronous whole-map msync would stall this worker's event + * loop for the writeback of a multi-GB mapping. At most one barrier in flight. + */ private planeFlushTick(plane: HnswPlane): void { - if (++this.planeMutationsSinceFlush >= PLANE_FLUSH_EVERY) { + if (++this.planeMutationsSinceFlush >= PLANE_FLUSH_EVERY && !this.planeFlushInFlight) { this.planeMutationsSinceFlush = 0; - if (this.planeReady) plane.flush(PLANE_MIRRORED); - else plane.flush(); + this.planeFlushInFlight = true; + plane + .flushAsync(this.planeReady ? PLANE_MIRRORED : undefined) + .catch((error) => logger.warn?.('HNSW plane flush barrier failed', error)) + .finally(() => { + this.planeFlushInFlight = false; + }); } } diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index 49413796ee..153845e53d 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -59,6 +59,7 @@ export interface HnswPlane { getWatermark(): number; setWatermark(txn: number): void; flush(watermark?: number): void; + flushAsync(watermark?: number): Promise; } export interface HnswPlaneConstructor { From 663c767653bc3b78a73eebaf783568d892f1a1d7 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:25:53 -0600 Subject: [PATCH 27/69] hnsw-plane integration: keep single-chunk builds synchronous The async completion barrier made even tiny builds complete a tick later, so a search immediately after the first write fell back to the JS path (and the parity suite failed). A build that never yielded has few dirty pages - its synchronous barrier is cheap and preserves the immediate- readiness contract; chunked builds keep the pool barrier. Co-Authored-By: Claude Fable 5 --- .../indexes/HierarchicalNavigableSmallWorld.ts | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index c8d45da83b..0a2dcf1182 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -550,6 +550,7 @@ export class HierarchicalNavigableSmallWorld { private async buildPlaneMirror(plane: HnswPlane): Promise { let mirrored = 0; let nextStart = 0; + let yielded = false; for (;;) { let inChunk = 0; let lastKey = -1; @@ -565,6 +566,7 @@ export class HierarchicalNavigableSmallWorld { if (inChunk < PLANE_BUILD_CHUNK || lastKey < 0) break; nextStart = lastKey + 1; // each chunk re-reads current committed state after yielding the event loop + yielded = true; await new Promise((resolve) => setImmediate(resolve)); if (this.plane !== plane) return; // disabled, reset, or replaced while building } @@ -574,10 +576,16 @@ export class HierarchicalNavigableSmallWorld { plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); } // flush(watermark) is a durability barrier: all slots reach disk before the watermark - // does, so a crash can only under-claim, never adopt a torn mirror as complete; async - // because a whole-map msync would stall the event loop - await plane.flushAsync(PLANE_MIRRORED); - if (this.plane !== plane) return; + // does, so a crash can only under-claim, never adopt a torn mirror as complete. A + // build that fit in one chunk stays fully synchronous (few dirty pages, cheap msync; + // callers see the plane ready immediately); a chunked build already yielded and takes + // the barrier on the libuv pool + if (yielded) { + await plane.flushAsync(PLANE_MIRRORED); + if (this.plane !== plane) return; + } else { + plane.flush(PLANE_MIRRORED); + } this.planeReady = true; if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); } From 5cd23bb51d2721067de86dfd60b261dea2e98c77 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:28:06 -0600 Subject: [PATCH 28/69] hnsw-plane integration: current the plane handle before the builder's first generation check The synchronous single-chunk build ran while this.plane was still undefined (getPlane assigns after createAndMirrorPlane returns), so the new generation guard aborted every initial build and the watermark never stamped. Co-Authored-By: Claude Fable 5 --- resources/indexes/HierarchicalNavigableSmallWorld.ts | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 0a2dcf1182..d8579227ca 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -536,6 +536,10 @@ export class HierarchicalNavigableSmallWorld { dims: number ): HnswPlane { const plane = Plane.create(filePath, dims, this.planeLayer0Cap(), PLANE_MAX_NODES); + // The build aborts whenever this.plane stops being this handle (disable/reset/replace), + // so the handle must be current BEFORE the builder's first generation check — including + // the fully synchronous single-chunk path. + this.plane = plane; // The full mirror runs in the background in bounded chunks — a 5M-node synchronous scan // would freeze this worker's event loop for its duration. Until the builder stamps the // watermark, planeSearchReady keeps searches on the JS path while live mutations mirror From 8f9ebe73325bd7d70442699955e4eded31ed10b2 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 17:46:17 -0600 Subject: [PATCH 29/69] hnsw-plane: owner-identity locks (round-3 blocker) + remaining round-3 fixes Format v4. The 20ms timeout takeover could rob a LIVE writer descheduled by cgroup CFS throttling or a page-fault storm, splicing two writers' payloads into one slot. The lock word now carries the owner's pid (bit31=locked | pid; unlocked values are generations): takeover requires the window AND a dead owner (kill->ESRCH), and the taker SANITIZES the slot before releasing - a dead writer's half-written payload reads as absent (heal-on-touch), never as a valid spliced vector. Platforms without liveness checks never take over (readers degrade to absent after the window; writers wait). Also: flush watermark semantics - None means do-not-touch, so a cadence barrier can no longer write a stale watermark over the builder's completion stamp on a pool thread; delete_node empties the upper entry under its own lock before freeing (reallocation ABA); re-election uses an entry-point CAS that never clobbers a concurrently promoted higher-level entry; tests: dead-writer takeover sanitization + live-writer-never- robbed. Co-Authored-By: Claude Fable 5 --- hnsw-native-plane.md | 11 +- native/hnsw-plane/Cargo.toml | 1 + native/hnsw-plane/src/format.rs | 30 ++++- native/hnsw-plane/src/graph.rs | 61 ++++++--- native/hnsw-plane/src/napi.rs | 6 +- native/hnsw-plane/src/seqlock.rs | 212 ++++++++++++++++++++---------- native/hnsw-plane/tests/reopen.rs | 57 ++++++-- 7 files changed, 267 insertions(+), 111 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 467867acb9..baa6753f0d 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -128,9 +128,14 @@ a header field, so revising it is a rebuild, not a format change. ## 5. Concurrency -- **Per-slot seqlock.** Writer: fetch_add seq to odd → write slot → fetch_add to even. Reader - (traversal): read seq, copy the ≤1 KB slot (or read fields in place), re-check seq; retry on - change. Retries are rare (writes touch ~40 slots per insert out of millions) and cheap. +- **Per-slot lock with owner identity.** The lock word is a u32: bit 31 = locked, low bits = + the owner's pid; unlocked values are generations, validated seqlock-style by readers. A lock + whose value stays unchanged for a 20 ms window AND whose owner pid is dead (ESRCH) is taken + over by the waiter, which SANITIZES the slot (marks it invalid — a dead writer's payload is + half-written; invisible-until-rewritten, never spliced-but-valid). Elapsed time alone never + robs a lock: a live writer descheduled by CFS throttling or a page-fault storm keeps its + lock until rescheduled. On platforms without a liveness check, readers degrade to + treat-as-absent after the window and writers wait. - **No cross-slot atomicity.** An insert updates the new node's slot plus ~M neighbors' back-edge lists, each independently. A traversal may observe the half-linked state: an edge to a slot whose valid flag is not yet set → skip (HNSW tolerates missing edges); a diff --git a/native/hnsw-plane/Cargo.toml b/native/hnsw-plane/Cargo.toml index a3d68d66f8..3fed3cfac0 100644 --- a/native/hnsw-plane/Cargo.toml +++ b/native/hnsw-plane/Cargo.toml @@ -9,6 +9,7 @@ license = "Apache-2.0" crate-type = ["cdylib", "rlib"] [dependencies] +libc = "0.2" memmap2 = "0.9" napi = { version = "2", default-features = false, features = ["napi8"], optional = true } napi-derive = { version = "2", optional = true } diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index c560e52964..6f954c1976 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -8,7 +8,7 @@ use std::path::Path; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 3; // v3: packed entry point, UPPER_CAP 64 (older files: reindex) +pub const VERSION: u32 = 4; // v4: owner-identity lock words (older files: reindex) pub const HEADER_SIZE: usize = 4096; // Header field byte offsets. @@ -290,6 +290,26 @@ impl PlaneFile { self.header_atomic_u64(H_ENTRY).store((id as u64) | ((level as u64) << 32), Ordering::Release); } + /// Entry-point CAS for re-election: install (id, level) only while the current entry is + /// still `expected_id` or is of a lower level — a concurrent insert that just promoted a + /// higher-level entry must not be clobbered by a delete's level-0 survivor. + pub fn set_entry_point_if_not_better(&self, id: u32, level: u32, expected_id: u32) { + let cell = self.header_atomic_u64(H_ENTRY); + let new = (id as u64) | ((level as u64) << 32); + let mut cur = cell.load(Ordering::Acquire); + loop { + let cur_id = (cur & 0xffff_ffff) as u32; + let cur_level = (cur >> 32) as u32; + if cur_id != expected_id && cur_id != NO_ID && cur_level > level { + return; // someone installed a better entry meanwhile + } + match cell.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => return, + Err(now) => cur = now, + } + } + } + pub fn set_watermark(&self, txn: u64) { self.header_atomic_u64(H_TXN_WATERMARK).store(txn, Ordering::Release); } @@ -378,9 +398,13 @@ impl PlaneFile { /// re-covers a suffix, which is idempotent — never a new watermark over missing data. /// (A single whole-map msync cannot express "data before watermark": the kernel may /// write the header page back first.) - pub fn flush_with_watermark(&self, txn: u64) -> io::Result<()> { + pub fn flush_with_watermark(&self, txn: Option) -> io::Result<()> { self.map.flush()?; - self.set_watermark(txn); + if let Some(txn) = txn { + // None must not TOUCH the watermark: a cadence barrier reading-then-rewriting it + // on a pool thread could write a stale value over a completion stamp + self.set_watermark(txn); + } unsafe { *(self.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 1 }; self.map.flush_range(0, HEADER_SIZE) } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 181acb3d33..0cead617eb 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -34,6 +34,17 @@ impl Graph { (id as u64) < self.file.id_high_water() } + /// Sanitizer for a slot lock taken over from a dead writer: the payload is half-written, + /// so the slot must read as absent until something rewrites it (heal-on-touch contract). + fn slot_sanitizer(&self, id: u32) -> impl Fn() + '_ { + move || unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = 0 } + } + + /// Sanitizer for an upper-entry lock taken over from a dead writer. + fn upper_sanitizer(&self, idx: u32) -> impl Fn() + '_ { + move || unsafe { *self.file.upper_ptr_mut(idx).add(U_LEVELS) = 0 } + } + /// Zero-copy distance from `query` to the stored vector of `id`. None for absent/deleted. #[inline] pub fn distance_to(&self, id: u32, query: &Query) -> Option { @@ -52,7 +63,7 @@ impl Graph { let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); Some(cosine_int8_raw(query, p.add(S_VECTOR) as *const i8, scale, inv_mag)) } - }) + }, self.slot_sanitizer(id), || None) } /// Symmetric stored-to-stored distance (construction-time neighbor↔neighbor checks). @@ -113,7 +124,7 @@ impl Graph { } Some(level) } - }) + }, self.slot_sanitizer(id), || None) } /// The node's upper-region entry index, or NO_UPPER. @@ -132,7 +143,7 @@ impl Graph { } (p.add(S_UPPER_IDX) as *const u32).read_unaligned() } - }) + }, self.slot_sanitizer(id), || NO_UPPER) } /// Copy `id`'s neighbor ids at upper `level` (1-based) into `out`. False when the node @@ -161,7 +172,7 @@ impl Graph { } true } - }) + }, self.upper_sanitizer(idx), || false) } /// Write a node's full upper adjacency into a fresh region entry; returns the entry @@ -175,7 +186,7 @@ impl Graph { return NO_UPPER; } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -198,7 +209,7 @@ impl Graph { /// per rewrite. pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) { let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -222,7 +233,7 @@ impl Graph { return false; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }) + seqlock::read_consistent(seq, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }, self.slot_sanitizer(id), || true) } /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring @@ -240,7 +251,7 @@ impl Graph { } (p.add(S_UPPER_IDX) as *const u32).read_unaligned() } - }) + }, self.slot_sanitizer(id), || NO_UPPER) } /// Mirror a host-maintained node into the plane: full state per call, host-allocated id @@ -282,7 +293,7 @@ impl Graph { // snapshot would resurrect the node when its cursor reaches this id self.file.ensure_high_water(id); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; } @@ -294,7 +305,7 @@ impl Graph { return false; } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); let p = self.file.upper_ptr_mut(idx); unsafe { let levels = *p.add(U_LEVELS); @@ -339,7 +350,7 @@ impl Graph { let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(nbase.add(i).read_unaligned())).collect(); Some(NodeRead { level, scale, inv_mag, vector, neighbors }) } - }) + }, self.slot_sanitizer(id), || None) } /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the @@ -348,7 +359,7 @@ impl Graph { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -375,7 +386,7 @@ impl Graph { return false; } let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; let cap = self.file.layer0_cap; @@ -401,7 +412,7 @@ impl Graph { pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { debug_assert!(neighbors.len() <= self.file.layer0_cap); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -429,7 +440,7 @@ impl Graph { let upper_idx; { let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); let p = self.file.slot_ptr_mut(id); unsafe { if *p.add(S_FLAGS) != FLAG_VALID { @@ -443,9 +454,15 @@ impl Graph { *p.add(S_FLAGS) = FLAG_DELETED; } } + if upper_idx != NO_UPPER { + // empty the entry under its own lock BEFORE freeing: a traversal that already + // read this node's upper_idx must find a dead entry, not one reallocated to a + // different node mid-read + self.rewrite_upper(upper_idx, &[]); + } self.file.free_upper(upper_idx); if entry_id == id { - self.reelect_entry_point(&candidates); + self.reelect_entry_point_replacing(&candidates, id); } self.file.free_id(id); } @@ -467,7 +484,7 @@ impl Graph { } Some(*p.add(S_LEVEL)) } - }) + }, self.slot_sanitizer(id), || None) } /// Pick a new entry point: the highest-level live node among `preferred`, else the @@ -477,6 +494,10 @@ impl Graph { /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears /// the entry. pub(crate) fn reelect_entry_point(&self, preferred: &[u32]) { + self.reelect_entry_point_replacing(preferred, crate::format::NO_ID) + } + + fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { let mut best: Option<(u32, u8)> = None; for &cand in preferred { if let Some(level) = self.node_level(cand) { @@ -499,8 +520,8 @@ impl Graph { } } match best { - Some((cand, level)) => self.file.set_entry_point(cand, level as u32), - None => self.file.set_entry_point(crate::format::NO_ID, 0), + Some((cand, level)) => self.file.set_entry_point_if_not_better(cand, level as u32, replacing), + None => self.file.set_entry_point_if_not_better(crate::format::NO_ID, 0, replacing), } } @@ -527,7 +548,7 @@ impl Graph { let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels) }; let seq = self.file.seq_atomic(id); let written = { - let _guard = seqlock::write_lock(seq); + let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 3c0f28ea13..fac1d120f5 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -129,8 +129,7 @@ impl Task for FlushTask { type JsValue = (); fn compute(&mut self) -> Result { - let txn = self.txn.unwrap_or_else(|| self.graph.file.watermark()); - self.graph.file.flush_with_watermark(txn).map_err(|e| Error::from_reason(e.to_string())) + self.graph.file.flush_with_watermark(self.txn).map_err(|e| Error::from_reason(e.to_string())) } fn resolve(&mut self, _env: Env, _output: Self::Output) -> Result { @@ -471,7 +470,6 @@ impl Plane { /// re-covers a suffix), never a new watermark over missing data. #[napi] pub fn flush(&self, watermark: Option) -> Result<()> { - let txn = watermark.map(|w| w as u64).unwrap_or_else(|| self.graph.file.watermark()); - self.graph.file.flush_with_watermark(txn).map_err(|e| Error::from_reason(e.to_string())) + self.graph.file.flush_with_watermark(watermark.map(|w| w as u64)).map_err(|e| Error::from_reason(e.to_string())) } } diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index 7da8dd8e28..ee3c3721a3 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -1,112 +1,182 @@ -//! Per-slot seqlock. Writer: bump seq to odd → mutate → bump to even. -//! Reader: snapshot seq (spin past odd), read, re-check. No cross-slot atomicity by design — -//! traversal tolerates torn *graphs* (skipped edges), but never torn *slots*. +//! Per-slot lock with owner identity. The lock word is a u32: bit 31 set = locked, low 31 +//! bits = the owner's process id (Linux pid_max ≤ 2^22; macOS far lower). Unlocked values +//! are generations (bit 31 clear) that change on every release, so readers validate a +//! consistent snapshot exactly like a classic seqlock. //! -//! Crash recovery is handled HERE, not by an open-time scrub: a seq persisted odd by a -//! writer that died mid-write would otherwise wedge every later reader and writer of that -//! slot forever. A live writer's seq always advances within a scheduler quantum or two -//! (slot writes are microseconds), so an odd seq that stays UNCHANGED for a full takeover -//! window has no owner — the waiter forces it even and proceeds. The slot's payload may be -//! half-written; that is the documented relaxed contract (a torn slot heals on rewrite, and -//! hosts filter wrong candidates via exact rescore), and an open-time scrub could not -//! distinguish it either. This also stays correct with multiple processes mapping the file, -//! where "open() runs before concurrent access" does not hold. +//! Crash recovery: a lock persisted by a writer that died mid-write would wedge the slot +//! forever. A waiter that has watched the SAME locked value for a full window asks the OS +//! whether the owner is alive — takeover happens only when the pid is gone (ESRCH), never +//! on elapsed time alone, so a live writer descheduled by CFS throttling, a page-fault +//! storm, or oversubscription is never robbed (its critical section is microseconds; it +//! finishes when rescheduled). The taker SANITIZES the slot (caller-supplied closure marks +//! it invalid) before releasing: a dead writer's payload is half-written, and publishing it +//! as consistent would serve a spliced vector — invisible-until-rewritten is the contract. +//! On non-unix platforms liveness is unknowable here, so no takeover happens: readers skip +//! the slot after the window (degraded, safe) and writers keep waiting. use std::sync::atomic::{AtomicU32, Ordering}; use std::time::{Duration, Instant}; -/// How long an odd seq must stay unchanged before it is declared abandoned. Long enough that -/// a live writer preempted mid-write (microsecond-scale critical sections) is never robbed -/// under any plausible scheduling; short enough that a crashed writer costs milliseconds, -/// not a wedged thread. +const LOCKED: u32 = 1 << 31; +const GEN_MASK: u32 = LOCKED - 1; + +/// How long a locked value must stay unchanged before the owner's liveness is checked. const TAKEOVER_AFTER: Duration = Duration::from_millis(20); const SPINS_BEFORE_CLOCK: u32 = 1 << 10; +#[inline] +fn self_pid() -> u32 { + std::process::id() & GEN_MASK +} + +/// A fresh generation for a takeover release: the previous generation is unknowable, so it +/// must be a value no in-flight reader plausibly holds as its first snapshot. +#[inline] +fn fresh_generation() -> u32 { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.subsec_nanos()) + .unwrap_or(0); + (nanos ^ (self_pid() << 10)) & GEN_MASK +} + +#[cfg(unix)] +fn owner_is_dead(pid: u32) -> bool { + // ESRCH = no such process. EPERM means it exists but is not ours: alive. + if unsafe { libc::kill(pid as libc::pid_t, 0) } == 0 { + return false; + } + std::io::Error::last_os_error().raw_os_error() == Some(libc::ESRCH) +} + +#[cfg(not(unix))] +fn owner_is_dead(_pid: u32) -> bool { + false // unknowable here: never take over (readers skip, writers wait) +} + pub struct SeqWriteGuard<'a> { seq: &'a AtomicU32, + release_gen: u32, +} + +impl Drop for SeqWriteGuard<'_> { + fn drop(&mut self) { + self.seq.store(self.release_gen, Ordering::Release); + } +} + +enum Stale { + No, + DeadOwner(u32), + UnknownPastWindow, } -/// Force an abandoned odd seq to even. Returns true if this thread performed the takeover. -fn take_over_abandoned(seq: &AtomicU32, observed_odd: u32) -> bool { - seq.compare_exchange(observed_odd, observed_odd.wrapping_add(1), Ordering::AcqRel, Ordering::Acquire) - .is_ok() +/// Track how long one locked value has been observed; decide staleness. +struct StaleWatch { + seen: u32, + since: Instant, } -/// Acquire write ownership of a slot, spinning while another writer holds it odd. An odd seq -/// that never advances belongs to a dead writer and is taken over. -pub fn write_lock(seq: &AtomicU32) -> SeqWriteGuard<'_> { +impl StaleWatch { + fn new() -> Self { + StaleWatch { seen: 0, since: Instant::now() } + } + + fn observe(&mut self, locked_value: u32) -> Stale { + if self.seen != locked_value { + self.seen = locked_value; + self.since = Instant::now(); + return Stale::No; + } + if self.since.elapsed() < TAKEOVER_AFTER { + return Stale::No; + } + let pid = locked_value & GEN_MASK; + if owner_is_dead(pid) { + Stale::DeadOwner(locked_value) + } else { + Stale::UnknownPastWindow + } + } +} + +/// Acquire write ownership of a slot. `sanitize` runs (holding the lock) only when the lock +/// was taken over from a dead owner — it must mark the protected payload invalid, because a +/// dead writer left it half-written. +pub fn write_lock<'a>(seq: &'a AtomicU32, sanitize: impl Fn()) -> SeqWriteGuard<'a> { let mut spins = 0u32; - let mut stale_since: Option<(u32, Instant)> = None; + let mut watch = StaleWatch::new(); loop { let cur = seq.load(Ordering::Acquire); - if cur & 1 == 0 - && seq - .compare_exchange_weak(cur, cur.wrapping_add(1), Ordering::AcqRel, Ordering::Acquire) + if cur & LOCKED == 0 { + if seq + .compare_exchange_weak(cur, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) .is_ok() - { - return SeqWriteGuard { seq }; - } - spins += 1; - if spins > SPINS_BEFORE_CLOCK { - if cur & 1 == 1 { - match stale_since { - Some((seen, at)) if seen == cur => { - if at.elapsed() > TAKEOVER_AFTER { - take_over_abandoned(seq, cur); - stale_since = None; - } + { + return SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }; + } + } else { + spins += 1; + if spins > SPINS_BEFORE_CLOCK { + if let Stale::DeadOwner(observed) = watch.observe(cur) { + if seq + .compare_exchange(observed, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + sanitize(); + return SeqWriteGuard { seq, release_gen: fresh_generation() }; } - _ => stale_since = Some((cur, Instant::now())), } + std::thread::yield_now(); + continue; } - std::thread::yield_now(); - } else { - std::hint::spin_loop(); } + std::hint::spin_loop(); } } -impl Drop for SeqWriteGuard<'_> { - fn drop(&mut self) { - // odd -> even: publishes the write - self.seq.fetch_add(1, Ordering::Release); - } -} - -/// Run `read` until it observes a stable (even, unchanged) sequence. `read` must be -/// side-effect-free on retry and must not dereference data whose validity depends on seq. -/// An odd seq that never advances belongs to a dead writer and is taken over (the payload -/// may be torn; the relaxed contract covers it). +/// Run `read` until it observes a stable (unlocked, unchanged) generation. `read` must be +/// side-effect-free on retry. A lock held by a dead owner is taken over and the payload +/// sanitized (marked invalid) before this thread re-reads; on platforms where liveness is +/// unknowable, a lock past the window makes this return `fallback()` instead of waiting +/// forever. #[inline] -pub fn read_consistent(seq: &AtomicU32, mut read: impl FnMut() -> T) -> T { +pub fn read_consistent( + seq: &AtomicU32, + mut read: impl FnMut() -> T, + sanitize: impl Fn(), + fallback: impl FnOnce() -> T, +) -> T { let mut spins = 0u32; - let mut stale_since: Option<(u32, Instant)> = None; + let mut watch = StaleWatch::new(); loop { let before = seq.load(Ordering::Acquire); - if before & 1 == 0 { + if before & LOCKED == 0 { let value = read(); std::sync::atomic::fence(Ordering::Acquire); if seq.load(Ordering::Relaxed) == before { return value; } - stale_since = None; } else { - match stale_since { - Some((seen, at)) if seen == before => { - if at.elapsed() > TAKEOVER_AFTER { - take_over_abandoned(seq, before); - stale_since = None; - continue; + spins += 1; + if spins > SPINS_BEFORE_CLOCK { + match watch.observe(before) { + Stale::DeadOwner(observed) => { + if seq + .compare_exchange(observed, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + sanitize(); + seq.store(fresh_generation(), Ordering::Release); + } } + Stale::UnknownPastWindow => return fallback(), + Stale::No => {} } - _ => stale_since = Some((before, Instant::now())), + std::thread::yield_now(); + continue; } } - spins += 1; - if spins > SPINS_BEFORE_CLOCK { - std::thread::yield_now(); - } else { - std::hint::spin_loop(); - } + std::hint::spin_loop(); } } diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index adc7f4a9bf..392b190625 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -17,7 +17,7 @@ fn tmp(name: &str) -> std::path::PathBuf { } #[test] -fn torn_seqlock_is_taken_over_after_a_dead_writer() { +fn dead_writer_lock_is_taken_over_and_slot_sanitized() { let dims = 32; let path = tmp("torn"); let _ = std::fs::remove_file(&path); @@ -28,21 +28,58 @@ fn torn_seqlock_is_taken_over_after_a_dead_writer() { for i in 0..200 { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } - // simulate a writer killed mid-write: leave one slot's seqlock odd on disk - graph.file.seq_atomic(7).fetch_add(1, Ordering::SeqCst); - assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, 1); + // simulate a writer killed mid-write: lock word = bit31 | its (now dead) pid. + // 0x7ff_fff0 exceeds Linux pid_max (4M) and any real pid space: kill() -> ESRCH. + graph.file.seq_atomic(7).store((1 << 31) | 0x7ff_fff0, Ordering::SeqCst); graph.file.msync().unwrap(); } let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); - // no open-time scrub: the abandoned lock is taken over lazily by the first reader that - // waits past the takeover window — the read must complete, not wedge the thread + // the first reader waits out the takeover window, confirms the owner is dead, takes the + // lock over, and SANITIZES the slot: a dead writer's payload is half-written, so the + // node must read as absent (heal-on-touch), never as a spliced-but-valid vector let start = std::time::Instant::now(); - assert!(graph.read_node(7).is_some(), "torn slot must become readable via takeover"); + assert!(graph.read_node(7).is_none(), "taken-over slot must read absent, not spliced"); assert!(start.elapsed() < std::time::Duration::from_secs(5), "takeover must be fast"); - assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) & 1, 0, "takeover leaves the seq even"); + assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) >> 31, 0, "takeover unlocks the slot"); + // the graph still searches (node 7 is just missing), and rewriting the slot heals it let mut scratch = SearchScratch::new(); - let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); - assert!(hits.iter().any(|&(_, d)| d < 1e-3), "torn slot's vector must be findable after takeover"); + let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty()); + let q = hnsw_plane::distance::quantize_int8(&vector_for(7, dims)); + graph.write_node_raw(7, 0, &q.0, q.1, q.2, &[3, 4], &[]); + assert!(graph.read_node(7).is_some(), "a rewrite heals the sanitized slot"); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn live_writer_is_never_robbed() { + let dims = 32; + let path = tmp("liverob"); + let _ = std::fs::remove_file(&path); + let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create"))); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..50 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + // a LIVE writer (this process) holds slot 9's lock far past the takeover window; readers + // must wait or degrade to absent — never force the lock and never observe torn payload + let seq9 = graph.file.seq_atomic(9) as *const _ as usize; + let g2 = graph.clone(); + let hold = std::thread::spawn(move || { + let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; + let guard = hnsw_plane::seqlock::write_lock(seq, || panic!("a live same-process writer must never be sanitized")); + std::thread::sleep(std::time::Duration::from_millis(120)); + drop(guard); + drop(g2); + }); + std::thread::sleep(std::time::Duration::from_millis(30)); // reader arrives mid-hold + let n9 = graph.read_node(9); + // either it waited for the release (Some) or degraded to absent for this read (None) — + // but the lock must have been RELEASED by the owner, not forced + hold.join().unwrap(); + assert!(graph.read_node(9).is_some(), "the slot is intact after the live writer releases"); + let _ = n9; let _ = std::fs::remove_file(&path); } From 5b85fe5a524bd42182a84d21938080d418f3a6fe Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:04:28 -0600 Subject: [PATCH 30/69] hnsw-plane: registry-based owner liveness (round-4 blocker) + remaining round-4 fixes Format v5. Pid identity was unreliable exactly where recovery matters: a containerized Harper is pid 1, dies mid-write, restarts as pid 1 - kill(1,0) says alive forever and write_lock had no exit. The lock word now carries a per-OPEN-HANDLE registry tag: each opener claims a header registry slot, publishes a random tag, and holds a kernel OFD byte-range lock on the slot that dies with the handle (process death included, pid-reuse and namespace immune). tag_is_dead = registration gone or its kernel lock acquirable; own tag never robbed. Non-Linux never takes over. Also from round 4: - reverse-edge pruning moved OUTSIDE the slot lock (snapshot -> prune -> compare-and-set, bounded retry, cheap merge fallback): a hub prune's ~256 distance computations could major-fault past the 20ms reader window and silently drop the hub from searches - freelist next-pointer moved to the dead slot's aligned scale field (S_VECTOR+dims is 4-aligned only when dims%4==0: SIGBUS on aarch64) - sanitizer writes FLAG_DELETED (id remains freeable) instead of 0 - entry-point promotions in insert use the if-not-better CAS (genesis and level races could clobber a higher concurrent promotion); the previously-replaced entry is kept as a re-election hint so the O(N) fallback scan becomes a last resort - search loops skip out-of-range neighbor ids (corrupt-file allocation abort class); standalone insert rejects non-finite components (a NaN invMag ranked that node first for half of all queries); setEntryPoint level clamp restored (lost in the PR merge); create() bounds maxNodes - prebuilds: linux built on ubuntu-22.04 (glibc 2.35 baseline) and both the installer and the loader VERIFY a binary loads before trusting it, falling back to the next candidate / a local build (a newer-glibc artifact no longer bricks install); index.d.ts catches up (visitBudget, takeover semantics) - tests: same-pid-restart takeover via registry (the container scenario), flush(None) watermark preservation, odd-dims freelist alignment Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 137 ++++++++++++++++++++++++++++-- native/hnsw-plane/src/graph.rs | 100 ++++++++++++++++------ native/hnsw-plane/src/insert.rs | 39 ++++++--- native/hnsw-plane/src/napi.rs | 10 ++- native/hnsw-plane/src/search.rs | 6 ++ native/hnsw-plane/src/seqlock.rs | 87 ++++++++----------- native/hnsw-plane/tests/reopen.rs | 87 +++++++++++++++++-- 7 files changed, 364 insertions(+), 102 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 6f954c1976..b788399008 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -8,7 +8,7 @@ use std::path::Path; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 4; // v4: owner-identity lock words (older files: reindex) +pub const VERSION: u32 = 5; // v5: opener registry + aligned freelist pointer (older files: reindex) pub const HEADER_SIZE: usize = 4096; // Header field byte offsets. @@ -26,6 +26,13 @@ const H_CLEAN_SHUTDOWN: usize = 56; // u8 const H_MAX_NODES: usize = 64; // u64 const H_UPPER_HIGH_WATER: usize = 72; // u64 atomic: upper-entry allocator const H_UPPER_FREELIST: usize = 80; // u64 atomic: (tag<<32)|idx; NO_UPPER = empty +const H_ENTRY_PREV: usize = 88; // u64: last replaced entry point (re-election hint) +// Opener registry: each live handle claims one slot, writes its random tag there, and holds +// a kernel OFD byte-range lock on the slot (released automatically when the handle - or its +// whole process - dies). A lock word's owner is dead iff its registry slot no longer carries +// its tag or the slot's byte range is lockable. Immune to pid reuse and pid namespaces. +const H_REGISTRY: usize = 128; // u32 x REGISTRY_SLOTS +pub const REGISTRY_SLOTS: usize = 64; /// Upper-layer region geometry: fixed entries covering levels 1..=MAX_UPPER_LEVELS at /// UPPER_CAP ids per level. P(level >= 1) = 1/M ~ 6.25%; the region reserves entries for @@ -56,6 +63,12 @@ pub const FLAG_DELETED: u8 = 2; pub const NO_ID: u32 = u32::MAX; pub struct PlaneFile { + /// Kept open for the lifetime of the mapping: the opener-registry OFD lock lives on it. + file: std::fs::File, + /// This handle's registry tag (low bits encode its registry slot). 0 = unregistered + /// (registry full or platform without OFD locks): this handle's own dead locks cannot be + /// reclaimed by others, and it never reclaims. + pub self_tag: u32, pub map: MmapMut, pub dims: usize, pub layer0_cap: usize, @@ -107,6 +120,9 @@ fn slots_per_page_for(slot_size: usize) -> usize { impl PlaneFile { /// Create a new plane file with capacity for `max_nodes` (sparse; pages materialize on write). pub fn create(path: &Path, dims: usize, layer0_cap: usize, max_nodes: u64) -> io::Result { + if max_nodes >= NO_ID as u64 { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "maxNodes must be below 2^32-1")); + } let slot_size = slot_size_for(dims, layer0_cap); let slots_per_page = slots_per_page_for(slot_size); let data_len = slot_region_len(max_nodes, slot_size, slots_per_page); @@ -132,7 +148,21 @@ impl PlaneFile { std::sync::atomic::fence(Ordering::Release); map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; - Ok(PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page, opened_clean: true }) + let mut plane = PlaneFile { + file, + self_tag: 0, + map, + dims, + layer0_cap, + slot_size, + max_nodes, + upper_offset, + upper_capacity, + slots_per_page, + opened_clean: true, + }; + plane.register_opener(); + Ok(plane) } pub fn open(path: &Path) -> io::Result { @@ -175,7 +205,20 @@ impl PlaneFile { )); } let opened_clean = map[H_CLEAN_SHUTDOWN] == 1; - let plane = PlaneFile { map, dims, layer0_cap, slot_size, max_nodes, upper_offset, upper_capacity, slots_per_page, opened_clean }; + let mut plane = PlaneFile { + file, + self_tag: 0, + map, + dims, + layer0_cap, + slot_size, + max_nodes, + upper_offset, + upper_capacity, + slots_per_page, + opened_clean, + }; + plane.register_opener(); let hw = plane.id_high_water(); if hw > max_nodes { return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header id high-water exceeds capacity: recreate the index")); @@ -233,10 +276,9 @@ impl PlaneFile { } return new as u32; } - // next-pointer lives in the dead slot's first neighbor word - let next = unsafe { - (*(self.slot_ptr(id).add(S_VECTOR + self.dims) as *const AtomicU32)).load(Ordering::Acquire) - }; + // next-pointer lives in the dead slot's scale field: offset 8, aligned for any + // dims (the first neighbor word at S_VECTOR+dims is 4-aligned only when dims%4==0) + let next = unsafe { (*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32)).load(Ordering::Acquire) }; let tag = (cur >> 32).wrapping_add(1); let new = (next as u64) | (tag << 32); if head.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire).is_ok() { @@ -249,7 +291,7 @@ impl PlaneFile { /// deleted (under its seqlock) so concurrent traversals skip it. pub fn free_id(&self, id: u32) { let head = self.header_atomic_u64(H_FREELIST_HEAD); - let next_word = unsafe { &*(self.slot_ptr(id).add(S_VECTOR + self.dims) as *const AtomicU32) }; + let next_word = unsafe { &*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32) }; loop { let cur = head.load(Ordering::Acquire); next_word.store((cur & 0xffff_ffff) as u32, Ordering::Release); @@ -287,7 +329,10 @@ impl PlaneFile { } pub fn set_entry_point(&self, id: u32, level: u32) { - self.header_atomic_u64(H_ENTRY).store((id as u64) | ((level as u64) << 32), Ordering::Release); + let prev = self.header_atomic_u64(H_ENTRY).swap((id as u64) | ((level as u64) << 32), Ordering::AcqRel); + if (prev & 0xffff_ffff) as u32 != NO_ID && (prev & 0xffff_ffff) as u32 != id { + self.header_atomic_u64(H_ENTRY_PREV).store(prev, Ordering::Release); + } } /// Entry-point CAS for re-election: install (id, level) only while the current entry is @@ -384,6 +429,80 @@ impl PlaneFile { } } + #[inline] + fn registry_tag_cell(&self, slot: usize) -> &AtomicU32 { + unsafe { &*(self.map.as_ptr().add(H_REGISTRY + slot * 4) as *const AtomicU32) } + } + + /// Claim a registry slot for this handle: take the slot's kernel byte-range lock (held + /// until this handle closes; released by the kernel if the process dies) and publish a + /// random tag whose low bits name the slot. On platforms without OFD locks, or with the + /// registry full, the handle stays unregistered (tag 0): it still works, but its own + /// abandoned locks are unreclaimable and it never reclaims others'. + fn register_opener(&mut self) { + #[cfg(target_os = "linux")] + for slot in 0..REGISTRY_SLOTS { + if !self.try_lock_registry_slot(slot, false) { + continue; + } + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.subsec_nanos()) + .unwrap_or(0); + let entropy = (nanos ^ std::process::id().rotate_left(16) ^ (self as *const _ as u32)) & crate::seqlock::GEN_MASK; + let tag = ((entropy | 1) & !(REGISTRY_SLOTS as u32 - 1)) | slot as u32; + let tag = if tag == 0 { REGISTRY_SLOTS as u32 | 1 << 30 | slot as u32 } else { tag }; + self.registry_tag_cell(slot).store(tag, Ordering::Release); + self.self_tag = tag; + return; + } + } + + /// Try to take the OFD write lock on a registry slot's byte range. `probe` releases it + /// immediately (liveness check); otherwise it is held for this handle's lifetime. + #[cfg(target_os = "linux")] + fn try_lock_registry_slot(&self, slot: usize, probe: bool) -> bool { + use std::os::unix::io::AsRawFd; + let mut fl: libc::flock = unsafe { std::mem::zeroed() }; + fl.l_type = libc::F_WRLCK as libc::c_short; + fl.l_whence = libc::SEEK_SET as libc::c_short; + fl.l_start = (H_REGISTRY + slot * 4) as libc::off_t; + fl.l_len = 4; + let got = unsafe { libc::fcntl(self.file.as_raw_fd(), libc::F_OFD_SETLK, &fl) } == 0; + if got && probe { + fl.l_type = libc::F_UNLCK as libc::c_short; + unsafe { libc::fcntl(self.file.as_raw_fd(), libc::F_OFD_SETLK, &fl) }; + } + got + } + + /// Whether the handle that minted `tag` is gone. True only with positive evidence: the + /// registry slot no longer carries the tag, or the slot's kernel lock is acquirable + /// (its holder's open handle is closed — process death included). This handle's own tag + /// is always alive (a thread of this process holds that lock; never rob it). + pub fn tag_is_dead(&self, tag: u32) -> bool { + if tag == 0 || tag == self.self_tag { + return false; + } + let slot = (tag as usize) & (REGISTRY_SLOTS - 1); + if self.registry_tag_cell(slot).load(Ordering::Acquire) != tag { + return true; // registration replaced or cleared: the minting handle is gone + } + #[cfg(target_os = "linux")] + { + self.try_lock_registry_slot(slot, true) + } + #[cfg(not(target_os = "linux"))] + { + false + } + } + + /// The re-election hint: the entry point most recently replaced by a promotion. + pub fn previous_entry_point(&self) -> u32 { + (self.header_atomic_u64(H_ENTRY_PREV).load(Ordering::Acquire) & 0xffff_ffff) as u32 + } + pub fn set_clean_shutdown(&mut self, clean: bool) { self.map[H_CLEAN_SHUTDOWN] = clean as u8; } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 0cead617eb..02f678617d 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -35,9 +35,14 @@ impl Graph { } /// Sanitizer for a slot lock taken over from a dead writer: the payload is half-written, - /// so the slot must read as absent until something rewrites it (heal-on-touch contract). + /// so the slot must read as deleted until something rewrites it (heal-on-touch contract; + /// FLAG_DELETED rather than 0 so hosts can still free/reuse the id). fn slot_sanitizer(&self, id: u32) -> impl Fn() + '_ { - move || unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = 0 } + move || unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED } + } + + fn owner_dead(&self) -> impl Fn(u32) -> bool + '_ { + move |tag| self.file.tag_is_dead(tag) } /// Sanitizer for an upper-entry lock taken over from a dead writer. @@ -52,7 +57,7 @@ impl Graph { return None; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { let flags = *p.add(S_FLAGS); @@ -63,7 +68,7 @@ impl Graph { let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); Some(cosine_int8_raw(query, p.add(S_VECTOR) as *const i8, scale, inv_mag)) } - }, self.slot_sanitizer(id), || None) + }, self.slot_sanitizer(id), || None, self.owner_dead()) } /// Symmetric stored-to-stored distance (construction-time neighbor↔neighbor checks). @@ -108,7 +113,7 @@ impl Graph { let seq = self.file.seq_atomic(id); let cap = self.file.layer0_cap; let dims = self.file.dims; - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { out.clear(); let p = self.file.slot_ptr(id); unsafe { @@ -124,7 +129,7 @@ impl Graph { } Some(level) } - }, self.slot_sanitizer(id), || None) + }, self.slot_sanitizer(id), || None, self.owner_dead()) } /// The node's upper-region entry index, or NO_UPPER. @@ -134,7 +139,7 @@ impl Graph { return NO_UPPER; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { let flags = *p.add(S_FLAGS); @@ -143,7 +148,7 @@ impl Graph { } (p.add(S_UPPER_IDX) as *const u32).read_unaligned() } - }, self.slot_sanitizer(id), || NO_UPPER) + }, self.slot_sanitizer(id), || NO_UPPER, self.owner_dead()) } /// Copy `id`'s neighbor ids at upper `level` (1-based) into `out`. False when the node @@ -156,7 +161,7 @@ impl Graph { return false; } let seq = self.file.upper_seq_atomic(idx); - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { out.clear(); let p = self.file.upper_ptr(idx); unsafe { @@ -172,7 +177,7 @@ impl Graph { } true } - }, self.upper_sanitizer(idx), || false) + }, self.upper_sanitizer(idx), || false, self.owner_dead()) } /// Write a node's full upper adjacency into a fresh region entry; returns the entry @@ -186,7 +191,7 @@ impl Graph { return NO_UPPER; } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -209,7 +214,7 @@ impl Graph { /// per rewrite. pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) { let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -233,7 +238,7 @@ impl Graph { return false; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }, self.slot_sanitizer(id), || true) + seqlock::read_consistent(seq, self.file.self_tag, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }, self.slot_sanitizer(id), || true, self.owner_dead()) } /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring @@ -243,7 +248,7 @@ impl Graph { return NO_UPPER; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { if *p.add(S_FLAGS) == 0 { @@ -251,7 +256,7 @@ impl Graph { } (p.add(S_UPPER_IDX) as *const u32).read_unaligned() } - }, self.slot_sanitizer(id), || NO_UPPER) + }, self.slot_sanitizer(id), || NO_UPPER, self.owner_dead()) } /// Mirror a host-maintained node into the plane: full state per call, host-allocated id @@ -293,7 +298,7 @@ impl Graph { // snapshot would resurrect the node when its cursor reaches this id self.file.ensure_high_water(id); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; } @@ -305,7 +310,7 @@ impl Graph { return false; } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.upper_sanitizer(idx)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); let p = self.file.upper_ptr_mut(idx); unsafe { let levels = *p.add(U_LEVELS); @@ -334,7 +339,7 @@ impl Graph { let seq = self.file.seq_atomic(id); let dims = self.file.dims; let cap = self.file.layer0_cap; - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { let flags = *p.add(S_FLAGS); @@ -350,7 +355,7 @@ impl Graph { let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(nbase.add(i).read_unaligned())).collect(); Some(NodeRead { level, scale, inv_mag, vector, neighbors }) } - }, self.slot_sanitizer(id), || None) + }, self.slot_sanitizer(id), || None, self.owner_dead()) } /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the @@ -359,7 +364,7 @@ impl Graph { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -386,7 +391,7 @@ impl Graph { return false; } let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; let cap = self.file.layer0_cap; @@ -408,11 +413,46 @@ impl Graph { true } + /// Apply a precomputed neighbor list only if the current list still equals `expected` — + /// the compare and the write share one lock acquisition, so heavy work (distance-based + /// pruning, which can major-fault) happens OUTSIDE the lock and the critical section + /// stays microseconds. Returns false when the list changed or the node is gone. + pub fn set_neighbors_if(&self, id: u32, expected: &[u32], next: &[u32]) -> bool { + debug_assert!(next.len() <= self.file.layer0_cap); + if !self.in_range(id) { + return false; + } + let seq = self.file.seq_atomic(id); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let p = self.file.slot_ptr_mut(id); + let dims = self.file.dims; + unsafe { + if *p.add(S_FLAGS) != FLAG_VALID { + return false; + } + let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; + if degree != expected.len() { + return false; + } + let base = p.add(S_VECTOR + dims) as *mut u32; + for (i, want) in expected.iter().enumerate() { + if u32::from_le(base.add(i).read_unaligned()) != *want { + return false; + } + } + (p.add(S_DEGREE) as *mut u16).write_unaligned((next.len() as u16).to_le()); + for (i, n) in next.iter().enumerate() { + base.add(i).write_unaligned(n.to_le()); + } + } + true + } + /// Replace only the neighbor list (single-writer construction path). pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { debug_assert!(neighbors.len() <= self.file.layer0_cap); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -440,7 +480,7 @@ impl Graph { let upper_idx; { let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); let p = self.file.slot_ptr_mut(id); unsafe { if *p.add(S_FLAGS) != FLAG_VALID { @@ -476,7 +516,7 @@ impl Graph { return None; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, || { + seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { if *p.add(S_FLAGS) != FLAG_VALID { @@ -484,7 +524,7 @@ impl Graph { } Some(*p.add(S_LEVEL)) } - }, self.slot_sanitizer(id), || None) + }, self.slot_sanitizer(id), || None, self.owner_dead()) } /// Pick a new entry point: the highest-level live node among `preferred`, else the @@ -499,6 +539,14 @@ impl Graph { fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { let mut best: Option<(u32, u8)> = None; + // the most recently replaced entry point is the best cheap candidate: usually alive, + // usually high-level — and it makes the full fallback scan a last resort + let prev = self.file.previous_entry_point(); + if prev != crate::format::NO_ID && prev != replacing { + if let Some(level) = self.node_level(prev) { + best = Some((prev, level)); + } + } for &cand in preferred { if let Some(level) = self.node_level(cand) { if best.map(|(_, l)| level > l).unwrap_or(true) { @@ -548,7 +596,7 @@ impl Graph { let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels) }; let seq = self.file.seq_atomic(id); let written = { - let _guard = seqlock::write_lock(seq, self.slot_sanitizer(id)); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 50d03168c4..ccc3546f89 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -93,17 +93,34 @@ fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize *list = scored.into_iter().map(|(cand, _)| cand).collect(); } -/// Add `new_id` to `nid`'s adjacency at `level`, coverage-pruning to `cap` when over. +/// Add `new_id` to `nid`'s adjacency at `level`, coverage-pruning to `cap` when over. The +/// prune's distance computations (which can major-fault on a cold mapping) run OUTSIDE the +/// slot lock: the list is snapshotted, pruned, and applied with a compare-and-set; after a +/// bounded retry the fallback merges under the lock with a cheap truncation instead. fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) { if level == 0 { - graph.update_neighbors(nid, |list| { - if list.contains(&new_id) { + for _ in 0..2 { + let mut snapshot: Vec = Vec::new(); + if graph.neighbors_into(nid, &mut snapshot).is_none() { return; } - list.push(new_id); - if list.len() > cap { - // distance_between reads other slots without locks; safe under this seqlock - prune_with_coverage(graph, nid, list, cap); + if snapshot.contains(&new_id) { + return; + } + let mut next = snapshot.clone(); + next.push(new_id); + if next.len() > cap { + prune_with_coverage(graph, nid, &mut next, cap); + } + if graph.set_neighbors_if(nid, &snapshot, &next) { + return; + } + } + // contended twice: merge cheaply under the lock (bounded critical section) + graph.update_neighbors(nid, |list| { + if !list.contains(&new_id) { + list.push(new_id); + list.truncate(cap); } }); } else { @@ -135,7 +152,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu if entry_id == NO_ID { let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); - graph.file.set_entry_point(id, level as u32); + // CAS: a concurrent first insert may have installed an entry already — never clobber + graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); return Some(id); } @@ -154,7 +172,7 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu None => { let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); - graph.file.set_entry_point(id, level as u32); + graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); return Some(id); } } @@ -247,7 +265,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu } if (level as u32) > entry_level { - graph.file.set_entry_point(id, level as u32); + // CAS against the observed entry: a concurrent higher-level promotion wins + graph.file.set_entry_point_if_not_better(id, level as u32, entry_id); } Some(id) } diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index fac1d120f5..fc100a3b41 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -185,6 +185,13 @@ impl Plane { self.graph.file.dims ))); } + for (i, v) in vector.iter().enumerate() { + if !v.is_finite() { + // a NaN component yields a huge invMag and -inf distances: that node would + // rank first for roughly half of all queries, permanently + return Err(Error::from_reason(format!("vector component {i} is not finite"))); + } + } let mut scratch = self.insert_scratch.lock().unwrap(); insert(&self.graph, &vector, &self.params, &mut scratch) .ok_or_else(|| Error::from_reason("plane is full (maxNodes reached)")) @@ -343,7 +350,8 @@ impl Plane { /// Set the graph entry point (dual-write mode mirrors the host's entry-point updates). #[napi] pub fn set_entry_point(&self, id: u32, level: u32) { - self.graph.file.set_entry_point(id, level); + // clamp: a garbage level would make every search iterate that many empty levels + self.graph.file.set_entry_point(id, level.min(crate::format::MAX_UPPER_LEVELS as u32)); } #[napi] diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index e6f5d23e06..2630879d75 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -156,6 +156,9 @@ pub fn search_layer( } for i in 0..nbuf.len() { let nid = nbuf[i]; + if (nid as u64) >= graph.file.max_nodes { + continue; // corrupt/torn neighbor id: skip rather than size allocations by it + } if !scratch.visit(nid) { continue; } @@ -374,6 +377,9 @@ pub fn search_predicated( } for i in 0..nbuf.len() { let nid = nbuf[i]; + if (nid as u64) >= graph.file.max_nodes { + continue; // corrupt/torn neighbor id: skip rather than size allocations by it + } if !scratch.visit(nid) { continue; } diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index ee3c3721a3..f831bad59e 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -1,34 +1,27 @@ //! Per-slot lock with owner identity. The lock word is a u32: bit 31 set = locked, low 31 -//! bits = the owner's process id (Linux pid_max ≤ 2^22; macOS far lower). Unlocked values -//! are generations (bit 31 clear) that change on every release, so readers validate a -//! consistent snapshot exactly like a classic seqlock. +//! bits = the owner handle's registry tag (see format.rs); unlocked values are generations +//! (bit 31 clear) that change on every release, so readers validate a consistent snapshot +//! seqlock-style. //! -//! Crash recovery: a lock persisted by a writer that died mid-write would wedge the slot -//! forever. A waiter that has watched the SAME locked value for a full window asks the OS -//! whether the owner is alive — takeover happens only when the pid is gone (ESRCH), never -//! on elapsed time alone, so a live writer descheduled by CFS throttling, a page-fault -//! storm, or oversubscription is never robbed (its critical section is microseconds; it -//! finishes when rescheduled). The taker SANITIZES the slot (caller-supplied closure marks -//! it invalid) before releasing: a dead writer's payload is half-written, and publishing it -//! as consistent would serve a spliced vector — invisible-until-rewritten is the contract. -//! On non-unix platforms liveness is unknowable here, so no takeover happens: readers skip -//! the slot after the window (degraded, safe) and writers keep waiting. +//! Crash recovery happens at the contended slot: a waiter that has watched the SAME locked +//! value for a full window asks `owner_dead(tag)` — implemented over kernel-owned file +//! locks that die with the owner's open handle, so it is immune to pid reuse, container +//! pid-1 restarts, and pid namespaces. Only a provably dead owner is taken over, and the +//! taker first runs `sanitize` (marking the payload deleted): a dead writer's payload is +//! half-written and must read as absent until rewritten. When liveness is unknowable +//! (non-Linux platforms, an unregistered handle), readers return `fallback()` after the +//! window instead of waiting forever, and writers keep waiting. use std::sync::atomic::{AtomicU32, Ordering}; use std::time::{Duration, Instant}; -const LOCKED: u32 = 1 << 31; -const GEN_MASK: u32 = LOCKED - 1; +pub const LOCKED: u32 = 1 << 31; +pub const GEN_MASK: u32 = LOCKED - 1; /// How long a locked value must stay unchanged before the owner's liveness is checked. const TAKEOVER_AFTER: Duration = Duration::from_millis(20); const SPINS_BEFORE_CLOCK: u32 = 1 << 10; -#[inline] -fn self_pid() -> u32 { - std::process::id() & GEN_MASK -} - /// A fresh generation for a takeover release: the previous generation is unknowable, so it /// must be a value no in-flight reader plausibly holds as its first snapshot. #[inline] @@ -37,21 +30,7 @@ fn fresh_generation() -> u32 { .duration_since(std::time::UNIX_EPOCH) .map(|d| d.subsec_nanos()) .unwrap_or(0); - (nanos ^ (self_pid() << 10)) & GEN_MASK -} - -#[cfg(unix)] -fn owner_is_dead(pid: u32) -> bool { - // ESRCH = no such process. EPERM means it exists but is not ours: alive. - if unsafe { libc::kill(pid as libc::pid_t, 0) } == 0 { - return false; - } - std::io::Error::last_os_error().raw_os_error() == Some(libc::ESRCH) -} - -#[cfg(not(unix))] -fn owner_is_dead(_pid: u32) -> bool { - false // unknowable here: never take over (readers skip, writers wait) + (nanos ^ (std::process::id() << 10)) & GEN_MASK } pub struct SeqWriteGuard<'a> { @@ -82,7 +61,7 @@ impl StaleWatch { StaleWatch { seen: 0, since: Instant::now() } } - fn observe(&mut self, locked_value: u32) -> Stale { + fn observe(&mut self, locked_value: u32, owner_dead: &impl Fn(u32) -> bool) -> Stale { if self.seen != locked_value { self.seen = locked_value; self.since = Instant::now(); @@ -91,8 +70,7 @@ impl StaleWatch { if self.since.elapsed() < TAKEOVER_AFTER { return Stale::No; } - let pid = locked_value & GEN_MASK; - if owner_is_dead(pid) { + if owner_dead(locked_value & GEN_MASK) { Stale::DeadOwner(locked_value) } else { Stale::UnknownPastWindow @@ -100,17 +78,23 @@ impl StaleWatch { } } -/// Acquire write ownership of a slot. `sanitize` runs (holding the lock) only when the lock -/// was taken over from a dead owner — it must mark the protected payload invalid, because a -/// dead writer left it half-written. -pub fn write_lock<'a>(seq: &'a AtomicU32, sanitize: impl Fn()) -> SeqWriteGuard<'a> { +/// Acquire write ownership of a slot. `self_tag` identifies this handle in the lock word; +/// `sanitize` runs (holding the lock) only after a takeover from a dead owner; `owner_dead` +/// decides takeover eligibility. A lock held past the window by an owner that is alive or +/// unknowable is simply waited on. +pub fn write_lock<'a>( + seq: &'a AtomicU32, + self_tag: u32, + sanitize: impl Fn(), + owner_dead: impl Fn(u32) -> bool, +) -> SeqWriteGuard<'a> { let mut spins = 0u32; let mut watch = StaleWatch::new(); loop { let cur = seq.load(Ordering::Acquire); if cur & LOCKED == 0 { if seq - .compare_exchange_weak(cur, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange_weak(cur, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) .is_ok() { return SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }; @@ -118,9 +102,9 @@ pub fn write_lock<'a>(seq: &'a AtomicU32, sanitize: impl Fn()) -> SeqWriteGuard< } else { spins += 1; if spins > SPINS_BEFORE_CLOCK { - if let Stale::DeadOwner(observed) = watch.observe(cur) { + if let Stale::DeadOwner(observed) = watch.observe(cur, &owner_dead) { if seq - .compare_exchange(observed, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) .is_ok() { sanitize(); @@ -136,16 +120,17 @@ pub fn write_lock<'a>(seq: &'a AtomicU32, sanitize: impl Fn()) -> SeqWriteGuard< } /// Run `read` until it observes a stable (unlocked, unchanged) generation. `read` must be -/// side-effect-free on retry. A lock held by a dead owner is taken over and the payload -/// sanitized (marked invalid) before this thread re-reads; on platforms where liveness is -/// unknowable, a lock past the window makes this return `fallback()` instead of waiting -/// forever. +/// side-effect-free on retry. A dead owner's lock is taken over (sanitizing the payload) +/// and the read retried; an alive-or-unknowable owner past the window makes this return +/// `fallback()` rather than stall a search indefinitely. #[inline] pub fn read_consistent( seq: &AtomicU32, + self_tag: u32, mut read: impl FnMut() -> T, sanitize: impl Fn(), fallback: impl FnOnce() -> T, + owner_dead: impl Fn(u32) -> bool, ) -> T { let mut spins = 0u32; let mut watch = StaleWatch::new(); @@ -160,10 +145,10 @@ pub fn read_consistent( } else { spins += 1; if spins > SPINS_BEFORE_CLOCK { - match watch.observe(before) { + match watch.observe(before, &owner_dead) { Stale::DeadOwner(observed) => { if seq - .compare_exchange(observed, LOCKED | self_pid(), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) .is_ok() { sanitize(); diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 392b190625..89eddd3ee5 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -28,9 +28,10 @@ fn dead_writer_lock_is_taken_over_and_slot_sanitized() { for i in 0..200 { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } - // simulate a writer killed mid-write: lock word = bit31 | its (now dead) pid. - // 0x7ff_fff0 exceeds Linux pid_max (4M) and any real pid space: kill() -> ESRCH. - graph.file.seq_atomic(7).store((1 << 31) | 0x7ff_fff0, Ordering::SeqCst); + // simulate a writer killed mid-write: lock word = bit31 | a tag whose registry slot + // carries no matching registration (the fabricated tag differs from any live tag, + // so tag_is_dead reports it dead immediately) + graph.file.seq_atomic(7).store((1 << 31) | 0x1234_5678 & 0x7fff_ffff, Ordering::SeqCst); graph.file.msync().unwrap(); } let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); @@ -68,10 +69,15 @@ fn live_writer_is_never_robbed() { let g2 = graph.clone(); let hold = std::thread::spawn(move || { let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; - let guard = hnsw_plane::seqlock::write_lock(seq, || panic!("a live same-process writer must never be sanitized")); + let g2 = &g2; + let guard = hnsw_plane::seqlock::write_lock( + seq, + g2.file.self_tag, + || panic!("a live same-process writer must never be sanitized"), + |tag| g2.file.tag_is_dead(tag), + ); std::thread::sleep(std::time::Duration::from_millis(120)); drop(guard); - drop(g2); }); std::thread::sleep(std::time::Duration::from_millis(30)); // reader arrives mid-hold let n9 = graph.read_node(9); @@ -170,3 +176,74 @@ fn full_plane_refuses_inserts_instead_of_corrupting() { assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_some()); let _ = std::fs::remove_file(&path); } + +#[test] +fn flush_without_watermark_preserves_a_completion_stamp() { + let dims = 32; + let path = tmp("flushnone"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create")); + graph.file.set_watermark(7); + graph.file.flush_with_watermark(None).unwrap(); + assert_eq!(graph.file.watermark(), 7, "a watermark-less barrier must not touch the stamp"); + graph.file.flush_with_watermark(Some(9)).unwrap(); + assert_eq!(graph.file.watermark(), 9); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn odd_dims_freelist_reuse_is_aligned() { + // dims 25: the old freelist next-pointer at S_VECTOR+dims was unaligned (SIGBUS on + // aarch64); it now lives at the dead slot's aligned scale field + let dims = 25; + let path = tmp("odddims"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..20 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + graph.delete_node(4); + graph.delete_node(9); + let a = insert(&graph, &vector_for(50, dims), ¶ms, &mut scratch).unwrap(); + let b = insert(&graph, &vector_for(51, dims), ¶ms, &mut scratch).unwrap(); + assert!(a == 9 || a == 4); + assert!(b == 9 || b == 4); + assert_ne!(a, b); + let _ = std::fs::remove_file(&path); +} + +#[cfg(target_os = "linux")] +#[test] +fn same_pid_restart_takeover_via_registry() { + // The container-pid-1 scenario: the process that died and the process that reopens have + // the SAME pid, so pid-based liveness would wedge forever. Registry liveness is keyed to + // the open handle (kernel lock dies with it), which a same-process reopen reproduces + // faithfully: drop the old handle, reopen, and the old tag must be reclaimable. + let dims = 32; + let path = tmp("samepid"); + let _ = std::fs::remove_file(&path); + let dead_tag; + { + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..100 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + dead_tag = graph.file.self_tag; + assert_ne!(dead_tag, 0, "linux handles must register"); + // die mid-write: lock word carries OUR tag, then the handle drops (kernel releases + // the registry lock exactly as process death would) + graph.file.seq_atomic(11).store((1 << 31) | dead_tag, Ordering::SeqCst); + graph.file.msync().unwrap(); + } + let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); + assert_ne!(graph.file.self_tag, dead_tag, "a new handle mints a new tag"); + let start = std::time::Instant::now(); + assert!(graph.read_node(11).is_none(), "taken-over slot reads deleted (sanitized), not spliced"); + assert!(start.elapsed() < std::time::Duration::from_secs(5)); + assert_eq!(graph.file.seq_atomic(11).load(Ordering::SeqCst) >> 31, 0, "lock reclaimed"); + let _ = std::fs::remove_file(&path); +} From 275690d760581c3eda7c65f5be97836b7e670852 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:19:21 -0600 Subject: [PATCH 31/69] =?UTF-8?q?hnsw-plane:=20bounded=20writer=20wedge=20?= =?UTF-8?q?(round-5=20blocker)=20=E2=80=94=20unreclaimable=20locks=20surfa?= =?UTF-8?q?ce=20as=20errors?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An abandoned lock whose owner tag is unregistered (tag 0: non-Linux, or a full 64-slot registry) or whose holder is alive-but-stuck could hang write_lock forever, silently. write_lock now returns Err(Wedged) after a hard 5s of one unchanged unreclaimable locked value; the error threads through every graph write API to NAPI, where Harper's existing mirror error handling disables and rebuilds the plane - a logged, self-healing degradation instead of a permanently hung thread. Readers already degraded via the window fallback. smoke.mjs temp paths made portable (Windows has no /tmp; with that fixed the win32 prebuild builds AND passes smoke, so it joins the matrix). Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/smoke.mjs | 12 ++-- native/hnsw-plane/src/graph.rs | 95 ++++++++++++++------------- native/hnsw-plane/src/insert.rs | 20 +++--- native/hnsw-plane/src/napi.rs | 27 ++++---- native/hnsw-plane/src/seqlock.rs | 41 +++++++++--- native/hnsw-plane/tests/concurrent.rs | 4 +- native/hnsw-plane/tests/reopen.rs | 16 ++--- 7 files changed, 122 insertions(+), 93 deletions(-) diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs index 79e6eb3657..cf7aa13383 100644 --- a/native/hnsw-plane/smoke.mjs +++ b/native/hnsw-plane/smoke.mjs @@ -1,12 +1,12 @@ -// NAPI smoke test: build with `cargo build --release --features napi --lib`, then -// (the bench bin cannot link against unresolved node-api symbols; build the lib alone) -// cp target/release/libhnsw_plane.so hnsw-plane.node && node smoke.mjs +// End-to-end smoke test: `npm run build && node smoke.mjs` (also the CI path). import { createRequire } from 'module'; const require = createRequire(import.meta.url); -const { Plane } = require('./hnsw-plane.node'); +const { Plane } = require('./index.js'); const dims = 64; -const path = `/tmp/smoke-${process.pid}.hnsw`; +const { tmpdir } = await import('node:os'); +const { join } = await import('node:path'); +const path = join(tmpdir(), `smoke-${process.pid}.hnsw`); const plane = Plane.create(path, dims, 32, 10_000); function vec(i) { @@ -49,7 +49,7 @@ if (pred.length === 0) throw new Error('predicate search returned nothing'); console.log(`predicate top hit: id ${pred[0].id} (calls: ${predicateCalls})`); // raw mirroring path (dual-write phase 1): host-allocated ids, full node state per call -const mirror = Plane.create(`/tmp/smoke-mirror-${process.pid}.hnsw`, dims, 32, 10_000); +const mirror = Plane.create(join(tmpdir(), `smoke-mirror-${process.pid}.hnsw`), dims, 32, 10_000); const q42 = vec(42); // quantize like the host: scale maps max|c| to 127, invMag = 1/|v| function quant(v) { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 02f678617d..7d68bf2ae6 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -10,6 +10,7 @@ use crate::format::{ S_UPPER_IDX, S_VECTOR, UPPER_CAP, UPPER_LEVEL_STRIDE, U_LEVELS, U_LISTS, }; use crate::seqlock; +use crate::seqlock::Wedged; pub struct Graph { pub file: PlaneFile, @@ -182,16 +183,16 @@ impl Graph { /// Write a node's full upper adjacency into a fresh region entry; returns the entry /// index to store in the slot (NO_UPPER when the region is exhausted or levels is empty). - pub fn write_upper(&self, levels: &[Vec]) -> u32 { + pub fn write_upper(&self, levels: &[Vec]) -> Result { if levels.is_empty() { - return NO_UPPER; + return Ok(NO_UPPER); } let idx = self.file.allocate_upper(); if idx == NO_UPPER { - return NO_UPPER; + return Ok(NO_UPPER); } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -206,15 +207,15 @@ impl Graph { } } } - idx + Ok(idx) } /// Rewrite an existing upper entry in place (full state). Used by the raw mirroring /// path so repeated updates to a high-level node reuse its entry instead of leaking one /// per rewrite. - pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) { + pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) -> Result<(), Wedged> { let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; let p = self.file.upper_ptr_mut(idx); unsafe { let n = levels.len().min(MAX_UPPER_LEVELS); @@ -229,6 +230,7 @@ impl Graph { } } } + Ok(()) } /// Whether a slot has ever been written (valid or deleted) — the builder scan's @@ -271,51 +273,53 @@ impl Graph { inv_mag: f32, neighbors: &[u32], upper_levels: &[Vec], - ) { + ) -> Result<(), Wedged> { self.file.ensure_high_water(id); let existing = self.upper_idx_raw(id); let upper_idx = if upper_levels.is_empty() { existing // keep an existing entry bound (level never shrinks in practice) } else if existing != NO_UPPER { - self.rewrite_upper(existing, upper_levels); + self.rewrite_upper(existing, upper_levels)?; existing } else { - self.write_upper(upper_levels) + self.write_upper(upper_levels)? }; let mut l0 = neighbors.to_vec(); l0.truncate(self.file.layer0_cap); - self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx); + self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx)?; + Ok(()) } /// Mark deleted WITHOUT returning the id to the plane freelist — dual-write mode, where /// the host owns id allocation and may re-mint or reuse ids on its own schedule. - pub fn clear_node(&self, id: u32) { + pub fn clear_node(&self, id: u32) -> Result<(), Wedged> { if (id as u64) >= self.file.max_nodes { - return; + return Ok(()); } // extend the high-water rather than skipping: a delete mirrored while a backfill // scan runs must leave a touched (deleted) slot behind, or the scan's older // snapshot would resurrect the node when its cursor reaches this id self.file.ensure_high_water(id); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; + Ok(()) } /// Atomic read-modify-write of `id`'s upper adjacency at `level` (1-based). Returns /// false when the node has no entry or level. `f` may read other slots. - pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> bool { + pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> Result { let idx = self.upper_idx_of(id); if idx == NO_UPPER || level as usize > MAX_UPPER_LEVELS { - return false; + return Ok(false); } let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; let p = self.file.upper_ptr_mut(idx); unsafe { let levels = *p.add(U_LEVELS); if level > levels { - return false; + return Ok(false); } let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); let degree = u16::from_le((lp as *const u16).read_unaligned()) as usize; @@ -328,7 +332,7 @@ impl Graph { base.add(i).write_unaligned(id.to_le()); } } - true + Ok(true) } /// Seqlock-consistent full copy (construction paths). @@ -360,11 +364,11 @@ impl Graph { /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the /// caller; `upper_idx` is a write_upper() result (NO_UPPER for level-0 nodes). - pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32], upper_idx: u32) { + pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32], upper_idx: u32) -> Result<(), Wedged> { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -380,25 +384,26 @@ impl Graph { // valid last within the locked section; the seqlock release publishes it *p.add(S_FLAGS) = FLAG_VALID; } + Ok(()) } /// Atomic read-modify-write of a node's layer-0 neighbor list under its seqlock. /// `f` may read OTHER slots (e.g. distance_between for pruning) — those are plain /// unlocked reads, so no lock ordering issue — but must not lock this graph's slots. /// Returns false for absent/deleted nodes. - pub fn update_neighbors)>(&self, id: u32, f: F) -> bool { + pub fn update_neighbors)>(&self, id: u32, f: F) -> Result { if !self.in_range(id) { - return false; + return Ok(false); } let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; let cap = self.file.layer0_cap; unsafe { let flags = *p.add(S_FLAGS); if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return false; + return Ok(false); } let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; let base = p.add(S_VECTOR + dims) as *mut u32; @@ -410,34 +415,34 @@ impl Graph { base.add(i).write_unaligned(n.to_le()); } } - true + Ok(true) } /// Apply a precomputed neighbor list only if the current list still equals `expected` — /// the compare and the write share one lock acquisition, so heavy work (distance-based /// pruning, which can major-fault) happens OUTSIDE the lock and the critical section /// stays microseconds. Returns false when the list changed or the node is gone. - pub fn set_neighbors_if(&self, id: u32, expected: &[u32], next: &[u32]) -> bool { + pub fn set_neighbors_if(&self, id: u32, expected: &[u32], next: &[u32]) -> Result { debug_assert!(next.len() <= self.file.layer0_cap); if !self.in_range(id) { - return false; + return Ok(false); } let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { if *p.add(S_FLAGS) != FLAG_VALID { - return false; + return Ok(false); } let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; if degree != expected.len() { - return false; + return Ok(false); } let base = p.add(S_VECTOR + dims) as *mut u32; for (i, want) in expected.iter().enumerate() { if u32::from_le(base.add(i).read_unaligned()) != *want { - return false; + return Ok(false); } } (p.add(S_DEGREE) as *mut u16).write_unaligned((next.len() as u16).to_le()); @@ -445,14 +450,14 @@ impl Graph { base.add(i).write_unaligned(n.to_le()); } } - true + Ok(true) } /// Replace only the neighbor list (single-writer construction path). - pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) { + pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) -> Result<(), Wedged> { debug_assert!(neighbors.len() <= self.file.layer0_cap); let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -461,15 +466,16 @@ impl Graph { (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); } } + Ok(()) } /// Mark deleted (traversals skip it), free its upper entry, and return the id to the /// plane freelist. Deleting the current entry point re-elects a replacement — without /// that, every search returns empty and every insert orphans itself against the dead /// entry. - pub fn delete_node(&self, id: u32) { + pub fn delete_node(&self, id: u32) -> Result<(), Wedged> { if !self.in_range(id) { - return; // never-allocated or out-of-range ids have nothing to delete + return Ok(()); // never-allocated or out-of-range ids have nothing to delete } // capture neighbors before invalidating: they are the best re-election candidates let (entry_id, _) = self.file.entry_point(); @@ -480,14 +486,14 @@ impl Graph { let upper_idx; { let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); unsafe { if *p.add(S_FLAGS) != FLAG_VALID { // deleting a never-written or already-deleted id must not free again: // a double-push makes the freelist a self-cycle that hands the same id // to every subsequent allocation - return; + return Ok(()); } upper_idx = (p.add(S_UPPER_IDX) as *const u32).read_unaligned(); (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); @@ -498,13 +504,14 @@ impl Graph { // empty the entry under its own lock BEFORE freeing: a traversal that already // read this node's upper_idx must find a dead entry, not one reallocated to a // different node mid-read - self.rewrite_upper(upper_idx, &[]); + self.rewrite_upper(upper_idx, &[])?; } self.file.free_upper(upper_idx); if entry_id == id { self.reelect_entry_point_replacing(&candidates, id); } self.file.free_id(id); + Ok(()) } /// Pick a new entry point: the highest-level live node among `preferred`, else the @@ -587,16 +594,16 @@ impl Graph { inv_mag: f32, neighbors: &[u32], upper_levels: &[Vec], - ) -> bool { + ) -> Result { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); self.file.ensure_high_water(id); // the upper entry is allocated before taking the slot lock (allocation is cheap and // an unused entry is freed below on the untouched-check failing) - let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels) }; + let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels)? }; let seq = self.file.seq_atomic(id); let written = { - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()); + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { @@ -620,6 +627,6 @@ impl Graph { if !written { self.file.free_upper(upper_idx); } - written + Ok(written) } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index ccc3546f89..e761778266 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -40,7 +40,7 @@ fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { } }); } else { - graph.update_upper_level(from, level, |list| { + let _ = graph.update_upper_level(from, level, |list| { if let Some(pos) = list.iter().position(|&x| x == to) { list.remove(pos); } @@ -112,19 +112,19 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) if next.len() > cap { prune_with_coverage(graph, nid, &mut next, cap); } - if graph.set_neighbors_if(nid, &snapshot, &next) { + if graph.set_neighbors_if(nid, &snapshot, &next).unwrap_or(false) { return; } } // contended twice: merge cheaply under the lock (bounded critical section) - graph.update_neighbors(nid, |list| { + let _ = graph.update_neighbors(nid, |list| { if !list.contains(&new_id) { list.push(new_id); list.truncate(cap); } }); } else { - graph.update_upper_level(nid, level, |list| { + let _ = graph.update_upper_level(nid, level, |list| { if list.contains(&new_id) { return; } @@ -150,8 +150,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { - let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; - graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); + let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; + if graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).is_err() { return None; } // CAS: a concurrent first insert may have installed an entry already — never clobber graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); return Some(id); @@ -170,8 +170,8 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu match (re_id != NO_ID).then(|| graph.distance_to(re_id, &query)).flatten() { Some(d) => (re_id, re_level, d), None => { - let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]) } else { NO_UPPER }; - graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx); + let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; + if graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).is_err() { return None; } graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); return Some(id); } @@ -248,13 +248,13 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu .unwrap_or_default() }) .collect(); - graph.write_upper(&levels) + graph.write_upper(&levels).unwrap_or(NO_UPPER) } else { NO_UPPER }; let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); l0.truncate(layer0_cap); - graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx); + if graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx).is_err() { return None; } // Reverse edges. for (l, conns) in connections.iter().enumerate() { diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index fc100a3b41..d00c13fe6c 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -200,8 +200,10 @@ impl Plane { /// Delete a node; its id returns to the plane freelist. Standalone-allocation mode only /// (pairs with insert()); dual-write hosts use clearNode instead. #[napi] - pub fn remove(&self, id: u32) { - self.graph.delete_node(id); + pub fn remove(&self, id: u32) -> Result<()> { + self.graph + .delete_node(id) + .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) } /// Mirror a host-maintained node into the plane (dual-write phase 1): full node state @@ -259,16 +261,9 @@ impl Plane { } } } - self.graph.write_node_raw( - id, - level, - vec_i8, - scale as f32, - inv_mag as f32, - &neighbors.to_vec(), - &upper_levels, - ); - Ok(()) + self.graph + .write_node_raw(id, level, vec_i8, scale as f32, inv_mag as f32, &neighbors.to_vec(), &upper_levels) + .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) } /// Builder-scan variant of writeNodeRaw: writes ONLY when the slot has never been @@ -321,7 +316,9 @@ impl Plane { l0.truncate(self.graph.file.layer0_cap); // the untouched check and the write share one seqlock acquisition inside the crate: // a live mirror's newer write can never be overwritten by this scan's older snapshot - Ok(self.graph.write_node_if_untouched(id, level, vec_i8, scale as f32, inv_mag as f32, &l0, &upper_levels)) + self.graph + .write_node_if_untouched(id, level, vec_i8, scale as f32, inv_mag as f32, &l0, &upper_levels) + .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) } /// Advisory: whether the file recorded a durability barrier (flush) as its last state @@ -344,7 +341,9 @@ impl Plane { /// owns id allocation). #[napi] pub fn clear_node(&self, id: u32) { - self.graph.clear_node(id); + if self.graph.clear_node(id).is_err() { + // bounded lock wedge: surfaced via the next write; clear is best-effort + } } /// Set the graph entry point (dual-write mode mirrors the host's entry-point updates). diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index f831bad59e..eaeba792eb 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -20,6 +20,15 @@ pub const GEN_MASK: u32 = LOCKED - 1; /// How long a locked value must stay unchanged before the owner's liveness is checked. const TAKEOVER_AFTER: Duration = Duration::from_millis(20); +/// Hard bound on waiting for a lock this thread cannot reclaim (owner alive-or-unknowable: +/// an unregistered handle's abandoned lock, a deadlocked live thread). A live writer's +/// critical section is microseconds, so five seconds of one unchanged locked value means +/// the slot is wedged — surfacing an error beats hanging a caller forever. +const WRITE_WEDGE_AFTER: Duration = Duration::from_secs(5); + +/// The slot's lock could not be acquired or reclaimed within the wedge bound. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Wedged; const SPINS_BEFORE_CLOCK: u32 = 1 << 10; /// A fresh generation for a takeover release: the previous generation is unknowable, so it @@ -87,9 +96,10 @@ pub fn write_lock<'a>( self_tag: u32, sanitize: impl Fn(), owner_dead: impl Fn(u32) -> bool, -) -> SeqWriteGuard<'a> { +) -> Result, Wedged> { let mut spins = 0u32; let mut watch = StaleWatch::new(); + let mut wedged_since: Option = None; loop { let cur = seq.load(Ordering::Acquire); if cur & LOCKED == 0 { @@ -97,19 +107,32 @@ pub fn write_lock<'a>( .compare_exchange_weak(cur, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) .is_ok() { - return SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }; + return Ok(SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }); } + wedged_since = None; } else { spins += 1; if spins > SPINS_BEFORE_CLOCK { - if let Stale::DeadOwner(observed) = watch.observe(cur, &owner_dead) { - if seq - .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - sanitize(); - return SeqWriteGuard { seq, release_gen: fresh_generation() }; + match watch.observe(cur, &owner_dead) { + Stale::DeadOwner(observed) => { + if seq + .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + sanitize(); + return Ok(SeqWriteGuard { seq, release_gen: fresh_generation() }); + } + wedged_since = None; } + Stale::UnknownPastWindow => { + // unreclaimable (unregistered owner tag, or an alive-but-stuck + // holder): bounded wait, then surface the wedge instead of hanging + let since = *wedged_since.get_or_insert_with(Instant::now); + if since.elapsed() > WRITE_WEDGE_AFTER { + return Err(Wedged); + } + } + Stale::No => {} } std::thread::yield_now(); continue; diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs index f821b726f5..4f1ea6ae02 100644 --- a/native/hnsw-plane/tests/concurrent.rs +++ b/native/hnsw-plane/tests/concurrent.rs @@ -96,8 +96,8 @@ fn concurrent_insert_search() { } // Delete + reinsert reuses ids (freelist; the #2182 fix). - graph.delete_node(5); - graph.delete_node(6); + let _ = graph.delete_node(5); + let _ = graph.delete_node(6); let params = InsertParams::default(); let a = insert(&graph, &vector_for(90_001, dims), ¶ms, &mut scratch).unwrap(); let b = insert(&graph, &vector_for(90_002, dims), ¶ms, &mut scratch).unwrap(); diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 89eddd3ee5..916a457678 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -47,7 +47,7 @@ fn dead_writer_lock_is_taken_over_and_slot_sanitized() { let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); assert!(!hits.is_empty()); let q = hnsw_plane::distance::quantize_int8(&vector_for(7, dims)); - graph.write_node_raw(7, 0, &q.0, q.1, q.2, &[3, 4], &[]); + graph.write_node_raw(7, 0, &q.0, q.1, q.2, &[3, 4], &[]).unwrap(); assert!(graph.read_node(7).is_some(), "a rewrite heals the sanitized slot"); let _ = std::fs::remove_file(&path); } @@ -100,9 +100,9 @@ fn double_remove_does_not_cycle_the_freelist() { for i in 0..20 { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } - graph.delete_node(5); - graph.delete_node(5); // second delete must be a no-op, not a second freelist push - graph.delete_node(2_000_000); // out-of-range must be a no-op, not an OOB write + let _ = graph.delete_node(5); + let _ = graph.delete_node(5); // second delete must be a no-op, not a second freelist push + let _ = graph.delete_node(2_000_000); // out-of-range must be a no-op, not an OOB write let a = insert(&graph, &vector_for(101, dims), ¶ms, &mut scratch).unwrap(); let b = insert(&graph, &vector_for(102, dims), ¶ms, &mut scratch).unwrap(); let c = insert(&graph, &vector_for(103, dims), ¶ms, &mut scratch).unwrap(); @@ -124,7 +124,7 @@ fn deleting_the_entry_point_reelects_and_recovers() { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } let (entry, _) = graph.file.entry_point(); - graph.delete_node(entry); + let _ = graph.delete_node(entry); let (new_entry, _) = graph.file.entry_point(); assert_ne!(new_entry, entry, "a new entry point must be elected"); let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); @@ -172,7 +172,7 @@ fn full_plane_refuses_inserts_instead_of_corrupting() { } assert!(insert(&graph, &vector_for(9, dims), ¶ms, &mut scratch).is_none(), "insert past maxNodes must fail cleanly"); // freed capacity is usable again - graph.delete_node(3); + let _ = graph.delete_node(3); assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_some()); let _ = std::fs::remove_file(&path); } @@ -204,8 +204,8 @@ fn odd_dims_freelist_reuse_is_aligned() { for i in 0..20 { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } - graph.delete_node(4); - graph.delete_node(9); + let _ = graph.delete_node(4); + let _ = graph.delete_node(9); let a = insert(&graph, &vector_for(50, dims), ¶ms, &mut scratch).unwrap(); let b = insert(&graph, &vector_for(51, dims), ¶ms, &mut scratch).unwrap(); assert!(a == 9 || a == 4); From b8dd488cef8f731628fd74861c46f0216c37fca8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:31:00 -0600 Subject: [PATCH 32/69] =?UTF-8?q?hnsw-plane:=20round-6=20residuals=20?= =?UTF-8?q?=E2=80=94=20wedge=20timer=20resets=20on=20owner=20churn,=20clea?= =?UTF-8?q?rNode=20surfaces=20wedges,=20insert=20distinguishes=20Full=20fr?= =?UTF-8?q?om=20Wedged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A busy slot with cycling owners must never trip the 5s wedge bound (the timer now resets whenever the lock value moves); clearNode no longer swallows the wedge error (NAPI throws; Harper's mirror path disables and rebuilds); standalone insert returns InsertError::{Full,Wedged} so a wedge is not reported as a full plane. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/bin/bench.rs | 2 +- native/hnsw-plane/src/insert.rs | 31 +++++++++++++++++++-------- native/hnsw-plane/src/napi.rs | 16 ++++++++------ native/hnsw-plane/src/seqlock.rs | 4 +++- native/hnsw-plane/tests/concurrent.rs | 2 +- native/hnsw-plane/tests/reopen.rs | 6 +++--- 6 files changed, 40 insertions(+), 21 deletions(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index e9d6307559..86fd7c4c91 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -212,7 +212,7 @@ fn main() { let mut count = 0u64; while !stop.load(Ordering::Relaxed) { let v = corpus.row(&mut rng); - if insert(&graph, &v, ¶ms, &mut scratch).is_none() { + if insert(&graph, &v, ¶ms, &mut scratch).is_err() { break; // plane full } count += 1; diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index e761778266..bc54ded2c1 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -136,12 +136,25 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) } } -/// Insert a vector, returning its node id — or None when the plane is full (max_nodes). -pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mut SearchScratch) -> Option { +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InsertError { + /// max_nodes reached (freeing capacity makes inserts possible again) + Full, + /// a slot lock could not be acquired or reclaimed within the wedge bound + Wedged, +} + +/// Insert a vector, returning its node id. +pub fn insert( + graph: &Graph, + vector: &[f32], + params: &InsertParams, + scratch: &mut SearchScratch, +) -> Result { let (bytes, scale, inv_mag) = quantize_int8(vector); let id = graph.file.allocate_id(); if id == NO_ID { - return None; + return Err(InsertError::Full); } let level = level_for(id, params.ml); let query = Query::new(vector.to_vec()); @@ -151,10 +164,10 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; - if graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).is_err() { return None; } + graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).map_err(|_| InsertError::Wedged)?; // CAS: a concurrent first insert may have installed an entry already — never clobber graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); - return Some(id); + return Ok(id); } let mut stats = SearchStats { visits: 0 }; @@ -171,9 +184,9 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu Some(d) => (re_id, re_level, d), None => { let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; - if graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).is_err() { return None; } + graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).map_err(|_| InsertError::Wedged)?; graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); - return Some(id); + return Ok(id); } } } @@ -254,7 +267,7 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu }; let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); l0.truncate(layer0_cap); - if graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx).is_err() { return None; } + graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx).map_err(|_| InsertError::Wedged)?; // Reverse edges. for (l, conns) in connections.iter().enumerate() { @@ -268,7 +281,7 @@ pub fn insert(graph: &Graph, vector: &[f32], params: &InsertParams, scratch: &mu // CAS against the observed entry: a concurrent higher-level promotion wins graph.file.set_entry_point_if_not_better(id, level as u32, entry_id); } - Some(id) + Ok(id) } #[inline] diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index d00c13fe6c..1b45499353 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -193,8 +193,12 @@ impl Plane { } } let mut scratch = self.insert_scratch.lock().unwrap(); - insert(&self.graph, &vector, &self.params, &mut scratch) - .ok_or_else(|| Error::from_reason("plane is full (maxNodes reached)")) + insert(&self.graph, &vector, &self.params, &mut scratch).map_err(|e| match e { + crate::insert::InsertError::Full => Error::from_reason("plane is full (maxNodes reached)"), + crate::insert::InsertError::Wedged => { + Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index") + } + }) } /// Delete a node; its id returns to the plane freelist. Standalone-allocation mode only @@ -340,10 +344,10 @@ impl Plane { /// Mark a node deleted without touching the plane freelist (dual-write mode: the host /// owns id allocation). #[napi] - pub fn clear_node(&self, id: u32) { - if self.graph.clear_node(id).is_err() { - // bounded lock wedge: surfaced via the next write; clear is best-effort - } + pub fn clear_node(&self, id: u32) -> Result<()> { + self.graph + .clear_node(id) + .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) } /// Set the graph entry point (dual-write mode mirrors the host's entry-point updates). diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index eaeba792eb..e3a8bd3e46 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -132,7 +132,9 @@ pub fn write_lock<'a>( return Err(Wedged); } } - Stale::No => {} + // the lock VALUE moved: owners are cycling, i.e. real progress — a busy + // slot must never trip the wedge bound + Stale::No => wedged_since = None, } std::thread::yield_now(); continue; diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs index 4f1ea6ae02..57122de035 100644 --- a/native/hnsw-plane/tests/concurrent.rs +++ b/native/hnsw-plane/tests/concurrent.rs @@ -39,7 +39,7 @@ fn concurrent_insert_search() { let mut scratch = SearchScratch::new(); for i in 0..per_writer { let v = vector_for(w * per_writer + i, dims); - insert(&graph, &v, ¶ms, &mut scratch).expect("plane full"); + insert(&graph, &v, ¶ms, &mut scratch).expect("insert"); } }); } diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 916a457678..04dee74fe1 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -168,12 +168,12 @@ fn full_plane_refuses_inserts_instead_of_corrupting() { let params = InsertParams::default(); let mut scratch = SearchScratch::new(); for i in 0..8 { - assert!(insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).is_some()); + assert!(insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).is_ok()); } - assert!(insert(&graph, &vector_for(9, dims), ¶ms, &mut scratch).is_none(), "insert past maxNodes must fail cleanly"); + assert!(insert(&graph, &vector_for(9, dims), ¶ms, &mut scratch).is_err(), "insert past maxNodes must fail cleanly"); // freed capacity is usable again let _ = graph.delete_node(3); - assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_some()); + assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_ok()); let _ = std::fs::remove_file(&path); } From 2d2755aeeacc7f09ed35b8b93142bbafa2ff0ffe Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:40:51 -0600 Subject: [PATCH 33/69] =?UTF-8?q?hnsw-plane:=20round-7=20refinements=20?= =?UTF-8?q?=E2=80=94=20epoch=20identity,=20salted=20acquisitions,=20lazy?= =?UTF-8?q?=20watch,=20corrupt-index=20guards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lock identity is now slot(6) + per-open epoch(12), with a per-acquisition salt(13) above it: back-to-back writers from one handle change the observed value every acquisition (no false wedge under healthy churn), and a dead lock pointing at a since-re-occupied registry slot is recognized dead by the epoch mismatch — the container same-pid restart recovers by takeover again instead of the 5s error path. StaleWatch initializes its clock lazily (no Instant::now() on uncontended reads). File-sourced freelist heads and upper indices are range-checked in release builds before any pointer math (a corrupt chain drops to the high-water allocator instead of computing out-of-mapping addresses). Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 50 ++++++++++++++++++++++++-------- native/hnsw-plane/src/graph.rs | 6 ++-- native/hnsw-plane/src/seqlock.rs | 36 ++++++++++++++++++----- 3 files changed, 69 insertions(+), 23 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index b788399008..672b759eba 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -276,6 +276,12 @@ impl PlaneFile { } return new as u32; } + if (id as u64) >= self.max_nodes { + // corrupt freelist head (file-sourced): drop the chain rather than compute + // out-of-mapping pointers; capacity continues via the high-water + let _ = head.compare_exchange(cur, NO_ID as u64, Ordering::AcqRel, Ordering::Acquire); + continue; + } // next-pointer lives in the dead slot's scale field: offset 8, aligned for any // dims (the first neighbor word at S_VECTOR+dims is 4-aligned only when dims%4==0) let next = unsafe { (*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32)).load(Ordering::Acquire) }; @@ -388,6 +394,11 @@ impl PlaneFile { loop { let cur = head.load(Ordering::Acquire); let idx = (cur & 0xffff_ffff) as u32; + if idx != NO_UPPER && (idx as u64) >= self.upper_capacity { + // corrupt upper freelist head (file-sourced): drop the chain + let _ = head.compare_exchange(cur, NO_UPPER as u64, Ordering::AcqRel, Ordering::Acquire); + continue; + } if idx == NO_UPPER { let hw = self.header_atomic_u64(H_UPPER_HIGH_WATER); let new = hw.fetch_add(1, Ordering::AcqRel); @@ -449,9 +460,14 @@ impl PlaneFile { .duration_since(std::time::UNIX_EPOCH) .map(|d| d.subsec_nanos()) .unwrap_or(0); - let entropy = (nanos ^ std::process::id().rotate_left(16) ^ (self as *const _ as u32)) & crate::seqlock::GEN_MASK; - let tag = ((entropy | 1) & !(REGISTRY_SLOTS as u32 - 1)) | slot as u32; - let tag = if tag == 0 { REGISTRY_SLOTS as u32 | 1 << 30 | slot as u32 } else { tag }; + // identity = slot (low 6 bits) + a random per-open epoch (12 bits, nonzero): + // a dead lock value pointing at a since-re-occupied slot is recognized as dead + // by the epoch mismatch — the container same-pid restart lands exactly here + let mut epoch = (nanos ^ std::process::id().rotate_left(16) ^ (self as *const _ as u32)) & 0xfff; + if epoch == 0 { + epoch = 1; + } + let tag = (epoch << 6) | slot as u32; self.registry_tag_cell(slot).store(tag, Ordering::Release); self.self_tag = tag; return; @@ -476,17 +492,27 @@ impl PlaneFile { got } - /// Whether the handle that minted `tag` is gone. True only with positive evidence: the - /// registry slot no longer carries the tag, or the slot's kernel lock is acquirable - /// (its holder's open handle is closed — process death included). This handle's own tag - /// is always alive (a thread of this process holds that lock; never rob it). - pub fn tag_is_dead(&self, tag: u32) -> bool { - if tag == 0 || tag == self.self_tag { + /// Whether the handle behind a lock value is gone. Lock values carry a per-acquisition + /// salt in their upper bits, so ownership is keyed on the registry SLOT (low bits): the + /// owner is dead only with positive evidence — no registration in the slot, or the + /// slot's kernel lock acquirable (its holder's open handle closed; process death + /// included). Our own slot is always alive (probing our own OFD lock would succeed and + /// lie). A dead value pointing at a slot since re-occupied by a NEW live handle reads + /// alive; the bounded writer wedge covers that rare mis-attribution. + pub fn tag_is_dead(&self, lock_value: u32) -> bool { + let identity = lock_value & crate::seqlock::TAG_MASK; + if identity == 0 { + return false; // unregistered owner: unknowable + } + if self.self_tag != 0 && identity == self.self_tag { + // ourselves: probing our own OFD lock from the same description would succeed + // and lie, so self is answered structurally return false; } - let slot = (tag as usize) & (REGISTRY_SLOTS - 1); - if self.registry_tag_cell(slot).load(Ordering::Acquire) != tag { - return true; // registration replaced or cleared: the minting handle is gone + let slot = (identity as usize) & (REGISTRY_SLOTS - 1); + let registered = self.registry_tag_cell(slot).load(Ordering::Acquire); + if registered == 0 || registered != identity { + return true; // slot empty, or re-occupied by a different epoch: owner departed } #[cfg(target_os = "linux")] { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 7d68bf2ae6..2c96a7ff9b 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -158,7 +158,7 @@ impl Graph { out.clear(); debug_assert!(level >= 1); let idx = self.upper_idx_of(id); - if idx == NO_UPPER || level as usize > MAX_UPPER_LEVELS { + if idx == NO_UPPER || (idx as u64) >= self.file.upper_capacity || level as usize > MAX_UPPER_LEVELS { return false; } let seq = self.file.upper_seq_atomic(idx); @@ -310,7 +310,7 @@ impl Graph { /// false when the node has no entry or level. `f` may read other slots. pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> Result { let idx = self.upper_idx_of(id); - if idx == NO_UPPER || level as usize > MAX_UPPER_LEVELS { + if idx == NO_UPPER || (idx as u64) >= self.file.upper_capacity || level as usize > MAX_UPPER_LEVELS { return Ok(false); } let seq = self.file.upper_seq_atomic(idx); @@ -500,7 +500,7 @@ impl Graph { *p.add(S_FLAGS) = FLAG_DELETED; } } - if upper_idx != NO_UPPER { + if upper_idx != NO_UPPER && (upper_idx as u64) < self.file.upper_capacity { // empty the entry under its own lock BEFORE freeing: a traversal that already // read this node's upper_idx must find a dead entry, not one reallocated to a // different node mid-read diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index e3a8bd3e46..f6cca05a72 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -17,6 +17,26 @@ use std::time::{Duration, Instant}; pub const LOCKED: u32 = 1 << 31; pub const GEN_MASK: u32 = LOCKED - 1; +/// A lock value decomposes as: bit 31 LOCKED | salt(13 bits) | handle identity(19 bits). +/// The identity — registry slot (6 bits) + the handle's per-open epoch (12 bits) — is what +/// liveness is keyed on; the salt changes every acquisition so back-to-back writers from ONE +/// handle still change the observed value (a waiter that never sees an unlocked window must +/// still see progress, or healthy same-handle churn would trip the wedge bound). +pub const TAG_MASK: u32 = (1 << 19) - 1; + +#[inline] +fn acquisition_value(self_tag: u32) -> u32 { + use std::cell::Cell; + thread_local! { + static SALT: Cell = const { Cell::new(0) }; + } + let salt = SALT.with(|c| { + let v = c.get().wrapping_add(1); + c.set(v); + v + }); + LOCKED | (((salt << 19) | (self_tag & TAG_MASK)) & GEN_MASK) +} /// How long a locked value must stay unchanged before the owner's liveness is checked. const TAKEOVER_AFTER: Duration = Duration::from_millis(20); @@ -62,21 +82,21 @@ enum Stale { /// Track how long one locked value has been observed; decide staleness. struct StaleWatch { seen: u32, - since: Instant, + since: Option, } impl StaleWatch { fn new() -> Self { - StaleWatch { seen: 0, since: Instant::now() } + StaleWatch { seen: 0, since: None } } fn observe(&mut self, locked_value: u32, owner_dead: &impl Fn(u32) -> bool) -> Stale { - if self.seen != locked_value { + if self.seen != locked_value || self.since.is_none() { self.seen = locked_value; - self.since = Instant::now(); + self.since = Some(Instant::now()); return Stale::No; } - if self.since.elapsed() < TAKEOVER_AFTER { + if self.since.map(|at| at.elapsed() < TAKEOVER_AFTER).unwrap_or(true) { return Stale::No; } if owner_dead(locked_value & GEN_MASK) { @@ -104,7 +124,7 @@ pub fn write_lock<'a>( let cur = seq.load(Ordering::Acquire); if cur & LOCKED == 0 { if seq - .compare_exchange_weak(cur, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange_weak(cur, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) .is_ok() { return Ok(SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }); @@ -116,7 +136,7 @@ pub fn write_lock<'a>( match watch.observe(cur, &owner_dead) { Stale::DeadOwner(observed) => { if seq - .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange(observed, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) .is_ok() { sanitize(); @@ -173,7 +193,7 @@ pub fn read_consistent( match watch.observe(before, &owner_dead) { Stale::DeadOwner(observed) => { if seq - .compare_exchange(observed, LOCKED | (self_tag & GEN_MASK), Ordering::AcqRel, Ordering::Acquire) + .compare_exchange(observed, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) .is_ok() { sanitize(); From acfae8e53b1ba04ae1abe6bfac417ee6b106bcd5 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:48:42 -0600 Subject: [PATCH 34/69] hnsw-plane: unique per-thread salt streams; guard file-sourced upper indices in the raw-write and free paths Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 2 +- native/hnsw-plane/src/graph.rs | 5 ++++- native/hnsw-plane/src/seqlock.rs | 7 ++++++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 672b759eba..976a7da88c 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -422,7 +422,7 @@ impl PlaneFile { /// Return a dead upper entry to the freelist. Caller must have unlinked it from its /// node's slot (or marked the node deleted) first. pub fn free_upper(&self, idx: u32) { - if idx == NO_UPPER { + if idx == NO_UPPER || (idx as u64) >= self.upper_capacity { return; } let head = self.header_atomic_u64(H_UPPER_FREELIST); diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 2c96a7ff9b..4b23cb4627 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -275,7 +275,10 @@ impl Graph { upper_levels: &[Vec], ) -> Result<(), Wedged> { self.file.ensure_high_water(id); - let existing = self.upper_idx_raw(id); + let existing = match self.upper_idx_raw(id) { + idx if idx != NO_UPPER && (idx as u64) >= self.file.upper_capacity => NO_UPPER, // corrupt stored index + idx => idx, + }; let upper_idx = if upper_levels.is_empty() { existing // keep an existing entry bound (level never shrinks in practice) } else if existing != NO_UPPER { diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs index f6cca05a72..310ec66f1f 100644 --- a/native/hnsw-plane/src/seqlock.rs +++ b/native/hnsw-plane/src/seqlock.rs @@ -27,8 +27,13 @@ pub const TAG_MASK: u32 = (1 << 19) - 1; #[inline] fn acquisition_value(self_tag: u32) -> u32 { use std::cell::Cell; + use std::sync::atomic::AtomicU32 as GlobalCounter; + // each thread's salt stream starts at a globally unique offset — identical thread-local + // streams across threads could publish identical lock values, making back-to-back + // acquisitions indistinguishable from one long hold + static NEXT_STREAM: GlobalCounter = GlobalCounter::new(1); thread_local! { - static SALT: Cell = const { Cell::new(0) }; + static SALT: Cell = Cell::new(NEXT_STREAM.fetch_add(0x2545_f491, Ordering::Relaxed)); } let salt = SALT.with(|c| { let v = c.get().wrapping_add(1); From 74efde7e94735c8ebcaddf2244051819de16db83 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:52:26 -0600 Subject: [PATCH 35/69] hnsw-plane: tombstoned virgin slots initialize upper_idx to NO_UPPER A clearNode on a never-written id left the zero-initialized upper index looking like the VALID entry 0; a later raw rewrite of that id would clobber another node's hierarchy. Regression test included. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/graph.rs | 11 ++++++++- native/hnsw-plane/tests/reopen.rs | 39 +++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 4b23cb4627..68248c772b 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -305,7 +305,16 @@ impl Graph { self.file.ensure_high_water(id); let seq = self.file.seq_atomic(id); let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED }; + let p = self.file.slot_ptr_mut(id); + unsafe { + if *p.add(S_FLAGS) == 0 { + // tombstoning a never-written slot: its zero-initialized upper_idx would + // otherwise read as the VALID index 0, and a later raw rewrite of this id + // would clobber upper entry 0 — another node's hierarchy + (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); + } + *p.add(S_FLAGS) = FLAG_DELETED; + } Ok(()) } diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 04dee74fe1..f82f95933f 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -247,3 +247,42 @@ fn same_pid_restart_takeover_via_registry() { assert_eq!(graph.file.seq_atomic(11).load(Ordering::SeqCst) >> 31, 0, "lock reclaimed"); let _ = std::fs::remove_file(&path); } + +#[test] +fn tombstoned_virgin_slot_does_not_alias_upper_entry_zero() { + let dims = 32; + let path = tmp("virgintomb"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + // make node 0's insert claim upper entry 0 (first level>=1 node allocates it); insert + // until some node has an upper entry + let mut upper_owner = None; + for i in 0..64 { + let id = insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + if graph.read_node(id).map(|n| n.level > 0).unwrap_or(false) { + upper_owner = Some(id); + break; + } + } + let upper_owner = upper_owner.expect("some node should have an upper level"); + let mut before = Vec::new(); + assert!(graph.upper_neighbors_into(upper_owner, 1, &mut before) || before.is_empty()); + + // tombstone a NEVER-written id (beyond anything inserted), then raw-write it with an + // upper list: it must allocate a fresh entry, not adopt the zero-initialized index 0 + let virgin = 900; + graph.clear_node(virgin).unwrap(); + let q = hnsw_plane::distance::quantize_int8(&vector_for(virgin, dims)); + graph + .write_node_raw(virgin, 1, &q.0, q.1, q.2, &[1, 2], &[vec![1, 2]]) + .unwrap(); + let mut after = Vec::new(); + let _ = graph.upper_neighbors_into(upper_owner, 1, &mut after); + assert_eq!(before, after, "raw-writing a tombstoned virgin slot must not clobber another node's upper entry"); + let mut virgin_upper = Vec::new(); + assert!(graph.upper_neighbors_into(virgin, 1, &mut virgin_upper)); + assert_eq!(virgin_upper, vec![1, 2]); + let _ = std::fs::remove_file(&path); +} From f73a1b5d92e351ce7ee3b89985d988cab56a93ad Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 18:56:05 -0600 Subject: [PATCH 36/69] hnsw-plane: takeover sanitizer also resets the slot's upper index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dead-writer path left a half-written (or zero-initialized) upper index behind FLAG_DELETED — the same entry-0 aliasing the clear_node fix closed, reachable through crash recovery instead. Sanitized slots now carry NO_UPPER. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/graph.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 68248c772b..ab756b09c3 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -39,7 +39,13 @@ impl Graph { /// so the slot must read as deleted until something rewrites it (heal-on-touch contract; /// FLAG_DELETED rather than 0 so hosts can still free/reuse the id). fn slot_sanitizer(&self, id: u32) -> impl Fn() + '_ { - move || unsafe { *self.file.slot_ptr_mut(id).add(S_FLAGS) = FLAG_DELETED } + move || unsafe { + let p = self.file.slot_ptr_mut(id); + // a dead writer's slot may hold a garbage (or zero-initialized) upper index; a + // later raw rewrite would reuse it and clobber another node's hierarchy + (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); + *p.add(S_FLAGS) = FLAG_DELETED; + } } fn owner_dead(&self) -> impl Fn(u32) -> bool + '_ { From 88fe0c727d89579177639db4f24fb43e555572d8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 19:32:32 -0600 Subject: [PATCH 37/69] hnsw-plane: prettier reflow for the two files Format Check flagged Import lists in `resources/databases.ts` and `resources/indexes/HierarchicalNavigableSmallWorld.ts` and one `getRange` call exceeded the print width. Formatting only; no behavior change. Co-Authored-By: Claude Opus --- resources/databases.ts | 11 ++++++++++- .../indexes/HierarchicalNavigableSmallWorld.ts | 14 ++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/resources/databases.ts b/resources/databases.ts index 0fd175d0af..27cb1cec83 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -3,7 +3,16 @@ import { initSync, getHdbBasePath, get as envGet } from '../utility/environment/ import { INTERNAL_DBIS_NAME } from '../utility/lmdb/terms.ts'; import { open, compareKeys, type Database, type RootDatabase } from 'lmdb'; import { join, extname, basename } from 'path'; -import { closeSync, existsSync, mkdirSync, openSync, readFileSync, readdirSync, realpathSync, unlinkSync } from 'node:fs'; +import { + closeSync, + existsSync, + mkdirSync, + openSync, + readFileSync, + readdirSync, + realpathSync, + unlinkSync, +} from 'node:fs'; import { unlink } from 'node:fs/promises'; import { getBaseSchemaPath, diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index d8579227ca..c783ad65e1 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -5,7 +5,13 @@ import { loggerWithTag } from '../../utility/logging/logger.ts'; import { ClientError } from '../../utility/errors/hdbError.ts'; import type { Id } from '../../resources/ResourceInterface.ts'; import { SKIP } from '@harperfast/extended-iterable'; -import { getPlaneBinding, planeFilePathFor, planeStalePathFor, PLANE_NO_ID, type HnswPlane } from './hnswPlaneBinding.ts'; +import { + getPlaneBinding, + planeFilePathFor, + planeStalePathFor, + PLANE_NO_ID, + type HnswPlane, +} from './hnswPlaneBinding.ts'; const logger = loggerWithTag('HNSW'); @@ -558,7 +564,11 @@ export class HierarchicalNavigableSmallWorld { for (;;) { let inChunk = 0; let lastKey = -1; - for (const { key, value } of this.indexStore.getRange({ start: nextStart, end: Infinity, limit: PLANE_BUILD_CHUNK })) { + for (const { key, value } of this.indexStore.getRange({ + start: nextStart, + end: Infinity, + limit: PLANE_BUILD_CHUNK, + })) { inChunk++; if (typeof key !== 'number') continue; lastKey = key; From 07de4a4ad6be6155572c380f7afdb266c65468e0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 19:57:00 -0600 Subject: [PATCH 38/69] hnsw-plane: drop a dead bounds check, repair smoke.mjs, refresh Cargo.lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `write_node_raw`'s napi wrapper repeated the `id >= max_nodes` rejection three lines after the guarded one; the copy in `write_node_raw_if_absent` is that function's only bounds check and stays. `smoke.mjs` required `./index.js`, which `build.mjs` never emits — the script could only ever throw MODULE_NOT_FOUND. It loads `hnsw-plane.node` now, and its header no longer claims CI runs it (CI runs cargo test plus the parity suite). The committed Cargo.lock predated `libc` becoming a dependency, so its `hnsw-plane` edge list was missing it. Co-Authored-By: Claude Opus --- native/hnsw-plane/Cargo.lock | 1 + native/hnsw-plane/smoke.mjs | 4 ++-- native/hnsw-plane/src/napi.rs | 3 --- 3 files changed, 3 insertions(+), 5 deletions(-) diff --git a/native/hnsw-plane/Cargo.lock b/native/hnsw-plane/Cargo.lock index e7a44ca599..91316d129a 100644 --- a/native/hnsw-plane/Cargo.lock +++ b/native/hnsw-plane/Cargo.lock @@ -46,6 +46,7 @@ dependencies = [ name = "hnsw-plane" version = "0.0.1" dependencies = [ + "libc", "memmap2", "napi", "napi-build", diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs index cf7aa13383..b2cbf796ce 100644 --- a/native/hnsw-plane/smoke.mjs +++ b/native/hnsw-plane/smoke.mjs @@ -1,7 +1,7 @@ -// End-to-end smoke test: `npm run build && node smoke.mjs` (also the CI path). +// End-to-end smoke test: `npm run build:hnsw-plane && node native/hnsw-plane/smoke.mjs`. import { createRequire } from 'module'; const require = createRequire(import.meta.url); -const { Plane } = require('./index.js'); +const { Plane } = require('./hnsw-plane.node'); const dims = 64; const { tmpdir } = await import('node:os'); diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 1b45499353..910dc2ddc0 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -240,9 +240,6 @@ impl Plane { id, self.graph.file.max_nodes ))); } - if (id as u64) >= self.graph.file.max_nodes { - return Err(Error::from_reason(format!("id {} exceeds plane capacity {}", id, self.graph.file.max_nodes))); - } if !(scale as f32).is_finite() || !(inv_mag as f32).is_finite() { return Err(Error::from_reason("scale/invMag must be finite")); } From e670636f11399596fccb15a0cc82d276deb85fc0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 20:03:40 -0600 Subject: [PATCH 39/69] hnsw-plane: stop the dual-write path guessing and inheriting upper entries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in `write_node_raw`, both ending with the fixed upper region draining, `write_upper` returning NO_UPPER, hierarchy edges silently not binding, and plane recall collapsing while the CF graph stays correct. `upper_idx_raw` read the slot's bound index through `read_consistent`, whose fallback is NO_UPPER. A peer worker holding that slot's lock past the 20 ms stale window therefore made the read report "nothing bound" for a node that had an entry, and a duplicate was minted — every contended mirror of a level>=1 node orphaned its predecessor, and on macOS `tag_is_dead` is always false, so that path is the only one available there. The read is now `upper_idx_locked`, taken under the slot write lock: it waits rather than guessing, and adds no failure mode `write_node` did not already have a line later. A node re-mirrored with no upper levels kept its old entry readable, on the reasoning that level never shrinks. It does: the shared id counter reseeds to largestNodeId + 1 on restart, so deleting the top ids hands them back out and the new record redraws its level, often 0. That entry is now emptied in place. Emptied, not freed: returning it to the shared freelist is not atomic with publishing the slot, so a concurrent mirror that read the index first could republish a slot pointing at an entry already handed to another node — trading a bounded retention for cross-node corruption. Keeping it bound costs at most one idle entry per id, which is the retention hnsw-native-plane.md §10 already accepts. Both are covered by tests that fail on the parent commit. The dead-owner takeover sanitizer discards a bound index the same way and is deliberately left alone; see the PR body. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/graph.rs | 38 ++++++++++----- native/hnsw-plane/tests/reopen.rs | 80 +++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+), 12 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index ab756b09c3..807df4f3bd 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -250,21 +250,24 @@ impl Graph { } /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring - /// path reuses a cleared node's entry when the host rewrites the same id. - fn upper_idx_raw(&self, id: u32) -> u32 { + /// path reuses a cleared node's entry when the host rewrites the same id. Read under the + /// slot write lock, not `read_consistent`: that read reports NO_UPPER when it cannot settle + /// within the stale window, and a caller that treats "cannot tell" as "none bound" mints a + /// second entry for an id that already owns one, orphaning the first. + fn upper_idx_locked(&self, id: u32) -> Result { if !self.in_range(id) { - return NO_UPPER; + return Ok(NO_UPPER); } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || { - let p = self.file.slot_ptr(id); - unsafe { - if *p.add(S_FLAGS) == 0 { - return NO_UPPER; // never written - } + let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; + let p = self.file.slot_ptr(id); + Ok(unsafe { + if *p.add(S_FLAGS) == 0 { + NO_UPPER // never written + } else { (p.add(S_UPPER_IDX) as *const u32).read_unaligned() } - }, self.slot_sanitizer(id), || NO_UPPER, self.owner_dead()) + }) } /// Mirror a host-maintained node into the plane: full state per call, host-allocated id @@ -281,12 +284,23 @@ impl Graph { upper_levels: &[Vec], ) -> Result<(), Wedged> { self.file.ensure_high_water(id); - let existing = match self.upper_idx_raw(id) { + let existing = match self.upper_idx_locked(id)? { idx if idx != NO_UPPER && (idx as u64) >= self.file.upper_capacity => NO_UPPER, // corrupt stored index idx => idx, }; let upper_idx = if upper_levels.is_empty() { - existing // keep an existing entry bound (level never shrinks in practice) + // an id re-minted at level 0 must not keep reading the previous node's hierarchy: + // the shared host counter reseeds to largestNodeId + 1 across a restart, so + // deleting the top ids hands them back out and the new record redraws its level. + // Emptied in place rather than freed — returning it to the shared freelist is not + // atomic with publishing the slot below, so a mirror that read this index first + // could republish a slot pointing at an entry already handed to another node. + // Keeping it bound costs at most one idle entry per id, the bounded upper + // retention hnsw-native-plane.md §10 already accepts. + if existing != NO_UPPER { + self.rewrite_upper(existing, &[])?; + } + existing } else if existing != NO_UPPER { self.rewrite_upper(existing, upper_levels)?; existing diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index f82f95933f..ad656b7b14 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -286,3 +286,83 @@ fn tombstoned_virgin_slot_does_not_alias_upper_entry_zero() { assert_eq!(virgin_upper, vec![1, 2]); let _ = std::fs::remove_file(&path); } + +/// The host's id counter reseeds to largestNodeId + 1 across a restart, so deleting the top +/// ids hands them back out and the new record redraws its level — often 0. The re-minted slot +/// must stop reading its predecessor's upper adjacency, and cycling a hot id through levels +/// must not consume a fresh entry each time. +#[test] +fn raw_rewrite_at_level_zero_clears_the_stale_hierarchy() { + let dims = 32; + let path = tmp("upperstale"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create")); + let q = hnsw_plane::distance::quantize_int8(&vector_for(9, dims)); + let write = |level: u8, upper: &[Vec]| { + graph.write_node_raw(9, level, &q.0, q.1, q.2, &[1, 2], upper).unwrap(); + }; + let mut nbrs = Vec::new(); + + write(1, &[vec![7]]); + assert!(graph.upper_neighbors_into(9, 1, &mut nbrs) && nbrs == vec![7]); + write(0, &[]); + assert!(!graph.upper_neighbors_into(9, 1, &mut nbrs), "a level-0 rewrite must not leave the old hierarchy readable"); + + // cycling the same id through level 0 and back must reuse its entry, not mint one per pass + for n in 0..graph.file.upper_capacity as u32 + 4 { + write(1, &[vec![n % 8]]); + write(0, &[]); + } + write(1, &[vec![5]]); + assert!( + graph.upper_neighbors_into(9, 1, &mut nbrs), + "upper region exhausted: level cycling minted a new entry per pass" + ); + assert_eq!(nbrs, vec![5]); + let _ = std::fs::remove_file(&path); +} + +/// A peer worker holding the slot lock past the 20 ms stale window makes the lock-free upper +/// read give up and report NO_UPPER. Treating that "cannot tell" as "nothing bound" mints a +/// second entry per contended mirror and orphans the first, so a hot node burns the fixed +/// upper region until level>=1 mirrors stop binding at all. +#[test] +fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { + use std::sync::atomic::{AtomicBool, Ordering as O}; + let dims = 32; + let path = tmp("uppercontend"); + let _ = std::fs::remove_file(&path); + let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); + let q = hnsw_plane::distance::quantize_int8(&vector_for(9, dims)); + graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![7]]).unwrap(); + + let seq9 = graph.file.seq_atomic(9) as *const _ as usize; + for n in 0..graph.file.upper_capacity as u32 + 4 { + let g2 = graph.clone(); + let held = std::sync::Arc::new(AtomicBool::new(false)); + let held2 = held.clone(); + let hold = std::thread::spawn(move || { + let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; + let g2 = &g2; + let guard = + hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { + g2.file.tag_is_dead(tag) + }); + held2.store(true, O::Release); + std::thread::sleep(std::time::Duration::from_millis(30)); // past TAKEOVER_AFTER + drop(guard); + }); + while !held.load(O::Acquire) { + std::hint::spin_loop(); // the mirror must arrive with the lock genuinely held + } + graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![n % 8]]).unwrap(); + hold.join().unwrap(); + } + + let mut nbrs = Vec::new(); + assert!( + graph.upper_neighbors_into(9, 1, &mut nbrs), + "upper region exhausted: each contended rewrite minted and orphaned an entry" + ); + let _ = std::fs::remove_file(&path); +} From 8d967c3f662fc3a4897c4d650acefa560d2b42c1 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 20:41:23 -0600 Subject: [PATCH 40/69] hnsw-plane: free a fresh upper entry when publishing the slot wedges Nothing points at an entry from `write_upper` until `write_node` publishes the slot, so a wedged publish stranded it outside both the freelist and the graph. `write_node_if_untouched` already frees on its own failed path; this matches it. Also trims `upper_idx_locked`'s doc comment to the invariant. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/graph.rs | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 807df4f3bd..ea6460164b 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -249,11 +249,10 @@ impl Graph { seqlock::read_consistent(seq, self.file.self_tag, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }, self.slot_sanitizer(id), || true, self.owner_dead()) } - /// The slot's stored upper idx regardless of valid/deleted flags — the raw mirroring - /// path reuses a cleared node's entry when the host rewrites the same id. Read under the - /// slot write lock, not `read_consistent`: that read reports NO_UPPER when it cannot settle - /// within the stale window, and a caller that treats "cannot tell" as "none bound" mints a - /// second entry for an id that already owns one, orphaning the first. + /// The slot's stored upper idx regardless of valid/deleted flags. Taken under the slot + /// write lock rather than `read_consistent`, whose NO_UPPER fallback cannot be told apart + /// from an unbound slot — reusing it as one mints a second entry for an id that already + /// owns one. fn upper_idx_locked(&self, id: u32) -> Result { if !self.in_range(id) { return Ok(NO_UPPER); @@ -288,15 +287,14 @@ impl Graph { idx if idx != NO_UPPER && (idx as u64) >= self.file.upper_capacity => NO_UPPER, // corrupt stored index idx => idx, }; + let mut fresh = NO_UPPER; let upper_idx = if upper_levels.is_empty() { - // an id re-minted at level 0 must not keep reading the previous node's hierarchy: - // the shared host counter reseeds to largestNodeId + 1 across a restart, so - // deleting the top ids hands them back out and the new record redraws its level. - // Emptied in place rather than freed — returning it to the shared freelist is not - // atomic with publishing the slot below, so a mirror that read this index first - // could republish a slot pointing at an entry already handed to another node. - // Keeping it bound costs at most one idle entry per id, the bounded upper - // retention hnsw-native-plane.md §10 already accepts. + // the host reseeds its id counter to largestNodeId + 1 on restart, so an id can be + // re-minted at level 0 over a slot that had a hierarchy; that entry must stop being + // readable. Emptied in place rather than freed: the freelist hand-off is not atomic + // with publishing the slot below, so a mirror that read this index first could + // republish a slot pointing at an entry already given to another node. One idle + // entry per id is the bounded retention hnsw-native-plane.md §10 accepts. if existing != NO_UPPER { self.rewrite_upper(existing, &[])?; } @@ -305,11 +303,17 @@ impl Graph { self.rewrite_upper(existing, upper_levels)?; existing } else { - self.write_upper(upper_levels)? + fresh = self.write_upper(upper_levels)?; + fresh }; let mut l0 = neighbors.to_vec(); l0.truncate(self.file.layer0_cap); - self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx)?; + if let Err(wedged) = self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx) { + // nothing points at a freshly allocated entry until the slot is published, so a + // wedged publish strands it outside both the freelist and the graph + self.file.free_upper(fresh); + return Err(wedged); + } Ok(()) } From b16362cbe6e3da64ebedfe07e8cc7676a45b04c0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 20:50:07 -0600 Subject: [PATCH 41/69] hnsw-plane: free the backfill scan's upper entry when its slot lock wedges MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `write_node_if_untouched` allocates before taking the slot lock, and the `?` on that lock returned without freeing — the same leak fixed one commit ago in `write_node_raw`, in the function that was cited as the model for getting it right. Nothing references a fresh entry until the write lands, so every path that does not publish it has to free it. Covered by a test that fails on the parent commit; it needs the 5 s writer wedge bound to elapse, so it runs alongside the rest rather than adding to the suite's wall clock. `write_node_raw`'s equivalent window survives only between `upper_idx_locked` releasing and `write_node` re-acquiring and cannot be driven deterministically — a holder that takes the lock first now wedges the read before anything is allocated — so its guard stays defensive and untested rather than pinned by a test that would pass on either side. `upper_high_water` gains the accessor `id_high_water` already had, which is what lets the test tell a reclaimed entry from a leaked one. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/format.rs | 4 +++ native/hnsw-plane/src/graph.rs | 18 +++++++---- native/hnsw-plane/tests/reopen.rs | 52 +++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+), 6 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 976a7da88c..e1cc0a8902 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -327,6 +327,10 @@ impl PlaneFile { self.header_atomic_u64(H_ID_HIGH_WATER).load(Ordering::Acquire) } + pub fn upper_high_water(&self) -> u64 { + self.header_atomic_u64(H_UPPER_HIGH_WATER).load(Ordering::Acquire) + } + /// Entry point (id, level), read as one atomic word — a torn (new id, old level) pair /// would blind a racing search. pub fn entry_point(&self) -> (u32, u32) { diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index ea6460164b..081ea74ad6 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -309,9 +309,7 @@ impl Graph { let mut l0 = neighbors.to_vec(); l0.truncate(self.file.layer0_cap); if let Err(wedged) = self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx) { - // nothing points at a freshly allocated entry until the slot is published, so a - // wedged publish strands it outside both the freelist and the graph - self.file.free_upper(fresh); + self.file.free_upper(fresh); // unreachable from any slot until publication succeeds return Err(wedged); } Ok(()) @@ -634,12 +632,20 @@ impl Graph { debug_assert!(neighbors.len() <= self.file.layer0_cap); debug_assert_eq!(vector.len(), self.file.dims); self.file.ensure_high_water(id); - // the upper entry is allocated before taking the slot lock (allocation is cheap and - // an unused entry is freed below on the untouched-check failing) + // the upper entry is allocated before taking the slot lock (allocation is cheap); it is + // unreachable from any slot until the write below lands, so every path that does not + // publish it — a wedged lock, a slot that turns out to be touched — has to free it let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels)? }; let seq = self.file.seq_atomic(id); let written = { - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; + let _guard = match seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()) + { + Ok(guard) => guard, + Err(wedged) => { + self.file.free_upper(upper_idx); + return Err(wedged); + } + }; let p = self.file.slot_ptr_mut(id); let dims = self.file.dims; unsafe { diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index ad656b7b14..e9eac19942 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -366,3 +366,55 @@ fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { ); let _ = std::fs::remove_file(&path); } + +/// The backfill scan allocates its upper entry before taking the slot lock, and nothing +/// references that entry until the write lands — so giving up on a wedged lock without freeing +/// strands it outside both the freelist and the graph. `upper_high_water` not advancing on the +/// next allocation is what proves it was reclaimed. +/// +/// `write_node_raw`'s equivalent window (between `upper_idx_locked` releasing and `write_node` +/// re-acquiring) is guarded the same way but cannot be driven deterministically: a holder that +/// takes the lock first now wedges the read, before anything is allocated. +#[test] +fn a_wedged_untouched_write_frees_its_upper_entry() { + use std::sync::atomic::{AtomicBool, Ordering as O}; + let dims = 32; + let path = tmp("wedgeuntouched"); + let _ = std::fs::remove_file(&path); + let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); + let write = |id: u32| { + let q = hnsw_plane::distance::quantize_int8(&vector_for(id, dims)); + let _ = graph.write_node_if_untouched(id, 1, &q.0, q.1, q.2, &[1, 2], &[vec![id]]); + }; + + write(1); + let baseline = graph.file.upper_high_water(); + + let seq9 = graph.file.seq_atomic(9) as *const _ as usize; + let g2 = graph.clone(); + let held = std::sync::Arc::new(AtomicBool::new(false)); + let held2 = held.clone(); + let hold = std::thread::spawn(move || { + let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; + let g2 = &g2; + let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { + g2.file.tag_is_dead(tag) + }); + held2.store(true, O::Release); + std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER + drop(guard); + }); + while !held.load(O::Acquire) { + std::hint::spin_loop(); + } + write(9); // wedges on the slot lock, having already allocated an upper entry + hold.join().unwrap(); + + write(2); + assert_eq!( + graph.file.upper_high_water(), + baseline + 1, + "the wedged write leaked its upper entry instead of returning it to the freelist" + ); + let _ = std::fs::remove_file(&path); +} From 365d7e2f2306b7af53a67ae6d7799d3a021bd4a6 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 20:55:50 -0600 Subject: [PATCH 42/69] hnsw-plane: bound the test lock handshakes and assert their write results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wedge and contention tests spun unbounded on the holder thread's signal, so a holder that panicked before signalling hung the run instead of failing it, and the wedge test discarded every write result — it could report success without the wedge or the reuse ever happening. Co-Authored-By: Claude Opus --- native/hnsw-plane/tests/reopen.rs | 42 ++++++++++++++++--------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index e9eac19942..f8464063db 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -12,6 +12,16 @@ fn vector_for(i: u32, dims: usize) -> Vec { (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect() } +/// Wait for a holder thread to report the lock taken, rather than spinning forever if it dies +/// before signalling. +fn await_lock(held: &std::sync::atomic::AtomicBool) { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); + while !held.load(std::sync::atomic::Ordering::Acquire) { + assert!(std::time::Instant::now() < deadline, "holder thread never acquired the lock"); + std::hint::spin_loop(); + } +} + fn tmp(name: &str) -> std::path::PathBuf { std::env::temp_dir().join(format!("hnsw-{name}-{}.hnsw", std::process::id())) } @@ -352,9 +362,7 @@ fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { std::thread::sleep(std::time::Duration::from_millis(30)); // past TAKEOVER_AFTER drop(guard); }); - while !held.load(O::Acquire) { - std::hint::spin_loop(); // the mirror must arrive with the lock genuinely held - } + await_lock(&held); // the mirror must arrive with the lock genuinely held graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![n % 8]]).unwrap(); hold.join().unwrap(); } @@ -367,32 +375,28 @@ fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { let _ = std::fs::remove_file(&path); } -/// The backfill scan allocates its upper entry before taking the slot lock, and nothing -/// references that entry until the write lands — so giving up on a wedged lock without freeing -/// strands it outside both the freelist and the graph. `upper_high_water` not advancing on the -/// next allocation is what proves it was reclaimed. -/// -/// `write_node_raw`'s equivalent window (between `upper_idx_locked` releasing and `write_node` -/// re-acquiring) is guarded the same way but cannot be driven deterministically: a holder that -/// takes the lock first now wedges the read, before anything is allocated. +/// Nothing references a freshly allocated upper entry until its write lands, so a path that +/// gives up on a wedged slot lock without freeing strands it outside both the freelist and the +/// graph. `upper_high_water` staying put on the next allocation is what distinguishes a +/// reclaimed entry from a leaked one. #[test] fn a_wedged_untouched_write_frees_its_upper_entry() { - use std::sync::atomic::{AtomicBool, Ordering as O}; + use std::sync::atomic::Ordering as O; let dims = 32; let path = tmp("wedgeuntouched"); let _ = std::fs::remove_file(&path); let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); let write = |id: u32| { let q = hnsw_plane::distance::quantize_int8(&vector_for(id, dims)); - let _ = graph.write_node_if_untouched(id, 1, &q.0, q.1, q.2, &[1, 2], &[vec![id]]); + graph.write_node_if_untouched(id, 1, &q.0, q.1, q.2, &[1, 2], &[vec![id]]) }; - write(1); + assert_eq!(write(1), Ok(true)); let baseline = graph.file.upper_high_water(); let seq9 = graph.file.seq_atomic(9) as *const _ as usize; let g2 = graph.clone(); - let held = std::sync::Arc::new(AtomicBool::new(false)); + let held = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); let held2 = held.clone(); let hold = std::thread::spawn(move || { let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; @@ -404,13 +408,11 @@ fn a_wedged_untouched_write_frees_its_upper_entry() { std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER drop(guard); }); - while !held.load(O::Acquire) { - std::hint::spin_loop(); - } - write(9); // wedges on the slot lock, having already allocated an upper entry + await_lock(&held); + assert_eq!(write(9), Err(hnsw_plane::seqlock::Wedged), "the held lock must wedge this write"); hold.join().unwrap(); - write(2); + assert_eq!(write(2), Ok(true)); assert_eq!( graph.file.upper_high_water(), baseline + 1, From e2d76de864722b978c9de65e34830adbe4acf518 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 31 Aug 2026 21:02:02 -0600 Subject: [PATCH 43/69] hnsw-plane: make the test holders prove they took the lock Both lock-contention holders published their "held" flag on the result of write_lock without checking it succeeded, so a wedged acquisition let the tests run uncontended and pass. Co-Authored-By: Claude Opus --- native/hnsw-plane/tests/reopen.rs | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index f8464063db..005c89860e 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -12,8 +12,6 @@ fn vector_for(i: u32, dims: usize) -> Vec { (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect() } -/// Wait for a holder thread to report the lock taken, rather than spinning forever if it dies -/// before signalling. fn await_lock(held: &std::sync::atomic::AtomicBool) { let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); while !held.load(std::sync::atomic::Ordering::Acquire) { @@ -357,12 +355,13 @@ fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { g2.file.tag_is_dead(tag) - }); + }) + .expect("the holder must actually take the lock, or the test proves nothing"); held2.store(true, O::Release); std::thread::sleep(std::time::Duration::from_millis(30)); // past TAKEOVER_AFTER drop(guard); }); - await_lock(&held); // the mirror must arrive with the lock genuinely held + await_lock(&held); graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![n % 8]]).unwrap(); hold.join().unwrap(); } @@ -377,8 +376,7 @@ fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { /// Nothing references a freshly allocated upper entry until its write lands, so a path that /// gives up on a wedged slot lock without freeing strands it outside both the freelist and the -/// graph. `upper_high_water` staying put on the next allocation is what distinguishes a -/// reclaimed entry from a leaked one. +/// graph. #[test] fn a_wedged_untouched_write_frees_its_upper_entry() { use std::sync::atomic::Ordering as O; @@ -403,7 +401,8 @@ fn a_wedged_untouched_write_frees_its_upper_entry() { let g2 = &g2; let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { g2.file.tag_is_dead(tag) - }); + }) + .expect("the holder must actually take the lock, or the test proves nothing"); held2.store(true, O::Release); std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER drop(guard); From 52e5fd540d1bb8c9b4544ca6fe4aeef70d37d57d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 05:50:14 -0600 Subject: [PATCH 44/69] hnsw-plane: restore MADV_RANDOM (dropped by a parallel-worktree replay of 97bb5c9eb) 36cd9be29 added it; the next commit's format.rs edits came from a worktree state that predated it and silently clobbered the function and both call sites. Same rationale as before: densely packed hosts live in permanent memory pressure; readahead on random re-faults taxes every tenant with no sequential reader to protect. Co-Authored-By: Claude Fable 5 --- native/hnsw-plane/src/format.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index e1cc0a8902..2457946863 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -90,6 +90,17 @@ pub struct PlaneFile { const PAGE: usize = 4096; const H_SLOTS_PER_PAGE: usize = 20; // u16 +/// MADV_RANDOM: hosts packing many instances live in permanent memory pressure, where +/// evict-and-refault is steady state; default readahead pulls ~16 unwanted pages per random +/// re-fault, taxing every tenant's page cache. The plane has no sequential reader to protect +/// (search is pointer-chasing, the builder writes, backfill scans read the host store). +fn advise_random(map: &MmapMut) { + #[cfg(unix)] + let _ = map.advise(memmap2::Advice::Random); + #[cfg(not(unix))] + let _ = map; +} + fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { let raw = S_VECTOR + dims + layer0_cap * 4; raw.next_multiple_of(64) // cache-line align @@ -131,6 +142,7 @@ impl PlaneFile { let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; file.set_len(len)?; let mut map = unsafe { MmapMut::map_mut(&file)? }; + advise_random(&map); // geometry and allocator state first; MAGIC+VERSION last, so a concurrent opener // in the create window sees an invalid header (retryable) rather than adopting a // half-initialized plane with max_nodes = 0 @@ -173,6 +185,7 @@ impl PlaneFile { return Err(io::Error::new(io::ErrorKind::InvalidData, "plane file shorter than its header: recreate the index")); } let map = unsafe { MmapMut::map_mut(&file)? }; + advise_random(&map); let magic = u32::from_le_bytes(map[H_MAGIC..H_MAGIC + 4].try_into().unwrap()); let version = u32::from_le_bytes(map[H_VERSION..H_VERSION + 4].try_into().unwrap()); if magic != MAGIC || version != VERSION { From bab065a8883eab668ccecf1bc9ca1fad081b8938 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 11:53:19 -0600 Subject: [PATCH 45/69] hnsw-plane: close the five open review threads on #2430 First-entry race: claim_entry_if_empty is a strict CAS from the empty encoding, so exactly one racer roots the graph and every loser joins it instead of returning an unlinked node. Both self-promotion sites go through it; a loser reuses the upper entry its slot already names. Concurrent reads: pad the vector so neighbor arrays are 4-aligned (and move the upper-list pad ahead of the ids), then read every field a reader acts on with an aligned read_volatile. The stored vector stays an ordinary load so the dot product keeps vectorizing. VERSION 6. Dead entry points: delete_node re-elects before the fallible upper cleanup, and searches repair an entry no writer will through the O(1) previous-entry hint, which promotions now record. Async iterator: one memoized iterator per iterate() call plus a closed flag, and a handler on the pending pipeline so an abandoned iterable cannot raise unhandledRejection. Stale planes: an undeletable plane is invalidated in band (watermark 0 under a durability barrier) before the .stale sidecar, the flag-off cleanup path marks instead of only logging, and the sidecar's own cleanup no longer permanently disables a plane whose file an operator already removed. Co-Authored-By: Claude Opus --- hnsw-native-plane.md | 22 ++-- native/hnsw-plane/src/format.rs | 74 ++++++++++-- native/hnsw-plane/src/graph.rs | 105 +++++++++++------- native/hnsw-plane/src/insert.rs | 81 +++++++++----- native/hnsw-plane/src/search.rs | 63 +++++------ native/hnsw-plane/tests/concurrent.rs | 104 ++++++++++++++--- native/hnsw-plane/tests/reopen.rs | 76 +++++++++++++ .../HierarchicalNavigableSmallWorld.ts | 47 ++++++-- resources/search.ts | 23 ++-- unitTests/resources/vectorIndexPlane.test.js | 80 ++++++++++++- 10 files changed, 517 insertions(+), 158 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index baa6753f0d..5b4df41c94 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -97,15 +97,19 @@ One file per index (per slice, once C2 lands): `.hnsw`. **Main region — layer-0 slots**, addressed `4096 + id × slot_size`: -| Field | Size (768-d int8, cap 64) | -| ------------------------------- | ---------------------------------- | -| seq (seqlock) | 4 B | -| flags (valid/deleted) + level | 2 B | -| scale (f32) + invMag (f32) | 8 B | -| degree | 2 B | -| vector (int8 × 768) | 768 B | -| neighbor ids (u32 × layer0_cap) | 256 B | -| **total, padded** | **1,040 B → 1 KB-aligned 1,088 B** | +| Field | Size (768-d int8, cap 64) | +| ------------------------------- | ----------------------------------- | +| seq (seqlock) | 4 B | +| flags (valid/deleted) + level | 2 B | +| scale (f32) + invMag (f32) | 8 B | +| degree | 2 B | +| vector (int8 × 768) | 768 B (padded to a 4-byte boundary) | +| neighbor ids (u32 × layer0_cap) | 256 B | +| **total, padded** | **1,040 B → 1 KB-aligned 1,088 B** | + +The vector's trailing pad keeps the neighbor array 4-aligned for every `dims`, so the search +hot path reads each neighbor id as one aligned volatile `u32`. Upper-layer id lists are padded +the same way (`degree u16 + pad u16 + ids`). At 100M nodes: ~109 GB (int8). A binary-code v2 slot (96 B codes + ids) is ~384 B → ~38 GB. For comparison, today's encoding averages 1,425 B/node _plus_ RocksDB overhead — so v1 is diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index 2457946863..d61843dab6 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -8,7 +8,7 @@ use std::path::Path; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 5; // v5: opener registry + aligned freelist pointer (older files: reindex) +pub const VERSION: u32 = 6; // v6: 4-aligned neighbor + upper id arrays (older files: reindex) pub const HEADER_SIZE: usize = 4096; // Header field byte offsets. @@ -39,11 +39,15 @@ pub const REGISTRY_SLOTS: usize = 64; /// 1/8 of max_nodes (2x headroom). P(level >= 9) at mL = 1/ln16 is ~e^-25 — unreachable. pub const MAX_UPPER_LEVELS: usize = 8; pub const UPPER_CAP: usize = 64; // matches the JS graph's upper cap (M<<2 under optimizeRouting) -// entry: seq u32 | levels u8 | pad | per-level (degree u16 + ids u32*UPPER_CAP) +// entry: seq u32 | levels u8 | pad | per-level (degree u16 + pad u16 + ids u32*UPPER_CAP) pub const U_SEQ: usize = 0; pub const U_LEVELS: usize = 4; pub const U_LISTS: usize = 8; -pub const UPPER_LEVEL_STRIDE: usize = 2 + UPPER_CAP * 4 + 2; // degree + ids + pad -> 132 +/// The pad follows the degree rather than the ids so every id array starts 4-aligned; the +/// stride (and so the entry size) is unchanged either way. +pub const UL_DEGREE: usize = 0; +pub const UL_IDS: usize = 4; +pub const UPPER_LEVEL_STRIDE: usize = UL_IDS + UPPER_CAP * 4; pub const NO_UPPER: u32 = u32::MAX; // Slot layout offsets (within a slot). @@ -55,8 +59,15 @@ pub const S_SCALE: usize = 8; // f32 pub const S_INV_MAG: usize = 12; // f32 pub const S_UPPER_IDX: usize = 16; // u32 index into the upper region; NO_UPPER = none pub const S_VECTOR: usize = 20; // dims bytes (int8) or dims*4 (f32) - // neighbors: u32 * layer0_cap, follows vector - // deleted slots reuse the first neighbor word as freelist next-pointer + // neighbors: u32 * layer0_cap, follows the 4-padded vector + +/// Byte offset of a slot's neighbor array. The vector is padded to a 4-byte boundary so this +/// is 4-aligned for every dims: the search hot path then reads each neighbor as one aligned +/// volatile u32 instead of four byte loads plus shifts. +#[inline] +pub const fn neighbor_offset(dims: usize) -> usize { + S_VECTOR + (dims + 3) / 4 * 4 +} pub const FLAG_VALID: u8 = 1; pub const FLAG_DELETED: u8 = 2; @@ -102,7 +113,7 @@ fn advise_random(map: &MmapMut) { } fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { - let raw = S_VECTOR + dims + layer0_cap * 4; + let raw = neighbor_offset(dims) + layer0_cap * 4; raw.next_multiple_of(64) // cache-line align } @@ -156,6 +167,9 @@ impl PlaneFile { .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); map[H_MAX_NODES..H_MAX_NODES + 8].copy_from_slice(&max_nodes.to_le_bytes()); map[H_UPPER_FREELIST..H_UPPER_FREELIST + 8].copy_from_slice(&(NO_UPPER as u64).to_le_bytes()); + // zero would read as "node 0 was the previous entry point" and hand every re-election + // and search-side repair a candidate that was never an entry point + map[H_ENTRY_PREV..H_ENTRY_PREV + 8].copy_from_slice(&(NO_ID as u64).to_le_bytes()); map[H_VERSION..H_VERSION + 4].copy_from_slice(&VERSION.to_le_bytes()); std::sync::atomic::fence(Ordering::Release); map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); @@ -295,8 +309,9 @@ impl PlaneFile { let _ = head.compare_exchange(cur, NO_ID as u64, Ordering::AcqRel, Ordering::Acquire); continue; } - // next-pointer lives in the dead slot's scale field: offset 8, aligned for any - // dims (the first neighbor word at S_VECTOR+dims is 4-aligned only when dims%4==0) + // next-pointer lives in the dead slot's scale field rather than its first neighbor + // word: the neighbor array is a live reader's aligned volatile load target, and a + // freelist pointer parked there would be decoded as a neighbor id let next = unsafe { (*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32)).load(Ordering::Acquire) }; let tag = (cur >> 32).wrapping_add(1); let new = (next as u64) | (tag << 32); @@ -353,15 +368,47 @@ impl PlaneFile { pub fn set_entry_point(&self, id: u32, level: u32) { let prev = self.header_atomic_u64(H_ENTRY).swap((id as u64) | ((level as u64) << 32), Ordering::AcqRel); - if (prev & 0xffff_ffff) as u32 != NO_ID && (prev & 0xffff_ffff) as u32 != id { - self.header_atomic_u64(H_ENTRY_PREV).store(prev, Ordering::Release); + self.record_previous_entry(prev, id); + } + + /// Remember the entry point a PROMOTION displaced. Only promotions are recorded: the node + /// they displace was live and high-level, which is what makes it a usable hint. Recording + /// a re-election's replacement instead would fill the hint with the dead node that forced + /// the re-election. + #[inline] + fn record_previous_entry(&self, prev_packed: u64, new_id: u32) { + let prev_id = (prev_packed & 0xffff_ffff) as u32; + if prev_id != NO_ID && prev_id != new_id { + self.header_atomic_u64(H_ENTRY_PREV).store(prev_packed, Ordering::Release); } } + /// Claim the entry point of an EMPTY graph: a strict compare-exchange from the empty + /// encoding, so exactly one racer wins. `set_entry_point_if_not_better` cannot serve here — + /// it is a not-worse install, so a second first-inserter would replace the winner with its + /// own edgeless node and orphan everything already rooted at the winner. A loser must join + /// the winner's graph instead of returning an unlinked node. + pub fn claim_entry_if_empty(&self, id: u32, level: u32) -> bool { + self.header_atomic_u64(H_ENTRY) + .compare_exchange(NO_ID as u64, (id as u64) | ((level as u64) << 32), Ordering::AcqRel, Ordering::Acquire) + .is_ok() + } + /// Entry-point CAS for re-election: install (id, level) only while the current entry is /// still `expected_id` or is of a lower level — a concurrent insert that just promoted a /// higher-level entry must not be clobbered by a delete's level-0 survivor. pub fn set_entry_point_if_not_better(&self, id: u32, level: u32, expected_id: u32) { + self.cas_entry_if_not_better(id, level, expected_id, false); + } + + /// The same CAS for an insert that PROMOTED itself above the entry it observed: the + /// displaced entry is live, so it is recorded as the previous-entry hint that re-election + /// and the search-side repair both consult before any O(high-water) scan. + pub fn promote_entry_point(&self, id: u32, level: u32, expected_id: u32) { + self.cas_entry_if_not_better(id, level, expected_id, true); + } + + fn cas_entry_if_not_better(&self, id: u32, level: u32, expected_id: u32, record_prev: bool) { let cell = self.header_atomic_u64(H_ENTRY); let new = (id as u64) | ((level as u64) << 32); let mut cur = cell.load(Ordering::Acquire); @@ -372,7 +419,12 @@ impl PlaneFile { return; // someone installed a better entry meanwhile } match cell.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire) { - Ok(_) => return, + Ok(_) => { + if record_prev { + self.record_previous_entry(cur, id); + } + return; + } Err(now) => cur = now, } } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 081ea74ad6..50a27ad6f2 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -6,12 +6,26 @@ use crate::distance::{cosine_i8_i8_raw, cosine_int8_raw, Query}; use crate::format::{ - PlaneFile, FLAG_DELETED, FLAG_VALID, MAX_UPPER_LEVELS, NO_UPPER, S_DEGREE, S_FLAGS, S_INV_MAG, S_LEVEL, S_SCALE, - S_UPPER_IDX, S_VECTOR, UPPER_CAP, UPPER_LEVEL_STRIDE, U_LEVELS, U_LISTS, + neighbor_offset, PlaneFile, FLAG_DELETED, FLAG_VALID, MAX_UPPER_LEVELS, NO_UPPER, S_DEGREE, S_FLAGS, S_INV_MAG, + S_LEVEL, S_SCALE, S_UPPER_IDX, S_VECTOR, UPPER_CAP, UPPER_LEVEL_STRIDE, UL_DEGREE, UL_IDS, U_LEVELS, U_LISTS, }; use crate::seqlock; use crate::seqlock::Wedged; +/// Aligned volatile load of a slot/upper-entry field another process may be mutating. +/// +/// Ordinary loads of concurrently-written mmap bytes are a data race the optimizer is free to +/// duplicate, split, or sink across the seqlock's validating fence — which would let a reader +/// act on bytes the generation check never covered. Volatile forbids exactly that. The vector +/// is deliberately NOT read this way: `cosine_int8_raw` must stay autovectorized, and a torn +/// vector only perturbs a distance the generation check then discards. Every field this is +/// applied to is naturally aligned (slots are 64-aligned and the neighbor/id arrays are +/// 4-padded by format.rs), so these compile to single loads. +#[inline(always)] +unsafe fn vread(p: *const T) -> T { + p.read_volatile() +} + pub struct Graph { pub file: PlaneFile, } @@ -67,12 +81,12 @@ impl Graph { seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { - let flags = *p.add(S_FLAGS); + let flags = vread(p.add(S_FLAGS)); if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { return None; } - let scale = (p.add(S_SCALE) as *const f32).read_unaligned(); - let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); + let scale = vread(p.add(S_SCALE) as *const f32); + let inv_mag = vread(p.add(S_INV_MAG) as *const f32); Some(cosine_int8_raw(query, p.add(S_VECTOR) as *const i8, scale, inv_mag)) } }, self.slot_sanitizer(id), || None, self.owner_dead()) @@ -119,20 +133,20 @@ impl Graph { } let seq = self.file.seq_atomic(id); let cap = self.file.layer0_cap; - let dims = self.file.dims; + let nbase = neighbor_offset(self.file.dims); seqlock::read_consistent(seq, self.file.self_tag, || { out.clear(); let p = self.file.slot_ptr(id); unsafe { - let flags = *p.add(S_FLAGS); + let flags = vread(p.add(S_FLAGS)); if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { return None; } - let level = *p.add(S_LEVEL); - let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; - let base = p.add(S_VECTOR + dims) as *const u32; + let level = vread(p.add(S_LEVEL)); + let degree = u16::from_le(vread(p.add(S_DEGREE) as *const u16)) as usize; + let base = p.add(nbase) as *const u32; for i in 0..degree.min(cap) { - out.push(u32::from_le(base.add(i).read_unaligned())); + out.push(u32::from_le(vread(base.add(i)))); } Some(level) } @@ -149,11 +163,11 @@ impl Graph { seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { - let flags = *p.add(S_FLAGS); + let flags = vread(p.add(S_FLAGS)); if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { return NO_UPPER; } - (p.add(S_UPPER_IDX) as *const u32).read_unaligned() + vread(p.add(S_UPPER_IDX) as *const u32) } }, self.slot_sanitizer(id), || NO_UPPER, self.owner_dead()) } @@ -172,15 +186,15 @@ impl Graph { out.clear(); let p = self.file.upper_ptr(idx); unsafe { - let levels = *p.add(U_LEVELS); + let levels = vread(p.add(U_LEVELS)); if level > levels { return false; } let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); - let degree = u16::from_le((lp as *const u16).read_unaligned()) as usize; - let base = lp.add(2) as *const u32; + let degree = u16::from_le(vread(lp.add(UL_DEGREE) as *const u16)) as usize; + let base = lp.add(UL_IDS) as *const u32; for i in 0..degree.min(UPPER_CAP) { - out.push(u32::from_le(base.add(i).read_unaligned())); + out.push(u32::from_le(vread(base.add(i)))); } true } @@ -206,8 +220,8 @@ impl Graph { for (l, list) in levels.iter().take(n).enumerate() { let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); let deg = list.len().min(UPPER_CAP); - (lp as *mut u16).write_unaligned((deg as u16).to_le()); - let base = lp.add(2) as *mut u32; + (lp.add(UL_DEGREE) as *mut u16).write_unaligned((deg as u16).to_le()); + let base = lp.add(UL_IDS) as *mut u32; for (i, id) in list.iter().take(deg).enumerate() { base.add(i).write_unaligned(id.to_le()); } @@ -229,8 +243,8 @@ impl Graph { for (l, list) in levels.iter().take(n).enumerate() { let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); let deg = list.len().min(UPPER_CAP); - (lp as *mut u16).write_unaligned((deg as u16).to_le()); - let base = lp.add(2) as *mut u32; + (lp.add(UL_DEGREE) as *mut u16).write_unaligned((deg as u16).to_le()); + let base = lp.add(UL_IDS) as *mut u32; for (i, id) in list.iter().take(deg).enumerate() { base.add(i).write_unaligned(id.to_le()); } @@ -246,7 +260,7 @@ impl Graph { return false; } let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || unsafe { *self.file.slot_ptr(id).add(S_FLAGS) != 0 }, self.slot_sanitizer(id), || true, self.owner_dead()) + seqlock::read_consistent(seq, self.file.self_tag, || unsafe { vread(self.file.slot_ptr(id).add(S_FLAGS)) != 0 }, self.slot_sanitizer(id), || true, self.owner_dead()) } /// The slot's stored upper idx regardless of valid/deleted flags. Taken under the slot @@ -356,12 +370,12 @@ impl Graph { return Ok(false); } let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); - let degree = u16::from_le((lp as *const u16).read_unaligned()) as usize; - let base = lp.add(2) as *mut u32; + let degree = u16::from_le((lp.add(UL_DEGREE) as *const u16).read_unaligned()) as usize; + let base = lp.add(UL_IDS) as *mut u32; let mut list: Vec = (0..degree.min(UPPER_CAP)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); f(&mut list); list.truncate(UPPER_CAP); - (lp as *mut u16).write_unaligned((list.len() as u16).to_le()); + (lp.add(UL_DEGREE) as *mut u16).write_unaligned((list.len() as u16).to_le()); for (i, id) in list.iter().enumerate() { base.add(i).write_unaligned(id.to_le()); } @@ -377,20 +391,21 @@ impl Graph { let seq = self.file.seq_atomic(id); let dims = self.file.dims; let cap = self.file.layer0_cap; + let nbase_off = neighbor_offset(dims); seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { - let flags = *p.add(S_FLAGS); + let flags = vread(p.add(S_FLAGS)); if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { return None; } - let level = *p.add(S_LEVEL); - let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; - let scale = (p.add(S_SCALE) as *const f32).read_unaligned(); - let inv_mag = (p.add(S_INV_MAG) as *const f32).read_unaligned(); + let level = vread(p.add(S_LEVEL)); + let degree = u16::from_le(vread(p.add(S_DEGREE) as *const u16)) as usize; + let scale = vread(p.add(S_SCALE) as *const f32); + let inv_mag = vread(p.add(S_INV_MAG) as *const f32); let vector = std::slice::from_raw_parts(p.add(S_VECTOR) as *const i8, dims).to_vec(); - let nbase = p.add(S_VECTOR + dims) as *const u32; - let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(nbase.add(i).read_unaligned())).collect(); + let nbase = p.add(nbase_off) as *const u32; + let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(vread(nbase.add(i)))).collect(); Some(NodeRead { level, scale, inv_mag, vector, neighbors }) } }, self.slot_sanitizer(id), || None, self.owner_dead()) @@ -413,7 +428,7 @@ impl Graph { (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); for (i, n) in neighbors.iter().enumerate() { - (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); + (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); } // valid last within the locked section; the seqlock release publishes it *p.add(S_FLAGS) = FLAG_VALID; @@ -440,7 +455,7 @@ impl Graph { return Ok(false); } let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; - let base = p.add(S_VECTOR + dims) as *mut u32; + let base = p.add(neighbor_offset(dims)) as *mut u32; let mut list: Vec = (0..degree.min(cap)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); f(&mut list); list.truncate(cap); @@ -473,7 +488,7 @@ impl Graph { if degree != expected.len() { return Ok(false); } - let base = p.add(S_VECTOR + dims) as *mut u32; + let base = p.add(neighbor_offset(dims)) as *mut u32; for (i, want) in expected.iter().enumerate() { if u32::from_le(base.add(i).read_unaligned()) != *want { return Ok(false); @@ -497,7 +512,7 @@ impl Graph { unsafe { (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); for (i, n) in neighbors.iter().enumerate() { - (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); + (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); } } Ok(()) @@ -534,6 +549,13 @@ impl Graph { *p.add(S_FLAGS) = FLAG_DELETED; } } + // re-elect BEFORE the fallible upper cleanup: the slot is tombstoned above and + // re-election is infallible, so ordering it first means no error path can leave the + // header naming a dead entry — which blinds every search until an insert happens to + // repair it + if entry_id == id { + self.reelect_entry_point_replacing(&candidates, id); + } if upper_idx != NO_UPPER && (upper_idx as u64) < self.file.upper_capacity { // empty the entry under its own lock BEFORE freeing: a traversal that already // read this node's upper_idx must find a dead entry, not one reallocated to a @@ -541,9 +563,6 @@ impl Graph { self.rewrite_upper(upper_idx, &[])?; } self.file.free_upper(upper_idx); - if entry_id == id { - self.reelect_entry_point_replacing(&candidates, id); - } self.file.free_id(id); Ok(()) } @@ -552,7 +571,7 @@ impl Graph { /// first live node found scanning the id range (rare path: only when the entry's whole /// neighborhood is gone). An empty graph clears the entry. /// A node's level without copying its vector or edges (cheap re-election scans). - fn node_level(&self, id: u32) -> Option { + pub(crate) fn node_level(&self, id: u32) -> Option { if !self.in_range(id) { return None; } @@ -560,10 +579,10 @@ impl Graph { seqlock::read_consistent(seq, self.file.self_tag, || { let p = self.file.slot_ptr(id); unsafe { - if *p.add(S_FLAGS) != FLAG_VALID { + if vread(p.add(S_FLAGS)) != FLAG_VALID { return None; } - Some(*p.add(S_LEVEL)) + Some(vread(p.add(S_LEVEL))) } }, self.slot_sanitizer(id), || None, self.owner_dead()) } @@ -659,7 +678,7 @@ impl Graph { (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); for (i, n) in neighbors.iter().enumerate() { - (p.add(S_VECTOR + dims + i * 4) as *mut u32).write_unaligned(n.to_le()); + (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); } *p.add(S_FLAGS) = FLAG_VALID; true diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index bc54ded2c1..e283410a67 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -161,35 +161,54 @@ pub fn insert( let layer0_cap = graph.file.layer0_cap; let m = params.m; - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { - let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; - graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).map_err(|_| InsertError::Wedged)?; - // CAS: a concurrent first insert may have installed an entry already — never clobber - graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); - return Ok(id); - } - let mut stats = SearchStats { visits: 0 }; - let (entry_id, entry_level, entry_dist) = match graph.distance_to(entry_id, &query) { - Some(d) => (entry_id, entry_level, d), - None => { - // The stored entry point is gone (e.g. a mirroring host cleared it without - // re-electing). Self-promoting an edgeless new node here would orphan the whole - // existing graph behind an unreachable root — re-elect from the live graph and - // continue; only a truly empty graph makes this node the first entry. - graph.reelect_entry_point(&[]); - let (re_id, re_level) = graph.file.entry_point(); - match (re_id != NO_ID).then(|| graph.distance_to(re_id, &query)).flatten() { - Some(d) => (re_id, re_level, d), - None => { - let upper_idx = if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; - graph.write_node(id, level, &bytes, scale, inv_mag, &[], upper_idx).map_err(|_| InsertError::Wedged)?; - graph.file.set_entry_point_if_not_better(id, level as u32, NO_ID); - return Ok(id); - } + // Upper entry this insert may already have published for `id` while trying to claim an + // empty graph. The slot points at it, so the join path below must REWRITE that index + // rather than mint a second one: freeing an index a live slot still names would let + // another node adopt it mid-traversal. + let mut published_upper = NO_UPPER; + let mut published = false; + let publish_edgeless = |published: &mut bool, published_upper: &mut u32| -> Result<(), InsertError> { + if *published { + return Ok(()); + } + *published_upper = + if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; + graph.write_node(id, level, &bytes, scale, inv_mag, &[], *published_upper).map_err(|_| InsertError::Wedged)?; + *published = true; + Ok(()) + }; + + // Resolve an entry point to grow from. Bounded because each turn either finishes, joins a + // live entry, or replaces one that is provably gone; the cap only guards a pathological + // insert/delete interleaving that keeps clearing the entry under us. + let mut joined = None; + for _ in 0..8 { + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id == NO_ID { + publish_edgeless(&mut published, &mut published_upper)?; + // Claim only from EMPTY. set_entry_point_if_not_better would install this edgeless + // node over a live equal-or-lower-level entry and orphan the graph behind it; and + // it never reports losing, so a loser used to return an unreachable node. + if graph.file.claim_entry_if_empty(id, level as u32) { + return Ok(id); } + continue; // another racer rooted the graph — join it rather than stand alone + } + if let Some(d) = graph.distance_to(entry_id, &query) { + joined = Some((entry_id, entry_level, d)); + break; } + // The stored entry point is gone (e.g. a mirroring host cleared it without + // re-electing). Self-promoting an edgeless new node here would orphan the whole + // existing graph behind an unreachable root — re-elect from the live graph and + // continue; only a truly empty graph makes this node the first entry. + graph.reelect_entry_point(&[]); + } + let Some((entry_id, entry_level, entry_dist)) = joined else { + publish_edgeless(&mut published, &mut published_upper)?; + graph.file.claim_entry_if_empty(id, level as u32); + return Ok(id); }; let top = level.min(entry_level as u8); let (mut ep, mut ep_dist) = @@ -261,7 +280,13 @@ pub fn insert( .unwrap_or_default() }) .collect(); - graph.write_upper(&levels).unwrap_or(NO_UPPER) + if published_upper != NO_UPPER { + // a lost first-entry claim already published this entry under the slot + graph.rewrite_upper(published_upper, &levels).map_err(|_| InsertError::Wedged)?; + published_upper + } else { + graph.write_upper(&levels).unwrap_or(NO_UPPER) + } } else { NO_UPPER }; @@ -279,7 +304,7 @@ pub fn insert( if (level as u32) > entry_level { // CAS against the observed entry: a concurrent higher-level promotion wins - graph.file.set_entry_point_if_not_better(id, level as u32, entry_id); + graph.file.promote_entry_point(id, level as u32, entry_id); } Ok(id) } diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 2630879d75..fd391976d2 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -219,6 +219,32 @@ pub fn greedy_descend( (current, current_dist) } +/// Resolve a live entry point for a read, repairing a dead one in place. +/// +/// A search that finds the header naming a deleted or sanitized node returns EMPTY, and on a +/// read-mostly table nothing ever repairs it: write-path re-election only runs on delete, and +/// a slot a reader sanitized after its writer died had no delete at all. Repair is bounded to +/// the O(1) previous-entry hint — `reelect_entry_point`'s fallback scan is O(high-water) and +/// would stampede the pool thread that runs every search. +fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Option<(u32, u32, f32)> { + let (entry_id, entry_level) = graph.file.entry_point(); + if entry_id != NO_ID { + if let Some(d) = graph.distance_to(entry_id, query) { + stats.visits += 1; + return Some((entry_id, entry_level, d)); + } + } + let prev = graph.file.previous_entry_point(); + if prev == NO_ID || prev == entry_id { + return None; + } + let level = graph.node_level(prev)?; + let d = graph.distance_to(prev, query)?; + stats.visits += 1; + graph.file.set_entry_point_if_not_better(prev, level as u32, entry_id); + Some((prev, level as u32, d)) +} + /// Full search: greedy descent through upper layers, then beam at layer 0. pub fn search( graph: &Graph, @@ -228,19 +254,10 @@ pub fn search( scratch: &mut SearchScratch, ) -> (Vec<(u32, f32)>, SearchStats) { let mut stats = SearchStats { visits: 0 }; - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { + let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { return (Vec::new(), stats); - } - scratch.begin(graph.file.id_high_water()); - - let entry_dist = match graph.distance_to(entry_id, query) { - Some(d) => { - stats.visits += 1; - d - } - None => return (Vec::new(), stats), }; + scratch.begin(graph.file.id_high_water()); let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, None, u64::MAX); out.truncate(k); @@ -259,18 +276,10 @@ pub fn search_filtered( scratch: &mut SearchScratch, ) -> (Vec<(u32, f32)>, SearchStats) { let mut stats = SearchStats { visits: 0 }; - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { + let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { return (Vec::new(), stats); - } - scratch.begin_public(graph.file.id_high_water()); - let entry_dist = match graph.distance_to(entry_id, query) { - Some(d) => { - stats.visits += 1; - d - } - None => return (Vec::new(), stats), }; + scratch.begin_public(graph.file.id_high_water()); let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); let budget = if filter.is_some() { (ef * filter_expansion) as u64 } else { u64::MAX }; let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, filter, budget); @@ -307,18 +316,10 @@ pub fn search_predicated( scratch: &mut SearchScratch, ) -> (Vec<(u32, f32)>, SearchStats) { let mut stats = SearchStats { visits: 0 }; - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { + let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { return (Vec::new(), stats); - } - scratch.begin_public(graph.file.id_high_water()); - let entry_dist = match graph.distance_to(entry_id, query) { - Some(d) => { - stats.visits += 1; - d - } - None => return (Vec::new(), stats), }; + scratch.begin_public(graph.file.id_high_water()); let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); use std::collections::HashMap; diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs index 57122de035..e20007fe81 100644 --- a/native/hnsw-plane/tests/concurrent.rs +++ b/native/hnsw-plane/tests/concurrent.rs @@ -6,7 +6,7 @@ use hnsw_plane::insert::{insert, InsertParams}; use hnsw_plane::search::{search, SearchScratch}; use hnsw_plane::{Graph, PlaneFile}; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Barrier}; fn vector_for(i: u32, dims: usize) -> Vec { // deterministic distinct unit-ish vectors on a few clusters @@ -16,6 +16,13 @@ fn vector_for(i: u32, dims: usize) -> Vec { let x = ((i as f32 * 0.37 + d as f32 * 1.13).sin() * 0.1) + if d % 7 == cluster { 1.0 } else { 0.0 }; v[d] = x; } + // Per-node signature (unique for i < dims^3). The cluster spike plus 0.1-amplitude noise + // alone leaves every member of a cluster inside int8 quantization noise of every other, so + // a self-query cannot tell "found this node" from "found some other node" — and a + // distance-only assertion over such a corpus passes even when the node is orphaned. + v[(i as usize) % dims] += 0.5; + v[(i as usize / dims) % dims] += 0.35; + v[(i as usize / (dims * dims)) % dims] += 0.22; v } @@ -31,18 +38,25 @@ fn concurrent_insert_search() { let per_writer = 2_000u32; let done = Arc::new(AtomicBool::new(false)); - std::thread::scope(|s| { - for w in 0..writers { - let graph = graph.clone(); - s.spawn(move || { - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..per_writer { - let v = vector_for(w * per_writer + i, dims); - insert(&graph, &v, ¶ms, &mut scratch).expect("insert"); - } - }); - } + // (corpus index, node id): ids come from the plane's own allocator, so writers interleave + // them — a self-query must be checked against the id its insert actually returned + let inserted: Vec<(u32, u32)> = std::thread::scope(|s| { + let writers_done: Vec<_> = (0..writers) + .map(|w| { + let graph = graph.clone(); + s.spawn(move || { + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + let mut mine = Vec::with_capacity(per_writer as usize); + for i in 0..per_writer { + let index = w * per_writer + i; + let v = vector_for(index, dims); + mine.push((index, insert(&graph, &v, ¶ms, &mut scratch).expect("insert"))); + } + mine + }) + }) + .collect(); for _ in 0..4 { let graph = graph.clone(); let done = done.clone(); @@ -70,6 +84,7 @@ fn concurrent_insert_search() { } done_ref.store(true, Ordering::Relaxed); }); + writers_done.into_iter().flat_map(|h| h.join().expect("writer panicked")).collect() }); let total = writers * per_writer; @@ -78,11 +93,12 @@ fn concurrent_insert_search() { // Every stored vector must be found as its own nearest neighbor at generous ef. let mut scratch = SearchScratch::new(); let mut misses = 0; - for i in (0..total).step_by(97) { - let query = Query::new(vector_for(i, dims)); + for &(index, id) in inserted.iter().step_by(97) { + let query = Query::new(vector_for(index, dims)); let (results, _) = search(&graph, &query, 10, 256, &mut scratch); - // identical vectors exist across ids (clusters), so accept any zero-ish distance hit - if !results.iter().any(|&(_, d)| d < 1e-3) { + // by ID, not by distance: this corpus is clustered near-duplicates, so a hit at + // distance ~0 is routinely a DIFFERENT node and would mask an orphaned one + if !results.iter().any(|&(rid, _)| rid == id) { misses += 1; } } @@ -107,3 +123,57 @@ fn concurrent_insert_search() { let _ = std::fs::remove_file(&path); } + +/// Orthogonal per-writer vector: every writer's self-query has exactly one right answer, so a +/// node that lost the first-entry race is unmissable rather than covered by a near-duplicate. +fn axis_vector(writer: u32, dims: usize) -> Vec { + let mut v = vec![0.0f32; dims]; + v[writer as usize % dims] = 1.0; + v +} + +/// The empty-graph entry-point claim, sampled where it actually races. +/// +/// Every writer that observes `entry_point() == NO_ID` publishes an edgeless node before the +/// claim, so a claim that only declines to clobber a winner leaves every loser unreachable: +/// nothing points at it and it is not the entry. One barrier per whole-suite run samples that +/// window about once; the regression needs many small fresh graphs, each racing the FIRST +/// insert, and each asserting reachability by id. +#[test] +fn racing_first_inserts_all_stay_reachable() { + let dims = 32; + let writers = 4u32; + let rounds = 200; + for round in 0..rounds { + let path = std::env::temp_dir().join(format!("hnsw-first-{}-{round}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let graph = Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create"))); + let barrier = Arc::new(Barrier::new(writers as usize)); + let ids: Vec<(u32, u32)> = std::thread::scope(|s| { + let handles: Vec<_> = (0..writers) + .map(|w| { + let graph = graph.clone(); + let barrier = barrier.clone(); + s.spawn(move || { + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + barrier.wait(); + (w, insert(&graph, &axis_vector(w, dims), ¶ms, &mut scratch).expect("insert")) + }) + }) + .collect(); + handles.into_iter().map(|h| h.join().expect("writer panicked")).collect() + }); + + let mut scratch = SearchScratch::new(); + for (w, id) in &ids { + let (results, _) = search(&graph, &Query::new(axis_vector(*w, dims)), 8, 64, &mut scratch); + assert!( + results.iter().any(|&(rid, _)| rid == *id), + "round {round}: writer {w}'s node {id} is unreachable from the entry point (found {results:?})" + ); + } + drop(graph); + let _ = std::fs::remove_file(&path); + } +} diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 005c89860e..68494243e2 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -419,3 +419,79 @@ fn a_wedged_untouched_write_frees_its_upper_entry() { ); let _ = std::fs::remove_file(&path); } + +/// A wedged upper-entry cleanup must not leave the header naming a deleted entry point. +/// +/// `delete_node` tombstones the slot, then rewrites the node's upper entry — a fallible step. +/// With re-election ordered after it, a wedged upper lock returned early and every subsequent +/// search routed through a dead entry (returning nothing) until some insert happened to +/// repair it. The observable, not the header word, is what this asserts. +#[test] +fn a_wedged_upper_cleanup_still_reelects_the_entry_point() { + use std::sync::atomic::Ordering as O; + let dims = 32; + let path = tmp("wedgedelete"); + let _ = std::fs::remove_file(&path); + let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); + let raw = |id: u32, level: u8, neighbors: &[u32], upper: &[Vec]| { + let q = hnsw_plane::distance::quantize_int8(&vector_for(id, dims)); + graph.write_node_raw(id, level, &q.0, q.1, q.2, neighbors, upper).expect("mirror"); + }; + // node 0 is the entry point and the only node with a hierarchy, so it owns upper entry 0 + raw(0, 1, &[1], &[vec![1]]); + raw(1, 0, &[0], &[]); + graph.file.set_entry_point(0, 1); + + let upper_seq = graph.file.upper_seq_atomic(0) as *const _ as usize; + let g2 = graph.clone(); + let held = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let held2 = held.clone(); + let hold = std::thread::spawn(move || { + let seq = unsafe { &*(upper_seq as *const std::sync::atomic::AtomicU32) }; + let g2 = &g2; + let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { + g2.file.tag_is_dead(tag) + }) + .expect("the holder must actually take the lock, or the test proves nothing"); + held2.store(true, O::Release); + std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER + drop(guard); + }); + await_lock(&held); + assert_eq!(graph.delete_node(0), Err(hnsw_plane::seqlock::Wedged), "the held upper lock must wedge the cleanup"); + hold.join().unwrap(); + + assert_eq!(graph.file.entry_point().0, 1, "the entry point must be re-elected before the fallible cleanup"); + let mut scratch = SearchScratch::new(); + let (hits, _) = search(&graph, &Query::new(vector_for(1, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty(), "searches must keep working after a wedged delete of the entry point"); + let _ = std::fs::remove_file(&path); +} + +/// A search must repair an entry point that no writer will: a host that cleared the entry, or +/// a slot a reader sanitized after its writer died, leaves no delete to run the write-path +/// re-election, so on a read-mostly table every search returns empty indefinitely. +#[test] +fn search_repairs_an_entry_point_no_writer_will() { + let dims = 32; + let path = tmp("entryheal"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..200 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let prev = graph.file.previous_entry_point(); + assert_ne!(prev, hnsw_plane::format::NO_ID, "promotions must record a previous-entry hint to repair from"); + + // the entry's slot reads as gone with no delete having run (dead-writer sanitization, or a + // mirroring host clearing the node) — nothing on the write path will ever re-elect + let (entry, _) = graph.file.entry_point(); + graph.clear_node(entry).expect("tombstone the entry slot"); + + let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty(), "a search must self-heal past a dead entry point instead of returning empty"); + assert_ne!(graph.file.entry_point().0, entry, "the repair must be published, not repeated per search"); + let _ = std::fs::remove_file(&path); +} diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index c783ad65e1..9d5d478565 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -1,4 +1,4 @@ -import { closeSync, existsSync, openSync, statSync, unlinkSync } from 'node:fs'; +import { closeSync, existsSync, openSync, rmSync, statSync, unlinkSync } from 'node:fs'; import { cosineDistance, euclideanDistance, dotProductDistance } from './vector.ts'; import { FLOAT32_OPTIONS } from 'msgpackr'; import { loggerWithTag } from '../../utility/logging/logger.ts'; @@ -381,7 +381,13 @@ export class HierarchicalNavigableSmallWorld { unlinkSync(filePath); logger.info?.('deleted the HNSW plane file of an index no longer using nativePlane'); } catch (error: any) { - if (error?.code !== 'ENOENT') logger.warn?.('could not delete the HNSW plane file', error); + if (error?.code !== 'ENOENT') { + // the file survives (Windows EBUSY while another process maps it), and nothing + // mirrors into it from here on: a later re-enable would adopt it at its nonzero + // watermark and silently miss every mutation made while the flag was off + logger.warn?.('could not delete the HNSW plane file; marking it stale', error); + this.invalidatePlaneFile(filePath); + } } } @@ -424,8 +430,11 @@ export class HierarchicalNavigableSmallWorld { const stalePath = planeStalePathFor(filePath); if (existsSync(stalePath)) { try { - unlinkSync(filePath); - unlinkSync(stalePath); + // force: an operator following the documented rollback deletes the .plane file by + // hand and leaves the sidecar; an ENOENT here used to trip the catch below on + // every attach forever, permanently disabling a plane that could just be rebuilt + rmSync(filePath, { force: true }); + rmSync(stalePath, { force: true }); } catch { this.planeRetryAt = now + NODE_COUNT_TTL; return null; @@ -723,6 +732,7 @@ export class HierarchicalNavigableSmallWorld { * inode keeps itself consistent until the schema-change/restart cycle rebuilds everything. */ private disablePlane(error: unknown): void { + const attached = this.plane; this.plane = null; this.planeReady = false; const filePath = this.planeFilePath(); @@ -731,8 +741,8 @@ export class HierarchicalNavigableSmallWorld { unlinkSync(filePath); } catch (unlinkError: any) { if (unlinkError?.code !== 'ENOENT') { - logger.warn?.('could not delete the disabled HNSW plane file; tombstoning it as stale', unlinkError); - this.tombstonePlane(filePath); + logger.warn?.('could not delete the disabled HNSW plane file; marking it stale', unlinkError); + this.invalidatePlaneFile(filePath, attached); } } } @@ -750,6 +760,7 @@ export class HierarchicalNavigableSmallWorld { * database instances. */ resetDerivedStorage(): void { + const attached = this.plane; this.plane = undefined; this.planeReady = false; this.planeRetryAt = 0; @@ -760,16 +771,30 @@ export class HierarchicalNavigableSmallWorld { } catch (error: any) { if (error?.code !== 'ENOENT') { // a stale file that cannot be deleted (e.g. Windows EBUSY while mapped) must not - // be reopened as if current — tombstone it so no process ever adopts it + // be reopened as if current — mark it so no process ever adopts it this.plane = null; - logger.warn?.('could not delete the HNSW plane file; tombstoning it as stale', error); - this.tombstonePlane(filePath); + logger.warn?.('could not delete the HNSW plane file; marking it stale', error); + this.invalidatePlaneFile(filePath, attached); } } } - /** Mark an undeletable plane file stale; getPlane refuses to open a tombstoned plane. */ - private tombstonePlane(filePath: string): void { + /** + * Make an undeletable plane file unadoptable, in band first and then with the `.stale` + * sidecar. Zeroing the watermark under a durability barrier marks the file an incomplete + * initial mirror, which planeSearchReady already refuses and PLANE_INCOMPLETE_REBUILD_MS + * already rebuilds — and unlike the sidecar (an empty file with no directory fsync) it is + * durable and cannot be separated from the plane it invalidates. The sidecar still follows + * because another process may still be mapping this inode and can re-stamp the watermark + * from its own mirror writes; it is checked at attach, before any such writer exists. + */ + private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { + try { + const plane = attached ?? (existsSync(filePath) ? getPlaneBinding()?.open(filePath) : undefined); + plane?.flush(0); + } catch (invalidateError) { + logger.warn?.('could not zero the watermark of the stale HNSW plane file', invalidateError); + } try { closeSync(openSync(planeStalePathFor(filePath), 'w')); } catch (tombstoneError) { diff --git a/resources/search.ts b/resources/search.ts index 3b69c4143c..8244377e8e 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -541,6 +541,9 @@ export function searchByIndex( // once it resolves, exposed as a lazily-resolving iterable — consumable through async // iteration only, like the promise-entry filter paths above. const pending = (searched as Promise).then(processEntries); + // nothing is required to consume this iterable (an aborted request, `limit: 0`), and + // a rejection nobody observed reaches Node's unhandledRejection and exits the process + pending.catch(() => {}); const results: any = new ExtendedIterable(); results.iterate = (options?: { async?: boolean }) => { // fail loudly rather than hand a synchronous consumer promise-shaped iterator @@ -550,17 +553,23 @@ export function searchByIndex( 'This index resolves search results asynchronously; the results must be consumed with async iteration' ); } - let inner: Iterator | null = null; + // ONE iterator per iterate() call, memoized before any next() runs: overlapping + // next() calls must advance a shared cursor, or each builds its own iterator over + // the same array and they both yield entry 0 while entry 1 is skipped + const iteratorPromise = pending.then((entries) => entries[Symbol.iterator]()); + iteratorPromise.catch(() => {}); + let closed = false; return { next() { - if (inner) return Promise.resolve(inner.next()); - return pending.then((entries) => { - inner = entries[Symbol.iterator](); - return inner.next(); - }); + if (closed) return Promise.resolve({ done: true, value: undefined }); + return iteratorPromise.then((inner) => (closed ? { done: true, value: undefined } : inner.next())); }, return(value?: any) { - (inner as any)?.return?.(value); + closed = true; + iteratorPromise.then( + (inner) => (inner as any).return?.(value), + () => {} + ); return Promise.resolve({ done: true, value }); }, }; diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index d625e279fa..a1a6ba293a 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -15,7 +15,7 @@ const fs = require('node:fs'); const { setupTestDBPath } = require('../testUtils'); const { table, resetDatabases } = require('#src/resources/databases'); const { HierarchicalNavigableSmallWorld } = require('#src/resources/indexes/HierarchicalNavigableSmallWorld'); -const { getPlaneBinding } = require('#src/resources/indexes/hnswPlaneBinding'); +const { getPlaneBinding, planeStalePathFor } = require('#src/resources/indexes/hnswPlaneBinding'); const { setMainIsWorker } = require('#js/server/threads/manageThreads'); async function fromAsync(iterable) { @@ -224,6 +224,52 @@ describe('HNSW native plane dual-write', function () { assert.throws(() => [...results], /async/i, 'sync iteration must throw, not loop on promise-shaped results'); }); + it('overlapping next() calls on plane-backed results advance one shared cursor', async () => { + const query = () => ({ + sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, + select: ['id'], + limit: 5, + }); + const sequential = (await fromAsync(PlaneTest.search(query()))).map((record) => record.id); + assert.ok(sequential.length > 2, 'need several results to detect a duplicate or a skip'); + const iterator = PlaneTest.search(query()).iterate({ async: true }); + // both issued before the first resolves: building an iterator per call returned entry 0 + // twice and dropped entry 1 + const [first, second] = await Promise.all([iterator.next(), iterator.next()]); + const seen = [first.value.id, second.value.id]; + for (let next = await iterator.next(); !next.done; next = await iterator.next()) seen.push(next.value.id); + assert.deepEqual(seen, sequential, 'overlapping next() calls must yield the sequential order exactly once'); + }); + + it('an abandoned plane-backed iterable does not raise an unhandled rejection', async () => { + const index = customIndex(); + const unhandled = []; + const onUnhandled = (reason) => unhandled.push(reason); + process.on('unhandledRejection', onUnhandled); + try { + // a post-load step that throws is the reachable way the pending pipeline rejects + index.rescoreResults = () => { + throw new Error('rescore boom'); + }; + const results = PlaneTest.search({ + sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, + select: ['id'], + limit: 5, + }); + // aborted request / limit 0: the consumer walks away without a single next() + await results.iterate({ async: true }).return(); + await new Promise((resolve) => setTimeout(resolve, 100)); + } finally { + delete index.rescoreResults; + process.off('unhandledRejection', onUnhandled); + } + assert.deepEqual( + unhandled.map((reason) => String(reason?.message ?? reason)), + [], + 'an unobserved rejection here exits the process under Node default policy' + ); + }); + it('reopens the same plane file across a restart', async () => { const planePath = customIndex().planeFilePath(); const inodeBefore = fs.statSync(planePath).ino; @@ -323,6 +369,38 @@ describe('HNSW native plane dual-write', function () { await Foreign.dropTable(); }); + it('a stale tombstone left without its plane file rebuilds instead of disabling the plane forever', async () => { + const Orphan = table({ + table: 'PlaneOrphan', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + const index = Orphan.indices.vector.customIndex; + for (let i = 0; i < 30; i++) await Orphan.put(i, { vector: makeVector(i + 40000) }); + const planePath = index.planeFilePath(); + const stalePath = planeStalePathFor(planePath); + try { + // the documented rollback, run by hand: the operator deletes the plane file while a + // tombstone from an earlier undeletable-plane path is still sitting next to it + index.resetDerivedStorage(); + fs.writeFileSync(stalePath, ''); + assert.ok(!fs.existsSync(planePath), 'precondition: tombstone present, plane file gone'); + const results = index.search( + { target: makeVector(40003), comparator: 'sort', distance: 'cosine', ef: 50 }, + { transaction: undefined } + ); + if (typeof results?.then === 'function') await results; + assert.ok(!fs.existsSync(stalePath), 'the tombstone must be cleared, not left to disable the plane forever'); + assert.ok(fs.existsSync(planePath), 'the plane must rebuild once the tombstone is cleared'); + } finally { + fs.rmSync(stalePath, { force: true }); + await Orphan.dropTable(); + } + }); + it('disabling the flag deletes the plane file so a re-enable rebuilds instead of adopting it stale', async () => { const planePath = customIndex().planeFilePath(); assert.ok(fs.existsSync(planePath)); From b434509af344fd49b23194801b9b4d6875a575f5 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 12:09:43 -0600 Subject: [PATCH 46/69] hnsw-plane: entry-point ordering and hint quality from the review round - delete_node re-elects while the node is still readable, so no window publishes a tombstoned entry point, and re-election never elects the node it is replacing. - a re-election that finds no candidate clears the entry only while it still names the node being replaced, instead of erasing an entry a concurrent insert installed. - an insert that cannot resolve an entry point within its retry bound returns an error rather than Ok for a node nothing points at. - the previous-entry hint records only a live displaced node, so a host-mirrored post-delete re-election cannot evict a usable hint. - the undeletable-plane invalidation stores the watermark inline and msyncs on the pool, off the event loop. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/format.rs | 26 ++++++++++- native/hnsw-plane/src/graph.rs | 43 ++++++++++--------- native/hnsw-plane/src/insert.rs | 32 +++++++------- native/hnsw-plane/tests/concurrent.rs | 9 +--- native/hnsw-plane/tests/reopen.rs | 9 ++-- .../HierarchicalNavigableSmallWorld.ts | 13 ++++-- resources/search.ts | 6 +-- 7 files changed, 79 insertions(+), 59 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index d61843dab6..ae9d8bc4db 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -378,9 +378,16 @@ impl PlaneFile { #[inline] fn record_previous_entry(&self, prev_packed: u64, new_id: u32) { let prev_id = (prev_packed & 0xffff_ffff) as u32; - if prev_id != NO_ID && prev_id != new_id { - self.header_atomic_u64(H_ENTRY_PREV).store(prev_packed, Ordering::Release); + if prev_id == NO_ID || prev_id == new_id || (prev_id as u64) >= self.max_nodes { + return; + } + // a hint is only worth keeping while its node is live: the host mirrors a post-delete + // re-election through this same call, and storing the node that died would evict a + // usable hint with one the repair path can never follow + if unsafe { *self.slot_ptr(prev_id).add(S_FLAGS) } != FLAG_VALID { + return; } + self.header_atomic_u64(H_ENTRY_PREV).store(prev_packed, Ordering::Release); } /// Claim the entry point of an EMPTY graph: a strict compare-exchange from the empty @@ -430,6 +437,21 @@ impl PlaneFile { } } + /// Clear the entry point, but only while it still names `expected_id`. A re-election that + /// found no candidate must not erase an entry a concurrent insert installed meanwhile — + /// `set_entry_point_if_not_better(NO_ID, 0, ..)` would, because a level-0 live entry is not + /// "better" than the level-0 clear. + pub fn clear_entry_point_if(&self, expected_id: u32) { + let cell = self.header_atomic_u64(H_ENTRY); + let mut cur = cell.load(Ordering::Acquire); + while (cur & 0xffff_ffff) as u32 == expected_id { + match cell.compare_exchange(cur, NO_ID as u64, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => return, + Err(now) => cur = now, + } + } + } + pub fn set_watermark(&self, txn: u64) { self.header_atomic_u64(H_TXN_WATERMARK).store(txn, Ordering::Release); } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 50a27ad6f2..00cde3a857 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -14,13 +14,14 @@ use crate::seqlock::Wedged; /// Aligned volatile load of a slot/upper-entry field another process may be mutating. /// -/// Ordinary loads of concurrently-written mmap bytes are a data race the optimizer is free to -/// duplicate, split, or sink across the seqlock's validating fence — which would let a reader -/// act on bytes the generation check never covered. Volatile forbids exactly that. The vector -/// is deliberately NOT read this way: `cosine_int8_raw` must stay autovectorized, and a torn -/// vector only perturbs a distance the generation check then discards. Every field this is -/// applied to is naturally aligned (slots are 64-aligned and the neighbor/id arrays are -/// 4-padded by format.rs), so these compile to single loads. +/// This forbids the optimizer from duplicating, splitting, or sinking the load across the +/// seqlock's validating fence, which would let a reader act on bytes the generation check +/// never covered. It does NOT make the access race-free under Rust's memory model — only +/// atomics would, and that is the format change hnsw-native-plane.md §10 records as +/// follow-up. The vector is deliberately not read this way: `cosine_int8_raw` must stay +/// autovectorized, and a torn vector only perturbs a distance the generation check discards. +/// Every field read here is naturally aligned (slots are 64-aligned; the neighbor and upper +/// id arrays are 4-padded by format.rs), so these compile to single loads. #[inline(always)] unsafe fn vread(p: *const T) -> T { p.read_volatile() @@ -532,6 +533,13 @@ impl Graph { if entry_id == id { self.neighbors_into(id, &mut candidates); } + // Re-elect before the tombstone, not after: between marking the slot deleted and + // installing a replacement, every concurrent search routes through a node that reads + // as absent and returns nothing. The node is still live here, so a crash inside the + // window leaves the header naming a live entry either way. + if entry_id == id { + self.reelect_entry_point_replacing(&candidates, id); + } let upper_idx; { let seq = self.file.seq_atomic(id); @@ -549,13 +557,6 @@ impl Graph { *p.add(S_FLAGS) = FLAG_DELETED; } } - // re-elect BEFORE the fallible upper cleanup: the slot is tombstoned above and - // re-election is infallible, so ordering it first means no error path can leave the - // header naming a dead entry — which blinds every search until an insert happens to - // repair it - if entry_id == id { - self.reelect_entry_point_replacing(&candidates, id); - } if upper_idx != NO_UPPER && (upper_idx as u64) < self.file.upper_capacity { // empty the entry under its own lock BEFORE freeing: a traversal that already // read this node's upper_idx must find a dead entry, not one reallocated to a @@ -593,11 +594,7 @@ impl Graph { /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears /// the entry. - pub(crate) fn reelect_entry_point(&self, preferred: &[u32]) { - self.reelect_entry_point_replacing(preferred, crate::format::NO_ID) - } - - fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { + pub(crate) fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { let mut best: Option<(u32, u8)> = None; // the most recently replaced entry point is the best cheap candidate: usually alive, // usually high-level — and it makes the full fallback scan a last resort @@ -608,6 +605,9 @@ impl Graph { } } for &cand in preferred { + if cand == replacing { + continue; // the node on its way out is never its own replacement + } if let Some(level) = self.node_level(cand) { if best.map(|(_, l)| level > l).unwrap_or(true) { best = Some((cand, level)); @@ -617,6 +617,9 @@ impl Graph { if best.is_none() { let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; for cand in 0..hw { + if cand == replacing { + continue; + } if let Some(level) = self.node_level(cand) { if best.map(|(_, l)| level > l).unwrap_or(true) { best = Some((cand, level)); @@ -629,7 +632,7 @@ impl Graph { } match best { Some((cand, level)) => self.file.set_entry_point_if_not_better(cand, level as u32, replacing), - None => self.file.set_entry_point_if_not_better(crate::format::NO_ID, 0, replacing), + None => self.file.clear_entry_point_if(replacing), } } diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index e283410a67..64bbe71092 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -162,10 +162,9 @@ pub fn insert( let m = params.m; let mut stats = SearchStats { visits: 0 }; - // Upper entry this insert may already have published for `id` while trying to claim an - // empty graph. The slot points at it, so the join path below must REWRITE that index - // rather than mint a second one: freeing an index a live slot still names would let - // another node adopt it mid-traversal. + // Upper entry a first-entry claim attempt already published for `id`. Its slot names the + // index, so the join path must rewrite it in place — freeing an index a live slot names + // would let another node adopt it mid-traversal. let mut published_upper = NO_UPPER; let mut published = false; let publish_edgeless = |published: &mut bool, published_upper: &mut u32| -> Result<(), InsertError> { @@ -179,21 +178,21 @@ pub fn insert( Ok(()) }; - // Resolve an entry point to grow from. Bounded because each turn either finishes, joins a - // live entry, or replaces one that is provably gone; the cap only guards a pathological - // insert/delete interleaving that keeps clearing the entry under us. + // Resolve an entry point to grow from. Every turn makes progress — it claims an empty + // graph, joins a live entry, or replaces one that is provably gone — so the cap only + // guards an insert/delete interleaving that keeps clearing the entry under us. let mut joined = None; - for _ in 0..8 { + for _ in 0..16 { let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { publish_edgeless(&mut published, &mut published_upper)?; - // Claim only from EMPTY. set_entry_point_if_not_better would install this edgeless - // node over a live equal-or-lower-level entry and orphan the graph behind it; and - // it never reports losing, so a loser used to return an unreachable node. + // Claim only from EMPTY: a not-worse install would put this edgeless node over a + // live equal-or-lower-level entry and orphan the graph behind it, and it never + // reports losing, so a loser used to return a node nothing points at. if graph.file.claim_entry_if_empty(id, level as u32) { return Ok(id); } - continue; // another racer rooted the graph — join it rather than stand alone + continue; // a racer rooted the graph — join it rather than stand alone } if let Some(d) = graph.distance_to(entry_id, &query) { joined = Some((entry_id, entry_level, d)); @@ -203,12 +202,12 @@ pub fn insert( // re-electing). Self-promoting an edgeless new node here would orphan the whole // existing graph behind an unreachable root — re-elect from the live graph and // continue; only a truly empty graph makes this node the first entry. - graph.reelect_entry_point(&[]); + graph.reelect_entry_point_replacing(&[], entry_id); } + // Reporting success for a node no search can reach is the failure this whole path exists + // to prevent, so an unresolvable entry point is an error the host can retry. let Some((entry_id, entry_level, entry_dist)) = joined else { - publish_edgeless(&mut published, &mut published_upper)?; - graph.file.claim_entry_if_empty(id, level as u32); - return Ok(id); + return Err(InsertError::Wedged); }; let top = level.min(entry_level as u8); let (mut ep, mut ep_dist) = @@ -281,7 +280,6 @@ pub fn insert( }) .collect(); if published_upper != NO_UPPER { - // a lost first-entry claim already published this entry under the slot graph.rewrite_upper(published_upper, &levels).map_err(|_| InsertError::Wedged)?; published_upper } else { diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs index e20007fe81..e3699e2925 100644 --- a/native/hnsw-plane/tests/concurrent.rs +++ b/native/hnsw-plane/tests/concurrent.rs @@ -132,13 +132,8 @@ fn axis_vector(writer: u32, dims: usize) -> Vec { v } -/// The empty-graph entry-point claim, sampled where it actually races. -/// -/// Every writer that observes `entry_point() == NO_ID` publishes an edgeless node before the -/// claim, so a claim that only declines to clobber a winner leaves every loser unreachable: -/// nothing points at it and it is not the entry. One barrier per whole-suite run samples that -/// window about once; the regression needs many small fresh graphs, each racing the FIRST -/// insert, and each asserting reachability by id. +/// Many small fresh graphs, each racing its FIRST insert: that window is where the empty-graph +/// entry-point claim races, and a single barrier in a long build samples it about once. #[test] fn racing_first_inserts_all_stay_reachable() { let dims = 32; diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 68494243e2..4aa735006c 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -420,12 +420,9 @@ fn a_wedged_untouched_write_frees_its_upper_entry() { let _ = std::fs::remove_file(&path); } -/// A wedged upper-entry cleanup must not leave the header naming a deleted entry point. -/// -/// `delete_node` tombstones the slot, then rewrites the node's upper entry — a fallible step. -/// With re-election ordered after it, a wedged upper lock returned early and every subsequent -/// search routed through a dead entry (returning nothing) until some insert happened to -/// repair it. The observable, not the header word, is what this asserts. +/// A wedged upper-entry cleanup must not leave the header naming a deleted entry point: the +/// cleanup is fallible, so an early return there strands every search on a dead entry. Asserts +/// the observable (searches still return hits), not the header word. #[test] fn a_wedged_upper_cleanup_still_reelects_the_entry_point() { use std::sync::atomic::Ordering as O; diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 9d5d478565..3001edf919 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -430,9 +430,8 @@ export class HierarchicalNavigableSmallWorld { const stalePath = planeStalePathFor(filePath); if (existsSync(stalePath)) { try { - // force: an operator following the documented rollback deletes the .plane file by - // hand and leaves the sidecar; an ENOENT here used to trip the catch below on - // every attach forever, permanently disabling a plane that could just be rebuilt + // force: either artifact may already be gone (the documented rollback deletes the + // plane file by hand), and an ENOENT here disables the plane on every later attach rmSync(filePath, { force: true }); rmSync(stalePath, { force: true }); } catch { @@ -791,7 +790,13 @@ export class HierarchicalNavigableSmallWorld { private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { try { const plane = attached ?? (existsSync(filePath) ? getPlaneBinding()?.open(filePath) : undefined); - plane?.flush(0); + // the header store takes effect for every process mapping the file immediately; the + // msync behind it goes to the pool, because on a multi-GB plane it would otherwise + // freeze this worker's event loop + plane?.setWatermark(0); + plane + ?.flushAsync(0) + .catch((flushError) => logger.warn?.('could not persist the stale HNSW plane watermark', flushError)); } catch (invalidateError) { logger.warn?.('could not zero the watermark of the stale HNSW plane file', invalidateError); } diff --git a/resources/search.ts b/resources/search.ts index 8244377e8e..225d45469d 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -553,9 +553,9 @@ export function searchByIndex( 'This index resolves search results asynchronously; the results must be consumed with async iteration' ); } - // ONE iterator per iterate() call, memoized before any next() runs: overlapping - // next() calls must advance a shared cursor, or each builds its own iterator over - // the same array and they both yield entry 0 while entry 1 is skipped + // one shared iterator per iterate() call: overlapping next() calls must advance the + // same cursor, or each builds its own over the same array and entry 0 is yielded + // twice while entry 1 is skipped const iteratorPromise = pending.then((entries) => entries[Symbol.iterator]()); iteratorPromise.catch(() => {}); let closed = false; From 9db12189dc7e00e242ec6d58d6b6acc2e2d4cb07 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 12:38:59 -0600 Subject: [PATCH 47/69] hnsw-plane: durable plane invalidation and a repair that cannot lose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Invalidation ordering: PlaneFile::invalidate zeroes the watermark and msyncs the header page alone, and the host calls it synchronously before creating the .stale sidecar. The sidecar is an empty file whose directory entry is never fsynced, so queuing an async whole-map flush and writing the sidecar first let a power loss keep the old nonzero watermark and lose the only marker — the next process then adopted a plane missing every mutation made while mirroring was off. Skipping the data flush is sound because the data is being discarded and lowering the watermark is the safe direction; it is also what makes a synchronous barrier affordable on a multi-GB plane. Entry repair: the previous-entry hint is one slot and can itself be dead (promote over a node, then lose both), which left every later search returning empty. A bounded probe of the dense low id range now backs it up, capped so a read never pays the write path's O(high-water) scan. The repair also publishes through replace_entry_if — strict on the entry it observed dead — because a not-worse install would displace a live level-0 root a concurrent first insert had just claimed, orphaning it. The hint's own liveness check reads FLAG_VALID volatile like every other field a concurrent writer mutates; it sits outside the slot seqlock, so a retry cannot catch a tear. §10 now records the atomic-slot-payload debt the volatile reads bound but do not discharge, which graph.rs cites. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011W6mChyPAUAoKEfxV1bSAo --- hnsw-native-plane.md | 8 ++ native/hnsw-plane/src/format.rs | 37 +++++++- native/hnsw-plane/src/graph.rs | 21 ++++ native/hnsw-plane/src/insert.rs | 10 +- native/hnsw-plane/src/napi.rs | 9 ++ native/hnsw-plane/src/search.rs | 37 +++++--- native/hnsw-plane/tests/reopen.rs | 95 +++++++++++++++++++ .../HierarchicalNavigableSmallWorld.ts | 29 +++--- resources/indexes/hnswPlaneBinding.ts | 2 + resources/search.ts | 3 +- unitTests/resources/vectorIndexPlane.test.js | 39 ++++++++ 11 files changed, 257 insertions(+), 33 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 5b4df41c94..92e229b969 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -279,6 +279,14 @@ Decided (Kris, 2026-08-31): Open: +- **Atomic slot payloads.** Fields a concurrent reader acts on (flags, level, degree, scale, + invMag, neighbor and upper ids) are read through aligned `read_volatile`, which forbids the + reload/split/sink across the seqlock's validating fence that `lto = true, codegen-units = 1` + otherwise licenses. That is not the same as being race-free under Rust's memory model: only + making those fields `AtomicU8`/`AtomicU16`/`AtomicU32` in the slot layout would be, and that + is a format change deferred past phase 1. The stored vector stays an ordinary load on + purpose — `cosine_int8_raw` must keep autovectorizing, and a torn vector only perturbs a + distance the generation check discards. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index ae9d8bc4db..f1d5b77b62 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -384,7 +384,9 @@ impl PlaneFile { // a hint is only worth keeping while its node is live: the host mirrors a post-delete // re-election through this same call, and storing the node that died would evict a // usable hint with one the repair path can never follow - if unsafe { *self.slot_ptr(prev_id).add(S_FLAGS) } != FLAG_VALID { + // volatile like every other read of a field a concurrent writer mutates (graph.rs's + // `vread`): this one is outside the slot seqlock, so the retry cannot even catch a tear + if unsafe { self.slot_ptr(prev_id).add(S_FLAGS).read_volatile() } != FLAG_VALID { return; } self.header_atomic_u64(H_ENTRY_PREV).store(prev_packed, Ordering::Release); @@ -437,6 +439,24 @@ impl PlaneFile { } } + /// Install `(id, level)` ONLY while the entry still names `expected_id`. The read-side + /// repair publishes through this rather than `set_entry_point_if_not_better`: the entry it + /// is replacing is dead, so "not worse" is the wrong test — a concurrent first insert that + /// just claimed the header with a level-0 root would lose to a higher-level repair + /// candidate and be orphaned with nothing pointing at it. + pub fn replace_entry_if(&self, expected_id: u32, id: u32, level: u32) -> bool { + let cell = self.header_atomic_u64(H_ENTRY); + let new = (id as u64) | ((level as u64) << 32); + let mut cur = cell.load(Ordering::Acquire); + while (cur & 0xffff_ffff) as u32 == expected_id { + match cell.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => return true, + Err(now) => cur = now, + } + } + false + } + /// Clear the entry point, but only while it still names `expected_id`. A re-election that /// found no candidate must not erase an entry a concurrent insert installed meanwhile — /// `set_entry_point_if_not_better(NO_ID, 0, ..)` would, because a level-0 live entry is not @@ -644,4 +664,19 @@ impl PlaneFile { unsafe { *(self.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 1 }; self.map.flush_range(0, HEADER_SIZE) } + + /// Mark the plane an incomplete mirror, durably, and nothing else: zero the watermark and + /// msync the header page alone. Every opener then refuses to search it and rebuilds. + /// + /// Deliberately NOT `flush_with_watermark(Some(0))`: that writes the whole mapping back + /// first, and the caller invalidating a multi-GB plane cannot pay a full msync inline — + /// which is why the host used to queue an async flush and create its `.stale` sidecar + /// before the flush had happened at all. Skipping the data flush is sound because the + /// data is being discarded, and because lowering the watermark is the safe direction: + /// the ordering hazard `flush_with_watermark` exists to prevent is a NEW watermark over + /// missing data, never an old one over durable data. + pub fn invalidate(&self) -> io::Result<()> { + self.set_watermark(0); + self.map.flush_range(0, HEADER_SIZE) + } } diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 00cde3a857..89b86f8055 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -594,6 +594,27 @@ impl Graph { /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears /// the entry. + /// Highest-level live node among the first `limit` ids, skipping `skip`. The read-side + /// repair's last resort: `reelect_entry_point_replacing` scans to the high-water mark, which + /// a search on the shared pool thread cannot afford, but ids are allocated densely from 0 so + /// a bounded prefix is where a live graph's nodes are. Best-level rather than first-live, + /// because a level-0 entry degrades every later search to a layer-0-only beam. + pub(crate) fn probe_for_entry(&self, limit: u32, skip: u32) -> Option<(u32, u8)> { + let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; + let mut best: Option<(u32, u8)> = None; + for cand in 0..hw.min(limit) { + if cand == skip { + continue; + } + if let Some(level) = self.node_level(cand) { + if best.map(|(_, l)| level > l).unwrap_or(true) { + best = Some((cand, level)); + } + } + } + best + } + pub(crate) fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { let mut best: Option<(u32, u8)> = None; // the most recently replaced entry point is the best cheap candidate: usually alive, diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 64bbe71092..52027e5f3a 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -186,9 +186,9 @@ pub fn insert( let (entry_id, entry_level) = graph.file.entry_point(); if entry_id == NO_ID { publish_edgeless(&mut published, &mut published_upper)?; - // Claim only from EMPTY: a not-worse install would put this edgeless node over a - // live equal-or-lower-level entry and orphan the graph behind it, and it never - // reports losing, so a loser used to return a node nothing points at. + // Claim only from EMPTY, and only the winner returns: a not-worse install would put + // this edgeless node over a live equal-or-lower-level entry and orphan the graph + // behind it, and it cannot report losing, which a loser must know to join instead. if graph.file.claim_entry_if_empty(id, level as u32) { return Ok(id); } @@ -204,8 +204,8 @@ pub fn insert( // continue; only a truly empty graph makes this node the first entry. graph.reelect_entry_point_replacing(&[], entry_id); } - // Reporting success for a node no search can reach is the failure this whole path exists - // to prevent, so an unresolvable entry point is an error the host can retry. + // An unresolvable entry point is an error the host retries: Ok here would report success + // for a node no search can reach. let Some((entry_id, entry_level, entry_dist)) = joined else { return Err(InsertError::Wedged); }; diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 910dc2ddc0..1bc7436ebd 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -480,4 +480,13 @@ impl Plane { pub fn flush(&self, watermark: Option) -> Result<()> { self.graph.file.flush_with_watermark(watermark.map(|w| w as u64)).map_err(|e| Error::from_reason(e.to_string())) } + + /// Durably mark this plane an incomplete mirror: zero the watermark and msync the header + /// page alone, so a host disabling a plane it cannot delete has the mark on disk before it + /// writes any out-of-band tombstone. Synchronous by design — it is a 4 KB msync, not the + /// whole-mapping writeback `flush` performs. + #[napi] + pub fn invalidate(&self) -> Result<()> { + self.graph.file.invalidate().map_err(|e| Error::from_reason(e.to_string())) + } } diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index fd391976d2..6bc33bf1b1 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -219,13 +219,24 @@ pub fn greedy_descend( (current, current_dist) } +/// Slots a read-side repair may probe when the previous-entry hint is dead too. Bounded so a +/// search never pays the write path's O(high-water) re-election scan. +const REPAIR_PROBE_LIMIT: u32 = 1024; + /// Resolve a live entry point for a read, repairing a dead one in place. /// /// A search that finds the header naming a deleted or sanitized node returns EMPTY, and on a /// read-mostly table nothing ever repairs it: write-path re-election only runs on delete, and -/// a slot a reader sanitized after its writer died had no delete at all. Repair is bounded to -/// the O(1) previous-entry hint — `reelect_entry_point`'s fallback scan is O(high-water) and -/// would stampede the pool thread that runs every search. +/// a slot a reader sanitized after its writer died had no delete at all. +/// +/// The candidate is the O(1) previous-entry hint, then a bounded probe. The hint is a single +/// slot and can itself be dead — promote B over C, delete B, then lose C to a dead writer, and +/// the hint names a deleted node — so falling back is what keeps that from returning empty +/// forever. The probe is capped at `REPAIR_PROBE_LIMIT` because +/// `reelect_entry_point_replacing`'s scan runs to the high-water mark, and every search paying +/// that would stampede the pool thread they all share; ids are dense from 0, so a live graph +/// resolves in the first few slots, and the repair publishes, so only the first search after a +/// wedge pays even that. fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Option<(u32, u32, f32)> { let (entry_id, entry_level) = graph.file.entry_point(); if entry_id != NO_ID { @@ -234,15 +245,19 @@ fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Optio return Some((entry_id, entry_level, d)); } } - let prev = graph.file.previous_entry_point(); - if prev == NO_ID || prev == entry_id { - return None; - } - let level = graph.node_level(prev)?; - let d = graph.distance_to(prev, query)?; + let hint = graph.file.previous_entry_point(); + let candidate = (hint != NO_ID && hint != entry_id) + .then(|| graph.node_level(hint).map(|level| (hint, level))) + .flatten() + .or_else(|| graph.probe_for_entry(REPAIR_PROBE_LIMIT, entry_id)); + let (id, level) = candidate?; + let d = graph.distance_to(id, query)?; stats.visits += 1; - graph.file.set_entry_point_if_not_better(prev, level as u32, entry_id); - Some((prev, level as u32, d)) + // Strict on the entry we observed dead, NOT a not-worse install: between the read above and + // here a first insert can have claimed the header with its own live level-0 root, and + // replacing that with a higher-level candidate would orphan a node nothing else points at. + graph.file.replace_entry_if(entry_id, id, level as u32); + Some((id, level as u32, d)) } /// Full search: greedy descent through upper layers, then beam at layer 0. diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 4aa735006c..f6c5227fcd 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -492,3 +492,98 @@ fn search_repairs_an_entry_point_no_writer_will() { assert_ne!(graph.file.entry_point().0, entry, "the repair must be published, not repeated per search"); let _ = std::fs::remove_file(&path); } + +/// `invalidate` demotes a plane that already looks like a complete mirror back to "incomplete, +/// rebuild me", and reports barrier failure to its caller instead of into a dropped promise — +/// which is what lets the host order it before creating a `.stale` sidecar. (Durability itself +/// is not observable in-process: the mapping is MAP_SHARED, so every store is already visible to +/// a reopen and to `read()` whether or not the msync ran. The ordering that a crash would expose +/// is asserted on the host side, in `vectorIndexPlane.test.js`.) +#[test] +fn invalidate_demotes_a_complete_looking_mirror_and_reports_failure() { + let dims = 32; + let path = tmp("invalidate"); + let _ = std::fs::remove_file(&path); + { + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..50 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + graph.file.flush_with_watermark(Some(4_096)).expect("barrier"); + assert_eq!(graph.file.watermark(), 4_096, "precondition: a complete-looking mirror"); + graph.file.invalidate().expect("invalidate must report its barrier, not swallow it"); + assert_eq!(graph.file.watermark(), 0, "invalidation must mark the mirror incomplete in band"); + } + let reopened = PlaneFile::open(&path).expect("reopen"); + assert_eq!(reopened.watermark(), 0, "a fresh opener must see the incomplete mark, not the old stamp"); + let _ = std::fs::remove_file(&path); +} + +/// The hint is one slot and can die too: promote over a node, then lose BOTH that node and the +/// entry it was promoted over. Without the bounded probe the repair has nowhere left to look and +/// every later search returns empty although most of the graph is live. +#[test] +fn search_repairs_an_entry_point_whose_hint_is_dead_too() { + let dims = 32; + let path = tmp("entryhealdeadhint"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..200 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let hint = graph.file.previous_entry_point(); + assert_ne!(hint, hnsw_plane::format::NO_ID, "precondition: a hint to invalidate"); + let (entry, _) = graph.file.entry_point(); + + // both sanitized with no delete having run, so no write-path re-election ever happens and + // the hint the repair would follow names a node that reads as gone + graph.clear_node(hint).expect("tombstone the hint slot"); + graph.clear_node(entry).expect("tombstone the entry slot"); + + let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty(), "a dead hint must fall back to the bounded probe, not return empty forever"); + let repaired = graph.file.entry_point().0; + assert_ne!(repaired, entry, "the repair must be published"); + assert_ne!(repaired, hint, "the repair must not publish the dead hint"); + let _ = std::fs::remove_file(&path); +} + +/// A repair publishes with a strict CAS on the entry it observed dead. A first insert that +/// claims the header in between owns the graph, and a higher-level repair candidate must lose to +/// it — installing the candidate would leave that insert's node with nothing pointing at it. +#[test] +fn a_repair_never_displaces_a_root_installed_while_it_ran() { + let dims = 32; + let path = tmp("entryhealrace"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..64 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let (observed, _) = graph.file.entry_point(); + let candidate = (0..64u32) + .find(|&id| id != observed && id != 7 && graph.read_node(id).is_some()) + .expect("a live repair candidate"); + let candidate_level = graph.read_node(candidate).expect("live").level; + assert!(graph.read_node(7).is_some(), "precondition: the racing root is a live node"); + + // the interleaving a repair races: the header no longer names the entry it read + graph.file.set_entry_point(7, 0); + assert!( + !graph.file.replace_entry_if(observed, candidate, candidate_level as u32), + "a repair must not publish over an entry installed after it read the dead one" + ); + assert_eq!(graph.file.entry_point().0, 7, "the root installed meanwhile stays"); + + // and it does publish when nothing raced it + let (current, _) = graph.file.entry_point(); + assert!(graph.file.replace_entry_if(current, candidate, candidate_level as u32)); + assert_eq!(graph.file.entry_point().0, candidate); + let _ = std::fs::remove_file(&path); +} diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 3001edf919..49370cfa4d 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -779,24 +779,25 @@ export class HierarchicalNavigableSmallWorld { } /** - * Make an undeletable plane file unadoptable, in band first and then with the `.stale` - * sidecar. Zeroing the watermark under a durability barrier marks the file an incomplete - * initial mirror, which planeSearchReady already refuses and PLANE_INCOMPLETE_REBUILD_MS - * already rebuilds — and unlike the sidecar (an empty file with no directory fsync) it is - * durable and cannot be separated from the plane it invalidates. The sidecar still follows - * because another process may still be mapping this inode and can re-stamp the watermark - * from its own mirror writes; it is checked at attach, before any such writer exists. + * Make an undeletable plane file unadoptable: in band first, then with the `.stale` sidecar. + * + * `invalidate()` zeroes the watermark and msyncs the header page alone, so the file reads as + * an incomplete initial mirror — which planeSearchReady already refuses and + * PLANE_INCOMPLETE_REBUILD_MS already rebuilds. It is synchronous on purpose: the sidecar is + * an empty file whose directory entry is never fsynced, so creating it before the watermark + * was durable would let a power loss keep the old nonzero watermark and lose the sidecar, + * and the next process would adopt a plane missing every mutation made while mirroring was + * off. A whole-mapping flush would give the same ordering but cannot run inline on a + * multi-GB plane, which is why this is a 4 KB header barrier rather than `flush(0)`. + * + * The sidecar still follows, because another process may still be mapping this inode and can + * re-stamp the watermark from its own mirror writes; it is checked at attach, before any + * such writer exists. */ private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { try { const plane = attached ?? (existsSync(filePath) ? getPlaneBinding()?.open(filePath) : undefined); - // the header store takes effect for every process mapping the file immediately; the - // msync behind it goes to the pool, because on a multi-GB plane it would otherwise - // freeze this worker's event loop - plane?.setWatermark(0); - plane - ?.flushAsync(0) - .catch((flushError) => logger.warn?.('could not persist the stale HNSW plane watermark', flushError)); + plane?.invalidate(); } catch (invalidateError) { logger.warn?.('could not zero the watermark of the stale HNSW plane file', invalidateError); } diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index 153845e53d..f8df28c3ad 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -60,6 +60,8 @@ export interface HnswPlane { setWatermark(txn: number): void; flush(watermark?: number): void; flushAsync(watermark?: number): Promise; + /** Zero the watermark and msync the header page alone — a 4 KB barrier, not a full flush. */ + invalidate(): void; } export interface HnswPlaneConstructor { diff --git a/resources/search.ts b/resources/search.ts index 225d45469d..fb360101a6 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -554,8 +554,7 @@ export function searchByIndex( ); } // one shared iterator per iterate() call: overlapping next() calls must advance the - // same cursor, or each builds its own over the same array and entry 0 is yielded - // twice while entry 1 is skipped + // same cursor const iteratorPromise = pending.then((entries) => entries[Symbol.iterator]()); iteratorPromise.catch(() => {}); let closed = false; diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index a1a6ba293a..7a62896934 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -369,6 +369,45 @@ describe('HNSW native plane dual-write', function () { await Foreign.dropTable(); }); + it('an undeletable plane is marked incomplete in band before its stale sidecar is created', async () => { + const Undeletable = table({ + table: 'PlaneUndeletable', + database: DB, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }); + const index = Undeletable.indices.vector.customIndex; + for (let i = 0; i < 30; i++) await Undeletable.put(i, { vector: makeVector(i + 60000) }); + const planePath = index.planeFilePath(); + const stalePath = planeStalePathFor(planePath); + fs.rmSync(stalePath, { force: true }); + const plane = getPlaneBinding().open(planePath); + plane.setWatermark(4096); // a plane that would read as a complete mirror on the next attach + const order = []; + // a stand-in rather than a monkeypatch: the binding's methods live on a non-writable + // prototype, so assigning over one silently does nothing in sloppy mode + const observed = { + invalidate() { + // the sidecar must not exist yet: it is an empty file whose directory entry is never + // fsynced, so creating it first lets a power loss keep the old watermark and lose the + // only marker + order.push(fs.existsSync(stalePath) ? 'sidecar-first' : 'in-band-first'); + plane.invalidate(); + }, + }; + try { + index.invalidatePlaneFile(planePath, observed); + assert.deepEqual(order, ['in-band-first'], 'the durable in-band mark must complete before the sidecar'); + assert.ok(fs.existsSync(stalePath), 'the sidecar still follows, for a process that cannot map the file'); + assert.equal(getPlaneBinding().open(planePath).getWatermark(), 0, 'the plane must read as an incomplete mirror'); + } finally { + fs.rmSync(stalePath, { force: true }); + await Undeletable.dropTable(); + } + }); + it('a stale tombstone left without its plane file rebuilds instead of disabling the plane forever', async () => { const Orphan = table({ table: 'PlaneOrphan', From b8912149573995524997a02a788ff5f30e7c1f48 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 12:51:27 -0600 Subject: [PATCH 48/69] hnsw-plane: probe the whole id range for a repair candidate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repair's last-resort probe scanned a fixed low prefix on the claim that ids are dense from 0. They are not: Harper allocates node ids monotonically through Atomics.add and never reuses them, so a table that has churned has its entire low prefix tombstoned and only its newest ids live — the probe would find nothing there and every search would return empty forever, which is the failure the probe exists to prevent. It now walks down from the newest id with a stride spanning the whole range, so it assumes nothing about where the live nodes are: the crate's own freelist does reuse ids and keeps live nodes low, and striding covers both. Same probe budget. probe_for_entry had also been inserted between reelect_entry_point_- replacing's doc comment and its body, so it carried documentation for parameters it does not have. replace_entry_if's comment claimed more than the code: it compares the id, not the incarnation, so under freelist reuse it can match a different node in the same slot. That is a routing-quality window, not a lost node — the edgeless claimer it names is structurally excluded, since claim_entry_if_empty fires only from NO_ID. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011W6mChyPAUAoKEfxV1bSAo --- native/hnsw-plane/src/format.rs | 11 +++++-- native/hnsw-plane/src/graph.rs | 50 ++++++++++++++++++++----------- native/hnsw-plane/src/search.rs | 17 ++++------- native/hnsw-plane/tests/reopen.rs | 32 ++++++++++++++++++++ 4 files changed, 79 insertions(+), 31 deletions(-) diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs index f1d5b77b62..3241b63e91 100644 --- a/native/hnsw-plane/src/format.rs +++ b/native/hnsw-plane/src/format.rs @@ -441,9 +441,14 @@ impl PlaneFile { /// Install `(id, level)` ONLY while the entry still names `expected_id`. The read-side /// repair publishes through this rather than `set_entry_point_if_not_better`: the entry it - /// is replacing is dead, so "not worse" is the wrong test — a concurrent first insert that - /// just claimed the header with a level-0 root would lose to a higher-level repair - /// candidate and be orphaned with nothing pointing at it. + /// is replacing is dead, so "not worse" is the wrong test — a level-0 root installed while + /// the repair ran would lose to a higher-level candidate and be orphaned. + /// + /// It compares the id, not the incarnation, so under the crate's own freelist reuse it can + /// match a different node that took the same slot. That is a routing-quality window, not a + /// lost node: the value it could displace is a live edged node, never the edgeless claimer + /// (`claim_entry_if_empty` fires only from NO_ID, which no reuse can produce). Harper's host + /// ids are monotonic and never reused, so this cannot arise there at all. pub fn replace_entry_if(&self, expected_id: u32, id: u32, level: u32) -> bool { let cell = self.header_atomic_u64(H_ENTRY); let new = (id as u64) | ((level as u64) << 32); diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 89b86f8055..714b6ce4cd 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -588,33 +588,49 @@ impl Graph { }, self.slot_sanitizer(id), || None, self.owner_dead()) } - /// Pick a new entry point: the highest-level live node among `preferred`, else the - /// highest-level live node found scanning the id range (level reads only — no per-node - /// vector copies; still O(high-water), which only runs when an entry point vanished - /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a - /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears - /// the entry. - /// Highest-level live node among the first `limit` ids, skipping `skip`. The read-side + /// Highest-level live node among at most `limit` probes, skipping `skip`. The read-side /// repair's last resort: `reelect_entry_point_replacing` scans to the high-water mark, which - /// a search on the shared pool thread cannot afford, but ids are allocated densely from 0 so - /// a bounded prefix is where a live graph's nodes are. Best-level rather than first-live, - /// because a level-0 entry degrades every later search to a layer-0-only beam. + /// a search on the shared pool thread cannot afford. + /// + /// It walks DOWN from the newest id with a stride that spans the whole range, so it makes no + /// assumption about where the live nodes are. Both directions matter: Harper allocates node + /// ids monotonically and never reuses them, so a churned table has its whole low prefix + /// tombstoned and only the newest ids are live — while the crate's own freelist does reuse + /// ids, which keeps live nodes low. Striding covers both, and a graph with any appreciable + /// live fraction is found in the first few probes either way. + /// + /// Best-level rather than first-live, because a level-0 entry degrades every later search to + /// a layer-0-only beam. pub(crate) fn probe_for_entry(&self, limit: u32, skip: u32) -> Option<(u32, u8)> { let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; + if hw == 0 || limit == 0 { + return None; + } + let stride = (hw / limit).max(1); let mut best: Option<(u32, u8)> = None; - for cand in 0..hw.min(limit) { - if cand == skip { - continue; - } - if let Some(level) = self.node_level(cand) { - if best.map(|(_, l)| level > l).unwrap_or(true) { - best = Some((cand, level)); + let mut cand = hw - 1; + for _ in 0..limit { + if cand != skip { + if let Some(level) = self.node_level(cand) { + if best.map(|(_, l)| level > l).unwrap_or(true) { + best = Some((cand, level)); + } } } + if cand < stride { + break; + } + cand -= stride; } best } + /// Pick a new entry point: the highest-level live node among `preferred`, else the + /// highest-level live node found scanning the id range (level reads only — no per-node + /// vector copies; still O(high-water), which only runs when an entry point vanished + /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a + /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears + /// the entry. pub(crate) fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { let mut best: Option<(u32, u8)> = None; // the most recently replaced entry point is the best cheap candidate: usually alive, diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 6bc33bf1b1..838dd8aed4 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -229,14 +229,10 @@ const REPAIR_PROBE_LIMIT: u32 = 1024; /// read-mostly table nothing ever repairs it: write-path re-election only runs on delete, and /// a slot a reader sanitized after its writer died had no delete at all. /// -/// The candidate is the O(1) previous-entry hint, then a bounded probe. The hint is a single -/// slot and can itself be dead — promote B over C, delete B, then lose C to a dead writer, and -/// the hint names a deleted node — so falling back is what keeps that from returning empty -/// forever. The probe is capped at `REPAIR_PROBE_LIMIT` because -/// `reelect_entry_point_replacing`'s scan runs to the high-water mark, and every search paying -/// that would stampede the pool thread they all share; ids are dense from 0, so a live graph -/// resolves in the first few slots, and the repair publishes, so only the first search after a -/// wedge pays even that. +/// The candidate is the O(1) previous-entry hint, then a probe capped at `REPAIR_PROBE_LIMIT` — +/// the hint is a single slot and can be dead itself. The cap is what keeps a read off the write +/// path's O(high-water) scan on the pool thread every search shares, and the repair publishes, +/// so only the first search after a wedge pays even the probe. fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Option<(u32, u32, f32)> { let (entry_id, entry_level) = graph.file.entry_point(); if entry_id != NO_ID { @@ -253,9 +249,8 @@ fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Optio let (id, level) = candidate?; let d = graph.distance_to(id, query)?; stats.visits += 1; - // Strict on the entry we observed dead, NOT a not-worse install: between the read above and - // here a first insert can have claimed the header with its own live level-0 root, and - // replacing that with a higher-level candidate would orphan a node nothing else points at. + // Strict on the entry we observed dead, not a not-worse install: a live level-0 root claimed + // since the read above must win, or it is orphaned with nothing pointing at it. graph.file.replace_entry_if(entry_id, id, level as u32); Some((id, level as u32, d)) } diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index f6c5227fcd..67c4fee143 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -552,6 +552,38 @@ fn search_repairs_an_entry_point_whose_hint_is_dead_too() { let _ = std::fs::remove_file(&path); } +/// Harper allocates node ids monotonically and never reuses them, so a table that has churned +/// has its whole low prefix tombstoned and only the newest ids live. A repair that probed a +/// fixed prefix would find nothing there and every search would return empty forever. +#[test] +fn search_repairs_an_entry_point_in_a_churned_graph_whose_low_ids_are_all_dead() { + let dims = 32; + let path = tmp("entryhealchurn"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..1_200 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + // every id a prefix probe would reach is gone, as it is for any long-lived churned table + for id in 0..1_100u32 { + let _ = graph.clear_node(id); + } + let (entry, _) = graph.file.entry_point(); + let hint = graph.file.previous_entry_point(); + let _ = graph.clear_node(entry); + if hint != hnsw_plane::format::NO_ID { + let _ = graph.clear_node(hint); + } + + let (hits, _) = search(&graph, &Query::new(vector_for(1_150, dims)), 5, 64, &mut scratch); + assert!(!hits.is_empty(), "the probe must reach the live tail, not only a dead low prefix"); + let repaired = graph.file.entry_point().0; + assert!(graph.read_node(repaired).is_some(), "the repair must publish a live node"); + let _ = std::fs::remove_file(&path); +} + /// A repair publishes with a strict CAS on the entry it observed dead. A first insert that /// claims the header in between owns the graph, and a higher-level repair candidate must lose to /// it — installing the candidate would leave that insert's node with nothing pointing at it. From faf1f99a549098972513c8785e392d42f0ec7039 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 12:57:51 -0600 Subject: [PATCH 49/69] hnsw-plane: rotate the repair probe so it cannot skip a residue class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fixed start walked hw-1, hw-1-stride, ... forever, which is one residue class of the stride. A live graph lying entirely between those samples was not merely missed once — it was invisible to every later repair too, which is the silent-empty-results mode the probe exists to prevent, reached by a different route than the low-prefix assumption the previous commit fixed. The start now rotates per call, so stride consecutive repairs cover every id while each stays capped at the same probe budget. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011W6mChyPAUAoKEfxV1bSAo --- native/hnsw-plane/src/graph.rs | 30 +++++++++++++--------- native/hnsw-plane/tests/reopen.rs | 41 +++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 11 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 714b6ce4cd..eaecaf0d86 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -27,6 +27,9 @@ unsafe fn vread(p: *const T) -> T { p.read_volatile() } +/// Rotates `probe_for_entry`'s starting offset so consecutive repairs sample different ids. +static PROBE_ROTATION: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); + pub struct Graph { pub file: PlaneFile, } @@ -589,26 +592,31 @@ impl Graph { } /// Highest-level live node among at most `limit` probes, skipping `skip`. The read-side - /// repair's last resort: `reelect_entry_point_replacing` scans to the high-water mark, which - /// a search on the shared pool thread cannot afford. + /// repair's last resort, bounded because `reelect_entry_point_replacing`'s scan runs to the + /// high-water mark and a search on the shared pool thread cannot afford it. + /// + /// Walks down from the newest id with a stride spanning the whole range, so it assumes + /// nothing about where the live nodes sit: Harper allocates ids monotonically and never + /// reuses them, so a churned table's low prefix is all tombstones, while the crate's own + /// freelist reuses ids and keeps live nodes low. /// - /// It walks DOWN from the newest id with a stride that spans the whole range, so it makes no - /// assumption about where the live nodes are. Both directions matter: Harper allocates node - /// ids monotonically and never reuses them, so a churned table has its whole low prefix - /// tombstoned and only the newest ids are live — while the crate's own freelist does reuse - /// ids, which keeps live nodes low. Striding covers both, and a graph with any appreciable - /// live fraction is found in the first few probes either way. + /// The start rotates. A fixed start would probe one residue class of the stride forever, so a + /// live graph lying entirely between its probes would stay invisible permanently rather than + /// for one search — the difference between a bounded miss and a silent-empty-results mode. + /// Rotating means `stride` consecutive repairs cover every id, while each stays capped at + /// `limit`. /// - /// Best-level rather than first-live, because a level-0 entry degrades every later search to - /// a layer-0-only beam. + /// Best-level rather than first-live: a level-0 entry degrades every later search to a + /// layer-0-only beam. pub(crate) fn probe_for_entry(&self, limit: u32, skip: u32) -> Option<(u32, u8)> { let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; if hw == 0 || limit == 0 { return None; } let stride = (hw / limit).max(1); + let offset = PROBE_ROTATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % stride; let mut best: Option<(u32, u8)> = None; - let mut cand = hw - 1; + let mut cand = hw - 1 - offset; for _ in 0..limit { if cand != skip { if let Some(level) = self.node_level(cand) { diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 67c4fee143..49d99325ac 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -584,6 +584,47 @@ fn search_repairs_an_entry_point_in_a_churned_graph_whose_low_ids_are_all_dead() let _ = std::fs::remove_file(&path); } +/// With a stride above 1 a fixed start probes one residue class forever, so a live graph lying +/// entirely between its probes would never be found. The rotation makes `stride` consecutive +/// repairs cover every id; here the sole survivor is deliberately in the residue the unrotated +/// walk skips. +#[test] +fn a_repair_probe_rotates_so_no_live_node_stays_between_its_samples() { + let dims = 32; + let path = tmp("entryhealrotate"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..2_100 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let hw = graph.file.id_high_water() as u32; + let stride = (hw / 1_024).max(1); // REPAIR_PROBE_LIMIT + assert!(stride > 1, "precondition: a stride the rotation actually has to cover, got {stride}"); + // an unrotated walk starts at hw-1 and steps by `stride`, so it only ever sees that residue; + // keep exactly one node alive in a different one + let survivor = (0..hw).rev().find(|id| (hw - 1 - id) % stride != 0).expect("a skipped residue"); + for id in 0..hw { + if id != survivor { + let _ = graph.clear_node(id); + } + } + assert!(graph.read_node(survivor).is_some(), "precondition: the survivor is live"); + + let mut found = false; + for _ in 0..stride { + let (hits, _) = search(&graph, &Query::new(vector_for(survivor, dims)), 5, 64, &mut scratch); + if !hits.is_empty() { + found = true; + break; + } + } + assert!(found, "a rotating probe must reach every residue within `stride` repairs"); + assert_eq!(graph.file.entry_point().0, survivor, "the only live node must be the repaired entry"); + let _ = std::fs::remove_file(&path); +} + /// A repair publishes with a strict CAS on the entry it observed dead. A first insert that /// claims the header in between owns the graph, and a higher-level repair candidate must lose to /// it — installing the candidate would leave that insert's node with nothing pointing at it. From 1a86e55949584d27ab3ffaaefb61158b43746407 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 13:02:27 -0600 Subject: [PATCH 50/69] hnsw-plane: make the repair-probe rotation per plane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rotation counter was process-global, so every other plane's repairs advanced it too: two planes repairing in turn each see offsets stepping by two, which pins each to one residue class indefinitely — the coverage the rotation was added to provide. It lives on the Graph handle now, so a plane's own consecutive repairs are what rotate it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011W6mChyPAUAoKEfxV1bSAo --- native/hnsw-plane/src/graph.rs | 20 ++++++------- native/hnsw-plane/tests/reopen.rs | 50 +++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 10 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index eaecaf0d86..15607ccc34 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -27,11 +27,13 @@ unsafe fn vread(p: *const T) -> T { p.read_volatile() } -/// Rotates `probe_for_entry`'s starting offset so consecutive repairs sample different ids. -static PROBE_ROTATION: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); - pub struct Graph { pub file: PlaneFile, + /// Rotates `probe_for_entry`'s starting offset so this plane's consecutive repairs sample + /// different ids. Per handle, not per process: a shared counter is advanced by every other + /// plane's repairs too, so one plane's calls can land on a single residue indefinitely — + /// which is the coverage the rotation exists to provide. + probe_rotation: std::sync::atomic::AtomicU32, } /// A consistent full copy of one node (construction paths only; search uses zero-copy). @@ -45,7 +47,7 @@ pub struct NodeRead { impl Graph { pub fn new(file: PlaneFile) -> Self { - Graph { file } + Graph { file, probe_rotation: std::sync::atomic::AtomicU32::new(0) } } #[inline] @@ -600,11 +602,9 @@ impl Graph { /// reuses them, so a churned table's low prefix is all tombstones, while the crate's own /// freelist reuses ids and keeps live nodes low. /// - /// The start rotates. A fixed start would probe one residue class of the stride forever, so a - /// live graph lying entirely between its probes would stay invisible permanently rather than - /// for one search — the difference between a bounded miss and a silent-empty-results mode. - /// Rotating means `stride` consecutive repairs cover every id, while each stays capped at - /// `limit`. + /// The start rotates per handle, so `stride` consecutive repairs of this plane cover every id + /// while each stays capped at `limit`; a fixed start would probe one residue class forever + /// and leave a graph lying between its samples invisible permanently, not for one search. /// /// Best-level rather than first-live: a level-0 entry degrades every later search to a /// layer-0-only beam. @@ -614,7 +614,7 @@ impl Graph { return None; } let stride = (hw / limit).max(1); - let offset = PROBE_ROTATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % stride; + let offset = self.probe_rotation.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % stride; let mut best: Option<(u32, u8)> = None; let mut cand = hw - 1 - offset; for _ in 0..limit { diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 49d99325ac..6044e7f3c0 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -625,6 +625,56 @@ fn a_repair_probe_rotates_so_no_live_node_stays_between_its_samples() { let _ = std::fs::remove_file(&path); } +/// Rotation has to be per handle. With one process-wide counter, every other plane's repairs +/// advance it too, so two planes repairing in turn each see offsets stepping by 2 — one residue +/// class apiece, indefinitely, which is exactly what rotating was meant to prevent. Both planes +/// here hide their survivor in the same residue, so a shared counter must strand one of them +/// whichever offset it starts on. +#[test] +fn repair_probe_rotation_is_per_plane_not_per_process() { + let dims = 32; + let mut graphs = Vec::new(); + let mut survivors = Vec::new(); + let mut stride = 0u32; + for which in 0..2 { + let path = tmp(&format!("entryhealperplane{which}")); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..2_100 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let hw = graph.file.id_high_water() as u32; + stride = (hw / 1_024).max(1); + assert!(stride > 1, "precondition: a stride the rotation has to cover"); + // the same skipped residue on both planes, so a shared counter cannot serve both + let survivor = (0..hw).rev().find(|id| (hw - 1 - id) % stride != 0).expect("a skipped residue"); + for id in 0..hw { + if id != survivor { + let _ = graph.clear_node(id); + } + } + graphs.push((graph, path)); + survivors.push(survivor); + } + + let mut scratch = SearchScratch::new(); + let mut found = [false; 2]; + for _ in 0..stride { + for (which, (graph, _)) in graphs.iter().enumerate() { + let (hits, _) = search(graph, &Query::new(vector_for(survivors[which], dims)), 5, 64, &mut scratch); + if !hits.is_empty() { + found[which] = true; + } + } + } + assert!(found[0] && found[1], "each plane must cover its own residues: {found:?}"); + for (_, path) in &graphs { + let _ = std::fs::remove_file(path); + } +} + /// A repair publishes with a strict CAS on the entry it observed dead. A first insert that /// claims the header in between owns the graph, and a higher-level repair candidate must lose to /// it — installing the candidate would leave that insert's node with nothing pointing at it. From 70a4fe685845d6f047804755f8936748d2f7f7c9 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 13:23:20 -0600 Subject: [PATCH 51/69] hnsw-plane: ceil the repair-probe stride so no low prefix is unreachable Floor division leaves stride * limit < hw whenever hw is not a multiple of limit, so every rotated probe stops above the lowest hw % limit ids. That is permanent, not per-search: no rotation offset reaches them, so a graph whose entry and hint die while its survivors sit in that prefix returns empty from every later search. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/graph.rs | 5 +++- native/hnsw-plane/src/insert.rs | 2 +- native/hnsw-plane/tests/reopen.rs | 47 +++++++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 4 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 15607ccc34..646c2935be 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -605,6 +605,9 @@ impl Graph { /// The start rotates per handle, so `stride` consecutive repairs of this plane cover every id /// while each stays capped at `limit`; a fixed start would probe one residue class forever /// and leave a graph lying between its samples invisible permanently, not for one search. + /// The stride is a ceiling division for the same reason: flooring it leaves + /// `stride * limit < hw`, so every rotated walk stops short of the lowest `hw % limit` ids + /// and a graph surviving only there stays invisible however many times the start rotates. /// /// Best-level rather than first-live: a level-0 entry degrades every later search to a /// layer-0-only beam. @@ -613,7 +616,7 @@ impl Graph { if hw == 0 || limit == 0 { return None; } - let stride = (hw / limit).max(1); + let stride = hw.div_ceil(limit); let offset = self.probe_rotation.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % stride; let mut best: Option<(u32, u8)> = None; let mut cand = hw - 1 - offset; diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 52027e5f3a..358ed18152 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -34,7 +34,7 @@ fn level_for(id: u32, ml: f64) -> u8 { /// Remove `to` from `from`'s adjacency at `level` (edge-replacement maintenance). fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { if level == 0 { - graph.update_neighbors(from, |list| { + let _ = graph.update_neighbors(from, |list| { if let Some(pos) = list.iter().position(|&x| x == to) { list.remove(pos); } diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs index 6044e7f3c0..7234ffc077 100644 --- a/native/hnsw-plane/tests/reopen.rs +++ b/native/hnsw-plane/tests/reopen.rs @@ -600,7 +600,7 @@ fn a_repair_probe_rotates_so_no_live_node_stays_between_its_samples() { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } let hw = graph.file.id_high_water() as u32; - let stride = (hw / 1_024).max(1); // REPAIR_PROBE_LIMIT + let stride = hw.div_ceil(1_024); // REPAIR_PROBE_LIMIT assert!(stride > 1, "precondition: a stride the rotation actually has to cover, got {stride}"); // an unrotated walk starts at hw-1 and steps by `stride`, so it only ever sees that residue; // keep exactly one node alive in a different one @@ -625,6 +625,49 @@ fn a_repair_probe_rotates_so_no_live_node_stays_between_its_samples() { let _ = std::fs::remove_file(&path); } +/// The stride must be a ceiling division. Flooring it leaves `stride * limit < hw` whenever `hw` +/// is not a multiple of `limit`, so every rotated walk stops above the lowest `hw % limit` ids — +/// a permanent blind spot, not a one-search one, since no offset ever reaches it. A graph whose +/// only survivors sit in that prefix would return empty from every later search; this one's does. +#[test] +fn a_repair_probe_reaches_the_low_ids_a_floored_stride_would_never_sample() { + let dims = 32; + let limit = 1_024u32; // REPAIR_PROBE_LIMIT + let path = tmp("entryheallowprefix"); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..2_100 { + insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); + } + let hw = graph.file.id_high_water() as u32; + // a floored walk bottoms out at `hw % limit` whatever its rotation offset, so ids below that + // are exactly what the ceiling buys + let floored_reach = hw % limit; + assert!(hw > limit && floored_reach > 1, "precondition: a low prefix a floored stride skips, hw {hw}"); + let survivor = floored_reach / 2; + for id in 0..hw { + if id != survivor { + let _ = graph.clear_node(id); + } + } + assert!(graph.read_node(survivor).is_some(), "precondition: the survivor is live"); + assert_ne!(graph.file.previous_entry_point(), survivor, "precondition: the probe must be what finds it"); + + let mut found = false; + for _ in 0..hw.div_ceil(limit) { + let (hits, _) = search(&graph, &Query::new(vector_for(survivor, dims)), 5, 64, &mut scratch); + if !hits.is_empty() { + found = true; + break; + } + } + assert!(found, "a full rotation must cover every id, the lowest included"); + assert_eq!(graph.file.entry_point().0, survivor, "the only live node must be the repaired entry"); + let _ = std::fs::remove_file(&path); +} + /// Rotation has to be per handle. With one process-wide counter, every other plane's repairs /// advance it too, so two planes repairing in turn each see offsets stepping by 2 — one residue /// class apiece, indefinitely, which is exactly what rotating was meant to prevent. Both planes @@ -646,7 +689,7 @@ fn repair_probe_rotation_is_per_plane_not_per_process() { insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); } let hw = graph.file.id_high_water() as u32; - stride = (hw / 1_024).max(1); + stride = hw.div_ceil(1_024); assert!(stride > 1, "precondition: a stride the rotation has to cover"); // the same skipped residue on both planes, so a shared counter cannot serve both let survivor = (0..hw).rev().find(|id| (hw - 1 - id) % stride != 0).expect("a skipped residue"); From e0f1ca30358b08dd596ba4fd98d52a456db67452 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 13:47:58 -0600 Subject: [PATCH 52/69] hnsw-plane: return a predicated search on its last verdict, not 50 ms later MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tail drain looped `while let Ok(..) = recv_timeout(50ms)`, so after the final verdict dropped `outstanding` to zero it waited out one more full timeout. Every filtered query paid 50 ms on the pool thread — 25-50x the search itself — and on the shared libuv pool that queues unrelated I/O too. Guarding the drain on `outstanding` also makes a stray verdict a break rather than a usize underflow. Also trims the ceiling-stride comment to the invariant it rests on. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/graph.rs | 5 ++- native/hnsw-plane/src/search.rs | 58 ++++++++++++++++++++++++++++++++- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs index 646c2935be..d86b3fdc6a 100644 --- a/native/hnsw-plane/src/graph.rs +++ b/native/hnsw-plane/src/graph.rs @@ -605,9 +605,8 @@ impl Graph { /// The start rotates per handle, so `stride` consecutive repairs of this plane cover every id /// while each stays capped at `limit`; a fixed start would probe one residue class forever /// and leave a graph lying between its samples invisible permanently, not for one search. - /// The stride is a ceiling division for the same reason: flooring it leaves - /// `stride * limit < hw`, so every rotated walk stops short of the lowest `hw % limit` ids - /// and a graph surviving only there stays invisible however many times the start rotates. + /// That coverage rests on `stride * limit >= hw`, which is why the stride is a ceiling + /// division: below it a walk stops short of id 0 and no offset ever reaches the tail. /// /// Best-level rather than first-live: a level-0 entry degrades every later search to a /// layer-0-only beam. diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 838dd8aed4..169868f800 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -347,9 +347,13 @@ pub fn search_predicated( let mut nbuf = std::mem::take(&mut scratch.neighbors); + // guarded on `outstanding` rather than draining until the channel is empty: with a blocking + // receive the unguarded form pays another full timeout after the last verdict lands, on every + // filtered query macro_rules! drain { ($recv:expr) => { - while let Ok((ids, flags)) = $recv { + while outstanding > 0 { + let Ok((ids, flags)) = $recv else { break }; outstanding -= 1; for (i, id) in ids.iter().enumerate() { verdicts.insert(*id, flags.get(i).copied().unwrap_or(0) != 0); @@ -485,4 +489,56 @@ mod predicate_tests { worker.join().unwrap(); let _ = std::fs::remove_file(&path); } + + /// The tail drain must stop receiving the moment the last verdict lands. Draining until the + /// channel reports empty sits out another full `recv_timeout` after `outstanding` reaches + /// zero — 50 ms added to every filtered query, against a sub-millisecond search. Measured + /// from the evaluator's last send so the search's own cost is not in the number. + #[test] + fn a_predicated_search_returns_as_soon_as_the_last_verdict_lands() { + let dims = 32; + let path = std::env::temp_dir().join(format!("hnsw-preddrain-{}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..1_000u32 { + let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + insert(&graph, &v, ¶ms, &mut scratch).unwrap(); + } + + let (req_tx, req_rx) = std::sync::mpsc::channel::>(); + let (res_tx, res_rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); + // stamped before the send, so the search can never observe a verdict newer than the stamp + let last_send = std::sync::Arc::new(std::sync::Mutex::new(None::)); + let stamps = last_send.clone(); + let worker = std::thread::spawn(move || { + while let Ok(ids) = req_rx.recv() { + let verdicts = vec![1u8; ids.len()]; + *stamps.lock().unwrap() = Some(std::time::Instant::now()); + if res_tx.send((ids, verdicts)).is_err() { + break; + } + } + }); + + let mut pipe = PredicatePipe { + dispatch: Box::new(move |ids| { + let _ = req_tx.send(ids); + }), + rx: res_rx, + }; + let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + let (hits, _) = + search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); + let tail = last_send.lock().unwrap().expect("the evaluator answered a batch").elapsed(); + assert!(!hits.is_empty(), "precondition: an admitting predicate returns results"); + assert!( + tail < std::time::Duration::from_millis(25), + "the drain sat {tail:?} past the last verdict instead of returning on it" + ); + drop(pipe); + worker.join().unwrap(); + let _ = std::fs::remove_file(&path); + } } From 2ea14fa4656461bbbff62c4add4810fe77eaaae7 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 13:51:18 -0600 Subject: [PATCH 53/69] hnsw-plane: take the best of several queries in the drain latency test A single wall-clock sample calls any 25 ms scheduler stall an extra receive. The defect it guards adds the full timeout to every query, so the minimum over a handful separates them: noise cannot hold all of them above the bound. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/search.rs | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index 169868f800..b168b86b6a 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -493,7 +493,9 @@ mod predicate_tests { /// The tail drain must stop receiving the moment the last verdict lands. Draining until the /// channel reports empty sits out another full `recv_timeout` after `outstanding` reaches /// zero — 50 ms added to every filtered query, against a sub-millisecond search. Measured - /// from the evaluator's last send so the search's own cost is not in the number. + /// from the evaluator's last send so the search's own cost is not in the number, and over + /// the best of several queries so scheduler noise on one of them cannot pass for the extra + /// receive, which every query would pay. #[test] fn a_predicated_search_returns_as_soon_as_the_last_verdict_lands() { let dims = 32; @@ -529,13 +531,24 @@ mod predicate_tests { rx: res_rx, }; let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - let (hits, _) = - search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); - let tail = last_send.lock().unwrap().expect("the evaluator answered a batch").elapsed(); - assert!(!hits.is_empty(), "precondition: an admitting predicate returns results"); + let mut best = std::time::Duration::MAX; + for _ in 0..5 { + let (hits, _) = search_predicated( + &graph, + &Query::new(q.clone()), + 10, + 64, + &mut pipe, + 64 * 24, + &mut scratch, + ); + let tail = last_send.lock().unwrap().expect("the evaluator answered a batch").elapsed(); + assert!(!hits.is_empty(), "precondition: an admitting predicate returns results"); + best = best.min(tail); + } assert!( - tail < std::time::Duration::from_millis(25), - "the drain sat {tail:?} past the last verdict instead of returning on it" + best < std::time::Duration::from_millis(25), + "the drain sat {best:?} past the last verdict on every query instead of returning on it" ); drop(pipe); worker.join().unwrap(); From 27eda01575d21266ae00ada3946bca3d6ab6e800 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 14:36:46 -0600 Subject: [PATCH 54/69] hnsw-plane: three defects from the recorded review backlog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit None of these were open review threads — all eight were closed and the last two bot rounds on this head were clean. They are findings earlier rounds recorded in the PR body as open-but-not-taken, each a defect with one right answer rather than a design question. add_reverse_edge's contended fallback dropped the edge it was adding: a push followed by truncate(cap) discards the tail, and at exactly cap the tail is the new id, so the reverse edge that keeps a newly inserted node reachable from that neighbor was lost precisely in the contended-and-full case the fallback exists to serve. It now evicts the farthest neighbor instead (lists are written in ascending distance). A refused predicate enqueue was counted outstanding. call_with_return_value drops its callback when the queue is closing or full, so no verdict can arrive, and the tail drain then waited out its whole 5 s deadline — on every in-flight filtered query during teardown. PredicatePipe::dispatch now reports whether the batch was handed off. A per-query distance override bypassed the plane cutover: a euclidean query against a cosine index was traversed cosine-first, and rescoreResults only corrects the reported distances of the candidates it is handed, not which candidates the beam kept. Such a query now takes the JS path, matching the adjacent dimension-mismatch precedent. Regressions: a_contended_merge_into_a_full_neighbor_list_keeps_the_edge_it_adds, a_refused_predicate_enqueue_does_not_hold_the_drain (5.02 s on the parent commit, sub-second here), and a JS parity test asserting the overridden metric never reaches the plane. Each checked to fail with its own fix reverted. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/insert.rs | 64 +++++++++++++++++-- native/hnsw-plane/src/napi.rs | 6 +- native/hnsw-plane/src/search.rs | 56 ++++++++++++---- .../HierarchicalNavigableSmallWorld.ts | 5 +- unitTests/resources/vectorIndexPlane.test.js | 15 +++++ 5 files changed, 126 insertions(+), 20 deletions(-) diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 358ed18152..0379eb0c05 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -93,6 +93,24 @@ fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize *list = scored.into_iter().map(|(cand, _)| cand).collect(); } +/// The contended fallback's merge: add `new_id` under the slot lock, evicting the farthest +/// existing neighbor once the list is at `cap`. Pushing and then truncating to `cap` instead +/// drops the tail — which at exactly `cap` is `new_id` itself, silently losing the reverse edge +/// that keeps the newly inserted node reachable from `nid`. Neighbor lists are written in +/// ascending distance (both the insert path and `prune_with_coverage` emit them sorted), so the +/// tail is the farthest neighbor and the cheapest one to give up under the lock. +fn merge_neighbor_capped(graph: &Graph, nid: u32, new_id: u32, cap: usize) { + let _ = graph.update_neighbors(nid, |list| { + if list.contains(&new_id) { + return; + } + if list.len() >= cap { + list.truncate(cap.saturating_sub(1)); + } + list.push(new_id); + }); +} + /// Add `new_id` to `nid`'s adjacency at `level`, coverage-pruning to `cap` when over. The /// prune's distance computations (which can major-fault on a cold mapping) run OUTSIDE the /// slot lock: the list is snapshotted, pruned, and applied with a compare-and-set; after a @@ -117,12 +135,7 @@ fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) } } // contended twice: merge cheaply under the lock (bounded critical section) - let _ = graph.update_neighbors(nid, |list| { - if !list.contains(&new_id) { - list.push(new_id); - list.truncate(cap); - } - }); + merge_neighbor_capped(graph, nid, new_id, cap); } else { let _ = graph.update_upper_level(nid, level, |list| { if list.contains(&new_id) { @@ -313,3 +326,42 @@ fn scratch_begin(graph: &Graph, scratch: &mut SearchScratch) { // via this helper to keep the public surface small. scratch.begin_public(graph.file.id_high_water()); } + +#[cfg(test)] +mod reverse_edge_tests { + use super::*; + use crate::PlaneFile; + + /// The contended fallback must still add the edge when the neighbor list is already full. + /// A push followed by `truncate(cap)` drops the tail, and at exactly `cap` the tail is the + /// id being added — so the reverse edge that keeps a newly inserted node reachable from + /// `nid` is lost precisely in the contended-and-full case the fallback exists to serve. + #[test] + fn a_contended_merge_into_a_full_neighbor_list_keeps_the_edge_it_adds() { + let dims = 8; + let cap = 8usize; + let path = std::env::temp_dir().join(format!("hnsw-revedge-{}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, cap, 4_096).expect("create")); + + let vector = vec![0i8; dims]; + let full: Vec = (1..=cap as u32).collect(); + graph.write_node_raw(0, 0, &vector, 1.0, 1.0, &full, &[]).expect("seed the full list"); + for &nid in &full { + graph.write_node_raw(nid, 0, &vector, 1.0, 1.0, &[], &[]).expect("seed a neighbor"); + } + let newcomer = cap as u32 + 1; + graph.write_node_raw(newcomer, 0, &vector, 1.0, 1.0, &[], &[]).expect("seed the newcomer"); + + merge_neighbor_capped(&graph, 0, newcomer, cap); + + let mut neighbors: Vec = Vec::new(); + graph.neighbors_into(0, &mut neighbors).expect("node 0 is live"); + assert!( + neighbors.contains(&newcomer), + "the contended merge dropped the edge it was adding: {neighbors:?}" + ); + assert_eq!(neighbors.len(), cap, "the merge must stay within the layer-0 cap"); + let _ = std::fs::remove_file(&path); + } +} diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs index 1bc7436ebd..d15700ffbf 100644 --- a/native/hnsw-plane/src/napi.rs +++ b/native/hnsw-plane/src/napi.rs @@ -92,7 +92,7 @@ impl Task for PredicateSearchTask { dispatch: Box::new(move |ids: Vec| { let tx = tx.clone(); let ids_echo = ids.clone(); - tsfn.call_with_return_value( + let status = tsfn.call_with_return_value( ids, ThreadsafeFunctionCallMode::NonBlocking, move |ret: Uint8Array| { @@ -102,6 +102,10 @@ impl Task for PredicateSearchTask { Ok(()) }, ); + // a closing or saturated queue drops the callback without invoking it, so this + // batch will never answer; reporting it lets the drain finish on the batches + // that will, instead of holding teardown for the full deadline + status == Status::Ok }), rx, }; diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs index b168b86b6a..9e8ca8b47b 100644 --- a/native/hnsw-plane/src/search.rs +++ b/native/hnsw-plane/src/search.rs @@ -303,8 +303,11 @@ pub fn search_filtered( /// steer result admission only; routing uses pure distance order, bounded by the visit /// budget, so a slow or saturated JS loop degrades speculative overshoot, not correctness. pub struct PredicatePipe { - /// Sends one batch of ids for evaluation. Must not block. - pub dispatch: Box) + Send>, + /// Sends one batch of ids for evaluation. Must not block. Returns whether the batch was + /// actually handed off: a refused enqueue never produces a verdict, so counting it as + /// outstanding would make the tail drain wait out its whole deadline for an answer that + /// cannot arrive. + pub dispatch: Box) -> bool + Send>, /// Receives (ids, verdicts) pairs; verdicts[i] != 0 admits ids[i]. pub rx: std::sync::mpsc::Receiver<(Vec, Vec)>, } @@ -405,8 +408,7 @@ pub fn search_predicated( candidates.push(Candidate { distance: d, id: nid }); speculative.push((nid, d)); batch.push(nid); - if batch.len() >= PREDICATE_BATCH { - (pipe.dispatch)(std::mem::take(&mut batch)); + if batch.len() >= PREDICATE_BATCH && (pipe.dispatch)(std::mem::take(&mut batch)) { outstanding += 1; } } @@ -416,8 +418,7 @@ pub fn search_predicated( scratch.neighbors = nbuf; // flush the tail batch and block-drain what's still in flight - if !batch.is_empty() { - (pipe.dispatch)(std::mem::take(&mut batch)); + if !batch.is_empty() && (pipe.dispatch)(std::mem::take(&mut batch)) { outstanding += 1; } let deadline = std::time::Instant::now() + DRAIN_TIMEOUT; @@ -473,9 +474,7 @@ mod predicate_tests { }); let mut pipe = PredicatePipe { - dispatch: Box::new(move |ids| { - let _ = req_tx.send(ids); - }), + dispatch: Box::new(move |ids| req_tx.send(ids).is_ok()), rx: res_rx, }; let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); @@ -525,9 +524,7 @@ mod predicate_tests { }); let mut pipe = PredicatePipe { - dispatch: Box::new(move |ids| { - let _ = req_tx.send(ids); - }), + dispatch: Box::new(move |ids| req_tx.send(ids).is_ok()), rx: res_rx, }; let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); @@ -554,4 +551,39 @@ mod predicate_tests { worker.join().unwrap(); let _ = std::fs::remove_file(&path); } + + /// A refused enqueue never answers. Counting it outstanding makes the tail drain wait out + /// its whole `DRAIN_TIMEOUT` for a verdict that cannot arrive — which is exactly the state + /// a closing environment puts every in-flight filtered query in, so teardown pays five + /// seconds per query instead of returning on the batches that did land. + #[test] + fn a_refused_predicate_enqueue_does_not_hold_the_drain() { + let dims = 32; + let path = std::env::temp_dir().join(format!("hnsw-refused-{}.hnsw", std::process::id())); + let _ = std::fs::remove_file(&path); + let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); + let params = InsertParams::default(); + let mut scratch = SearchScratch::new(); + for i in 0..1_000u32 { + let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + insert(&graph, &v, ¶ms, &mut scratch).unwrap(); + } + + // the sender stays alive for the whole search, so a drain that believes a batch is + // outstanding blocks on the deadline rather than on a disconnected channel + let (tx, rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); + let mut pipe = PredicatePipe { dispatch: Box::new(|_ids| false), rx }; + let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); + let started = std::time::Instant::now(); + let (hits, _) = + search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); + let elapsed = started.elapsed(); + drop(tx); + assert!(hits.is_empty(), "no verdict can arrive for a refused batch, so nothing may be admitted"); + assert!( + elapsed < std::time::Duration::from_secs(1), + "the search waited {elapsed:?} on batches that were never enqueued (deadline is {DRAIN_TIMEOUT:?})" + ); + let _ = std::fs::remove_file(&path); + } } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 49370cfa4d..7ad8e76612 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -1779,7 +1779,10 @@ export class HierarchicalNavigableSmallWorld { filterEvaluations: 0, } : undefined; - if (this.planeEligible) { + // The plane traverses the index's own metric (cosine — the eligibility requirement), so a + // query overriding `distance` has to take the JS path: rescoreResults only corrects the + // reported distances of whatever candidates came back, not which candidates the beam kept. + if (this.planeEligible && distanceFunction === this.distance) { const plane = this.getPlane(target.length, false); // a query whose dimensionality differs from the graph's takes the JS path (which // tolerates the mismatch) rather than erroring or disabling the healthy plane diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index 7a62896934..c725bc8647 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -132,6 +132,21 @@ describe('HNSW native plane dual-write', function () { } }); + // The plane only ever traverses the index's own metric, so a query asking for a different one + // has to fall back: rescoring fixes the reported distances of the candidates it is handed, not + // which candidates a cosine beam kept. + it('a query overriding the distance metric takes the JS path rather than a cosine traversal', () => { + const condition = { target: vectors.get(3), comparator: 'sort', distance: 'euclidean', ef: EF }; + const flagged = customIndex().search(condition, { transaction: undefined }); + assert.equal(typeof flagged?.then, 'undefined', 'a euclidean query must not be answered by the cosine plane'); + const jsEntries = jsReference().search(condition, { transaction: undefined }); + assert.deepEqual( + flagged.map((entry) => entry.key), + jsEntries.map((entry) => entry.key), + 'an overridden metric must return exactly what the JS traversal returns' + ); + }); + it('parity holds after update-in-place and delete (including neighbor repair)', async () => { for (let i = 0; i < 100; i++) { const vector = makeVector(i + 5000); From c0d3299595df557efe13523dfdde63e99fcc82e7 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 14:53:17 -0600 Subject: [PATCH 55/69] hnsw-plane: state the contended merge's victim as arbitrary, not farthest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pre-push review's one finding on the delta: the merge's premise was wrong. Neighbor lists are distance-ordered only right after a prune — non-overflow reverse-edge appends push at the tail — so the displaced neighbor is arbitrary, not the farthest. The behavior stands: an arbitrary existing edge is the right thing to give up over the edge being added, whose loss is systematic and costs a freshly inserted node its in-edge, and any better victim needs distances this path keeps outside the lock. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/insert.rs | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs index 0379eb0c05..73070b1bea 100644 --- a/native/hnsw-plane/src/insert.rs +++ b/native/hnsw-plane/src/insert.rs @@ -93,12 +93,13 @@ fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize *list = scored.into_iter().map(|(cand, _)| cand).collect(); } -/// The contended fallback's merge: add `new_id` under the slot lock, evicting the farthest -/// existing neighbor once the list is at `cap`. Pushing and then truncating to `cap` instead -/// drops the tail — which at exactly `cap` is `new_id` itself, silently losing the reverse edge -/// that keeps the newly inserted node reachable from `nid`. Neighbor lists are written in -/// ascending distance (both the insert path and `prune_with_coverage` emit them sorted), so the -/// tail is the farthest neighbor and the cheapest one to give up under the lock. +/// The contended fallback's merge: add `new_id` under the slot lock, displacing the tail once the +/// list is at `cap`. Which neighbor that is is arbitrary — appends push at the tail, so a list is +/// distance-ordered only immediately after a prune — but it must not be `new_id` itself, which is +/// what a push followed by `truncate(cap)` drops. That loss is the systematic one: the edge being +/// added is the in-edge keeping a freshly inserted node reachable from `nid`, and it disappears +/// every time the list is full and the CAS path is contended. Picking a better victim needs +/// distances, which this path deliberately keeps outside the lock. fn merge_neighbor_capped(graph: &Graph, nid: u32, new_id: u32, cap: usize) { let _ = graph.update_neighbors(nid, |list| { if list.contains(&new_id) { @@ -332,10 +333,9 @@ mod reverse_edge_tests { use super::*; use crate::PlaneFile; - /// The contended fallback must still add the edge when the neighbor list is already full. - /// A push followed by `truncate(cap)` drops the tail, and at exactly `cap` the tail is the - /// id being added — so the reverse edge that keeps a newly inserted node reachable from - /// `nid` is lost precisely in the contended-and-full case the fallback exists to serve. + /// The contended fallback must still add the edge when the neighbor list is already full — + /// the one case where a push-then-`truncate(cap)` discards `new_id` rather than a neighbor, + /// losing the in-edge exactly in the contended-and-full case the fallback exists to serve. #[test] fn a_contended_merge_into_a_full_neighbor_list_keeps_the_edge_it_adds() { let dims = 8; From ad198eb07f3cea14aa68411bcd02b1c860ab99c8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 14:54:20 -0600 Subject: [PATCH 56/69] hnsw-plane: let the bench's build loop fail loudly on a rejected insert The crate's remaining build warning, and the reason it is worth silencing rather than allowing: a benchmark that discards insert errors reports build throughput for rows it never indexed, and then measures recall against a corpus the graph does not contain. The concurrent-writer loop two hundred lines down already breaks on the same error. Co-Authored-By: Claude Opus --- native/hnsw-plane/src/bin/bench.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs index 86fd7c4c91..f3fe4d2f47 100644 --- a/native/hnsw-plane/src/bin/bench.rs +++ b/native/hnsw-plane/src/bin/bench.rs @@ -122,7 +122,7 @@ fn main() { let build_start = Instant::now(); for i in 0..n { let v = corpus.row(&mut rng); - insert(&graph, &v, ¶ms, &mut scratch); + insert(&graph, &v, ¶ms, &mut scratch).expect("build insert"); if (i + 1) % 50_000 == 0 { let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); println!(" built {} ({:.0} inserts/s)", i + 1, rate); From a55adf01fc93485e91ebec6ca807380dcaee1f4b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 16:20:51 -0600 Subject: [PATCH 57/69] chore: rebase native HNSW plane onto main Co-Authored-By: GPT-5 Codex From a0f735d545ffb78b32f0d290d0cb4fda0e8c1f8c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 1 Sep 2026 16:49:08 -0600 Subject: [PATCH 58/69] style: format query array scoping test Co-Authored-By: GPT-5 Codex --- unitTests/resources/query-array-scoping.test.js | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/unitTests/resources/query-array-scoping.test.js b/unitTests/resources/query-array-scoping.test.js index 9cf50d8e31..c089829dec 100644 --- a/unitTests/resources/query-array-scoping.test.js +++ b/unitTests/resources/query-array-scoping.test.js @@ -140,10 +140,7 @@ describe('Array-valued property scoping', () => { it('indexed array: same matching records as unindexed', async function () { // unique ids: which leg leads the scan is estimate-dependent, so the harper#2434 // duplicate pattern is not stable here - assert.deepStrictEqual( - await collectUniqueIds(searchRest('sizesIdx=ge=175&sizesIdx=le=180')), - [1, 2, 4, 6] - ); + assert.deepStrictEqual(await collectUniqueIds(searchRest('sizesIdx=ge=175&sizesIdx=le=180')), [1, 2, 4, 6]); }); it.skip('harper#2434: indexed lead condition must not duplicate records into the result', async function () { assert.deepStrictEqual(await collectIds(searchRest('sizesIdx=ge=175&sizesIdx=le=180')), [1, 2, 4, 6]); @@ -202,9 +199,7 @@ describe('Array-valued property scoping', () => { }); it('programmatic string value against numeric elements: same result', async function () { assert.deepStrictEqual( - await collectIds( - Widgets.search({ conditions: [{ attribute: 'sizes', comparator: 'contains', value: '17' }] }) - ), + await collectIds(Widgets.search({ conditions: [{ attribute: 'sizes', comparator: 'contains', value: '17' }] })), [1, 2, 4] ); }); From 0028bb3bfea1bea13e36fd34364786a87aae91dc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 4 Sep 2026 11:34:47 -0600 Subject: [PATCH 59/69] Consume the native HNSW plane as @harperfast/hnsw 0.2.1 instead of an in-repo crate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Rust crate now lives in HarperFast/hnsw and ships as an Apache-2.0 npm package with platform prebuilds, so `native/hnsw-plane/` and its `build:hnsw-plane` script are removed and the adapter requires the exact-pinned optional dependency. 0.2.1 also moves plane invalidation into the package: `invalidateFile()` / `invalidatePlane()` set the one-way header latch, zero the watermark, and write the fsync'd `.stale` sidecar in that order, and `open()` now refuses a plane carrying either marker. The ordering invariant and its power-loss rationale move with them, so the adapter keeps only availability, fallback, and integration policy — including a sidecar-only path for a plane that outlives the package that made it. CI runs the published prebuild, so the job proves that prebuild loads (a failed load would leave the suite self-skipping and green) and no longer re-runs the crate's own tests against a pinned source checkout, which HarperFast/hnsw's CI already covers on three platforms and which would grade something other than the binary under test. The phase-2 plan in hnsw-native-plane.md now targets #2489's shared post-commit `DerivedIndexBackend` delivery rather than an HNSW-specific commit callback. Refs #2489 Co-Authored-By: Claude Opus 5 --- .github/workflows/unit-test.yml | 13 +- .gitignore | 4 - dependencies.md | 12 + hnsw-native-plane.md | 31 +- native/hnsw-plane/Cargo.lock | 230 ------ native/hnsw-plane/Cargo.toml | 30 - native/hnsw-plane/build.mjs | 19 - native/hnsw-plane/build.rs | 6 - native/hnsw-plane/smoke.mjs | 84 -- native/hnsw-plane/src/bin/bench.rs | 246 ------ native/hnsw-plane/src/distance.rs | 138 ---- native/hnsw-plane/src/format.rs | 687 ---------------- native/hnsw-plane/src/graph.rs | 743 ----------------- native/hnsw-plane/src/insert.rs | 367 --------- native/hnsw-plane/src/lib.rs | 15 - native/hnsw-plane/src/napi.rs | 496 ------------ native/hnsw-plane/src/search.rs | 589 -------------- native/hnsw-plane/src/seqlock.rs | 217 ----- native/hnsw-plane/tests/concurrent.rs | 174 ---- native/hnsw-plane/tests/reopen.rs | 755 ------------------ package-lock.json | 89 +++ package.json | 2 +- .../HierarchicalNavigableSmallWorld.ts | 30 +- resources/indexes/hnswPlaneBinding.ts | 76 +- unitTests/resources/vectorIndexPlane.test.js | 33 +- 25 files changed, 203 insertions(+), 4883 deletions(-) delete mode 100644 native/hnsw-plane/Cargo.lock delete mode 100644 native/hnsw-plane/Cargo.toml delete mode 100644 native/hnsw-plane/build.mjs delete mode 100644 native/hnsw-plane/build.rs delete mode 100644 native/hnsw-plane/smoke.mjs delete mode 100644 native/hnsw-plane/src/bin/bench.rs delete mode 100644 native/hnsw-plane/src/distance.rs delete mode 100644 native/hnsw-plane/src/format.rs delete mode 100644 native/hnsw-plane/src/graph.rs delete mode 100644 native/hnsw-plane/src/insert.rs delete mode 100644 native/hnsw-plane/src/lib.rs delete mode 100644 native/hnsw-plane/src/napi.rs delete mode 100644 native/hnsw-plane/src/search.rs delete mode 100644 native/hnsw-plane/src/seqlock.rs delete mode 100644 native/hnsw-plane/tests/concurrent.rs delete mode 100644 native/hnsw-plane/tests/reopen.rs diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml index 638d6bf2b3..fe0c960d42 100644 --- a/.github/workflows/unit-test.yml +++ b/.github/workflows/unit-test.yml @@ -119,8 +119,10 @@ jobs: timeout-minutes: 30 run: npm run test:unit:windows - # The vectorIndexPlane suite self-skips when the optional native artifact is absent; this - # job builds it so the dual-write mirroring and native search cutover run in CI. + # The vectorIndexPlane suite self-skips when the optional native package is absent, so the + # load probe below is what keeps that skip from reading as a green run. The crate's own + # tests belong to HarperFast/hnsw's CI, not here: this job runs the published prebuild, so + # testing the crate source would grade something other than the binary under test. hnsw-plane: name: HNSW native plane (Node.js v24) runs-on: ubuntu-latest @@ -141,11 +143,8 @@ jobs: - name: Build run: npm run build || true # we currently have type errors so just ignore that - - name: Build native hnsw-plane module - run: npm run build:hnsw-plane - - - name: Crate tests - run: cargo test --release --manifest-path native/hnsw-plane/Cargo.toml + - name: Verify the published native binding loads + run: node -e "if (typeof require('@harperfast/hnsw').Plane?.open !== 'function') throw new Error('@harperfast/hnsw loaded without a Plane constructor');" - name: Setup Harper env: diff --git a/.gitignore b/.gitignore index 20ec7714c3..3eae7855b2 100644 --- a/.gitignore +++ b/.gitignore @@ -67,7 +67,3 @@ test_export_terminology_test.json # dev-mode boots (harper dev ) symlink node_modules/harper into the # fixture component dir; keep those out of commits integrationTests/**/node_modules/ - -# hnsw-plane native build outputs (optional module; build locally with npm run build:hnsw-plane) -native/hnsw-plane/target/ -native/hnsw-plane/hnsw-plane.node diff --git a/dependencies.md b/dependencies.md index 566ef59c29..47be015a55 100644 --- a/dependencies.md +++ b/dependencies.md @@ -243,3 +243,15 @@ This is the inverse of the entries below — a dependency we take deliberate ste - Security: Microsoft-maintained TypeScript compiler, same publisher/package as the 5.x devDependency. - Overlap: Complements, does not replace, the `typescript` devDependency — TypeScript 7.0 ships no compiler API yet (planned for 7.1), so `@typescript-eslint/parser` still needs `typescript` 5.x. The 5.x and 7.x versions never coexist in `node_modules` at once: 5.x is the installed devDependency, 7.x is fetched on-demand by `npx` purely for `typecheck:fast`. - Eventual removal: Once TypeScript 7 stabilizes as the primary `typescript` devDependency (post-7.1's compiler API), this becomes redundant and `typecheck:fast` can be dropped. + +## @harperfast/hnsw (optional dependency) + +- Need for usage: Supplies the native memory-mapped HNSW traversal plane used only by indexes that opt in with `nativePlane: true`. Harper keeps the RocksDB graph authoritative in phase 1 and falls back to its JS traversal when the package cannot load. +- Size/memory cost: The JS/package metadata is about 250 KB unpacked plus one platform-specific native binary. Runtime mapped-file size is approximately 1,344 bytes per 768-dimensional int8 node at layer-0 cap 128; mappings are shared by the OS page cache across workers. +- Security: First-party Apache-2.0 Harper package. It runs native code in-process, so Harper exact-pins the package and its own manifest exact-pins every platform prebuild to the same version. The dedicated CI job loads the registry prebuild and tests Harper against it. +- Environment interaction: Lazily loaded only when an eligible index enables `nativePlane`; it creates a memory-mapped `.hnsw` derived-index file next to the index store and may create a `.stale` invalidation sidecar. It does not modify globals or install polyfills. +- Overlap: It mirrors the existing JS HNSW graph during the opt-in validation phase. The overlap is deliberate: native traversal removes per-node RocksDB reads and JS object bookkeeping while the existing graph remains the rollback path. +- Transitive dependencies: Only exact-version, platform-specific optional prebuild packages; no JS runtime dependency tree. +- Binary compilation: Supported Linux glibc x64/arm64, macOS arm64, and Windows x64 targets use prebuilds. Other targets attempt a Rust source build; because the root package is optional, a failed build leaves the JS path available. +- Can be deferred: Yes. The adapter requires it lazily and caches an unavailable result after one warning. +- Eventual removal: Remove the optional dependency and adapter integration, delete derived `.hnsw` files, and use the existing JS/CF graph path. The `nativePlane` flag is explicitly rollback-safe in phase 1. diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 92e229b969..f0fd87364c 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -68,7 +68,7 @@ Non-goals (this phase): │ HierarchicalNavigableSmallWorld.ts │ │ hnsw-plane │ │ • pk→nodeId mapping (stays RocksDB) │ │ • mmap'd slot file (per index/slice)│ │ • insert/update/delete logic (phase 1) ├──►│ • slot read/write API (seqlocked) │ - │ • commit callback → slot writes │ │ • search(query, k, ef, filter) → │ + │ • phase-1 mirror → slot writes │ │ • search(query, k, ef, filter) → │ │ • record load + exact rescore (as-is) │◄──┤ top-k ids, own thread pool │ │ • runIndexing replay from watermark │ │ • TSFN batch filter callback │ └─────────────────────────────────────────┘ └─────────────────────────────────────┘ @@ -222,12 +222,18 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d exact `indexStore.put/remove` sites via `writeNodeRaw`/`clearNode`/`setEntryPoint` with host-allocated ids; the plane file (`/
..hnsw`, layer0 cap 128, 16M-node sparse reservation) is created lazily with a full mirror of the existing CF graph on - first enable, reopened on restart, deleted on drop/clear/reindex. The compiled module is - optional (`npm run build:hnsw-plane`); absence falls back to the JS path with one warning. + first enable, reopened on restart, deleted on drop/clear/reindex. The exact-pinned + `@harperfast/hnsw` package is optional; absence falls back to the JS path with one warning. Parity, predicate, restart, and lifecycle coverage in `unitTests/resources/vectorIndexPlane.test.js`. Watermark/replay wiring, slicing, and msync-cadence flushes are not wired yet (open items). -- **Phase 2 — file-primary.** Drop the CF writes; the file is the only graph store. JS insert +- **Phase 2 — shared post-commit delivery, then file-primary.** Implement #2489's + `DerivedIndexBackend` contract rather than an HNSW-specific commit callback: every worker + enqueues its own committed transaction-log entries, the backend advances a durable log-position + watermark, and open/rebuild uses the shared retention-aware replay driver. Moving the mirror + from the pre-commit `indexStore.put/remove` sites into that delivery path removes rollback + phantoms without creating a second protocol alongside full-text indexing. Then drop the CF + graph writes; the file is the only graph store. JS insert reads nodes through a native `getNode(id)` (one NAPI crossing per read, ~1 µs — comparable to today's decode path). Migration for existing indexes: reindex (accepted contract), or a one-shot CF→file bulk conversion since it is a pure format transform. @@ -270,12 +276,13 @@ Decided (Kris, 2026-08-31): - **Packaging: independent open-source package.** The core has zero Harper coupling — the crate compiles standalone and its NAPI surface is generic (create/open plane, insert(id, vector), remove(id), search(query, k, ef, filter), watermark get/set). Harper-specific glue — the - pk→nodeId mapping, commit-callback integration, txnlog-anchored replay, auto-ef policy - constants — stays in Harper regardless of packaging. Plan: develop in-repo under - `native/hnsw-plane/` until the NAPI surface stabilizes (end of phase 1), then split to its own - repo in the symphony/lmdb-js mold and consume via npm. The pitch as a community package: a - persistent, incrementally-maintained, concurrently-searchable HNSW for Node — hnswlib-node has - no durable incremental persistence, no off-loop batched filtering, no seqlock concurrency. + pk→nodeId mapping, #2489's `DerivedIndexBackend` delivery, txnlog-anchored replay, auto-ef + policy constants — stays in Harper regardless of packaging. Published as the exact-pinned + optional dependency `@harperfast/hnsw` 0.2.1 (Apache-2.0, HarperFast/hnsw), with platform + prebuilds and a source-build fallback; the Harper adapter owns only availability/fallback and + integration policy. The pitch as a community package: a persistent, + incrementally-maintained, concurrently-searchable HNSW for Node — hnswlib-node has no durable + incremental persistence, no off-loop batched filtering, no seqlock concurrency. Open: @@ -307,8 +314,8 @@ bounded by the phase-1 contract (opt-in flag, CF authoritative, plane derived): transaction can leave phantom nodes in the plane (the CF never had them). Phantom ids are filtered at record load (missing record → SKIP), so results can be transiently short by the phantom count; the garbage accumulates only at the rollback rate. The structural fix — - driving the mirror from committed state (commit callback / txnlog consumer) — is the - phase-2 "watermark/replay wiring" work item and also subsumes the residual lost-write + driving the mirror through #2489's shared post-commit `DerivedIndexBackend` runtime — is the + phase-2 delivery/watermark/replay work item and also subsumes the residual lost-write window during attach retry (mirror calls during the 250 ms backoff are dropped and heal only on the node's next touch). - **The async custom-index search contract** (`resources/search.ts`): a plane-backed search diff --git a/native/hnsw-plane/Cargo.lock b/native/hnsw-plane/Cargo.lock deleted file mode 100644 index 91316d129a..0000000000 --- a/native/hnsw-plane/Cargo.lock +++ /dev/null @@ -1,230 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "bitflags" -version = "2.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "convert_case" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec182b0ca2f35d8fc196cf3404988fd8b8c739a4d270ff118a398feb0cbec1ca" -dependencies = [ - "unicode-segmentation", -] - -[[package]] -name = "ctor" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" -dependencies = [ - "quote", - "syn", -] - -[[package]] -name = "hnsw-plane" -version = "0.0.1" -dependencies = [ - "libc", - "memmap2", - "napi", - "napi-build", - "napi-derive", -] - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "libloading" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" -dependencies = [ - "cfg-if", - "windows-link", -] - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "memmap2" -version = "0.9.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" -dependencies = [ - "libc", -] - -[[package]] -name = "napi" -version = "2.16.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55740c4ae1d8696773c78fdafd5d0e5fe9bc9f1b071c7ba493ba5c413a9184f3" -dependencies = [ - "bitflags", - "ctor", - "napi-derive", - "napi-sys", - "once_cell", -] - -[[package]] -name = "napi-build" -version = "2.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60fdf9b392c50e7c4170fa633bd909490ed7835cea4c046776d1a4dd8d2ae0ab" - -[[package]] -name = "napi-derive" -version = "2.16.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cbe2585d8ac223f7d34f13701434b9d5f4eb9c332cccce8dee57ea18ab8ab0c" -dependencies = [ - "cfg-if", - "convert_case", - "napi-derive-backend", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "napi-derive-backend" -version = "1.0.75" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1639aaa9eeb76e91c6ae66da8ce3e89e921cd3885e99ec85f4abacae72fc91bf" -dependencies = [ - "convert_case", - "once_cell", - "proc-macro2", - "quote", - "regex", - "semver", - "syn", -] - -[[package]] -name = "napi-sys" -version = "2.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "427802e8ec3a734331fec1035594a210ce1ff4dc5bc1950530920ab717964ea3" -dependencies = [ - "libloading", -] - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "regex" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "unicode-segmentation" -version = "1.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" diff --git a/native/hnsw-plane/Cargo.toml b/native/hnsw-plane/Cargo.toml deleted file mode 100644 index 3fed3cfac0..0000000000 --- a/native/hnsw-plane/Cargo.toml +++ /dev/null @@ -1,30 +0,0 @@ -[package] -name = "hnsw-plane" -version = "0.0.1" -edition = "2021" -description = "Native HNSW traversal plane: mmap fixed-slot graph file + off-loop search" -license = "Apache-2.0" - -[lib] -crate-type = ["cdylib", "rlib"] - -[dependencies] -libc = "0.2" -memmap2 = "0.9" -napi = { version = "2", default-features = false, features = ["napi8"], optional = true } -napi-derive = { version = "2", optional = true } - -[build-dependencies] -napi-build = "2" - -[features] -default = [] -napi = ["dep:napi", "dep:napi-derive"] - -[[bin]] -name = "bench" -path = "src/bin/bench.rs" - -[profile.release] -lto = true -codegen-units = 1 diff --git a/native/hnsw-plane/build.mjs b/native/hnsw-plane/build.mjs deleted file mode 100644 index af82c4bf0f..0000000000 --- a/native/hnsw-plane/build.mjs +++ /dev/null @@ -1,19 +0,0 @@ -// Builds the optional hnsw-plane NAPI module in place (`npm run build:hnsw-plane`); harper -// installs never require a cargo toolchain — without the artifact, nativePlane falls back to -// the JS path. -import { execSync } from 'node:child_process'; -import { copyFileSync } from 'node:fs'; -import { dirname, join } from 'node:path'; -import { fileURLToPath } from 'node:url'; - -const crateRoot = dirname(fileURLToPath(import.meta.url)); -// build the lib alone: the bench bin cannot link against unresolved node-api symbols -execSync('cargo build --release --features napi --lib', { cwd: crateRoot, stdio: 'inherit' }); -const cdylib = - process.platform === 'win32' - ? 'hnsw_plane.dll' - : process.platform === 'darwin' - ? 'libhnsw_plane.dylib' - : 'libhnsw_plane.so'; -copyFileSync(join(crateRoot, 'target', 'release', cdylib), join(crateRoot, 'hnsw-plane.node')); -console.log('built native/hnsw-plane/hnsw-plane.node'); diff --git a/native/hnsw-plane/build.rs b/native/hnsw-plane/build.rs deleted file mode 100644 index 89463eb5e6..0000000000 --- a/native/hnsw-plane/build.rs +++ /dev/null @@ -1,6 +0,0 @@ -fn main() { - // napi_build wires the node-api link args for the cdylib; only needed for the napi feature - if std::env::var("CARGO_FEATURE_NAPI").is_ok() { - napi_build::setup(); - } -} diff --git a/native/hnsw-plane/smoke.mjs b/native/hnsw-plane/smoke.mjs deleted file mode 100644 index b2cbf796ce..0000000000 --- a/native/hnsw-plane/smoke.mjs +++ /dev/null @@ -1,84 +0,0 @@ -// End-to-end smoke test: `npm run build:hnsw-plane && node native/hnsw-plane/smoke.mjs`. -import { createRequire } from 'module'; -const require = createRequire(import.meta.url); -const { Plane } = require('./hnsw-plane.node'); - -const dims = 64; -const { tmpdir } = await import('node:os'); -const { join } = await import('node:path'); -const path = join(tmpdir(), `smoke-${process.pid}.hnsw`); -const plane = Plane.create(path, dims, 32, 10_000); - -function vec(i) { - const v = new Float32Array(dims); - for (let d = 0; d < dims; d++) v[d] = Math.sin(i * 0.37 + d * 1.13) * 0.1 + (d % 7 === i % 7 ? 1 : 0); - return v; -} - -const ids = []; -for (let i = 0; i < 2000; i++) ids.push(plane.insert(vec(i))); -console.log('inserted 2000, highWater =', plane.idHighWater()); - -// async search: nearest neighbor of an inserted vector is itself (distance ~0) -const hits = await plane.search(vec(42), 5, 128); -console.log('top hit:', hits[0]); -if (hits[0].distance > 1e-3) throw new Error('self-query failed'); - -// filtered search: allow only even ids -const bitset = new Uint8Array(Math.ceil(plane.idHighWater() / 8)); -for (const id of ids) if (id % 2 === 0) bitset[id >> 3] |= 1 << (id & 7); -const filtered = await plane.search(vec(43), 5, 128, bitset); -for (const h of filtered) if (h.id % 2 !== 0) throw new Error(`filter leak: id ${h.id}`); -console.log('filtered top hit:', filtered[0]); - -// delete + reinsert reuses the id (the #2182 fix) -plane.remove(ids[7]); -const reused = plane.insert(vec(9001)); -if (reused !== ids[7]) throw new Error(`expected id reuse of ${ids[7]}, got ${reused}`); -console.log('freelist reuse OK, highWater still', plane.idHighWater()); - -// pipelined JS predicate: admit only ids divisible by 3; verdicts computed on the JS -// event loop while traversal runs on the libuv pool -let predicateCalls = 0; -const pred = await plane.searchWithPredicate(vec(44), 5, 128, (ids) => { - predicateCalls++; - return Uint8Array.from(ids, (id) => (id % 3 === 0 ? 1 : 0)); -}); -for (const h of pred) if (h.id % 3 !== 0) throw new Error(`predicate leak: id ${h.id}`); -if (pred.length === 0) throw new Error('predicate search returned nothing'); -console.log(`predicate top hit: id ${pred[0].id} (calls: ${predicateCalls})`); - -// raw mirroring path (dual-write phase 1): host-allocated ids, full node state per call -const mirror = Plane.create(join(tmpdir(), `smoke-mirror-${process.pid}.hnsw`), dims, 32, 10_000); -const q42 = vec(42); -// quantize like the host: scale maps max|c| to 127, invMag = 1/|v| -function quant(v) { - let maxAbs = 0, - magSq = 0; - for (const x of v) { - maxAbs = Math.max(maxAbs, Math.abs(x)); - magSq += x * x; - } - const scale = maxAbs === 0 ? 1 : maxAbs / 127; - const bytes = Buffer.from(Int8Array.from(v, (x) => Math.max(-127, Math.min(127, Math.round(x / scale)))).buffer); - return { bytes, scale, invMag: 1 / Math.sqrt(magSq) }; -} -// two nodes linked to each other, host ids 10 and 20; node 10 is the entry at level 1 -const a = quant(q42), - b = quant(vec(43)); -mirror.writeNodeRaw(10, 1, a.bytes, a.scale, a.invMag, Uint32Array.from([20]), [Uint32Array.from([])]); -mirror.writeNodeRaw(20, 0, b.bytes, b.scale, b.invMag, Uint32Array.from([10]), null); -mirror.setEntryPoint(10, 1); -const mhits = mirror.searchSync(q42, 2, 16); -if (mhits[0].id !== 10 || mhits[0].distance > 1e-3) - throw new Error(`mirror self-query failed: ${JSON.stringify(mhits)}`); -mirror.clearNode(20); -const mhits2 = mirror.searchSync(vec(43), 2, 16); -if (mhits2.some((h) => h.id === 20)) throw new Error('cleared node still returned'); -console.log('raw mirroring OK'); - -plane.flush(); -const reopened = Plane.open(path); -const hits2 = reopened.searchSync(vec(42), 5, 128); -if (hits2[0].distance > 1e-3) throw new Error('reopened self-query failed'); -console.log('reopen + sidecar OK. smoke PASSED'); diff --git a/native/hnsw-plane/src/bin/bench.rs b/native/hnsw-plane/src/bin/bench.rs deleted file mode 100644 index f3fe4d2f47..0000000000 --- a/native/hnsw-plane/src/bin/bench.rs +++ /dev/null @@ -1,246 +0,0 @@ -//! Standalone cost benchmark: build an N-node graph in the plane file, run queries, report -//! per-visit cost — the number that decides whether the native plane hits its 0.25–0.4 µs -//! budget (JS baseline: 4.34 µs/visit at 5M/ef 512). -//! -//! Usage: bench [n=100000] [dims=768] [queries=200] [ef=512] [path=/tmp/bench.hnsw] [cap=128] [threads=0] -//! threads > 0 adds a concurrent-throughput pass: T searcher threads (queries each) + one -//! background writer inserting throughout, reporting aggregate QPS and per-thread p50/p99. - -use hnsw_plane::distance::Query; -use hnsw_plane::insert::{insert, InsertParams}; -use hnsw_plane::search::{search, SearchScratch}; -use hnsw_plane::{Graph, PlaneFile}; -use std::path::PathBuf; -use std::time::Instant; - -// xorshift for reproducible synthetic vectors without a rand dependency -struct Rng(u64); -impl Rng { - fn next_unit(&mut self) -> f32 { - self.0 ^= self.0 << 13; - self.0 ^= self.0 >> 7; - self.0 ^= self.0 << 17; - (self.0 >> 40) as f32 / (1u64 << 24) as f32 - } - // Box-Muller - fn next_gauss(&mut self) -> f32 { - let u1 = self.next_unit().max(f32::MIN_POSITIVE); - let u2 = self.next_unit(); - (-2.0 * u1.ln()).sqrt() * (2.0 * std::f32::consts::PI * u2).cos() - } -} - -/// Gaussian-mixture corpus matching benchmarks/hnsw-scale.js: unit centroids, per-dim noise -/// derived from an intra-cluster cosine target of 0.75 (uniform-random 768-d is a corpus -/// "no ANN can index" per that benchmark's own calibration notes). -struct Corpus { - centroids: Vec, - n_clusters: usize, - dims: usize, - noise: f32, -} - -impl Corpus { - fn new(n: u64, dims: usize, rng: &mut Rng) -> Self { - let intra_cos = 0.75f32; - let noise = ((1.0 / (intra_cos * intra_cos) - 1.0) / dims as f32).sqrt(); - let n_clusters = 8.max((n as f64 / 500.0).round() as usize); - let mut centroids = vec![0.0f32; n_clusters * dims]; - for c in 0..n_clusters { - let mut mag = 0.0f32; - for d in 0..dims { - let x = rng.next_gauss(); - centroids[c * dims + d] = x; - mag += x * x; - } - let mag = mag.sqrt().max(f32::MIN_POSITIVE); - for d in 0..dims { - centroids[c * dims + d] /= mag; - } - } - Corpus { centroids, n_clusters, dims, noise } - } - - fn row(&self, rng: &mut Rng) -> Vec { - let c = (rng.next_unit() * self.n_clusters as f32) as usize % self.n_clusters; - let mut v = vec![0.0f32; self.dims]; - let mut mag = 0.0f32; - for d in 0..self.dims { - let x = self.centroids[c * self.dims + d] + rng.next_gauss() * self.noise; - v[d] = x; - mag += x * x; - } - let mag = mag.sqrt().max(f32::MIN_POSITIVE); - for d in 0..self.dims { - v[d] /= mag; - } - v - } -} - -fn main() { - let args: Vec = std::env::args().collect(); - let n: u64 = args.get(1).and_then(|a| a.parse().ok()).unwrap_or(100_000); - let dims: usize = args.get(2).and_then(|a| a.parse().ok()).unwrap_or(768); - let queries: usize = args.get(3).and_then(|a| a.parse().ok()).unwrap_or(200); - let ef: usize = args.get(4).and_then(|a| a.parse().ok()).unwrap_or(512); - let path: PathBuf = args.get(5).map(Into::into).unwrap_or_else(|| "/tmp/bench.hnsw".into()); - let layer0_cap: usize = args.get(6).and_then(|a| a.parse().ok()).unwrap_or(128); - - // Reuse an existing plane file when it already holds exactly n nodes at the same cap - // (ef sweeps without rebuilding). The corpus RNG below replays identically. - let reuse = PlaneFile::open(&path) - .ok() - .filter(|f| f.id_high_water() == n && f.layer0_cap == layer0_cap) - .is_some(); - let file = if reuse { - println!("reusing existing plane at {}", path.display()); - PlaneFile::open(&path).expect("open") - } else { - PlaneFile::create(&path, dims, layer0_cap, n + 1024).expect("create") - }; - println!( - "plane: {} nodes x {} dims, slot {} B, file {:.1} GB (sparse)", - n, - dims, - file.slot_size, - (n * file.slot_size as u64) as f64 / 1e9 - ); - let graph = Graph::new(file); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - let mut rng = Rng(0x1234_5678_9abc_def0); - let corpus = Corpus::new(n, dims, &mut rng); - - if reuse { - // replay the build's RNG draws so query rows match a fresh run; the upper region - // persists inside the plane file - for _ in 0..n { - let _ = corpus.row(&mut rng); - } - } else { - let build_start = Instant::now(); - for i in 0..n { - let v = corpus.row(&mut rng); - insert(&graph, &v, ¶ms, &mut scratch).expect("build insert"); - if (i + 1) % 50_000 == 0 { - let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64(); - println!(" built {} ({:.0} inserts/s)", i + 1, rate); - } - } - let build = build_start.elapsed(); - println!("build: {:.1}s ({:.0} inserts/s)", build.as_secs_f64(), n as f64 / build.as_secs_f64()); - graph.file.msync().expect("msync"); - } - - // Query with held-out vectors; measure latency and set-recall@10 vs brute-force truth - // (same asymmetric metric, so recall isolates graph quality, not quantization). - let mut latencies = Vec::with_capacity(queries); - let mut total_visits = 0u64; - let mut recall_hits = 0usize; - let mut recall_total = 0usize; - for _ in 0..queries { - let q = Query::new(corpus.row(&mut rng)); - let start = Instant::now(); - let (results, stats) = search(&graph, &q, 10, ef, &mut scratch); - latencies.push(start.elapsed()); - total_visits += stats.visits; - assert!(!results.is_empty()); - - let mut truth: Vec<(u32, f32)> = (0..n as u32) - .filter_map(|id| graph.distance_to(id, &q).map(|d| (id, d))) - .collect(); - truth.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap()); - truth.truncate(10); - recall_total += truth.len(); - recall_hits += truth.iter().filter(|(tid, _)| results.iter().any(|(rid, _)| rid == tid)).count(); - } - latencies.sort(); - let p50 = latencies[queries / 2]; - let p95 = latencies[queries * 95 / 100]; - let p99 = latencies[(queries * 99 / 100).min(queries - 1)]; - let mean_visits = total_visits as f64 / queries as f64; - let us_per_visit = p50.as_micros() as f64 / mean_visits; - println!( - "search (ef {}): p50 {:.2} ms p95 {:.2} ms p99 {:.2} ms visits/query {:.0} -> {:.3} us/visit (JS baseline 4.34)", - ef, - p50.as_secs_f64() * 1e3, - p95.as_secs_f64() * 1e3, - p99.as_secs_f64() * 1e3, - mean_visits, - us_per_visit - ); - println!("recall@10 (set): {:.3}", recall_hits as f64 / recall_total as f64); - - let threads: usize = args.get(7).and_then(|a| a.parse().ok()).unwrap_or(0); - if threads > 0 { - use std::sync::atomic::{AtomicBool, Ordering}; - use std::sync::Arc; - let graph = Arc::new(graph); - let corpus = Arc::new(corpus); - let stop = Arc::new(AtomicBool::new(false)); - let per_thread = queries.max(100); - let start = Instant::now(); - let mut handles = Vec::new(); - for t in 0..threads { - let graph = graph.clone(); - let corpus = corpus.clone(); - handles.push(std::thread::spawn(move || { - let mut scratch = SearchScratch::new(); - let mut rng = Rng(0x9e37_79b9 ^ (t as u64 + 1) * 0x1234_5677); - let mut lat: Vec = Vec::with_capacity(per_thread); - for _ in 0..per_thread { - let q = Query::new(corpus.row(&mut rng)); - let s = Instant::now(); - let (r, _) = search(&graph, &q, 10, ef, &mut scratch); - lat.push(s.elapsed()); - assert!(!r.is_empty()); - } - lat.sort(); - (lat[per_thread / 2], lat[(per_thread * 99 / 100).min(per_thread - 1)]) - })); - } - // background writer: sustained inserts while searchers run - let writer = { - let graph = graph.clone(); - let corpus = corpus.clone(); - let stop = stop.clone(); - std::thread::spawn(move || { - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - let mut rng = Rng(0xdead_beef_cafe_f00d); - let mut count = 0u64; - while !stop.load(Ordering::Relaxed) { - let v = corpus.row(&mut rng); - if insert(&graph, &v, ¶ms, &mut scratch).is_err() { - break; // plane full - } - count += 1; - } - count - }) - }; - let mut p50s = Vec::new(); - let mut p99s = Vec::new(); - for h in handles { - let (p50, p99) = h.join().unwrap(); - p50s.push(p50); - p99s.push(p99); - } - let wall = start.elapsed(); - stop.store(true, Ordering::Relaxed); - let inserted = writer.join().unwrap(); - let total_q = (threads * per_thread) as f64; - p50s.sort(); - p99s.sort(); - println!( - "concurrent: {} threads x {} queries + writer -> {:.0} QPS aggregate p50(med) {:.2} ms p99(worst) {:.2} ms writer {:.0} inserts/s", - threads, - per_thread, - total_q / wall.as_secs_f64(), - p50s[threads / 2].as_secs_f64() * 1e3, - p99s[threads - 1].as_secs_f64() * 1e3, - inserted as f64 / wall.as_secs_f64() - ); - } -} diff --git a/native/hnsw-plane/src/distance.rs b/native/hnsw-plane/src/distance.rs deleted file mode 100644 index 949378aced..0000000000 --- a/native/hnsw-plane/src/distance.rs +++ /dev/null @@ -1,138 +0,0 @@ -//! Distance kernels. Asymmetric: full-precision f32 query × int8-stored vector (matches the JS -//! quantizeInt8 scale + cached 1/|v| model). Symmetric int8×int8 for construction-time -//! neighbor↔neighbor checks (stored per-edge distances were dropped from the format; recompute). -//! AVX2 with scalar fallback; Linux x86_64 is the performance target, other platforms take the -//! scalar path (fine for dev). - -/// Precomputed query state, built once per search. -pub struct Query { - pub vector: Vec, - pub inv_mag: f32, -} - -impl Query { - pub fn new(vector: Vec) -> Self { - let mag_sq: f32 = vector.iter().map(|v| v * v).sum(); - let inv_mag = 1.0 / mag_sq.sqrt().max(f32::MIN_POSITIVE); - Query { vector, inv_mag } - } -} - -#[inline] -fn dot_f32_i8_scalar(q: &[f32], v: *const i8) -> f32 { - let mut acc = [0.0f32; 8]; - let chunks = q.len() / 8; - for c in 0..chunks { - let base = c * 8; - for lane in 0..8 { - acc[lane] += q[base + lane] * unsafe { *v.add(base + lane) } as f32; - } - } - let mut dot: f32 = acc.iter().sum(); - for i in chunks * 8..q.len() { - dot += q[i] * unsafe { *v.add(i) } as f32; - } - dot -} - -#[cfg(target_arch = "x86_64")] -#[target_feature(enable = "avx2", enable = "fma")] -unsafe fn dot_f32_i8_avx2(q: &[f32], v: *const i8) -> f32 { - use std::arch::x86_64::*; - let mut acc0 = _mm256_setzero_ps(); - let mut acc1 = _mm256_setzero_ps(); - let chunks = q.len() / 16; - for c in 0..chunks { - let base = c * 16; - let v16 = _mm_loadu_si128(v.add(base) as *const __m128i); - let lo = _mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(v16)); - let hi = _mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(_mm_srli_si128(v16, 8))); - acc0 = _mm256_fmadd_ps(_mm256_loadu_ps(q.as_ptr().add(base)), lo, acc0); - acc1 = _mm256_fmadd_ps(_mm256_loadu_ps(q.as_ptr().add(base + 8)), hi, acc1); - } - let acc = _mm256_add_ps(acc0, acc1); - let s = _mm_add_ps(_mm256_extractf128_ps(acc, 1), _mm256_castps256_ps128(acc)); - let s = _mm_hadd_ps(s, s); - let s = _mm_hadd_ps(s, s); - let mut dot = _mm_cvtss_f32(s); - for i in chunks * 16..q.len() { - dot += q[i] * *v.add(i) as f32; - } - dot -} - -#[inline] -fn dot_f32_i8(q: &[f32], v: *const i8) -> f32 { - #[cfg(target_arch = "x86_64")] - { - if std::arch::is_x86_feature_detected!("avx2") && std::arch::is_x86_feature_detected!("fma") { - return unsafe { dot_f32_i8_avx2(q, v) }; - } - } - dot_f32_i8_scalar(q, v) -} - -/// Cosine distance: f32 query × raw int8 vector at `stored` (dims = query.vector.len()). -/// Zero-copy: `stored` points into the mmap; the caller's seqlock read discards torn results. -#[inline] -pub fn cosine_int8_raw(query: &Query, stored: *const i8, scale: f32, stored_inv_mag: f32) -> f32 { - let dot = dot_f32_i8(&query.vector, stored); - 1.0 - dot * scale * stored_inv_mag * query.inv_mag -} - -#[inline] -fn dot_i8_i8_scalar(a: *const i8, b: *const i8, len: usize) -> i32 { - let mut dot = 0i32; - for i in 0..len { - dot += unsafe { *a.add(i) as i32 * *b.add(i) as i32 }; - } - dot -} - -#[cfg(target_arch = "x86_64")] -#[target_feature(enable = "avx2")] -unsafe fn dot_i8_i8_avx2(a: *const i8, b: *const i8, len: usize) -> i32 { - use std::arch::x86_64::*; - let mut acc = _mm256_setzero_si256(); - let chunks = len / 16; - for c in 0..chunks { - let av = _mm256_cvtepi8_epi16(_mm_loadu_si128(a.add(c * 16) as *const __m128i)); - let bv = _mm256_cvtepi8_epi16(_mm_loadu_si128(b.add(c * 16) as *const __m128i)); - acc = _mm256_add_epi32(acc, _mm256_madd_epi16(av, bv)); - } - let lo = _mm256_castsi256_si128(acc); - let hi = _mm256_extracti128_si256(acc, 1); - let s = _mm_add_epi32(lo, hi); - let s = _mm_add_epi32(s, _mm_srli_si128(s, 8)); - let s = _mm_add_epi32(s, _mm_srli_si128(s, 4)); - let mut dot = _mm_cvtsi128_si32(s); - for i in chunks * 16..len { - dot += *a.add(i) as i32 * *b.add(i) as i32; - } - dot -} - -/// Cosine distance between two int8-stored vectors (construction-time neighbor checks). -#[inline] -pub fn cosine_i8_i8_raw(a: *const i8, scale_a: f32, inv_mag_a: f32, b: *const i8, scale_b: f32, inv_mag_b: f32, len: usize) -> f32 { - #[cfg(target_arch = "x86_64")] - let dot = if std::arch::is_x86_feature_detected!("avx2") { - unsafe { dot_i8_i8_avx2(a, b, len) } - } else { - dot_i8_i8_scalar(a, b, len) - }; - #[cfg(not(target_arch = "x86_64"))] - let dot = dot_i8_i8_scalar(a, b, len); - 1.0 - dot as f32 * scale_a * scale_b * inv_mag_a * inv_mag_b -} - -/// Symmetric int8 quantization matching the JS quantizeInt8: scale maps max |component| to 127. -pub fn quantize_int8(vector: &[f32]) -> (Vec, f32, f32) { - let max_abs = vector.iter().fold(0.0f32, |m, v| m.max(v.abs())); - let scale = if max_abs == 0.0 { 1.0 } else { max_abs / 127.0 }; - let inv_scale = 1.0 / scale; - let bytes: Vec = vector.iter().map(|v| (v * inv_scale).round().clamp(-127.0, 127.0) as i8).collect(); - let mag_sq: f32 = vector.iter().map(|v| v * v).sum(); - let inv_mag = 1.0 / mag_sq.sqrt().max(f32::MIN_POSITIVE); - (bytes, scale, inv_mag) -} diff --git a/native/hnsw-plane/src/format.rs b/native/hnsw-plane/src/format.rs deleted file mode 100644 index 3241b63e91..0000000000 --- a/native/hnsw-plane/src/format.rs +++ /dev/null @@ -1,687 +0,0 @@ -//! On-disk format: 4 KB header + fixed-size layer-0 slot array + upper-layer region. -//! See ../../../hnsw-native-plane.md §4. Format changes bump VERSION and require reindex. - -use memmap2::MmapMut; -use std::fs::OpenOptions; -use std::io; -use std::path::Path; -use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; - -pub const MAGIC: u32 = 0x484e_5357; // "HNSW" -pub const VERSION: u32 = 6; // v6: 4-aligned neighbor + upper id arrays (older files: reindex) -pub const HEADER_SIZE: usize = 4096; - -// Header field byte offsets. -const H_MAGIC: usize = 0; -const H_VERSION: usize = 4; -const H_DIMS: usize = 8; // u16 -const H_QUANT: usize = 10; // u8: 0 = int8, 1 = f32 -const H_LAYER0_CAP: usize = 12; // u16 -const H_SLOT_SIZE: usize = 16; // u32 -const H_ENTRY: usize = 24; // u64 atomic: (level << 32) | id, one word so readers never see a torn pair -const H_ID_HIGH_WATER: usize = 32; // u64 atomic -const H_FREELIST_HEAD: usize = 40; // u64 atomic: (tag << 32) | id; id u32::MAX = empty -const H_TXN_WATERMARK: usize = 48; // u64 -const H_CLEAN_SHUTDOWN: usize = 56; // u8 -const H_MAX_NODES: usize = 64; // u64 -const H_UPPER_HIGH_WATER: usize = 72; // u64 atomic: upper-entry allocator -const H_UPPER_FREELIST: usize = 80; // u64 atomic: (tag<<32)|idx; NO_UPPER = empty -const H_ENTRY_PREV: usize = 88; // u64: last replaced entry point (re-election hint) -// Opener registry: each live handle claims one slot, writes its random tag there, and holds -// a kernel OFD byte-range lock on the slot (released automatically when the handle - or its -// whole process - dies). A lock word's owner is dead iff its registry slot no longer carries -// its tag or the slot's byte range is lockable. Immune to pid reuse and pid namespaces. -const H_REGISTRY: usize = 128; // u32 x REGISTRY_SLOTS -pub const REGISTRY_SLOTS: usize = 64; - -/// Upper-layer region geometry: fixed entries covering levels 1..=MAX_UPPER_LEVELS at -/// UPPER_CAP ids per level. P(level >= 1) = 1/M ~ 6.25%; the region reserves entries for -/// 1/8 of max_nodes (2x headroom). P(level >= 9) at mL = 1/ln16 is ~e^-25 — unreachable. -pub const MAX_UPPER_LEVELS: usize = 8; -pub const UPPER_CAP: usize = 64; // matches the JS graph's upper cap (M<<2 under optimizeRouting) -// entry: seq u32 | levels u8 | pad | per-level (degree u16 + pad u16 + ids u32*UPPER_CAP) -pub const U_SEQ: usize = 0; -pub const U_LEVELS: usize = 4; -pub const U_LISTS: usize = 8; -/// The pad follows the degree rather than the ids so every id array starts 4-aligned; the -/// stride (and so the entry size) is unchanged either way. -pub const UL_DEGREE: usize = 0; -pub const UL_IDS: usize = 4; -pub const UPPER_LEVEL_STRIDE: usize = UL_IDS + UPPER_CAP * 4; -pub const NO_UPPER: u32 = u32::MAX; - -// Slot layout offsets (within a slot). -pub const S_SEQ: usize = 0; // u32 seqlock -pub const S_FLAGS: usize = 4; // u8: bit0 = valid, bit1 = deleted -pub const S_LEVEL: usize = 5; // u8 -pub const S_DEGREE: usize = 6; // u16 -pub const S_SCALE: usize = 8; // f32 -pub const S_INV_MAG: usize = 12; // f32 -pub const S_UPPER_IDX: usize = 16; // u32 index into the upper region; NO_UPPER = none -pub const S_VECTOR: usize = 20; // dims bytes (int8) or dims*4 (f32) - // neighbors: u32 * layer0_cap, follows the 4-padded vector - -/// Byte offset of a slot's neighbor array. The vector is padded to a 4-byte boundary so this -/// is 4-aligned for every dims: the search hot path then reads each neighbor as one aligned -/// volatile u32 instead of four byte loads plus shifts. -#[inline] -pub const fn neighbor_offset(dims: usize) -> usize { - S_VECTOR + (dims + 3) / 4 * 4 -} - -pub const FLAG_VALID: u8 = 1; -pub const FLAG_DELETED: u8 = 2; -pub const NO_ID: u32 = u32::MAX; - -pub struct PlaneFile { - /// Kept open for the lifetime of the mapping: the opener-registry OFD lock lives on it. - file: std::fs::File, - /// This handle's registry tag (low bits encode its registry slot). 0 = unregistered - /// (registry full or platform without OFD locks): this handle's own dead locks cannot be - /// reclaimed by others, and it never reclaims. - pub self_tag: u32, - pub map: MmapMut, - pub dims: usize, - pub layer0_cap: usize, - pub slot_size: usize, - pub max_nodes: u64, - upper_offset: usize, - pub upper_capacity: u64, - /// Whether the file recorded a clean shutdown when opened (create() reports true). - /// An unclean open has had its torn seqlocks scrubbed, but individual slots may hold - /// unflushed/partial states — hosts should rebuild rather than trust completeness. - pub opened_clean: bool, - /// Slots per 4 KB page under page-grouped addressing; 0 = packed (slots may straddle - /// pages). Grouped is chosen at create when the per-page waste is small (e.g. 1,344 B - /// slots: 3/page, 64 B waste). Straddling only costs on cold faults, but the layout is - /// header-pinned so it must be decided before any data exists. - pub slots_per_page: usize, -} - -const PAGE: usize = 4096; -const H_SLOTS_PER_PAGE: usize = 20; // u16 - -/// MADV_RANDOM: hosts packing many instances live in permanent memory pressure, where -/// evict-and-refault is steady state; default readahead pulls ~16 unwanted pages per random -/// re-fault, taxing every tenant's page cache. The plane has no sequential reader to protect -/// (search is pointer-chasing, the builder writes, backfill scans read the host store). -fn advise_random(map: &MmapMut) { - #[cfg(unix)] - let _ = map.advise(memmap2::Advice::Random); - #[cfg(not(unix))] - let _ = map; -} - -fn slot_size_for(dims: usize, layer0_cap: usize) -> usize { - let raw = neighbor_offset(dims) + layer0_cap * 4; - raw.next_multiple_of(64) // cache-line align -} - -fn upper_entry_size() -> usize { - (U_LISTS + MAX_UPPER_LEVELS * UPPER_LEVEL_STRIDE).next_multiple_of(64) -} - -fn slot_region_len(max_nodes: u64, slot_size: usize, slots_per_page: usize) -> u64 { - if slots_per_page > 0 { - max_nodes.div_ceil(slots_per_page as u64) * PAGE as u64 - } else { - max_nodes * slot_size as u64 - } -} - -fn slots_per_page_for(slot_size: usize) -> usize { - if slot_size > PAGE { - return 0; - } - let per = PAGE / slot_size; - let waste = PAGE - per * slot_size; - // group when waste is under ~3% of the page; otherwise pack - if waste <= 128 { per } else { 0 } -} - -impl PlaneFile { - /// Create a new plane file with capacity for `max_nodes` (sparse; pages materialize on write). - pub fn create(path: &Path, dims: usize, layer0_cap: usize, max_nodes: u64) -> io::Result { - if max_nodes >= NO_ID as u64 { - return Err(io::Error::new(io::ErrorKind::InvalidInput, "maxNodes must be below 2^32-1")); - } - let slot_size = slot_size_for(dims, layer0_cap); - let slots_per_page = slots_per_page_for(slot_size); - let data_len = slot_region_len(max_nodes, slot_size, slots_per_page); - let upper_capacity = max_nodes / 8 + 64; - let len = HEADER_SIZE as u64 + data_len + upper_capacity * upper_entry_size() as u64; - let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?; - file.set_len(len)?; - let mut map = unsafe { MmapMut::map_mut(&file)? }; - advise_random(&map); - // geometry and allocator state first; MAGIC+VERSION last, so a concurrent opener - // in the create window sees an invalid header (retryable) rather than adopting a - // half-initialized plane with max_nodes = 0 - map[H_DIMS..H_DIMS + 2].copy_from_slice(&(dims as u16).to_le_bytes()); - map[H_QUANT] = 0; - map[H_LAYER0_CAP..H_LAYER0_CAP + 2].copy_from_slice(&(layer0_cap as u16).to_le_bytes()); - map[H_SLOT_SIZE..H_SLOT_SIZE + 4].copy_from_slice(&(slot_size as u32).to_le_bytes()); - map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].copy_from_slice(&(slots_per_page as u16).to_le_bytes()); - map[H_ENTRY..H_ENTRY + 8].copy_from_slice(&(NO_ID as u64).to_le_bytes()); - map[H_FREELIST_HEAD..H_FREELIST_HEAD + 8] - .copy_from_slice(&((NO_ID as u64) | 0u64 << 32).to_le_bytes()); - map[H_MAX_NODES..H_MAX_NODES + 8].copy_from_slice(&max_nodes.to_le_bytes()); - map[H_UPPER_FREELIST..H_UPPER_FREELIST + 8].copy_from_slice(&(NO_UPPER as u64).to_le_bytes()); - // zero would read as "node 0 was the previous entry point" and hand every re-election - // and search-side repair a candidate that was never an entry point - map[H_ENTRY_PREV..H_ENTRY_PREV + 8].copy_from_slice(&(NO_ID as u64).to_le_bytes()); - map[H_VERSION..H_VERSION + 4].copy_from_slice(&VERSION.to_le_bytes()); - std::sync::atomic::fence(Ordering::Release); - map[H_MAGIC..H_MAGIC + 4].copy_from_slice(&MAGIC.to_le_bytes()); - let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; - let mut plane = PlaneFile { - file, - self_tag: 0, - map, - dims, - layer0_cap, - slot_size, - max_nodes, - upper_offset, - upper_capacity, - slots_per_page, - opened_clean: true, - }; - plane.register_opener(); - Ok(plane) - } - - pub fn open(path: &Path) -> io::Result { - let file = OpenOptions::new().read(true).write(true).open(path)?; - let file_len = file.metadata()?.len(); - if file_len < HEADER_SIZE as u64 { - // a truncated or interrupted create must be a catchable error, not a slice panic - return Err(io::Error::new(io::ErrorKind::InvalidData, "plane file shorter than its header: recreate the index")); - } - let map = unsafe { MmapMut::map_mut(&file)? }; - advise_random(&map); - let magic = u32::from_le_bytes(map[H_MAGIC..H_MAGIC + 4].try_into().unwrap()); - let version = u32::from_le_bytes(map[H_VERSION..H_VERSION + 4].try_into().unwrap()); - if magic != MAGIC || version != VERSION { - return Err(io::Error::new(io::ErrorKind::InvalidData, "format mismatch: reindex required")); - } - let dims = u16::from_le_bytes(map[H_DIMS..H_DIMS + 2].try_into().unwrap()) as usize; - let layer0_cap = u16::from_le_bytes(map[H_LAYER0_CAP..H_LAYER0_CAP + 2].try_into().unwrap()) as usize; - let slot_size = u32::from_le_bytes(map[H_SLOT_SIZE..H_SLOT_SIZE + 4].try_into().unwrap()) as usize; - let slots_per_page = u16::from_le_bytes(map[H_SLOTS_PER_PAGE..H_SLOTS_PER_PAGE + 2].try_into().unwrap()) as usize; - let max_nodes = u64::from_le_bytes(map[H_MAX_NODES..H_MAX_NODES + 8].try_into().unwrap()); - if dims == 0 || slot_size == 0 || slot_size != slot_size_for(dims, layer0_cap) { - return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header geometry is inconsistent: recreate the index")); - } - if max_nodes > NO_ID as u64 || slots_per_page != slots_per_page_for(slot_size) { - return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header geometry is inconsistent: recreate the index")); - } - let upper_offset = HEADER_SIZE + slot_region_len(max_nodes, slot_size, slots_per_page) as usize; - let upper_capacity = max_nodes / 8 + 64; - let expected = (upper_offset as u64) - .checked_add(upper_capacity.checked_mul(upper_entry_size() as u64).ok_or_else(|| { - io::Error::new(io::ErrorKind::InvalidData, "plane header geometry overflows: recreate the index") - })?) - .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "plane header geometry overflows: recreate the index"))?; - if file_len < expected { - // header-valid but short (rsync/backup truncation): mid-range slot_ptr/upper_ptr - // would otherwise read off the mapping - return Err(io::Error::new( - io::ErrorKind::InvalidData, - format!("plane file is {file_len} bytes but its header implies {expected}: recreate the index"), - )); - } - let opened_clean = map[H_CLEAN_SHUTDOWN] == 1; - let mut plane = PlaneFile { - file, - self_tag: 0, - map, - dims, - layer0_cap, - slot_size, - max_nodes, - upper_offset, - upper_capacity, - slots_per_page, - opened_clean, - }; - plane.register_opener(); - let hw = plane.id_high_water(); - if hw > max_nodes { - return Err(io::Error::new(io::ErrorKind::InvalidData, "plane header id high-water exceeds capacity: recreate the index")); - } - // No open-time repair: seqlocks persisted odd by a dead writer are taken over lazily - // at the contended slot (seqlock.rs) — a whole-file scrub would page in the entire - // mapping and, with another process still mapping the file, could force a LIVE - // writer's lock. The clean-shutdown byte remains advisory metadata only. - Ok(plane) - } - - /// Force any persisted-odd seqlocks (slot + upper regions) back to even after an unclean - /// shutdown. Safe because open() runs before any concurrent access exists. - #[inline] - pub fn slot_ptr(&self, id: u32) -> *const u8 { - let off = if self.slots_per_page > 0 { - (id as usize / self.slots_per_page) * PAGE + (id as usize % self.slots_per_page) * self.slot_size - } else { - id as usize * self.slot_size - }; - unsafe { self.map.as_ptr().add(HEADER_SIZE + off) } - } - - #[inline] - pub fn slot_ptr_mut(&self, id: u32) -> *mut u8 { - // Mutation through a shared map: all mutable slot access is mediated by the seqlock - // (seqlock.rs) and atomics; the mmap itself is plain memory. - self.slot_ptr(id) as *mut u8 - } - - #[inline] - fn header_atomic_u64(&self, offset: usize) -> &AtomicU64 { - unsafe { &*(self.map.as_ptr().add(offset) as *const AtomicU64) } - } - - #[inline] - pub fn seq_atomic(&self, id: u32) -> &AtomicU32 { - unsafe { &*(self.slot_ptr(id).add(S_SEQ) as *const AtomicU32) } - } - - /// Allocate a node id: pop the freelist, else bump the high-water. Returns NO_ID when - /// the plane is full (max_nodes reached) — an unchecked bump would address into the - /// upper-layer region and, past that, off the mapping. - pub fn allocate_id(&self) -> u32 { - let head = self.header_atomic_u64(H_FREELIST_HEAD); - loop { - let cur = head.load(Ordering::Acquire); - let id = (cur & 0xffff_ffff) as u32; - if id == NO_ID { - let hw = self.header_atomic_u64(H_ID_HIGH_WATER); - let new = hw.fetch_add(1, Ordering::AcqRel); - if new >= self.max_nodes { - hw.fetch_sub(1, Ordering::AcqRel); - return NO_ID; - } - return new as u32; - } - if (id as u64) >= self.max_nodes { - // corrupt freelist head (file-sourced): drop the chain rather than compute - // out-of-mapping pointers; capacity continues via the high-water - let _ = head.compare_exchange(cur, NO_ID as u64, Ordering::AcqRel, Ordering::Acquire); - continue; - } - // next-pointer lives in the dead slot's scale field rather than its first neighbor - // word: the neighbor array is a live reader's aligned volatile load target, and a - // freelist pointer parked there would be decoded as a neighbor id - let next = unsafe { (*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32)).load(Ordering::Acquire) }; - let tag = (cur >> 32).wrapping_add(1); - let new = (next as u64) | (tag << 32); - if head.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire).is_ok() { - return id; - } - } - } - - /// Return a deleted node's id to the freelist. Caller must have already marked the slot - /// deleted (under its seqlock) so concurrent traversals skip it. - pub fn free_id(&self, id: u32) { - let head = self.header_atomic_u64(H_FREELIST_HEAD); - let next_word = unsafe { &*(self.slot_ptr(id).add(S_SCALE) as *const AtomicU32) }; - loop { - let cur = head.load(Ordering::Acquire); - next_word.store((cur & 0xffff_ffff) as u32, Ordering::Release); - let tag = (cur >> 32).wrapping_add(1); - let new = (id as u64) | (tag << 32); - if head.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire).is_ok() { - return; - } - } - } - - /// Raise the high-water to at least `id + 1` (dual-write mode: ids are allocated by the - /// host's existing allocator and mirrored in; the plane allocator is bypassed). - pub fn ensure_high_water(&self, id: u32) { - let hw = self.header_atomic_u64(H_ID_HIGH_WATER); - let want = id as u64 + 1; - let mut cur = hw.load(Ordering::Acquire); - while cur < want { - match hw.compare_exchange_weak(cur, want, Ordering::AcqRel, Ordering::Acquire) { - Ok(_) => break, - Err(now) => cur = now, - } - } - } - - pub fn id_high_water(&self) -> u64 { - self.header_atomic_u64(H_ID_HIGH_WATER).load(Ordering::Acquire) - } - - pub fn upper_high_water(&self) -> u64 { - self.header_atomic_u64(H_UPPER_HIGH_WATER).load(Ordering::Acquire) - } - - /// Entry point (id, level), read as one atomic word — a torn (new id, old level) pair - /// would blind a racing search. - pub fn entry_point(&self) -> (u32, u32) { - let packed = self.header_atomic_u64(H_ENTRY).load(Ordering::Acquire); - ((packed & 0xffff_ffff) as u32, (packed >> 32) as u32) - } - - pub fn set_entry_point(&self, id: u32, level: u32) { - let prev = self.header_atomic_u64(H_ENTRY).swap((id as u64) | ((level as u64) << 32), Ordering::AcqRel); - self.record_previous_entry(prev, id); - } - - /// Remember the entry point a PROMOTION displaced. Only promotions are recorded: the node - /// they displace was live and high-level, which is what makes it a usable hint. Recording - /// a re-election's replacement instead would fill the hint with the dead node that forced - /// the re-election. - #[inline] - fn record_previous_entry(&self, prev_packed: u64, new_id: u32) { - let prev_id = (prev_packed & 0xffff_ffff) as u32; - if prev_id == NO_ID || prev_id == new_id || (prev_id as u64) >= self.max_nodes { - return; - } - // a hint is only worth keeping while its node is live: the host mirrors a post-delete - // re-election through this same call, and storing the node that died would evict a - // usable hint with one the repair path can never follow - // volatile like every other read of a field a concurrent writer mutates (graph.rs's - // `vread`): this one is outside the slot seqlock, so the retry cannot even catch a tear - if unsafe { self.slot_ptr(prev_id).add(S_FLAGS).read_volatile() } != FLAG_VALID { - return; - } - self.header_atomic_u64(H_ENTRY_PREV).store(prev_packed, Ordering::Release); - } - - /// Claim the entry point of an EMPTY graph: a strict compare-exchange from the empty - /// encoding, so exactly one racer wins. `set_entry_point_if_not_better` cannot serve here — - /// it is a not-worse install, so a second first-inserter would replace the winner with its - /// own edgeless node and orphan everything already rooted at the winner. A loser must join - /// the winner's graph instead of returning an unlinked node. - pub fn claim_entry_if_empty(&self, id: u32, level: u32) -> bool { - self.header_atomic_u64(H_ENTRY) - .compare_exchange(NO_ID as u64, (id as u64) | ((level as u64) << 32), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - } - - /// Entry-point CAS for re-election: install (id, level) only while the current entry is - /// still `expected_id` or is of a lower level — a concurrent insert that just promoted a - /// higher-level entry must not be clobbered by a delete's level-0 survivor. - pub fn set_entry_point_if_not_better(&self, id: u32, level: u32, expected_id: u32) { - self.cas_entry_if_not_better(id, level, expected_id, false); - } - - /// The same CAS for an insert that PROMOTED itself above the entry it observed: the - /// displaced entry is live, so it is recorded as the previous-entry hint that re-election - /// and the search-side repair both consult before any O(high-water) scan. - pub fn promote_entry_point(&self, id: u32, level: u32, expected_id: u32) { - self.cas_entry_if_not_better(id, level, expected_id, true); - } - - fn cas_entry_if_not_better(&self, id: u32, level: u32, expected_id: u32, record_prev: bool) { - let cell = self.header_atomic_u64(H_ENTRY); - let new = (id as u64) | ((level as u64) << 32); - let mut cur = cell.load(Ordering::Acquire); - loop { - let cur_id = (cur & 0xffff_ffff) as u32; - let cur_level = (cur >> 32) as u32; - if cur_id != expected_id && cur_id != NO_ID && cur_level > level { - return; // someone installed a better entry meanwhile - } - match cell.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire) { - Ok(_) => { - if record_prev { - self.record_previous_entry(cur, id); - } - return; - } - Err(now) => cur = now, - } - } - } - - /// Install `(id, level)` ONLY while the entry still names `expected_id`. The read-side - /// repair publishes through this rather than `set_entry_point_if_not_better`: the entry it - /// is replacing is dead, so "not worse" is the wrong test — a level-0 root installed while - /// the repair ran would lose to a higher-level candidate and be orphaned. - /// - /// It compares the id, not the incarnation, so under the crate's own freelist reuse it can - /// match a different node that took the same slot. That is a routing-quality window, not a - /// lost node: the value it could displace is a live edged node, never the edgeless claimer - /// (`claim_entry_if_empty` fires only from NO_ID, which no reuse can produce). Harper's host - /// ids are monotonic and never reused, so this cannot arise there at all. - pub fn replace_entry_if(&self, expected_id: u32, id: u32, level: u32) -> bool { - let cell = self.header_atomic_u64(H_ENTRY); - let new = (id as u64) | ((level as u64) << 32); - let mut cur = cell.load(Ordering::Acquire); - while (cur & 0xffff_ffff) as u32 == expected_id { - match cell.compare_exchange(cur, new, Ordering::AcqRel, Ordering::Acquire) { - Ok(_) => return true, - Err(now) => cur = now, - } - } - false - } - - /// Clear the entry point, but only while it still names `expected_id`. A re-election that - /// found no candidate must not erase an entry a concurrent insert installed meanwhile — - /// `set_entry_point_if_not_better(NO_ID, 0, ..)` would, because a level-0 live entry is not - /// "better" than the level-0 clear. - pub fn clear_entry_point_if(&self, expected_id: u32) { - let cell = self.header_atomic_u64(H_ENTRY); - let mut cur = cell.load(Ordering::Acquire); - while (cur & 0xffff_ffff) as u32 == expected_id { - match cell.compare_exchange(cur, NO_ID as u64, Ordering::AcqRel, Ordering::Acquire) { - Ok(_) => return, - Err(now) => cur = now, - } - } - } - - pub fn set_watermark(&self, txn: u64) { - self.header_atomic_u64(H_TXN_WATERMARK).store(txn, Ordering::Release); - } - - pub fn watermark(&self) -> u64 { - self.header_atomic_u64(H_TXN_WATERMARK).load(Ordering::Acquire) - } - - #[inline] - pub fn upper_ptr(&self, idx: u32) -> *const u8 { - debug_assert!((idx as u64) < self.upper_capacity); - unsafe { self.map.as_ptr().add(self.upper_offset + idx as usize * upper_entry_size()) } - } - - #[inline] - pub fn upper_ptr_mut(&self, idx: u32) -> *mut u8 { - self.upper_ptr(idx) as *mut u8 - } - - #[inline] - pub fn upper_seq_atomic(&self, idx: u32) -> &AtomicU32 { - unsafe { &*(self.upper_ptr(idx).add(U_SEQ) as *const AtomicU32) } - } - - /// Allocate an upper-region entry: pop the upper freelist, else bump the high-water. - /// Returns NO_UPPER when exhausted — the node then simply has no upper links, which - /// degrades routing, not correctness. A dead entry's next-pointer lives in its first - /// list bytes (offset U_LISTS), clobbered on reuse by the full rewrite. - pub fn allocate_upper(&self) -> u32 { - let head = self.header_atomic_u64(H_UPPER_FREELIST); - loop { - let cur = head.load(Ordering::Acquire); - let idx = (cur & 0xffff_ffff) as u32; - if idx != NO_UPPER && (idx as u64) >= self.upper_capacity { - // corrupt upper freelist head (file-sourced): drop the chain - let _ = head.compare_exchange(cur, NO_UPPER as u64, Ordering::AcqRel, Ordering::Acquire); - continue; - } - if idx == NO_UPPER { - let hw = self.header_atomic_u64(H_UPPER_HIGH_WATER); - let new = hw.fetch_add(1, Ordering::AcqRel); - if new >= self.upper_capacity { - hw.fetch_sub(1, Ordering::AcqRel); - return NO_UPPER; - } - return new as u32; - } - let next = unsafe { (*(self.upper_ptr(idx).add(U_LISTS) as *const AtomicU32)).load(Ordering::Acquire) }; - let tag = (cur >> 32).wrapping_add(1); - if head - .compare_exchange(cur, (next as u64) | (tag << 32), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - return idx; - } - } - } - - /// Return a dead upper entry to the freelist. Caller must have unlinked it from its - /// node's slot (or marked the node deleted) first. - pub fn free_upper(&self, idx: u32) { - if idx == NO_UPPER || (idx as u64) >= self.upper_capacity { - return; - } - let head = self.header_atomic_u64(H_UPPER_FREELIST); - let next_word = unsafe { &*(self.upper_ptr(idx).add(U_LISTS) as *const AtomicU32) }; - loop { - let cur = head.load(Ordering::Acquire); - next_word.store((cur & 0xffff_ffff) as u32, Ordering::Release); - let tag = (cur >> 32).wrapping_add(1); - if head - .compare_exchange(cur, (idx as u64) | (tag << 32), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - return; - } - } - } - - #[inline] - fn registry_tag_cell(&self, slot: usize) -> &AtomicU32 { - unsafe { &*(self.map.as_ptr().add(H_REGISTRY + slot * 4) as *const AtomicU32) } - } - - /// Claim a registry slot for this handle: take the slot's kernel byte-range lock (held - /// until this handle closes; released by the kernel if the process dies) and publish a - /// random tag whose low bits name the slot. On platforms without OFD locks, or with the - /// registry full, the handle stays unregistered (tag 0): it still works, but its own - /// abandoned locks are unreclaimable and it never reclaims others'. - fn register_opener(&mut self) { - #[cfg(target_os = "linux")] - for slot in 0..REGISTRY_SLOTS { - if !self.try_lock_registry_slot(slot, false) { - continue; - } - let nanos = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.subsec_nanos()) - .unwrap_or(0); - // identity = slot (low 6 bits) + a random per-open epoch (12 bits, nonzero): - // a dead lock value pointing at a since-re-occupied slot is recognized as dead - // by the epoch mismatch — the container same-pid restart lands exactly here - let mut epoch = (nanos ^ std::process::id().rotate_left(16) ^ (self as *const _ as u32)) & 0xfff; - if epoch == 0 { - epoch = 1; - } - let tag = (epoch << 6) | slot as u32; - self.registry_tag_cell(slot).store(tag, Ordering::Release); - self.self_tag = tag; - return; - } - } - - /// Try to take the OFD write lock on a registry slot's byte range. `probe` releases it - /// immediately (liveness check); otherwise it is held for this handle's lifetime. - #[cfg(target_os = "linux")] - fn try_lock_registry_slot(&self, slot: usize, probe: bool) -> bool { - use std::os::unix::io::AsRawFd; - let mut fl: libc::flock = unsafe { std::mem::zeroed() }; - fl.l_type = libc::F_WRLCK as libc::c_short; - fl.l_whence = libc::SEEK_SET as libc::c_short; - fl.l_start = (H_REGISTRY + slot * 4) as libc::off_t; - fl.l_len = 4; - let got = unsafe { libc::fcntl(self.file.as_raw_fd(), libc::F_OFD_SETLK, &fl) } == 0; - if got && probe { - fl.l_type = libc::F_UNLCK as libc::c_short; - unsafe { libc::fcntl(self.file.as_raw_fd(), libc::F_OFD_SETLK, &fl) }; - } - got - } - - /// Whether the handle behind a lock value is gone. Lock values carry a per-acquisition - /// salt in their upper bits, so ownership is keyed on the registry SLOT (low bits): the - /// owner is dead only with positive evidence — no registration in the slot, or the - /// slot's kernel lock acquirable (its holder's open handle closed; process death - /// included). Our own slot is always alive (probing our own OFD lock would succeed and - /// lie). A dead value pointing at a slot since re-occupied by a NEW live handle reads - /// alive; the bounded writer wedge covers that rare mis-attribution. - pub fn tag_is_dead(&self, lock_value: u32) -> bool { - let identity = lock_value & crate::seqlock::TAG_MASK; - if identity == 0 { - return false; // unregistered owner: unknowable - } - if self.self_tag != 0 && identity == self.self_tag { - // ourselves: probing our own OFD lock from the same description would succeed - // and lie, so self is answered structurally - return false; - } - let slot = (identity as usize) & (REGISTRY_SLOTS - 1); - let registered = self.registry_tag_cell(slot).load(Ordering::Acquire); - if registered == 0 || registered != identity { - return true; // slot empty, or re-occupied by a different epoch: owner departed - } - #[cfg(target_os = "linux")] - { - self.try_lock_registry_slot(slot, true) - } - #[cfg(not(target_os = "linux"))] - { - false - } - } - - /// The re-election hint: the entry point most recently replaced by a promotion. - pub fn previous_entry_point(&self) -> u32 { - (self.header_atomic_u64(H_ENTRY_PREV).load(Ordering::Acquire) & 0xffff_ffff) as u32 - } - - pub fn set_clean_shutdown(&mut self, clean: bool) { - self.map[H_CLEAN_SHUTDOWN] = clean as u8; - } - - pub fn msync(&self) -> io::Result<()> { - self.map.flush() - } - - /// Durability barrier with watermark ordering: flush all data, then advance the - /// watermark and mark the shutdown clean, then flush the header page alone. A crash - /// between the two flushes leaves the OLD watermark over fully-durable data — replay - /// re-covers a suffix, which is idempotent — never a new watermark over missing data. - /// (A single whole-map msync cannot express "data before watermark": the kernel may - /// write the header page back first.) - pub fn flush_with_watermark(&self, txn: Option) -> io::Result<()> { - self.map.flush()?; - if let Some(txn) = txn { - // None must not TOUCH the watermark: a cadence barrier reading-then-rewriting it - // on a pool thread could write a stale value over a completion stamp - self.set_watermark(txn); - } - unsafe { *(self.map.as_ptr().add(H_CLEAN_SHUTDOWN) as *mut u8) = 1 }; - self.map.flush_range(0, HEADER_SIZE) - } - - /// Mark the plane an incomplete mirror, durably, and nothing else: zero the watermark and - /// msync the header page alone. Every opener then refuses to search it and rebuilds. - /// - /// Deliberately NOT `flush_with_watermark(Some(0))`: that writes the whole mapping back - /// first, and the caller invalidating a multi-GB plane cannot pay a full msync inline — - /// which is why the host used to queue an async flush and create its `.stale` sidecar - /// before the flush had happened at all. Skipping the data flush is sound because the - /// data is being discarded, and because lowering the watermark is the safe direction: - /// the ordering hazard `flush_with_watermark` exists to prevent is a NEW watermark over - /// missing data, never an old one over durable data. - pub fn invalidate(&self) -> io::Result<()> { - self.set_watermark(0); - self.map.flush_range(0, HEADER_SIZE) - } -} diff --git a/native/hnsw-plane/src/graph.rs b/native/hnsw-plane/src/graph.rs deleted file mode 100644 index d86b3fdc6a..0000000000 --- a/native/hnsw-plane/src/graph.rs +++ /dev/null @@ -1,743 +0,0 @@ -//! Slot-level node access over the plane file, mediated by per-slot seqlocks. Hot-path -//! reads (distance, neighbor ids) are zero-copy against the mmap; full-copy read_node -//! exists for construction paths. Upper-layer adjacency lives in a fixed-entry region of -//! the same file (per-entry seqlocks), so the hierarchy persists with the graph and -//! concurrent searches share nothing mutable. - -use crate::distance::{cosine_i8_i8_raw, cosine_int8_raw, Query}; -use crate::format::{ - neighbor_offset, PlaneFile, FLAG_DELETED, FLAG_VALID, MAX_UPPER_LEVELS, NO_UPPER, S_DEGREE, S_FLAGS, S_INV_MAG, - S_LEVEL, S_SCALE, S_UPPER_IDX, S_VECTOR, UPPER_CAP, UPPER_LEVEL_STRIDE, UL_DEGREE, UL_IDS, U_LEVELS, U_LISTS, -}; -use crate::seqlock; -use crate::seqlock::Wedged; - -/// Aligned volatile load of a slot/upper-entry field another process may be mutating. -/// -/// This forbids the optimizer from duplicating, splitting, or sinking the load across the -/// seqlock's validating fence, which would let a reader act on bytes the generation check -/// never covered. It does NOT make the access race-free under Rust's memory model — only -/// atomics would, and that is the format change hnsw-native-plane.md §10 records as -/// follow-up. The vector is deliberately not read this way: `cosine_int8_raw` must stay -/// autovectorized, and a torn vector only perturbs a distance the generation check discards. -/// Every field read here is naturally aligned (slots are 64-aligned; the neighbor and upper -/// id arrays are 4-padded by format.rs), so these compile to single loads. -#[inline(always)] -unsafe fn vread(p: *const T) -> T { - p.read_volatile() -} - -pub struct Graph { - pub file: PlaneFile, - /// Rotates `probe_for_entry`'s starting offset so this plane's consecutive repairs sample - /// different ids. Per handle, not per process: a shared counter is advanced by every other - /// plane's repairs too, so one plane's calls can land on a single residue indefinitely — - /// which is the coverage the rotation exists to provide. - probe_rotation: std::sync::atomic::AtomicU32, -} - -/// A consistent full copy of one node (construction paths only; search uses zero-copy). -pub struct NodeRead { - pub level: u8, - pub scale: f32, - pub inv_mag: f32, - pub vector: Vec, - pub neighbors: Vec, -} - -impl Graph { - pub fn new(file: PlaneFile) -> Self { - Graph { file, probe_rotation: std::sync::atomic::AtomicU32::new(0) } - } - - #[inline] - fn in_range(&self, id: u32) -> bool { - (id as u64) < self.file.id_high_water() - } - - /// Sanitizer for a slot lock taken over from a dead writer: the payload is half-written, - /// so the slot must read as deleted until something rewrites it (heal-on-touch contract; - /// FLAG_DELETED rather than 0 so hosts can still free/reuse the id). - fn slot_sanitizer(&self, id: u32) -> impl Fn() + '_ { - move || unsafe { - let p = self.file.slot_ptr_mut(id); - // a dead writer's slot may hold a garbage (or zero-initialized) upper index; a - // later raw rewrite would reuse it and clobber another node's hierarchy - (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); - *p.add(S_FLAGS) = FLAG_DELETED; - } - } - - fn owner_dead(&self) -> impl Fn(u32) -> bool + '_ { - move |tag| self.file.tag_is_dead(tag) - } - - /// Sanitizer for an upper-entry lock taken over from a dead writer. - fn upper_sanitizer(&self, idx: u32) -> impl Fn() + '_ { - move || unsafe { *self.file.upper_ptr_mut(idx).add(U_LEVELS) = 0 } - } - - /// Zero-copy distance from `query` to the stored vector of `id`. None for absent/deleted. - #[inline] - pub fn distance_to(&self, id: u32, query: &Query) -> Option { - if !self.in_range(id) { - return None; - } - let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || { - let p = self.file.slot_ptr(id); - unsafe { - let flags = vread(p.add(S_FLAGS)); - if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return None; - } - let scale = vread(p.add(S_SCALE) as *const f32); - let inv_mag = vread(p.add(S_INV_MAG) as *const f32); - Some(cosine_int8_raw(query, p.add(S_VECTOR) as *const i8, scale, inv_mag)) - } - }, self.slot_sanitizer(id), || None, self.owner_dead()) - } - - /// Symmetric stored-to-stored distance (construction-time neighbor↔neighbor checks). - /// Plain unlocked reads: a torn read only perturbs a construction heuristic. - pub fn distance_between(&self, a: u32, b: u32) -> Option { - if !self.in_range(a) || !self.in_range(b) { - return None; - } - let dims = self.file.dims; - let pa = self.file.slot_ptr(a); - let pb = self.file.slot_ptr(b); - unsafe { - let fa = *pa.add(S_FLAGS); - let fb = *pb.add(S_FLAGS); - if fa & FLAG_VALID == 0 || fa & FLAG_DELETED != 0 || fb & FLAG_VALID == 0 || fb & FLAG_DELETED != 0 { - return None; - } - let scale_a = (pa.add(S_SCALE) as *const f32).read_unaligned(); - let inv_a = (pa.add(S_INV_MAG) as *const f32).read_unaligned(); - let scale_b = (pb.add(S_SCALE) as *const f32).read_unaligned(); - let inv_b = (pb.add(S_INV_MAG) as *const f32).read_unaligned(); - Some(cosine_i8_i8_raw( - pa.add(S_VECTOR) as *const i8, - scale_a, - inv_a, - pb.add(S_VECTOR) as *const i8, - scale_b, - inv_b, - dims, - )) - } - } - - /// Copy layer-0 neighbor ids into `out` (cleared first). Returns the node's level, - /// or None for absent/deleted. - #[inline] - pub fn neighbors_into(&self, id: u32, out: &mut Vec) -> Option { - out.clear(); - if !self.in_range(id) { - return None; - } - let seq = self.file.seq_atomic(id); - let cap = self.file.layer0_cap; - let nbase = neighbor_offset(self.file.dims); - seqlock::read_consistent(seq, self.file.self_tag, || { - out.clear(); - let p = self.file.slot_ptr(id); - unsafe { - let flags = vread(p.add(S_FLAGS)); - if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return None; - } - let level = vread(p.add(S_LEVEL)); - let degree = u16::from_le(vread(p.add(S_DEGREE) as *const u16)) as usize; - let base = p.add(nbase) as *const u32; - for i in 0..degree.min(cap) { - out.push(u32::from_le(vread(base.add(i)))); - } - Some(level) - } - }, self.slot_sanitizer(id), || None, self.owner_dead()) - } - - /// The node's upper-region entry index, or NO_UPPER. - #[inline] - fn upper_idx_of(&self, id: u32) -> u32 { - if !self.in_range(id) { - return NO_UPPER; - } - let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || { - let p = self.file.slot_ptr(id); - unsafe { - let flags = vread(p.add(S_FLAGS)); - if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return NO_UPPER; - } - vread(p.add(S_UPPER_IDX) as *const u32) - } - }, self.slot_sanitizer(id), || NO_UPPER, self.owner_dead()) - } - - /// Copy `id`'s neighbor ids at upper `level` (1-based) into `out`. False when the node - /// has no upper entry or no such level. - pub fn upper_neighbors_into(&self, id: u32, level: u8, out: &mut Vec) -> bool { - out.clear(); - debug_assert!(level >= 1); - let idx = self.upper_idx_of(id); - if idx == NO_UPPER || (idx as u64) >= self.file.upper_capacity || level as usize > MAX_UPPER_LEVELS { - return false; - } - let seq = self.file.upper_seq_atomic(idx); - seqlock::read_consistent(seq, self.file.self_tag, || { - out.clear(); - let p = self.file.upper_ptr(idx); - unsafe { - let levels = vread(p.add(U_LEVELS)); - if level > levels { - return false; - } - let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); - let degree = u16::from_le(vread(lp.add(UL_DEGREE) as *const u16)) as usize; - let base = lp.add(UL_IDS) as *const u32; - for i in 0..degree.min(UPPER_CAP) { - out.push(u32::from_le(vread(base.add(i)))); - } - true - } - }, self.upper_sanitizer(idx), || false, self.owner_dead()) - } - - /// Write a node's full upper adjacency into a fresh region entry; returns the entry - /// index to store in the slot (NO_UPPER when the region is exhausted or levels is empty). - pub fn write_upper(&self, levels: &[Vec]) -> Result { - if levels.is_empty() { - return Ok(NO_UPPER); - } - let idx = self.file.allocate_upper(); - if idx == NO_UPPER { - return Ok(NO_UPPER); - } - let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; - let p = self.file.upper_ptr_mut(idx); - unsafe { - let n = levels.len().min(MAX_UPPER_LEVELS); - *p.add(U_LEVELS) = n as u8; - for (l, list) in levels.iter().take(n).enumerate() { - let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); - let deg = list.len().min(UPPER_CAP); - (lp.add(UL_DEGREE) as *mut u16).write_unaligned((deg as u16).to_le()); - let base = lp.add(UL_IDS) as *mut u32; - for (i, id) in list.iter().take(deg).enumerate() { - base.add(i).write_unaligned(id.to_le()); - } - } - } - Ok(idx) - } - - /// Rewrite an existing upper entry in place (full state). Used by the raw mirroring - /// path so repeated updates to a high-level node reuse its entry instead of leaking one - /// per rewrite. - pub fn rewrite_upper(&self, idx: u32, levels: &[Vec]) -> Result<(), Wedged> { - let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; - let p = self.file.upper_ptr_mut(idx); - unsafe { - let n = levels.len().min(MAX_UPPER_LEVELS); - *p.add(U_LEVELS) = n as u8; - for (l, list) in levels.iter().take(n).enumerate() { - let lp = p.add(U_LISTS + l * UPPER_LEVEL_STRIDE); - let deg = list.len().min(UPPER_CAP); - (lp.add(UL_DEGREE) as *mut u16).write_unaligned((deg as u16).to_le()); - let base = lp.add(UL_IDS) as *mut u32; - for (i, id) in list.iter().take(deg).enumerate() { - base.add(i).write_unaligned(id.to_le()); - } - } - } - Ok(()) - } - - /// Whether a slot has ever been written (valid or deleted) — the builder scan's - /// skip-if-touched check. - pub fn node_touched(&self, id: u32) -> bool { - if !self.in_range(id) { - return false; - } - let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || unsafe { vread(self.file.slot_ptr(id).add(S_FLAGS)) != 0 }, self.slot_sanitizer(id), || true, self.owner_dead()) - } - - /// The slot's stored upper idx regardless of valid/deleted flags. Taken under the slot - /// write lock rather than `read_consistent`, whose NO_UPPER fallback cannot be told apart - /// from an unbound slot — reusing it as one mints a second entry for an id that already - /// owns one. - fn upper_idx_locked(&self, id: u32) -> Result { - if !self.in_range(id) { - return Ok(NO_UPPER); - } - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr(id); - Ok(unsafe { - if *p.add(S_FLAGS) == 0 { - NO_UPPER // never written - } else { - (p.add(S_UPPER_IDX) as *const u32).read_unaligned() - } - }) - } - - /// Mirror a host-maintained node into the plane: full state per call, host-allocated id - /// (high-water is raised, the plane allocator is bypassed), upper entry reused in place - /// when present. This is the dual-write phase-1 write path. - pub fn write_node_raw( - &self, - id: u32, - level: u8, - vector: &[i8], - scale: f32, - inv_mag: f32, - neighbors: &[u32], - upper_levels: &[Vec], - ) -> Result<(), Wedged> { - self.file.ensure_high_water(id); - let existing = match self.upper_idx_locked(id)? { - idx if idx != NO_UPPER && (idx as u64) >= self.file.upper_capacity => NO_UPPER, // corrupt stored index - idx => idx, - }; - let mut fresh = NO_UPPER; - let upper_idx = if upper_levels.is_empty() { - // the host reseeds its id counter to largestNodeId + 1 on restart, so an id can be - // re-minted at level 0 over a slot that had a hierarchy; that entry must stop being - // readable. Emptied in place rather than freed: the freelist hand-off is not atomic - // with publishing the slot below, so a mirror that read this index first could - // republish a slot pointing at an entry already given to another node. One idle - // entry per id is the bounded retention hnsw-native-plane.md §10 accepts. - if existing != NO_UPPER { - self.rewrite_upper(existing, &[])?; - } - existing - } else if existing != NO_UPPER { - self.rewrite_upper(existing, upper_levels)?; - existing - } else { - fresh = self.write_upper(upper_levels)?; - fresh - }; - let mut l0 = neighbors.to_vec(); - l0.truncate(self.file.layer0_cap); - if let Err(wedged) = self.write_node(id, level, vector, scale, inv_mag, &l0, upper_idx) { - self.file.free_upper(fresh); // unreachable from any slot until publication succeeds - return Err(wedged); - } - Ok(()) - } - - /// Mark deleted WITHOUT returning the id to the plane freelist — dual-write mode, where - /// the host owns id allocation and may re-mint or reuse ids on its own schedule. - pub fn clear_node(&self, id: u32) -> Result<(), Wedged> { - if (id as u64) >= self.file.max_nodes { - return Ok(()); - } - // extend the high-water rather than skipping: a delete mirrored while a backfill - // scan runs must leave a touched (deleted) slot behind, or the scan's older - // snapshot would resurrect the node when its cursor reaches this id - self.file.ensure_high_water(id); - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - unsafe { - if *p.add(S_FLAGS) == 0 { - // tombstoning a never-written slot: its zero-initialized upper_idx would - // otherwise read as the VALID index 0, and a later raw rewrite of this id - // would clobber upper entry 0 — another node's hierarchy - (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); - } - *p.add(S_FLAGS) = FLAG_DELETED; - } - Ok(()) - } - - /// Atomic read-modify-write of `id`'s upper adjacency at `level` (1-based). Returns - /// false when the node has no entry or level. `f` may read other slots. - pub fn update_upper_level)>(&self, id: u32, level: u8, f: F) -> Result { - let idx = self.upper_idx_of(id); - if idx == NO_UPPER || (idx as u64) >= self.file.upper_capacity || level as usize > MAX_UPPER_LEVELS { - return Ok(false); - } - let seq = self.file.upper_seq_atomic(idx); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.upper_sanitizer(idx), self.owner_dead())?; - let p = self.file.upper_ptr_mut(idx); - unsafe { - let levels = *p.add(U_LEVELS); - if level > levels { - return Ok(false); - } - let lp = p.add(U_LISTS + (level as usize - 1) * UPPER_LEVEL_STRIDE); - let degree = u16::from_le((lp.add(UL_DEGREE) as *const u16).read_unaligned()) as usize; - let base = lp.add(UL_IDS) as *mut u32; - let mut list: Vec = (0..degree.min(UPPER_CAP)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); - f(&mut list); - list.truncate(UPPER_CAP); - (lp.add(UL_DEGREE) as *mut u16).write_unaligned((list.len() as u16).to_le()); - for (i, id) in list.iter().enumerate() { - base.add(i).write_unaligned(id.to_le()); - } - } - Ok(true) - } - - /// Seqlock-consistent full copy (construction paths). - pub fn read_node(&self, id: u32) -> Option { - if !self.in_range(id) { - return None; - } - let seq = self.file.seq_atomic(id); - let dims = self.file.dims; - let cap = self.file.layer0_cap; - let nbase_off = neighbor_offset(dims); - seqlock::read_consistent(seq, self.file.self_tag, || { - let p = self.file.slot_ptr(id); - unsafe { - let flags = vread(p.add(S_FLAGS)); - if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return None; - } - let level = vread(p.add(S_LEVEL)); - let degree = u16::from_le(vread(p.add(S_DEGREE) as *const u16)) as usize; - let scale = vread(p.add(S_SCALE) as *const f32); - let inv_mag = vread(p.add(S_INV_MAG) as *const f32); - let vector = std::slice::from_raw_parts(p.add(S_VECTOR) as *const i8, dims).to_vec(); - let nbase = p.add(nbase_off) as *const u32; - let neighbors = (0..degree.min(cap)).map(|i| u32::from_le(vread(nbase.add(i)))).collect(); - Some(NodeRead { level, scale, inv_mag, vector, neighbors }) - } - }, self.slot_sanitizer(id), || None, self.owner_dead()) - } - - /// Write a full slot under its seqlock. `neighbors` is pruned to layer0_cap by the - /// caller; `upper_idx` is a write_upper() result (NO_UPPER for level-0 nodes). - pub fn write_node(&self, id: u32, level: u8, vector: &[i8], scale: f32, inv_mag: f32, neighbors: &[u32], upper_idx: u32) -> Result<(), Wedged> { - debug_assert!(neighbors.len() <= self.file.layer0_cap); - debug_assert_eq!(vector.len(), self.file.dims); - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - let dims = self.file.dims; - unsafe { - *p.add(S_LEVEL) = level; - (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); - (p.add(S_SCALE) as *mut f32).write_unaligned(scale); - (p.add(S_INV_MAG) as *mut f32).write_unaligned(inv_mag); - (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); - std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); - for (i, n) in neighbors.iter().enumerate() { - (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); - } - // valid last within the locked section; the seqlock release publishes it - *p.add(S_FLAGS) = FLAG_VALID; - } - Ok(()) - } - - /// Atomic read-modify-write of a node's layer-0 neighbor list under its seqlock. - /// `f` may read OTHER slots (e.g. distance_between for pruning) — those are plain - /// unlocked reads, so no lock ordering issue — but must not lock this graph's slots. - /// Returns false for absent/deleted nodes. - pub fn update_neighbors)>(&self, id: u32, f: F) -> Result { - if !self.in_range(id) { - return Ok(false); - } - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - let dims = self.file.dims; - let cap = self.file.layer0_cap; - unsafe { - let flags = *p.add(S_FLAGS); - if flags & FLAG_VALID == 0 || flags & FLAG_DELETED != 0 { - return Ok(false); - } - let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; - let base = p.add(neighbor_offset(dims)) as *mut u32; - let mut list: Vec = (0..degree.min(cap)).map(|i| u32::from_le(base.add(i).read_unaligned())).collect(); - f(&mut list); - list.truncate(cap); - (p.add(S_DEGREE) as *mut u16).write_unaligned((list.len() as u16).to_le()); - for (i, n) in list.iter().enumerate() { - base.add(i).write_unaligned(n.to_le()); - } - } - Ok(true) - } - - /// Apply a precomputed neighbor list only if the current list still equals `expected` — - /// the compare and the write share one lock acquisition, so heavy work (distance-based - /// pruning, which can major-fault) happens OUTSIDE the lock and the critical section - /// stays microseconds. Returns false when the list changed or the node is gone. - pub fn set_neighbors_if(&self, id: u32, expected: &[u32], next: &[u32]) -> Result { - debug_assert!(next.len() <= self.file.layer0_cap); - if !self.in_range(id) { - return Ok(false); - } - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - let dims = self.file.dims; - unsafe { - if *p.add(S_FLAGS) != FLAG_VALID { - return Ok(false); - } - let degree = u16::from_le((p.add(S_DEGREE) as *const u16).read_unaligned()) as usize; - if degree != expected.len() { - return Ok(false); - } - let base = p.add(neighbor_offset(dims)) as *mut u32; - for (i, want) in expected.iter().enumerate() { - if u32::from_le(base.add(i).read_unaligned()) != *want { - return Ok(false); - } - } - (p.add(S_DEGREE) as *mut u16).write_unaligned((next.len() as u16).to_le()); - for (i, n) in next.iter().enumerate() { - base.add(i).write_unaligned(n.to_le()); - } - } - Ok(true) - } - - /// Replace only the neighbor list (single-writer construction path). - pub fn write_neighbors(&self, id: u32, neighbors: &[u32]) -> Result<(), Wedged> { - debug_assert!(neighbors.len() <= self.file.layer0_cap); - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - let dims = self.file.dims; - unsafe { - (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); - for (i, n) in neighbors.iter().enumerate() { - (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); - } - } - Ok(()) - } - - /// Mark deleted (traversals skip it), free its upper entry, and return the id to the - /// plane freelist. Deleting the current entry point re-elects a replacement — without - /// that, every search returns empty and every insert orphans itself against the dead - /// entry. - pub fn delete_node(&self, id: u32) -> Result<(), Wedged> { - if !self.in_range(id) { - return Ok(()); // never-allocated or out-of-range ids have nothing to delete - } - // capture neighbors before invalidating: they are the best re-election candidates - let (entry_id, _) = self.file.entry_point(); - let mut candidates: Vec = Vec::new(); - if entry_id == id { - self.neighbors_into(id, &mut candidates); - } - // Re-elect before the tombstone, not after: between marking the slot deleted and - // installing a replacement, every concurrent search routes through a node that reads - // as absent and returns nothing. The node is still live here, so a crash inside the - // window leaves the header naming a live entry either way. - if entry_id == id { - self.reelect_entry_point_replacing(&candidates, id); - } - let upper_idx; - { - let seq = self.file.seq_atomic(id); - let _guard = seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead())?; - let p = self.file.slot_ptr_mut(id); - unsafe { - if *p.add(S_FLAGS) != FLAG_VALID { - // deleting a never-written or already-deleted id must not free again: - // a double-push makes the freelist a self-cycle that hands the same id - // to every subsequent allocation - return Ok(()); - } - upper_idx = (p.add(S_UPPER_IDX) as *const u32).read_unaligned(); - (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(NO_UPPER); - *p.add(S_FLAGS) = FLAG_DELETED; - } - } - if upper_idx != NO_UPPER && (upper_idx as u64) < self.file.upper_capacity { - // empty the entry under its own lock BEFORE freeing: a traversal that already - // read this node's upper_idx must find a dead entry, not one reallocated to a - // different node mid-read - self.rewrite_upper(upper_idx, &[])?; - } - self.file.free_upper(upper_idx); - self.file.free_id(id); - Ok(()) - } - - /// Pick a new entry point: the highest-level live node among `preferred`, else the - /// first live node found scanning the id range (rare path: only when the entry's whole - /// neighborhood is gone). An empty graph clears the entry. - /// A node's level without copying its vector or edges (cheap re-election scans). - pub(crate) fn node_level(&self, id: u32) -> Option { - if !self.in_range(id) { - return None; - } - let seq = self.file.seq_atomic(id); - seqlock::read_consistent(seq, self.file.self_tag, || { - let p = self.file.slot_ptr(id); - unsafe { - if vread(p.add(S_FLAGS)) != FLAG_VALID { - return None; - } - Some(vread(p.add(S_LEVEL))) - } - }, self.slot_sanitizer(id), || None, self.owner_dead()) - } - - /// Highest-level live node among at most `limit` probes, skipping `skip`. The read-side - /// repair's last resort, bounded because `reelect_entry_point_replacing`'s scan runs to the - /// high-water mark and a search on the shared pool thread cannot afford it. - /// - /// Walks down from the newest id with a stride spanning the whole range, so it assumes - /// nothing about where the live nodes sit: Harper allocates ids monotonically and never - /// reuses them, so a churned table's low prefix is all tombstones, while the crate's own - /// freelist reuses ids and keeps live nodes low. - /// - /// The start rotates per handle, so `stride` consecutive repairs of this plane cover every id - /// while each stays capped at `limit`; a fixed start would probe one residue class forever - /// and leave a graph lying between its samples invisible permanently, not for one search. - /// That coverage rests on `stride * limit >= hw`, which is why the stride is a ceiling - /// division: below it a walk stops short of id 0 and no offset ever reaches the tail. - /// - /// Best-level rather than first-live: a level-0 entry degrades every later search to a - /// layer-0-only beam. - pub(crate) fn probe_for_entry(&self, limit: u32, skip: u32) -> Option<(u32, u8)> { - let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; - if hw == 0 || limit == 0 { - return None; - } - let stride = hw.div_ceil(limit); - let offset = self.probe_rotation.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % stride; - let mut best: Option<(u32, u8)> = None; - let mut cand = hw - 1 - offset; - for _ in 0..limit { - if cand != skip { - if let Some(level) = self.node_level(cand) { - if best.map(|(_, l)| level > l).unwrap_or(true) { - best = Some((cand, level)); - } - } - } - if cand < stride { - break; - } - cand -= stride; - } - best - } - - /// Pick a new entry point: the highest-level live node among `preferred`, else the - /// highest-level live node found scanning the id range (level reads only — no per-node - /// vector copies; still O(high-water), which only runs when an entry point vanished - /// with no live neighborhood). Preferring level keeps the hierarchy navigable — a - /// level-0 entry degrades every search to a layer-0-only beam. An empty graph clears - /// the entry. - pub(crate) fn reelect_entry_point_replacing(&self, preferred: &[u32], replacing: u32) { - let mut best: Option<(u32, u8)> = None; - // the most recently replaced entry point is the best cheap candidate: usually alive, - // usually high-level — and it makes the full fallback scan a last resort - let prev = self.file.previous_entry_point(); - if prev != crate::format::NO_ID && prev != replacing { - if let Some(level) = self.node_level(prev) { - best = Some((prev, level)); - } - } - for &cand in preferred { - if cand == replacing { - continue; // the node on its way out is never its own replacement - } - if let Some(level) = self.node_level(cand) { - if best.map(|(_, l)| level > l).unwrap_or(true) { - best = Some((cand, level)); - } - } - } - if best.is_none() { - let hw = self.file.id_high_water().min(self.file.max_nodes) as u32; - for cand in 0..hw { - if cand == replacing { - continue; - } - if let Some(level) = self.node_level(cand) { - if best.map(|(_, l)| level > l).unwrap_or(true) { - best = Some((cand, level)); - if level as usize >= MAX_UPPER_LEVELS { - break; // cannot do better - } - } - } - } - } - match best { - Some((cand, level)) => self.file.set_entry_point_if_not_better(cand, level as u32, replacing), - None => self.file.clear_entry_point_if(replacing), - } - } - - /// write_node, but only when the slot has never been touched — the check and the write - /// share ONE seqlock acquisition, so a concurrent live mirror's newer write can never be - /// overwritten by a backfill scan's older snapshot (a two-step check-then-write left - /// exactly that window). Returns true when this state was written. - #[allow(clippy::too_many_arguments)] - pub fn write_node_if_untouched( - &self, - id: u32, - level: u8, - vector: &[i8], - scale: f32, - inv_mag: f32, - neighbors: &[u32], - upper_levels: &[Vec], - ) -> Result { - debug_assert!(neighbors.len() <= self.file.layer0_cap); - debug_assert_eq!(vector.len(), self.file.dims); - self.file.ensure_high_water(id); - // the upper entry is allocated before taking the slot lock (allocation is cheap); it is - // unreachable from any slot until the write below lands, so every path that does not - // publish it — a wedged lock, a slot that turns out to be touched — has to free it - let upper_idx = if upper_levels.is_empty() { NO_UPPER } else { self.write_upper(upper_levels)? }; - let seq = self.file.seq_atomic(id); - let written = { - let _guard = match seqlock::write_lock(seq, self.file.self_tag, self.slot_sanitizer(id), self.owner_dead()) - { - Ok(guard) => guard, - Err(wedged) => { - self.file.free_upper(upper_idx); - return Err(wedged); - } - }; - let p = self.file.slot_ptr_mut(id); - let dims = self.file.dims; - unsafe { - if *p.add(S_FLAGS) != 0 { - false - } else { - *p.add(S_LEVEL) = level; - (p.add(S_DEGREE) as *mut u16).write_unaligned((neighbors.len() as u16).to_le()); - (p.add(S_SCALE) as *mut f32).write_unaligned(scale); - (p.add(S_INV_MAG) as *mut f32).write_unaligned(inv_mag); - (p.add(S_UPPER_IDX) as *mut u32).write_unaligned(upper_idx); - std::ptr::copy_nonoverlapping(vector.as_ptr() as *const u8, p.add(S_VECTOR), dims); - for (i, n) in neighbors.iter().enumerate() { - (p.add(neighbor_offset(dims) + i * 4) as *mut u32).write_unaligned(n.to_le()); - } - *p.add(S_FLAGS) = FLAG_VALID; - true - } - } - }; - if !written { - self.file.free_upper(upper_idx); - } - Ok(written) - } -} diff --git a/native/hnsw-plane/src/insert.rs b/native/hnsw-plane/src/insert.rs deleted file mode 100644 index 73070b1bea..0000000000 --- a/native/hnsw-plane/src/insert.rs +++ /dev/null @@ -1,367 +0,0 @@ -//! HNSW insert with parity to the JS implementation's optimizeRouting selection -//! (HierarchicalNavigableSmallWorld.ts): candidate i is skipped when an already-added -//! connection reaches it indirectly at comparable cost, and inferior indirect edges are -//! replaced by the new direct route. Stored per-edge distances were dropped from the file -//! format, so neighbor↔neighbor distances are recomputed (int8×int8) on id-match hits only. - -use crate::distance::{quantize_int8, Query}; -use crate::format::{NO_ID, NO_UPPER}; -use crate::graph::Graph; -use crate::search::{greedy_descend, search_layer, SearchScratch, SearchStats}; - -pub struct InsertParams { - pub m: usize, // base connection count (JS M, default 16) - pub ef_construction: usize, // candidate list size - pub ml: f64, // level normalization: 1 / ln(M) - pub optimize_routing: f32, // JS optimizeRouting, default 0.5; 0 disables -} - -impl Default for InsertParams { - fn default() -> Self { - InsertParams { m: 16, ef_construction: 200, ml: 1.0 / (16f64).ln(), optimize_routing: 0.5 } - } -} - -/// Deterministic pseudo-random level from the node id (reproducible benchmark builds). -fn level_for(id: u32, ml: f64) -> u8 { - let mut x = (id as u64).wrapping_mul(0x9e37_79b9_7f4a_7c15).wrapping_add(0x2545_f491_4f6c_dd1d); - x ^= x >> 33; - let unit = (x as f64) / (u64::MAX as f64); - let level = (-unit.max(f64::MIN_POSITIVE).ln() * ml).floor(); - (level as u8).min(crate::format::MAX_UPPER_LEVELS as u8) -} - -/// Remove `to` from `from`'s adjacency at `level` (edge-replacement maintenance). -fn remove_edge(graph: &Graph, from: u32, to: u32, level: u8) { - if level == 0 { - let _ = graph.update_neighbors(from, |list| { - if let Some(pos) = list.iter().position(|&x| x == to) { - list.remove(pos); - } - }); - } else { - let _ = graph.update_upper_level(from, level, |list| { - if let Some(pos) = list.iter().position(|&x| x == to) { - list.remove(pos); - } - }); - } -} - -/// Neighbor ids of `id` at `level` (level 0 from the slot, upper from the resident map). -fn neighbors_at(graph: &Graph, id: u32, level: u8, buf: &mut Vec) { - if level == 0 { - graph.neighbors_into(id, buf); - } else { - graph.upper_neighbors_into(id, level, buf); - } -} - -/// Prune an over-cap adjacency list by evicting the most REDUNDANT far member rather than -/// blindly the farthest: plain closest-keep can strip a node's last in-edge in dense -/// near-duplicate clusters, orphaning it from the graph (observed as unfindable self-queries -/// under concurrent builds). A far member e is redundant when some kept nearer member k has -/// d(e, k) < d(base, e) — searches reaching k still reach e. Bounded: farthest 16 candidates -/// checked against the nearest 16 keepers (~30us per overflow event); falls back to evicting -/// the plain farthest when nothing is provably redundant. -fn prune_with_coverage(graph: &Graph, base: u32, list: &mut Vec, cap: usize) { - let mut scored: Vec<(u32, f32)> = list - .iter() - .filter_map(|&cand| graph.distance_between(base, cand).map(|d| (cand, d))) - .collect(); - scored.sort_by(|a, b| a.1.total_cmp(&b.1)); - while scored.len() > cap { - let check_from = scored.len().saturating_sub(16); - let keepers = &scored[..16.min(check_from)]; - let mut evict = scored.len() - 1; // fallback: farthest - 'hunt: for i in (check_from..scored.len()).rev() { - let (e, d_base_e) = scored[i]; - for &(k, _) in keepers { - if k == e { - continue; - } - if let Some(d_ek) = graph.distance_between(e, k) { - if d_ek < d_base_e { - evict = i; - break 'hunt; - } - } - } - } - scored.remove(evict); - } - *list = scored.into_iter().map(|(cand, _)| cand).collect(); -} - -/// The contended fallback's merge: add `new_id` under the slot lock, displacing the tail once the -/// list is at `cap`. Which neighbor that is is arbitrary — appends push at the tail, so a list is -/// distance-ordered only immediately after a prune — but it must not be `new_id` itself, which is -/// what a push followed by `truncate(cap)` drops. That loss is the systematic one: the edge being -/// added is the in-edge keeping a freshly inserted node reachable from `nid`, and it disappears -/// every time the list is full and the CAS path is contended. Picking a better victim needs -/// distances, which this path deliberately keeps outside the lock. -fn merge_neighbor_capped(graph: &Graph, nid: u32, new_id: u32, cap: usize) { - let _ = graph.update_neighbors(nid, |list| { - if list.contains(&new_id) { - return; - } - if list.len() >= cap { - list.truncate(cap.saturating_sub(1)); - } - list.push(new_id); - }); -} - -/// Add `new_id` to `nid`'s adjacency at `level`, coverage-pruning to `cap` when over. The -/// prune's distance computations (which can major-fault on a cold mapping) run OUTSIDE the -/// slot lock: the list is snapshotted, pruned, and applied with a compare-and-set; after a -/// bounded retry the fallback merges under the lock with a cheap truncation instead. -fn add_reverse_edge(graph: &Graph, nid: u32, new_id: u32, level: u8, cap: usize) { - if level == 0 { - for _ in 0..2 { - let mut snapshot: Vec = Vec::new(); - if graph.neighbors_into(nid, &mut snapshot).is_none() { - return; - } - if snapshot.contains(&new_id) { - return; - } - let mut next = snapshot.clone(); - next.push(new_id); - if next.len() > cap { - prune_with_coverage(graph, nid, &mut next, cap); - } - if graph.set_neighbors_if(nid, &snapshot, &next).unwrap_or(false) { - return; - } - } - // contended twice: merge cheaply under the lock (bounded critical section) - merge_neighbor_capped(graph, nid, new_id, cap); - } else { - let _ = graph.update_upper_level(nid, level, |list| { - if list.contains(&new_id) { - return; - } - list.push(new_id); - if list.len() > cap { - prune_with_coverage(graph, nid, list, cap); - } - }); - } -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum InsertError { - /// max_nodes reached (freeing capacity makes inserts possible again) - Full, - /// a slot lock could not be acquired or reclaimed within the wedge bound - Wedged, -} - -/// Insert a vector, returning its node id. -pub fn insert( - graph: &Graph, - vector: &[f32], - params: &InsertParams, - scratch: &mut SearchScratch, -) -> Result { - let (bytes, scale, inv_mag) = quantize_int8(vector); - let id = graph.file.allocate_id(); - if id == NO_ID { - return Err(InsertError::Full); - } - let level = level_for(id, params.ml); - let query = Query::new(vector.to_vec()); - let layer0_cap = graph.file.layer0_cap; - let m = params.m; - - let mut stats = SearchStats { visits: 0 }; - // Upper entry a first-entry claim attempt already published for `id`. Its slot names the - // index, so the join path must rewrite it in place — freeing an index a live slot names - // would let another node adopt it mid-traversal. - let mut published_upper = NO_UPPER; - let mut published = false; - let publish_edgeless = |published: &mut bool, published_upper: &mut u32| -> Result<(), InsertError> { - if *published { - return Ok(()); - } - *published_upper = - if level > 0 { graph.write_upper(&vec![Vec::new(); level as usize]).unwrap_or(NO_UPPER) } else { NO_UPPER }; - graph.write_node(id, level, &bytes, scale, inv_mag, &[], *published_upper).map_err(|_| InsertError::Wedged)?; - *published = true; - Ok(()) - }; - - // Resolve an entry point to grow from. Every turn makes progress — it claims an empty - // graph, joins a live entry, or replaces one that is provably gone — so the cap only - // guards an insert/delete interleaving that keeps clearing the entry under us. - let mut joined = None; - for _ in 0..16 { - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id == NO_ID { - publish_edgeless(&mut published, &mut published_upper)?; - // Claim only from EMPTY, and only the winner returns: a not-worse install would put - // this edgeless node over a live equal-or-lower-level entry and orphan the graph - // behind it, and it cannot report losing, which a loser must know to join instead. - if graph.file.claim_entry_if_empty(id, level as u32) { - return Ok(id); - } - continue; // a racer rooted the graph — join it rather than stand alone - } - if let Some(d) = graph.distance_to(entry_id, &query) { - joined = Some((entry_id, entry_level, d)); - break; - } - // The stored entry point is gone (e.g. a mirroring host cleared it without - // re-electing). Self-promoting an edgeless new node here would orphan the whole - // existing graph behind an unreachable root — re-elect from the live graph and - // continue; only a truly empty graph makes this node the first entry. - graph.reelect_entry_point_replacing(&[], entry_id); - } - // An unresolvable entry point is an error the host retries: Ok here would report success - // for a node no search can reach. - let Some((entry_id, entry_level, entry_dist)) = joined else { - return Err(InsertError::Wedged); - }; - let top = level.min(entry_level as u8); - let (mut ep, mut ep_dist) = - greedy_descend(graph, &query, entry_id, entry_dist, entry_level, top as u32, &mut stats); - - // Per-level connection lists for the new node, selection-ordered. - let mut connections: Vec> = vec![Vec::new(); level as usize + 1]; - let mut nbuf: Vec = Vec::new(); - - for l in (0..=top).rev() { - scratch_begin(graph, scratch); - let mut neighbors = - search_layer(graph, &query, ep, ep_dist, params.ef_construction, l, scratch, &mut stats, None, u64::MAX); - neighbors.truncate(m << 1); - if let Some(&(best, best_d)) = neighbors.first() { - ep = best; - ep_dist = best_d; - } - - // JS optimizeRouting selection over rank-ordered candidates. - let take_conns = std::mem::take(&mut connections[l as usize]); - let mut conns = take_conns; - for (i, &(nid, ndist)) in neighbors.iter().enumerate() { - if nid == id { - continue; - } - let mut skipping = false; - let mut replaced: Vec<(u32, u32)> = Vec::new(); // (from, to) edge removals - if params.optimize_routing > 0.0 { - let distance_threshold = 1.0 + params.optimize_routing * (1.0 + (0.5 * i as f32) / m as f32); - neighbors_at(graph, nid, l, &mut nbuf); - for (i2, &nnid) in nbuf.iter().enumerate() { - let neighbor_threshold = 1.0 + params.optimize_routing * (1.0 + (0.5 * i2 as f32) / m as f32); - if let Some(&(added_id, added_dist)) = conns.iter().find(|(aid, _)| *aid == nnid) { - // recompute the stored neighbor↔neighbor distance (not persisted) - let neighbor_distance = graph.distance_between(nid, nnid).unwrap_or(f32::INFINITY); - if ndist * distance_threshold > added_dist + neighbor_distance { - skipping = true; - break; // JS: `if (skipping) break` ends the neighbor scan - } else if neighbor_distance * neighbor_threshold > ndist + added_dist { - replaced.push((added_id, nid)); - replaced.push((nid, added_id)); - } - // JS breaks only the inner connections scan; keep scanning neighbors - } - } - if skipping { - continue; - } - } else if i >= if l > 0 { m } else { m << 1 } { - continue; - } - conns.push((nid, ndist)); - for (from, to) in replaced { - remove_edge(graph, from, to, l); - } - } - connections[l as usize] = conns; - } - - // Write the new node: upper entry first so a reader that sees the node sees its - // hierarchy; layer-0 list pruned to the file cap (selection order = rank order). - let upper_idx = if level > 0 { - let levels: Vec> = (1..=level as usize) - .map(|l| { - connections - .get(l) - .map(|c| c.iter().map(|&(nid, _)| nid).collect()) - .unwrap_or_default() - }) - .collect(); - if published_upper != NO_UPPER { - graph.rewrite_upper(published_upper, &levels).map_err(|_| InsertError::Wedged)?; - published_upper - } else { - graph.write_upper(&levels).unwrap_or(NO_UPPER) - } - } else { - NO_UPPER - }; - let mut l0: Vec = connections[0].iter().map(|&(nid, _)| nid).collect(); - l0.truncate(layer0_cap); - graph.write_node(id, level, &bytes, scale, inv_mag, &l0, upper_idx).map_err(|_| InsertError::Wedged)?; - - // Reverse edges. - for (l, conns) in connections.iter().enumerate() { - let cap = if l == 0 { layer0_cap } else { m << 1 }; - for &(nid, _) in conns { - add_reverse_edge(graph, nid, id, l as u8, cap); - } - } - - if (level as u32) > entry_level { - // CAS against the observed entry: a concurrent higher-level promotion wins - graph.file.promote_entry_point(id, level as u32, entry_id); - } - Ok(id) -} - -#[inline] -fn scratch_begin(graph: &Graph, scratch: &mut SearchScratch) { - // search_layer assumes a fresh epoch per sweep; SearchScratch::begin is crate-private - // via this helper to keep the public surface small. - scratch.begin_public(graph.file.id_high_water()); -} - -#[cfg(test)] -mod reverse_edge_tests { - use super::*; - use crate::PlaneFile; - - /// The contended fallback must still add the edge when the neighbor list is already full — - /// the one case where a push-then-`truncate(cap)` discards `new_id` rather than a neighbor, - /// losing the in-edge exactly in the contended-and-full case the fallback exists to serve. - #[test] - fn a_contended_merge_into_a_full_neighbor_list_keeps_the_edge_it_adds() { - let dims = 8; - let cap = 8usize; - let path = std::env::temp_dir().join(format!("hnsw-revedge-{}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, cap, 4_096).expect("create")); - - let vector = vec![0i8; dims]; - let full: Vec = (1..=cap as u32).collect(); - graph.write_node_raw(0, 0, &vector, 1.0, 1.0, &full, &[]).expect("seed the full list"); - for &nid in &full { - graph.write_node_raw(nid, 0, &vector, 1.0, 1.0, &[], &[]).expect("seed a neighbor"); - } - let newcomer = cap as u32 + 1; - graph.write_node_raw(newcomer, 0, &vector, 1.0, 1.0, &[], &[]).expect("seed the newcomer"); - - merge_neighbor_capped(&graph, 0, newcomer, cap); - - let mut neighbors: Vec = Vec::new(); - graph.neighbors_into(0, &mut neighbors).expect("node 0 is live"); - assert!( - neighbors.contains(&newcomer), - "the contended merge dropped the edge it was adding: {neighbors:?}" - ); - assert_eq!(neighbors.len(), cap, "the merge must stay within the layer-0 cap"); - let _ = std::fs::remove_file(&path); - } -} diff --git a/native/hnsw-plane/src/lib.rs b/native/hnsw-plane/src/lib.rs deleted file mode 100644 index 654bdd4cdb..0000000000 --- a/native/hnsw-plane/src/lib.rs +++ /dev/null @@ -1,15 +0,0 @@ -//! hnsw-plane: native HNSW traversal plane over a memory-mapped fixed-slot file. -//! Design: ../../hnsw-native-plane.md. NAPI bindings land behind the `napi` feature in -//! phase-1 integration; the core is buildable and benchmarkable standalone. - -pub mod distance; -pub mod format; -pub mod graph; -pub mod insert; -#[cfg(feature = "napi")] -mod napi; -pub mod search; -pub mod seqlock; - -pub use format::PlaneFile; -pub use graph::Graph; diff --git a/native/hnsw-plane/src/napi.rs b/native/hnsw-plane/src/napi.rs deleted file mode 100644 index d15700ffbf..0000000000 --- a/native/hnsw-plane/src/napi.rs +++ /dev/null @@ -1,496 +0,0 @@ -//! NAPI surface (feature = "napi"). One boundary crossing per operation; searches run on -//! the libuv thread pool via AsyncTask so the JS event loop is never blocked (C1). -//! The surface is deliberately Harper-agnostic — pk↔id mapping, commit-callback glue, and -//! txnlog-anchored replay live in the host application. - -use crate::distance::Query; -use crate::insert::{insert, InsertParams}; -use crate::search::{search_filtered, search_predicated, PredicatePipe, SearchScratch}; -use crate::{Graph, PlaneFile}; -use napi::bindgen_prelude::*; -use napi::threadsafe_function::{ErrorStrategy, ThreadsafeFunction, ThreadsafeFunctionCallMode}; -use napi::JsFunction; -use napi_derive::napi; -use std::sync::{Arc, Mutex}; - -/// Pooled per-query scratch (the visited array is O(nodes); never allocate per query). -struct ScratchPool(Mutex>); - -impl ScratchPool { - fn take(&self) -> SearchScratch { - self.0.lock().unwrap().pop().unwrap_or_default() - } - fn put(&self, s: SearchScratch) { - let mut pool = self.0.lock().unwrap(); - if pool.len() < 64 { - pool.push(s); - } - } -} - -#[napi(object)] -pub struct SearchHit { - pub id: u32, - pub distance: f64, -} - -pub struct SearchTask { - graph: Arc, - pool: Arc, - query: Vec, - k: usize, - ef: usize, - filter: Option>, - filter_expansion: usize, -} - -#[napi] -impl Task for SearchTask { - type Output = Vec<(u32, f32)>; - type JsValue = Vec; - - fn compute(&mut self) -> Result { - let mut scratch = self.pool.take(); - let query = Query::new(std::mem::take(&mut self.query)); - let (hits, _stats) = search_filtered( - &self.graph, - &query, - self.k, - self.ef, - self.filter.as_deref(), - self.filter_expansion, - &mut scratch, - ); - self.pool.put(scratch); - Ok(hits) - } - - fn resolve(&mut self, _env: Env, output: Self::Output) -> Result { - Ok(output.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) - } -} - -pub struct PredicateSearchTask { - graph: Arc, - pool: Arc, - query: Vec, - k: usize, - ef: usize, - tsfn: Option, ErrorStrategy::Fatal>>, - visit_budget: u64, -} - -#[napi] -impl Task for PredicateSearchTask { - type Output = Vec<(u32, f32)>; - type JsValue = Vec; - - fn compute(&mut self) -> Result { - let tsfn = self.tsfn.take().ok_or_else(|| Error::from_reason("task reused"))?; - let (tx, rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); - let mut pipe = PredicatePipe { - dispatch: Box::new(move |ids: Vec| { - let tx = tx.clone(); - let ids_echo = ids.clone(); - let status = tsfn.call_with_return_value( - ids, - ThreadsafeFunctionCallMode::NonBlocking, - move |ret: Uint8Array| { - // predicate errors / env teardown surface as a missing send; the - // drain deadline in search_predicated treats absent verdicts as deny - let _ = tx.send((ids_echo, ret.to_vec())); - Ok(()) - }, - ); - // a closing or saturated queue drops the callback without invoking it, so this - // batch will never answer; reporting it lets the drain finish on the batches - // that will, instead of holding teardown for the full deadline - status == Status::Ok - }), - rx, - }; - let mut scratch = self.pool.take(); - let query = Query::new(std::mem::take(&mut self.query)); - let (hits, _stats) = - search_predicated(&self.graph, &query, self.k, self.ef, &mut pipe, self.visit_budget, &mut scratch); - self.pool.put(scratch); - Ok(hits) - } - - fn resolve(&mut self, _env: Env, output: Self::Output) -> Result { - Ok(output.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) - } -} - -pub struct FlushTask { - graph: Arc, - txn: Option, -} - -#[napi] -impl Task for FlushTask { - type Output = (); - type JsValue = (); - - fn compute(&mut self) -> Result { - self.graph.file.flush_with_watermark(self.txn).map_err(|e| Error::from_reason(e.to_string())) - } - - fn resolve(&mut self, _env: Env, _output: Self::Output) -> Result { - Ok(()) - } -} - -#[napi] -pub struct Plane { - graph: Arc, - pool: Arc, - params: InsertParams, - // insert scratch, serialized: phase-1 hosts call insert from a single writer at a time - // per index (Harper's commit path); a Mutex keeps misuse safe rather than fast. - insert_scratch: Mutex, -} - -#[napi] -impl Plane { - /// Create a new plane file. `maxNodes` bounds the sparse reservation (pages materialize - /// on write). - #[napi(factory)] - pub fn create(path: String, dims: u32, layer0_cap: u32, max_nodes: f64) -> Result { - let file = PlaneFile::create(std::path::Path::new(&path), dims as usize, layer0_cap as usize, max_nodes as u64) - .map_err(|e| Error::from_reason(e.to_string()))?; - Ok(Self::wrap(file)) - } - - /// Open an existing plane file (the upper-layer region lives in the same file). - #[napi(factory)] - pub fn open(path: String) -> Result { - let file = PlaneFile::open(std::path::Path::new(&path)).map_err(|e| Error::from_reason(e.to_string()))?; - Ok(Self::wrap(file)) - } - - fn wrap(file: PlaneFile) -> Plane { - Plane { - graph: Arc::new(Graph::new(file)), - pool: Arc::new(ScratchPool(Mutex::new(Vec::new()))), - params: InsertParams::default(), - insert_scratch: Mutex::new(SearchScratch::new()), - } - } - - /// Insert a vector; returns the allocated node id (freelist ids are reused). Throws on - /// a dimension mismatch or a full plane (maxNodes reached). - #[napi] - pub fn insert(&self, vector: Float32Array) -> Result { - if vector.len() != self.graph.file.dims { - return Err(Error::from_reason(format!( - "vector has {} dims; plane was created with {}", - vector.len(), - self.graph.file.dims - ))); - } - for (i, v) in vector.iter().enumerate() { - if !v.is_finite() { - // a NaN component yields a huge invMag and -inf distances: that node would - // rank first for roughly half of all queries, permanently - return Err(Error::from_reason(format!("vector component {i} is not finite"))); - } - } - let mut scratch = self.insert_scratch.lock().unwrap(); - insert(&self.graph, &vector, &self.params, &mut scratch).map_err(|e| match e { - crate::insert::InsertError::Full => Error::from_reason("plane is full (maxNodes reached)"), - crate::insert::InsertError::Wedged => { - Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index") - } - }) - } - - /// Delete a node; its id returns to the plane freelist. Standalone-allocation mode only - /// (pairs with insert()); dual-write hosts use clearNode instead. - #[napi] - pub fn remove(&self, id: u32) -> Result<()> { - self.graph - .delete_node(id) - .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) - } - - /// Mirror a host-maintained node into the plane (dual-write phase 1): full node state - /// per call, host-allocated id, int8 vector bin + quantization scale + cached 1/|v|, - /// layer-0 neighbor ids, and per-upper-level neighbor id arrays (level 1 first). An - /// existing upper entry is rewritten in place. Idempotent per (id, state). - #[napi] - pub fn write_node_raw( - &self, - id: u32, - level: u8, - vector: Buffer, - scale: f64, - inv_mag: f64, - neighbors: Uint32Array, - upper: Option>, - ) -> Result<()> { - if vector.len() != self.graph.file.dims { - return Err(Error::from_reason(format!( - "vector is {} bytes; plane dims = {}", - vector.len(), - self.graph.file.dims - ))); - } - // ensure_high_water + slot_ptr have no bounds check, so a host id past the fixed - // reservation would address past the slot region (mmap overrun) — reject it here. - if id as u64 >= self.graph.file.max_nodes { - return Err(Error::from_reason(format!( - "node id {} exceeds the plane's maxNodes reservation ({})", - id, self.graph.file.max_nodes - ))); - } - if !(scale as f32).is_finite() || !(inv_mag as f32).is_finite() { - return Err(Error::from_reason("scale/invMag must be finite")); - } - let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; - let upper_levels: Vec> = - upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); - // reject out-of-range neighbor ids rather than letting them poison traversal - // (SearchScratch::visit would size its array from them; distance_to skips them, but - // a u32::MAX id costs a huge allocation before it is skipped) - let max = self.graph.file.max_nodes; - for &n in neighbors.iter() { - if (n as u64) >= max { - return Err(Error::from_reason(format!("neighbor id {n} exceeds plane capacity {max}"))); - } - } - for level in &upper_levels { - for &n in level { - if (n as u64) >= max { - return Err(Error::from_reason(format!("upper neighbor id {n} exceeds plane capacity {max}"))); - } - } - } - self.graph - .write_node_raw(id, level, vec_i8, scale as f32, inv_mag as f32, &neighbors.to_vec(), &upper_levels) - .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) - } - - /// Builder-scan variant of writeNodeRaw: writes ONLY when the slot has never been - /// touched (valid or deleted). A backfill scan mirroring a snapshot must not overwrite - /// a node a concurrent live mirror already wrote with newer state — the check and the - /// write happen under the slot's seqlock, so the race is closed across workers too. - /// Returns true when the scan's state was written. - #[napi] - #[allow(clippy::too_many_arguments)] - pub fn write_node_raw_if_absent( - &self, - id: u32, - level: u8, - vector: Buffer, - scale: f64, - inv_mag: f64, - neighbors: Uint32Array, - upper: Option>, - ) -> Result { - if vector.len() != self.graph.file.dims { - return Err(Error::from_reason(format!( - "vector is {} bytes; plane dims = {}", - vector.len(), - self.graph.file.dims - ))); - } - if (id as u64) >= self.graph.file.max_nodes { - return Err(Error::from_reason(format!("id {} exceeds plane capacity {}", id, self.graph.file.max_nodes))); - } - if !(scale as f32).is_finite() || !(inv_mag as f32).is_finite() { - return Err(Error::from_reason("scale/invMag must be finite")); - } - let max = self.graph.file.max_nodes; - for &n in neighbors.iter() { - if (n as u64) >= max { - return Err(Error::from_reason(format!("neighbor id {n} exceeds plane capacity {max}"))); - } - } - let upper_levels: Vec> = - upper.map(|ls| ls.iter().map(|l| l.to_vec()).collect()).unwrap_or_default(); - for level_ids in &upper_levels { - for &n in level_ids { - if (n as u64) >= max { - return Err(Error::from_reason(format!("upper neighbor id {n} exceeds plane capacity {max}"))); - } - } - } - let vec_i8 = unsafe { std::slice::from_raw_parts(vector.as_ptr() as *const i8, vector.len()) }; - let mut l0 = neighbors.to_vec(); - l0.truncate(self.graph.file.layer0_cap); - // the untouched check and the write share one seqlock acquisition inside the crate: - // a live mirror's newer write can never be overwritten by this scan's older snapshot - self.graph - .write_node_if_untouched(id, level, vec_i8, scale as f32, inv_mag as f32, &l0, &upper_levels) - .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) - } - - /// Advisory: whether the file recorded a durability barrier (flush) as its last state - /// when this handle opened it. Crash recovery does not depend on it — torn per-slot - /// locks are taken over lazily at the affected slot. - #[napi] - pub fn opened_clean(&self) -> bool { - self.graph.file.opened_clean - } - - /// Async durability barrier on the libuv pool: same ordering contract as flush(), off - /// the event loop — a whole-map msync over a large mapping stalls its calling thread. - #[napi(ts_return_type = "Promise")] - pub fn flush_async(&self, watermark: Option) -> AsyncTask { - let txn = watermark.map(|w| w as u64); - AsyncTask::new(FlushTask { graph: self.graph.clone(), txn }) - } - - /// Mark a node deleted without touching the plane freelist (dual-write mode: the host - /// owns id allocation). - #[napi] - pub fn clear_node(&self, id: u32) -> Result<()> { - self.graph - .clear_node(id) - .map_err(|_| Error::from_reason("plane slot lock is wedged (unreclaimable holder); rebuild the index")) - } - - /// Set the graph entry point (dual-write mode mirrors the host's entry-point updates). - #[napi] - pub fn set_entry_point(&self, id: u32, level: u32) { - // clamp: a garbage level would make every search iterate that many empty levels - self.graph.file.set_entry_point(id, level.min(crate::format::MAX_UPPER_LEVELS as u32)); - } - - #[napi] - pub fn get_entry_point(&self) -> Vec { - let (id, level) = self.graph.file.entry_point(); - vec![id as f64, level as f64] - } - - /// Query dimensionality must match the plane: the distance kernel streams - /// `query.len()` bytes from each slot's vector, so an oversized query would read past - /// it into adjacent slot bytes (or off the mapping entirely). - fn check_query_dims(&self, len: usize) -> Result<()> { - if len != self.graph.file.dims { - return Err(Error::from_reason(format!( - "query vector has {} dimensions; plane dims = {}", - len, self.graph.file.dims - ))); - } - Ok(()) - } - - #[napi(getter)] - pub fn dims(&self) -> u32 { - self.graph.file.dims as u32 - } - - #[napi(getter)] - pub fn layer0_cap(&self) -> u32 { - self.graph.file.layer0_cap as u32 - } - - /// Async k-NN search on the libuv thread pool. `filter` is an optional allow-bitset - /// over node ids (bit i of byte i>>3); filtered searches are visit-bounded by - /// ef * filterExpansion (default 24). - #[napi(ts_return_type = "Promise>")] - pub fn search( - &self, - vector: Float32Array, - k: u32, - ef: u32, - filter: Option, - filter_expansion: Option, - ) -> Result> { - self.check_query_dims(vector.len())?; - Ok(AsyncTask::new(SearchTask { - graph: self.graph.clone(), - pool: self.pool.clone(), - query: vector.to_vec(), - k: k as usize, - ef: ef as usize, - filter: filter.map(|f| f.to_vec()), - filter_expansion: filter_expansion.unwrap_or(24) as usize, - })) - } - - /// Async k-NN search with a JS predicate: `predicate(ids: number[]) => Uint8Array` - /// (one 0/1 byte per id, evaluated synchronously). Batches of candidate ids stream to - /// the predicate over a ThreadsafeFunction while traversal keeps expanding — the search - /// thread never blocks on the JS event loop until the beam itself is done, so a busy - /// loop costs speculative overshoot (bounded by the visit budget), not latency. - /// `visitBudget` caps layer-0 visits absolutely (a host budget may sit below ef, which a - /// multiplier cannot express); when absent the budget is ef * filterExpansion. - /// Must not be awaited synchronously from code the predicate itself blocks. - #[napi(ts_return_type = "Promise>")] - pub fn search_with_predicate( - &self, - vector: Float32Array, - k: u32, - ef: u32, - #[napi(ts_arg_type = "(ids: Array) => Uint8Array")] predicate: JsFunction, - filter_expansion: Option, - visit_budget: Option, - ) -> Result> { - self.check_query_dims(vector.len())?; - let tsfn: ThreadsafeFunction, ErrorStrategy::Fatal> = predicate - .create_threadsafe_function(0, |ctx: napi::threadsafe_function::ThreadSafeCallContext>| { - let ids: Vec = ctx.value.iter().map(|&v| v as f64).collect(); - Ok(vec![ids]) - })?; - let ef = ef as usize; - Ok(AsyncTask::new(PredicateSearchTask { - graph: self.graph.clone(), - pool: self.pool.clone(), - query: vector.to_vec(), - k: k as usize, - ef, - tsfn: Some(tsfn), - visit_budget: visit_budget - .map(|b| b.max(1.0) as u64) - .unwrap_or((ef * filter_expansion.unwrap_or(24) as usize) as u64), - })) - } - - /// Synchronous search (benchmarks/tests; blocks the calling thread). - #[napi] - pub fn search_sync(&self, vector: Float32Array, k: u32, ef: u32) -> Result> { - self.check_query_dims(vector.len())?; - let mut scratch = self.pool.take(); - let query = Query::new(vector.to_vec()); - let (hits, _) = search_filtered(&self.graph, &query, k as usize, ef as usize, None, 24, &mut scratch); - self.pool.put(scratch); - Ok(hits.into_iter().map(|(id, d)| SearchHit { id, distance: d as f64 }).collect()) - } - - /// Lifetime id high-water (allocated ids, including freed ones awaiting reuse). - #[napi] - pub fn id_high_water(&self) -> f64 { - self.graph.file.id_high_water() as f64 - } - - #[napi] - pub fn get_watermark(&self) -> f64 { - self.graph.file.watermark() as f64 - } - - #[napi] - pub fn set_watermark(&self, txn: f64) { - self.graph.file.set_watermark(txn as u64); - } - - /// Durability barrier: flush all data, then advance the watermark (defaults to the - /// current one) and the clean-shutdown flag, then flush the header alone — so a crash - /// between the flushes can only leave an OLD watermark over durable data (replay - /// re-covers a suffix), never a new watermark over missing data. - #[napi] - pub fn flush(&self, watermark: Option) -> Result<()> { - self.graph.file.flush_with_watermark(watermark.map(|w| w as u64)).map_err(|e| Error::from_reason(e.to_string())) - } - - /// Durably mark this plane an incomplete mirror: zero the watermark and msync the header - /// page alone, so a host disabling a plane it cannot delete has the mark on disk before it - /// writes any out-of-band tombstone. Synchronous by design — it is a 4 KB msync, not the - /// whole-mapping writeback `flush` performs. - #[napi] - pub fn invalidate(&self) -> Result<()> { - self.graph.file.invalidate().map_err(|e| Error::from_reason(e.to_string())) - } -} diff --git a/native/hnsw-plane/src/search.rs b/native/hnsw-plane/src/search.rs deleted file mode 100644 index 9e8ca8b47b..0000000000 --- a/native/hnsw-plane/src/search.rs +++ /dev/null @@ -1,589 +0,0 @@ -//! Beam search over the plane, zero-copy: per-visit cost is one seqlock-guarded distance -//! against mmap bytes plus primitive heap/visited ops. Visited tracking is an epoch-stamped -//! array; neighbor ids stream through a reusable scratch buffer. - -use crate::distance::Query; -use crate::format::NO_ID; -use crate::graph::Graph; -use std::cmp::Ordering as CmpOrdering; -use std::collections::BinaryHeap; - -#[derive(PartialEq)] -struct Candidate { - distance: f32, - id: u32, -} -impl Eq for Candidate {} -impl Ord for Candidate { - fn cmp(&self, other: &Self) -> CmpOrdering { - // min-heap by distance via reverse - other.distance.partial_cmp(&self.distance).unwrap_or(CmpOrdering::Equal) - } -} -impl PartialOrd for Candidate { - fn partial_cmp(&self, other: &Self) -> Option { - Some(self.cmp(other)) - } -} - -#[derive(PartialEq)] -struct Result_ { - distance: f32, - id: u32, -} -impl Eq for Result_ {} -impl Ord for Result_ { - fn cmp(&self, other: &Self) -> CmpOrdering { - // max-heap by distance (worst result on top for eviction) - self.distance.partial_cmp(&other.distance).unwrap_or(CmpOrdering::Equal) - } -} -impl PartialOrd for Result_ { - fn partial_cmp(&self, other: &Self) -> Option { - Some(self.cmp(other)) - } -} - -/// Reusable per-thread search scratch. -pub struct SearchScratch { - visited: Vec, - epoch: u32, - neighbors: Vec, -} - -impl SearchScratch { - pub fn new() -> Self { - SearchScratch { visited: Vec::new(), epoch: 0, neighbors: Vec::new() } - } - - pub fn begin_public(&mut self, capacity: u64) { - self.begin(capacity) - } - - fn begin(&mut self, capacity: u64) { - if self.visited.len() < capacity as usize { - self.visited.resize(capacity as usize, 0); - } - self.epoch = self.epoch.wrapping_add(1); - if self.epoch == 0 { - self.visited.fill(0); - self.epoch = 1; - } - } - - #[inline] - fn visit(&mut self, id: u32) -> bool { - // ids minted by concurrent inserts after begin() can exceed the sizing snapshot; - // growth is bounded by the id itself, which write paths bound by max_nodes - if id as usize >= self.visited.len() { - self.visited.resize(id as usize + 1024, 0); - } - let slot = &mut self.visited[id as usize]; - if *slot == self.epoch { - false - } else { - *slot = self.epoch; - true - } - } -} - -impl Default for SearchScratch { - fn default() -> Self { - Self::new() - } -} - -pub struct SearchStats { - pub visits: u64, -} - -#[inline] -fn bit_allowed(filter: Option<&[u8]>, id: u32) -> bool { - match filter { - None => true, - Some(bits) => { - let byte = (id >> 3) as usize; - byte < bits.len() && bits[byte] & (1 << (id & 7)) != 0 - } - } -} - -/// Beam search within one layer, starting from `entry`. Level 0 reads slot adjacency; -/// upper levels read the resident upper map. Returns (id, distance) ascending by distance. -/// Assumes scratch.begin() was called for this query; entry is marked visited here. -/// -/// `filter`: optional allow-bitset over node ids (bit i of byte i>>3). Filtered-out nodes -/// are traversed (their edges route) but excluded from results — ACORN-style — with -/// `visit_budget` bounding total visits so a selective filter terminates. -pub fn search_layer( - graph: &Graph, - query: &Query, - entry: u32, - entry_dist: f32, - ef: usize, - level: u8, - scratch: &mut SearchScratch, - stats: &mut SearchStats, - filter: Option<&[u8]>, - visit_budget: u64, -) -> Vec<(u32, f32)> { - let mut candidates = BinaryHeap::new(); - let mut results: BinaryHeap = BinaryHeap::new(); - scratch.visit(entry); - candidates.push(Candidate { distance: entry_dist, id: entry }); - if bit_allowed(filter, entry) { - results.push(Result_ { distance: entry_dist, id: entry }); - } - - // take() the scratch neighbor buffer to sidestep the double-borrow of scratch - let mut nbuf = std::mem::take(&mut scratch.neighbors); - - while let Some(c) = candidates.pop() { - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() >= ef && c.distance > worst { - break; - } - if stats.visits >= visit_budget { - break; - } - if level == 0 { - if graph.neighbors_into(c.id, &mut nbuf).is_none() { - continue; - } - } else { - graph.upper_neighbors_into(c.id, level, &mut nbuf); - } - for i in 0..nbuf.len() { - let nid = nbuf[i]; - if (nid as u64) >= graph.file.max_nodes { - continue; // corrupt/torn neighbor id: skip rather than size allocations by it - } - if !scratch.visit(nid) { - continue; - } - if let Some(d) = graph.distance_to(nid, query) { - stats.visits += 1; - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() < ef || d < worst { - candidates.push(Candidate { distance: d, id: nid }); - if bit_allowed(filter, nid) { - results.push(Result_ { distance: d, id: nid }); - if results.len() > ef { - results.pop(); - } - } - } - } - } - } - scratch.neighbors = nbuf; - - let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); - out.sort_by(|a, b| a.1.total_cmp(&b.1)); - out -} - -/// Greedy single-candidate descent through upper layers from `from_level` down to -/// `to_level` (exclusive lower bound handled by caller loops). Returns improved entry. -pub fn greedy_descend( - graph: &Graph, - query: &Query, - mut current: u32, - mut current_dist: f32, - from_level: u32, - to_level: u32, - stats: &mut SearchStats, -) -> (u32, f32) { - let mut nbuf: Vec = Vec::new(); - let mut level = from_level; - while level > to_level { - let mut improved = true; - while improved { - improved = false; - graph.upper_neighbors_into(current, level.min(255) as u8, &mut nbuf); - for i in 0..nbuf.len() { - let nid = nbuf[i]; - if let Some(d) = graph.distance_to(nid, query) { - stats.visits += 1; - if d < current_dist { - current = nid; - current_dist = d; - improved = true; - } - } - } - } - level -= 1; - } - (current, current_dist) -} - -/// Slots a read-side repair may probe when the previous-entry hint is dead too. Bounded so a -/// search never pays the write path's O(high-water) re-election scan. -const REPAIR_PROBE_LIMIT: u32 = 1024; - -/// Resolve a live entry point for a read, repairing a dead one in place. -/// -/// A search that finds the header naming a deleted or sanitized node returns EMPTY, and on a -/// read-mostly table nothing ever repairs it: write-path re-election only runs on delete, and -/// a slot a reader sanitized after its writer died had no delete at all. -/// -/// The candidate is the O(1) previous-entry hint, then a probe capped at `REPAIR_PROBE_LIMIT` — -/// the hint is a single slot and can be dead itself. The cap is what keeps a read off the write -/// path's O(high-water) scan on the pool thread every search shares, and the repair publishes, -/// so only the first search after a wedge pays even the probe. -fn resolve_entry(graph: &Graph, query: &Query, stats: &mut SearchStats) -> Option<(u32, u32, f32)> { - let (entry_id, entry_level) = graph.file.entry_point(); - if entry_id != NO_ID { - if let Some(d) = graph.distance_to(entry_id, query) { - stats.visits += 1; - return Some((entry_id, entry_level, d)); - } - } - let hint = graph.file.previous_entry_point(); - let candidate = (hint != NO_ID && hint != entry_id) - .then(|| graph.node_level(hint).map(|level| (hint, level))) - .flatten() - .or_else(|| graph.probe_for_entry(REPAIR_PROBE_LIMIT, entry_id)); - let (id, level) = candidate?; - let d = graph.distance_to(id, query)?; - stats.visits += 1; - // Strict on the entry we observed dead, not a not-worse install: a live level-0 root claimed - // since the read above must win, or it is orphaned with nothing pointing at it. - graph.file.replace_entry_if(entry_id, id, level as u32); - Some((id, level as u32, d)) -} - -/// Full search: greedy descent through upper layers, then beam at layer 0. -pub fn search( - graph: &Graph, - query: &Query, - k: usize, - ef: usize, - scratch: &mut SearchScratch, -) -> (Vec<(u32, f32)>, SearchStats) { - let mut stats = SearchStats { visits: 0 }; - let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { - return (Vec::new(), stats); - }; - scratch.begin(graph.file.id_high_water()); - let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); - let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, None, u64::MAX); - out.truncate(k); - (out, stats) -} - -/// Full search with an optional allow-bitset filter. `filter_expansion` multiplies ef into -/// the visit budget when a filter is present (matching the JS filterExpansion semantics). -pub fn search_filtered( - graph: &Graph, - query: &Query, - k: usize, - ef: usize, - filter: Option<&[u8]>, - filter_expansion: usize, - scratch: &mut SearchScratch, -) -> (Vec<(u32, f32)>, SearchStats) { - let mut stats = SearchStats { visits: 0 }; - let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { - return (Vec::new(), stats); - }; - scratch.begin_public(graph.file.id_high_water()); - let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); - let budget = if filter.is_some() { (ef * filter_expansion) as u64 } else { u64::MAX }; - let mut out = search_layer(graph, query, ep, ep_dist, ef, 0, scratch, &mut stats, filter, budget); - out.truncate(k); - (out, stats) -} - -/// Pipelined predicate filtering: candidate ids are batched to an external evaluator (the -/// NAPI layer wires this to a JS ThreadsafeFunction) while traversal continues expanding — -/// the search thread never blocks on the evaluator until the beam itself is done. Verdicts -/// steer result admission only; routing uses pure distance order, bounded by the visit -/// budget, so a slow or saturated JS loop degrades speculative overshoot, not correctness. -pub struct PredicatePipe { - /// Sends one batch of ids for evaluation. Must not block. Returns whether the batch was - /// actually handed off: a refused enqueue never produces a verdict, so counting it as - /// outstanding would make the tail drain wait out its whole deadline for an answer that - /// cannot arrive. - pub dispatch: Box) -> bool + Send>, - /// Receives (ids, verdicts) pairs; verdicts[i] != 0 admits ids[i]. - pub rx: std::sync::mpsc::Receiver<(Vec, Vec)>, -} - -const PREDICATE_BATCH: usize = 64; -const DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); - -/// Full search with a pipelined predicate filter (upper-layer descent is unfiltered, as in -/// the JS implementation — predicates gate results, not routing). `visit_budget` is the -/// absolute layer-0 visit cap: hosts pass their own resolved budget directly, since a -/// multiplier-of-ef encoding cannot express a budget below ef. -pub fn search_predicated( - graph: &Graph, - query: &Query, - k: usize, - ef: usize, - pipe: &mut PredicatePipe, - visit_budget: u64, - scratch: &mut SearchScratch, -) -> (Vec<(u32, f32)>, SearchStats) { - let mut stats = SearchStats { visits: 0 }; - let Some((entry_id, entry_level, entry_dist)) = resolve_entry(graph, query, &mut stats) else { - return (Vec::new(), stats); - }; - scratch.begin_public(graph.file.id_high_water()); - let (ep, ep_dist) = greedy_descend(graph, query, entry_id, entry_dist, entry_level, 0, &mut stats); - - use std::collections::HashMap; - let mut verdicts: HashMap = HashMap::new(); - let mut speculative: Vec<(u32, f32)> = Vec::new(); // awaiting verdicts - let mut batch: Vec = Vec::new(); - let mut outstanding = 0usize; - - let mut candidates = BinaryHeap::new(); - let mut results: BinaryHeap = BinaryHeap::new(); - scratch.visit(ep); - candidates.push(Candidate { distance: ep_dist, id: ep }); - speculative.push((ep, ep_dist)); - batch.push(ep); - - let mut nbuf = std::mem::take(&mut scratch.neighbors); - - // guarded on `outstanding` rather than draining until the channel is empty: with a blocking - // receive the unguarded form pays another full timeout after the last verdict lands, on every - // filtered query - macro_rules! drain { - ($recv:expr) => { - while outstanding > 0 { - let Ok((ids, flags)) = $recv else { break }; - outstanding -= 1; - for (i, id) in ids.iter().enumerate() { - verdicts.insert(*id, flags.get(i).copied().unwrap_or(0) != 0); - } - } - }; - } - - loop { - // non-blocking verdict intake each iteration - drain!(pipe.rx.try_recv()); - if !verdicts.is_empty() && !speculative.is_empty() { - speculative.retain(|&(id, d)| match verdicts.get(&id) { - Some(true) => { - results.push(Result_ { distance: d, id }); - if results.len() > ef { - results.pop(); - } - false - } - Some(false) => false, - None => true, - }); - } - - let Some(c) = candidates.pop() else { break }; - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() >= ef && c.distance > worst { - break; - } - if stats.visits >= visit_budget { - break; - } - if graph.neighbors_into(c.id, &mut nbuf).is_none() { - continue; - } - for i in 0..nbuf.len() { - let nid = nbuf[i]; - if (nid as u64) >= graph.file.max_nodes { - continue; // corrupt/torn neighbor id: skip rather than size allocations by it - } - if !scratch.visit(nid) { - continue; - } - if let Some(d) = graph.distance_to(nid, query) { - stats.visits += 1; - let worst = results.peek().map(|r| r.distance).unwrap_or(f32::INFINITY); - if results.len() < ef || d < worst { - candidates.push(Candidate { distance: d, id: nid }); - speculative.push((nid, d)); - batch.push(nid); - if batch.len() >= PREDICATE_BATCH && (pipe.dispatch)(std::mem::take(&mut batch)) { - outstanding += 1; - } - } - } - } - } - scratch.neighbors = nbuf; - - // flush the tail batch and block-drain what's still in flight - if !batch.is_empty() && (pipe.dispatch)(std::mem::take(&mut batch)) { - outstanding += 1; - } - let deadline = std::time::Instant::now() + DRAIN_TIMEOUT; - while outstanding > 0 && std::time::Instant::now() < deadline { - drain!(pipe.rx.recv_timeout(std::time::Duration::from_millis(50))); - } - speculative.retain(|&(id, d)| { - if verdicts.get(&id).copied().unwrap_or(false) { - results.push(Result_ { distance: d, id }); - if results.len() > ef { - results.pop(); - } - } - false - }); - - let mut out: Vec<(u32, f32)> = results.into_iter().map(|r| (r.id, r.distance)).collect(); - out.sort_by(|a, b| a.1.total_cmp(&b.1)); - out.truncate(k); - (out, stats) -} - -#[cfg(test)] -mod predicate_tests { - use super::*; - use crate::insert::{insert, InsertParams}; - use crate::PlaneFile; - - #[test] - fn pipelined_predicate_filters_results() { - let dims = 32; - let path = std::env::temp_dir().join(format!("hnsw-pred-{}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let file = PlaneFile::create(&path, dims, 16, 4_096).expect("create"); - let graph = Graph::new(file); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..1_000u32 { - let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - insert(&graph, &v, ¶ms, &mut scratch).unwrap(); - } - - // evaluator thread: admit even ids only, answering over a channel like the TSFN does - let (req_tx, req_rx) = std::sync::mpsc::channel::>(); - let (res_tx, res_rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); - let worker = std::thread::spawn(move || { - while let Ok(ids) = req_rx.recv() { - let verdicts: Vec = ids.iter().map(|id| (id % 2 == 0) as u8).collect(); - if res_tx.send((ids, verdicts)).is_err() { - break; - } - } - }); - - let mut pipe = PredicatePipe { - dispatch: Box::new(move |ids| req_tx.send(ids).is_ok()), - rx: res_rx, - }; - let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - let (hits, _) = - search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); - assert!(!hits.is_empty()); - for (id, _) in &hits { - assert_eq!(id % 2, 0, "odd id {id} leaked through the predicate"); - } - drop(pipe); - worker.join().unwrap(); - let _ = std::fs::remove_file(&path); - } - - /// The tail drain must stop receiving the moment the last verdict lands. Draining until the - /// channel reports empty sits out another full `recv_timeout` after `outstanding` reaches - /// zero — 50 ms added to every filtered query, against a sub-millisecond search. Measured - /// from the evaluator's last send so the search's own cost is not in the number, and over - /// the best of several queries so scheduler noise on one of them cannot pass for the extra - /// receive, which every query would pay. - #[test] - fn a_predicated_search_returns_as_soon_as_the_last_verdict_lands() { - let dims = 32; - let path = std::env::temp_dir().join(format!("hnsw-preddrain-{}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..1_000u32 { - let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - insert(&graph, &v, ¶ms, &mut scratch).unwrap(); - } - - let (req_tx, req_rx) = std::sync::mpsc::channel::>(); - let (res_tx, res_rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); - // stamped before the send, so the search can never observe a verdict newer than the stamp - let last_send = std::sync::Arc::new(std::sync::Mutex::new(None::)); - let stamps = last_send.clone(); - let worker = std::thread::spawn(move || { - while let Ok(ids) = req_rx.recv() { - let verdicts = vec![1u8; ids.len()]; - *stamps.lock().unwrap() = Some(std::time::Instant::now()); - if res_tx.send((ids, verdicts)).is_err() { - break; - } - } - }); - - let mut pipe = PredicatePipe { - dispatch: Box::new(move |ids| req_tx.send(ids).is_ok()), - rx: res_rx, - }; - let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - let mut best = std::time::Duration::MAX; - for _ in 0..5 { - let (hits, _) = search_predicated( - &graph, - &Query::new(q.clone()), - 10, - 64, - &mut pipe, - 64 * 24, - &mut scratch, - ); - let tail = last_send.lock().unwrap().expect("the evaluator answered a batch").elapsed(); - assert!(!hits.is_empty(), "precondition: an admitting predicate returns results"); - best = best.min(tail); - } - assert!( - best < std::time::Duration::from_millis(25), - "the drain sat {best:?} past the last verdict on every query instead of returning on it" - ); - drop(pipe); - worker.join().unwrap(); - let _ = std::fs::remove_file(&path); - } - - /// A refused enqueue never answers. Counting it outstanding makes the tail drain wait out - /// its whole `DRAIN_TIMEOUT` for a verdict that cannot arrive — which is exactly the state - /// a closing environment puts every in-flight filtered query in, so teardown pays five - /// seconds per query instead of returning on the batches that did land. - #[test] - fn a_refused_predicate_enqueue_does_not_hold_the_drain() { - let dims = 32; - let path = std::env::temp_dir().join(format!("hnsw-refused-{}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..1_000u32 { - let v: Vec = (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - insert(&graph, &v, ¶ms, &mut scratch).unwrap(); - } - - // the sender stays alive for the whole search, so a drain that believes a batch is - // outstanding blocks on the deadline rather than on a disconnected channel - let (tx, rx) = std::sync::mpsc::channel::<(Vec, Vec)>(); - let mut pipe = PredicatePipe { dispatch: Box::new(|_ids| false), rx }; - let q: Vec = (0..dims).map(|d| ((41.0f32 * 0.31 + d as f32) * 0.7).sin()).collect(); - let started = std::time::Instant::now(); - let (hits, _) = - search_predicated(&graph, &Query::new(q), 10, 64, &mut pipe, 64 * 24, &mut scratch); - let elapsed = started.elapsed(); - drop(tx); - assert!(hits.is_empty(), "no verdict can arrive for a refused batch, so nothing may be admitted"); - assert!( - elapsed < std::time::Duration::from_secs(1), - "the search waited {elapsed:?} on batches that were never enqueued (deadline is {DRAIN_TIMEOUT:?})" - ); - let _ = std::fs::remove_file(&path); - } -} diff --git a/native/hnsw-plane/src/seqlock.rs b/native/hnsw-plane/src/seqlock.rs deleted file mode 100644 index 310ec66f1f..0000000000 --- a/native/hnsw-plane/src/seqlock.rs +++ /dev/null @@ -1,217 +0,0 @@ -//! Per-slot lock with owner identity. The lock word is a u32: bit 31 set = locked, low 31 -//! bits = the owner handle's registry tag (see format.rs); unlocked values are generations -//! (bit 31 clear) that change on every release, so readers validate a consistent snapshot -//! seqlock-style. -//! -//! Crash recovery happens at the contended slot: a waiter that has watched the SAME locked -//! value for a full window asks `owner_dead(tag)` — implemented over kernel-owned file -//! locks that die with the owner's open handle, so it is immune to pid reuse, container -//! pid-1 restarts, and pid namespaces. Only a provably dead owner is taken over, and the -//! taker first runs `sanitize` (marking the payload deleted): a dead writer's payload is -//! half-written and must read as absent until rewritten. When liveness is unknowable -//! (non-Linux platforms, an unregistered handle), readers return `fallback()` after the -//! window instead of waiting forever, and writers keep waiting. - -use std::sync::atomic::{AtomicU32, Ordering}; -use std::time::{Duration, Instant}; - -pub const LOCKED: u32 = 1 << 31; -pub const GEN_MASK: u32 = LOCKED - 1; -/// A lock value decomposes as: bit 31 LOCKED | salt(13 bits) | handle identity(19 bits). -/// The identity — registry slot (6 bits) + the handle's per-open epoch (12 bits) — is what -/// liveness is keyed on; the salt changes every acquisition so back-to-back writers from ONE -/// handle still change the observed value (a waiter that never sees an unlocked window must -/// still see progress, or healthy same-handle churn would trip the wedge bound). -pub const TAG_MASK: u32 = (1 << 19) - 1; - -#[inline] -fn acquisition_value(self_tag: u32) -> u32 { - use std::cell::Cell; - use std::sync::atomic::AtomicU32 as GlobalCounter; - // each thread's salt stream starts at a globally unique offset — identical thread-local - // streams across threads could publish identical lock values, making back-to-back - // acquisitions indistinguishable from one long hold - static NEXT_STREAM: GlobalCounter = GlobalCounter::new(1); - thread_local! { - static SALT: Cell = Cell::new(NEXT_STREAM.fetch_add(0x2545_f491, Ordering::Relaxed)); - } - let salt = SALT.with(|c| { - let v = c.get().wrapping_add(1); - c.set(v); - v - }); - LOCKED | (((salt << 19) | (self_tag & TAG_MASK)) & GEN_MASK) -} - -/// How long a locked value must stay unchanged before the owner's liveness is checked. -const TAKEOVER_AFTER: Duration = Duration::from_millis(20); -/// Hard bound on waiting for a lock this thread cannot reclaim (owner alive-or-unknowable: -/// an unregistered handle's abandoned lock, a deadlocked live thread). A live writer's -/// critical section is microseconds, so five seconds of one unchanged locked value means -/// the slot is wedged — surfacing an error beats hanging a caller forever. -const WRITE_WEDGE_AFTER: Duration = Duration::from_secs(5); - -/// The slot's lock could not be acquired or reclaimed within the wedge bound. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct Wedged; -const SPINS_BEFORE_CLOCK: u32 = 1 << 10; - -/// A fresh generation for a takeover release: the previous generation is unknowable, so it -/// must be a value no in-flight reader plausibly holds as its first snapshot. -#[inline] -fn fresh_generation() -> u32 { - let nanos = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.subsec_nanos()) - .unwrap_or(0); - (nanos ^ (std::process::id() << 10)) & GEN_MASK -} - -pub struct SeqWriteGuard<'a> { - seq: &'a AtomicU32, - release_gen: u32, -} - -impl Drop for SeqWriteGuard<'_> { - fn drop(&mut self) { - self.seq.store(self.release_gen, Ordering::Release); - } -} - -enum Stale { - No, - DeadOwner(u32), - UnknownPastWindow, -} - -/// Track how long one locked value has been observed; decide staleness. -struct StaleWatch { - seen: u32, - since: Option, -} - -impl StaleWatch { - fn new() -> Self { - StaleWatch { seen: 0, since: None } - } - - fn observe(&mut self, locked_value: u32, owner_dead: &impl Fn(u32) -> bool) -> Stale { - if self.seen != locked_value || self.since.is_none() { - self.seen = locked_value; - self.since = Some(Instant::now()); - return Stale::No; - } - if self.since.map(|at| at.elapsed() < TAKEOVER_AFTER).unwrap_or(true) { - return Stale::No; - } - if owner_dead(locked_value & GEN_MASK) { - Stale::DeadOwner(locked_value) - } else { - Stale::UnknownPastWindow - } - } -} - -/// Acquire write ownership of a slot. `self_tag` identifies this handle in the lock word; -/// `sanitize` runs (holding the lock) only after a takeover from a dead owner; `owner_dead` -/// decides takeover eligibility. A lock held past the window by an owner that is alive or -/// unknowable is simply waited on. -pub fn write_lock<'a>( - seq: &'a AtomicU32, - self_tag: u32, - sanitize: impl Fn(), - owner_dead: impl Fn(u32) -> bool, -) -> Result, Wedged> { - let mut spins = 0u32; - let mut watch = StaleWatch::new(); - let mut wedged_since: Option = None; - loop { - let cur = seq.load(Ordering::Acquire); - if cur & LOCKED == 0 { - if seq - .compare_exchange_weak(cur, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - return Ok(SeqWriteGuard { seq, release_gen: cur.wrapping_add(1) & GEN_MASK }); - } - wedged_since = None; - } else { - spins += 1; - if spins > SPINS_BEFORE_CLOCK { - match watch.observe(cur, &owner_dead) { - Stale::DeadOwner(observed) => { - if seq - .compare_exchange(observed, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - sanitize(); - return Ok(SeqWriteGuard { seq, release_gen: fresh_generation() }); - } - wedged_since = None; - } - Stale::UnknownPastWindow => { - // unreclaimable (unregistered owner tag, or an alive-but-stuck - // holder): bounded wait, then surface the wedge instead of hanging - let since = *wedged_since.get_or_insert_with(Instant::now); - if since.elapsed() > WRITE_WEDGE_AFTER { - return Err(Wedged); - } - } - // the lock VALUE moved: owners are cycling, i.e. real progress — a busy - // slot must never trip the wedge bound - Stale::No => wedged_since = None, - } - std::thread::yield_now(); - continue; - } - } - std::hint::spin_loop(); - } -} - -/// Run `read` until it observes a stable (unlocked, unchanged) generation. `read` must be -/// side-effect-free on retry. A dead owner's lock is taken over (sanitizing the payload) -/// and the read retried; an alive-or-unknowable owner past the window makes this return -/// `fallback()` rather than stall a search indefinitely. -#[inline] -pub fn read_consistent( - seq: &AtomicU32, - self_tag: u32, - mut read: impl FnMut() -> T, - sanitize: impl Fn(), - fallback: impl FnOnce() -> T, - owner_dead: impl Fn(u32) -> bool, -) -> T { - let mut spins = 0u32; - let mut watch = StaleWatch::new(); - loop { - let before = seq.load(Ordering::Acquire); - if before & LOCKED == 0 { - let value = read(); - std::sync::atomic::fence(Ordering::Acquire); - if seq.load(Ordering::Relaxed) == before { - return value; - } - } else { - spins += 1; - if spins > SPINS_BEFORE_CLOCK { - match watch.observe(before, &owner_dead) { - Stale::DeadOwner(observed) => { - if seq - .compare_exchange(observed, acquisition_value(self_tag), Ordering::AcqRel, Ordering::Acquire) - .is_ok() - { - sanitize(); - seq.store(fresh_generation(), Ordering::Release); - } - } - Stale::UnknownPastWindow => return fallback(), - Stale::No => {} - } - std::thread::yield_now(); - continue; - } - } - std::hint::spin_loop(); - } -} diff --git a/native/hnsw-plane/tests/concurrent.rs b/native/hnsw-plane/tests/concurrent.rs deleted file mode 100644 index e3699e2925..0000000000 --- a/native/hnsw-plane/tests/concurrent.rs +++ /dev/null @@ -1,174 +0,0 @@ -//! Concurrent-write torture: writers insert while readers search; then verify the graph is -//! coherent (every stored vector findable, edge lists within cap, freelist reuse works). - -use hnsw_plane::distance::Query; -use hnsw_plane::insert::{insert, InsertParams}; -use hnsw_plane::search::{search, SearchScratch}; -use hnsw_plane::{Graph, PlaneFile}; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Barrier}; - -fn vector_for(i: u32, dims: usize) -> Vec { - // deterministic distinct unit-ish vectors on a few clusters - let mut v = vec![0.0f32; dims]; - let cluster = (i % 7) as usize; - for d in 0..dims { - let x = ((i as f32 * 0.37 + d as f32 * 1.13).sin() * 0.1) + if d % 7 == cluster { 1.0 } else { 0.0 }; - v[d] = x; - } - // Per-node signature (unique for i < dims^3). The cluster spike plus 0.1-amplitude noise - // alone leaves every member of a cluster inside int8 quantization noise of every other, so - // a self-query cannot tell "found this node" from "found some other node" — and a - // distance-only assertion over such a corpus passes even when the node is orphaned. - v[(i as usize) % dims] += 0.5; - v[(i as usize / dims) % dims] += 0.35; - v[(i as usize / (dims * dims)) % dims] += 0.22; - v -} - -#[test] -fn concurrent_insert_search() { - let dims = 64; - let path = std::env::temp_dir().join(format!("hnsw-torture-{}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let file = PlaneFile::create(&path, dims, 32, 40_000).expect("create"); - let graph = Arc::new(Graph::new(file)); - - let writers = 4u32; - let per_writer = 2_000u32; - let done = Arc::new(AtomicBool::new(false)); - - // (corpus index, node id): ids come from the plane's own allocator, so writers interleave - // them — a self-query must be checked against the id its insert actually returned - let inserted: Vec<(u32, u32)> = std::thread::scope(|s| { - let writers_done: Vec<_> = (0..writers) - .map(|w| { - let graph = graph.clone(); - s.spawn(move || { - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - let mut mine = Vec::with_capacity(per_writer as usize); - for i in 0..per_writer { - let index = w * per_writer + i; - let v = vector_for(index, dims); - mine.push((index, insert(&graph, &v, ¶ms, &mut scratch).expect("insert"))); - } - mine - }) - }) - .collect(); - for _ in 0..4 { - let graph = graph.clone(); - let done = done.clone(); - s.spawn(move || { - let mut scratch = SearchScratch::new(); - let mut q = 0u32; - while !done.load(Ordering::Relaxed) { - let query = Query::new(vector_for(q % 1000, dims)); - let (results, _) = search(&graph, &query, 10, 64, &mut scratch); - // once anything is inserted, results must be non-empty and finite - for (_, d) in &results { - assert!(d.is_finite()); - } - q += 1; - } - }); - } - // scope joins writers when their closures end; signal readers afterward via a - // dedicated waiter thread - let graph_ref = graph.clone(); - let done_ref = done.clone(); - s.spawn(move || { - while graph_ref.file.id_high_water() < (writers * per_writer) as u64 { - std::thread::yield_now(); - } - done_ref.store(true, Ordering::Relaxed); - }); - writers_done.into_iter().flat_map(|h| h.join().expect("writer panicked")).collect() - }); - - let total = writers * per_writer; - assert_eq!(graph.file.id_high_water(), total as u64); - - // Every stored vector must be found as its own nearest neighbor at generous ef. - let mut scratch = SearchScratch::new(); - let mut misses = 0; - for &(index, id) in inserted.iter().step_by(97) { - let query = Query::new(vector_for(index, dims)); - let (results, _) = search(&graph, &query, 10, 256, &mut scratch); - // by ID, not by distance: this corpus is clustered near-duplicates, so a hit at - // distance ~0 is routinely a DIFFERENT node and would mask an orphaned one - if !results.iter().any(|&(rid, _)| rid == id) { - misses += 1; - } - } - assert_eq!(misses, 0, "self-queries missing after concurrent build"); - - // Edge lists respect the cap. - for id in (0..total).step_by(53) { - if let Some(n) = graph.read_node(id) { - assert!(n.neighbors.len() <= graph.file.layer0_cap); - } - } - - // Delete + reinsert reuses ids (freelist; the #2182 fix). - let _ = graph.delete_node(5); - let _ = graph.delete_node(6); - let params = InsertParams::default(); - let a = insert(&graph, &vector_for(90_001, dims), ¶ms, &mut scratch).unwrap(); - let b = insert(&graph, &vector_for(90_002, dims), ¶ms, &mut scratch).unwrap(); - assert!(a == 5 || a == 6, "expected freelist reuse, got {a}"); - assert!(b == 5 || b == 6, "expected freelist reuse, got {b}"); - assert_eq!(graph.file.id_high_water(), total as u64, "high-water must not grow on reuse"); - - let _ = std::fs::remove_file(&path); -} - -/// Orthogonal per-writer vector: every writer's self-query has exactly one right answer, so a -/// node that lost the first-entry race is unmissable rather than covered by a near-duplicate. -fn axis_vector(writer: u32, dims: usize) -> Vec { - let mut v = vec![0.0f32; dims]; - v[writer as usize % dims] = 1.0; - v -} - -/// Many small fresh graphs, each racing its FIRST insert: that window is where the empty-graph -/// entry-point claim races, and a single barrier in a long build samples it about once. -#[test] -fn racing_first_inserts_all_stay_reachable() { - let dims = 32; - let writers = 4u32; - let rounds = 200; - for round in 0..rounds { - let path = std::env::temp_dir().join(format!("hnsw-first-{}-{round}.hnsw", std::process::id())); - let _ = std::fs::remove_file(&path); - let graph = Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create"))); - let barrier = Arc::new(Barrier::new(writers as usize)); - let ids: Vec<(u32, u32)> = std::thread::scope(|s| { - let handles: Vec<_> = (0..writers) - .map(|w| { - let graph = graph.clone(); - let barrier = barrier.clone(); - s.spawn(move || { - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - barrier.wait(); - (w, insert(&graph, &axis_vector(w, dims), ¶ms, &mut scratch).expect("insert")) - }) - }) - .collect(); - handles.into_iter().map(|h| h.join().expect("writer panicked")).collect() - }); - - let mut scratch = SearchScratch::new(); - for (w, id) in &ids { - let (results, _) = search(&graph, &Query::new(axis_vector(*w, dims)), 8, 64, &mut scratch); - assert!( - results.iter().any(|&(rid, _)| rid == *id), - "round {round}: writer {w}'s node {id} is unreachable from the entry point (found {results:?})" - ); - } - drop(graph); - let _ = std::fs::remove_file(&path); - } -} diff --git a/native/hnsw-plane/tests/reopen.rs b/native/hnsw-plane/tests/reopen.rs deleted file mode 100644 index 7234ffc077..0000000000 --- a/native/hnsw-plane/tests/reopen.rs +++ /dev/null @@ -1,755 +0,0 @@ -//! Crash-window and lifecycle coverage: torn-seqlock scrub on unclean reopen, entry-point -//! deletion recovery, truncated-file rejection, and full-plane behavior. These are the paths -//! a test that never crashes cannot verify. - -use hnsw_plane::distance::Query; -use hnsw_plane::insert::{insert, InsertParams}; -use hnsw_plane::search::{search, SearchScratch}; -use hnsw_plane::{Graph, PlaneFile}; -use std::sync::atomic::Ordering; - -fn vector_for(i: u32, dims: usize) -> Vec { - (0..dims).map(|d| ((i as f32 * 0.31 + d as f32) * 0.7).sin()).collect() -} - -fn await_lock(held: &std::sync::atomic::AtomicBool) { - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); - while !held.load(std::sync::atomic::Ordering::Acquire) { - assert!(std::time::Instant::now() < deadline, "holder thread never acquired the lock"); - std::hint::spin_loop(); - } -} - -fn tmp(name: &str) -> std::path::PathBuf { - std::env::temp_dir().join(format!("hnsw-{name}-{}.hnsw", std::process::id())) -} - -#[test] -fn dead_writer_lock_is_taken_over_and_slot_sanitized() { - let dims = 32; - let path = tmp("torn"); - let _ = std::fs::remove_file(&path); - { - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..200 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - // simulate a writer killed mid-write: lock word = bit31 | a tag whose registry slot - // carries no matching registration (the fabricated tag differs from any live tag, - // so tag_is_dead reports it dead immediately) - graph.file.seq_atomic(7).store((1 << 31) | 0x1234_5678 & 0x7fff_ffff, Ordering::SeqCst); - graph.file.msync().unwrap(); - } - let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); - // the first reader waits out the takeover window, confirms the owner is dead, takes the - // lock over, and SANITIZES the slot: a dead writer's payload is half-written, so the - // node must read as absent (heal-on-touch), never as a spliced-but-valid vector - let start = std::time::Instant::now(); - assert!(graph.read_node(7).is_none(), "taken-over slot must read absent, not spliced"); - assert!(start.elapsed() < std::time::Duration::from_secs(5), "takeover must be fast"); - assert_eq!(graph.file.seq_atomic(7).load(Ordering::SeqCst) >> 31, 0, "takeover unlocks the slot"); - // the graph still searches (node 7 is just missing), and rewriting the slot heals it - let mut scratch = SearchScratch::new(); - let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty()); - let q = hnsw_plane::distance::quantize_int8(&vector_for(7, dims)); - graph.write_node_raw(7, 0, &q.0, q.1, q.2, &[3, 4], &[]).unwrap(); - assert!(graph.read_node(7).is_some(), "a rewrite heals the sanitized slot"); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn live_writer_is_never_robbed() { - let dims = 32; - let path = tmp("liverob"); - let _ = std::fs::remove_file(&path); - let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create"))); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..50 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - // a LIVE writer (this process) holds slot 9's lock far past the takeover window; readers - // must wait or degrade to absent — never force the lock and never observe torn payload - let seq9 = graph.file.seq_atomic(9) as *const _ as usize; - let g2 = graph.clone(); - let hold = std::thread::spawn(move || { - let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; - let g2 = &g2; - let guard = hnsw_plane::seqlock::write_lock( - seq, - g2.file.self_tag, - || panic!("a live same-process writer must never be sanitized"), - |tag| g2.file.tag_is_dead(tag), - ); - std::thread::sleep(std::time::Duration::from_millis(120)); - drop(guard); - }); - std::thread::sleep(std::time::Duration::from_millis(30)); // reader arrives mid-hold - let n9 = graph.read_node(9); - // either it waited for the release (Some) or degraded to absent for this read (None) — - // but the lock must have been RELEASED by the owner, not forced - hold.join().unwrap(); - assert!(graph.read_node(9).is_some(), "the slot is intact after the live writer releases"); - let _ = n9; - let _ = std::fs::remove_file(&path); -} - -#[test] -fn double_remove_does_not_cycle_the_freelist() { - let dims = 32; - let path = tmp("dblrm"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..20 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let _ = graph.delete_node(5); - let _ = graph.delete_node(5); // second delete must be a no-op, not a second freelist push - let _ = graph.delete_node(2_000_000); // out-of-range must be a no-op, not an OOB write - let a = insert(&graph, &vector_for(101, dims), ¶ms, &mut scratch).unwrap(); - let b = insert(&graph, &vector_for(102, dims), ¶ms, &mut scratch).unwrap(); - let c = insert(&graph, &vector_for(103, dims), ¶ms, &mut scratch).unwrap(); - assert_eq!(a, 5, "freed id is reused once"); - assert_ne!(b, a, "a double-freed id must not be handed out twice"); - assert_ne!(c, b); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn deleting_the_entry_point_reelects_and_recovers() { - let dims = 32; - let path = tmp("entrydel"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..100 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let (entry, _) = graph.file.entry_point(); - let _ = graph.delete_node(entry); - let (new_entry, _) = graph.file.entry_point(); - assert_ne!(new_entry, entry, "a new entry point must be elected"); - let (hits, _) = search(&graph, &Query::new(vector_for(3, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty(), "search must survive entry-point deletion"); - // subsequent inserts must not orphan themselves against the dead entry - let id = insert(&graph, &vector_for(500, dims), ¶ms, &mut scratch).unwrap(); - let (hits, _) = search(&graph, &Query::new(vector_for(500, dims)), 5, 128, &mut scratch); - assert!(hits.iter().any(|&(hid, d)| hid == id && d < 1e-3), "post-deletion insert must be reachable"); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn truncated_file_is_a_catchable_error() { - let path = tmp("trunc"); - std::fs::write(&path, vec![0u8; 100]).unwrap(); - assert!(PlaneFile::open(&path).is_err(), "a 100-byte file must be rejected, not panic"); - // header-valid but body-truncated: create a real plane, then cut it short - let path2 = tmp("trunc2"); - let _ = std::fs::remove_file(&path2); - { - let graph = Graph::new(PlaneFile::create(&path2, 32, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - insert(&graph, &vector_for(1, 32), ¶ms, &mut scratch).unwrap(); - } - let full = std::fs::metadata(&path2).unwrap().len(); - let f = std::fs::OpenOptions::new().write(true).open(&path2).unwrap(); - f.set_len(full / 2).unwrap(); - drop(f); - assert!(PlaneFile::open(&path2).is_err(), "a body-truncated file must be rejected, not read off the map"); - let _ = std::fs::remove_file(&path); - let _ = std::fs::remove_file(&path2); -} - -#[test] -fn full_plane_refuses_inserts_instead_of_corrupting() { - let dims = 32; - let path = tmp("full"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 8).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..8 { - assert!(insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).is_ok()); - } - assert!(insert(&graph, &vector_for(9, dims), ¶ms, &mut scratch).is_err(), "insert past maxNodes must fail cleanly"); - // freed capacity is usable again - let _ = graph.delete_node(3); - assert!(insert(&graph, &vector_for(10, dims), ¶ms, &mut scratch).is_ok()); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn flush_without_watermark_preserves_a_completion_stamp() { - let dims = 32; - let path = tmp("flushnone"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create")); - graph.file.set_watermark(7); - graph.file.flush_with_watermark(None).unwrap(); - assert_eq!(graph.file.watermark(), 7, "a watermark-less barrier must not touch the stamp"); - graph.file.flush_with_watermark(Some(9)).unwrap(); - assert_eq!(graph.file.watermark(), 9); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn odd_dims_freelist_reuse_is_aligned() { - // dims 25: the old freelist next-pointer at S_VECTOR+dims was unaligned (SIGBUS on - // aarch64); it now lives at the dead slot's aligned scale field - let dims = 25; - let path = tmp("odddims"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 256).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..20 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let _ = graph.delete_node(4); - let _ = graph.delete_node(9); - let a = insert(&graph, &vector_for(50, dims), ¶ms, &mut scratch).unwrap(); - let b = insert(&graph, &vector_for(51, dims), ¶ms, &mut scratch).unwrap(); - assert!(a == 9 || a == 4); - assert!(b == 9 || b == 4); - assert_ne!(a, b); - let _ = std::fs::remove_file(&path); -} - -#[cfg(target_os = "linux")] -#[test] -fn same_pid_restart_takeover_via_registry() { - // The container-pid-1 scenario: the process that died and the process that reopens have - // the SAME pid, so pid-based liveness would wedge forever. Registry liveness is keyed to - // the open handle (kernel lock dies with it), which a same-process reopen reproduces - // faithfully: drop the old handle, reopen, and the old tag must be reclaimable. - let dims = 32; - let path = tmp("samepid"); - let _ = std::fs::remove_file(&path); - let dead_tag; - { - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..100 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - dead_tag = graph.file.self_tag; - assert_ne!(dead_tag, 0, "linux handles must register"); - // die mid-write: lock word carries OUR tag, then the handle drops (kernel releases - // the registry lock exactly as process death would) - graph.file.seq_atomic(11).store((1 << 31) | dead_tag, Ordering::SeqCst); - graph.file.msync().unwrap(); - } - let graph = Graph::new(PlaneFile::open(&path).expect("reopen")); - assert_ne!(graph.file.self_tag, dead_tag, "a new handle mints a new tag"); - let start = std::time::Instant::now(); - assert!(graph.read_node(11).is_none(), "taken-over slot reads deleted (sanitized), not spliced"); - assert!(start.elapsed() < std::time::Duration::from_secs(5)); - assert_eq!(graph.file.seq_atomic(11).load(Ordering::SeqCst) >> 31, 0, "lock reclaimed"); - let _ = std::fs::remove_file(&path); -} - -#[test] -fn tombstoned_virgin_slot_does_not_alias_upper_entry_zero() { - let dims = 32; - let path = tmp("virgintomb"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - // make node 0's insert claim upper entry 0 (first level>=1 node allocates it); insert - // until some node has an upper entry - let mut upper_owner = None; - for i in 0..64 { - let id = insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - if graph.read_node(id).map(|n| n.level > 0).unwrap_or(false) { - upper_owner = Some(id); - break; - } - } - let upper_owner = upper_owner.expect("some node should have an upper level"); - let mut before = Vec::new(); - assert!(graph.upper_neighbors_into(upper_owner, 1, &mut before) || before.is_empty()); - - // tombstone a NEVER-written id (beyond anything inserted), then raw-write it with an - // upper list: it must allocate a fresh entry, not adopt the zero-initialized index 0 - let virgin = 900; - graph.clear_node(virgin).unwrap(); - let q = hnsw_plane::distance::quantize_int8(&vector_for(virgin, dims)); - graph - .write_node_raw(virgin, 1, &q.0, q.1, q.2, &[1, 2], &[vec![1, 2]]) - .unwrap(); - let mut after = Vec::new(); - let _ = graph.upper_neighbors_into(upper_owner, 1, &mut after); - assert_eq!(before, after, "raw-writing a tombstoned virgin slot must not clobber another node's upper entry"); - let mut virgin_upper = Vec::new(); - assert!(graph.upper_neighbors_into(virgin, 1, &mut virgin_upper)); - assert_eq!(virgin_upper, vec![1, 2]); - let _ = std::fs::remove_file(&path); -} - -/// The host's id counter reseeds to largestNodeId + 1 across a restart, so deleting the top -/// ids hands them back out and the new record redraws its level — often 0. The re-minted slot -/// must stop reading its predecessor's upper adjacency, and cycling a hot id through levels -/// must not consume a fresh entry each time. -#[test] -fn raw_rewrite_at_level_zero_clears_the_stale_hierarchy() { - let dims = 32; - let path = tmp("upperstale"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create")); - let q = hnsw_plane::distance::quantize_int8(&vector_for(9, dims)); - let write = |level: u8, upper: &[Vec]| { - graph.write_node_raw(9, level, &q.0, q.1, q.2, &[1, 2], upper).unwrap(); - }; - let mut nbrs = Vec::new(); - - write(1, &[vec![7]]); - assert!(graph.upper_neighbors_into(9, 1, &mut nbrs) && nbrs == vec![7]); - write(0, &[]); - assert!(!graph.upper_neighbors_into(9, 1, &mut nbrs), "a level-0 rewrite must not leave the old hierarchy readable"); - - // cycling the same id through level 0 and back must reuse its entry, not mint one per pass - for n in 0..graph.file.upper_capacity as u32 + 4 { - write(1, &[vec![n % 8]]); - write(0, &[]); - } - write(1, &[vec![5]]); - assert!( - graph.upper_neighbors_into(9, 1, &mut nbrs), - "upper region exhausted: level cycling minted a new entry per pass" - ); - assert_eq!(nbrs, vec![5]); - let _ = std::fs::remove_file(&path); -} - -/// A peer worker holding the slot lock past the 20 ms stale window makes the lock-free upper -/// read give up and report NO_UPPER. Treating that "cannot tell" as "nothing bound" mints a -/// second entry per contended mirror and orphans the first, so a hot node burns the fixed -/// upper region until level>=1 mirrors stop binding at all. -#[test] -fn contended_raw_rewrite_does_not_mint_a_second_upper_entry() { - use std::sync::atomic::{AtomicBool, Ordering as O}; - let dims = 32; - let path = tmp("uppercontend"); - let _ = std::fs::remove_file(&path); - let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); - let q = hnsw_plane::distance::quantize_int8(&vector_for(9, dims)); - graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![7]]).unwrap(); - - let seq9 = graph.file.seq_atomic(9) as *const _ as usize; - for n in 0..graph.file.upper_capacity as u32 + 4 { - let g2 = graph.clone(); - let held = std::sync::Arc::new(AtomicBool::new(false)); - let held2 = held.clone(); - let hold = std::thread::spawn(move || { - let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; - let g2 = &g2; - let guard = - hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { - g2.file.tag_is_dead(tag) - }) - .expect("the holder must actually take the lock, or the test proves nothing"); - held2.store(true, O::Release); - std::thread::sleep(std::time::Duration::from_millis(30)); // past TAKEOVER_AFTER - drop(guard); - }); - await_lock(&held); - graph.write_node_raw(9, 1, &q.0, q.1, q.2, &[1, 2], &[vec![n % 8]]).unwrap(); - hold.join().unwrap(); - } - - let mut nbrs = Vec::new(); - assert!( - graph.upper_neighbors_into(9, 1, &mut nbrs), - "upper region exhausted: each contended rewrite minted and orphaned an entry" - ); - let _ = std::fs::remove_file(&path); -} - -/// Nothing references a freshly allocated upper entry until its write lands, so a path that -/// gives up on a wedged slot lock without freeing strands it outside both the freelist and the -/// graph. -#[test] -fn a_wedged_untouched_write_frees_its_upper_entry() { - use std::sync::atomic::Ordering as O; - let dims = 32; - let path = tmp("wedgeuntouched"); - let _ = std::fs::remove_file(&path); - let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); - let write = |id: u32| { - let q = hnsw_plane::distance::quantize_int8(&vector_for(id, dims)); - graph.write_node_if_untouched(id, 1, &q.0, q.1, q.2, &[1, 2], &[vec![id]]) - }; - - assert_eq!(write(1), Ok(true)); - let baseline = graph.file.upper_high_water(); - - let seq9 = graph.file.seq_atomic(9) as *const _ as usize; - let g2 = graph.clone(); - let held = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); - let held2 = held.clone(); - let hold = std::thread::spawn(move || { - let seq = unsafe { &*(seq9 as *const std::sync::atomic::AtomicU32) }; - let g2 = &g2; - let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { - g2.file.tag_is_dead(tag) - }) - .expect("the holder must actually take the lock, or the test proves nothing"); - held2.store(true, O::Release); - std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER - drop(guard); - }); - await_lock(&held); - assert_eq!(write(9), Err(hnsw_plane::seqlock::Wedged), "the held lock must wedge this write"); - hold.join().unwrap(); - - assert_eq!(write(2), Ok(true)); - assert_eq!( - graph.file.upper_high_water(), - baseline + 1, - "the wedged write leaked its upper entry instead of returning it to the freelist" - ); - let _ = std::fs::remove_file(&path); -} - -/// A wedged upper-entry cleanup must not leave the header naming a deleted entry point: the -/// cleanup is fallible, so an early return there strands every search on a dead entry. Asserts -/// the observable (searches still return hits), not the header word. -#[test] -fn a_wedged_upper_cleanup_still_reelects_the_entry_point() { - use std::sync::atomic::Ordering as O; - let dims = 32; - let path = tmp("wedgedelete"); - let _ = std::fs::remove_file(&path); - let graph = std::sync::Arc::new(Graph::new(PlaneFile::create(&path, dims, 16, 64).expect("create"))); - let raw = |id: u32, level: u8, neighbors: &[u32], upper: &[Vec]| { - let q = hnsw_plane::distance::quantize_int8(&vector_for(id, dims)); - graph.write_node_raw(id, level, &q.0, q.1, q.2, neighbors, upper).expect("mirror"); - }; - // node 0 is the entry point and the only node with a hierarchy, so it owns upper entry 0 - raw(0, 1, &[1], &[vec![1]]); - raw(1, 0, &[0], &[]); - graph.file.set_entry_point(0, 1); - - let upper_seq = graph.file.upper_seq_atomic(0) as *const _ as usize; - let g2 = graph.clone(); - let held = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); - let held2 = held.clone(); - let hold = std::thread::spawn(move || { - let seq = unsafe { &*(upper_seq as *const std::sync::atomic::AtomicU32) }; - let g2 = &g2; - let guard = hnsw_plane::seqlock::write_lock(seq, g2.file.self_tag, || panic!("live owner sanitized"), |tag| { - g2.file.tag_is_dead(tag) - }) - .expect("the holder must actually take the lock, or the test proves nothing"); - held2.store(true, O::Release); - std::thread::sleep(std::time::Duration::from_millis(6_500)); // past WRITE_WEDGE_AFTER - drop(guard); - }); - await_lock(&held); - assert_eq!(graph.delete_node(0), Err(hnsw_plane::seqlock::Wedged), "the held upper lock must wedge the cleanup"); - hold.join().unwrap(); - - assert_eq!(graph.file.entry_point().0, 1, "the entry point must be re-elected before the fallible cleanup"); - let mut scratch = SearchScratch::new(); - let (hits, _) = search(&graph, &Query::new(vector_for(1, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty(), "searches must keep working after a wedged delete of the entry point"); - let _ = std::fs::remove_file(&path); -} - -/// A search must repair an entry point that no writer will: a host that cleared the entry, or -/// a slot a reader sanitized after its writer died, leaves no delete to run the write-path -/// re-election, so on a read-mostly table every search returns empty indefinitely. -#[test] -fn search_repairs_an_entry_point_no_writer_will() { - let dims = 32; - let path = tmp("entryheal"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..200 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let prev = graph.file.previous_entry_point(); - assert_ne!(prev, hnsw_plane::format::NO_ID, "promotions must record a previous-entry hint to repair from"); - - // the entry's slot reads as gone with no delete having run (dead-writer sanitization, or a - // mirroring host clearing the node) — nothing on the write path will ever re-elect - let (entry, _) = graph.file.entry_point(); - graph.clear_node(entry).expect("tombstone the entry slot"); - - let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty(), "a search must self-heal past a dead entry point instead of returning empty"); - assert_ne!(graph.file.entry_point().0, entry, "the repair must be published, not repeated per search"); - let _ = std::fs::remove_file(&path); -} - -/// `invalidate` demotes a plane that already looks like a complete mirror back to "incomplete, -/// rebuild me", and reports barrier failure to its caller instead of into a dropped promise — -/// which is what lets the host order it before creating a `.stale` sidecar. (Durability itself -/// is not observable in-process: the mapping is MAP_SHARED, so every store is already visible to -/// a reopen and to `read()` whether or not the msync ran. The ordering that a crash would expose -/// is asserted on the host side, in `vectorIndexPlane.test.js`.) -#[test] -fn invalidate_demotes_a_complete_looking_mirror_and_reports_failure() { - let dims = 32; - let path = tmp("invalidate"); - let _ = std::fs::remove_file(&path); - { - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..50 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - graph.file.flush_with_watermark(Some(4_096)).expect("barrier"); - assert_eq!(graph.file.watermark(), 4_096, "precondition: a complete-looking mirror"); - graph.file.invalidate().expect("invalidate must report its barrier, not swallow it"); - assert_eq!(graph.file.watermark(), 0, "invalidation must mark the mirror incomplete in band"); - } - let reopened = PlaneFile::open(&path).expect("reopen"); - assert_eq!(reopened.watermark(), 0, "a fresh opener must see the incomplete mark, not the old stamp"); - let _ = std::fs::remove_file(&path); -} - -/// The hint is one slot and can die too: promote over a node, then lose BOTH that node and the -/// entry it was promoted over. Without the bounded probe the repair has nowhere left to look and -/// every later search returns empty although most of the graph is live. -#[test] -fn search_repairs_an_entry_point_whose_hint_is_dead_too() { - let dims = 32; - let path = tmp("entryhealdeadhint"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..200 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let hint = graph.file.previous_entry_point(); - assert_ne!(hint, hnsw_plane::format::NO_ID, "precondition: a hint to invalidate"); - let (entry, _) = graph.file.entry_point(); - - // both sanitized with no delete having run, so no write-path re-election ever happens and - // the hint the repair would follow names a node that reads as gone - graph.clear_node(hint).expect("tombstone the hint slot"); - graph.clear_node(entry).expect("tombstone the entry slot"); - - let (hits, _) = search(&graph, &Query::new(vector_for(7, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty(), "a dead hint must fall back to the bounded probe, not return empty forever"); - let repaired = graph.file.entry_point().0; - assert_ne!(repaired, entry, "the repair must be published"); - assert_ne!(repaired, hint, "the repair must not publish the dead hint"); - let _ = std::fs::remove_file(&path); -} - -/// Harper allocates node ids monotonically and never reuses them, so a table that has churned -/// has its whole low prefix tombstoned and only the newest ids live. A repair that probed a -/// fixed prefix would find nothing there and every search would return empty forever. -#[test] -fn search_repairs_an_entry_point_in_a_churned_graph_whose_low_ids_are_all_dead() { - let dims = 32; - let path = tmp("entryhealchurn"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..1_200 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - // every id a prefix probe would reach is gone, as it is for any long-lived churned table - for id in 0..1_100u32 { - let _ = graph.clear_node(id); - } - let (entry, _) = graph.file.entry_point(); - let hint = graph.file.previous_entry_point(); - let _ = graph.clear_node(entry); - if hint != hnsw_plane::format::NO_ID { - let _ = graph.clear_node(hint); - } - - let (hits, _) = search(&graph, &Query::new(vector_for(1_150, dims)), 5, 64, &mut scratch); - assert!(!hits.is_empty(), "the probe must reach the live tail, not only a dead low prefix"); - let repaired = graph.file.entry_point().0; - assert!(graph.read_node(repaired).is_some(), "the repair must publish a live node"); - let _ = std::fs::remove_file(&path); -} - -/// With a stride above 1 a fixed start probes one residue class forever, so a live graph lying -/// entirely between its probes would never be found. The rotation makes `stride` consecutive -/// repairs cover every id; here the sole survivor is deliberately in the residue the unrotated -/// walk skips. -#[test] -fn a_repair_probe_rotates_so_no_live_node_stays_between_its_samples() { - let dims = 32; - let path = tmp("entryhealrotate"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..2_100 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let hw = graph.file.id_high_water() as u32; - let stride = hw.div_ceil(1_024); // REPAIR_PROBE_LIMIT - assert!(stride > 1, "precondition: a stride the rotation actually has to cover, got {stride}"); - // an unrotated walk starts at hw-1 and steps by `stride`, so it only ever sees that residue; - // keep exactly one node alive in a different one - let survivor = (0..hw).rev().find(|id| (hw - 1 - id) % stride != 0).expect("a skipped residue"); - for id in 0..hw { - if id != survivor { - let _ = graph.clear_node(id); - } - } - assert!(graph.read_node(survivor).is_some(), "precondition: the survivor is live"); - - let mut found = false; - for _ in 0..stride { - let (hits, _) = search(&graph, &Query::new(vector_for(survivor, dims)), 5, 64, &mut scratch); - if !hits.is_empty() { - found = true; - break; - } - } - assert!(found, "a rotating probe must reach every residue within `stride` repairs"); - assert_eq!(graph.file.entry_point().0, survivor, "the only live node must be the repaired entry"); - let _ = std::fs::remove_file(&path); -} - -/// The stride must be a ceiling division. Flooring it leaves `stride * limit < hw` whenever `hw` -/// is not a multiple of `limit`, so every rotated walk stops above the lowest `hw % limit` ids — -/// a permanent blind spot, not a one-search one, since no offset ever reaches it. A graph whose -/// only survivors sit in that prefix would return empty from every later search; this one's does. -#[test] -fn a_repair_probe_reaches_the_low_ids_a_floored_stride_would_never_sample() { - let dims = 32; - let limit = 1_024u32; // REPAIR_PROBE_LIMIT - let path = tmp("entryheallowprefix"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..2_100 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let hw = graph.file.id_high_water() as u32; - // a floored walk bottoms out at `hw % limit` whatever its rotation offset, so ids below that - // are exactly what the ceiling buys - let floored_reach = hw % limit; - assert!(hw > limit && floored_reach > 1, "precondition: a low prefix a floored stride skips, hw {hw}"); - let survivor = floored_reach / 2; - for id in 0..hw { - if id != survivor { - let _ = graph.clear_node(id); - } - } - assert!(graph.read_node(survivor).is_some(), "precondition: the survivor is live"); - assert_ne!(graph.file.previous_entry_point(), survivor, "precondition: the probe must be what finds it"); - - let mut found = false; - for _ in 0..hw.div_ceil(limit) { - let (hits, _) = search(&graph, &Query::new(vector_for(survivor, dims)), 5, 64, &mut scratch); - if !hits.is_empty() { - found = true; - break; - } - } - assert!(found, "a full rotation must cover every id, the lowest included"); - assert_eq!(graph.file.entry_point().0, survivor, "the only live node must be the repaired entry"); - let _ = std::fs::remove_file(&path); -} - -/// Rotation has to be per handle. With one process-wide counter, every other plane's repairs -/// advance it too, so two planes repairing in turn each see offsets stepping by 2 — one residue -/// class apiece, indefinitely, which is exactly what rotating was meant to prevent. Both planes -/// here hide their survivor in the same residue, so a shared counter must strand one of them -/// whichever offset it starts on. -#[test] -fn repair_probe_rotation_is_per_plane_not_per_process() { - let dims = 32; - let mut graphs = Vec::new(); - let mut survivors = Vec::new(); - let mut stride = 0u32; - for which in 0..2 { - let path = tmp(&format!("entryhealperplane{which}")); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 4_096).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..2_100 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let hw = graph.file.id_high_water() as u32; - stride = hw.div_ceil(1_024); - assert!(stride > 1, "precondition: a stride the rotation has to cover"); - // the same skipped residue on both planes, so a shared counter cannot serve both - let survivor = (0..hw).rev().find(|id| (hw - 1 - id) % stride != 0).expect("a skipped residue"); - for id in 0..hw { - if id != survivor { - let _ = graph.clear_node(id); - } - } - graphs.push((graph, path)); - survivors.push(survivor); - } - - let mut scratch = SearchScratch::new(); - let mut found = [false; 2]; - for _ in 0..stride { - for (which, (graph, _)) in graphs.iter().enumerate() { - let (hits, _) = search(graph, &Query::new(vector_for(survivors[which], dims)), 5, 64, &mut scratch); - if !hits.is_empty() { - found[which] = true; - } - } - } - assert!(found[0] && found[1], "each plane must cover its own residues: {found:?}"); - for (_, path) in &graphs { - let _ = std::fs::remove_file(path); - } -} - -/// A repair publishes with a strict CAS on the entry it observed dead. A first insert that -/// claims the header in between owns the graph, and a higher-level repair candidate must lose to -/// it — installing the candidate would leave that insert's node with nothing pointing at it. -#[test] -fn a_repair_never_displaces_a_root_installed_while_it_ran() { - let dims = 32; - let path = tmp("entryhealrace"); - let _ = std::fs::remove_file(&path); - let graph = Graph::new(PlaneFile::create(&path, dims, 16, 1_024).expect("create")); - let params = InsertParams::default(); - let mut scratch = SearchScratch::new(); - for i in 0..64 { - insert(&graph, &vector_for(i, dims), ¶ms, &mut scratch).unwrap(); - } - let (observed, _) = graph.file.entry_point(); - let candidate = (0..64u32) - .find(|&id| id != observed && id != 7 && graph.read_node(id).is_some()) - .expect("a live repair candidate"); - let candidate_level = graph.read_node(candidate).expect("live").level; - assert!(graph.read_node(7).is_some(), "precondition: the racing root is a live node"); - - // the interleaving a repair races: the header no longer names the entry it read - graph.file.set_entry_point(7, 0); - assert!( - !graph.file.replace_entry_if(observed, candidate, candidate_level as u32), - "a repair must not publish over an entry installed after it read the dead one" - ); - assert_eq!(graph.file.entry_point().0, 7, "the root installed meanwhile stays"); - - // and it does publish when nothing raced it - let (current, _) = graph.file.entry_point(); - assert!(graph.file.replace_entry_if(current, candidate, candidate_level as u32)); - assert_eq!(graph.file.entry_point().0, candidate); - let _ = std::fs::remove_file(&path); -} diff --git a/package-lock.json b/package-lock.json index 46d8b05811..3d678e7425 100644 --- a/package-lock.json +++ b/package-lock.json @@ -17,6 +17,7 @@ "@fastify/cors": "^11.2.0", "@fastify/static": "^9.1.3", "@harperfast/extended-iterable": "1.0.3", + "@harperfast/hnsw": "0.2.1", "@harperfast/rocksdb-js": "2.8.0", "@harperfast/skills": "^1.10.8", "@turf/area": "6.5.0", @@ -141,6 +142,7 @@ "node": "^22.18.0 || >=24.0.0" }, "optionalDependencies": { + "@harperfast/hnsw": "0.2.1", "bufferutil": "4.1.0", "segfault-handler": "1.3.0", "utf-8-validate": "5.0.10" @@ -2454,6 +2456,93 @@ "version": "1.0.3", "license": "Apache-2.0" }, + "node_modules/@harperfast/hnsw": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/@harperfast/hnsw/-/hnsw-0.2.1.tgz", + "integrity": "sha512-ryUJYE9p7secerYYelVCfhc4kChBv6TZDvDFQUU7NQ99TC2dcPIXQjpOVxne2P6mQPQfh/WP5YmOYNjLn0UNbQ==", + "hasInstallScript": true, + "license": "Apache-2.0", + "optional": true, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@harperfast/hnsw-darwin-arm64": "0.2.1", + "@harperfast/hnsw-linux-arm64-glibc": "0.2.1", + "@harperfast/hnsw-linux-x64-glibc": "0.2.1", + "@harperfast/hnsw-win32-x64": "0.2.1" + } + }, + "node_modules/@harperfast/hnsw-darwin-arm64": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/@harperfast/hnsw-darwin-arm64/-/hnsw-darwin-arm64-0.2.1.tgz", + "integrity": "sha512-dCSrj+eTWyD0qvaN6zHGqo1cpwTOxyfXxP6Kl+EqpuiEcModb9gl1lG+dI6jzCBMJakjmWTgI8WiWrLSWm9p5w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@harperfast/hnsw-linux-arm64-glibc": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/@harperfast/hnsw-linux-arm64-glibc/-/hnsw-linux-arm64-glibc-0.2.1.tgz", + "integrity": "sha512-scLouN0oKR7s4Gv6h1RMzCfS7E0Lca/Tw4MeaLFD5rjQQh2YWvAfhwzPDh/HkPAiaUxEIgCXbc5KlRIzV8jZBQ==", + "cpu": [ + "arm64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@harperfast/hnsw-linux-x64-glibc": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/@harperfast/hnsw-linux-x64-glibc/-/hnsw-linux-x64-glibc-0.2.1.tgz", + "integrity": "sha512-yAReLrNpdRrs0HpPnG/nykSjz4ycMgJOAjMPMVyjkDCg9q2nO+4utn8mSBu59rNXAGdfqUHOYzXvIHrPc2E0Rg==", + "cpu": [ + "x64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@harperfast/hnsw-win32-x64": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/@harperfast/hnsw-win32-x64/-/hnsw-win32-x64-0.2.1.tgz", + "integrity": "sha512-TXbJhvg/7wIYrPFwGnjsOsvKcnevMXEctbDifZEiWLTUgSsepIHYKREaQXIIt+d7l08E3KGdfUl3vNaSZjGqjg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, "node_modules/@harperfast/integration-testing": { "version": "0.7.1", "resolved": "https://registry.npmjs.org/@harperfast/integration-testing/-/integration-testing-0.7.1.tgz", diff --git a/package.json b/package.json index 65c4ce9e05..11e5775fa9 100644 --- a/package.json +++ b/package.json @@ -54,7 +54,6 @@ "scripts": { "build": "tsc --project tsconfig.build.json", "build:watch": "npm run build -- --watch --incremental", - "build:hnsw-plane": "node native/hnsw-plane/build.mjs", "typecheck": "tsc --project tsconfig.json", "typecheck:fast": "npx -y -p typescript@7.0.2 tsc --noEmit --project tsconfig.json", "test:types": "tsc --project unitTests/types/tsconfig.json", @@ -263,6 +262,7 @@ } }, "optionalDependencies": { + "@harperfast/hnsw": "0.2.1", "bufferutil": "4.1.0", "segfault-handler": "1.3.0", "utf-8-validate": "5.0.10" diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 7ad8e76612..8b8791c824 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -7,6 +7,7 @@ import type { Id } from '../../resources/ResourceInterface.ts'; import { SKIP } from '@harperfast/extended-iterable'; import { getPlaneBinding, + invalidatePlaneFile as invalidateHnswPlaneFile, planeFilePathFor, planeStalePathFor, PLANE_NO_ID, @@ -778,33 +779,12 @@ export class HierarchicalNavigableSmallWorld { } } - /** - * Make an undeletable plane file unadoptable: in band first, then with the `.stale` sidecar. - * - * `invalidate()` zeroes the watermark and msyncs the header page alone, so the file reads as - * an incomplete initial mirror — which planeSearchReady already refuses and - * PLANE_INCOMPLETE_REBUILD_MS already rebuilds. It is synchronous on purpose: the sidecar is - * an empty file whose directory entry is never fsynced, so creating it before the watermark - * was durable would let a power loss keep the old nonzero watermark and lose the sidecar, - * and the next process would adopt a plane missing every mutation made while mirroring was - * off. A whole-mapping flush would give the same ordering but cannot run inline on a - * multi-GB plane, which is why this is a 4 KB header barrier rather than `flush(0)`. - * - * The sidecar still follows, because another process may still be mapping this inode and can - * re-stamp the watermark from its own mirror writes; it is checked at attach, before any - * such writer exists. - */ + /** Make an undeletable plane unadoptable before this process stops mirroring it. */ private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { try { - const plane = attached ?? (existsSync(filePath) ? getPlaneBinding()?.open(filePath) : undefined); - plane?.invalidate(); - } catch (invalidateError) { - logger.warn?.('could not zero the watermark of the stale HNSW plane file', invalidateError); - } - try { - closeSync(openSync(planeStalePathFor(filePath), 'w')); - } catch (tombstoneError) { - logger.warn?.('could not tombstone the stale HNSW plane file', tombstoneError); + invalidateHnswPlaneFile(filePath, attached); + } catch (error) { + logger.warn?.('could not invalidate the stale HNSW plane file', error); } } diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index f8df28c3ad..9727d1d898 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -1,5 +1,5 @@ -import { join } from 'node:path'; -import { PACKAGE_ROOT } from '../../utility/packageUtils.js'; +import { closeSync, fsyncSync, openSync } from 'node:fs'; +import { dirname, join } from 'node:path'; import { loggerWithTag } from '../../utility/logging/logger.ts'; const logger = loggerWithTag('HNSW'); @@ -10,7 +10,7 @@ export interface PlaneSearchHit { } /** - * NAPI surface of the native HNSW traversal plane (native/hnsw-plane). Dual-write phase 1 uses + * NAPI surface of the native HNSW traversal plane (`@harperfast/hnsw`). Dual-write phase 1 uses * only the raw mirroring calls (host-allocated ids; the plane's own insert()/remove() allocator * path is bypassed by design) plus the search entry points. */ @@ -60,8 +60,8 @@ export interface HnswPlane { setWatermark(txn: number): void; flush(watermark?: number): void; flushAsync(watermark?: number): Promise; - /** Zero the watermark and msync the header page alone — a 4 KB barrier, not a full flush. */ - invalidate(): void; + invalidateFile(): PlaneInvalidationOutcome; + invalidated(): boolean; } export interface HnswPlaneConstructor { @@ -69,6 +69,19 @@ export interface HnswPlaneConstructor { open(path: string): HnswPlane; } +export interface PlaneInvalidationOutcome { + inBand: boolean; + sidecar: boolean; + inBandError?: string; + sidecarError?: string; +} + +interface HnswPlanePackage { + Plane: HnswPlaneConstructor; + invalidatePlane(path: string): PlaneInvalidationOutcome; + stalePathFor(path: string): string; +} + /** Entry-point id meaning "none" (u32::MAX in the plane header). */ export const PLANE_NO_ID = 0xffffffff; @@ -90,27 +103,56 @@ export function planeFilePathFor(storePath: string, storeName: string): string { * possible and rebuild. */ export function planeStalePathFor(planePath: string): string { - return `${planePath}.stale`; + // the package owns the convention; the literal covers the calls that precede its load + return binding?.stalePathFor(planePath) ?? `${planePath}.stale`; } -// The compiled artifact is optional: harper installs carry no cargo toolchain, so absence just -// means nativePlane-flagged indexes run the existing JS path. Build locally with -// `npm run build:hnsw-plane`. -const BINDING_PATH = join(PACKAGE_ROOT, 'native', 'hnsw-plane', 'hnsw-plane.node'); - -let binding: HnswPlaneConstructor | null | undefined; +let binding: HnswPlanePackage | null | undefined; -/** The native plane constructor, or null when the compiled artifact is unavailable (warns once). */ -export function getPlaneBinding(): HnswPlaneConstructor | null { +function getHnswPackage(): HnswPlanePackage | null { if (binding !== undefined) return binding; try { - binding = require(BINDING_PATH).Plane as HnswPlaneConstructor; + binding = require('@harperfast/hnsw') as HnswPlanePackage; } catch (error) { binding = null; logger.warn?.( - `The hnsw-plane native module is not available (${(error as Error).message}); ` + - `indexes with nativePlane enabled will use the JS search path. Build it with: npm run build:hnsw-plane` + `The @harperfast/hnsw native module is not available (${(error as Error).message}); ` + + 'indexes with nativePlane enabled will use the JS search path' ); } return binding; } + +/** The native plane constructor, or null when the compiled artifact is unavailable (warns once). */ +export function getPlaneBinding(): HnswPlaneConstructor | null { + return getHnswPackage()?.Plane ?? null; +} + +/** Make a derived plane unadoptable before mirroring stops. */ +export function invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): PlaneInvalidationOutcome { + if (attached) return attached.invalidateFile(); + const hnswPackage = getHnswPackage(); + if (hnswPackage) return hnswPackage.invalidatePlane(filePath); + // A plane can outlive the package that made it (uninstall, or a prebuild that stopped + // loading), and a reinstall would then adopt it. Only the sidecar is reachable without the + // package, and it has to survive a power loss to be worth writing, so fsync it and the + // directory entry that names it — the same durability the package's own sidecar has. + const fd = openSync(planeStalePathFor(filePath), 'w'); + try { + fsyncSync(fd); + } finally { + closeSync(fd); + } + try { + const dirFd = openSync(dirname(filePath), 'r'); + try { + fsyncSync(dirFd); + } finally { + closeSync(dirFd); + } + } catch { + // Windows cannot open a directory as a file; it also does not need this — the metadata + // journal already orders the create ahead of anything that could read the sidecar + } + return { inBand: false, sidecar: true }; +} diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index c725bc8647..c5c02af83c 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -6,8 +6,7 @@ * the CF graph (same ids/levels/edges), so a native search over it must return the same * candidates as the JS traversal of the CF graph at equal ef. * - * The whole suite is skipped when the optional native artifact is absent — build it with - * `npm run build:hnsw-plane`. + * The whole suite is skipped when the optional native package is absent. */ require('../testUtils'); const assert = require('node:assert'); @@ -52,7 +51,7 @@ function makeVector(i) { describe('HNSW native plane dual-write', function () { if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; // custom object index is RocksDB-only here if (!getPlaneBinding()) { - it.skip('skipped: native hnsw-plane module not built (npm run build:hnsw-plane)', () => {}); + it.skip('skipped: @harperfast/hnsw native module is unavailable', () => {}); return; } let PlaneTest; @@ -384,7 +383,7 @@ describe('HNSW native plane dual-write', function () { await Foreign.dropTable(); }); - it('an undeletable plane is marked incomplete in band before its stale sidecar is created', async () => { + it('an undeletable plane is invalidated both in band and through a durable sidecar', async () => { const Undeletable = table({ table: 'PlaneUndeletable', database: DB, @@ -399,24 +398,16 @@ describe('HNSW native plane dual-write', function () { const stalePath = planeStalePathFor(planePath); fs.rmSync(stalePath, { force: true }); const plane = getPlaneBinding().open(planePath); - plane.setWatermark(4096); // a plane that would read as a complete mirror on the next attach - const order = []; - // a stand-in rather than a monkeypatch: the binding's methods live on a non-writable - // prototype, so assigning over one silently does nothing in sloppy mode - const observed = { - invalidate() { - // the sidecar must not exist yet: it is an empty file whose directory entry is never - // fsynced, so creating it first lets a power loss keep the old watermark and lose the - // only marker - order.push(fs.existsSync(stalePath) ? 'sidecar-first' : 'in-band-first'); - plane.invalidate(); - }, - }; + plane.setWatermark(4096); // a plane that would otherwise read as complete on the next attach try { - index.invalidatePlaneFile(planePath, observed); - assert.deepEqual(order, ['in-band-first'], 'the durable in-band mark must complete before the sidecar'); - assert.ok(fs.existsSync(stalePath), 'the sidecar still follows, for a process that cannot map the file'); - assert.equal(getPlaneBinding().open(planePath).getWatermark(), 0, 'the plane must read as an incomplete mirror'); + index.invalidatePlaneFile(planePath, plane); + assert.ok(plane.invalidated(), 'the open mapping must observe the one-way in-band invalidation latch'); + assert.ok(fs.existsSync(stalePath), 'the package must also persist the out-of-band sidecar'); + assert.throws( + () => getPlaneBinding().open(planePath), + /invalidat|stale/i, + 'a new process must refuse the invalidated plane' + ); } finally { fs.rmSync(stalePath, { force: true }); await Undeletable.dropTable(); From 87e0fd71afa54a5b371ae1d158d2fa88e396924c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 4 Sep 2026 12:34:22 -0600 Subject: [PATCH 60/69] Address the pre-push review: strict build in the plane CI job, misplaced JSDoc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hnsw-plane job's `npm run build || true` was copied from the jobs that tolerate type errors, but this one runs mocha against dist/ — a tolerated build failure grades a stale dist and still reports green, which is the same hole the load probe closes on the package side. It builds clean on this branch, so the job builds strictly. The createAndMirrorPlane doc block sat above planeLayer0Cap, describing the wrong method. Co-Authored-By: Claude Opus 5 --- .github/workflows/unit-test.yml | 4 +++- resources/indexes/HierarchicalNavigableSmallWorld.ts | 10 +++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml index fe0c960d42..5f41696035 100644 --- a/.github/workflows/unit-test.yml +++ b/.github/workflows/unit-test.yml @@ -140,8 +140,10 @@ jobs: - name: Install dependencies run: npm install --ignore-scripts + # strict here, unlike the other jobs: the plane suite runs against dist/, so a tolerated + # build failure would grade a stale dist and still report green - name: Build - run: npm run build || true # we currently have type errors so just ignore that + run: npm run build - name: Verify the published native binding loads run: node -e "if (typeof require('@harperfast/hnsw').Plane?.open !== 'function') throw new Error('@harperfast/hnsw loaded without a Plane constructor');" diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 8b8791c824..c3cf313a9e 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -531,6 +531,11 @@ export class HierarchicalNavigableSmallWorld { return false; } + /** The JS graph's effective layer-0 cap for this configuration; sizes the plane's slots. */ + private planeLayer0Cap(): number { + return this.optimizeRouting ? this.M << 3 : this.M << 1; + } + /** * Create the plane file and fully mirror the existing CF graph into it (the "first enable" * build — a pure copy of the same graph, so plane and CF are bit-identical by construction; @@ -540,11 +545,6 @@ export class HierarchicalNavigableSmallWorld { * scan's older snapshot of that node — it re-syncs on the node's next touch, and the exact * rescore + record load already filter stale candidates (relaxed adherence, design §5). */ - /** The JS graph's effective layer-0 cap for this configuration; sizes the plane's slots. */ - private planeLayer0Cap(): number { - return this.optimizeRouting ? this.M << 3 : this.M << 1; - } - private createAndMirrorPlane( Plane: NonNullable>, filePath: string, From 17b8a34e929b0ad739646302dc80e8b3e85ae445 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 14:20:13 -0600 Subject: [PATCH 61/69] Make native HNSW index file-primary Co-Authored-By: GPT-6 Codex --- dependencies.md | 10 +- hnsw-native-plane.md | 245 +++++-- resources/DerivedIndexBackend.ts | 398 ++++++++++++ resources/RocksTransactionLogStore.ts | 3 +- resources/Table.ts | 2 + resources/databases.ts | 14 + .../HierarchicalNavigableSmallWorld.ts | 487 +++++--------- resources/indexes/hnswPlaneBinding.ts | 12 +- .../resources/vectorIndexPlane-thread.js | 46 ++ unitTests/resources/vectorIndexPlane.test.js | 611 ++++++++---------- 10 files changed, 1099 insertions(+), 729 deletions(-) create mode 100644 resources/DerivedIndexBackend.ts create mode 100644 unitTests/resources/vectorIndexPlane-thread.js diff --git a/dependencies.md b/dependencies.md index 47be015a55..3156e697ad 100644 --- a/dependencies.md +++ b/dependencies.md @@ -246,12 +246,12 @@ This is the inverse of the entries below — a dependency we take deliberate ste ## @harperfast/hnsw (optional dependency) -- Need for usage: Supplies the native memory-mapped HNSW traversal plane used only by indexes that opt in with `nativePlane: true`. Harper keeps the RocksDB graph authoritative in phase 1 and falls back to its JS traversal when the package cannot load. +- Need for usage: Supplies the file-primary memory-mapped HNSW index used only by indexes that opt in with `nativePlane: true`. Harper keeps primary-key mappings and replay cursors in RocksDB; graph nodes and adjacency exist only in the native file. - Size/memory cost: The JS/package metadata is about 250 KB unpacked plus one platform-specific native binary. Runtime mapped-file size is approximately 1,344 bytes per 768-dimensional int8 node at layer-0 cap 128; mappings are shared by the OS page cache across workers. - Security: First-party Apache-2.0 Harper package. It runs native code in-process, so Harper exact-pins the package and its own manifest exact-pins every platform prebuild to the same version. The dedicated CI job loads the registry prebuild and tests Harper against it. - Environment interaction: Lazily loaded only when an eligible index enables `nativePlane`; it creates a memory-mapped `.hnsw` derived-index file next to the index store and may create a `.stale` invalidation sidecar. It does not modify globals or install polyfills. -- Overlap: It mirrors the existing JS HNSW graph during the opt-in validation phase. The overlap is deliberate: native traversal removes per-node RocksDB reads and JS object bookkeeping while the existing graph remains the rollback path. +- Overlap: None for an opted-in index. Ordinary HNSW indexes continue to use the JS/RocksDB graph; `nativePlane` indexes use native insert, mutation, and search through the shared post-commit derived-index runtime. - Transitive dependencies: Only exact-version, platform-specific optional prebuild packages; no JS runtime dependency tree. -- Binary compilation: Supported Linux glibc x64/arm64, macOS arm64, and Windows x64 targets use prebuilds. Other targets attempt a Rust source build; because the root package is optional, a failed build leaves the JS path available. -- Can be deferred: Yes. The adapter requires it lazily and caches an unavailable result after one warning. -- Eventual removal: Remove the optional dependency and adapter integration, delete derived `.hnsw` files, and use the existing JS/CF graph path. The `nativePlane` flag is explicitly rollback-safe in phase 1. +- Binary compilation: Supported Linux glibc x64/arm64, macOS arm64, and Windows x64 targets use prebuilds. Other targets attempt a Rust source build. Because the root package is optional Harper still installs if that build fails, but opted-in indexes remain unavailable until the module is present. +- Can be deferred: Yes for ordinary HNSW indexes. The adapter loads lazily; an opted-in index returns 503 and retries rebuild when the native module is unavailable. +- Eventual removal: Disable `nativePlane`, allow the schema reindex to rebuild the ordinary JS/CF graph, then remove the optional dependency and adapter integration. diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index f0fd87364c..45685829fd 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -55,8 +55,7 @@ Non-goals (this phase): - Binary quantization / Matryoshka truncation (benchmark-gated per the Reflex study; the format reserves a quantization-mode field so a binary plane is a format v2, not a redesign). -- Native insert loop (phase 3; insert logic stays in JS initially, persisting through the - native slot-write API). +- Native batch insertion; version 0.2.1 supplies the single-record native insert used here. - Cross-node ANN protocol. Out of scope entirely. - Lexical/BM25 anything. @@ -65,18 +64,19 @@ Non-goals (this phase): ``` JS (worker threads) native (Rust, napi-rs) ┌─────────────────────────────────────────┐ ┌─────────────────────────────────────┐ - │ HierarchicalNavigableSmallWorld.ts │ │ hnsw-plane │ - │ • pk→nodeId mapping (stays RocksDB) │ │ • mmap'd slot file (per index/slice)│ - │ • insert/update/delete logic (phase 1) ├──►│ • slot read/write API (seqlocked) │ - │ • phase-1 mirror → slot writes │ │ • search(query, k, ef, filter) → │ - │ • record load + exact rescore (as-is) │◄──┤ top-k ids, own thread pool │ - │ • runIndexing replay from watermark │ │ • TSFN batch filter callback │ + │ DerivedIndexBackend + HNSW adapter │ │ @harperfast/hnsw 0.2.1 │ + │ • pk↔nodeId mapping (RocksDB) │ │ • mmap'd graph file (per index) │ + │ • per-origin audit cursor (RocksDB) │ │ • insert/remove (seqlocked) │ + │ • committed-log replay + rebuild ├──►│ • search(query, k, ef, filter) → │ + │ • record load + exact rescore │◄──┤ top-k ids, libuv worker pool │ + │ • bounded keyed wake-up queues │ │ • TSFN batch filter callback │ └─────────────────────────────────────────┘ └─────────────────────────────────────┘ ``` -What stays in RocksDB: the pk→nodeId mapping (transactional with record writes — it is the -authority on which node id a record owns), records themselves, and all other indexes. What -moves to the file: node vectors, per-layer adjacency, entry point, id allocator, freelist. +What stays in RocksDB: the primary records, pk↔nodeId mappings, durable replay cursors, and all +other indexes. The mappings are post-commit derived state and are recoverable from records plus +the retained transaction log. Node vectors, per-layer adjacency, entry point, id allocator, and +freelist live only in the native file. ## 4. File format (v1) @@ -210,36 +210,154 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d ## 8. Write path phasing -- **Phase 1 — dual-write, search cutover.** Insert/update/delete logic stays in JS +- **Phase 1 prototype — dual-write, search cutover (superseded before merge).** Insert/update/delete logic stayed in JS (`HierarchicalNavigableSmallWorld.ts` unchanged algorithmically); mutations persist to BOTH the index CF (as today) and the file via native slot-write calls. Search runs native from the file. Validation = compare native results against the JS path on the same graph; rollback = flip search back to JS, drop the file. The double-write cost is bounded (index writes are a fraction of insert cost) and temporary. - _Integrated_ behind the opt-in `nativePlane: true` index option (search-only: toggling never - reindexes; int8 + cosine indexes only — the flag no-ops elsewhere). Mutations mirror at the - exact `indexStore.put/remove` sites via `writeNodeRaw`/`clearNode`/`setEntryPoint` with - host-allocated ids; the plane file (`/
..hnsw`, layer0 cap 128, - 16M-node sparse reservation) is created lazily with a full mirror of the existing CF graph on - first enable, reopened on restart, deleted on drop/clear/reindex. The exact-pinned - `@harperfast/hnsw` package is optional; absence falls back to the JS path with one warning. - Parity, predicate, restart, and lifecycle coverage in `unitTests/resources/vectorIndexPlane.test.js`. - Watermark/replay wiring, slicing, and msync-cadence flushes are not wired yet (open items). - -- **Phase 2 — shared post-commit delivery, then file-primary.** Implement #2489's - `DerivedIndexBackend` contract rather than an HNSW-specific commit callback: every worker - enqueues its own committed transaction-log entries, the backend advances a durable log-position - watermark, and open/rebuild uses the shared retention-aware replay driver. Moving the mirror - from the pre-commit `indexStore.put/remove` sites into that delivery path removes rollback - phantoms without creating a second protocol alongside full-text indexing. Then drop the CF - graph writes; the file is the only graph store. JS insert - reads nodes through a native `getNode(id)` (one NAPI crossing per read, ~1 µs — comparable to - today's decode path). Migration for existing indexes: reindex (accepted contract), or a - one-shot CF→file bulk conversion since it is a pure format transform. -- **Phase 3 — native insert.** Move the insert search + neighbor selection native (same - traversal core), leaving JS a thin `index(pk, vector)` call. Unlocks bulk build (C5) at - native speed and removes the ~tens-of-ms event-loop pin per insert (#895). + This implementation was removed when phase 2 landed on the same draft PR. It never shipped, so + the PR has no deployed dual-write format to preserve or migrate in place. + +- **Phase 2 — shared post-commit delivery, file-primary (current implementation).** Implement #2489's + `DerivedIndexBackend` runtime rather than an HNSW-specific commit callback. This remains opt-in + behind `nativePlane: true`; ordinary HNSW indexes keep the RocksDB graph and do not acquire an + audit dependency. A table with an opted-in backend must explicitly declare `audit: true`; inheriting + either true or false from the global setting is rejected so a vector-index option cannot silently + expand the audit-readable security surface. Auditing adds a durable full-record audit entry to + every commit on the table, including commits that do not change the vector, for the configured + retention window. This storage and data-retention cost is part of opting in. + Enablement warns that the audit API can now expose full table history for that window. + + The pre-commit index hook compares the old and new projection, validates it, and stages only the + changed backend target on the transaction; unrelated field changes never enqueue an HNSW update. + Direct attributes use reference/length equality first and compare numeric components only when + identities differ; resolver-backed projections necessarily pay the component comparison. + Pending work is coalesced by `(index, primaryKey)`: rapid updates replace that key's queued target + with one reconciliation against current primary state. The runtime retains the log-position tickets + that caused each wake-up, while replay advances each origin cursor through the audit log only after + the native durability barrier. Before commit it checks the aggregate distinct-key and position-ticket + depth already published by every worker. A bounded number of simultaneously committing transactions + can pass the same check, so the configured limit is a scheduling threshold rather than a strict memory + ceiling. Once the observed depth reaches either threshold, a new vector-changing write receives a + retryable 503 until the backend catches up; unrelated writes continue. This admission control is + required because accepting unique-key load above native insert throughput and then rebuilding at + that same throughput cannot converge. + The initial limit is 65,536 changed records or 262,144 queued log-position tickets (approximately + 16 MiB of identifiers/metadata), whichever comes first; current depth and a process-wide rejection + count are logged at exponentially spaced rejection counts so the fixed limit can be tuned without + flooding logs. + + `RocksTransactionLogStore.aftercommit` only schedules the staged targets and returns. Each worker + has a queue, but one cross-thread per-index writer lock serializes complete batches, and each queue + is FIFO, so two updates to one key cannot apply in reverse order. The stored mapping also carries + the last reconciled primary-record version and discards an older observation. If applying or + flushing an already-committed entry fails, it drops the complete in-flight batch and memory queue, + leaves the durable cursor before the failed entry, logs the failure with index identity, marks the + index unavailable/rebuild-required, and rebuilds from current + records. It cannot crash the record writer, and no rejected promise escapes the detached drain. + Persistent rebuild failures retry with capped exponential backoff (1 second through 5 minutes), + keep search unavailable, and emit one error per failed retry rather than spinning. + + The backend advances a durable cursor for each origin log only after its own durability barrier. + On open, a cursor older than `logging.auditRetention` causes a full rebuild; replay never advances + across a retained gap. The default retention means a node unavailable beyond that window takes the + measured rebuild path and returns 503 for its duration. Operators can size retention for the + expected outage, but correctness does not depend on doing so: expiry changes recovery cost, not + outcome. During replay or rebuild, + searches return the existing index-in-progress 503 rather than reading a partial file. Cleanup + may delete old segments; the recovery contract is rebuild rather than pinning audit indefinitely. + + HNSW uses the package's standalone `insert`/`remove` API. The RocksDB index CF retains only + `primaryKey ↔ nativeNodeId` identity and per-origin replay cursors; graph nodes and adjacency + exist only in the mmap file. A changed mapping is marked pending, hidden from search, and published + only after the mmap flush succeeds; the cursor advances after the mapping publication. A crash can + therefore leave replay extra work, but cannot leave a published mapping to an undurable node. + Hot delivery and replay both re-read the current authoritative record + after commit. This gives multi-origin/source-resolution writes the same reconciliation rule and + makes an old entry idempotent as "delete current native id, then add the current value". The + audit object's in-memory record is an allowed optimization only when its version is still the + primary store's current version. Rebuild records a + log boundary, scans current records into a fresh file, and replays from that boundary before the + index becomes queryable. Existing phase-1 graph CFs are migrated by this rebuild. + + The current package fixes standalone construction at M=16, efConstruction=200, mL=1/ln(16), + and optimizeRouting=0.5, so file-primary mode accepts only that geometry. Its sparse reservation + is fixed at create time; `nativePlaneMaxNodes` is therefore a structural option (16M default), + and exhaustion makes the index unavailable until it is enlarged and rebuilt. The native file + is an approximate derived index: concurrent CRDT/source-resolution arrivals are not promised to + reproduce a single total order. Exact record load and rescore still reject stale candidates. + A crash between native allocation and identity persistence can leave an unreachable native node; + replay restores the live record and exact filtering hides the orphan, while rebuild reclaims it. + A schema requesting `nativePlane: true` with non-native construction geometry is rejected rather + than silently changing M/efConstruction/mL/optimizeRouting during upgrade. + + Phase 1 has not shipped: this PR is draft, so no deployed index is silently migrated from its + configurable JS geometry. Version 0.2.1 already implements the insertion search and graph mutation + inside the native `insert()` call; phase 2 uses that path for rebuild as well as incremental writes, + rather than the ~263 inserts/s JS anchor. Before merge, a 100k-record native rebuild benchmark must + sustain at least 1,000 inserts/s on the Linux CI runner and publish progress (records, rate, ETA). + At the 16M default reservation this floor bounds a full rebuild to about 4.5 hours; configurations + above it explicitly accept the proportionally longer 503 recovery window until a batch API exists. + Worker shutdown is graceful and waits for a synchronous N-API insert to return; a worker is never + force-terminated in the middle of a plane mutation while the process survives. + +- **Phase 3 — native batch insert.** Add a bulk API so rebuild can cross N-API once per chunk and + report progress while native code owns the insertion loop. Single-record insertion search and + graph mutation already run natively in 0.2.1. + +### Approaches considered for phase 2 + +**Chosen: transaction log plus rebuild on retention gap.** Correctness: record and audit entry +commit atomically; post-commit work cannot leak an aborted write; per-origin cursors advance after +the plane flush; retained gaps force rebuild; replay reconciles against current records. Performance: +the record transaction adds no marker or mmap work, but opting in does add the required audit-log +write to every table commit; projection comparison prevents unrelated commits from reaching HNSW, +rapid same-key changes coalesce, and changed-vector writes are admitted only while the bounded queue +has capacity. The commit hook only schedules work and the backend batches its durability barrier. +Operational complexity: it reuses Harper's existing audit log, +aftercommit stream, retention setting, and rebuild path. Scope: the runtime and interface are shared +with future Tantivy work, while this PR supplies only the HNSW backend. + +**Rejected: transactional dirty-key outbox.** Correctness is attractive because a marker cannot be +lost to audit retention and current-state reconciliation is naturally idempotent. It adds a second +durable delivery fact beside the transaction log, however, plus marker fencing, reclamation, and a +write to every indexed record transaction. That contradicts #2489's single protocol and raises hot +write amplification for every backend. + +**Rejected: keep the phase-1 CF graph as authority.** This preserves rollback and package fallback +and makes native state disposable. It retains the duplicate graph writes and graph decode/storage +cost this phase exists to remove, so write throughput and storage continue to scale with two graphs. + +**Rejected: ship post-commit delivery first and retain the CF graph until a later cutover.** This is +the safer rollout when phase 1 is already deployed, but phase 1 exists only on this draft PR and has +no production population to protect. The 0.2.1 dependency already moved insertion native, and the +opt-in flag plus 503-during-build behavior remains the rollout gate. Shipping another temporary disk +format would add a migration and prolong the duplicate graph cost without gathering compatibility +evidence from any existing deployment. File-primary therefore lands before this PR's first merge. + +**Rejected: synchronous native mutation inside the record transaction.** This removes the CF graph +with little new runtime code. An aborted transaction can still publish an mmap mutation, and native +construction remains on request latency. Recovery then needs the same log protocol anyway. + +**Rejected: build on `transactionBroadcast`'s subscription registry.** That module supplies the +scheduling and transaction-grouping precedent, but its cursors are live-subscription state +(`lastTxnTime` and one shared thread-local position), not durable per derived index and origin. Its +same-thread Rocks path also drains synchronously while holding `thread-local-writes`, so it cannot +contain an async native durability barrier. Adapting it would couple client-subscription lifetime to +index availability and still require the cursor, queue bound, gap detection, and rebuild machinery. +The derived runtime therefore subscribes directly to the same lower-level `aftercommit` event. + +**Chosen for overload: coalesce by key, then admission-control unique work.** Applying every +intermediate vector wastes construction work because delivery always reconciles against current +primary state. A bounded keyed queue replaces pending work for the same key while retaining ordered +position tickets for cursor-prefix accounting. Per-key rate limiting is unnecessary after coalescing; +serving the removed CF graph would reintroduce dual storage and cannot include new keys. If distinct +keys or tickets still fill the queue, accepting an unbounded deficit violates eventual convergence +and dropping work forces rebuild at the same inadequate rate, so the pre-commit hook returns a +retryable 503 before that vector-changing record commits. This couples that record to its declared +index, as ordinary transactional indexes already do, while unrelated writes remain independent. ## 9. Validation plan @@ -258,6 +376,34 @@ p50 7.2 ms / recall@10-set 0.997 @ ef 512). Acceptance for phase 1: 5. **Churn:** delete/reinsert cycles hold node count stable (freelist reuse; #2182 regression test). +Phase-2 acceptance: + +1. An aborted record transaction produces no native mutation; committed insert/update/delete reaches + native search only after `aftercommit`. +2. An opted-in table without an explicit `audit: true` is rejected, while an ordinary HNSW index + keeps inheriting the global audit setting. The schema error explains that the audit API retains + full table history for the configured window. +3. Restart replays retained entries from each origin cursor; a cursor before retention rebuilds and + search stays unavailable until publication. +4. Concurrent multi-origin delivery re-reads the winning primary record, independent of delivery + order. +5. An unrelated-attribute commit schedules zero plane operations. Repeated writes to one key + coalesce while retaining a correct contiguous cursor; saturating with distinct keys rejects a + changed-vector write with a retryable 503 before commit and still permits an unrelated write. +6. A forced apply/flush failure retains the earlier cursor, produces no unhandled rejection, logs + once for that episode, marks the index unavailable, and converges by rebuild. +7. A full rebuild and retained-log replay both meet the same recall@k baseline as a from-scratch JS + graph on the same deterministic corpus; duplicate and deleted ids are absent from returned keys. +8. A non-default M/efConstruction/mL/optimizeRouting with `nativePlane: true` is rejected during + schema setup, not silently rebuilt under native defaults. +9. CI's existing `HNSW native plane` job installs 0.2.1 and runs a separate load probe before mocha; + absence already fails the job instead of producing a skipped green suite (verified on 87e0fd71). +10. Two rapid updates to one key across workers cannot apply in reverse version order, and graceful + worker recycle waits until an in-flight native mutation has returned. +11. A 100k-record native rebuild reports progress and sustains at least 1,000 inserts/s on Linux CI. +12. A soak combines concurrent search, queue admission, retry, and restart during delivery; after + recovery its result quality meets the same deterministic recall baseline. + ## 10. Decisions & open questions Decided (Kris, 2026-08-31): @@ -271,16 +417,17 @@ Decided (Kris, 2026-08-31): diversity-preserving prune at lower cap is worth engineering). - **Platform policy.** Performance is a Linux target only. macOS must work (mmap/msync semantics differ — msync alone is a weaker barrier there; an `F_FULLFSYNC` pass is a known follow-up, - and sparse-file behavior varies by filesystem — functional, not optimized). Windows may fall back to the JS implementation - entirely; the native plane is allowed to be absent there. + and sparse-file behavior varies by filesystem — functional, not optimized). Windows is supported + through the package prebuild. If the optional native package is unavailable on any platform, + `nativePlane` indexes stay unavailable; ordinary HNSW indexes continue to use the JS implementation. - **Packaging: independent open-source package.** The core has zero Harper coupling — the crate - compiles standalone and its NAPI surface is generic (create/open plane, insert(id, vector), + compiles standalone and its NAPI surface is generic (create/open plane, insert(vector), remove(id), search(query, k, ef, filter), watermark get/set). Harper-specific glue — the pk→nodeId mapping, #2489's `DerivedIndexBackend` delivery, txnlog-anchored replay, auto-ef policy constants — stays in Harper regardless of packaging. Published as the exact-pinned optional dependency `@harperfast/hnsw` 0.2.1 (Apache-2.0, HarperFast/hnsw), with platform - prebuilds and a source-build fallback; the Harper adapter owns only availability/fallback and - integration policy. The pitch as a community package: a persistent, + prebuilds and a source-build fallback; the Harper adapter owns availability and integration + policy. The pitch as a community package: a persistent, incrementally-maintained, concurrently-searchable HNSW for Node — hnswlib-node has no durable incremental persistence, no off-loop batched filtering, no seqlock concurrency. @@ -304,20 +451,14 @@ Open: - ~~Reservation growth~~ — decided (Kris, 2026-08-31): a generous sparse reservation at create is the model; mremap-based growth is a possible later enhancement, not a requirement. -## 11. Known phase-1 limitations (reviewed, accepted, tracked) +## 11. Phase-1 findings resolved by phase 2 -From the round-1 cross-model review (codex + gemini + cursor-grok + harper-domain), two -architectural findings are deliberately deferred rather than fixed in phase 1 — both are -bounded by the phase-1 contract (opt-in flag, CF authoritative, plane derived): +The phase-1 review found two integration constraints. Phase 2 resolves the first and retains the +second as part of the custom-index search contract: -- **Mirroring runs at the indexStore.put sites, inside the transaction.** A rolled-back - transaction can leave phantom nodes in the plane (the CF never had them). Phantom ids are - filtered at record load (missing record → SKIP), so results can be transiently short by - the phantom count; the garbage accumulates only at the rollback rate. The structural fix — - driving the mirror through #2489's shared post-commit `DerivedIndexBackend` runtime — is the - phase-2 delivery/watermark/replay work item and also subsumes the residual lost-write - window during attach retry (mirror calls during the 250 ms backoff are dropped and heal - only on the node's next touch). +- **Pre-commit mmap writes:** resolved. Transactions only stage changed primary keys; native mutation + starts from `RocksTransactionLogStore.aftercommit`, and abort coverage verifies that no node is + allocated for a rolled-back record write. - **The async custom-index search contract** (`resources/search.ts`): a plane-backed search returns a promise-backed, async-only iterable; synchronous consumers of custom-index results would throw. Harper's search paths tolerate MaybePromise, and one full-stack test diff --git a/resources/DerivedIndexBackend.ts b/resources/DerivedIndexBackend.ts new file mode 100644 index 0000000000..cfd06fd77c --- /dev/null +++ b/resources/DerivedIndexBackend.ts @@ -0,0 +1,398 @@ +import { ClientError, ServerError } from '../utility/errors/hdbError.ts'; +import { loggerWithTag } from '../utility/logging/logger.ts'; +import { getWorkerIndex } from '../server/threads/manageThreads.js'; +import type { AuditRecord } from './auditStore.ts'; + +const logger = loggerWithTag('DerivedIndex'); +const MAX_PENDING_KEYS = 65_536; +const MAX_PENDING_TICKETS = 262_144; +const APPLY_BATCH_SIZE = 128; +const REBUILD_PROGRESS_INTERVAL = 10_000; +const MAX_REBUILD_BACKOFF = 300_000; +const MAX_WORKER_SLOTS = 256; +const DEPTH_VALUES_PER_WORKER = 3; // pending keys, position tickets, rejected writes +const warnedAuditIndexes = new Set(); + +type Position = { nodeId: number; txnLogKey: number }; +type Ticket = Position & { done: boolean }; +type Pending = { id: any; tickets: Ticket[] }; +type StagedTarget = { runtime: DerivedIndexRuntime; id: any }; + +export function derivedIndexCursorKey(indexName: string, nodeId: number): symbol { + return Symbol.for(`derived-index-cursor:${indexName}:${nodeId}`); +} + +export interface DerivedIndexBackend { + readonly postCommit: true; + attachDerivedRuntime(runtime: DerivedIndexRuntime): void; + applyDerivedValue(id: any, value: any, version?: number): void; + flushDerived(watermark?: number): Promise; + resetDerivedStorage(): void; +} + +function valuesEqual(a: any, b: any): boolean { + if (a === b) return true; + if (a == null || b == null || a.length !== b.length) return false; + for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) return false; + return true; +} + +function lock(store: any, key: string, callback: () => Promise): Promise { + return new Promise((resolve, reject) => { + let started = false; + const acquired = () => { + if (started) return; + started = true; + Promise.resolve() + .then(callback) + .then(resolve, reject) + .finally(() => store.unlock(key)); + }; + if (store.tryLock(key, acquired)) acquired(); + }); +} + +export class DerivedIndexRuntime { + readonly backend: DerivedIndexBackend; + readonly indexStore: any; + readonly table: any; + readonly attribute: any; + readonly lockKey: string; + private sharedDepth: Int32Array; + private sharedDepthOffset: number; + private pending = new Map(); + private ticketsByOrigin = new Map(); + private pendingTickets = 0; + private drainScheduled = false; + private draining = false; + private closed = false; + private ready = false; + private retryScheduled = false; + private rebuildBackoff = 1_000; + private listener: (entries: AuditRecord[], targets?: StagedTarget[]) => void; + + constructor(table: any, attribute: any, indexStore: any, backend: DerivedIndexBackend) { + this.table = table; + this.attribute = attribute; + this.indexStore = indexStore; + this.backend = backend; + this.lockKey = `derived-index-writer:${indexStore.name}`; + const workerIndex = getWorkerIndex() ?? 0; + this.sharedDepth = new Int32Array( + indexStore.getUserSharedBuffer( + `derived-index-depth:${indexStore.name}`, + new ArrayBuffer(MAX_WORKER_SLOTS * DEPTH_VALUES_PER_WORKER * Int32Array.BYTES_PER_ELEMENT) + ) + ); + this.sharedDepthOffset = workerIndex * DEPTH_VALUES_PER_WORKER; + if (this.sharedDepthOffset + DEPTH_VALUES_PER_WORKER - 1 >= this.sharedDepth.length) { + throw new Error(`Derived indexes support worker indexes below ${MAX_WORKER_SLOTS}`); + } + // A replacement worker reuses the dead worker's slot. Resetting only this slot removes + // stale admission pressure without disturbing live peers. + Atomics.store(this.sharedDepth, this.sharedDepthOffset, 0); + Atomics.store(this.sharedDepth, this.sharedDepthOffset + 1, 0); + Atomics.store(this.sharedDepth, this.sharedDepthOffset + 2, 0); + backend.attachDerivedRuntime(this); + this.listener = (entries, targets) => this.committed(entries, targets); + table.auditStore.on('aftercommit', this.listener); + indexStore.isIndexing = true; + void this.initialize(); + } + + stage(transaction: any, id: any, value: any, existingValue: any): void { + if (valuesEqual(value, existingValue)) return; + let totalKeys = 0; + let totalTickets = 0; + let rejectedWrites = 0; + for (let offset = 0; offset < this.sharedDepth.length; offset += DEPTH_VALUES_PER_WORKER) { + totalKeys += Atomics.load(this.sharedDepth, offset); + totalTickets += Atomics.load(this.sharedDepth, offset + 1); + rejectedWrites += Atomics.load(this.sharedDepth, offset + 2); + } + if ((!this.pending.has(id) && totalKeys >= MAX_PENDING_KEYS) || totalTickets >= MAX_PENDING_TICKETS) { + const localRejections = Atomics.add(this.sharedDepth, this.sharedDepthOffset + 2, 1) + 1; + if ((localRejections & (localRejections - 1)) === 0) { + logger.warn?.( + `Derived index ${this.indexStore.name} rejected ${rejectedWrites + 1} writes while ${totalKeys} keys and ${totalTickets} log positions were pending` + ); + } + throw new ServerError( + `Derived index ${this.indexStore.name} is catching up; retry this vector-changing write`, + 503 + ); + } + const targets: StagedTarget[] = (transaction.derivedIndexTargets ??= []); + if (!targets.some((target) => target.runtime === this && target.id === id)) targets.push({ runtime: this, id }); + } + + private committed(entries: AuditRecord[], targets?: StagedTarget[]): void { + if (this.closed || !targets) return; + const positions = new Map(); + for (const entry of entries) { + if (entry.tableId === this.table.tableId && entry.recordId != null) positions.set(entry.recordId, entry); + } + for (const target of targets) { + if (target.runtime !== this) continue; + const entry = positions.get(target.id); + if (!entry?.txnLogKey) { + this.fail(new Error(`Committed derived-index target ${String(target.id)} has no transaction-log entry`)); + return; + } + this.enqueue(target.id, { nodeId: entry.nodeId ?? 0, txnLogKey: entry.txnLogKey }); + } + this.scheduleDrain(); + } + + private enqueue(id: any, position: Position): void { + const ticket: Ticket = { ...position, done: false }; + let pending = this.pending.get(id); + if (!pending) { + this.pending.set(id, (pending = { id, tickets: [] })); + Atomics.add(this.sharedDepth, this.sharedDepthOffset, 1); + } + pending.tickets.push(ticket); + let originTickets = this.ticketsByOrigin.get(position.nodeId); + if (!originTickets) this.ticketsByOrigin.set(position.nodeId, (originTickets = [])); + originTickets.push(ticket); + this.pendingTickets++; + Atomics.add(this.sharedDepth, this.sharedDepthOffset + 1, 1); + } + + private scheduleDrain(): void { + if (this.closed || !this.ready || this.draining || this.drainScheduled || this.pending.size === 0) return; + this.drainScheduled = true; + setImmediate(() => { + this.drainScheduled = false; + void this.drain(); + }); + } + + private async drain(): Promise { + if (this.closed || this.draining || !this.ready) return; + this.draining = true; + try { + await lock(this.indexStore, this.lockKey, async () => { + if (this.closed) return; + const batch = Array.from(this.pending.values()) + .slice(0, APPLY_BATCH_SIZE) + .map((pending) => ({ pending, ticketCount: pending.tickets.length })); + await this.reconcile(this.table.auditStore.loadLogs?.() ?? []); + if (!this.ready || this.closed) return; + for (const { pending, ticketCount } of batch) { + for (const ticket of pending.tickets.slice(0, ticketCount)) ticket.done = true; + pending.tickets.splice(0, ticketCount); + if (pending.tickets.length === 0) { + this.pending.delete(pending.id); + Atomics.sub(this.sharedDepth, this.sharedDepthOffset, 1); + } + } + this.discardCompletedTickets(); + }); + } catch (error) { + this.fail(error); + } finally { + this.draining = false; + this.scheduleDrain(); + } + } + + private cursorKey(nodeId: number): symbol { + return derivedIndexCursorKey(this.indexStore.name, nodeId); + } + + private discardCompletedTickets(): void { + for (const [nodeId, tickets] of this.ticketsByOrigin) { + while (tickets[0]?.done) { + tickets.shift(); + this.pendingTickets--; + Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, 1); + } + if (tickets.length === 0) this.ticketsByOrigin.delete(nodeId); + } + } + + private async initialize(): Promise { + if (this.closed) return; + try { + await this.table.indexingOperation; + if (this.closed) return; + // runIndexing clears this flag when its schema scan completes. Reassert ownership + // before replay/rebuild so queries cannot observe a partial native generation. + this.indexStore.isIndexing = true; + await lock(this.indexStore, this.lockKey, async () => { + if (this.closed) return; + const logs = this.table.auditStore.loadLogs?.() ?? []; + await this.reconcile(logs); + }); + if (this.closed) return; + this.ready = true; + this.indexStore.isIndexing = false; + this.rebuildBackoff = 1_000; + this.scheduleDrain(); + } catch (error) { + this.fail(error); + } + } + + private cursorExists(nodeId: number, cursor: number): boolean { + for (const entry of this.table.auditStore.getRange({ start: cursor, exactStart: true, log: nodeId })) { + return entry.txnLogKey === cursor; + } + return false; + } + + private async reconcile(logs: any[]): Promise { + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + if (!logs[nodeId]) continue; + const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); + const cursorExists = cursor && this.cursorExists(nodeId, cursor); + if (!cursorExists) { + await this.rebuild(logs); + return; + } + } + await this.replay(logs); + } + + private async replay(logs: any[]): Promise { + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + if (!logs[nodeId]) continue; + const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); + if (!cursor || !this.cursorExists(nodeId, cursor)) { + throw new Error(`Derived-index cursor for origin ${nodeId} is outside audit retention`); + } + let latest = cursor; + let applied = 0; + for (const entry of this.table.auditStore.getRange({ + start: cursor, + exactStart: true, + exclusiveStart: true, + log: nodeId, + })) { + latest = entry.txnLogKey; + if (entry.tableId === this.table.tableId && entry.recordId != null) { + const current = this.table.primaryStore.getEntry(entry.recordId); + const record = current?.value; + const value = + record && (this.attribute.resolve ? this.attribute.resolve(record) : record[this.attribute.name]); + this.backend.applyDerivedValue(entry.recordId, value, current?.localTime ?? current?.version); + if (++applied % APPLY_BATCH_SIZE === 0) await this.backend.flushDerived(latest); + } + } + // exactStart follows physical log order after locating the cursor. A transaction that + // began earlier can commit later with a numerically smaller timestamp, so inequality, + // rather than a numeric greater-than comparison, identifies forward progress. + if (latest !== cursor) { + await this.backend.flushDerived(latest); + this.indexStore.putSync(this.cursorKey(nodeId), latest); + } + } + } + + private async rebuild(logs: any[]): Promise { + // Capture an entry that actually exists in each log before scanning. `exactStart` can then + // resume by physical position and include transactions that started before this boundary + // but committed after it. A wall-clock timestamp is not a log position. + const boundaries: Array = []; + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + if (!logs[nodeId]) continue; + for (const entry of this.table.auditStore.getRange({ start: 0, log: nodeId })) { + boundaries[nodeId] = entry.txnLogKey; + } + } + this.backend.resetDerivedStorage(); + await this.indexStore.clear(); + let indexed = 0; + const total = this.table.primaryStore.getKeysCount?.() ?? 0; + const startedAt = Date.now(); + for (const { key, value, version, localTime } of this.table.primaryStore.getRange({ + versions: true, + snapshot: false, + })) { + if (!value) continue; + const projected = this.attribute.resolve ? this.attribute.resolve(value) : value[this.attribute.name]; + this.backend.applyDerivedValue(key, projected, localTime ?? version); + indexed++; + if (indexed % REBUILD_PROGRESS_INTERVAL === 0) { + await this.backend.flushDerived(); + const elapsedSeconds = Math.max((Date.now() - startedAt) / 1_000, 0.001); + const rate = Math.round(indexed / elapsedSeconds); + const etaSeconds = rate > 0 ? Math.max(0, Math.ceil((total - indexed) / rate)) : undefined; + logger.info?.(`Rebuilding ${this.indexStore.name}: ${indexed}/${total} records, ${rate}/s, ETA ${etaSeconds}s`); + } + if (indexed % APPLY_BATCH_SIZE === 0) await new Promise((resolve) => setImmediate(resolve)); + } + await this.backend.flushDerived(Math.max(1, ...boundaries.filter((value) => value != null))); + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + const boundary = boundaries[nodeId]; + if (boundary) this.indexStore.putSync(this.cursorKey(nodeId), boundary); + } + if (indexed) { + const elapsedSeconds = Math.max((Date.now() - startedAt) / 1_000, 0.001); + logger.info?.( + `Rebuilt ${this.indexStore.name} from ${indexed} records at ${Math.round(indexed / elapsedSeconds)}/s` + ); + } + } + + private fail(error: unknown): void { + if (this.closed || this.retryScheduled) return; + this.ready = false; + this.indexStore.isIndexing = true; + Atomics.sub(this.sharedDepth, this.sharedDepthOffset, this.pending.size); + Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, this.pendingTickets); + this.pending.clear(); + this.ticketsByOrigin.clear(); + this.pendingTickets = 0; + logger.error?.(`Derived index ${this.indexStore.name} is unavailable and will rebuild`, error); + this.retryScheduled = true; + setTimeout(() => { + this.retryScheduled = false; + void this.initialize(); + }, this.rebuildBackoff).unref(); + this.rebuildBackoff = Math.min(this.rebuildBackoff * 2, MAX_REBUILD_BACKOFF); + } + + requestRebuild(error: unknown): void { + this.fail(error); + } + + close(): void { + if (this.closed) return; + this.closed = true; + Atomics.sub(this.sharedDepth, this.sharedDepthOffset, this.pending.size); + Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, this.pendingTickets); + this.pending.clear(); + this.ticketsByOrigin.clear(); + this.pendingTickets = 0; + this.table.auditStore.removeListener('aftercommit', this.listener); + } +} + +export function attachDerivedIndexBackends(table: any): { close(): void } | undefined { + const runtimes: DerivedIndexRuntime[] = []; + const attachedStores = new Set(); + for (const attribute of table.attributes) { + const indexStore = table.indices[attribute.name]; + const backend = indexStore?.customIndex as DerivedIndexBackend; + if (backend?.postCommit && !attachedStores.has(indexStore)) { + if (table.audit !== true) { + throw new ClientError( + `Table '${table.databaseName}.${table.tableName}' must enable audit logging before using a post-commit derived index` + ); + } + attachedStores.add(indexStore); + const warningKey = `${table.databaseName}.${table.tableName}.${indexStore.name}`; + if (!warnedAuditIndexes.has(warningKey)) { + warnedAuditIndexes.add(warningKey); + logger.warn?.( + `Derived index ${indexStore.name} requires auditing; the audit API retains full record history for the configured retention window` + ); + } + runtimes.push(new DerivedIndexRuntime(table, attribute, indexStore, backend)); + } + } + if (runtimes.length === 0) return; + return { close: () => runtimes.forEach((runtime) => runtime.close()) }; +} diff --git a/resources/RocksTransactionLogStore.ts b/resources/RocksTransactionLogStore.ts index a692687377..e2b4e003fd 100644 --- a/resources/RocksTransactionLogStore.ts +++ b/resources/RocksTransactionLogStore.ts @@ -152,7 +152,7 @@ export class RocksTransactionLogStore extends EventEmitter { } } const entries = options.transaction.logEntries; - if (entries) this.emit('aftercommit', entries); + if (entries) this.emit('aftercommit', entries, options.transaction.derivedIndexTargets); }; } log.addEntry(entryBinary, options.transaction.id); @@ -242,6 +242,7 @@ export class RocksTransactionLogStore extends EventEmitter { getRange(options: { start?: number; exactStart?: boolean; + exclusiveStart?: boolean; end?: number; log?: string | number; excludeLogs?: string[]; diff --git a/resources/Table.ts b/resources/Table.ts index a8ad153bfe..3fdbffc4ac 100644 --- a/resources/Table.ts +++ b/resources/Table.ts @@ -780,6 +780,7 @@ export function makeTable(options) { static tableName = tableName; static tableId = tableId; static indices = indices; + static derivedIndexRuntime: { close(): void } | undefined; static audit = audit; static databasePath = databasePath; static databaseName = databaseName; @@ -6191,6 +6192,7 @@ export function makeTable(options) { /** Release everything makeTable() registered process-wide; the class must not be used afterwards. */ static cleanup() { disposed = true; + TableResource.derivedIndexRuntime?.close(); clearTimeout(cleanupTimer); settlePendingCleanup(); clearInterval(recordExpirationInterval); diff --git a/resources/databases.ts b/resources/databases.ts index 17af3638a6..0ad0abd9a7 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -48,6 +48,7 @@ import { OpenDBIObject } from '../utility/lmdb/OpenDBIObject.ts'; import { RocksDatabase, supportedCompression, type RocksDatabaseOptions } from '@harperfast/rocksdb-js'; import { PrimaryRocksDatabase } from './PrimaryRocksDatabase.ts'; import { replayLogs } from './replayLogs.ts'; +import { attachDerivedIndexBackends } from './DerivedIndexBackend.ts'; import { totalmem } from 'node:os'; import { RocksIndexStore } from './RocksIndexStore.ts'; import { when } from '../utility/when.ts'; @@ -1311,6 +1312,8 @@ function initStores( table.schemaVersion = 1; if (!destination) databaseEventsEmitter.emit('updateTable', table); } + table.derivedIndexRuntime?.close(); + table.derivedIndexRuntime = attachDerivedIndexBackends(table); if (Array.isArray(primaryAttribute.relationships)) { relationshipsToHydrate.push({ table, databaseName, tableName, definitions: primaryAttribute.relationships }); } else if (primaryAttribute.relationships !== undefined) { @@ -2291,6 +2294,15 @@ export function table(tableDefinition: TableDefinition): Tabl // flag must be left as-is. Only an explicit value can re-assert on the existing-Table branch. const schemaDefinedExplicit = tableDefinition.schemaDefined !== undefined; if (schemaDefined == undefined) schemaDefined = true; + if ( + attributes.some((attribute) => attribute.indexed?.type === 'HNSW' && attribute.indexed.nativePlane) && + audit !== true && + Table?.audit !== true + ) { + throw new ClientError( + `Table '${databaseName}.${tableName}' must explicitly enable audit logging before using nativePlane because its transaction log is the derived-index recovery source` + ); + } const relationshipDefinitions = schemaRelationshipsDefined ? normalizeRelationships(attributes) : undefined; const internalDbiInit = createOpenDBIObject(false); @@ -2866,6 +2878,8 @@ export function table(tableDefinition: TableDefinition): Tabl signalling.signalSchemaChange( new SchemaEventMsg(process.pid, 'schema-change', Table.databaseName, Table.tableName) ); + Table.derivedIndexRuntime?.close(); + Table.derivedIndexRuntime = attachDerivedIndexBackends(Table); Table.origin = origin; if (hasChanges || refreshRelationshipAttributes) { diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index c3cf313a9e..115100368e 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -2,9 +2,12 @@ import { closeSync, existsSync, openSync, rmSync, statSync, unlinkSync } from 'n import { cosineDistance, euclideanDistance, dotProductDistance } from './vector.ts'; import { FLOAT32_OPTIONS } from 'msgpackr'; import { loggerWithTag } from '../../utility/logging/logger.ts'; -import { ClientError } from '../../utility/errors/hdbError.ts'; +import { ClientError, ServerError } from '../../utility/errors/hdbError.ts'; import type { Id } from '../../resources/ResourceInterface.ts'; +import type { DerivedIndexRuntime } from '../DerivedIndexBackend.ts'; import { SKIP } from '@harperfast/extended-iterable'; +import { RocksDatabase } from '@harperfast/rocksdb-js'; +import { createHash } from 'node:crypto'; import { getPlaneBinding, invalidatePlaneFile as invalidateHnswPlaneFile, @@ -50,33 +53,6 @@ function dequantizeInt8(q: Int8Array, scale: number): number[] { return out; } -/** - * Connection ids for the plane mirror: ids only — per-edge distances are dropped (recomputed - * natively). A list past `cap` keeps the NEAREST edges rather than an arbitrary array prefix, - * so transient JS overshoot (grace above the cap) and the plane's tighter upper cap truncate by - * the same distance policy the JS prune uses. - */ -function planeConnectionIds(connections: Connection[] | undefined, cap: number): Uint32Array { - if (!connections?.length) return new Uint32Array(0); - let usableCount = 0; - for (const connection of connections) { - if (typeof connection.id === 'number' && connection.id >= 0) usableCount++; - } - if (usableCount > cap) { - // rare (transient overshoot / the tighter upper cap): worth the intermediate copies - const nearest = connections - .filter((connection) => typeof connection.id === 'number' && connection.id >= 0) - .sort((a, b) => a.distance - b.distance); - return Uint32Array.from({ length: cap }, (_, i) => nearest[i].id); - } - const ids = new Uint32Array(usableCount); - let at = 0; - for (const connection of connections) { - if (typeof connection.id === 'number' && connection.id >= 0) ids[at++] = connection.id; - } - return ids; -} - // Auto-scaled search ef, used only when an index does not explicitly configure efConstructionSearch // and a query does not pass its own ef. A fixed ef makes recall decay as the graph grows (it explores // a shrinking fraction of the graph), so ef grows with sqrt(node count) in two regimes, with a @@ -147,9 +123,6 @@ const NODE_COUNT_TTL = 10_000; // rather than silently truncated. maxNodes is a fixed sparse reservation — pages materialize on // write — and ids at or past it are rejected by the crate, which disables the plane. const PLANE_LAYER0_CAP_MAX = 1024; -// Ids kept per upper level — the crate's format-level UPPER_CAP, which truncates whatever is -// passed; pre-sorting to this cap keeps the nearest edges instead of an array prefix. -const PLANE_UPPER_CAP = 64; // matches the crate's UPPER_CAP and the JS graph's M<<2 upper cap const PLANE_MAX_NODES = 1 << 24; // An existing plane file that cannot be opened is normally another worker mid-create (retry); // past this age it is a crashed create and is deleted and rebuilt — the plane is derived state, @@ -158,18 +131,6 @@ const PLANE_STALE_CREATE_MS = 60_000; // Retry cadence while another worker holds the create: its header lands within moments of the // exclusive open, so a long deferral would silently drop this worker's mirror writes. const PLANE_ATTACH_RETRY_MS = 250; -// A plane whose initial mirror never completed (watermark still 0) is never searched; past this -// age the builder is taken as crashed and the file is rebuilt from the CF. -const PLANE_INCOMPLETE_REBUILD_MS = 3_600_000; -// Watermark stamped when the initial full mirror completes; 0 = still building (or crashed -// mid-build). Phase-2 replay wiring will carry real transaction ids, which are also nonzero. -const PLANE_MIRRORED = 1; -// Steady-state durability cadence: a flush barrier every N mirrored mutations, so a crash -// loses a bounded tail (scrubbed + rebuilt on unclean reopen) instead of everything since the -// initial build's flush. -const PLANE_FLUSH_EVERY = 4096; -// Builder yield cadence: nodes mirrored per event-loop turn during the first-enable build. -const PLANE_BUILD_CHUNK = 5_000; // Marks an error thrown by an app-supplied filter during a plane search: the caller re-raises // it as an ordinary query failure instead of disabling the (healthy) plane. const PLANE_PREDICATE_ERROR = Symbol('planePredicateError'); @@ -264,9 +225,7 @@ export class HierarchicalNavigableSmallWorld { // reindex (databases.ts persists the new value but skips rebuilding). efConstructionSearch is the // search-time candidate-list size; the build uses efConstruction/M/distance, which are structural. // filterExpansion is the visit-budget multiplier for predicate-aware (filtered) traversal. - // nativePlane never changes the stored CF graph either: enabling it builds the derived plane - // file by mirroring the existing graph (see getPlane), so a toggle must not force a rebuild. - static searchOnlyOptions = ['efConstructionSearch', 'filterExpansion', 'nativePlane']; + static searchOnlyOptions = ['efConstructionSearch', 'filterExpansion']; // Signals to search.ts that this index accepts a per-record predicate in search() and applies it // during traversal (predicate-aware / ACORN-style filtering), so companion conditions and RBAC can // be pushed down instead of post-filtering an under-filled candidate set (#1241). @@ -306,17 +265,19 @@ export class HierarchicalNavigableSmallWorld { private convertedNodes = new WeakMap(); private nodeCount = 0; private nodeCountAt = 0; - // Native traversal plane (dual-write): the CF graph stays authoritative; every graph mutation - // is mirrored into the plane file and search runs native when the flag is on. + // Native file-primary index. The RocksDB index store holds identity mappings and replay cursors; + // graph nodes and adjacency exist only in this file. // undefined = not yet attached (may retry), null = unavailable or disabled for this process. private plane: HnswPlane | null | undefined; - private planeBuild: Promise | undefined; // in-flight first-enable mirror (tests await it) - private planeMutationsSinceFlush = 0; - private planeFlushInFlight = false; private planeEligible = false; private planeReady = false; private planeRetryAt = 0; private planeDisabledLogged = false; + private filePrimary = false; + private nativePlaneMaxNodes = PLANE_MAX_NODES; + private derivedRuntime?: DerivedIndexRuntime; + private pendingDerivedMappings = new Map(); + postCommit?: true; constructor(indexStore: any, options: any) { this.indexStore = indexStore; if (indexStore) { @@ -348,32 +309,41 @@ export class HierarchicalNavigableSmallWorld { if (options.filterExpansion !== undefined) this.filterExpansion = options.filterExpansion; } if (options?.nativePlane) { + if (!(indexStore?.rootStore instanceof RocksDatabase)) { + throw new ClientError('nativePlane requires the RocksDB storage engine'); + } + const nativeML = 1 / Math.log(16); + if ( + (options.M !== undefined && options.M !== 16) || + (options.efConstruction !== undefined && options.efConstruction !== 200) || + (options.mL !== undefined && options.mL !== nativeML) || + (options.optimizeRouting !== undefined && options.optimizeRouting !== 0.5) + ) { + throw new ClientError('nativePlane requires M=16, efConstruction=200, mL=1/ln(16), and optimizeRouting=0.5'); + } + this.efConstruction = options.efConstruction ?? 200; + this.nativePlaneMaxNodes = options.nativePlaneMaxNodes ?? PLANE_MAX_NODES; + if ( + !Number.isSafeInteger(this.nativePlaneMaxNodes) || + this.nativePlaneMaxNodes < 1 || + this.nativePlaneMaxNodes >= PLANE_NO_ID + ) { + throw new ClientError('nativePlaneMaxNodes must be a positive integer below 2^32-1'); + } // The plane stores int8 bins and computes asymmetric cosine only, so the flag is a // no-op for float (quantization: "none") and non-cosine indexes; a graph whose derived // layer-0 cap exceeds the plane maximum is refused rather than silently truncated. this.planeEligible = - this.int8 && - this.distance === cosineDistance && - this.planeLayer0Cap() <= PLANE_LAYER0_CAP_MAX && - // the plane's fixed upper-level cap must hold this graph's full upper adjacency - // (M<<2 under optimizeRouting) or hierarchy edges would be silently truncated - (this.optimizeRouting ? this.M << 2 : this.M) <= PLANE_UPPER_CAP; + this.int8 && this.distance === cosineDistance && this.planeLayer0Cap() <= PLANE_LAYER0_CAP_MAX; if (!this.planeEligible) { - logger.info?.( - 'nativePlane requires an int8-quantized cosine HNSW index whose M fits the plane geometry; using the JS search path' - ); + throw new ClientError('nativePlane requires an int8-quantized cosine HNSW index'); } + this.filePrimary = true; + this.postCommit = true; } } - /** - * Called by openIndex on the authoritative index instance. With the flag off (or the index - * ineligible) nothing mirrors, so an existing plane file only goes stale — and a later - * re-enable would adopt it; deleting it here makes re-enabling rebuild from the CF (the - * documented rollback: flag off + file deleted). Not done in the constructor: auxiliary - * instances over the same store (e.g. a flag-off reference in tests) must not delete the - * live index's plane. - */ + /** Remove derived native state after the option is disabled. */ cleanupDisabledPlane(): void { if (this.planeEligible) return; const filePath = this.planeFilePath(); @@ -401,17 +371,8 @@ export class HierarchicalNavigableSmallWorld { } /** - * Attach (open or lazily create) the native plane for this index. `dims` must be provided by - * callers that may CREATE the file (a node mirror or a search, which know the vector length); - * without it the call is open-only — if no file exists yet there is nothing to sync, and the - * eventual creation's full mirror reads the then-current CF state. - * - * Multi-worker create races are settled by an exclusive open ('wx') of the file itself: the - * winner creates and mirrors, losers see EEXIST and open (retrying on a short cadence while - * the winner is still writing the header, so their mirror writes drop for at most moments). - * A loser attached mid-build mirrors its own writes immediately but planeSearchReady keeps - * its searches on the JS path until the builder stamps the mirror complete. A crashed create - * leaves an unopenable file; once older than PLANE_STALE_CREATE_MS it is deleted and rebuilt. + * Open or lazily create the native file. An exclusive create resolves multi-worker races; the + * derived-index runtime owns population, replay, and publication. */ private getPlane(dims?: number, dimsFromVector = false): HnswPlane | null { if (this.plane !== undefined) return this.plane; @@ -467,24 +428,14 @@ export class HierarchicalNavigableSmallWorld { return null; } closeSync(fd); - // a populated graph's own dimensionality sizes the file — a caller-supplied dims - // (possibly a malformed search target) must not; the file format pins dims forever - let dimsFromStore = false; - for (const { value } of this.indexStore.getRange({ start: 0, end: Infinity, limit: 1 })) { - const storedVector = value?.level !== undefined ? value.vector : undefined; - if (storedVector) { - dims = Array.isArray(storedVector) ? storedVector.length : storedVector.byteLength; - dimsFromStore = true; - } - } - if (!dimsFromStore && !dimsFromVector) { - // empty graph and the dims came from a search target: a malformed query must not - // pin the file's dimensionality forever — defer creation to the first real write + if (!dimsFromVector) { + // A search target must not pin an empty index's dimensionality forever. Creation is + // deferred to the first committed vector or to the rebuild scan. unlinkSync(filePath); return null; } try { - return (this.plane = this.createAndMirrorPlane(Plane, filePath, dims)); + return (this.plane = Plane.create(filePath, dims, this.planeLayer0Cap(), this.nativePlaneMaxNodes)); } catch (createError) { // never leave a file a later open would trust as a complete mirror try { @@ -502,32 +453,19 @@ export class HierarchicalNavigableSmallWorld { } } - /** - * True once the plane's initial full mirror completed (watermark stamped nonzero at the end - * of createAndMirrorPlane). A plane opened mid-build keeps receiving this worker's mirror - * writes but must not serve searches — its graph is incomplete; one abandoned by a crashed - * builder would stay unusable forever, so past a generous age it is rebuilt from the CF. - */ + /** True only after rebuild/replay has crossed a native durability barrier. */ private planeSearchReady(plane: HnswPlane): boolean { + if (plane.invalidated()) { + this.plane = undefined; + this.planeReady = false; + return false; + } + if (this.indexStore.isIndexing) return false; if (this.planeReady) return true; - if (plane.getWatermark() >= PLANE_MIRRORED) { + if (plane.getWatermark() > 0) { this.planeReady = true; return true; } - const filePath = this.planeFilePath(); - if (filePath) { - try { - // age by creation time: ongoing dual-writes into an abandoned build keep - // refreshing mtime, which would defer this rebuild forever - const stat = statSync(filePath); - if (Date.now() - (stat.birthtimeMs || stat.mtimeMs) > PLANE_INCOMPLETE_REBUILD_MS) { - logger.warn?.('rebuilding an HNSW plane file whose initial mirror never completed'); - this.resetDerivedStorage(); - } - } catch { - // stat raced a concurrent delete; the next attach sorts it out - } - } return false; } @@ -537,199 +475,8 @@ export class HierarchicalNavigableSmallWorld { } /** - * Create the plane file and fully mirror the existing CF graph into it (the "first enable" - * build — a pure copy of the same graph, so plane and CF are bit-identical by construction; - * a reindex would rebuild a different random-level graph at far higher cost). The scan reads - * committed state: mutations committed while it runs mirror themselves through their own - * dual-write calls, though a write racing the scan can transiently be overwritten with the - * scan's older snapshot of that node — it re-syncs on the node's next touch, and the exact - * rescore + record load already filter stale candidates (relaxed adherence, design §5). - */ - private createAndMirrorPlane( - Plane: NonNullable>, - filePath: string, - dims: number - ): HnswPlane { - const plane = Plane.create(filePath, dims, this.planeLayer0Cap(), PLANE_MAX_NODES); - // The build aborts whenever this.plane stops being this handle (disable/reset/replace), - // so the handle must be current BEFORE the builder's first generation check — including - // the fully synchronous single-chunk path. - this.plane = plane; - // The full mirror runs in the background in bounded chunks — a 5M-node synchronous scan - // would freeze this worker's event loop for its duration. Until the builder stamps the - // watermark, planeSearchReady keeps searches on the JS path while live mutations mirror - // write-through; the scan uses writeNodeRawIfAbsent so an older snapshot can never - // overwrite what a concurrent live mirror (this worker's or another's) already wrote. - this.planeBuild = this.buildPlaneMirror(plane).catch((error) => { - this.disablePlane(error); - }); - return plane; - } - - private async buildPlaneMirror(plane: HnswPlane): Promise { - let mirrored = 0; - let nextStart = 0; - let yielded = false; - for (;;) { - let inChunk = 0; - let lastKey = -1; - for (const { key, value } of this.indexStore.getRange({ - start: nextStart, - end: Infinity, - limit: PLANE_BUILD_CHUNK, - })) { - inChunk++; - if (typeof key !== 'number') continue; - lastKey = key; - if (!value || value.level === undefined) continue; - if (this.plane !== plane) return; // disabled, reset, or replaced while building - this.writeNodeToPlane(plane, key, value, true); - mirrored++; - } - if (inChunk < PLANE_BUILD_CHUNK || lastKey < 0) break; - nextStart = lastKey + 1; - // each chunk re-reads current committed state after yielding the event loop - yielded = true; - await new Promise((resolve) => setImmediate(resolve)); - if (this.plane !== plane) return; // disabled, reset, or replaced while building - } - if (this.plane !== plane) return; // never stamp a replacement plane ready - const entryPointId = this.indexStore.getSync(ENTRY_POINT); - if (typeof entryPointId === 'number' && plane.getEntryPoint()[0] === PLANE_NO_ID) { - plane.setEntryPoint(entryPointId, this.safeGetSync(entryPointId)?.level ?? 0); - } - // flush(watermark) is a durability barrier: all slots reach disk before the watermark - // does, so a crash can only under-claim, never adopt a torn mirror as complete. A - // build that fit in one chunk stays fully synchronous (few dirty pages, cheap msync; - // callers see the plane ready immediately); a chunked build already yielded and takes - // the barrier on the libuv pool - if (yielded) { - await plane.flushAsync(PLANE_MIRRORED); - if (this.plane !== plane) return; - } else { - plane.flush(PLANE_MIRRORED); - } - this.planeReady = true; - if (mirrored > 0) logger.info?.(`built the HNSW plane file from ${mirrored} existing graph nodes`); - } - - /** Write one JS graph node's full state into the plane (throws on ineligible node state). */ - private writeNodeToPlane(plane: HnswPlane, nodeId: number, node: any, ifAbsent = false): void { - if (!Number.isInteger(nodeId) || nodeId < 0 || nodeId >= PLANE_NO_ID) { - throw new Error(`node id ${nodeId} is outside the plane's u32 id space`); - } - const vector = node.vector; - let bin: Buffer; - let scale: number; - let invMag: number | undefined = node.invMag; - if (Array.isArray(vector)) { - // legacy float node inside an int8 index: quantize the mirror copy only (the CF node - // is untouched); its distances in the plane are then quantized like every other node - const q = quantizeInt8(vector); - bin = q.bytes; - scale = q.scale; - if (invMag === undefined) { - let magSq = 0; - for (const v of vector) magSq += v * v; - invMag = 1 / (Math.sqrt(magSq) || 1); - } - } else { - // Int8Array (converted) or raw bin view straight from the store decode — same bytes; - // pass them through untouched (never re-quantize) - bin = Buffer.from(vector.buffer, vector.byteOffset, vector.byteLength); - scale = node.scale ?? 1; - if (invMag === undefined) { - // legacy pre-invMag node: |v| ~= scale * |q|, the same fallback searchLayer uses - const q = - vector instanceof Int8Array ? vector : new Int8Array(vector.buffer, vector.byteOffset, vector.byteLength); - let magSq = 0; - for (let i = 0; i < q.length; i++) magSq += q[i] * q[i]; - invMag = 1 / ((Math.sqrt(magSq) || 1) * scale); - } - } - const level = node.level ?? 0; - const layer0 = planeConnectionIds(node[0], plane.layer0Cap); - let upper: Uint32Array[] | null = null; - if (level >= 1) { - upper = []; - for (let l = 1; l <= level; l++) upper.push(planeConnectionIds(node[l], PLANE_UPPER_CAP)); - } - if (ifAbsent) plane.writeNodeRawIfAbsent(nodeId, level, bin, scale, invMag, layer0, upper); - else plane.writeNodeRaw(nodeId, level, bin, scale, invMag, layer0, upper); - } - - /** Mirror a node put into the plane; a plane failure never fails the CF write. */ - private mirrorNodePut(nodeId: number, node: any): void { - if (!this.planeEligible) return; - const vector = node?.vector; - const dims = Array.isArray(vector) ? vector.length : vector?.byteLength; - const plane = this.getPlane(dims, true); - if (!plane) return; - try { - this.writeNodeToPlane(plane, nodeId, node); - this.planeFlushTick(plane); - } catch (error) { - this.disablePlane(error); - } - } - - private mirrorNodeRemove(nodeId: number): void { - if (!this.planeEligible) return; - const plane = this.getPlane(); - if (!plane) return; - try { - plane.clearNode(nodeId); - this.planeFlushTick(plane); - } catch (error) { - this.disablePlane(error); - } - } - - /** - * Bounded-lag durability: a flush barrier every PLANE_FLUSH_EVERY mirrored mutations, - * on the libuv pool — a synchronous whole-map msync would stall this worker's event - * loop for the writeback of a multi-GB mapping. At most one barrier in flight. - */ - private planeFlushTick(plane: HnswPlane): void { - if (++this.planeMutationsSinceFlush >= PLANE_FLUSH_EVERY && !this.planeFlushInFlight) { - this.planeMutationsSinceFlush = 0; - this.planeFlushInFlight = true; - plane - .flushAsync(this.planeReady ? PLANE_MIRRORED : undefined) - .catch((error) => logger.warn?.('HNSW plane flush barrier failed', error)) - .finally(() => { - this.planeFlushInFlight = false; - }); - } - } - - private mirrorEntryPoint(entryPointId: number, level: number | undefined, options?: any): void { - if (!this.planeEligible) return; - const plane = this.getPlane(); - if (!plane) return; - try { - plane.setEntryPoint(entryPointId, level ?? this.safeGetSync(entryPointId, options)?.level ?? 0); - } catch (error) { - this.disablePlane(error); - } - } - - private mirrorEntryPointCleared(): void { - if (!this.planeEligible) return; - const plane = this.getPlane(); - if (!plane) return; - try { - plane.setEntryPoint(PLANE_NO_ID, 0); - } catch (error) { - this.disablePlane(error); - } - } - - /** - * Disable the plane for this process; searches and writes fall back to the JS/CF path. The - * file is deleted too: dual-write stops here, so a mirror kept on disk would be reopened - * after a restart missing every post-disable mutation. Another worker still mapping the old - * inode keeps itself consistent until the schema-change/restart cycle rebuilds everything. + * Disable the file-primary index for this process. Invalidation makes the old mapping unusable + * in peer workers; the runtime keeps search unavailable until a rebuild succeeds. */ private disablePlane(error: unknown): void { const attached = this.plane; @@ -737,6 +484,7 @@ export class HierarchicalNavigableSmallWorld { this.planeReady = false; const filePath = this.planeFilePath(); if (filePath) { + if (this.filePrimary && attached) this.invalidatePlaneFile(filePath, attached); try { unlinkSync(filePath); } catch (unlinkError: any) { @@ -748,8 +496,14 @@ export class HierarchicalNavigableSmallWorld { } if (!this.planeDisabledLogged) { this.planeDisabledLogged = true; - logger.error?.('disabling the HNSW native plane for this index (falling back to the JS path)', error); + logger.error?.( + this.filePrimary + ? 'disabling the HNSW native index until it is rebuilt' + : 'disabling the HNSW native plane for this index (falling back to the JS path)', + error + ); } + if (this.filePrimary) this.derivedRuntime?.requestRebuild(error); } /** @@ -760,12 +514,14 @@ export class HierarchicalNavigableSmallWorld { * database instances. */ resetDerivedStorage(): void { + this.pendingDerivedMappings.clear(); const attached = this.plane; this.plane = undefined; this.planeReady = false; this.planeRetryAt = 0; const filePath = this.planeFilePath(); if (!filePath) return; + if (this.filePrimary && attached) this.invalidatePlaneFile(filePath, attached); try { unlinkSync(filePath); } catch (error: any) { @@ -779,7 +535,7 @@ export class HierarchicalNavigableSmallWorld { } } - /** Make an undeletable plane unadoptable before this process stops mirroring it. */ + /** Make an undeletable plane unadoptable before this process releases it. */ private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { try { invalidateHnswPlaneFile(filePath, attached); @@ -841,7 +597,9 @@ export class HierarchicalNavigableSmallWorld { } const entries: any[] = []; for (const hit of hits) { - const primaryKey = this.safeGetSync(hit.id, options)?.primaryKey; + const mapping = this.safeGetSync(hit.id, options); + if (mapping?.pending) continue; + const primaryKey = mapping?.primaryKey; if (primaryKey === undefined) continue; // deleted/reused id raced the search entries.push({ key: primaryKey, distance: hit.distance }); } @@ -851,7 +609,96 @@ export class HierarchicalNavigableSmallWorld { }); } + attachDerivedRuntime(runtime: DerivedIndexRuntime): void { + this.derivedRuntime = runtime; + } + + prepareCommitted(primaryKey: Id, vector: number[], existingVector: number[], options: any): void { + this.validateVector(primaryKey, vector); + this.derivedRuntime?.stage(options.transaction, primaryKey, vector, existingVector); + } + + private validateVector(primaryKey: Id, vector?: number[]): void { + if (!vector) return; + for (let i = 0; i < vector.length; i++) { + if (!Number.isFinite(vector[i])) { + throw new ClientError( + `Vector for attribute "${String(primaryKey)}" contains non-finite component at index ${i}: ${vector[i]}. Ensure the embedding produces only finite values.` + ); + } + } + } + + applyDerivedValue(primaryKey: Id, vector: number[], version?: number): void { + this.validateVector(primaryKey, vector); + const safeKey = typeof primaryKey === 'number' ? [KEY_PREFIX, primaryKey] : primaryKey; + const storedMapping = this.pendingDerivedMappings.get(primaryKey) ?? this.indexStore.getSync(safeKey); + const oldNodeId = typeof storedMapping === 'number' ? storedMapping : storedMapping?.id; + if (storedMapping?.version != null && version != null && storedMapping.version > version) return; + const nativeVector = vector ? Float32Array.from(vector) : undefined; + const signature = nativeVector + ? createHash('sha256') + .update(Buffer.from(nativeVector.buffer, nativeVector.byteOffset, nativeVector.byteLength)) + .digest('base64url') + : undefined; + if (oldNodeId != null && signature && storedMapping.signature === signature && !storedMapping.pending) { + this.indexStore.putSync(safeKey, { id: oldNodeId, signature, version }); + this.indexStore.putSync(oldNodeId, { primaryKey, version }); + return; + } + let plane = vector ? this.getPlane(vector.length, true) : this.getPlane(); + if (plane?.invalidated()) { + this.plane = undefined; + plane = vector ? this.getPlane(vector.length, true) : this.getPlane(); + } + if (oldNodeId != null) { + plane?.remove(oldNodeId); + this.indexStore.removeSync(oldNodeId); + } + if (!vector) { + this.indexStore.removeSync(safeKey); + this.pendingDerivedMappings.set(primaryKey, { version }); + return; + } + if (!plane) throw new ServerError('The native HNSW module is unavailable for a file-primary index', 503); + const nodeId = plane.insert(nativeVector!); + this.indexStore.putSync(safeKey, { id: nodeId, signature, version, pending: true }); + this.indexStore.putSync(nodeId, { primaryKey, version, pending: true }); + this.pendingDerivedMappings.set(primaryKey, { id: nodeId, signature, version }); + } + + async flushDerived(watermark?: number): Promise { + const plane = this.getPlane(); + if (!plane) { + if (!getPlaneBinding()) throw new ServerError('The native HNSW module is unavailable', 503); + return this.publishDerivedMappings(); + } + await plane.flushAsync(watermark); + this.publishDerivedMappings(); + this.planeReady = true; + } + + private publishDerivedMappings(): void { + for (const [primaryKey, mapping] of this.pendingDerivedMappings) { + const safeKey = typeof primaryKey === 'number' ? [KEY_PREFIX, primaryKey] : primaryKey; + if (mapping.id === undefined) { + this.indexStore.removeSync(safeKey); + } else { + this.indexStore.putSync(safeKey, mapping); + this.indexStore.putSync(mapping.id, { primaryKey, version: mapping.version }); + } + } + this.pendingDerivedMappings.clear(); + } + index(primaryKey: Id, vector: number[], existingVector?: number[], options: any = {}) { + if (this.filePrimary) { + if (options.transaction) return this.prepareCommitted(primaryKey, vector, existingVector, options); + // runIndexing invokes custom indexes without a transaction. The shared runtime waits for + // that schema scan, then owns one primary-record rebuild plus log replay; populating here + // would duplicate native construction and race the runtime's generation reset. + return; + } // Reject non-finite components before touching the graph. NaN in particular poisons // bisectInsert (arr[mid].distance <= NaN is always false → returns 0, pinning the // candidate to rank 1 of every future search). Infinity causes analogous ordering @@ -943,15 +790,12 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', nodeId); this.indexStore.put(ENTRY_POINT, nodeId, options); - this.mirrorNodePut(nodeId, node); - this.mirrorEntryPoint(nodeId, level, options); return; } // Generate random level for this new element const level = oldNode.level ?? Math.min(Math.floor(-Math.log(this.random()) * this.mL), MAX_LEVEL); let currentLevel = entryPoint.level; - let mirrorEntryPointAfterPut = false; if (level > currentLevel) { // if we are at a higher level, make this the new entry point if (typeof nodeId !== 'number') { @@ -959,10 +803,6 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', nodeId); this.indexStore.put(ENTRY_POINT, nodeId, options); - // the CF put is invisible until commit, but a plane write is immediately visible — - // mirror the promotion only after the node's own slot lands (below), so a concurrent - // native search never descends from a not-yet-written entry slot - mirrorEntryPointAfterPut = true; } // Pure descent — only neighbors[0] is used — so it runs greedily for the same reason @@ -1109,8 +949,6 @@ export class HierarchicalNavigableSmallWorld { ...connections, }; this.indexStore.put(nodeId, storedNode, options); - this.mirrorNodePut(nodeId, storedNode); - if (mirrorEntryPointAfterPut) this.mirrorEntryPoint(nodeId, level, options); } else { // removal of this node, but first make sure we have a valid entry point if (entryPointId === nodeId) { @@ -1144,7 +982,6 @@ export class HierarchicalNavigableSmallWorld { if (entryPointId === undefined) { // no nodes left in index this.indexStore.remove(ENTRY_POINT, options); - this.mirrorEntryPointCleared(); } else { // set the new entry point if (typeof entryPointId !== 'number') { @@ -1152,11 +989,9 @@ export class HierarchicalNavigableSmallWorld { } logger.debug?.('setting entry point to', entryPointId); this.indexStore.put(ENTRY_POINT, entryPointId, options); - this.mirrorEntryPoint(entryPointId, undefined, options); } } this.indexStore.remove(nodeId, options); - this.mirrorNodeRemove(nodeId); // A re-insert of this primary key must get a fresh node rather than the deleted node's id. this.indexStore.remove(safeKey, options); } @@ -1212,7 +1047,6 @@ export class HierarchicalNavigableSmallWorld { } for (const [id, updatedNode] of updatedNodes) { this.indexStore.put(id, updatedNode, options); - this.mirrorNodePut(id, updatedNode); } for (const [key, orphanVector] of needsReindexing) { // If the orphan IS the current entry point, re-running @@ -1246,7 +1080,6 @@ export class HierarchicalNavigableSmallWorld { } if (replacementEP !== undefined) { this.indexStore.put(ENTRY_POINT, replacementEP, options); - this.mirrorEntryPoint(replacementEP, undefined, options); } } this.index(key, orphanVector, orphanVector, options); @@ -1759,6 +1592,9 @@ export class HierarchicalNavigableSmallWorld { filterEvaluations: 0, } : undefined; + if (this.filePrimary && distanceFunction !== this.distance) { + throw new ClientError('A nativePlane index only supports its configured cosine distance'); + } // The plane traverses the index's own metric (cosine — the eligibility requirement), so a // query overriding `distance` has to take the JS path: rescoreResults only corrects the // reported distances of whatever candidates came back, not which candidates the beam kept. @@ -1776,6 +1612,7 @@ export class HierarchicalNavigableSmallWorld { if (error?.[PLANE_PREDICATE_ERROR]) throw error; // a failed native search disables the plane and re-runs this query on the JS path this.disablePlane(error); + if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); return this.search( { target, value, descending, distance, comparator, ef, filterExpansion }, context, @@ -1786,9 +1623,11 @@ export class HierarchicalNavigableSmallWorld { } catch (error) { // a synchronous NAPI throw (before any promise exists) degrades to the JS path below this.disablePlane(error); + if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); } } } + if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); let entryPoint = this.getEntryPoint(options); if (!entryPoint) return withStats([], filterState); let entryPointId = entryPoint.id; diff --git a/resources/indexes/hnswPlaneBinding.ts b/resources/indexes/hnswPlaneBinding.ts index 9727d1d898..d11a256803 100644 --- a/resources/indexes/hnswPlaneBinding.ts +++ b/resources/indexes/hnswPlaneBinding.ts @@ -9,14 +9,12 @@ export interface PlaneSearchHit { distance: number; } -/** - * NAPI surface of the native HNSW traversal plane (`@harperfast/hnsw`). Dual-write phase 1 uses - * only the raw mirroring calls (host-allocated ids; the plane's own insert()/remove() allocator - * path is bypassed by design) plus the search entry points. - */ +/** NAPI surface of the native file-primary HNSW index (`@harperfast/hnsw`). */ export interface HnswPlane { readonly dims: number; readonly layer0Cap: number; + insert(vector: Float32Array): number; + remove(id: number): void; writeNodeRaw( id: number, level: number, @@ -117,7 +115,7 @@ function getHnswPackage(): HnswPlanePackage | null { binding = null; logger.warn?.( `The @harperfast/hnsw native module is not available (${(error as Error).message}); ` + - 'indexes with nativePlane enabled will use the JS search path' + 'indexes with nativePlane enabled will remain unavailable until the module can load' ); } return binding; @@ -128,7 +126,7 @@ export function getPlaneBinding(): HnswPlaneConstructor | null { return getHnswPackage()?.Plane ?? null; } -/** Make a derived plane unadoptable before mirroring stops. */ +/** Make a derived plane unadoptable before it is replaced or removed. */ export function invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): PlaneInvalidationOutcome { if (attached) return attached.invalidateFile(); const hnswPackage = getHnswPackage(); diff --git a/unitTests/resources/vectorIndexPlane-thread.js b/unitTests/resources/vectorIndexPlane-thread.js new file mode 100644 index 0000000000..94215e03b8 --- /dev/null +++ b/unitTests/resources/vectorIndexPlane-thread.js @@ -0,0 +1,46 @@ +require('../testUtils'); +const { parentPort } = require('node:worker_threads'); +const { setupTestDBPath } = require('../testUtils'); +const { table } = require('#src/resources/databases'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +if (parentPort) { + setupTestDBPath(); + setMainIsWorker(true); + const PlaneTest = table({ + table: 'PlaneTest', + database: 'vector-plane', + audit: true, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { + name: 'vector', + indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, + type: 'Array', + }, + ], + }); + + async function waitUntilReady() { + await PlaneTest.indexingOperation; + while (PlaneTest.indices.vector.isIndexing) await new Promise((resolve) => setTimeout(resolve, 10)); + } + + void waitUntilReady().then(() => + parentPort.postMessage({ + type: 'ready', + retainedMarker: PlaneTest.indices.vector.getSync('__native-plane-reopen-marker__'), + }) + ); + parentPort.on('message', async (message) => { + if (message.type === 'shutdown') process.exit(0); + if (message.type !== 'put') return; + try { + for (const record of message.records) await PlaneTest.put(record.id, record); + parentPort.postMessage({ type: 'done' }); + } catch (error) { + parentPort.postMessage({ type: 'error', message: error.message, stack: error.stack }); + } + }); +} diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index c5c02af83c..a9deb815fe 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -1,20 +1,13 @@ -/** - * Coverage for the native HNSW traversal plane, phase 1 (dual-write + opt-in search cutover, - * hnsw-native-plane.md §8): with `nativePlane: true` every graph mutation is mirrored into a - * plane file next to the index store and searches run through the native module, while the - * RocksDB column family stays authoritative. The plane graph must be a bit-identical mirror of - * the CF graph (same ids/levels/edges), so a native search over it must return the same - * candidates as the JS traversal of the CF graph at equal ef. - * - * The whole suite is skipped when the optional native package is absent. - */ require('../testUtils'); const assert = require('node:assert'); const fs = require('node:fs'); +const { Worker } = require('node:worker_threads'); const { setupTestDBPath } = require('../testUtils'); +const { waitFor } = require('../waitFor'); const { table, resetDatabases } = require('#src/resources/databases'); -const { HierarchicalNavigableSmallWorld } = require('#src/resources/indexes/HierarchicalNavigableSmallWorld'); -const { getPlaneBinding, planeStalePathFor } = require('#src/resources/indexes/hnswPlaneBinding'); +const { DatabaseTransaction } = require('#src/resources/DatabaseTransaction'); +const { getPlaneBinding } = require('#src/resources/indexes/hnswPlaneBinding'); +const { derivedIndexCursorKey } = require('#src/resources/DerivedIndexBackend'); const { setMainIsWorker } = require('#js/server/threads/manageThreads'); async function fromAsync(iterable) { @@ -24,160 +17,203 @@ async function fromAsync(iterable) { } const DIMS = 24; -const N = 1200; +const N = 500; const EF = 200; -const DB = 'test'; - -// Deterministic clustered corpus: 20 well-separated centers plus per-vector noise, so graphs are -// meaningful (uniform-random high-dim corpora defeat ANN) and runs are reproducible. +const DB = 'vector-plane'; +const RETAINED_MARKER = '__native-plane-reopen-marker__'; let seedState = 42; function rand() { seedState = (seedState * 1103515245 + 12345) % 2147483648; return seedState / 2147483648; } -const centers = []; -for (let c = 0; c < 20; c++) { - const center = new Array(DIMS); - for (let d = 0; d < DIMS; d++) center[d] = rand() * 2 - 1; - centers.push(center); -} +const centers = Array.from({ length: 20 }, () => Array.from({ length: DIMS }, () => rand() * 2 - 1)); function makeVector(i) { const center = centers[i % centers.length]; - const v = new Array(DIMS); - for (let d = 0; d < DIMS; d++) v[d] = center[d] + (rand() - 0.5) * 0.2; - return v; + return center.map((value) => value + (rand() - 0.5) * 0.2); +} +function cosineDistance(a, b) { + let dot = 0; + let aa = 0; + let bb = 0; + for (let i = 0; i < a.length; i++) { + dot += a[i] * b[i]; + aa += a[i] * a[i]; + bb += b[i] * b[i]; + } + return 1 - dot / Math.sqrt(aa * bb); } -describe('HNSW native plane dual-write', function () { - if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; // custom object index is RocksDB-only here +describe('HNSW native plane file-primary delivery', function () { + if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; if (!getPlaneBinding()) { it.skip('skipped: @harperfast/hnsw native module is unavailable', () => {}); return; } + this.timeout(30_000); let PlaneTest; - const vectors = new Map(); // id → current vector - before(async () => { - setupTestDBPath(); - setMainIsWorker(true); - PlaneTest = table({ + const vectors = new Map(); + + function defineTable() { + return table({ table: 'PlaneTest', database: DB, + audit: true, attributes: [ { name: 'id', isPrimaryKey: true }, { name: 'name', indexed: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + { + name: 'vector', + indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, + type: 'Array', + }, ], }); - for (let i = 0; i < N; i++) { - const vector = makeVector(i); - vectors.set(i, vector); - await PlaneTest.put(i, { name: 'rec' + i, vector }); - } - }); - + } function customIndex() { return PlaneTest.indices.vector.customIndex; } - - // A flag-off HierarchicalNavigableSmallWorld over the SAME index store: the pure JS traversal - // of the CF graph, the reference the plane must match. - function jsReference() { - return new HierarchicalNavigableSmallWorld(PlaneTest.indices.vector, {}); + async function nativeSearch(target, filter) { + return await customIndex().search( + { target, comparator: 'sort', distance: 'cosine', ef: EF }, + { transaction: undefined }, + filter + ); } - - async function searchBoth(target, filter) { - const condition = { target, comparator: 'sort', distance: 'cosine', ef: EF }; - const planeResult = customIndex().search(condition, { transaction: undefined }, filter); - assert.equal(typeof planeResult?.then, 'function', 'the flagged index should search through the plane (async)'); - const planeEntries = await planeResult; - const jsEntries = jsReference().search(condition, { transaction: undefined }, filter); - assert.equal(typeof jsEntries?.then, 'undefined', 'the reference instance must use the JS path'); - return { planeEntries, jsEntries }; + async function readySearch(target, filter) { + return waitFor( + async () => { + if (PlaneTest.indices.vector.isIndexing) return false; + try { + return await nativeSearch(target, filter); + } catch (error) { + if (/rebuilding/.test(error.message)) return false; + throw error; + } + }, + { timeout: 15_000, message: 'native plane did not become searchable' } + ); } - - // Same candidate set; same order wherever consecutive distances are distinct (the plane - // computes f32 distances vs the JS f64, so exact ties may swap — and post-load rescoring - // restores exact order for real queries anyway). - function assertParity(planeEntries, jsEntries) { - const planeKeys = planeEntries.map((e) => e.key); - const jsKeys = jsEntries.map((e) => e.key); - assert.deepEqual( - [...planeKeys].sort((a, b) => a - b), - [...jsKeys].sort((a, b) => a - b), - 'plane and JS searches must return the same candidate set' + async function waitForKey(id, target) { + return waitFor( + async () => { + const results = await readySearch(target); + return results.some((entry) => entry.key === id); + }, + { timeout: 15_000, message: `native plane did not index record ${id}` } + ); + } + async function waitForCursors() { + return waitFor( + () => { + const logs = PlaneTest.auditStore.loadLogs(); + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + if (!logs[nodeId]) continue; + let latest; + for (const entry of PlaneTest.auditStore.getRange({ start: 0, log: nodeId })) latest = entry.txnLogKey; + if ( + latest !== undefined && + PlaneTest.indices.vector.getSync(derivedIndexCursorKey(PlaneTest.indices.vector.name, nodeId)) !== latest + ) + return false; + } + return true; + }, + { timeout: 15_000, message: 'native plane cursors did not reach the retained audit-log tail' } ); - for (let i = 0; i < planeEntries.length; i++) { - if (planeKeys[i] === jsKeys[i]) continue; - const distanceGap = Math.abs(planeEntries[i].distance - jsEntries[i].distance); - assert.ok( - distanceGap < 1e-4, - `order diverged at rank ${i} (${planeKeys[i]} vs ${jsKeys[i]}) with distance gap ${distanceGap}` - ); - } } - it('dual-writes into a plane file next to the index store', () => { - const planePath = customIndex().planeFilePath(); - assert.ok(planePath, 'the index should resolve a plane file path'); - assert.ok(fs.existsSync(planePath), 'inserts through the flagged index should have created the plane file'); + before(async () => { + setupTestDBPath(); + setMainIsWorker(true); + PlaneTest = defineTable(); + await PlaneTest.indexingOperation; + const firstVector = makeVector(0); + vectors.set(0, firstVector); + await PlaneTest.put(0, { name: 'rec0', vector: firstVector }); + await waitForKey(0, firstVector); + await waitForCursors(); + for (let i = 1; i < N; i++) { + const vector = makeVector(i); + vectors.set(i, vector); + await PlaneTest.put(i, { name: `rec${i}`, vector }); + } + await waitForKey(3, vectors.get(3)); + await waitFor( + () => { + let mappings = 0; + for (const { key } of PlaneTest.indices.vector.getRange()) if (typeof key === 'number') mappings++; + return mappings >= N; + }, + { timeout: 15_000, message: 'post-commit native delivery did not drain' } + ); + await waitForCursors(); }); - it('plane search returns the same candidates as the JS path at equal ef', async () => { - for (const probe of [3, 77, 500]) { - const { planeEntries, jsEntries } = await searchBoth(vectors.get(probe)); - assert.ok(planeEntries.length >= 100, `expected a full candidate list, got ${planeEntries.length}`); - assertParity(planeEntries, jsEntries); - assert.equal(planeEntries[0].key, probe, 'the probe vector should be its own nearest neighbor'); + it('stores only primary-key mappings and cursors in RocksDB', () => { + assert.ok(fs.existsSync(customIndex().planeFilePath())); + let mappings = 0; + for (const { key, value } of PlaneTest.indices.vector.getRange()) { + if (typeof key !== 'number') continue; + mappings++; + assert.equal(value.level, undefined, 'the CF must not retain HNSW graph nodes'); + assert.equal(value.vector, undefined, 'the CF must not retain graph vectors'); + assert.equal(value.pending, undefined, 'published mappings must follow the native durability barrier'); + assert.notEqual(value.primaryKey, undefined, 'numeric entries are native-id to primary-key mappings'); } + assert.ok(mappings >= N); }); - // The plane only ever traverses the index's own metric, so a query asking for a different one - // has to fall back: rescoring fixes the reported distances of the candidates it is handed, not - // which candidates a cosine beam kept. - it('a query overriding the distance metric takes the JS path rather than a cosine traversal', () => { - const condition = { target: vectors.get(3), comparator: 'sort', distance: 'euclidean', ef: EF }; - const flagged = customIndex().search(condition, { transaction: undefined }); - assert.equal(typeof flagged?.then, 'undefined', 'a euclidean query must not be answered by the cosine plane'); - const jsEntries = jsReference().search(condition, { transaction: undefined }); - assert.deepEqual( - flagged.map((entry) => entry.key), - jsEntries.map((entry) => entry.key), - 'an overridden metric must return exactly what the JS traversal returns' - ); + it('builds searchable native state with deterministic recall', async () => { + for (const probe of [3, 77, 300]) { + const entries = await nativeSearch(vectors.get(probe)); + assert.equal(entries[0].key, probe); + const expected = [...vectors] + .sort((a, b) => cosineDistance(vectors.get(probe), a[1]) - cosineDistance(vectors.get(probe), b[1])) + .slice(0, 10) + .map(([id]) => id); + const returned = new Set(entries.slice(0, 20).map((entry) => entry.key)); + assert.ok( + expected.filter((id) => returned.has(id)).length >= 9, + 'recall@10 in the first 20 must be at least 0.9' + ); + } }); - it('parity holds after update-in-place and delete (including neighbor repair)', async () => { - for (let i = 0; i < 100; i++) { - const vector = makeVector(i + 5000); - vectors.set(i, vector); - await PlaneTest.put(i, { name: 'rec' + i, vector }); - } - for (let i = 100; i < 200; i++) { - vectors.delete(i); - await PlaneTest.delete(i); - } - for (const probe of [0, 50, 300]) { - const { planeEntries, jsEntries } = await searchBoth(vectors.get(probe)); - assertParity(planeEntries, jsEntries); - for (const entry of planeEntries) { - assert.ok(entry.key < 100 || entry.key >= 200, `deleted record ${entry.key} returned by the plane`); - } - } + it('does not publish an aborted transaction to the plane', async () => { + const vector = makeVector(50_000); + const plane = customIndex().getPlane(); + const highWater = plane.idHighWater(); + const context = { transaction: new DatabaseTransaction() }; + await PlaneTest.put(50_000, { name: 'aborted', vector }, context); + context.transaction.abort(); + assert.equal(await PlaneTest.get(50_000), undefined); + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(plane.idHighWater(), highWater, 'an aborted write must allocate no native node'); }); - it('predicate-filtered plane search returns only predicate-passing records', async () => { - const filter = (primaryKey) => primaryKey % 3 === 0; - const { planeEntries, jsEntries } = await searchBoth(vectors.get(21), filter); - assert.ok(planeEntries.length > 0, 'filtered plane search should return results'); - for (const entry of planeEntries) { - assert.equal(entry.key % 3, 0, `record ${entry.key} does not pass the predicate`); - } - // budget/pipelining semantics differ slightly under selective filters, so assert the head - // of the ranking agrees rather than the full candidate set - assert.equal(planeEntries[0].key, jsEntries[0].key, 'best filtered match should agree with the JS path'); + it('ignores unrelated field changes and applies committed update/delete', async () => { + const plane = customIndex().getPlane(); + const highWater = plane.idHighWater(); + await PlaneTest.put(3, { name: 'renamed', vector: vectors.get(3) }); + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(plane.idHighWater(), highWater, 'an unchanged vector must schedule no native insert'); + + const updated = makeVector(60_003); + vectors.set(3, updated); + await PlaneTest.put(3, { name: 'renamed', vector: updated }); + await waitForKey(3, updated); + await PlaneTest.delete(77); + vectors.delete(77); + await waitFor(async () => !(await readySearch(updated)).some((entry) => entry.key === 77), { + timeout: 10_000, + message: 'deleted native id remained searchable', + }); }); - it('full-stack query runs through the plane and rescoring restores exact order', async () => { + it('applies predicates and full-stack exact rescoring', async () => { + const filtered = await readySearch(vectors.get(21), (id) => id % 3 === 0); + assert.ok(filtered.length > 0); + for (const entry of filtered) assert.equal(entry.key % 3, 0); const target = vectors.get(42); const results = await fromAsync( PlaneTest.search({ @@ -186,13 +222,9 @@ describe('HNSW native plane dual-write', function () { limit: 10, }) ); - assert.equal(results.length, 10); - assert.equal(results[0].id, 42, 'the probe vector should be its own nearest neighbor'); - for (let i = 1; i < results.length; i++) { - assert.ok(results[i].$distance >= results[i - 1].$distance, 'rescored results must be ordered'); - } - // conditions alongside the vector sort exercise the predicate pushdown through search.ts - const filtered = await fromAsync( + assert.equal(results[0].id, 42); + for (let i = 1; i < results.length; i++) assert.ok(results[i].$distance >= results[i - 1].$distance); + const withCondition = await fromAsync( PlaneTest.search({ sort: { attribute: 'vector', target, distance: 'cosine' }, conditions: [{ attribute: 'name', comparator: 'gt', value: 'rec9' }], @@ -200,10 +232,8 @@ describe('HNSW native plane dual-write', function () { limit: 20, }) ); - assert.ok(filtered.length > 0); - for (const record of filtered) assert.ok(record.name > 'rec9'); - // threshold comparator: int8 suppresses the traversal-time limit and rescoreResults - // re-filters on exact distances post-load — via the plane path + assert.ok(withCondition.length > 0); + for (const record of withCondition) assert.ok(record.name > 'rec9'); const within = await fromAsync( PlaneTest.search({ conditions: [{ attribute: 'vector', comparator: 'le', value: 0.05, target }], @@ -214,54 +244,47 @@ describe('HNSW native plane dual-write', function () { for (const record of within) assert.ok(record.$distance <= 0.05, `distance ${record.$distance} exceeds threshold`); }); - it('a throwing app filter surfaces as the query error without disabling the plane', async () => { - const condition = { target: vectors.get(3), comparator: 'sort', distance: 'cosine', ef: EF }; + it('surfaces a throwing app filter without disabling the native plane', async () => { + const target = vectors.get(3); await assert.rejects( - Promise.resolve( - customIndex().search(condition, { transaction: undefined }, () => { - throw new Error('filter boom'); - }) - ), + nativeSearch(target, () => { + throw new Error('filter boom'); + }), /filter boom/ ); - const after = customIndex().search(condition, { transaction: undefined }); - assert.equal(typeof after?.then, 'function', 'the plane must stay enabled after an app-filter throw'); - assert.ok((await after).length > 0); + assert.ok((await readySearch(target)).length > 0); }); - it('synchronous iteration of plane-backed results fails loudly instead of spinning', () => { + it('rejects synchronous iteration of asynchronous plane-backed results', () => { const results = PlaneTest.search({ sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, select: ['id'], limit: 5, }); - assert.throws(() => [...results], /async/i, 'sync iteration must throw, not loop on promise-shaped results'); + assert.throws(() => [...results], /async/i, 'sync iteration must throw instead of spinning'); }); - it('overlapping next() calls on plane-backed results advance one shared cursor', async () => { + it('serializes overlapping next calls on one plane-backed result cursor', async () => { const query = () => ({ sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, select: ['id'], limit: 5, }); const sequential = (await fromAsync(PlaneTest.search(query()))).map((record) => record.id); - assert.ok(sequential.length > 2, 'need several results to detect a duplicate or a skip'); + assert.ok(sequential.length > 2, 'need several results to detect a duplicate or skip'); const iterator = PlaneTest.search(query()).iterate({ async: true }); - // both issued before the first resolves: building an iterator per call returned entry 0 - // twice and dropped entry 1 const [first, second] = await Promise.all([iterator.next(), iterator.next()]); const seen = [first.value.id, second.value.id]; for (let next = await iterator.next(); !next.done; next = await iterator.next()) seen.push(next.value.id); - assert.deepEqual(seen, sequential, 'overlapping next() calls must yield the sequential order exactly once'); + assert.deepEqual(seen, sequential, 'overlapping next calls must yield the sequential order exactly once'); }); - it('an abandoned plane-backed iterable does not raise an unhandled rejection', async () => { + it('does not raise an unhandled rejection when a plane-backed iterable is abandoned', async () => { const index = customIndex(); const unhandled = []; const onUnhandled = (reason) => unhandled.push(reason); process.on('unhandledRejection', onUnhandled); try { - // a post-load step that throws is the reachable way the pending pipeline rejects index.rescoreResults = () => { throw new Error('rescore boom'); }; @@ -270,7 +293,6 @@ describe('HNSW native plane dual-write', function () { select: ['id'], limit: 5, }); - // aborted request / limit 0: the consumer walks away without a single next() await results.iterate({ async: true }).return(); await new Promise((resolve) => setTimeout(resolve, 100)); } finally { @@ -284,206 +306,115 @@ describe('HNSW native plane dual-write', function () { ); }); - it('reopens the same plane file across a restart', async () => { - const planePath = customIndex().planeFilePath(); - const inodeBefore = fs.statSync(planePath).ino; - resetDatabases(); - PlaneTest = table({ - table: 'PlaneTest', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'name', indexed: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], - }); - const { planeEntries, jsEntries } = await searchBoth(vectors.get(7)); - assertParity(planeEntries, jsEntries); - assert.equal(fs.statSync(planePath).ino, inodeBefore, 'restart should reopen the plane file, not recreate it'); - }); - - it('builds the plane lazily when the flag is enabled on an existing index', async () => { - let Later = table({ - table: 'PlaneLater', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'vector', indexed: { type: 'HNSW' }, type: 'Array' }, - ], + it('serializes post-commit delivery from two workers into one native file', async () => { + PlaneTest.indices.vector.putSync(RETAINED_MARKER, true); + const worker = new Worker(require.resolve('./vectorIndexPlane-thread.js'), { + workerData: { workerIndex: 1, workerCount: 2 }, }); - const laterVectors = new Map(); - for (let i = 0; i < 300; i++) { - const vector = makeVector(i + 9000); - laterVectors.set(i, vector); - await Later.put(i, { vector }); - } - assert.ok( - !Later.indices.vector.customIndex.planeFilePath() || - !fs.existsSync(Later.indices.vector.customIndex.planeFilePath()), - 'no plane file before the flag is enabled' - ); - resetDatabases(); - Later = table({ - table: 'PlaneLater', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], - }); - assert.ok(!Later.indexingOperation, 'nativePlane is search-only: enabling it must not trigger a reindex'); - const condition = { target: laterVectors.get(5), comparator: 'sort', distance: 'cosine', ef: EF }; - const planeEntries = await Later.indices.vector.customIndex.search(condition, { transaction: undefined }); - const jsEntries = new HierarchicalNavigableSmallWorld(Later.indices.vector, {}).search(condition, { - transaction: undefined, - }); - assert.ok(fs.existsSync(Later.indices.vector.customIndex.planeFilePath()), 'first search should build the plane'); - assert.deepEqual( - planeEntries.map((e) => e.key).sort((a, b) => a - b), - jsEntries.map((e) => e.key).sort((a, b) => a - b), - 'the lazily-mirrored plane must return the same candidate set' - ); - await Later.dropTable(); - }); - - it('a plane whose initial mirror never completed is not searched', async () => { - const condition = { target: vectors.get(11), comparator: 'sort', distance: 'cosine', ef: EF }; - const index = customIndex(); - const plane = index.getPlane(); - assert.ok(plane, 'the plane should be attached'); - plane.setWatermark(0); // simulate a crashed/incomplete initial mirror - index.planeReady = false; - const jsResults = index.search(condition, { transaction: undefined }); - assert.equal(typeof jsResults?.then, 'undefined', 'an incomplete mirror must fall back to the JS path'); - assert.ok(jsResults.length > 0); - plane.setWatermark(1); - const planeResults = index.search(condition, { transaction: undefined }); - assert.equal(typeof planeResults?.then, 'function', 'a completed mirror serves searches again'); - await planeResults; - }); - - it('an unopenable plane file degrades to the JS path without erroring', async () => { - const Foreign = table({ - table: 'PlaneForeign', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], - }); - const index = Foreign.indices.vector.customIndex; - fs.writeFileSync(index.planeFilePath(), 'not a plane'); // e.g. a crashed create's leftovers - for (let i = 0; i < 20; i++) await Foreign.put(i, { vector: makeVector(i + 20000) }); - const results = index.search( - { target: makeVector(20003), comparator: 'sort', distance: 'cosine', ef: 50 }, - { transaction: undefined } - ); - assert.equal(typeof results?.then, 'undefined', 'writes and searches must run on the JS path meanwhile'); - assert.ok(results.length > 0); - await Foreign.dropTable(); - }); - - it('an undeletable plane is invalidated both in band and through a durable sidecar', async () => { - const Undeletable = table({ - table: 'PlaneUndeletable', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], - }); - const index = Undeletable.indices.vector.customIndex; - for (let i = 0; i < 30; i++) await Undeletable.put(i, { vector: makeVector(i + 60000) }); - const planePath = index.planeFilePath(); - const stalePath = planeStalePathFor(planePath); - fs.rmSync(stalePath, { force: true }); - const plane = getPlaneBinding().open(planePath); - plane.setWatermark(4096); // a plane that would otherwise read as complete on the next attach + const nextMessage = () => + new Promise((resolve, reject) => { + worker.once('message', resolve); + worker.once('error', reject); + }); try { - index.invalidatePlaneFile(planePath, plane); - assert.ok(plane.invalidated(), 'the open mapping must observe the one-way in-band invalidation latch'); - assert.ok(fs.existsSync(stalePath), 'the package must also persist the out-of-band sidecar'); - assert.throws( - () => getPlaneBinding().open(planePath), - /invalidat|stale/i, - 'a new process must refuse the invalidated plane' - ); + const ready = await nextMessage(); + assert.equal(ready.type, 'ready'); + assert.equal(ready.retainedMarker, true, 'a peer worker should reopen a current plane without rebuilding it'); + const workerRecords = []; + for (let i = 0; i < 20; i++) { + const id = 2_000 + i; + const vector = makeVector(id); + vectors.set(id, vector); + workerRecords.push({ id, name: `worker${i}`, vector }); + } + const workerDone = nextMessage(); + worker.postMessage({ type: 'put', records: workerRecords }); + for (let i = 0; i < 20; i++) { + const id = 3_000 + i; + const vector = makeVector(id); + vectors.set(id, vector); + await PlaneTest.put(id, { name: `main${i}`, vector }); + } + const result = await workerDone; + assert.equal(result.type, 'done', result.stack ?? result.message); + await waitForKey(2_003, vectors.get(2_003)); + await waitForKey(3_003, vectors.get(3_003)); } finally { - fs.rmSync(stalePath, { force: true }); - await Undeletable.dropTable(); + await worker.terminate(); } - }); - - it('a stale tombstone left without its plane file rebuilds instead of disabling the plane forever', async () => { - const Orphan = table({ - table: 'PlaneOrphan', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], + const replacement = new Worker(require.resolve('./vectorIndexPlane-thread.js'), { + workerData: { workerIndex: 1, workerCount: 2 }, }); - const index = Orphan.indices.vector.customIndex; - for (let i = 0; i < 30; i++) await Orphan.put(i, { vector: makeVector(i + 40000) }); - const planePath = index.planeFilePath(); - const stalePath = planeStalePathFor(planePath); try { - // the documented rollback, run by hand: the operator deletes the plane file while a - // tombstone from an earlier undeletable-plane path is still sitting next to it - index.resetDerivedStorage(); - fs.writeFileSync(stalePath, ''); - assert.ok(!fs.existsSync(planePath), 'precondition: tombstone present, plane file gone'); - const results = index.search( - { target: makeVector(40003), comparator: 'sort', distance: 'cosine', ef: 50 }, - { transaction: undefined } - ); - if (typeof results?.then === 'function') await results; - assert.ok(!fs.existsSync(stalePath), 'the tombstone must be cleared, not left to disable the plane forever'); - assert.ok(fs.existsSync(planePath), 'the plane must rebuild once the tombstone is cleared'); + const ready = await new Promise((resolve, reject) => { + replacement.once('message', resolve); + replacement.once('error', reject); + }); + assert.equal(ready.type, 'ready'); + assert.equal(ready.retainedMarker, true, 'a replacement worker should reopen rather than rebuild the plane'); + await waitForCursors(); } finally { - fs.rmSync(stalePath, { force: true }); - await Orphan.dropTable(); + await replacement.terminate(); } }); - it('disabling the flag deletes the plane file so a re-enable rebuilds instead of adopting it stale', async () => { + it('recovers searchable native state across a database reset', async () => { const planePath = customIndex().planeFilePath(); - assert.ok(fs.existsSync(planePath)); resetDatabases(); - PlaneTest = table({ - table: 'PlaneTest', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'name', indexed: true }, - { name: 'vector', indexed: { type: 'HNSW' }, type: 'Array' }, - ], - }); - assert.ok(!PlaneTest.indexingOperation, 'removing the search-only flag must not reindex'); - assert.ok(!fs.existsSync(planePath), 'the derived plane file should be deleted with the flag off'); - // mutate while the flag is off, then re-enable: the rebuilt plane must see the mutation - const vector = makeVector(30001); - vectors.set(1201, vector); - await PlaneTest.put(1201, { name: 'rec1201', vector }); + PlaneTest = defineTable(); + await waitForKey(42, vectors.get(42)); + assert.ok(fs.existsSync(planePath)); + }); + + it('rebuilds from primary records when its durable cursor is outside audit retention', async () => { + const index = customIndex(); + const planePath = index.planeFilePath(); + PlaneTest.indices.vector.putSync(999_999, { primaryKey: 'stale-derived-mapping' }); + PlaneTest.indices.vector.putSync(derivedIndexCursorKey(PlaneTest.indices.vector.name, 0), 1); + resetDatabases(); - PlaneTest = table({ - table: 'PlaneTest', - database: DB, - attributes: [ - { name: 'id', isPrimaryKey: true }, - { name: 'name', indexed: true }, - { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, - ], - }); - const { planeEntries, jsEntries } = await searchBoth(vector); - assertParity(planeEntries, jsEntries); - assert.equal(planeEntries[0].key, 1201, 'a record written while the flag was off must be in the rebuilt plane'); + PlaneTest = defineTable(); + await waitForKey(42, vectors.get(42)); + assert.equal( + PlaneTest.indices.vector.getSync(999_999), + undefined, + 'the retention gap must replace stale derived mappings from primary records' + ); + assert.ok(fs.existsSync(planePath)); + }); + + it('requires audit logging and native construction geometry', () => { + assert.throws( + () => + table({ + table: 'PlaneNoAudit', + database: DB, + audit: false, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true }, type: 'Array' }, + ], + }), + /audit logging/ + ); + assert.throws( + () => + table({ + table: 'PlaneBadGeometry', + database: DB, + audit: true, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true, M: 8 }, type: 'Array' }, + ], + }), + /requires M=16/ + ); }); - it('index drop removes the plane file', async () => { + it('removes the native file when the table is dropped', async () => { const planePath = customIndex().planeFilePath(); - assert.ok(fs.existsSync(planePath)); await PlaneTest.dropTable(); - assert.ok(!fs.existsSync(planePath), 'dropping the table should delete the plane file'); + assert.ok(!fs.existsSync(planePath)); }); }); From bfab525d0b415e22cc77a10dddbaa4ffd918694d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 14:54:01 -0600 Subject: [PATCH 62/69] Harden native HNSW recovery Co-Authored-By: GPT-6 Codex --- .github/workflows/unit-test.yml | 5 +- hnsw-native-plane.md | 19 ++-- package-lock.json | 8 +- resources/DerivedIndexBackend.ts | 93 +++++++++++----- .../HierarchicalNavigableSmallWorld.ts | 63 ++++++----- resources/search.ts | 12 +-- .../resources/vectorIndexPlane-thread.js | 8 +- unitTests/resources/vectorIndexPlane.test.js | 100 ++++++++++++++++++ 8 files changed, 237 insertions(+), 71 deletions(-) diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml index 296b7aa826..e428c2d6d4 100644 --- a/.github/workflows/unit-test.yml +++ b/.github/workflows/unit-test.yml @@ -87,6 +87,7 @@ jobs: - name: Run tests timeout-minutes: 15 run: npm run test:unit:all + # Windows had no unit-level coverage at all before this job: every job above pins # ubuntu-latest, and integration-tests.yml was the only workflow touching Windows. # Platform-gated code and the tests written to prove it therefore never executed. @@ -172,5 +173,7 @@ jobs: LOGGING_LEVEL: 'info' run: node --enable-source-maps ./dist/bin/harper.js install - - name: Plane dual-write + parity tests + - name: File-primary native plane tests + env: + HNSW_NATIVE_REBUILD_BENCHMARK: '1' run: npx mocha unitTests/resources/vectorIndexPlane.test.js diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 45685829fd..05586055ce 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -262,7 +262,9 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d The backend advances a durable cursor for each origin log only after its own durability barrier. On open, a cursor older than `logging.auditRetention` causes a full rebuild; replay never advances - across a retained gap. The default retention means a node unavailable beyond that window takes the + across a retained gap or a corrupt frame. A whole-table reload marker (used by replica snapshot copy) + also forces a rebuild because its copied rows deliberately have no individual audit entries. The + default retention means a node unavailable beyond that window takes the measured rebuild path and returns 503 for its duration. Operators can size retention for the expected outage, but correctness does not depend on doing so: expiry changes recovery cost, not outcome. During replay or rebuild, @@ -278,9 +280,10 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d after commit. This gives multi-origin/source-resolution writes the same reconciliation rule and makes an old entry idempotent as "delete current native id, then add the current value". The audit object's in-memory record is an allowed optimization only when its version is still the - primary store's current version. Rebuild records a - log boundary, scans current records into a fresh file, and replays from that boundary before the - index becomes queryable. Existing phase-1 graph CFs are migrated by this rebuild. + primary store's current version. Rebuild records the oldest retained physical entry in each log, + scans current records into a fresh file, and replays from those entries before the index becomes + queryable. This avoids both a timestamp race with already-open transactions and a synchronous scan + to discover the log tail. Existing phase-1 graph CFs are migrated by this rebuild. The current package fixes standalone construction at M=16, efConstruction=200, mL=1/ln(16), and optimizeRouting=0.5, so file-primary mode accepts only that geometry. Its sparse reservation @@ -296,8 +299,9 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d Phase 1 has not shipped: this PR is draft, so no deployed index is silently migrated from its configurable JS geometry. Version 0.2.1 already implements the insertion search and graph mutation inside the native `insert()` call; phase 2 uses that path for rebuild as well as incremental writes, - rather than the ~263 inserts/s JS anchor. Before merge, a 100k-record native rebuild benchmark must - sustain at least 1,000 inserts/s on the Linux CI runner and publish progress (records, rate, ETA). + rather than the ~263 inserts/s JS anchor. The native CI job runs a gated 100k-record rebuild-insertion + benchmark, publishes progress (records, rate, ETA), and requires at least 1,000 inserts/s. The local + verification for this revision sustained 4,937 inserts/s including mapping writes and flush barriers. At the 16M default reservation this floor bounds a full rebuild to about 4.5 hours; configurations above it explicitly accept the proportionally longer 503 recovery window until a batch API exists. Worker shutdown is graceful and waits for a synchronous N-API insert to return; a worker is never @@ -400,7 +404,8 @@ Phase-2 acceptance: absence already fails the job instead of producing a skipped green suite (verified on 87e0fd71). 10. Two rapid updates to one key across workers cannot apply in reverse version order, and graceful worker recycle waits until an in-flight native mutation has returned. -11. A 100k-record native rebuild reports progress and sustains at least 1,000 inserts/s on Linux CI. +11. The gated 100k-record native rebuild benchmark reports progress and enforces at least 1,000 + inserts/s in the Linux native-plane CI job. 12. A soak combines concurrent search, queue admission, retry, and restart during delivery; after recovery its result quality meets the same deterministic recall baseline. diff --git a/package-lock.json b/package-lock.json index 4a616fdc6d..b7387ec43c 100644 --- a/package-lock.json +++ b/package-lock.json @@ -28,7 +28,7 @@ "@turf/distance": "6.5.0", "@turf/helpers": "6.5.0", "@turf/length": "6.5.0", - "alasql": "4.18.0", + "alasql": "4.19.0", "amaro": "^1.1.8", "argon2": "0.45.1", "asn1js": "3.0.10", @@ -5476,9 +5476,9 @@ "license": "MIT" }, "node_modules/alasql": { - "version": "4.18.0", - "resolved": "https://registry.npmjs.org/alasql/-/alasql-4.18.0.tgz", - "integrity": "sha512-Hb1gffGoiAlCVg+3sApIqegFtpa8ITcspEPbdwvbQV7vmKNWbteCPpEsCr95F2BN4Cl7kXNswJL6Bv30lGp6hg==", + "version": "4.19.0", + "resolved": "https://registry.npmjs.org/alasql/-/alasql-4.19.0.tgz", + "integrity": "sha512-HpoGEwaoGJl/L9OGs9aah1QFhC4tPYBLCR5P1DUIjP68IZ41G6+hhBPxbNCF/XBvuoWeDLomqD+b+0IwlcYGdA==", "license": "MIT", "dependencies": { "cross-fetch": "4.1.0", diff --git a/resources/DerivedIndexBackend.ts b/resources/DerivedIndexBackend.ts index cfd06fd77c..077737c3bb 100644 --- a/resources/DerivedIndexBackend.ts +++ b/resources/DerivedIndexBackend.ts @@ -1,6 +1,6 @@ import { ClientError, ServerError } from '../utility/errors/hdbError.ts'; import { loggerWithTag } from '../utility/logging/logger.ts'; -import { getWorkerIndex } from '../server/threads/manageThreads.js'; +import { getWorkerCount, getWorkerIndex } from '../server/threads/manageThreads.js'; import type { AuditRecord } from './auditStore.ts'; const logger = loggerWithTag('DerivedIndex'); @@ -28,6 +28,7 @@ export interface DerivedIndexBackend { applyDerivedValue(id: any, value: any, version?: number): void; flushDerived(watermark?: number): Promise; resetDerivedStorage(): void; + hasDerivedStorage(): boolean; } function valuesEqual(a: any, b: any): boolean { @@ -37,19 +38,18 @@ function valuesEqual(a: any, b: any): boolean { return true; } -function lock(store: any, key: string, callback: () => Promise): Promise { - return new Promise((resolve, reject) => { - let started = false; - const acquired = () => { - if (started) return; - started = true; - Promise.resolve() - .then(callback) - .then(resolve, reject) - .finally(() => store.unlock(key)); - }; - if (store.tryLock(key, acquired)) acquired(); - }); +async function lock(store: any, key: string, callback: () => Promise): Promise { + while (true) { + let unlocked: () => void; + const released = new Promise((resolve) => (unlocked = resolve)); + if (store.tryLock(key, unlocked!)) break; + await released; + } + try { + await callback(); + } finally { + store.unlock(key); + } } export class DerivedIndexRuntime { @@ -68,6 +68,7 @@ export class DerivedIndexRuntime { private closed = false; private ready = false; private retryScheduled = false; + private forceRebuild = false; private rebuildBackoff = 1_000; private listener: (entries: AuditRecord[], targets?: StagedTarget[]) => void; @@ -105,7 +106,11 @@ export class DerivedIndexRuntime { let totalKeys = 0; let totalTickets = 0; let rejectedWrites = 0; - for (let offset = 0; offset < this.sharedDepth.length; offset += DEPTH_VALUES_PER_WORKER) { + const activeDepthLength = Math.min( + this.sharedDepth.length, + Math.max(1, getWorkerCount() ?? 1) * DEPTH_VALUES_PER_WORKER + ); + for (let offset = 0; offset < activeDepthLength; offset += DEPTH_VALUES_PER_WORKER) { totalKeys += Atomics.load(this.sharedDepth, offset); totalTickets += Atomics.load(this.sharedDepth, offset + 1); rejectedWrites += Atomics.load(this.sharedDepth, offset + 2); @@ -123,11 +128,22 @@ export class DerivedIndexRuntime { ); } const targets: StagedTarget[] = (transaction.derivedIndexTargets ??= []); - if (!targets.some((target) => target.runtime === this && target.id === id)) targets.push({ runtime: this, id }); + const targetsByRuntime: Map> = (transaction.derivedIndexTargetIds ??= new Map()); + let ids = targetsByRuntime.get(this); + if (!ids) targetsByRuntime.set(this, (ids = new Set())); + if (!ids.has(id)) { + ids.add(id); + targets.push({ runtime: this, id }); + } } private committed(entries: AuditRecord[], targets?: StagedTarget[]): void { - if (this.closed || !targets) return; + if (this.closed) return; + if (entries.some((entry) => entry.tableId === this.table.tableId && entry.type === 'reload')) { + this.requestRebuild(new Error(`Table ${this.table.tableName} received a whole-table reload`)); + return; + } + if (!targets) return; const positions = new Map(); for (const entry of entries) { if (entry.tableId === this.table.tableId && entry.recordId != null) positions.set(entry.recordId, entry); @@ -243,6 +259,11 @@ export class DerivedIndexRuntime { } private async reconcile(logs: any[]): Promise { + if (this.forceRebuild || !this.backend.hasDerivedStorage()) { + await this.rebuild(logs); + this.forceRebuild = false; + return; + } for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); @@ -255,22 +276,35 @@ export class DerivedIndexRuntime { await this.replay(logs); } - private async replay(logs: any[]): Promise { + private async replay(logs: any[], ignoreReloadsBefore?: number): Promise { for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); + if (!cursor) { + let hasEntries = false; + for (const _entry of this.table.auditStore.getRange({ start: 0, log: nodeId })) { + hasEntries = true; + break; + } + if (!hasEntries) continue; + } if (!cursor || !this.cursorExists(nodeId, cursor)) { throw new Error(`Derived-index cursor for origin ${nodeId} is outside audit retention`); } let latest = cursor; let applied = 0; - for (const entry of this.table.auditStore.getRange({ + const entries = this.table.auditStore.getRange({ start: cursor, exactStart: true, exclusiveStart: true, log: nodeId, - })) { + }); + for (const entry of entries) { latest = entry.txnLogKey; + if (entry.tableId === this.table.tableId && entry.type === 'reload') { + if (ignoreReloadsBefore !== undefined && entry.txnLogKey < ignoreReloadsBefore) continue; + throw new Error(`Table ${this.table.tableName} requires reconstruction after a whole-table reload`); + } if (entry.tableId === this.table.tableId && entry.recordId != null) { const current = this.table.primaryStore.getEntry(entry.recordId); const record = current?.value; @@ -280,6 +314,9 @@ export class DerivedIndexRuntime { if (++applied % APPLY_BATCH_SIZE === 0) await this.backend.flushDerived(latest); } } + if (entries.corruptFrameStop.breaks) { + throw new Error(`Audit log ${nodeId} ended at a corrupt frame while updating ${this.indexStore.name}`); + } // exactStart follows physical log order after locating the cursor. A transaction that // began earlier can commit later with a numerically smaller timestamp, so inequality, // rather than a numeric greater-than comparison, identifies forward progress. @@ -291,21 +328,24 @@ export class DerivedIndexRuntime { } private async rebuild(logs: any[]): Promise { - // Capture an entry that actually exists in each log before scanning. `exactStart` can then - // resume by physical position and include transactions that started before this boundary - // but committed after it. A wall-clock timestamp is not a log position. + const startedAt = Date.now(); + this.ready = false; + this.indexStore.isIndexing = true; + // Keep the oldest retained physical entry as the catch-up boundary. Starting at an actual + // entry includes transactions that were already open when reconstruction began; their + // timestamps can predate any wall-clock boundary even when they commit during the scan. const boundaries: Array = []; for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; for (const entry of this.table.auditStore.getRange({ start: 0, log: nodeId })) { boundaries[nodeId] = entry.txnLogKey; + break; } } this.backend.resetDerivedStorage(); await this.indexStore.clear(); let indexed = 0; const total = this.table.primaryStore.getKeysCount?.() ?? 0; - const startedAt = Date.now(); for (const { key, value, version, localTime } of this.table.primaryStore.getRange({ versions: true, snapshot: false, @@ -328,6 +368,9 @@ export class DerivedIndexRuntime { const boundary = boundaries[nodeId]; if (boundary) this.indexStore.putSync(this.cursorKey(nodeId), boundary); } + await this.replay(logs, startedAt); + this.ready = true; + this.indexStore.isIndexing = false; if (indexed) { const elapsedSeconds = Math.max((Date.now() - startedAt) / 1_000, 0.001); logger.info?.( @@ -339,6 +382,7 @@ export class DerivedIndexRuntime { private fail(error: unknown): void { if (this.closed || this.retryScheduled) return; this.ready = false; + this.forceRebuild = true; this.indexStore.isIndexing = true; Atomics.sub(this.sharedDepth, this.sharedDepthOffset, this.pending.size); Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, this.pendingTickets); @@ -355,6 +399,7 @@ export class DerivedIndexRuntime { } requestRebuild(error: unknown): void { + this.forceRebuild = true; this.fail(error); } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 115100368e..899af7ea9d 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -125,11 +125,9 @@ const NODE_COUNT_TTL = 10_000; const PLANE_LAYER0_CAP_MAX = 1024; const PLANE_MAX_NODES = 1 << 24; // An existing plane file that cannot be opened is normally another worker mid-create (retry); -// past this age it is a crashed create and is deleted and rebuilt — the plane is derived state, -// the index column family stays authoritative. +// past this age it is a crashed create and is deleted so the audit-backed runtime can rebuild it. const PLANE_STALE_CREATE_MS = 60_000; -// Retry cadence while another worker holds the create: its header lands within moments of the -// exclusive open, so a long deferral would silently drop this worker's mirror writes. +// Retry cadence while another worker holds the create; its header lands shortly after exclusive open. const PLANE_ATTACH_RETRY_MS = 250; // Marks an error thrown by an app-supplied filter during a plane search: the caller re-raises // it as an ordinary query failure instead of disabling the (healthy) plane. @@ -276,7 +274,10 @@ export class HierarchicalNavigableSmallWorld { private filePrimary = false; private nativePlaneMaxNodes = PLANE_MAX_NODES; private derivedRuntime?: DerivedIndexRuntime; - private pendingDerivedMappings = new Map(); + private pendingDerivedMappings = new Map< + Id, + { id?: number; signature?: string; version?: number; pending?: boolean } + >(); postCommit?: true; constructor(indexStore: any, options: any) { this.indexStore = indexStore; @@ -349,13 +350,13 @@ export class HierarchicalNavigableSmallWorld { const filePath = this.planeFilePath(); if (!filePath) return; try { + if (existsSync(filePath)) this.invalidatePlaneFile(filePath, this.plane); unlinkSync(filePath); logger.info?.('deleted the HNSW plane file of an index no longer using nativePlane'); } catch (error: any) { if (error?.code !== 'ENOENT') { // the file survives (Windows EBUSY while another process maps it), and nothing - // mirrors into it from here on: a later re-enable would adopt it at its nonzero - // watermark and silently miss every mutation made while the flag was off + // A later re-enable must not adopt a file that missed mutations while disabled. logger.warn?.('could not delete the HNSW plane file; marking it stale', error); this.invalidatePlaneFile(filePath); } @@ -403,9 +404,8 @@ export class HierarchicalNavigableSmallWorld { } if (existsSync(filePath)) { try { - // crash recovery is per-slot inside the crate (abandoned seqlocks are taken - // over lazily); the clean flag is advisory only — acting on it here would let - // a second worker unlink a plane the first worker is live-mirroring into + // Crash recovery is per-slot inside the crate. The clean flag is advisory; + // another worker may still be constructing this shared file. return (this.plane = Plane.open(filePath)); } catch (openError) { if (now - statSync(filePath).mtimeMs <= PLANE_STALE_CREATE_MS) { @@ -437,7 +437,7 @@ export class HierarchicalNavigableSmallWorld { try { return (this.plane = Plane.create(filePath, dims, this.planeLayer0Cap(), this.nativePlaneMaxNodes)); } catch (createError) { - // never leave a file a later open would trust as a complete mirror + // Never leave a partial file that a later process could trust as current. try { unlinkSync(filePath); } catch { @@ -484,7 +484,7 @@ export class HierarchicalNavigableSmallWorld { this.planeReady = false; const filePath = this.planeFilePath(); if (filePath) { - if (this.filePrimary && attached) this.invalidatePlaneFile(filePath, attached); + if (this.filePrimary && existsSync(filePath)) this.invalidatePlaneFile(filePath, attached); try { unlinkSync(filePath); } catch (unlinkError: any) { @@ -507,11 +507,8 @@ export class HierarchicalNavigableSmallWorld { } /** - * Delete the derived plane state. Called when the backing store is dropped or cleared - * (index drop, table drop/clear, reindex-from-scratch); the plane lazily rebuilds from the - * CF on next use. Unlinking while another worker still maps the old file is safe on POSIX — - * that worker keeps writing the orphaned inode until the schema-change signal resets its - * database instances. + * Delete the derived plane state before an audit-backed reconstruction. Path invalidation + * makes peers stop using an old mapping even when unlink leaves their mmap inode alive. */ resetDerivedStorage(): void { this.pendingDerivedMappings.clear(); @@ -521,7 +518,7 @@ export class HierarchicalNavigableSmallWorld { this.planeRetryAt = 0; const filePath = this.planeFilePath(); if (!filePath) return; - if (this.filePrimary && attached) this.invalidatePlaneFile(filePath, attached); + if (this.filePrimary && existsSync(filePath)) this.invalidatePlaneFile(filePath, attached); try { unlinkSync(filePath); } catch (error: any) { @@ -535,6 +532,13 @@ export class HierarchicalNavigableSmallWorld { } } + hasDerivedStorage(): boolean { + const filePath = this.planeFilePath(); + if (!filePath || !existsSync(filePath)) return false; + const plane = this.getPlane(); + return Boolean(plane && !plane.invalidated()); + } + /** Make an undeletable plane unadoptable before this process releases it. */ private invalidatePlaneFile(filePath: string, attached?: HnswPlane | null): void { try { @@ -620,6 +624,9 @@ export class HierarchicalNavigableSmallWorld { private validateVector(primaryKey: Id, vector?: number[]): void { if (!vector) return; + if (vector.length === 0) { + throw new ClientError(`Vector for attribute "${String(primaryKey)}" must contain at least one component.`); + } for (let i = 0; i < vector.length; i++) { if (!Number.isFinite(vector[i])) { throw new ClientError( @@ -632,7 +639,8 @@ export class HierarchicalNavigableSmallWorld { applyDerivedValue(primaryKey: Id, vector: number[], version?: number): void { this.validateVector(primaryKey, vector); const safeKey = typeof primaryKey === 'number' ? [KEY_PREFIX, primaryKey] : primaryKey; - const storedMapping = this.pendingDerivedMappings.get(primaryKey) ?? this.indexStore.getSync(safeKey); + const pendingMapping = this.pendingDerivedMappings.get(primaryKey); + const storedMapping = pendingMapping ?? this.indexStore.getSync(safeKey); const oldNodeId = typeof storedMapping === 'number' ? storedMapping : storedMapping?.id; if (storedMapping?.version != null && version != null && storedMapping.version > version) return; const nativeVector = vector ? Float32Array.from(vector) : undefined; @@ -641,7 +649,13 @@ export class HierarchicalNavigableSmallWorld { .update(Buffer.from(nativeVector.buffer, nativeVector.byteOffset, nativeVector.byteLength)) .digest('base64url') : undefined; - if (oldNodeId != null && signature && storedMapping.signature === signature && !storedMapping.pending) { + if ( + oldNodeId != null && + signature && + storedMapping.signature === signature && + !pendingMapping && + !storedMapping.pending + ) { this.indexStore.putSync(safeKey, { id: oldNodeId, signature, version }); this.indexStore.putSync(oldNodeId, { primaryKey, version }); return; @@ -664,7 +678,7 @@ export class HierarchicalNavigableSmallWorld { const nodeId = plane.insert(nativeVector!); this.indexStore.putSync(safeKey, { id: nodeId, signature, version, pending: true }); this.indexStore.putSync(nodeId, { primaryKey, version, pending: true }); - this.pendingDerivedMappings.set(primaryKey, { id: nodeId, signature, version }); + this.pendingDerivedMappings.set(primaryKey, { id: nodeId, signature, version, pending: true }); } async flushDerived(watermark?: number): Promise { @@ -684,7 +698,8 @@ export class HierarchicalNavigableSmallWorld { if (mapping.id === undefined) { this.indexStore.removeSync(safeKey); } else { - this.indexStore.putSync(safeKey, mapping); + const published = { id: mapping.id, signature: mapping.signature, version: mapping.version }; + this.indexStore.putSync(safeKey, published); this.indexStore.putSync(mapping.id, { primaryKey, version: mapping.version }); } } @@ -1610,7 +1625,7 @@ export class HierarchicalNavigableSmallWorld { return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { // an app filter's own throw is the query's failure, not the plane's if (error?.[PLANE_PREDICATE_ERROR]) throw error; - // a failed native search disables the plane and re-runs this query on the JS path + // File-primary mode stays unavailable until its audit-backed rebuild succeeds. this.disablePlane(error); if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); return this.search( @@ -1621,7 +1636,7 @@ export class HierarchicalNavigableSmallWorld { ); }); } catch (error) { - // a synchronous NAPI throw (before any promise exists) degrades to the JS path below + // Handle a throw raised before the asynchronous native search returns its promise. this.disablePlane(error); if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); } diff --git a/resources/search.ts b/resources/search.ts index 9a87d12070..9f978b215b 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -576,25 +576,17 @@ export function searchByIndex( return loaded; }; if (typeof (searched as any)?.then === 'function') { - // An async custom-index search (the native HNSW plane runs off the event loop and - // resolves its candidate list as a promise). Apply the same load + rescore pipeline - // once it resolves, exposed as a lazily-resolving iterable — consumable through async - // iteration only, like the promise-entry filter paths above. const pending = (searched as Promise).then(processEntries); - // nothing is required to consume this iterable (an aborted request, `limit: 0`), and - // a rejection nobody observed reaches Node's unhandledRejection and exits the process + // A consumer may abandon this lazy iterable without calling next(). pending.catch(() => {}); const results: any = new ExtendedIterable(); results.iterate = (options?: { async?: boolean }) => { - // fail loudly rather than hand a synchronous consumer promise-shaped iterator - // results (which a bare for-of would spin on forever) if (!options?.async) { throw new Error( 'This index resolves search results asynchronously; the results must be consumed with async iteration' ); } - // one shared iterator per iterate() call: overlapping next() calls must advance the - // same cursor + // Overlapping next() calls share one cursor and cannot duplicate its first entry. const iteratorPromise = pending.then((entries) => entries[Symbol.iterator]()); iteratorPromise.catch(() => {}); let closed = false; diff --git a/unitTests/resources/vectorIndexPlane-thread.js b/unitTests/resources/vectorIndexPlane-thread.js index 94215e03b8..c9dcb86949 100644 --- a/unitTests/resources/vectorIndexPlane-thread.js +++ b/unitTests/resources/vectorIndexPlane-thread.js @@ -35,8 +35,14 @@ if (parentPort) { ); parentPort.on('message', async (message) => { if (message.type === 'shutdown') process.exit(0); - if (message.type !== 'put') return; try { + if (message.type === 'commitAndBlock') { + await PlaneTest.put(message.record.id, message.record); + parentPort.postMessage({ type: 'committed' }); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 60_000); + return; + } + if (message.type !== 'put') return; for (const record of message.records) await PlaneTest.put(record.id, record); parentPort.postMessage({ type: 'done' }); } catch (error) { diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index a9deb815fe..8e20f29228 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -191,6 +191,19 @@ describe('HNSW native plane file-primary delivery', function () { assert.equal(plane.idHighWater(), highWater, 'an aborted write must allocate no native node'); }); + it('keeps repeated same-key replay mappings pending until the native flush', async () => { + const id = 50_001; + const vector = makeVector(id); + const index = customIndex(); + index.applyDerivedValue(id, vector, 1); + index.applyDerivedValue(id, vector, 2); + assert.ok(!(await nativeSearch(vector)).some((entry) => entry.key === id)); + await index.flushDerived(); + assert.ok((await nativeSearch(vector)).some((entry) => entry.key === id)); + index.applyDerivedValue(id, undefined, 3); + await index.flushDerived(); + }); + it('ignores unrelated field changes and applies committed update/delete', async () => { const plane = customIndex().getPlane(); const highWater = plane.idHighWater(); @@ -358,6 +371,67 @@ describe('HNSW native plane file-primary delivery', function () { } }); + it('replays a committed write after its worker terminates before delivery', async () => { + const id = 4_000; + const vector = makeVector(id); + vectors.set(id, vector); + const worker = new Worker(require.resolve('./vectorIndexPlane-thread.js'), { + workerData: { workerIndex: 1, workerCount: 2 }, + }); + try { + const ready = await new Promise((resolve, reject) => { + worker.once('message', resolve); + worker.once('error', reject); + }); + assert.equal(ready.type, 'ready'); + const committed = new Promise((resolve, reject) => { + worker.once('message', resolve); + worker.once('error', reject); + }); + worker.postMessage({ type: 'commitAndBlock', record: { id, name: 'crash-window', vector } }); + assert.equal((await committed).type, 'committed'); + } finally { + await worker.terminate(); + } + + const replacement = new Worker(require.resolve('./vectorIndexPlane-thread.js'), { + workerData: { workerIndex: 1, workerCount: 2 }, + }); + try { + const ready = await new Promise((resolve, reject) => { + replacement.once('message', resolve); + replacement.once('error', reject); + }); + assert.equal(ready.type, 'ready'); + await waitForKey(id, vector); + } finally { + await replacement.terminate(); + } + }); + + it('rebuilds after a whole-table snapshot reload marker', async () => { + PlaneTest.indices.vector.putSync(999_998, { primaryKey: 'stale-snapshot-mapping' }); + await PlaneTest.writeReloadMarker(); + await waitFor( + () => !PlaneTest.indices.vector.isIndexing && PlaneTest.indices.vector.getSync(999_998) === undefined, + { timeout: 15_000, message: 'whole-table reload marker did not replace derived mappings' } + ); + await waitForKey(42, vectors.get(42)); + }); + + it('rebuilds when a replacement worker replays a snapshot reload marker', async () => { + PlaneTest.indices.vector.putSync(999_997, { primaryKey: 'stale-offline-snapshot-mapping' }); + PlaneTest.derivedIndexRuntime.close(); + await PlaneTest.writeReloadMarker(); + resetDatabases(); + PlaneTest = defineTable(); + await waitFor( + () => !PlaneTest.indices.vector.isIndexing && PlaneTest.indices.vector.getSync(999_997) === undefined, + { timeout: 15_000, message: 'replayed reload marker did not replace derived mappings' } + ); + await waitForKey(42, vectors.get(42)); + }); + it('recovers searchable native state across a database reset', async () => { const planePath = customIndex().planeFilePath(); resetDatabases(); @@ -410,8 +484,34 @@ describe('HNSW native plane file-primary delivery', function () { }), /requires M=16/ ); + assert.throws( + () => customIndex().prepareCommitted('empty-vector', [], undefined, { transaction: {} }), + /must contain at least one component/ + ); }); + (process.env.HNSW_NATIVE_REBUILD_BENCHMARK ? it : it.skip)( + 'sustains the native rebuild insertion floor for 100k records', + async function () { + this.timeout(180_000); + const total = 100_000; + const index = customIndex(); + const startedAt = Date.now(); + for (let id = 100_000; id < 100_000 + total; id++) { + index.applyDerivedValue(id, makeVector(id), id); + if (id % 10_000 === 9_999) { + await index.flushDerived(id); + const indexed = id - 100_000 + 1; + const rate = Math.round(indexed / Math.max((Date.now() - startedAt) / 1_000, 0.001)); + const eta = Math.ceil((total - indexed) / Math.max(rate, 1)); + console.log(`Native rebuild benchmark: ${indexed}/${total} records, ${rate}/s, ETA ${eta}s`); + } + } + const rate = total / Math.max((Date.now() - startedAt) / 1_000, 0.001); + assert.ok(rate >= 1_000, `native rebuild rate ${Math.round(rate)}/s is below the 1,000/s floor`); + } + ); + it('removes the native file when the table is dropped', async () => { const planePath = customIndex().planeFilePath(); await PlaneTest.dropTable(); From e7adb5df3034ad69d298ce92338920ddf3e9a7e8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 15:18:52 -0600 Subject: [PATCH 63/69] Document the file-primary requirements and non-guarantees Consolidates what nativePlane: true now requires (explicit audit, RocksDB, fixed construction geometry, fixed reservation, native package, audit retention, backlog admission) and what it does not promise (a single total order across concurrent CRDT/source-resolution arrivals, identical graphs, in-place format upgrades). Corrects the rebuild-duration claim: the CI gate is a 100k regression floor, not a 16M guarantee. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 05586055ce..5912fef5bb 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -301,9 +301,12 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d inside the native `insert()` call; phase 2 uses that path for rebuild as well as incremental writes, rather than the ~263 inserts/s JS anchor. The native CI job runs a gated 100k-record rebuild-insertion benchmark, publishes progress (records, rate, ETA), and requires at least 1,000 inserts/s. The local - verification for this revision sustained 4,937 inserts/s including mapping writes and flush barriers. - At the 16M default reservation this floor bounds a full rebuild to about 4.5 hours; configurations - above it explicitly accept the proportionally longer 503 recovery window until a batch API exists. + verification for this revision sustained 4,625 inserts/s at 100k including mapping writes and flush + barriers. That gate is a regression floor at 100k, not a rebuild-duration guarantee: insert rate falls + as the graph grows (8,482/s at 30k → 4,625/s at 100k here; §12's crate anchor is 1,242/s at 1M), and + nothing above 1M is measured. A 16M rebuild is therefore at least ~3.6 hours at the 1M rate and in + practice longer, so a deployment sizing above the default reservation is accepting a 503 recovery + window of that order until a batch API exists. Worker shutdown is graceful and waits for a synchronous N-API insert to return; a worker is never force-terminated in the middle of a plane mutation while the process survives. @@ -311,6 +314,34 @@ search(sliceHandles, queryVector: Float32Array, k, ef, filter?): Promise<{ids, d report progress while native code owns the insertion loop. Single-record insertion search and graph mutation already run natively in 0.2.1. +### What `nativePlane: true` requires, and what it does not promise + +Removing the RocksDB graph moves several phase-1 conveniences into hard requirements. All of them +are enforced or surfaced in code, not left as advice. + +| Requirement | Enforcement | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The table declares `audit: true` explicitly. Inheriting either value from the global setting is rejected, so a vector-index option cannot silently widen the audit-readable surface. | `resources/databases.ts` `table()` and `attachDerivedIndexBackends()` both throw a `ClientError`; enabling one logs once that the audit API now retains full record history for the retention window. | +| The RocksDB storage engine. | `ClientError` at index construction. | +| `M=16`, `efConstruction=200`, `mL=1/ln(16)`, `optimizeRouting=0.5`, int8-quantized cosine. Standalone `insert` in 0.2.1 fixes this geometry, so a schema asking for another one is rejected rather than silently rebuilt under native defaults. | `ClientError` at index construction. | +| `nativePlaneMaxNodes` (16M default) is structural — the sparse reservation is fixed at file create. Exhausting it makes the index unavailable until the value is raised and the index rebuilt. | Reservation is a create-time header field; growth is a possible later enhancement (§10). | +| `@harperfast/hnsw` must load on the platform. There is no JS graph to fall back to, so an absent or failing native module means the index is unavailable, not degraded. | Search returns 503; delivery throws a 503 `ServerError`. Ordinary HNSW indexes are unaffected. | +| The audit log must retain entries back to each origin cursor. A cursor before retention, a corrupt frame, or a whole-table reload marker forces a full rebuild from current records. | Detected in `reconcile`/`replay`; the index reports 503 for the rebuild's duration. | +| Vector-changing writes are admissible only while the shared backlog is under its bound. | The pre-commit hook returns a retryable 503 once the aggregate pending-key or log-position depth reaches its threshold; unrelated writes continue. | + +Not promised: + +- **A single total order across concurrent CRDT or source-resolution arrivals.** Hot delivery and + replay both re-read the current authoritative record, so the index converges on whatever the + primary store resolved, but two origins landing concurrently are not promised to produce the + index state a single serial order would. This is an approximate nearest-neighbour index and the + exact record load plus rescore on the read path already rejects stale candidates, so the + divergence is bounded by candidate selection, not by returned data. +- **Byte-identical graphs across nodes or across rebuilds.** Insertion order and the concurrent + prune both affect edge selection; only recall is held to a baseline (§9). +- **In-place format upgrades.** A plane whose format version does not match is rejected at open + and reindexed (§4). + ### Approaches considered for phase 2 **Chosen: transaction log plus rebuild on retention gap.** Correctness: record and audit entry From 4ce6259d111eed759b5fae05dc36d324c142221a Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 22:52:16 -0600 Subject: [PATCH 64/69] Address the pre-push review of the file-primary derived index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes confirmed against source from the gemini, cursor-grok and Harper domain legs: - dropTable() left the derived runtime's aftercommit listener and rebuild retry armed, so a same-name recreate raced two runtimes on one plane file. - A search-target getPlane() probe exclusively created and unlinked an empty plane file; a concurrent first insert read it as another worker's in-progress create and rejected the write for PLANE_STALE_CREATE_MS. - A ready index with no plane file answered every query with a rebuilding 503 that no consumer converts back into a scan; it now returns no results, pinned by a new test. - hasDerivedStorage() answered from a cached invalidated handle after a peer's rebuild, so every worker reconstructed over the last worker's replacement. - reconcile() read an origin with an empty local log as a retention gap and rebuilt the index on every pass; it now shares replay's exemption. - drain() copied the whole pending map per batch and retired tickets with a shift per ticket. - valuesEqual() reported two non-array-like values as equal, which would silently unstage a scalar change from a future backend. - Removed the phase-1 JS fallback in the plane search error paths: filePrimary is unconditionally true wherever planeEligible is. Gemini's exclusiveStart finding is refuted, not deferred: options reach log.query() verbatim (RocksTransactionLogStore.ts) and rocksdb-js implements exclusive traversal, as transactionBroadcast.ts already relies on. Records in hnsw-native-plane.md §10 the three recovery majors this change does not fix — the corrupt-frame reconstruction loop, per-key replay coalescing, and decoder-sentinel escalation — each of which needs the safely-captured physical catch-up boundary that section now also explains why rebuild cannot take today. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 26 ++++++++++ resources/DerivedIndexBackend.ts | 51 +++++++++++-------- resources/Table.ts | 5 ++ .../HierarchicalNavigableSmallWorld.ts | 43 ++++++++++------ unitTests/resources/vectorIndexPlane.test.js | 26 ++++++++++ 5 files changed, 114 insertions(+), 37 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 5912fef5bb..5ca5692cca 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -479,6 +479,32 @@ Open: distance the generation check discards. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. +- **An interior corrupt audit frame can loop reconstruction.** `replay()` throws on + `corruptFrameStop.breaks`, and `rebuild()`'s own catch-up replay starts at the oldest retained + entry, so it meets the same frame and throws again: the backoff retries reconstruction + indefinitely and the index stays 503 until retention removes the corruption. Bounding this + needs the same safely-captured physical boundary as the item below — a rebuild that starts + after the break has no reason to read it. +- **Replay does not coalesce native mutations for one key.** 128 queued updates to a single key + replay its final vector 128 times, because a mapping still pending publication bypasses the + signature shortcut. Deduplicating within a batch is not free: `applyDerivedValue` reads current + record state at first touch, so a change committing later in the same batch would be skipped + and its entry then passed by the advancing cursor. A correct version re-reads deduplicated keys + at the batch boundary. +- **An undecodable audit header is skipped, not escalated.** Replay matches on + `tableId`/`recordId`; an entry that decodes to a sentinel carries neither, so it is passed over + while the cursor advances past it. If that entry was a record's last vector-changing commit the + index stays stale indefinitely. Decoder sentinels should force reconstruction instead. +- **Rebuild replays the whole retained log, not just the scan window.** After the primary-store + scan, `rebuild()` sets each origin's cursor to the _oldest_ retained entry rather than the log + tail at scan start, so the follow-up `replay()` re-reads and re-hashes the entire retention + window before the index leaves 503. It is conservative on purpose and cannot be tightened to + the tail: `RocksTransactionLogStore.addEntry` stages an entry into the log at write time and + the commit hook only publishes it, so a transaction still open when the scan begins already + occupies a physical position behind that tail and a tail-anchored cursor would skip its + entries entirely. The correct tighter bound is the oldest entry belonging to a transaction + uncommitted at scan start; Harper does not expose that position today. Until it does, rebuild + cost scales with retention rather than with scan duration. - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the format (dims × mode in header) but int8 is the default and the optimization target. - ~~Upper-layer region persistence~~ — done (format v2): fixed-entry region in the same file, diff --git a/resources/DerivedIndexBackend.ts b/resources/DerivedIndexBackend.ts index 077737c3bb..5fc82f59f3 100644 --- a/resources/DerivedIndexBackend.ts +++ b/resources/DerivedIndexBackend.ts @@ -33,7 +33,9 @@ export interface DerivedIndexBackend { function valuesEqual(a: any, b: any): boolean { if (a === b) return true; - if (a == null || b == null || a.length !== b.length) return false; + // Element-wise comparison is only defined for the array-like projections backends index. + // Anything else that already failed `===` counts as changed rather than silently unstaged. + if (a == null || b == null || typeof a.length !== 'number' || a.length !== b.length) return false; for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) return false; return true; } @@ -190,9 +192,11 @@ export class DerivedIndexRuntime { try { await lock(this.indexStore, this.lockKey, async () => { if (this.closed) return; - const batch = Array.from(this.pending.values()) - .slice(0, APPLY_BATCH_SIZE) - .map((pending) => ({ pending, ticketCount: pending.tickets.length })); + const batch: Array<{ pending: Pending; ticketCount: number }> = []; + for (const pending of this.pending.values()) { + batch.push({ pending, ticketCount: pending.tickets.length }); + if (batch.length === APPLY_BATCH_SIZE) break; + } await this.reconcile(this.table.auditStore.loadLogs?.() ?? []); if (!this.ready || this.closed) return; for (const { pending, ticketCount } of batch) { @@ -219,10 +223,12 @@ export class DerivedIndexRuntime { private discardCompletedTickets(): void { for (const [nodeId, tickets] of this.ticketsByOrigin) { - while (tickets[0]?.done) { - tickets.shift(); - this.pendingTickets--; - Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, 1); + let completed = 0; + while (completed < tickets.length && tickets[completed].done) completed++; + if (completed > 0) { + tickets.splice(0, completed); + this.pendingTickets -= completed; + Atomics.sub(this.sharedDepth, this.sharedDepthOffset + 1, completed); } if (tickets.length === 0) this.ticketsByOrigin.delete(nodeId); } @@ -251,6 +257,11 @@ export class DerivedIndexRuntime { } } + private originHasEntries(nodeId: number): boolean { + for (const _entry of this.table.auditStore.getRange({ start: 0, log: nodeId })) return true; + return false; + } + private cursorExists(nodeId: number, cursor: number): boolean { for (const entry of this.table.auditStore.getRange({ start: cursor, exactStart: true, log: nodeId })) { return entry.txnLogKey === cursor; @@ -267,8 +278,11 @@ export class DerivedIndexRuntime { for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); - const cursorExists = cursor && this.cursorExists(nodeId, cursor); - if (!cursorExists) { + // A replica that only ever received remote writes keeps an empty local log, and rebuild + // writes no cursor for it. Without this exemption every reconciliation would read that + // missing cursor as a retention gap and reconstruct the whole index again. + if (!cursor && !this.originHasEntries(nodeId)) continue; + if (!this.cursorExists(nodeId, cursor)) { await this.rebuild(logs); return; } @@ -280,14 +294,7 @@ export class DerivedIndexRuntime { for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); - if (!cursor) { - let hasEntries = false; - for (const _entry of this.table.auditStore.getRange({ start: 0, log: nodeId })) { - hasEntries = true; - break; - } - if (!hasEntries) continue; - } + if (!cursor && !this.originHasEntries(nodeId)) continue; if (!cursor || !this.cursorExists(nodeId, cursor)) { throw new Error(`Derived-index cursor for origin ${nodeId} is outside audit retention`); } @@ -331,9 +338,11 @@ export class DerivedIndexRuntime { const startedAt = Date.now(); this.ready = false; this.indexStore.isIndexing = true; - // Keep the oldest retained physical entry as the catch-up boundary. Starting at an actual - // entry includes transactions that were already open when reconstruction began; their - // timestamps can predate any wall-clock boundary even when they commit during the scan. + // Keep the oldest retained physical entry as the catch-up boundary. addEntry stages an entry + // into the log at write time and the commit hook only publishes it, so a transaction still + // open when this scan begins already sits physically behind the current tail. Anchoring on + // the tail — or on any wall-clock instant — would skip its entries once it commits. The cost + // is that replay re-walks the whole retention window; see hnsw-native-plane.md §10. const boundaries: Array = []; for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; diff --git a/resources/Table.ts b/resources/Table.ts index 3fdbffc4ac..dfe5abde77 100644 --- a/resources/Table.ts +++ b/resources/Table.ts @@ -1622,6 +1622,11 @@ export function makeTable(options) { static async dropTable() { TableResource.assertSchemaMutable('drop a table'); + // Retire post-commit derived-index delivery before any destructive work. Its aftercommit + // listener and rebuild retry outlive the stores otherwise, and a same-name recreate would + // leave the orphan racing the new table's runtime on the same derived file and mappings. + TableResource.derivedIndexRuntime?.close(); + TableResource.derivedIndexRuntime = undefined; const rootStore = primaryStore.rootStore; if (databaseName === databasePath) { // Persist a drop tombstone on the primary catalog entry BEFORE any diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index 899af7ea9d..be9f63956e 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -419,6 +419,11 @@ export class HierarchicalNavigableSmallWorld { } } if (!dims) return null; // open-only call and no file: nothing to attach yet + // A search target must not pin an empty index's dimensionality: creation is deferred to + // the first committed vector or to the rebuild scan. Return before the exclusive create + // rather than creating and unlinking — a concurrent insert that saw the empty file would + // read it as another worker's in-progress create and 503 for PLANE_STALE_CREATE_MS. + if (!dimsFromVector) return null; let fd: number; try { fd = openSync(filePath, 'wx'); @@ -428,12 +433,6 @@ export class HierarchicalNavigableSmallWorld { return null; } closeSync(fd); - if (!dimsFromVector) { - // A search target must not pin an empty index's dimensionality forever. Creation is - // deferred to the first committed vector or to the rebuild scan. - unlinkSync(filePath); - return null; - } try { return (this.plane = Plane.create(filePath, dims, this.planeLayer0Cap(), this.nativePlaneMaxNodes)); } catch (createError) { @@ -535,6 +534,14 @@ export class HierarchicalNavigableSmallWorld { hasDerivedStorage(): boolean { const filePath = this.planeFilePath(); if (!filePath || !existsSync(filePath)) return false; + // A peer's rebuild invalidates the old inode and creates a replacement. This process's + // cached handle still maps the old one, so answering from it reports storage as absent and + // makes every worker rebuild over the last worker's replacement. Reattach first. + if (this.plane?.invalidated()) { + this.plane = undefined; + this.planeReady = false; + this.planeRetryAt = 0; + } const plane = this.getPlane(); return Boolean(plane && !plane.invalidated()); } @@ -1625,24 +1632,28 @@ export class HierarchicalNavigableSmallWorld { return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { // an app filter's own throw is the query's failure, not the plane's if (error?.[PLANE_PREDICATE_ERROR]) throw error; - // File-primary mode stays unavailable until its audit-backed rebuild succeeds. + // There is no JS graph behind a file-primary index: it stays unavailable until + // its audit-backed rebuild succeeds. this.disablePlane(error); - if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); - return this.search( - { target, value, descending, distance, comparator, ef, filterExpansion }, - context, - filter, - minResults - ); + throw new ServerError('The native HNSW index is rebuilding', 503); }); } catch (error) { // Handle a throw raised before the asynchronous native search returns its promise. this.disablePlane(error); - if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); + throw new ServerError('The native HNSW index is rebuilding', 503); } } } - if (this.filePrimary) throw new ServerError('The native HNSW index is rebuilding', 503); + if (this.filePrimary) { + const planePath = this.planeFilePath(); + if (this.indexStore?.isIndexing || (planePath && existsSync(planePath))) { + throw new ServerError('The native HNSW index is rebuilding', 503); + } + // The runtime finished and created no file: nothing has been committed to this index. + // Match the JS path's empty result rather than reporting a reconstruction that is not + // running, which no consumer converts back into a scan. + return withStats([], filterState); + } let entryPoint = this.getEntryPoint(options); if (!entryPoint) return withStats([], filterState); let entryPointId = entryPoint.id; diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index 8e20f29228..dd2fc615ac 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -457,6 +457,32 @@ describe('HNSW native plane file-primary delivery', function () { assert.ok(fs.existsSync(planePath)); }); + it('answers an empty index with no results instead of a rebuilding 503', async () => { + const EmptyTable = table({ + table: 'PlaneEmpty', + database: DB, + audit: true, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, type: 'Array' }, + ], + }); + const index = EmptyTable.indices.vector.customIndex; + await waitFor(() => !EmptyTable.indices.vector.isIndexing, { + timeout: 15_000, + message: 'the empty derived runtime never became ready', + }); + const results = await index.search( + { target: makeVector(0), comparator: 'sort', distance: 'cosine', ef: EF }, + { transaction: undefined } + ); + assert.deepEqual([...results], []); + // The search target must not publish a placeholder file: a concurrent first insert would + // read it as another worker's in-progress create and reject the write for a full minute. + assert.ok(!fs.existsSync(index.planeFilePath())); + await EmptyTable.dropTable(); + }); + it('requires audit logging and native construction geometry', () => { assert.throws( () => From 91b74028d194a2222c137d90e49c2a17f80c2acc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 23:04:19 -0600 Subject: [PATCH 65/69] Prove emptiness before answering a file-primary vector query with no results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit read "no plane file and this process is not indexing" as "the index is empty". That is also the state immediately after any process removes an unopenable file, so a populated index could answer a vector query with no results instead of reporting that it was unavailable — a successful looking query hiding every indexed vector. Emptiness is now taken only from the surviving node mappings, which is the one fact that proves it, and a new test pins both directions. Two further recovery defects from the same review round: - getPlane() unlinked an unopenable plane and fell through to creating a fresh empty one. In file-primary mode the surviving mappings still name node ids from the removed file, so that silently resolves them against an empty graph. Reconstruction, which clears the mappings first, is now the only path. - The rebuild scan called applyDerivedValue without a guard, so a single record the backend rejects — a malformed vector stored before the index existed — aborted every reconstruction attempt at the same record and left the index permanently unavailable. Such records are now skipped and counted. Also records in §10 that a new origin's first local write forces a full rebuild where replaying that origin's retained log would converge just as well. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 5 +++ resources/DerivedIndexBackend.ts | 15 +++++++- .../HierarchicalNavigableSmallWorld.ts | 23 +++++++++-- unitTests/resources/vectorIndexPlane.test.js | 38 +++++++++++++++++++ 4 files changed, 75 insertions(+), 6 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 5ca5692cca..cd95277464 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -479,6 +479,11 @@ Open: distance the generation check discards. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. +- **A new origin's first local write forces a full rebuild.** An origin log that gains entries + while the index holds no cursor for it cannot be replayed, so `reconcile()` reconstructs from + primary records. Replaying that origin's retained log from its oldest entry would converge + just as well — delivery reconciles against current record state — and would avoid taking + vector search offline the first time each cluster peer writes to the table. - **An interior corrupt audit frame can loop reconstruction.** `replay()` throws on `corruptFrameStop.breaks`, and `rebuild()`'s own catch-up replay starts at the oldest retained entry, so it meets the same frame and throws again: the backoff retries reconstruction diff --git a/resources/DerivedIndexBackend.ts b/resources/DerivedIndexBackend.ts index 5fc82f59f3..b61d00cdaa 100644 --- a/resources/DerivedIndexBackend.ts +++ b/resources/DerivedIndexBackend.ts @@ -354,6 +354,7 @@ export class DerivedIndexRuntime { this.backend.resetDerivedStorage(); await this.indexStore.clear(); let indexed = 0; + let skipped = 0; const total = this.table.primaryStore.getKeysCount?.() ?? 0; for (const { key, value, version, localTime } of this.table.primaryStore.getRange({ versions: true, @@ -361,7 +362,16 @@ export class DerivedIndexRuntime { })) { if (!value) continue; const projected = this.attribute.resolve ? this.attribute.resolve(value) : value[this.attribute.name]; - this.backend.applyDerivedValue(key, projected, localTime ?? version); + try { + this.backend.applyDerivedValue(key, projected, localTime ?? version); + } catch (error) { + // A record the backend rejects (a malformed vector stored before this index existed) + // would otherwise abort every reconstruction attempt at the same record, leaving the + // index permanently unavailable. Skip it and report the count. + if (!(error instanceof ClientError)) throw error; + if (skipped++ === 0) logger.warn?.(`${this.indexStore.name} skipped a record it cannot index`, error); + continue; + } indexed++; if (indexed % REBUILD_PROGRESS_INTERVAL === 0) { await this.backend.flushDerived(); @@ -383,7 +393,8 @@ export class DerivedIndexRuntime { if (indexed) { const elapsedSeconds = Math.max((Date.now() - startedAt) / 1_000, 0.001); logger.info?.( - `Rebuilt ${this.indexStore.name} from ${indexed} records at ${Math.round(indexed / elapsedSeconds)}/s` + `Rebuilt ${this.indexStore.name} from ${indexed} records at ${Math.round(indexed / elapsedSeconds)}/s` + + (skipped ? `, skipping ${skipped} it cannot index` : '') ); } } diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index be9f63956e..d162dcd625 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -415,6 +415,13 @@ export class HierarchicalNavigableSmallWorld { } logger.warn?.('deleting an unopenable HNSW plane file left by an interrupted create', openError); unlinkSync(filePath); + if (this.filePrimary) { + // The surviving mappings still name node ids from the file just removed. + // Creating a fresh plane here would resolve them against an empty graph; + // only reconstruction, which clears the mappings first, is safe. + this.planeRetryAt = now + PLANE_ATTACH_RETRY_MS; + return null; + } // fall through to the create path below } } @@ -531,6 +538,14 @@ export class HierarchicalNavigableSmallWorld { } } + /** True while any node-id mapping survives, which is the only proof this index holds nodes. */ + private hasNodeMappings(): boolean { + for (const { key } of this.indexStore.getRange()) { + if (typeof key === 'number') return true; + } + return false; + } + hasDerivedStorage(): boolean { const filePath = this.planeFilePath(); if (!filePath || !existsSync(filePath)) return false; @@ -1646,12 +1661,12 @@ export class HierarchicalNavigableSmallWorld { } if (this.filePrimary) { const planePath = this.planeFilePath(); - if (this.indexStore?.isIndexing || (planePath && existsSync(planePath))) { + // The absence of a file is not proof of an empty index — it is also the state just after + // any process removes an unopenable one. Only the surviving node mappings prove it, so + // answer with results the JS path would give solely when none remain. + if (this.indexStore?.isIndexing || (planePath && existsSync(planePath)) || this.hasNodeMappings()) { throw new ServerError('The native HNSW index is rebuilding', 503); } - // The runtime finished and created no file: nothing has been committed to this index. - // Match the JS path's empty result rather than reporting a reconstruction that is not - // running, which no consumer converts back into a scan. return withStats([], filterState); } let entryPoint = this.getEntryPoint(options); diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index dd2fc615ac..eadecf604e 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -483,6 +483,44 @@ describe('HNSW native plane file-primary delivery', function () { await EmptyTable.dropTable(); }); + it('reports 503 rather than no results when a populated index has lost its file', async () => { + const LostFile = table({ + table: 'PlaneLostFile', + database: DB, + audit: true, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'vector', indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, type: 'Array' }, + ], + }); + await LostFile.indexingOperation; + const index = LostFile.indices.vector.customIndex; + const probe = makeVector(5); + await LostFile.put(1, { vector: probe }); + await waitFor( + async () => { + if (LostFile.indices.vector.isIndexing) return false; + const hits = await index.search( + { target: probe, comparator: 'sort', distance: 'cosine', ef: EF }, + { + transaction: undefined, + } + ); + return [...hits].some((entry) => entry.key === 1); + }, + { timeout: 15_000, message: 'the populated index never became searchable' } + ); + index.plane = undefined; + fs.rmSync(index.planeFilePath(), { force: true }); + // The node mappings survive the file, so emptiness is not proven: answering [] here would + // hide every indexed vector behind a query that looks successful. + assert.throws( + () => index.search({ target: probe, comparator: 'sort', distance: 'cosine', ef: EF }, { transaction: undefined }), + /rebuilding/ + ); + await LostFile.dropTable(); + }); + it('requires audit logging and native construction geometry', () => { assert.throws( () => From 5514c0b89fbbb8749fa8b77266a67b1458b6a7e8 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 23:17:59 -0600 Subject: [PATCH 66/69] Schedule the repair a lost native file needs, and close the audit-guard hole MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit made a query 503 when a populated index had lost its file, but nothing scheduled the reconstruction that clears it: a table taking no further writes never drains, so the index would have stayed unavailable indefinitely. The query now requests the rebuild it is reporting. An explicit `audit: false` could also slip past the nativePlane guard whenever the Table static already had auditing on, because nothing clears that static. The descriptor then persisted `audit: false` and the next process start failed catalog load on the derived-index attach — a database that will not open. The guard now rejects an explicit false regardless, covered by a test. Also bounds the emptiness probe to the numeric node-id key space instead of scanning the primary-key mappings beside it, and records in §10 the residual this leaves: reconstruction clears the mappings that prove non-emptiness, so a peer worker querying in that window still sees an empty index. `isIndexing` is per-worker, so closing it needs a shared readiness signal. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 4 ++++ resources/databases.ts | 5 ++++- .../HierarchicalNavigableSmallWorld.ts | 17 +++++++++++---- unitTests/resources/vectorIndexPlane.test.js | 21 +++++++++++++++++++ 4 files changed, 42 insertions(+), 5 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index cd95277464..7bdec559e0 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -479,6 +479,10 @@ Open: distance the generation check discards. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. +- **A peer worker can answer an empty result during another worker's rebuild.** Reconstruction + removes the file and then clears the mappings that prove non-emptiness, so a query on a + different worker between that clear and the first re-inserted record sees neither and returns + no results. `isIndexing` is per-worker, so closing this needs a shared readiness signal. - **A new origin's first local write forces a full rebuild.** An origin log that gains entries while the index holds no cursor for it cannot be replayed, so `reconcile()` reconstructs from primary records. Replaying that origin's retained log from its oldest entry would converge diff --git a/resources/databases.ts b/resources/databases.ts index 0ad0abd9a7..9daaf4f62e 100644 --- a/resources/databases.ts +++ b/resources/databases.ts @@ -2297,7 +2297,10 @@ export function table(tableDefinition: TableDefinition): Tabl if ( attributes.some((attribute) => attribute.indexed?.type === 'HNSW' && attribute.indexed.nativePlane) && audit !== true && - Table?.audit !== true + // An explicit false must fail here even for an already-audited Table. Nothing clears the + // static, so the runtime would stay attached while the descriptor persists audit: false, + // and the next process start would fail catalog load on the derived-index attach. + (audit === false || Table?.audit !== true) ) { throw new ClientError( `Table '${databaseName}.${tableName}' must explicitly enable audit logging before using nativePlane because its transaction log is the derived-index recovery source` diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index d162dcd625..e7bdc473bc 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -540,7 +540,9 @@ export class HierarchicalNavigableSmallWorld { /** True while any node-id mapping survives, which is the only proof this index holds nodes. */ private hasNodeMappings(): boolean { - for (const { key } of this.indexStore.getRange()) { + // Node ids are the store's numeric keys, so bounding the probe to that key space keeps this + // off a full scan of the primary-key mappings beside them. + for (const { key } of this.indexStore.getRange({ start: 0, end: Number.MAX_SAFE_INTEGER })) { if (typeof key === 'number') return true; } return false; @@ -1661,10 +1663,17 @@ export class HierarchicalNavigableSmallWorld { } if (this.filePrimary) { const planePath = this.planeFilePath(); + if (this.indexStore?.isIndexing || (planePath && existsSync(planePath))) { + throw new ServerError('The native HNSW index is rebuilding', 503); + } // The absence of a file is not proof of an empty index — it is also the state just after - // any process removes an unopenable one. Only the surviving node mappings prove it, so - // answer with results the JS path would give solely when none remain. - if (this.indexStore?.isIndexing || (planePath && existsSync(planePath)) || this.hasNodeMappings()) { + // any process removes an unopenable one. Only the surviving node mappings prove it. + if (this.hasNodeMappings()) { + // A table taking no further writes never drains, so a query is the only thing left + // that can notice the graph is gone and ask for it back. + this.derivedRuntime?.requestRebuild( + new Error(`${this.indexStore.name} lost its native file while its node mappings survive`) + ); throw new ServerError('The native HNSW index is rebuilding', 503); } return withStats([], filterState); diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index eadecf604e..726f47713c 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -552,6 +552,27 @@ describe('HNSW native plane file-primary delivery', function () { () => customIndex().prepareCommitted('empty-vector', [], undefined, { transaction: {} }), /must contain at least one component/ ); + // An already-audited table must not be able to turn auditing off underneath a nativePlane + // index: nothing clears Table.audit, so the descriptor would persist audit: false and the + // next process start would fail catalog load on the derived-index attach. + assert.throws( + () => + table({ + table: 'PlaneTest', + database: DB, + audit: false, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { name: 'name', indexed: true }, + { + name: 'vector', + indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, + type: 'Array', + }, + ], + }), + /audit logging/ + ); }); (process.env.HNSW_NATIVE_REBUILD_BENCHMARK ? it : it.skip)( From 1eac171d32c05589f20310412c08bb9b35ba1c46 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 23:27:33 -0600 Subject: [PATCH 67/69] Record the shared-readiness gap behind both cross-worker rebuild residuals Both the empty answer a peer can give during a rebuild and the redundant reconstruction a peer can request just before the mappings are cleared come from the same missing fact: isIndexing is per-worker. Names the fix rather than leaving two separate symptoms. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 7bdec559e0..9cfde9efbd 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -479,10 +479,13 @@ Open: distance the generation check discards. - **msync cadence default** — bounded-lag durability window vs write amplification; needs a workload measurement, not a guess. -- **A peer worker can answer an empty result during another worker's rebuild.** Reconstruction - removes the file and then clears the mappings that prove non-emptiness, so a query on a - different worker between that clear and the first re-inserted record sees neither and returns - no results. `isIndexing` is per-worker, so closing this needs a shared readiness signal. +- **Workers have no shared readiness signal during reconstruction.** `isIndexing` is per-worker, + and the two gaps that follow both come from that. Reconstruction removes the file and then + clears the mappings that prove non-emptiness, so a query on a different worker between the + clear and the first re-inserted record sees neither and returns no results; and in the shorter + window before the clear, that worker sees mappings with no file and requests a reconstruction + of its own, which the writer lock serializes into one redundant rebuild rather than a loop. + A shared flag beside the existing `getUserSharedBuffer` depth counters would close both. - **A new origin's first local write forces a full rebuild.** An origin log that gains entries while the index holds no cursor for it cannot be replayed, so `reconcile()` reconstructs from primary records. Replaying that origin's retained log from its oldest entry would converge From 4a38085cd8b47a21d348e5def675134966b4cb94 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 8 Sep 2026 23:31:33 -0600 Subject: [PATCH 68/69] Correct the recurrence claim in the shared-readiness note The writer lock bounds concurrency, not recurrence: each rebuild re-opens the same unlink-then-clear window on the other worker, so the sequence stopping depends on query timing rather than on a guarantee. The previous wording claimed a bound the lock does not provide. Co-Authored-By: Claude Opus 5 --- hnsw-native-plane.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 9cfde9efbd..50b41daa2a 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -484,7 +484,9 @@ Open: clears the mappings that prove non-emptiness, so a query on a different worker between the clear and the first re-inserted record sees neither and returns no results; and in the shorter window before the clear, that worker sees mappings with no file and requests a reconstruction - of its own, which the writer lock serializes into one redundant rebuild rather than a loop. + of its own. The writer lock keeps two workers from reconstructing concurrently, but it does + not bound recurrence: each rebuild re-opens the same unlink-then-clear window on the other + worker, so whether the sequence stops depends on query timing rather than on a guarantee. A shared flag beside the existing `getUserSharedBuffer` depth counters would close both. - **A new origin's first local write forces a full rebuild.** An origin log that gains entries while the index holds no cursor for it cannot be replayed, so `reconcile()` reconstructs from From dd3bb398a6293587d7c8ec9d913a7ae8ce7fe2be Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 9 Sep 2026 07:35:25 -0600 Subject: [PATCH 69/69] Make one plane-vector invariant, and plan the shared-runtime convergence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ordinary input could take a file-primary index down for hours. A stored vector whose length differs from the plane's create-time dims, a component finite as a double but not as the f32 the plane stores, one whose square overflows f32 even though it does not (storing invMag 0, which makes every distance involving that node NaN), a decoded BigInt, or a value with no length at all: each reached the crate or Float32Array.from as an error neither the search path nor reconstruction can attribute to the input. A record aborted every rebuild attempt at the same entry; a query was read as plane corruption and unlinked a healthy file, costing a reconstruction the design measures in hours at 16M nodes. Validate the invariant once, in assertPlaneVector, and apply it at both entry points: the committed projection and the search target. The dimension check runs again before the native insert, where a worker that had no plane to compare against finally has one, and before the old node is removed. Rejecting bad input is not enough on its own: only the rebuild scan skipped records the backend cannot index, while replay rethrew them, and rebuild finishes by replaying from the oldest retained entry — so every record the scan skipped was met again and reconstruction could never finish. Give replay the same skip-and-count guard. Retiring a runtime now fences its in-flight rebuild and replay: neither checked `closed` at any await, so on the table() redefine path the retired runtime's final `isIndexing = false` landed after its replacement had set it and this worker served searches from the half-built generation; a retired runtime no longer advances a durable cursor either. A failure in the id-mapping reads that follow a successful traversal also no longer counts as a plane failure, so a transient RocksDB read or a store closed under an in-flight search cannot unlink the file. hnsw-native-plane.md §13 records an integrated ingest benchmark and the convergence plan it argues for: adopt #2533's runtime, add the coalesced delivery view, bounded collection and delivery, an explicit durability cadence, a rebuild phase, generation fencing and shared readiness, and reduce this PR to an HNSW backend. §13.6 retires the rebuild-anchor question the first draft deferred to the storage layer: a committed read is bounded at one physical offset and walks every byte up to it, so nothing sits behind the tail it yields and the tail is already a safe anchor. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VhMkk4iZBE8zEQcnvu1syc --- hnsw-native-plane.md | 244 +++++++++++++++++- resources/DerivedIndexBackend.ts | 41 ++- .../HierarchicalNavigableSmallWorld.ts | 99 +++++-- .../resources/hnswDerivedIngest.bench.js | 244 ++++++++++++++++++ unitTests/resources/vectorIndexPlane.test.js | 74 +++++- 5 files changed, 661 insertions(+), 41 deletions(-) create mode 100644 unitTests/resources/hnswDerivedIngest.bench.js diff --git a/hnsw-native-plane.md b/hnsw-native-plane.md index 50b41daa2a..d69667cbbd 100644 --- a/hnsw-native-plane.md +++ b/hnsw-native-plane.md @@ -512,13 +512,11 @@ Open: - **Rebuild replays the whole retained log, not just the scan window.** After the primary-store scan, `rebuild()` sets each origin's cursor to the _oldest_ retained entry rather than the log tail at scan start, so the follow-up `replay()` re-reads and re-hashes the entire retention - window before the index leaves 503. It is conservative on purpose and cannot be tightened to - the tail: `RocksTransactionLogStore.addEntry` stages an entry into the log at write time and - the commit hook only publishes it, so a transaction still open when the scan begins already - occupies a physical position behind that tail and a tail-anchored cursor would skip its - entries entirely. The correct tighter bound is the oldest entry belonging to a transaction - uncommitted at scan start; Harper does not expose that position today. Until it does, rebuild - cost scales with retention rather than with scan duration. + window before the index leaves 503. It is more conservative than it needs to be, and §13.6 has + the reading of rocksdb-js that says so: a committed-only query is bounded at one physical offset + and reads every byte up to it, so nothing sits behind the tail it yields and that tail is a safe + anchor. Tightening it is work for the shared runtime's rebuild phase; until then rebuild cost + scales with retention rather than with scan duration. - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the format (dims × mode in header) but int8 is the default and the optimization target. - ~~Upper-layer region persistence~~ — done (format v2): fixed-entry region in the same file, @@ -575,3 +573,235 @@ optimizeRouting-parity insert (including the recomputed neighbor↔neighbor dist recall from 0.49 (placeholder insert) to JS parity. Uniform-random 768-d corpora produce meaningless recall numbers (the JS benchmark's own calibration note: a corpus "no ANN can index") — all comparisons use the mixture corpus. + +## 13. Convergence onto the shared derived-index runtime (#2533 / #2535) + +[#2533](https://github.com/HarperFast/harper/pull/2533) implements #2489's shared transaction-log +runtime (`resources/derivedIndexRuntime.ts`); [#2535](https://github.com/HarperFast/harper/pull/2535) +is stacked on it and adds a RocksDB storage adapter for native derived indexes. +`resources/DerivedIndexBackend.ts` on this branch is a second, HNSW-shaped implementation of the +same protocol. One runtime is kept — #2533's — and HNSW becomes a backend on it. + +### 13.1 What the measurements decide + +`unitTests/resources/hnswDerivedIngest.bench.js` runs a real audited table with a file-primary +HNSW index through three shapes. Measured at 384 dimensions, `@harperfast/hnsw` 0.2.1, one worker: + +| shape | graph ≈3,000 | graph ≈25,000 | +| -------------------------------------------- | ------------------------------- | ------------------------------- | +| foreground `put` | 0.320 ms (3,125/s), p99 10.1 ms | 0.509 ms (1,964/s), p99 17.0 ms | +| `applyDerivedValue` | 0.214 ms/call | 0.389 ms/call | +| `flushDerived` | 3.59 ms × 50 calls | 4.18 ms × 129 calls | +| backend share of the write-and-index wall | 66.9–94.9% of 641 ms | 76.2–97.2% of 2,554 ms | +| end-to-end indexed rate | 3,121/s | 1,957/s | +| event loop, max/p99 **while writing** | 19.9 / 18.0 ms | 31.9 / 25.7 ms | +| event loop, max/p99 **while draining alone** | 0.0 / 0.0 ms | 2.6 / 2.6 ms | +| serialized single writes | 4.97 ms/record, ≤88.8% barrier | 4.05 ms/record, ≤79.1% barrier | +| 50 keys × 20 rounds: repeated `apply` calls | 95% | 95% | + +Package cost in isolation (N = 10,000): insert 208 µs (128-d) / 315 µs (384-d) / 835 µs (1536-d); +update (remove + insert) 287 / 468 / 1,399 µs. `flushAsync` scales with the **dirty set**, not with +index size, and has a floor: after a single insert it costs 2.3 / 3.4 / 4.3 ms, and after 10,000 +inserts at 384-d it costs 181.7 ms — 3.4 ms per record against 18 µs per record, a ~190× spread +that is the whole case for a cadence. + +What the instrumentation does and does not separate: the meter wraps whole backend methods, so +`applyDerivedValue` carries the vector hash and the RocksDB mapping writes as well as the native +insert, and `flushDerived` carries publishing those mappings as well as the msync. Those rows +therefore bound the _backend's_ share, not the native share; the isolated package numbers above are +what establish the native term inside it. The backend-share row is a range for a second reason: +`applyDerivedValue` is synchronous, so its time is exact and is the floor, while `flushDerived` is +timed across an `await` and so charges the backend for anything the loop ran during the barrier — +the ceiling. The serialized row's barrier share is bounded the same way, for the same reason. Two +rows also measure less than they look like: + +- The serialized row awaits full drain between writes, so nothing is available to combine. It + bounds the cost of one isolated write — one barrier per record, ~80–90% of it — and does **not** + measure what a flush cadence could amortize. Arrivals paced independently of the drain are the + missing experiment. +- The 95% is repeated keys across the whole run, not within one delivery window. Batch coalescing + removes only the repeats that land in the same batch, so 95% is the ceiling, not the saving. + +Three conclusions do hold. + +1. **Per-transaction bookkeeping is not what decides HNSW throughput.** The backend's synchronous + apply alone is 67–76% of the wall clock of a write-and-index cycle, and adding the barrier's + wall-clock time — an over-count, since it spans an await — reaches 95–97%. Even at the floor, + everything else, foreground write work and runtime bookkeeping together, has at most a third of + the wall to share. The concern that #2533's collect/resolve + path is less optimized than this branch's is real in the small but cannot pay for a second + runtime: both implementations decode the same log entries and read the same authoritative + record. +2. **This branch forces a barrier per drain; #2533 permits amortization but does not schedule it.** + `replay()` awaits `flushDerived()` at every origin's final cursor advance. #2533 separates + _offered_ from _durable_ progress and lets a backend accept up to `maxAcceptedBatchesAhead` + batches before its barrier — but that is a ceiling, not a scheduler. A backend that flushes each + accepted batch keeps this branch's cost, and one that waits only for the ceiling can leave a lone + write undurable indefinitely. The cadence has to be specified, not inherited. +3. **The drain's blocking lands on the foreground write path, and neither runtime bounds it.** + Draining alone barely touches the event loop (0.0–2.6 ms), but while writes are in flight the + drain runs inside their awaits and the loop blocks for 20–32 ms, showing up as a `put` p99 of + 10–17 ms against a 0.06 ms median. `drain()` applies up to 128 keys between awaits, so a turn is + up to 128 × 0.39 ms of synchronous native work. #2533 is not better placed: `#collectBatch` + completes a whole transaction before checking any budget, `#resolveTransactions` then resolves + every distinct key and projection outside that check, and `deliver()` runs after it — so + `maxTransactionsPerTurn: 256` and `maxMillisecondsPerTurn: 5` bound neither one large transaction + nor the applied cost of a batch. A queue-and-accept backend converts the applied cost into queue + depth, which is why §13.2 bounds collection and resolved payload rather than application alone. + +### 13.2 Changes the shared runtime needs (stacked on #2535) + +- **Coalesced delivery view.** Add a batch-level, last-write-wins view over distinct + `(tableId, recordId)` alongside the existing `transactions` array, keeping `writeKeyId` identity + and the whole `through` cursor vector intact. `#resolveTransactions` already resolves each key + once and hands every occurrence the same resolved object; what repeats is the mutation wrapper, + which for a backend costing 0.2–1.4 ms per mutation is the expensive part. The coalesced entry + keeps the last `logVersion` in batch order, the only field that differs between occurrences. +- **Bounded collection, resolution and delivery.** Make the budget cover the work that is actually + unbounded: incremental collection with no cursor publication until a complete transaction is + covered, or an explicit oversized-transaction policy; payload accounting that includes pending, + deferred and accepted bytes, with a stated way to estimate projection size without serializing it + twice; and time-sliced yielding rather than one `setImmediate` per mutation. Alongside it, make + `maxTransactionsPerTurn`, `maxBytesPerTurn`, `maxMillisecondsPerTurn` and + `maxAcceptedBatchesAhead` settable per registration — a vector backend and a full-text backend + want different values — and correct the Stage 1 sentence claiming `deliver()` is "included in the + runner's wall-time budget", which the code does not enforce. +- **An explicit durability cadence.** Specify what obliges a backend to flush: a maximum flush age + so an isolated write becomes durable promptly, work and byte thresholds so a burst amortizes, idle + completion, and shutdown behaviour. Without it, "adopt #2533" buys the _permission_ to amortize + and none of the amortization. The thresholds are not free choices: the barrier grows with the + dirty set (§13.1), so a cadence that waits for a large one trades per-record cost for a longer + single stall, and both ends need a stated bound. +- **Rebuild as a runtime phase, on the conservative boundary.** #2533 stops at `needs-rebuild`: + `#needsRebuild()` logs, drops the iterator, releases the lock, and the index stays unavailable. + Reset storage → scan the primary store → project → deliver in bounded batches → replay → resume is + identical for HNSW and full-text and belongs in the runtime. Anchor catch-up on the tail of a + committed read taken at scan start — §13.6 shows why that is safe and why this branch's + oldest-retained anchor is a cost with no correctness return — and keep the exact-boundary + validation and fail-closed behaviour when retention or corruption prevents proof. +- **Generation fencing and cancellation.** Asynchronous acceptance makes ownership handoff unsafe + without it: worker A can accept a batch, release the runner lock, and later run a scheduled + mutation or a flush completion after worker B has reset the index, publishing A's mappings or + readiness into B's generation. The backend interface needs a queue-drain/cancellation handshake; + the runtime needs epoch checks before mutation and after every await, queue shutdown ordered + before lock release, `rebuilding` published before any destructive reset, and `ready` published + only after scan, catch-up and the final barrier. +- **Shared cross-worker readiness.** `indexStore.isIndexing` is per-worker and `getStatus()` is only + meaningful on the owner, so a non-owning worker can answer from a partially built index. Publish + readiness and its reason in a shared buffer beside the owner-epoch counter, and fence peer readers + with it — this is the same mechanism the fencing above needs, and it closes the two cross-worker + rebuild-window residuals in §10. +- **A lag policy, or an explicit decision not to have one.** Indexing capacity is one insert per + changed vector plus the barrier — about 2,000 records/s per index at 384 dimensions, falling with + dimension and graph size — while the foreground work of a `put` is 0.06 ms. A write stream that + does not share the owning worker's event loop, which is any peer worker or any client pipelined + across workers, therefore exceeds indexing capacity by a wide margin. The single-worker benchmark + cannot show that gap directly, because there the writer and the drain contend for one loop and + both land near 2,000/s. Deleting this branch's runtime deletes its retryable 503, and #2533 has no + replacement, so sustained overload runs the cursor past audit retention, rebuilds, and falls + behind again. Preserve the admission behaviour or approve the changed availability contract + explicitly, with observable lag; queue memory pressure and retention lag are separate signals. + +### 13.3 What #2430 becomes + +`resources/DerivedIndexBackend.ts` is deleted. `HierarchicalNavigableSmallWorld` implements #2533's +`DerivedIndexBackend` interface: `deliver()` queues and returns accepted, an applier drains it in +bounded time slices, `flushAsync()` is the barrier, and the cursor vector is stored with the plane +generation. HNSW does **not** use #2535's `RocksDerivedIndexStorage`: its durability barrier is a +database-wide `flushSync({ allowWriteStall: true })`, whereas an mmap index needs only `msync` of +its own file. + +Three constraints the async shape imposes, none of which exist in today's synchronous path: + +- **The flush cut must be immutable.** `flushDerived()` awaits the plane barrier and then publishes + the live `pendingDerivedMappings` map. Once application can continue during that await, a later + mutation's mapping enters that map and is published by an earlier barrier, so a crash can leave a + published mapping naming native state the barrier never covered. Snapshot the pending set at + barrier entry, or serialize application against flushing. +- **`msync` is not the whole durability unit.** The graph, both mapping directions, the generation + metadata and the offered cursor vector all have to have a stated persistence order, and `msync` + alone does not establish durability of the RocksDB half. Restart tests belong at each publication + boundary; an ordinary database reset does not stand in for power-loss ordering. +- **Cursor migration is versioned.** Per-origin `Symbol.for('derived-index-cursor:…')` keys become + the shared cursor vector. An old cursor read against a fresh plane generation must force + reconstruction rather than resume. + +### 13.4 Approaches considered + +**Invariant:** one delivery/recovery implementation, in which a published cursor certifies a +complete durable prefix for that index generation and every committed change outside that prefix +stays replayable. + +**Different layer.** Keep both runtimes and share only the transaction-log reader changes. Rejected: +ownership election, cursor validation and recovery stay duplicated, which is the failure #2489 +exists to prevent. + +**Deeper cause.** Move HNSW insertion into the package's own thread pool so delivery cost stops +being event-loop cost. Genuinely the deeper fix for the event-loop term, but 0.2.1 exposes only a +synchronous `insert`, it removes neither the duplicated runtime nor the repeated-key work, and it is +phase-3. + +**Do less.** Adopt #2533 unchanged and put coalescing, chunking and rebuild inside the HNSW backend. +Rejected for rebuild, which is not backend-specific and whose duplication is how two runtimes came +to exist. The coalescing half of this argument is narrower than it first looked: #2533 already +resolves each key once, so backend-private coalescing would discard repeated mutation wrappers, not +repeated primary reads — still worth doing in the runtime, but as an allocation and dispatch saving. + +**Chosen (revised after planning review).** One stacked PR on #2535 adds the coalesced view, bounded +collection/resolution/delivery, an explicit durability cadence, the rebuild phase, generation +fencing with cancellation, and shared readiness. #2430 then rebases onto it and becomes an HNSW +backend only. + +The first draft of this plan also put a tighter rebuild boundary in the same PR — the earliest +position uncommitted at scan start, derived from `readUncommitted` or from per-worker staged +timestamps. The planning review disqualified it on a fact rather than a preference: the shared +runner resumes _after_ a complete transaction at its exact cursor, so installing the oldest staged +transaction A as the cursor skips A's own transaction entirely, and if A aborts, that exact +committed boundary may never exist and cursor validation fails. A correct version needs a distinct +_inclusive_ rebuild anchor with its own transition into a durable cursor, plus synchronization +between capture, staging publication, commit visibility, abort, worker replacement and retention — +a correctness-sensitive change to every audited writer, in exchange for an unmeasured rebuild-cost +optimization. Reading the pinned rocksdb-js afterwards showed the whole construction to be +unnecessary rather than merely misplaced — a committed read is a contiguous byte prefix, so the tail +is already a safe anchor (§13.6). The rejected alternative was right that the proposal did not +belong in this PR; it was solving a problem that is not there. + +### 13.5 Sequencing + +1. Stacked PR on `codex/fulltext-storage-adapter` with §13.2. Its gate is #2533's existing + derived-index suite plus a fake backend carrying a synthetic per-mutation cost, exercising one + oversized transaction, ownership handoff while an apply is scheduled and while a flush is + pending, and independently paced arrivals for the cadence. +2. Rebase #2430 onto that branch; delete `DerivedIndexBackend.ts`; port `vectorIndexPlane.test.js` + alongside the runtime suite and require native availability in the designated gate rather than + accepting a skip. Targets are numeric — write latency, indexed throughput, queue memory and + maximum durability age — because a falling flush _share_ can also mean unrelated work got slower. +3. #2533 and #2535 merge first; #2430 stays draft until then. + +### 13.6 The rebuild anchor, resolved + +The conservative anchor makes rebuild replay the whole retention window (§10). The first draft of +this plan proposed tightening it with a boundary derived from staged/uncommitted positions, the +planning review disqualified that construction, and it was deferred to the storage layer. Reading +the pinned rocksdb-js retires the whole line of work: no storage change is needed, and the tail is +already safe. + +`TransactionLog.query()` resolves its end through `loadLastPosition()`, which decodes +`_lastCommittedPosition` into a single `{ logId, size }` physical offset; the iterator then walks +`while (position < size)`. A committed-only read is therefore a **contiguous byte prefix** of the +log, not a per-transaction filter over a sparse range. Nothing can sit physically behind the last +entry it yields, so anchoring rebuild catch-up on that entry cannot skip anything — the hazard the +conservative anchor exists to avoid does not exist. + +One question is left, and it is native-side: whether `_lastCommittedPosition` advances only to a +contiguous committed frontier, or to the end of whichever transaction committed most recently. Under +the second reading a committed read can return entries of a transaction that has not committed yet, +and would return entries of one that later aborts. That is not a rebuild-anchor problem — it is a +property every committed reader in Harper already has, replay and replication included — and this +protocol absorbs it: delivery re-reads the authoritative record rather than applying the log entry's +body (§8), so a phantom entry resolves to current state and is idempotent. + +So the shared runtime's rebuild phase (§13.2) should anchor on the tail of a committed read taken at +scan start, and this branch's oldest-retained anchor is a cost with no correctness return. It is not +changed here because this runtime is the one being deleted; the tail anchor belongs in the +replacement, where it is a few lines rather than a storage-layer project. diff --git a/resources/DerivedIndexBackend.ts b/resources/DerivedIndexBackend.ts index b61d00cdaa..a3b12eedff 100644 --- a/resources/DerivedIndexBackend.ts +++ b/resources/DerivedIndexBackend.ts @@ -31,7 +31,7 @@ export interface DerivedIndexBackend { hasDerivedStorage(): boolean; } -function valuesEqual(a: any, b: any): boolean { +export function valuesEqual(a: any, b: any): boolean { if (a === b) return true; // Element-wise comparison is only defined for the array-like projections backends index. // Anything else that already failed `===` counts as changed rather than silently unstaged. @@ -291,6 +291,7 @@ export class DerivedIndexRuntime { } private async replay(logs: any[], ignoreReloadsBefore?: number): Promise { + let skipped = 0; for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; const cursor = this.indexStore.getSync(this.cursorKey(nodeId)); @@ -317,8 +318,21 @@ export class DerivedIndexRuntime { const record = current?.value; const value = record && (this.attribute.resolve ? this.attribute.resolve(record) : record[this.attribute.name]); - this.backend.applyDerivedValue(entry.recordId, value, current?.localTime ?? current?.version); - if (++applied % APPLY_BATCH_SIZE === 0) await this.backend.flushDerived(latest); + try { + this.backend.applyDerivedValue(entry.recordId, value, current?.localTime ?? current?.version); + } catch (error) { + // Same rule as the rebuild scan: a record the backend rejects is unindexable, not a + // delivery failure. Without this the two disagree — rebuild ends by replaying from + // the oldest retained entry, so every record its scan skipped is met again here and + // rethrown, and reconstruction can never finish while that entry is retained. + if (!(error instanceof ClientError)) throw error; + if (skipped++ === 0) logger.warn?.(`${this.indexStore.name} skipped a record it cannot index`, error); + continue; + } + if (++applied % APPLY_BATCH_SIZE === 0) { + await this.backend.flushDerived(latest); + if (this.closed) return; + } } } if (entries.corruptFrameStop.breaks) { @@ -329,6 +343,9 @@ export class DerivedIndexRuntime { // rather than a numeric greater-than comparison, identifies forward progress. if (latest !== cursor) { await this.backend.flushDerived(latest); + // A retired runtime must not advance a durable cursor: its replacement may already + // have reset the generation this position describes. + if (this.closed) return; this.indexStore.putSync(this.cursorKey(nodeId), latest); } } @@ -338,11 +355,9 @@ export class DerivedIndexRuntime { const startedAt = Date.now(); this.ready = false; this.indexStore.isIndexing = true; - // Keep the oldest retained physical entry as the catch-up boundary. addEntry stages an entry - // into the log at write time and the commit hook only publishes it, so a transaction still - // open when this scan begins already sits physically behind the current tail. Anchoring on - // the tail — or on any wall-clock instant — would skip its entries once it commits. The cost - // is that replay re-walks the whole retention window; see hnsw-native-plane.md §10. + // A committed read is bounded at one physical offset and returns every byte up to it, so the + // tail it yields is already a safe anchor; the oldest retained entry is chosen only so this + // runtime, which is being replaced, is not the place that changes recovery semantics. const boundaries: Array = []; for (let nodeId = 0; nodeId < logs.length; nodeId++) { if (!logs[nodeId]) continue; @@ -380,14 +395,22 @@ export class DerivedIndexRuntime { const etaSeconds = rate > 0 ? Math.max(0, Math.ceil((total - indexed) / rate)) : undefined; logger.info?.(`Rebuilding ${this.indexStore.name}: ${indexed}/${total} records, ${rate}/s, ETA ${etaSeconds}s`); } - if (indexed % APPLY_BATCH_SIZE === 0) await new Promise((resolve) => setImmediate(resolve)); + if (indexed % APPLY_BATCH_SIZE === 0) { + await new Promise((resolve) => setImmediate(resolve)); + if (this.closed) return; + } } await this.backend.flushDerived(Math.max(1, ...boundaries.filter((value) => value != null))); + if (this.closed) return; for (let nodeId = 0; nodeId < logs.length; nodeId++) { const boundary = boundaries[nodeId]; if (boundary) this.indexStore.putSync(this.cursorKey(nodeId), boundary); } await this.replay(logs, startedAt); + // A retired runtime must not publish readiness: on the table() redefine path its replacement + // has already set isIndexing, and clearing it here would let this worker serve searches from + // the half-built generation the replacement is still filling. + if (this.closed) return; this.ready = true; this.indexStore.isIndexing = false; if (indexed) { diff --git a/resources/indexes/HierarchicalNavigableSmallWorld.ts b/resources/indexes/HierarchicalNavigableSmallWorld.ts index e7bdc473bc..90f9d4cb7a 100644 --- a/resources/indexes/HierarchicalNavigableSmallWorld.ts +++ b/resources/indexes/HierarchicalNavigableSmallWorld.ts @@ -4,6 +4,7 @@ import { FLOAT32_OPTIONS } from 'msgpackr'; import { loggerWithTag } from '../../utility/logging/logger.ts'; import { ClientError, ServerError } from '../../utility/errors/hdbError.ts'; import type { Id } from '../../resources/ResourceInterface.ts'; +import { valuesEqual as derivedValuesEqual } from '../DerivedIndexBackend.ts'; import type { DerivedIndexRuntime } from '../DerivedIndexBackend.ts'; import { SKIP } from '@harperfast/extended-iterable'; import { RocksDatabase } from '@harperfast/rocksdb-js'; @@ -131,7 +132,7 @@ const PLANE_STALE_CREATE_MS = 60_000; const PLANE_ATTACH_RETRY_MS = 250; // Marks an error thrown by an app-supplied filter during a plane search: the caller re-raises // it as an ordinary query failure instead of disabling the (healthy) plane. -const PLANE_PREDICATE_ERROR = Symbol('planePredicateError'); +const NOT_A_PLANE_FAILURE = Symbol('notAPlaneFailure'); class MinHeap { private data: Candidate[] = []; @@ -355,7 +356,6 @@ export class HierarchicalNavigableSmallWorld { logger.info?.('deleted the HNSW plane file of an index no longer using nativePlane'); } catch (error: any) { if (error?.code !== 'ENOENT') { - // the file survives (Windows EBUSY while another process maps it), and nothing // A later re-enable must not adopt a file that missed mutations while disabled. logger.warn?.('could not delete the HNSW plane file; marking it stale', error); this.invalidatePlaneFile(filePath); @@ -422,7 +422,6 @@ export class HierarchicalNavigableSmallWorld { this.planeRetryAt = now + PLANE_ATTACH_RETRY_MS; return null; } - // fall through to the create path below } } if (!dims) return null; // open-only call and no file: nothing to attach yet @@ -617,19 +616,32 @@ export class HierarchicalNavigableSmallWorld { // the plane itself is healthy; mark the failure as the application's so the caller // re-raises it rather than disabling the plane and retrying try { - (predicateError as any)[PLANE_PREDICATE_ERROR] = true; + (predicateError as any)[NOT_A_PLANE_FAILURE] = true; } catch { // a frozen/primitive throw still propagates, it just also disables the plane } throw predicateError; } const entries: any[] = []; - for (const hit of hits) { - const mapping = this.safeGetSync(hit.id, options); - if (mapping?.pending) continue; - const primaryKey = mapping?.primaryKey; - if (primaryKey === undefined) continue; // deleted/reused id raced the search - entries.push({ key: primaryKey, distance: hit.distance }); + try { + for (const hit of hits) { + const mapping = this.safeGetSync(hit.id, options); + if (mapping?.pending) continue; + const primaryKey = mapping?.primaryKey; + if (primaryKey === undefined) continue; // deleted/reused id raced the search + entries.push({ key: primaryKey, distance: hit.distance }); + } + } catch (error) { + // The traversal already succeeded; this is a RocksDB read of the id mappings, which + // can fail transiently or because the store closed under an in-flight search. Tagging + // it keeps the caller from reading it as a plane failure and unlinking a healthy file, + // which costs a full reconstruction. + try { + (error as any)[NOT_A_PLANE_FAILURE] = true; + } catch { + // a frozen/primitive throw still propagates, it just also disables the plane + } + throw error; } // nodesVisited stays 0 here: layer-0 visits happen inside the native traversal // (filterEvaluations is still counted by the predicate adapter) @@ -642,21 +654,56 @@ export class HierarchicalNavigableSmallWorld { } prepareCommitted(primaryKey: Id, vector: number[], existingVector: number[], options: any): void { + // Validation is O(dims) and runs on the commit path of every write to the table, so skip it + // when the projection did not change — an unchanged vector was validated when it was first + // written. It must still precede stage(): a rejected write that leaves a target behind makes + // committed() find no audit entry for the id and escalate to a full rebuild. + if (derivedValuesEqual(vector, existingVector)) return; this.validateVector(primaryKey, vector); this.derivedRuntime?.stage(options.transaction, primaryKey, vector, existingVector); } private validateVector(primaryKey: Id, vector?: number[]): void { if (!vector) return; - if (vector.length === 0) { - throw new ClientError(`Vector for attribute "${String(primaryKey)}" must contain at least one component.`); + this.assertPlaneVector(vector, `Vector for attribute "${String(primaryKey)}"`); + } + + /** + * Everything reaching the plane — a committed record's projection or a query target — has to + * convert to the f32 it stores, and to a representable magnitude. What fails here would instead + * throw out of `Float32Array.from` or out of the crate, as an error neither the search path nor + * reconstruction can attribute to the record: a query would unlink a healthy file, and a record + * would abort every rebuild attempt at the same entry. + */ + private assertPlaneVector(vector: number[], label: string): void { + // A positive integer length, not merely a numeric one: a negative or fractional length skips + // the component loop and the emptiness check, and reaches Plane.create with it. + const length = (vector as any)?.length; + if (!Number.isInteger(length) || length < 1) { + throw new ClientError(`${label} must be an array of at least one number.`); } + let sumOfSquares = 0; for (let i = 0; i < vector.length; i++) { - if (!Number.isFinite(vector[i])) { + const component = vector[i]; + // The type check precedes Math.fround, which throws a TypeError on the BigInt a msgpackr + // or cbor-x decode produces for a large int64. + if (typeof component !== 'number' || !Number.isFinite(Math.fround(component))) { throw new ClientError( - `Vector for attribute "${String(primaryKey)}" contains non-finite component at index ${i}: ${vector[i]}. Ensure the embedding produces only finite values.` + `${label} has a component at index ${i} that is not a finite 32-bit float: ${String(component)}.` ); } + const asFloat32 = Math.fround(component); + sumOfSquares += asFloat32 * asFloat32; + } + // Components can each be f32-finite while their squares are not, which stores invMag 0 and + // makes every distance involving that node NaN. + if (!Number.isFinite(Math.fround(sumOfSquares))) { + throw new ClientError(`${label} has a magnitude too large to represent in 32-bit floats.`); + } + // The plane's dimensionality is fixed at create time by the first committed vector. + const dims = this.plane?.dims; + if (dims !== undefined && vector.length !== dims) { + throw new ClientError(`${label} has ${vector.length} components, but this index stores ${dims}.`); } } @@ -689,6 +736,14 @@ export class HierarchicalNavigableSmallWorld { this.plane = undefined; plane = vector ? this.getPlane(vector.length, true) : this.getPlane(); } + // The pre-commit check above ran before this worker had a plane to compare against, so it + // cannot have caught a mismatch. Reject before the removal below, or the record loses its + // old node on the way to failing. + if (vector && plane && vector.length !== plane.dims) { + throw new ClientError( + `Vector for attribute "${String(primaryKey)}" has ${vector.length} components, but this index stores ${plane.dims}.` + ); + } if (oldNodeId != null) { plane?.remove(oldNodeId); this.indexStore.removeSync(oldNodeId); @@ -1639,16 +1694,18 @@ export class HierarchicalNavigableSmallWorld { // reported distances of whatever candidates came back, not which candidates the beam kept. if (this.planeEligible && distanceFunction === this.distance) { const plane = this.getPlane(target.length, false); - // a query whose dimensionality differs from the graph's takes the JS path (which - // tolerates the mismatch) rather than erroring or disabling the healthy plane + // A file-primary index has no JS path to fall through to, so a target the plane cannot + // accept has to fail as the client error it is. Otherwise it throws out of + // Float32Array.from or the traversal, is read as plane corruption, and unlinks a healthy + // file — one malformed query costing a full reconstruction. + if (this.filePrimary && plane) this.assertPlaneVector(target, 'Search target'); + // a non-file-primary query whose dimensionality differs from the graph's takes the JS + // path, which tolerates the mismatch, rather than disabling the healthy plane if (plane && plane.dims === target.length && this.planeSearchReady(plane)) { - // Native cutover: same resolved ef, same predicate semantics; resolves to the same - // entries shape ({ key, distance }) the JS path returns, so rescoreResults and all - // post-load behavior are unchanged. searchByIndex handles the promise. try { return this.searchPlane(plane, target, effectiveEf, filter, filterState, options).catch((error) => { - // an app filter's own throw is the query's failure, not the plane's - if (error?.[PLANE_PREDICATE_ERROR]) throw error; + // the query failed for a reason outside the traversal: re-raise instead of disabling the file + if (error?.[NOT_A_PLANE_FAILURE]) throw error; // There is no JS graph behind a file-primary index: it stays unavailable until // its audit-backed rebuild succeeds. this.disablePlane(error); diff --git a/unitTests/resources/hnswDerivedIngest.bench.js b/unitTests/resources/hnswDerivedIngest.bench.js new file mode 100644 index 0000000000..27bbb96697 --- /dev/null +++ b/unitTests/resources/hnswDerivedIngest.bench.js @@ -0,0 +1,244 @@ +// Integrated ingest benchmark for the file-primary HNSW derived index. Excluded from +// `test:unit:resources` by its `.bench.js` name; run it directly: +// +// HOME= npx mocha unitTests/resources/hnswDerivedIngest.bench.js +// +// The meter wraps whole backend methods: `applyDerivedValue` includes the vector hash and the +// RocksDB mapping writes as well as the native insert, and `flushDerived` includes publishing +// those mappings as well as the msync. It therefore bounds the backend's share, and does not +// separate native from JS inside it — the isolated package numbers do that. +// +// The serialized case awaits full drain between writes, so it measures the cost of one isolated +// write, not what a flush cadence could amortize. The repeated-key case counts distinct keys +// across the whole run, not within one delivery window. +require('../testUtils'); +const assert = require('node:assert'); +const { monitorEventLoopDelay } = require('node:perf_hooks'); +const { setupTestDBPath } = require('../testUtils'); +const { waitFor } = require('../waitFor'); +const { table } = require('#src/resources/databases'); +const { getPlaneBinding } = require('#src/resources/indexes/hnswPlaneBinding'); +const { derivedIndexCursorKey } = require('#src/resources/DerivedIndexBackend'); +const { setMainIsWorker } = require('#js/server/threads/manageThreads'); + +const DIMS = Number(process.env.BENCH_DIMS ?? 384); +const SEED = Number(process.env.BENCH_SEED ?? 1000); +const BURST = Number(process.env.BENCH_BURST ?? 2000); +const TRICKLE = Number(process.env.BENCH_TRICKLE ?? 40); +const HOT_KEYS = Number(process.env.BENCH_HOT_KEYS ?? 50); +const HOT_ROUNDS = Number(process.env.BENCH_HOT_ROUNDS ?? 20); +const DB = 'vector-ingest-bench'; + +let seedState = 42; +function rand() { + seedState = (seedState * 1103515245 + 12345) % 2147483648; + return seedState / 2147483648; +} +function makeVector() { + const vector = new Array(DIMS); + for (let i = 0; i < DIMS; i++) vector[i] = rand() * 2 - 1; + return vector; +} + +function percentile(samples, fraction) { + if (samples.length === 0) return 0; + const sorted = [...samples].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * fraction))]; +} + +describe('HNSW file-primary ingest cost', function () { + if (process.env.HARPER_STORAGE_ENGINE === 'lmdb') return; + if (!getPlaneBinding()) { + it.skip('skipped: @harperfast/hnsw native module is unavailable', () => {}); + return; + } + this.timeout(20 * 60_000); + let Bench; + const meter = { applyCount: 0, applyMs: 0, flushCount: 0, flushMs: 0 }; + + function customIndex() { + return Bench.indices.vector.customIndex; + } + + function instrument() { + const index = customIndex(); + const apply = index.applyDerivedValue.bind(index); + const flush = index.flushDerived.bind(index); + index.applyDerivedValue = (key, vector, version) => { + const started = process.hrtime.bigint(); + try { + return apply(key, vector, version); + } finally { + meter.applyMs += Number(process.hrtime.bigint() - started) / 1e6; + meter.applyCount++; + } + }; + // flushDerived awaits the native barrier, so this spans an await and charges anything the + // loop ran meanwhile to the backend. It is an upper bound on barrier cost, and the reason + // the reported backend share is printed as a range with applyMs — which is synchronous and + // exact — as its floor. + index.flushDerived = async (watermark) => { + const started = process.hrtime.bigint(); + try { + return await flush(watermark); + } finally { + meter.flushMs += Number(process.hrtime.bigint() - started) / 1e6; + meter.flushCount++; + } + }; + } + + function resetMeter() { + meter.applyCount = 0; + meter.applyMs = 0; + meter.flushCount = 0; + meter.flushMs = 0; + } + + // Scanning the audit log to find the tail costs more than the drain being measured, so callers + // read the tail before starting a clock and pass it to drained(), which only polls cursors. + function auditTails() { + const tails = []; + const logs = Bench.auditStore.loadLogs(); + for (let nodeId = 0; nodeId < logs.length; nodeId++) { + if (!logs[nodeId]) continue; + let latest; + for (const entry of Bench.auditStore.getRange({ start: 0, log: nodeId })) latest = entry.txnLogKey; + if (latest !== undefined) tails.push([nodeId, latest]); + } + return tails; + } + + const POLL_MS = 1; + + async function drained(tails = auditTails()) { + return waitFor( + () => + tails.every( + ([nodeId, latest]) => + Bench.indices.vector.getSync(derivedIndexCursorKey(Bench.indices.vector.name, nodeId)) === latest + ), + { timeout: 15 * 60_000, interval: POLL_MS, message: 'derived index did not drain' } + ); + } + + before(async () => { + setupTestDBPath(); + setMainIsWorker(true); + Bench = table({ + table: 'IngestBench', + database: DB, + audit: true, + attributes: [ + { name: 'id', isPrimaryKey: true }, + { + name: 'vector', + indexed: { type: 'HNSW', nativePlane: true, efConstruction: 200 }, + type: 'Array', + }, + ], + }); + await Bench.indexingOperation; + await Bench.put(0, { vector: makeVector() }); + await drained(); + instrument(); + // Seed a graph large enough that insert cost reflects a populated index rather than + // the near-empty one a fresh table would measure. + for (let id = 1; id <= SEED; id++) await Bench.put(id, { vector: makeVector() }); + await drained(); + }); + + it('burst ingest: foreground cost, drain cost, and event-loop occupancy', async () => { + resetMeter(); + const loop = monitorEventLoopDelay({ resolution: 1 }); + const putSamples = []; + loop.enable(); + const startedWrites = process.hrtime.bigint(); + for (let id = SEED + 1; id <= SEED + BURST; id++) { + const started = process.hrtime.bigint(); + await Bench.put(id, { vector: makeVector() }); + putSamples.push(Number(process.hrtime.bigint() - started) / 1e6); + } + const writeMs = Number(process.hrtime.bigint() - startedWrites) / 1e6; + loop.disable(); + const writeLoopMax = loop.max / 1e6; + const writeLoopP99 = loop.percentile(99) / 1e6; + // Read the tail before the drain clock starts: scanning the retained log costs more than + // the drain, and inside the window it would be charged to the runtime. + const tails = auditTails(); + loop.reset(); + loop.enable(); + const startedDrain = process.hrtime.bigint(); + await drained(tails); + const drainMs = Number(process.hrtime.bigint() - startedDrain) / 1e6; + const totalMs = writeMs + drainMs; + loop.disable(); + + console.log(`\n== burst ${BURST} puts, dims=${DIMS}, graph≈${SEED + BURST} ==`); + console.log( + ` foreground put: ${(writeMs / BURST).toFixed(3)} ms/put (${Math.round(BURST / (writeMs / 1000))}/s)` + ); + console.log( + ` put p50/p99: ${percentile(putSamples, 0.5).toFixed(3)} / ${percentile(putSamples, 0.99).toFixed(3)} ms` + ); + console.log(` write+drain wall: ${totalMs.toFixed(0)} ms (${Math.round(BURST / (totalMs / 1000))} indexed/s)`); + console.log( + ` applyDerivedValue: ${meter.applyCount} calls, ${meter.applyMs.toFixed(0)} ms total, ${(meter.applyMs / Math.max(1, meter.applyCount)).toFixed(3)} ms/call` + ); + console.log( + ` flushDerived: ${meter.flushCount} calls, ${meter.flushMs.toFixed(0)} ms total, ${(meter.flushMs / Math.max(1, meter.flushCount)).toFixed(3)} ms/call` + ); + console.log( + ` flush share of drain:${((meter.flushMs / Math.max(1, meter.applyMs + meter.flushMs)) * 100).toFixed(1)} %` + ); + console.log( + ` backend share of wall:${((meter.applyMs / Math.max(1, totalMs)) * 100).toFixed(1)}–${(((meter.applyMs + meter.flushMs) / Math.max(1, totalMs)) * 100).toFixed(1)} % of ${totalMs.toFixed(0)} ms (floor = synchronous apply ${meter.applyMs.toFixed(0)} ms; ceiling adds the barrier's ${meter.flushMs.toFixed(0)} ms, timed across an await)` + ); + console.log(` loop max/p99 writing:${writeLoopMax.toFixed(1)} / ${writeLoopP99.toFixed(1)} ms`); + console.log( + ` loop max/p99 draining:${(loop.max / 1e6).toFixed(1)} / ${(loop.percentile(99) / 1e6).toFixed(1)} ms` + ); + assert.ok(meter.applyCount >= BURST); + }); + + it('serialized writes: freshness cost of one isolated write', async () => { + resetMeter(); + const base = SEED + BURST + 1; + for (let n = 0; n < TRICKLE; n++) { + await Bench.put(base + n, { vector: makeVector() }); + await drained(); + } + + // The wall clock here would carry a retained-log scan and a poll interval per record, both + // larger than the work; the meter is what this case is for. + console.log(`\n== serialized ${TRICKLE} puts, full drain between each ==`); + console.log(` indexing work per record: ${((meter.applyMs + meter.flushMs) / TRICKLE).toFixed(2)} ms`); + console.log( + ` applyDerivedValue: ${meter.applyCount} calls, ${(meter.applyMs / Math.max(1, meter.applyCount)).toFixed(3)} ms/call` + ); + console.log( + ` flushDerived: ${meter.flushCount} calls, ${(meter.flushMs / Math.max(1, meter.flushCount)).toFixed(3)} ms/call` + ); + console.log( + ` flush share: ${((meter.flushMs / Math.max(1, meter.applyMs + meter.flushMs)) * 100).toFixed(1)} %` + ); + console.log(` flushes per record: ${(meter.flushCount / TRICKLE).toFixed(2)}`); + }); + + it('repeated keys: upper bound on what per-key coalescing could remove', async () => { + resetMeter(); + const first = SEED + BURST + TRICKLE + 10; + for (let round = 0; round < HOT_ROUNDS; round++) { + for (let k = 0; k < HOT_KEYS; k++) await Bench.put(first + k, { vector: makeVector() }); + } + await drained(); + const writes = HOT_KEYS * HOT_ROUNDS; + console.log(`\n== ${HOT_KEYS} keys × ${HOT_ROUNDS} rounds = ${writes} commits ==`); + console.log(` applyDerivedValue: ${meter.applyCount} calls (${HOT_KEYS} distinct keys)`); + console.log(` time in apply: ${meter.applyMs.toFixed(0)} ms`); + console.log( + ` repeated keys: ${(((meter.applyCount - HOT_KEYS) / Math.max(1, meter.applyCount)) * 100).toFixed(1)} % of apply calls` + ); + console.log(` flushDerived: ${meter.flushCount} calls, ${meter.flushMs.toFixed(0)} ms total`); + }); +}); diff --git a/unitTests/resources/vectorIndexPlane.test.js b/unitTests/resources/vectorIndexPlane.test.js index 726f47713c..cf14f42fdb 100644 --- a/unitTests/resources/vectorIndexPlane.test.js +++ b/unitTests/resources/vectorIndexPlane.test.js @@ -268,6 +268,69 @@ describe('HNSW native plane file-primary delivery', function () { assert.ok((await readySearch(target)).length > 0); }); + it('rejects a wrong-dimension write and query as client errors, not index failures', async () => { + const wrong = makeVector(0).concat(1); + await assert.rejects( + async () => PlaneTest.put(90_001, { name: 'wrong-dims', vector: wrong }), + (error) => { + assert.match(error.message, /components, but this index stores/); + assert.equal(error.statusCode, 400, 'a wrong-length vector is the caller mistake, not a 503'); + return true; + } + ); + // The native insert would throw an unclassifiable error instead, which reconstruction + // rethrows — one such record would abort every rebuild and strand the index at 503. + assert.equal(await PlaneTest.get(90_001), undefined, 'the rejected write must not commit'); + await assert.rejects( + async () => nativeSearch(wrong), + (error) => { + assert.match(error.message, /Search target has/); + assert.equal(error.statusCode, 400); + return true; + } + ); + // A BigInt component would throw out of Float32Array.from, which the caller reads as plane + // corruption and answers by unlinking a healthy file. + const bigintTarget = vectors.get(3).slice(); + bigintTarget[0] = 1n; + await assert.rejects(async () => nativeSearch(bigintTarget), /not a finite 32-bit float/); + assert.ok(fs.existsSync(customIndex().planeFilePath()), 'a malformed query must not unlink the plane'); + assert.ok((await readySearch(vectors.get(3))).length > 0, 'the index stays healthy'); + }); + + it('rejects a component outside the f32 range instead of stranding reconstruction', async () => { + const overflowing = makeVector(0).slice(); + overflowing[0] = 1e39; // finite as a double, Infinity as the f32 the plane stores + await assert.rejects( + async () => PlaneTest.put(90_002, { name: 'f32-overflow', vector: overflowing }), + (error) => { + assert.match(error.message, /not a finite 32-bit float/); + assert.equal(error.statusCode, 400); + return true; + } + ); + assert.equal(await PlaneTest.get(90_002), undefined, 'the rejected write must not commit'); + + // f32-finite components whose squares are not: invMag would store 0 and every distance + // involving the node would be NaN. + const huge = makeVector(0).map(() => 1e20); + await assert.rejects( + async () => PlaneTest.put(90_003, { name: 'f32-magnitude', vector: huge }), + /magnitude too large/ + ); + + // Math.fround throws a TypeError on a BigInt, which is not a ClientError and would strand + // reconstruction exactly as the native throw did. + const bigint = makeVector(0).slice(); + bigint[0] = 1n; + await assert.rejects( + async () => PlaneTest.put(90_004, { name: 'f32-bigint', vector: bigint }), + /not a finite 32-bit float/ + ); + + assert.ok((await readySearch(vectors.get(3))).length > 0, 'the index stays healthy'); + }); + it('rejects synchronous iteration of asynchronous plane-backed results', () => { const results = PlaneTest.search({ sort: { attribute: 'vector', target: vectors.get(42), distance: 'cosine' }, @@ -548,10 +611,13 @@ describe('HNSW native plane file-primary delivery', function () { }), /requires M=16/ ); - assert.throws( - () => customIndex().prepareCommitted('empty-vector', [], undefined, { transaction: {} }), - /must contain at least one component/ - ); + for (const bad of [[], { length: -1 }, { length: 1.5 }, {}]) { + assert.throws( + () => customIndex().prepareCommitted('bad-vector', bad, undefined, { transaction: {} }), + /must be an array of at least one number/, + `${JSON.stringify(bad)} must not reach the plane` + ); + } // An already-audited table must not be able to turn auditing off underneath a nativePlane // index: nothing clears Table.audit, so the descriptor would persist audit: false and the // next process start would fail catalog load on the derived-index attach.