Durable path-level plane invalidation (invalidatePlane, format v7 latch) and the harper#2430 native delta

.github/workflows/ci.yml

@@ -8,7 +8,7 @@ jobs:
   test:
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
+        os: [ubuntu-latest, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     timeout-minutes: 20
     steps:

DESIGN.md

@@ -96,18 +96,24 @@ One file per index (per slice, once C2 lands): `<index-path>.hnsw`.
 | freelist_head                     | u64 atomic | CAS push/pop; ABA-guarded with a 32-bit tag                  |
 | txn_watermark                     | u64        | last durably indexed transaction; advanced by msync cadence  |
 | clean_shutdown flag               | u8         | torn-state detection on open                                 |
+| invalidated latch                 | u8         | one-way (v7): watermark reads 0 on every handle, open refuses |
+| write_epoch                       | u64 atomic | bumped by every node write; re-arms the read-side repair probe |
 
 **Main region — layer-0 slots**, addressed `4096 + id × slot_size`:
 
-| Field                           | Size (768-d int8, cap 64)          |
-| ------------------------------- | ---------------------------------- |
-| seq (seqlock)                   | 4 B                                |
-| flags (valid/deleted) + level   | 2 B                                |
-| scale (f32) + invMag (f32)      | 8 B                                |
-| degree                          | 2 B                                |
-| vector (int8 × 768)             | 768 B                              |
-| neighbor ids (u32 × layer0_cap) | 256 B                              |
-| **total, padded**               | **1,040 B → 1 KB-aligned 1,088 B** |
+| Field                           | Size (768-d int8, cap 64)           |
+| ------------------------------- | ----------------------------------- |
+| seq (seqlock)                   | 4 B                                 |
+| flags (valid/deleted) + level   | 2 B                                 |
+| scale (f32) + invMag (f32)      | 8 B                                 |
+| degree                          | 2 B                                 |
+| vector (int8 × 768)             | 768 B (padded to a 4-byte boundary) |
+| neighbor ids (u32 × layer0_cap) | 256 B                               |
+| **total, padded**               | **1,040 B → 1 KB-aligned 1,088 B**  |
+
+The vector's trailing pad keeps the neighbor array 4-aligned for every `dims`, so the search
+hot path reads each neighbor id as one aligned volatile `u32`. Upper-layer id lists are padded
+the same way (`degree u16 + pad u16 + ids`).
 
 At 100M nodes: ~109 GB (int8). A binary-code v2 slot (96 B codes + ids) is ~384 B → ~38 GB.
 For comparison, today's encoding averages 1,425 B/node _plus_ RocksDB overhead — so v1 is
@@ -167,6 +173,24 @@ bounded-lag durability with deterministic catch-up. For an approximate index who
 truth (records + pk→nodeId) remains fully transactional, bounded lag is the right trade — it
 buys the entire performance model.
 
+**Invalidation (a plane the host cannot delete).** Disabling a plane deletes its file; when the
+unlink fails (Windows sharing violation while another process maps it) the file must not be
+adopted later at its nonzero watermark, or it silently serves searches missing every mutation
+made while mirroring was off. `invalidate_plane(path)` / `invalidate_file(&handle)` leave two
+markers, always attempting both: in band — `PlaneFile::invalidate` sets a one-way header latch,
+zeroes the watermark, and msyncs the header page alone (a whole-mapping flush cannot run inline
+on a multi-GB plane, and lowering the watermark is the safe direction) — then a `<path>.stale`
+sidecar, created with create-new semantics (a planted symlink is never followed) and fsync'd
+together with its directory entry (the directory fsync is skipped on Windows, where `std` has no
+directory handle and `FlushFileBuffers` on the marker covers its creation). The package enforces
+both markers: `open` refuses a file carrying either, `create` refuses a path with a leftover
+sidecar, and `watermark()` reads 0 on every handle while the latch is set — so a flush already
+in flight on another handle, which still stamps the word, cannot revive the plane. In band
+first: the sidecar is what a process that cannot map the file checks, the latch is what covers a
+plane whose sidecar a crash lost. A temporary handle opened for the in-band mark is unmapped
+and closed before the sidecar step — its own mapping would keep the file undeletable — and the
+call fails only when neither marker is durable, leaving the file exactly as found.
+
 **Backup/copy-db/reseed:** the file is node-local derived state. Backup either includes it
 (consistent-enough after an msync barrier) or marks the index rebuild-on-restore. Replica
 reseed = rebuild from records (C5 bulk construction makes this fast; until then, the existing
@@ -272,6 +296,14 @@ Decided (Kris, 2026-08-31):
 
 Open:
 
+- **Atomic slot payloads.** Fields a concurrent reader acts on (flags, level, degree, scale,
+  invMag, neighbor and upper ids) are read through aligned `read_volatile`, which forbids the
+  reload/split/sink across the seqlock's validating fence that `lto = true, codegen-units = 1`
+  otherwise licenses. That is not the same as being race-free under Rust's memory model: only
+  making those fields `AtomicU8`/`AtomicU16`/`AtomicU32` in the slot layout would be, and that
+  is a format change deferred past phase 1. The stored vector stays an ordinary load on
+  purpose — `cosine_int8_raw` must keep autovectorizing, and a torn vector only perturbs a
+  distance the generation check discards.
 - **msync cadence default** — bounded-lag durability window vs write amplification; needs a
   workload measurement, not a guess.
 - **f32 (quantization:"none") slot variant** — 3,072 B vectors → 3.4 KB slots; supported by the

README.md

@@ -68,6 +68,13 @@ const predicated = await plane.searchWithPredicate(queryVector, 10, 512, (ids) =
 );
 ```
 
+A plane is derived state; when the host must stop maintaining one and cannot delete the file
+(Windows sharing violations while another process maps it), `invalidatePlane(path)` — or
+`plane.invalidateFile()` through a handle the host already holds — durably marks it
+unadoptable: a one-way in-band latch (watermark reads 0, `Plane.open` refuses) plus a fsync'd
+`<path>.stale` sidecar (`stalePathFor(path)`, which `open` also refuses). It throws only when
+neither marker lands. Hosts delete both files and rebuild.
+
 Full API in [index.d.ts](index.d.ts).
 
 ## Benchmarks
@@ -81,8 +88,8 @@ equal recall.
 
 ## Status
 
-Extracted from the Harper vector-index engine; the format (v2) and API are young and may
-change with a major version + reindex. Roadmap: prebuilds, binary-quantized slot format
+Extracted from the Harper vector-index engine; the format (v7) and API are young and may
+change with a version bump + reindex (an older format version fails to open; rebuild). Roadmap: prebuilds, binary-quantized slot format
 (~4× smaller traversal plane), Matryoshka dimension truncation, mremap growth, index
 slicing with native top-k merge.
 

index.d.ts

@@ -12,7 +12,10 @@ export interface SearchHit {
 export declare class Plane {
 	/** Create a new plane file. `maxNodes` is a sparse reservation — pages materialize on write. */
 	static create(path: string, dims: number, layer0Cap: number, maxNodes: number): Plane;
-	/** Open an existing plane file (format-version mismatch throws: rebuild the index). */
+	/**
+	 * Open an existing plane file. Throws on a format-version mismatch and on an invalidated
+	 * plane (header latch or `.stale` sidecar): delete the file and its sidecar, rebuild.
+	 */
 	static open(path: string): Plane;
 
 	/**
@@ -101,4 +104,42 @@ export declare class Plane {
 	flush(watermark?: number): void;
 	/** flush() on the libuv thread pool — a whole-map msync can stall its calling thread. */
 	flushAsync(watermark?: number): Promise<void>;
+	/**
+	 * In-band half of invalidateFile() only — no sidecar, so a process that cannot map the
+	 * file sees nothing; prefer invalidateFile(). Sets the one-way header latch, zeroes the
+	 * watermark, msyncs the header page (a 4 KB barrier, not a whole-mapping flush). From then
+	 * on every handle reads watermark 0, whatever a racing flush stamps, and open() throws.
+	 */
+	invalidate(): void;
+	/**
+	 * invalidatePlane() through this handle: the in-band mark via this mapping (no second open,
+	 * no second registry slot — on Windows this mapping is why the unlink failed) and the
+	 * `.stale` sidecar next to the path it opened. The path must not have been replaced since.
+	 */
+	invalidateFile(): InvalidationOutcome;
+	/** Whether the plane was invalidated, by any handle, since this one opened. */
+	invalidated(): boolean;
 }
+
+export interface InvalidationOutcome {
+	/** The watermark was zeroed and its header page msync'd. */
+	inBand: boolean;
+	/** `<path>.stale` exists and is fsync'd (on POSIX, so is its directory entry). */
+	sidecar: boolean;
+	inBandError?: string;
+	sidecarError?: string;
+}
+
+/**
+ * Make a plane file that could not be deleted unadoptable, durably, through a temporary
+ * handle that is unmapped and closed before this returns. Both markers are always attempted:
+ * the in-band latch and the fsync'd `.stale` sidecar; open() refuses a file carrying either.
+ * Throws only when neither marker became durable; nothing is deleted or renamed, and an
+ * in-band mark whose msync failed may still have landed in the shared mapping (the safe
+ * direction: it reads as incomplete). Idempotent. Synchronous (three small fsyncs on a cold path).
+ */
+export declare function invalidatePlane(path: string): InvalidationOutcome;
+/** invalidatePlane() on the libuv thread pool. */
+export declare function invalidatePlaneAsync(path: string): Promise<InvalidationOutcome>;
+/** The sidecar convention: `<path>.stale`. */
+export declare function stalePathFor(path: string): string;

package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/hnsw",
-	"version": "0.1.0",
+	"version": "0.2.0",
 	"description": "Persistent, incrementally-maintained, concurrently-searchable native HNSW for Node.js: a memory-mapped fixed-slot graph file with off-event-loop search, seqlock concurrency, int8 asymmetric distance, and bitset/predicate filtering.",
 	"license": "Apache-2.0",
 	"repository": {
@@ -41,9 +41,9 @@
 		"rust"
 	],
 	"optionalDependencies": {
-		"@harperfast/hnsw-darwin-arm64": "0.1.0",
-		"@harperfast/hnsw-linux-arm64-glibc": "0.1.0",
-		"@harperfast/hnsw-linux-x64-glibc": "0.1.0",
-		"@harperfast/hnsw-win32-x64": "0.1.0"
+		"@harperfast/hnsw-darwin-arm64": "0.2.0",
+		"@harperfast/hnsw-linux-arm64-glibc": "0.2.0",
+		"@harperfast/hnsw-linux-x64-glibc": "0.2.0",
+		"@harperfast/hnsw-win32-x64": "0.2.0"
 	}
 }

smoke.mjs

@@ -1,7 +1,7 @@
 // End-to-end smoke test: `npm run build && node smoke.mjs` (also the CI path).
 import { createRequire } from 'module';
 const require = createRequire(import.meta.url);
-const { Plane } = require('./index.js');
+const { Plane, invalidatePlane, invalidatePlaneAsync, stalePathFor } = require('./index.js');
 
 const dims = 64;
 const { tmpdir } = await import('node:os');
@@ -81,4 +81,43 @@ plane.flush();
 const reopened = Plane.open(path);
 const hits2 = reopened.searchSync(vec(42), 5, 128);
 if (hits2[0].distance > 1e-3) throw new Error('reopened self-query failed');
-console.log('reopen + sidecar OK. smoke PASSED');
+console.log('reopen OK');
+
+// invalidation through the caller's own handle: both markers land, the latch survives a
+// later flush, and every later open is refused
+const { existsSync, mkdirSync, rmSync } = await import('node:fs');
+reopened.setWatermark(4096);
+reopened.flush();
+const outcome = reopened.invalidateFile();
+if (!outcome.inBand || !outcome.sidecar) throw new Error(`invalidation incomplete: ${JSON.stringify(outcome)}`);
+if (stalePathFor(path) !== `${path}.stale` || !existsSync(stalePathFor(path))) throw new Error('no .stale sidecar');
+reopened.flush(900);
+if (reopened.getWatermark() !== 0 || !reopened.invalidated()) throw new Error('a later flush revived the plane');
+rmSync(stalePathFor(path));
+let refused;
+try {
+	Plane.open(path);
+} catch (error) {
+	refused = error;
+}
+if (!refused || !/invalidated/.test(refused.message)) throw new Error(`open must refuse an invalidated plane, got ${refused}`);
+// by path: a temporary open that must not survive the call (idempotent on a latched plane)
+const byPath = invalidatePlane(path);
+if (!byPath.inBand || !byPath.sidecar) throw new Error(`path invalidation incomplete: ${JSON.stringify(byPath)}`);
+const byPathAsync = await invalidatePlaneAsync(path);
+if (!byPathAsync.inBand || !byPathAsync.sidecar) throw new Error(`async path invalidation incomplete: ${JSON.stringify(byPathAsync)}`);
+rmSync(stalePathFor(path));
+// neither marker possible: not a plane, and a directory squatting the sidecar path
+const bogus = join(tmpdir(), `smoke-bogus-${process.pid}.hnsw`);
+const { writeFileSync } = await import('node:fs');
+writeFileSync(bogus, 'not a plane');
+mkdirSync(stalePathFor(bogus));
+let threw;
+try {
+	invalidatePlane(bogus);
+} catch (error) {
+	threw = error;
+}
+if (!threw || !/in-band:.*sidecar:/.test(threw.message)) throw new Error(`double failure must throw naming both causes, got ${threw}`);
+rmSync(stalePathFor(bogus), { recursive: true });
+console.log('invalidatePlane OK. smoke PASSED');

src/bin/bench.rs

@@ -122,7 +122,7 @@ fn main() {
         let build_start = Instant::now();
         for i in 0..n {
             let v = corpus.row(&mut rng);
-            insert(&graph, &v, &params, &mut scratch);
+            insert(&graph, &v, &params, &mut scratch).expect("build insert");
             if (i + 1) % 50_000 == 0 {
                 let rate = (i + 1) as f64 / build_start.elapsed().as_secs_f64();
                 println!("  built {} ({:.0} inserts/s)", i + 1, rate);