Native HNSW index: file-primary mmap graph over a shared post-commit derived-index runtime (#2489)

.github/workflows/unit-test.yml

@@ -133,3 +133,47 @@ jobs:
       - name: Run unit tests
         timeout-minutes: 30
         run: npm run test:unit:windows
+
+  # The vectorIndexPlane suite self-skips when the optional native package is absent, so the
+  # load probe below is what keeps that skip from reading as a green run. The crate's own
+  # tests belong to HarperFast/hnsw's CI, not here: this job runs the published prebuild, so
+  # testing the crate source would grade something other than the binary under test.
+  hnsw-plane:
+    name: HNSW native plane (Node.js v24)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
+
+      - name: Setup Node.js 24
+        uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0
+        with:
+          node-version: 24
+          package-manager-cache: false
+
+      - name: Install dependencies
+        run: npm install --ignore-scripts
+
+      # strict here, unlike the other jobs: the plane suite runs against dist/, so a tolerated
+      # build failure would grade a stale dist and still report green
+      - name: Build
+        run: npm run build
+
+      - name: Verify the published native binding loads
+        run: node -e "if (typeof require('@harperfast/hnsw').Plane?.open !== 'function') throw new Error('@harperfast/hnsw loaded without a Plane constructor');"
+
+      - name: Setup Harper
+        env:
+          DEFAULTS_MODE: 'dev'
+          HDB_ADMIN_USERNAME: 'admin'
+          HDB_ADMIN_PASSWORD: 'password'
+          ROOTPATH: '/tmp/hdb'
+          NODE_HOSTNAME: 'localhost'
+          LOGGING_LEVEL: 'info'
+        run: node --enable-source-maps ./dist/bin/harper.js install
+
+      - name: File-primary native plane tests
+        env:
+          HNSW_NATIVE_REBUILD_BENCHMARK: '1'
+        run: npx mocha unitTests/resources/vectorIndexPlane.test.js

dependencies.md

@@ -243,3 +243,15 @@ This is the inverse of the entries below — a dependency we take deliberate ste
 - Security: Microsoft-maintained TypeScript compiler, same publisher/package as the 5.x devDependency.
 - Overlap: Complements, does not replace, the `typescript` devDependency — TypeScript 7.0 ships no compiler API yet (planned for 7.1), so `@typescript-eslint/parser` still needs `typescript` 5.x. The 5.x and 7.x versions never coexist in `node_modules` at once: 5.x is the installed devDependency, 7.x is fetched on-demand by `npx` purely for `typecheck:fast`.
 - Eventual removal: Once TypeScript 7 stabilizes as the primary `typescript` devDependency (post-7.1's compiler API), this becomes redundant and `typecheck:fast` can be dropped.
+
+## @harperfast/hnsw (optional dependency)
+
+- Need for usage: Supplies the file-primary memory-mapped HNSW index used only by indexes that opt in with `nativePlane: true`. Harper keeps primary-key mappings and replay cursors in RocksDB; graph nodes and adjacency exist only in the native file.
+- Size/memory cost: The JS/package metadata is about 250 KB unpacked plus one platform-specific native binary. Runtime mapped-file size is approximately 1,344 bytes per 768-dimensional int8 node at layer-0 cap 128; mappings are shared by the OS page cache across workers.
+- Security: First-party Apache-2.0 Harper package. It runs native code in-process, so Harper exact-pins the package and its own manifest exact-pins every platform prebuild to the same version. The dedicated CI job loads the registry prebuild and tests Harper against it.
+- Environment interaction: Lazily loaded only when an eligible index enables `nativePlane`; it creates a memory-mapped `.hnsw` derived-index file next to the index store and may create a `.stale` invalidation sidecar. It does not modify globals or install polyfills.
+- Overlap: None for an opted-in index. Ordinary HNSW indexes continue to use the JS/RocksDB graph; `nativePlane` indexes use native insert, mutation, and search through the shared post-commit derived-index runtime.
+- Transitive dependencies: Only exact-version, platform-specific optional prebuild packages; no JS runtime dependency tree.
+- Binary compilation: Supported Linux glibc x64/arm64, macOS arm64, and Windows x64 targets use prebuilds. Other targets attempt a Rust source build. Because the root package is optional Harper still installs if that build fails, but opted-in indexes remain unavailable until the module is present.
+- Can be deferred: Yes for ordinary HNSW indexes. The adapter loads lazily; an opted-in index returns 503 and retries rebuild when the native module is unavailable.
+- Eventual removal: Disable `nativePlane`, allow the schema reindex to rebuild the ordinary JS/CF graph, then remove the optional dependency and adapter integration.