Meta: note indexing & index-query backlog — priority order and burn-down

[samkeen]

One place for everything still open that touches Flow ① (ingest / chunk / embed) and Flows ② / ③ (search / similar) — 11 of the 25 open issues. Goal: work these in priority order and drive the open-issue count down.

The ordering principle: most of this backlog is eval-gated ("measure on the hand-labelled retrieval eval, never on intuition"), so the two issues that strengthen the eval itself came first — every tier-3 experiment is only as trustworthy as the gate it's measured against. Both have now landed (§1), so the gate is sharp and the eval-gated items below are unblocked.

1. ✅ Landed — the gate is sharpened

• ✅ Discovery quality floor: enforce "limit is a cap, not a promise" in discover::candidates #150 — Discovery quality floor in discover::candidates. Closed; the ruling in index-engine.md §3 is now enforced.
• ✅ qmd chunker: validate retrieval quality vs. the old paragraph chunker (the §7 gate) #44 — qmd chunker validated against the old paragraph chunker (the §7 gate). Closed. This was the prerequisite for every eval-gated item below (Search fusion tuning: RRF weighting + stopword filter for short queries #105, Tree-sitter / AST-aware code chunking #104, Ranking: cross-encoder reranker over the fused top-N (+ query expansion seam) #28, Embedder spike: faster/smaller model (bge-small / quantized / ONNX) behind the seam #17, Vector-scan scaling levers (conditional on scale) #106) — the eval's query set grew with it, so the gate can no longer lie to you the way the original note warned.

Also closed since this issue was filed: #16 (CLI cross-process background reindex + b2 status + Ctrl-C cancel), #43 (model-relative chunk size — closed with the trigger recorded), and #97 (post-v1 large-scale perf — absorbed into #106 as suggested here, for the free −1).

2. Cheap, model-free wins (indexing ops & UX)

• Search fusion tuning: RRF weighting + stopword filter for short queries #105 — RRF weighting + stopword filter for short queries. Ordering-only change; now measurable, since qmd chunker: validate retrieval quality vs. the old paragraph chunker (the §7 gate) #44 has made the eval sensitive.
• Embedding: embed the currently-open note and its neighbors first, not strict path order #27 — Embed the currently-open note and its 1-hop neighbors first. Pure ordering change within the embed pass; converges to identical vectors, so low-risk. Implementation sketch in the issue, with two post-#170 corrections (the anchor is a path; the pending set is hash-derived, so an anchor can contribute no pending rows).
• Links: dst_alias_raw + cosmetic alias-refresh on title rename #29 — edges.dst_alias_raw + cosmetic alias-refresh on title rename. Small, display-only; not a correctness issue. Note the standing cost caveat on it: a schema bump drops the vector tables, so this wants batching behind a change that re-embeds anyway, or an additive migration path.

3. Experiments & spikes — enter only with a measured (or dogfooded) reason

• Experiment: should b2 similar also weight graph distance, not just semantic similarity? #20 — Should b2 similar weight graph distance? Dogfood-judged, not eval-scorable. Explicitly not the fix for the discovery floor (it reorders candidates; it can't make a bad list shorter).
• Embedder spike: faster/smaller model (bge-small / quantized / ONNX) behind the seam #17 — Embedder spike: bge-small / quantized / ONNX. Raw-speed lever; eval-gate before changing the default. Also the issue that would introduce a >512-token-window model, which is Chunker: model-relative chunk size (per-model context window) — deferred Embedder::max_tokens() alternative #43's recorded trigger.
• Ranking: cross-encoder reranker over the fused top-N (+ query expansion seam) #28 — Cross-encoder reranker over the fused top-N (search only, never similar). Gate on a measured RRF precision@k gap first; value grows with vault size and with agent consumers (the serve adapter, serve HTTP adapter — a third dumb adapter over the façade #24).

4. Parked with explicit triggers

• Tree-sitter / AST-aware code chunking #104 — Tree-sitter / AST-aware code chunking. Low priority; the prose-mode guard (Chunker: guard against splitting inside a fenced code block / table #41) already shipped.
• Vector-scan scaling levers (conditional on scale) #106 — Vector-scan scaling levers (quantization → partition → materialized table → LanceDB, in that order; ANN rejected), now also carrying Post-v1: large-scale performance beyond personal-vault scale #97's open-ended "beyond personal scale" direction. Trigger: the whole-space scan stops being interactive on a real vault. Lever 2 needs re-speccing before it can be costed — see §6.

5. Index expansion: resources become searchable (roadmap order from #66)

• Searchable resources (text extraction, FTS/vectors, resource centroids) #108 — Searchable resources (text extraction → FTS/vectors through the existing bge space + resource_centroids). Depends on Render mechanisms (asset protocol, inline images, viewers) #107 (render mechanisms) landing first per the roadmap. Its locked DDL needs a post-#170 re-anchor — and content-addressing hands the slice a small win.
• PDF text extraction #109 — PDF text extraction (feeds Searchable resources (text extraction, FTS/vectors, resource centroids) #108; the extraction-crate decision lands there by design).
• Semantic seams (describer + multimodal embedder) #110 — Semantic seams (describer + multimodal embedder). Future, unscheduled — fine to leave parked at the bottom.

6. Post-#170 re-spec debts inside this backlog

GH #170 (path-keyed note identity + content-addressed vectors) landed after most of these were written. Three carry design text that the pivot invalidated — none is a re-prioritization, all are "read this before costing the work":

• Vector-scan scaling levers (conditional on scale) #106 lever 2 (partition / filtered-KNN) — embeddings is now keyed by text_hash with no note or path column, and a vector may be shared across notes, so there is no owner to partition on. The lever now means partitioning the join through chunks, a different design with a different cost profile. Content-addressing also pre-paid part of lever 1 (quantization), by however much duplicate chunk text the vault holds. Detail.
• Searchable resources (text extraction, FTS/vectors, resource centroids) #108's locked chunk DDL — note_b2id is note_path; both sides of the one-of are now the same kind of key, which is strictly less work. Resource chunks also join the content-addressed store with no new column. Detail.
• Embedding: embed the currently-open note and its neighbors first, not strict path order #27's anchor parameter — a path, and the hash-derived pending set changes what "the anchor's group" contains. Detail.

Outside this backlog's scope but part of the same sweep: #103 (the b2:// URI is now the rejected alternative in data-model.md §5) and #166 (highlights — note_b2id and the retired IdGen seam are both live design questions again).

Related, deliberately not tracked here

#107 (UI prerequisite of #108, not itself index work), #102 (resource-sourced edges — data-model/authoring surface, not indexing or query), and the LLM MVP chain #154/#155 (they consume the index; #151/#153 are closed).