From 2c86c8aab838d8c018a47f6fbeb87c6480147605 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 05:21:54 +0000 Subject: [PATCH 1/6] docs: re-author the three specs into docs/, plain style, anchors kept MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit invariants.md, data-model.md, and index-engine.md move from design/ to docs/ re-authored per the documentation style guide: short sentences, plain words, you-voice, no prose em dashes. Every invariant id and every section number the code cites (data-model §0-§10, index-engine §1-§8) keeps its meaning. index-engine §3/§4/§6 absorb the unique content of the indexing and retrieval HTML deep dives (incremental heuristic, stage-by-stage pipelines, verdict states, constants table, batching and the model swap). design/ itself is removed in a follow-up commit once every reference points here. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CnZh7mztkPnvXJ4GnfrZiE --- docs/data-model.md | 519 +++++++++++++++++++++++++++++++++++++ docs/index-engine.md | 602 +++++++++++++++++++++++++++++++++++++++++++ docs/invariants.md | 328 +++++++++++++++++++++++ 3 files changed, 1449 insertions(+) create mode 100644 docs/data-model.md create mode 100644 docs/index-engine.md create mode 100644 docs/invariants.md diff --git a/docs/data-model.md b/docs/data-model.md new file mode 100644 index 0000000..9f98556 --- /dev/null +++ b/docs/data-model.md @@ -0,0 +1,519 @@ +# Data model + +What a note and a connection are, in plain Markdown, for anyone changing how B2 reads or +writes them. This is the yardstick the engine is measured against: the SQLite schema in +[index-engine.md](index-engine.md) §3 is derived from this model and must satisfy it, never +the reverse. + +Related pages: [invariants.md](invariants.md) is the normative register, cited by id. +[index-engine.md](index-engine.md) is the *how*. The *why* behind each choice is an +[ADR](../ADRs/README.md): why connections live where they do (ADR-0010), why B2 never writes +the body (ADR-0004), why identity is the path (ADR-0003), why there is no suggestion queue +(ADR-0009). + +The model has exactly two source-of-truth objects, both plain Markdown: + +1. **A note.** One `.md` file: YAML frontmatter plus a Markdown body. +2. **A connection (edge).** A directed link from one note to another: a plain link you write + in the body, or a typed relation in frontmatter `b2_relations:` (§0). + +Both are authored by a human. A real vault also holds **resources**: every non-`.md` file. A +resource is a peer vault member, not a third authored object. B2 can read it but cannot author +structure into it, because Markdown is the only format whose bytes B2 may write. So the source +*tier* is the whole vault directory, while the two authored objects stay note plus edge. +Resources are defined in §10; they change nothing in §0 to §9. + +### The two storage tiers + +1. **Markdown: the source of truth for knowledge.** Your notes plus every committed edge, on + your disk, fully usable with no B2. The files stay pristine, and the body is 100% yours + (W2). The short list of on-command writes is W3. Reading your vault writes nothing at all + (W1). +2. **The index (`.b2/b2.sqlite`): a disposable cache.** The search indexes and the keyed + graph. It holds nothing that cannot be rebuilt from the Markdown. + +The rules: two tiers, sharply split (S1); `index = projection of (the vault directory)` (S2, +ADR-0002). Drop `b2.sqlite`, reindex, get an identical index (S3). No durable B2-derived state +lives outside your notes (S4). A resource (§10) contributes only derived rows, so the +guarantee is unchanged. + +### Folders + +Your vault directory carries two kinds of authored material: the Markdown files (content) and +the directory tree itself (structure). A folder, empty or not, is user-authored exactly like a +note, and the filesystem is authoritative for it. B2 proxies the OS rather than modeling +folders: + +- `create_dir` makes missing parents but refuses an occupied target. There is no `mkdir -p` + idempotence: you asked to *create*. +- `move_dir` is one `rename`. `delete_dir` is `remove_dir_all`. +- Each resolves its target against the disk, never the index, so empty folders work everywhere. + +Folders are never projected into the index. They carry nothing to chunk, embed, or link. The +tree listing (`Vault::list_dirs`) is a live filesystem walk (dot-folders skipped, §1), so the +tree matches the vault's managed subtree by construction, in both directions. S4 scopes to +*B2-derived* data; your own structure is vault material, not B2 state. + +--- + +## 0. Where a connection lives + +Ruled by [ADR-0010](../ADRs/0010-typed-graph-two-homes-closed-vocabulary.md) and +[ADR-0004](../ADRs/0004-markdown-is-the-only-surface-b2-writes.md). A connection lives in +exactly one of two homes, by origin. The homes split by *what they can say*, not just by who +writes them: + +| Origin of the edge | Where it lives | Source of truth | +|---|---|---| +| A plain body link | The body: a bare `[[path\|title]]`, a Markdown `[text](path)`, an embed. Ordinary Markdown, always an untyped `references` edge | The body. B2 reads it, never writes it | +| A typed relation (committed by `b2 link`, or written by you or an importer) | Frontmatter `b2_relations:`, as a typed-link string `- " [[path\|title]] — …"` (§2). The only home of a verb plus explanation | Frontmatter, B2's managed metadata zone | + +**`b2 link` writes frontmatter, not the body.** Committing appends one typed-link string to +the source note's `b2_relations:` (Markdown first, index reconciled after). The edge then +materializes as an `origin=frontmatter` edge derived from that Markdown line. Committing is +the projection of an authored line, not a bespoke index write (§3). + +In one line: the body holds the plain links you write (all `references`); frontmatter +`b2_relations:` holds every *typed* relation, verb and explanation; both are authored +Markdown, and the graph is their union (G2). + +Each edge has exactly one home, and B2 never copies between them. So there is nothing to keep +in sync, only a one-way projection to rebuild. A `b2_relations:` entry may deliberately target +a note the body already links: that is the augment flow (§2). The overlap case, the *same* +`(target, type)` in both homes (necessarily `references`, the only type a body link can +carry), is resolved at projection time by frontmatter-wins dedup. The frontmatter row is kept, +because only it can carry an explanation. The redundant body reference is ignored as a +duplicate, never auto-removed from your file. + +--- + +## 1. The note + +A note is one `.md` file whose vault-relative path has no dot-prefixed segment: YAML +frontmatter, then a Markdown body. Any segment counts, so `notes/.templates/daily.md` is no +more a note than `.scratch.md` is. + +```markdown +--- +type: concept # optional, defaults to `note`; OKF entity discriminator +title: "Spaced repetition" # optional, INERT — the title is the filename (L2) +description: "Why expanding intervals beat massed practice." +tags: [learning, memory] +created: 2026-06-20 +updated: 2026-06-29 +aliases: [SRS] # optional Obsidian-native extra titles +b2_relations: # B2's managed zone: typed edges (§2). origin=frontmatter + - "contradicts [[notes/cramming-works|Cramming works]] — short-term recall only" +provenance: # optional; defaults to {by: human} + by: human +--- + +Spaced repetition schedules reviews at expanding intervals… + +It builds on [[concepts/memory|Human memory]] — the forgetting curve is the mechanism. +``` + +The body link is human-authored (`origin=inline`) and untyped: an ordinary `references` edge. +The surrounding prose is just prose. The `b2_relations:` entry is a typed edge +(`origin=frontmatter`). + +### Hidden means hidden + +A dot-prefixed name is outside the managed subtree, whatever it is: a folder (`.git/`, +`.obsidian/`, B2's own `.b2/`), a resource (`.DS_Store`), or a Markdown file (`.scratch.md`). +The convention is the filesystem's, not B2's, so B2 reads it the same way everywhere (S2, +[GH #136](https://github.com/AlteredCraft/B2/issues/136)): + +- The walk skips it before it routes it (`pathspec::is_hidden`, applied above the + note/resource dispatch in `collect_vault_files`). So "hidden" cannot mean one thing for a + PDF and another for a Markdown draft. A dot-prefixed `.md` is never a note: no chunks, + embeddings, graph presence, listing, search, or file-tree appearance. +- The file itself is untouched. Skipping is not deleting (W4). That is the point: keep a + scratch draft out of B2's way without leaving the folder you work in. +- B2 will not author one either. Every authoring destination refuses a dot-prefixed segment, + because creating a member the walk would never see is a silent desync between disk and + index. + +Migration note: rows for a previously indexed dot-prefixed `.md` are pruned on the next +reindex, and inbound links re-dangle (G5). Renaming it back restores it exactly. + +### Frontmatter keys + +**Required: nothing.** A note is a `.md` file; that is the whole entry requirement. Identity +is the vault-relative path (L1, ADR-0003), not a key in the file. There is no frontmatter B2 +needs present, none it will add, and a note with no frontmatter block at all is an ordinary, +fully indexed note. + +Optional keys B2 recognizes: + +- **`type`**: what kind of note this is (`note`, `concept`, `source`, `person`, `daily`, …). + Controlled but extensible; unknown values are tolerated; the OKF entity discriminator (§5). + Defaults to `note`. Its only consumer is display, so nothing keys on its presence and the + new-note template does not seed it (GH #80: the template stamps only what cannot be + reconstructed later). Not `b2`-namespaced on purpose: a courtesy you own, not a key B2 + machines on. +- **`title`**: recognized but inert (L2). A note's title *is* its filename (basename minus + `.md`). The key is parsed and round-tripped losslessly like any other, and never drives + display, link aliases, or search. +- **`description`**: a one-line summary. Feeds the embedding prompt and OKF export. +- **`tags`**: a list of strings. +- **`created` / `updated`**: ISO-8601 date or datetime. `created` is set by B2 at creation + (`b2 add`); `updated` is yours (or another tool's) to maintain. B2 does not stamp it. +- **`aliases`**: Obsidian-native additional titles; treated as alternate link aliases. +- **`provenance`**: optional, opt-in note-level authorship: + `{by: human | agent:, source?, confidence?}`. Absent means `{by: human}`. B2 + neither requires nor manages it. Edges carry no provenance (§4). +- **`b2_relations`**: B2's managed zone for typed edges (§2). A YAML list of typed-link + strings, the only place a relation verb and explanation live. Namespaced, and B2's *only* + key, so it can never collide with your `relations:` key or another tool's. For the same + reason a generic un-namespaced `relations:` is *not* read; it is just another unknown key, + preserved verbatim. B2 appends here on `b2 link` (never the body); you and importers may + write it too. + +**Unknown keys** are preserved verbatim and byte-for-byte on round trip (W5, §6). A `b2id:` +line an older B2 left behind is exactly that: an unknown key, never read, never removed +(ADR-0003). + +--- + +## 2. Authored links and typed relations + +### A bare wikilink is an untyped `references` edge + +A normal `[[path|title]]` anywhere in prose is a connection of type `references`, +`origin=inline`: the untyped graph Obsidian already gives you, typed and materialized. It is +directed (A→B, the literal fact that A's text points at B). That preserves the split between +backlinks and forward links (`b2 neighbors` shows it as *referenced-by* from B's side), is the +information-preserving default, and keeps the explicit symmetric verb (`contradicts`) +meaningful as a deliberate choice. + +### A frontmatter `b2_relations:` entry is a typed edge + +The typed-link syntax is ` [[path|title]] — explanation`, as a quoted string in a +`b2_relations:` list. This is the one and only home of a typed relation. The optional trailing +text after an em dash (or `:`) is the edge's `explanation`. + +```yaml +b2_relations: + - "supports [[concepts/forgetting-curve|Forgetting curve]] — the schedule exploits it" + - "contradicts [[notes/cramming-works|Cramming works]]" +``` + +- Quoted, so `[[`, `|`, and `:` are always YAML-safe. The reader accepts quoted or unquoted. +- An entry that is just a bare `[[path|title]]` (no verb) is accepted and reads as + `references`. +- You and importers may write this block too. B2 appends to it on `b2 link` and never authors + the body. + +### Typing a body link: frontmatter augments the body + +A `b2_relations:` entry may target a note the body already links. It *augments* that +connection: the body keeps its plain, clickable link exactly as written, and the frontmatter +carries the stance. A different verb (`supports [[x]]` over a body `[[x]]`) adds the typed +edge alongside the untyped reference; both are real, separately authored facts. The same verb +(`references [[x]] — why`) collapses into one edge, frontmatter-wins (§0, §3), so the +explanation survives. This is the intended UI gesture: select a body link, choose a verb and +optionally an explanation, and B2 appends one `b2_relations:` entry. The body is never +touched. + +### The relation vocabulary: a stance core plus a tolerated tail + +Small, orthogonal, stable core; expressiveness in the tail (G3, ADR-0010). + +The core (a closed set; the `b2 link` palette, and what queries can rely on): + +| Verb | Stance | Direction | Inverse (display only) | +|---|---|---|---| +| `references` | neutral | directed | referenced-by | +| `supports` | for | directed | supported-by | +| `contradicts` | against | symmetric | contradicts | + +Boundary notes: `references` is both the automatic type of a bare link and the deliberate +"see also". `supports` is a directed "A backs B". `contradicts` is a deliberate symmetric +"these state opposites"; tension has no aggressor, so no direction is recorded. + +Extensibility: the core is stable across versions. Any other verb you write (`elaborates`, +`part-of`, `supersedes`, …) is tolerated and stored verbatim, never dropped. Tooling treats +tail verbs as opaque strings: no inverse label, no special traversal. A tail verb that proves +common can be promoted into the core later (gaining an inverse label). Demotion is just +removal from the palette; stored data is untouched. + +Typing guidance: use a stance verb whenever the notes take a position on each other. +`references` is the honest default when they don't. + +Conventions: lowercase kebab-case, named from the source's perspective (`derived-from`, not +`DerivedFrom`). Edges are directed and stored once (G4): inbound edges are computed by +scanning `dst_path` and labelled with the inverse; the symmetric verb is its own inverse. B2 +never writes a reciprocal link into the target file. That would amplify writes and edit a note +you didn't touch. + +### Edge identity is derived, so the file stays clean + +An authored edge, body or frontmatter, is identified by the tuple +`(src path, dst path, type, occurrence-index)`, all recoverable from the Markdown alone. No +edge id is ever written into the file. A committed edge carries no provenance at all: it is a +pristine authored line, nothing more (§4). + +--- + +## 3. The edge record (derived projection) + +Every edge projects to one record. This is the shape the [index-engine.md](index-engine.md) §3 +`edges` table holds. The Markdown is the source; this is the index. + +| Field | Values | Source | +|---|---|---| +| `id` | derived | edge identity, from `(src, dst, type, occurrence_index)` | +| `src_path` | note path | the authoring note (always a note, G6) | +| `dst_path` | note path, or NULL | resolved from the authored `[[path]]` at parse time | +| `dst_resource_path` | resource path, or NULL | set instead of `dst_path` when the target is a non-`.md` file (§10) | +| `dst_path_raw` | text | the target exactly as authored; what a dangling edge keeps | +| `type` | relation verb (§2) | the `b2_relations:` verb; `references` for every body link | +| `origin` | `inline` \| `frontmatter` | which of the two homes (§0) the edge came from | +| `explanation` | free text, optional | trailing text after `—` or `:` (frontmatter entries only) | +| `caption` | text, optional | a Markdown link's text, or an embed's alt text | + +- **Every edge is authored and active** (G1). `origin` records which home it came from. There + is no lifecycle and no `status` column: an edge exists exactly when it is written in the + Markdown, which is what keeps `index = projection of (Markdown)` exact. +- **`src` and `dst` are vault-relative paths, resolved at parse time.** The authored + `[[path]]` is normalized by the resolver (the wikilink `+ ".md"` ladder) and stored as the + path it named. `dst_path_raw` keeps the text exactly as written. So an edge stores what you + authored, and the graph carries no identifier the vault does not. A B2-performed move is + therefore one transaction: rewrite the inbound `[[path|title]]` text, re-key the moved + note's rows, re-project the inbound sources. That is how L1's "rename keeps every backlink + resolving" holds. A move made outside B2 is a delete plus a create, and its inbound links + project as surfaced dangling edges (G5). +- **The edge set is the union of the two homes, deduped** (G2). A frontmatter entry with a + *different* verb than a body link to the same target is no duplicate; that is the augment + case (§2), and both edges project. If the *same* `(src, dst, type)` is authored in both + homes (necessarily `references`), projection keeps the frontmatter row. Frontmatter wins + because only it can carry an explanation. The file is never auto-edited. +- **A `dst` may be a resource, not a note.** A body embed or link to a non-`.md` file + (`![[photo.png]]`, `[[papers/x.pdf]]`) resolves against the `resources` table and records + `dst_resource_path`. `src` is still a note (§10). +- **A `dst` that resolves to nothing is a surfaced dangling edge, not a dropped one** (G5). A + note is one `.md` file (§1), so a `[[Hermes]]` naming a folder, or a plain typo, matches no + note and no resource. The edge is still projected, with both `dst_path` and + `dst_resource_path` NULL. `b2 neighbors` and `b2 explain` (and the desktop Connections pane) + present these distinctly, as *unresolved* with the authored target, so a mistyped link reads + as broken rather than silently vanishing + ([GH #12](https://github.com/AlteredCraft/B2/issues/12)). Resolving the target turns the + same edge into an ordinary connection on the next reindex. Folder-note resolution + (Obsidian-style `Hermes/Hermes.md`) is a possible later refinement, deliberately out of + scope. + +Why this projection is materialized rather than computed on read, and why that does not make +it a third source of truth (G6), is [index-engine.md](index-engine.md) §3. The standing cost +is its §8. + +--- + +## 4. Committing a connection + +There is no suggestion lifecycle, no review queue, no rejection memory, and no event log +(ADR-0009). A connection becomes real in exactly two ways, both authored in Markdown: + +1. **A body link you write.** A plain `[[path|title]]`, Markdown link, or embed: an untyped + `references` edge (§2). B2 reads it on the next reindex; it never writes the body. +2. **`b2 link [--type ] [--explanation …]`.** B2 appends one typed-link + string to the source note's frontmatter `b2_relations:` (Markdown first, never the body), + then re-projects the note so the edge materializes as `origin=frontmatter`. `--type` + defaults to `references`; the palette is the core vocabulary (§2). B2 writes the target as + a bare `[[path]]` with no `|alias`: the filename is the note's title (L2), so the path + already reads as the title. (You may still write any `|alias` you like in a body link; B2 + reads it and never rewrites it.) + +The GUI adds one more authoring *gesture* over the same model: dragging a Similar card onto a +line of the note being edited types a `[[wikilink]]` at that line's end. That is the untyped, +body kind of link, landing in the editor's buffer exactly as `[[` completion does. B2 still +authors nothing (W1), and you are still the precision gate. + +**No provenance tier.** A committed edge is pristine: no `by`, no `confidence`, no `source`. +Nothing is stapled to the note beyond the ` [[path|title]]` line itself. Provenance is +decision fuel for a review step B2 doesn't have. (The optional note-level `provenance:` +frontmatter stays yours to write, §1, and is separate from edges.) + +--- + +## 5. OKF compatibility + +Build *like* OKF for cheap interop; don't depend on it. Export is a no-op, not a migration. +The model already lines up: + +- **`type`** is the OKF entity discriminator: recognized frontmatter, defaulting to `note` + (§1). +- **Resource URI**: a per-note URI is derivable from its vault-relative path under a + configured base. It is exactly as durable as the path (L1), the same handle Obsidian, a + static-site generator, and a file manager all give you (ADR-0003). +- **`index.md`**: a vault-root manifest listing notes and types, derivable from the + frontmatter, so an OKF consumer has a collection entry point. Generated, never a second + source of truth. + +Net: "export to OKF" is selecting and re-shaping fields that already exist. The export surface +itself (minting URIs, emitting the manifest) is not built; +[GH #103](https://github.com/AlteredCraft/B2/issues/103) tracks it. + +--- + +## 6. Serialization rules + +W5's lossless round trip is what makes the two-tier split safe. It is a property of the +parser/serializer, so it is specified here: + +- Unknown frontmatter keys are preserved verbatim and in order. Body text, whitespace, and + comment tokens are byte-preserved. +- The only bytes B2 ever changes are the specific mechanical edits it is asked to make (W3). + Exactly one of those touches the body: rewriting an inbound `[[oldpath|title]]` to + `[[newpath|title]]` on a move, aliases preserved verbatim. An operation you did not invoke + changes no byte at all (W1). +- Nothing is reformatted, reordered, or normalized in passing. Not YAML quoting style, not + list indentation, not line endings. + +These are the tripwires [index-engine.md](index-engine.md) §8 budgets: this doc defines them, +that one enforces them in the store. + +--- + +## 7. Rejected and deferred alternatives + +The reasoning for each rejection lives in the ADR that made the call; this is the index. + +| Rejected | Why, in one line | Recorded in | +|---|---|---| +| B2 authoring the body (a `## Relations` section) | The body is the rendered/exported document and must stay 100% yours | ADR-0004 | +| Typed-line syntax parsed from body prose | Would make B2 an interpreter of prose *shape*: `- see [[x]] for background` becoming verb `see` (L4) | ADR-0004, ADR-0010 | +| The body as the home for committed edges | Accepted trade: a frontmatter edge isn't guaranteed clickable in vanilla Obsidian, which can't render edge *types* regardless (§0) | ADR-0010 | +| A suggestion review layer / per-pair LLM relator | Notes × candidates model calls, and exactly the model-compensating machinery M1 defers | ADR-0009, ADR-0005 | +| A durable event-log tier (`.b2/log/`) | It would exist to hold a queue and rejection memory B2 doesn't have; anything durable outside the notes weakens S2/S4 | ADR-0002 | +| A stamped machine identity (`b2id:`), removed 2026-08-13 ([GH #170](https://github.com/AlteredCraft/B2/issues/170)) | Its one real buy (re-binding after an out-of-band move) cost an unbidden write, a machine key in every file, a collision subsystem, and a carve-out on S3 | ADR-0003 | +| Content-hash identity | Churns the graph on a typo fix. (Hash-keying the *vector store* is the opposite case and shipped with the same change: derived data, where content-addressing is exactly right, M4) | ADR-0003, ADR-0006 | +| Stored reciprocal links | Inverse edges are derived at query time (G4); writing them back amplifies writes and edits notes you didn't touch | ADR-0010 | +| Per-edge ULIDs in the file | Authored edge identity is derived (§2); explicit ids clutter the note for no gain | ADR-0010 | +| Edge provenance in Markdown | With no review step there is no provenance to keep (§4) | ADR-0009 | + +--- + +## 8. The golden vault (the test fixture's shape) + +The smallest fixture that exercises the whole model: an authored typed edge and a bare +reference. + +`concepts/memory.md` +```markdown +--- +type: concept +title: "Human memory" +created: 2026-06-20 +--- +The brain encodes, stores, and retrieves information… +``` + +`notes/spaced-repetition.md` +```markdown +--- +type: concept +title: "Spaced repetition" +created: 2026-06-20 +b2_relations: + - "supports [[concepts/memory|Human memory]] — applies the forgetting curve" +--- +Spaced repetition exploits the [[concepts/memory|Human memory]] retrieval curve. + +Expanding review intervals exploit the forgetting curve. +``` + +The body holds one plain link (`origin=inline`, `references`), and the `b2_relations:` entry +types the same connection with a stance (`origin=frontmatter`, `supports`): the augment shape +from §2, exercising both homes at once. The derived graph (no live model needed to assert): + +- `references`: spaced-repetition → memory (`origin=inline`), from the prose wikilink. +- `supports`: spaced-repetition → memory (`origin=frontmatter`, explanation "applies…"). + +`b2 neighbors concepts/memory` returns spaced-repetition twice (referenced-by, supported-by). +Both files round-trip byte-identical. Dropping and rebuilding the index reproduces the +identical graph. + +--- + +## 9. Judgment calls: all resolved + +The data model is locked; nothing is open. Every decision and its rejected alternatives are in +[ADRs/](../ADRs/README.md), chiefly ADR-0002 (two tiers), ADR-0003 (path identity), ADR-0004 +(write discipline), ADR-0009 (no suggestion queue), and ADR-0010 (the two homes, the dedup +rule, and the verb core). The normative claims are the register's S, W, L, and G entries. + +--- + +## 10. Resources: the second kind of vault member + +§0 to §9 define the authored objects. A real vault also holds resources: every non-`.md` +file. This section defines what a resource *is* in the model; the schema is +[index-engine.md](index-engine.md) §3. + +A resource is a peer vault member, not a lesser one, and not a generalized note. The single +asymmetry is the authoring surface, not status: + +> A note is where structure is *authored*; a resource is a peer document B2 cannot write. +> Notes have frontmatter, authored edges, and B2's write guarantees, because Markdown is the +> one format whose bytes B2 may touch. Resources have bytes, derivable text and vectors, and +> inbound links. Identity is not part of the asymmetry: both are keyed by their vault-relative +> path (L3), so the note rules are the resource rules with an authoring surface added, not a +> second identity model. + +What the asymmetry does *not* mean: a resource is never required to be attached to a note. An +unlinked resource fully exists: walked, classified, in the file tree, in the index, openable. + +**Identity: path-keyed, index-only.** There is nowhere to stamp a machine id (binary bytes are +not B2's to edit; a sidecar file would be durable state outside Markdown, violating S4) and +nothing one would protect. That reasoning is now the whole vault's; ADR-0003 read it back onto +notes. + +- **`b2 mv` on a resource** is simply *the* move: rewrite the inbound `[[path]]` / + `![alt](path)` text, move the file, re-project. Identical terms to a note (§3). +- **Placing one is not authoring one.** "B2 cannot write a resource" is about its *content*: + B2 never edits the bytes, and there is no format-specific writer. Putting a file *into* the + vault on explicit command (the desktop's drag-from-Finder import, `Vault::import_file`) is + file management, the same act as moving or deleting it (W3). The copy is byte-honest: B2 + places exactly the bytes it was handed, then projects them, routing on the extension exactly + as the walk does. A dropped `.md` is therefore a *note* arriving, frontmatter and all. +- **An out-of-band move degrades exactly as a note's does:** identification, not repair. The + old path's rows prune, the new path projects as a new member, and every inbound link + surfaces as a dangling edge (G5). The index keeps a blake3 content hash per resource, and + notes carry a `body_hash`, so a *proposed*-repair feature stays buildable on data already + stored. It is recorded as future investigation in GH #170 and deliberately not built: a + proposal is still yours to accept (W4). + +**Edges: `src` is a note in v1; `dst` may be anything** (G6). This is a consequence of the +invariant, not a status rule: every edge must trace to an authored line in Markdown (§3), and +a resource has no writable home for one. Two relief valves keep this from becoming an +expressiveness wall: (a) today, the tolerated tail already authors the inverse direction from +the note side (`- "supported-by [[papers/x.pdf]]"`); (b) if needed, resource-sourced edges get +a designed future home, a vault-level B2-managed relations file, so the edge is still authored +Markdown and the invariant holds ([GH #102](https://github.com/AlteredCraft/B2/issues/102), +deferred until the need is real). + +**What is built, and what is designed.** The inventory-and-graph half ships: the walk +classifies every non-`.md` file by extension into one of six classes (`text`, `html`, `pdf`, +`image`, `media`, `binary`, the total fallback), records +`(path, class, size, mtime, content_hash)`, prunes what the walk no longer meets, and resolves +body embeds and links at them into `dst_resource_path` edges. Classification is by extension +only: deterministic, and a misclassification degrades rather than mis-executes. + +The content half is locked design, not shipped: every class funnels to *text*. Native for +`text`, extracted for `html` (tag-strip) and `pdf` (text layer), and, for an `image`, the +aggregated alt-text and captions from the notes that embed it (a pure projection of authored +Markdown). That text then flows through the existing bge space with zero new discipline +(chunks plus a per-document centroid). +[GH #108](https://github.com/AlteredCraft/B2/issues/108) tracks it, +[#109](https://github.com/AlteredCraft/B2/issues/109) the PDF extraction dependency, and +[#107](https://github.com/AlteredCraft/B2/issues/107) render mechanisms. Until they land, a +resource is an inventoried, linkable, openable peer that search and discovery do not reach. +Multimodal image embedding and an LLM/OCR describer are documented future seams, off by +default (M3, [GH #110](https://github.com/AlteredCraft/B2/issues/110)). + +**Why a separate object, not a `kind` column on the note.** Two tables, two contracts, zero +"unless it's a resource" clauses. Generalizing `notes` to hold resources would staple a caveat +onto every invariant, write guarantee, and frontmatter behavior in §0 to §9. A distinct +`resources` table isolates the different *write* contract instead of threading it through the +note rules. diff --git a/docs/index-engine.md b/docs/index-engine.md new file mode 100644 index 0000000..21655d9 --- /dev/null +++ b/docs/index-engine.md @@ -0,0 +1,602 @@ +# Index engine + +How B2 turns a vault into a searchable index and serves reads over it, for anyone changing +the engine. This page specifies the disposable SQLite index (FTS5, an in-process vector scan, +and the typed graph) and the four flows over it. + +Related pages: [invariants.md](invariants.md) is the normative register, cited by id. +[data-model.md](data-model.md) defines what the index projects. The *why* behind each choice +is an [ADR](../ADRs/README.md): why SQLite and not qmd (ADR-0019), why plain vector tables +(ADR-0006), why the graph is materialized (ADR-0010), why discovery always serves (ADR-0014), +why a served result claims evidence (ADR-0015), why candle (ADR-0020). Each section below +names the record that decided it. + +Words used on this page: a **chunk** is a slice of a note, roughly 450 tokens, the unit +everything compares. An **embedding** (or vector) is the list of numbers a model produces for +a chunk: its position on a map of meaning. **BM25** is classic keyword ranking (rare words +count more). **RRF** (Reciprocal Rank Fusion) merges two ranked lists by position. + +## 1. qmd, the reference, and the chunker + +[qmd](https://github.com/tobi/qmd) is a local CLI search engine for Markdown, all on-device: +SQLite + FTS5 + `sqlite-vec`; ~900-token chunks with ~15% overlap and Markdown-aware +break-point scoring; three search modes; a pipeline of LLM query expansion → parallel +retrieval → RRF fusion (`Σ 1/(k+rank+1)`, k=60) → cross-encoder rerank → position-aware +blend; local GGUF models; TypeScript on Node, MIT. + +We rebuilt rather than depended (ADR-0019). Taken wholesale: the chunking heuristic, the RRF +formula and `RRF_K = 60`, the position-aware blend, the asymmetric query/document prompt +discipline (each model brings its own prefix; B2 ships bge's, §6), and the JSON/`--explain` +agent-output discipline. Discarded: the npm/Node packaging and the "DB is the product" +framing. + +**The chunker as adapted** (`chunk.rs`, [GH #19](https://github.com/AlteredCraft/B2/issues/19)). +Four model-free changes to qmd's heuristic: + +- A **450**-token target: headroom under bge's hard 512-token truncation. +- A `chars/4` proxy for token sizing, so the core stays tokenizer-free (E1). +- An unconditional stored `heading_path` breadcrumb on every chunk. +- Every lever on a `ChunkConfig`: overlap 0.15, backscan 200, and the break weights (H1=100 + down to list=5). + +The chunker prefers to cut at headings, then paragraph breaks. A forced cut is pushed past a +fenced code block or a Markdown table rather than bisecting it +([GH #41](https://github.com/AlteredCraft/B2/issues/41)). Consecutive chunks overlap by about +15%, so an idea straddling a boundary is not sliced in half. Every chunk records +`char_start..char_end` for the exact body slice that produced it, so it stays addressable for +highlight and explain. The config is eval-validated: GH #44 ran a seven-variant sweep +(`make eval-sweep`) and kept `ChunkConfig::default()`; the `prepend_heading_path` knob +measured rank-neutral twice and ships off. Tree-sitter AST chunking for code stays deferred +([GH #104](https://github.com/AlteredCraft/B2/issues/104)). + +## 2. Why we rebuilt instead of depending on qmd + +Recorded in full as +[ADR-0019](../ADRs/0019-build-our-own-sqlite-index-engine.md). In one line: qmd is a search +engine, and B2 is a typed graph with hybrid retrieval over it. qmd models no typed edges, no +backlinks, and nothing that repairs links on a move (G1 to G6, L1), and SQLite holds all three +queryable concerns in one transactional store. + +## 3. The storage architecture + +One artifact, per S1/S2 and ADR-0002: a disposable SQLite index holding every queryable +concern transactionally, rebuildable from the vault at any time (S3). + +The precise DDL and build order live in `crates/b2-core/src/db.rs` (schema) and `ingest.rs` +(flows). The sketch below is the orientation; the code is the buildable contract. + +``` +b2.sqlite — DISPOSABLE CACHE (= projection of the vault directory; drop & rebuild any time) +├── MIRROR OF THE VAULT (lets us diff vs. disk) +│ ├── meta(key, value) -- schema_version, embed_model_id, embed_dim +│ ├── notes(path PK, type, title, description, -- the path IS the identity (L1) +│ │ created, updated, body_hash, mtime, indexed_at) +│ ├── note_aliases(note_path, alias) -- frontmatter `aliases:` +│ └── resources(path PK, class, size, mtime, -- non-.md peers; class by extension (§10 dm) +│ content_hash, indexed_at) +│ +├── DERIVED: SEARCH +│ ├── chunks(id, note_path, seq, char_start, char_end, token_count, heading_path, text, text_hash) +│ ├── chunks_fts -- FTS5 over chunk text, `porter unicode61` (BM25) +│ ├── embeddings(text_hash PK, vector) -- CONTENT-ADDRESSED plain BLOB vectors (768-dim) +│ └── note_centroids(note_path, centroid) -- per-note centroid (discovery's coarse stage) +│ +└── DERIVED: TYPED GRAPH + └── edges(id PK, src_path, dst_path, dst_resource_path, dst_path_raw, + type, origin, explanation, caption, embed, occurrence_index) +``` + +Every table is derived from the vault; there is no third home. The two vector tables are +created at *embed* time rather than in the base migration: their existence *is* the "this +vault has an embedding space" signal the BM25-only fallbacks key on (M4, ADR-0006). + +Why this shape fits B2: + +- **Everything keys on the vault-relative path** (L1, ADR-0003). `notes.path` is the primary + key; `chunks.note_path`, `note_aliases.note_path`, `note_centroids.note_path`, and + `edges.src_path` are `REFERENCES notes(path) ON DELETE CASCADE ON UPDATE CASCADE`. That + makes a B2-performed move a path re-key rather than a rebuild: `UPDATE notes SET path = …` + cascades through every child in one transaction, alongside the inbound link-text rewrite + and a re-projection of the inbound sources. `edges.dst_path` is deliberately *not* a + foreign key: it must be allowed to be NULL, the dangling case (G5). "Rename keeps every + backlink resolving" is therefore a property of the move operation, not of the key; the + price is that a move made outside B2 is a delete plus a create (§8). +- **Vectors are content-addressed** (M4, ADR-0006), keyed by blake3 of the chunk text, which + *is* the embed input, verbatim. A note that moves re-embeds nothing, identical text + anywhere shares one vector, and the only invalidation rule is "a hash no chunk references + is garbage", pruned by the whole-vault pass on the same derived-data lifecycle as + centroids. +- **Every `edges` row derives from Markdown** (G1, G2, ADR-0010), deduped frontmatter-wins. + There is no `status` column and no suggestion queue: `b2 link` appends a typed-link string + to the source note's frontmatter and re-projects that note. Committing is the projection of + an authored line, not an in-place index write. +- **Hybrid retrieval and graph queries compose in one query.** "Semantic-nearest chunks whose + note is within 2 typed hops of note X" is a join across `embeddings`, `chunks`, and + `edges`. This is the substrate `b2 similar` runs on. +- **Deterministic seams for tests.** A fake embedder writes to `embeddings`, so the whole + pipeline is assertable with no live model (E2). + +**Resources widen the projection without disturbing any statement above.** Path-keyed peers, +so `index = projection of (the vault directory)` still holds. `resources` is a separate table +from `notes`, not a `kind` column on it (two tables, two contracts, zero "unless it's a +resource" clauses), and `edges.dst_resource_path` lets a body `![[photo.png]]` or +`[[papers/x.pdf]]` resolve against it while `src` stays a note (G6). What is built is +inventory plus graph; resource *content* search is designed, not shipped +([data-model.md](data-model.md) §10). Either way there is no migration: a schema change is a +`schema_version` bump plus a rebuild (S5). + +### How a reindex runs + +Ingest (`ingest.rs`) is the write path that realizes `index = projection of (the vault +directory)`. A reindex composes two separately invokable passes +([GH #15](https://github.com/AlteredCraft/B2/issues/15)): + +1. **Project** (`project_vault`, model-free): notes, resources, chunks, FTS, edges. +2. **Embed** (`embed_vault`): fill the DB-derived set of chunks still missing a vector. + +Projection itself runs in two phases so link resolution never depends on file order: phase 1 +projects every note and its chunks (filling the resolver, the set of known note paths, for +the whole vault); phase 2 derives edges against that now-complete resolver. Chunk text lands +in `chunks_fts` at projection time, so a projected-but-unembedded vault is already +keyword-searchable (search degrades to BM25-only). The desktop uses exactly this: project, +paint the tree, embed in the background. S2 is untouched: a projected-but-unembedded index is +a smaller projection, never a wrong one. + +**The incremental skip: embed only what changed.** A full re-embed is the one genuinely slow +step (a real transformer, on CPU), so a routine reindex must not redo finished work. Per +note: if the body is unchanged and its chunks already all have vectors, reuse them and embed +nothing. Otherwise re-chunk and re-embed. Two signals, one per pass: + +| Signal | Source | Why it is needed | +|---|---|---| +| body unchanged | `db::note_body_hash`, the stored hash read before the upsert overwrites it | same body ⇒ `chunk_body` yields identical chunks, deterministically | +| fully embedded | `db::note_fully_embedded`, every chunk has an `embeddings` row | catches the model-swap case: body unchanged, but the vector space was just emptied | + +`--force` bypasses the skip; a model swap forces it implicitly (§6). Edges are the asymmetry: +they are *always* re-derived in phase 2 (cheap), because a link's target may have moved even +when this note's body didn't. A frontmatter-only edit (a `b2_relations:` entry written by +`b2 link`) therefore re-projects the note and its edges but embeds nothing, which is what +makes `b2 link` cheap. + +The skip only reuses vectors a fresh embed would reproduce byte-for-byte (the embedder is a +pure function of chunk text), so incremental ≡ full rebuild holds (S3). Both paths replace +rather than accumulate: re-projecting a note deletes its old chunks (FTS triggers fire) and +all its edges, then re-derives everything from the current Markdown. The whole-vault pass +owns every *reconciliation*: pruning rows for files the walk no longer met, collecting +vectors no chunk references. Single-note paths never prune. An interrupted embed heals on the +next reindex, because the embed pass fills whatever chunks lack vectors, whyever they lack +them. + +**The operator view.** The embed phase reports `ReindexProgress` after every batch; the CLI +renders a live line on an interactive stderr only, so `--json` and piped output stay pure +data. The count tracks notes that actually embed, not position in the full list. `reindex` +reports what it did: `indexed`, `embedded` (the count that re-ran), `resources`, plus any +unreadable files it skipped. `b2 reindex --dry-run` previews the same decision, writing +nothing. `b2 reindex &` backgrounds through the shell; `b2 status` reports coverage plus the +running pid; `b2 reindex --cancel` signals that pid onto the same cooperative-cancel path +Ctrl-C uses. Readers keep working throughout (C1). + +### Opening the index concurrently: many readers, one builder + +Several things open one index at once: `b2 reindex &` racing a `b2 status`, the desktop app +launching while a CLI reindex runs, the desktop host's own threads. The locked stance is C1, +enforced in `db::open` in three layers, each answering a different failure: + +- **The WAL flip is retried** ([GH #111](https://github.com/AlteredCraft/B2/issues/111)). + Setting `journal_mode = WAL` is the one statement in `open` that takes a write lock and the + one `busy_timeout` cannot cover: SQLite skips the busy handler for a write lock upgraded + from an already-open read transaction, so a second opener took an immediate `SQLITE_BUSY`. + The wait is ours. +- **The schema migration is one `BEGIN IMMEDIATE` transaction, entered only when there is + work** ([GH #114](https://github.com/AlteredCraft/B2/issues/114)). Unserialized, two openers + that both read a stale `schema_version` interleave: one's `DROP TABLE` lands after the + other's `CREATE`, and the current version gets stamped over a half-demolished schema. + `busy_timeout` was irrelevant: nothing contended; every statement succeeded, in the wrong + order. The check that decides whether to enter is a *read*, so the common open against a + current schema takes no write lock and can never be refused. +- **Completeness is checked, not assumed.** A stamp is believed only alongside the tables it + vouches for; a current stamp over missing tables is stale and rebuilt from empty. + Recreating just the missing tables would be worse than useless: an incremental reindex + skips notes whose `body_hash` matches, so they would stay empty and S3 would quietly fail. + +The vector tables are the same drop-and-rebuild shape and get the same treatment. Not an +advisory lock file: that would be a third concurrency mechanism guarding state the database +already guards, and the weaker one where it counts (on a network share or synced folder, +`flock` quietly stops meaning anything). The `reindex` lock +([GH #55](https://github.com/AlteredCraft/B2/issues/55)) answers a question SQLite cannot +(is another *process* already doing this expensive work?). It is taken by `b2-cli` alone, +never by the desktop host or by readers, so it cannot cover schema atomicity. + +### Why materialize the graph at all + +A note's *outbound* links are parseable from that one file on demand, so edge metadata is not +the reason. Inversion and composition are. Materializing turns three things from full-vault +scans (or impossibilities) into indexed lookups: + +- **Backlinks.** "Who points at X" cannot be read from X, only from every *other* note: + O(vault) per query at runtime, one lookup here. This also services L1: the edges name the + exact files to rewrite on a move instead of scanning the vault to find them (§8). +- **Typed multi-hop traversal.** "Notes within 2 hops of X via `supports`" is a scan per hop + at runtime; over `edges` it is one SQL traversal. +- **The graph⨝vector join.** "Semantic-nearest chunks whose note is within k typed hops of X" + is a single join `embeddings ⨝ chunks ⨝ edges`, not expressible as a per-note parse. + `b2 similar`'s candidate generation is its complement: notes semantically near an anchor + but *not* within 1 hop, where the materialized graph supplies the exclusion. + +G6 keeps this cheap: runtime outbound-parsing is the correctness *definition*, and the +`edges` table is its *cache*. One more disposable table in the same store, populated by the +same parse pass that already walks each body for chunking. The standing cost is the +move-repair amplification in §8. + +### Discovery serves the ranked list: `limit` is a cap, not a promise + +The standing rule is invariants.md D1; the decision, the seven issues of measurement behind +it, and the retired z-gate are +[ADR-0014](../ADRs/0014-discovery-always-serves-the-ranked-prefix.md). Mechanically, in this +engine: + +- `similar` serves the ranked top-N whenever candidates exist. `limit` under-fills only for + want of scorable notes, and no statistical bar truncates the list. +- The per-candidate **z survives as a statistic that gates nothing**: computed after stage 2 + on the best-passage distances, non-increasing down the row order (strictly monotonic in the + score; tied scores share a z and order by the path tie-break), painted as the within-list + strength band. Because the judged z is affine in the squared best-pair distance the score + negates the root of, score order, z order, and band are one number: a card can never show a + weaker band above a stronger one. +- Below `STATS_MIN_POPULATION` (12), in a space with no spread, or under the fake embedder, + no z exists at all, and a surface must *say* the list is ungraded. Silence there would read + as "all judged, all scored low", the opposite of what happened. +- Candidate *generation* is unchanged and stays recall-oriented: the two-stage scan (§4) + over-produces, nothing auto-links, and the human commits every edge (W4). + +The **pair-scorer escalation** stays the named long-term seam. An anchor-local rule cannot +catch a *pair-level* miscalibration (a single stranger the model scores like a cluster-mate, +the standing `encryption ↔ phishing` residue). A discovery-side pair-scorer would be a second +model seam, sibling of §5's reranker but distinct from it (that seam needs query text, and +`similar` has none). It would still only filter what is surfaced, never author a link. Under +always-serve that residue is ordering quality, not existence. + +## 4. Retrieval: search, fusion, and discovery + +Semantic search is in v1 (ADR-0019): exact, in-process, no vector extension, no approximate +nearest-neighbor index. + +**Storage and scoring.** Vectors live in plain tables (`embeddings(text_hash, vector)` plus +per-note `note_centroids`), read with one statement and scored in-process (`embed::l2_sq`). +Content-addressing costs the scan one indexed join back to `chunks` to recover which chunk +each vector ranks for. A `vec0`-style virtual table charges a per-row shadow-table probe on +every scan, which dominates at real-vault scale; the plain-table scan does not +([GH #38](https://github.com/AlteredCraft/B2/issues/38), ADR-0006). + +### Flow ②: search, from query to served rows + +`search.rs`; the entry point is `Vault::search_evidence` (the CLI's `cmd_search` adds +`--exclude` via `search_evidence_excluding`; the desktop's `search` command is the same +read). The stages, in order: + +1. **Fail fast on a model swap.** `ensure_query_space_matches` returns + `Error::ModelMismatch` ("run `b2 reindex`") rather than fusing incomparable vectors (M2). +2. **Dispatch.** If the `embeddings` table does not exist, run `keyword_only_search`: the + BM25 leg alone, same fusion, so scores stay on one scale. BM25-only is a smaller + projection, never an error, and `best_cos` is honestly `None`. +3. **The keyword leg.** `fts5_query` sanitizes raw natural language into a safe FTS5 `MATCH` + expression: each alphanumeric run double-quoted so nothing reads as an operator, OR-joined + for recall. Punctuation is FTS5 syntax and would otherwise crash the parse. The index it + matches is stemmed (`porter unicode61`, below). +4. **The semantic leg.** `embedder.embed_query` (bge's asymmetric query prefix), then + `db::vector_search`: a full in-process scan, `embed::l2_sq` per chunk vector, nearest + first. +5. **Provenance.** `provenance_of` records, per chunk, its BM25 rank, its vector rank, and + its own distance, plus the query's best cosine: the absolute signals RRF is about to + discard (D2). +6. **Fusion.** `rrf_fuse`: `score = Σ 1/(60 + rank + 1)` over both lists (`RRF_K = 60`). A + fused-score tie breaks on the dense list's rank, and that is a policy, not walk order: + RRF over integer ranks lands every fused score on a lattice, so mirrored rank pairs tie + structurally, and the eval corpus produced one where the semantic half named the labelled + answer and BM25 named the wrong one ([GH #156](https://github.com/AlteredCraft/B2/issues/156)). + Absent ranks sort below present ones; id is last, purely for determinism. +7. **Resolve to notes.** `resolve_hits` / `resolve_note_hits`: dedup chunks onto each note's + best one, cut a query-windowed snippet, stop at `limit`. +8. **The evidence verdict.** `lexical_evidence` reads IDF-weighted term coverage; + `EvidenceBar::for_model` supplies the calibrated bar (or none); + `vouched = coverage clears it OR best_cos clears it`. The result is a + `SearchEvidenceView`: the rows, whole and in fused order (never reordered), plus + `{vouched, chunk_total, terms, best_cos}`. + +**The three verdict states are three behaviors** (D2, ADR-0015, +[GH #202](https://github.com/AlteredCraft/B2/issues/202)): + +| Verdict | CLI human mode | CLI `--json` | Desktop | +|---|---|---|---| +| `vouched: true` (evidence found) | the rows | the object: rows + verdict | the rows | +| `vouched: false` (no evidence) | "No matches." and none of the rows. Strict: no reveal, no `--all` | the rows are served, beside `vouched: false`; an agent handed an explicit verdict can be honest about them | no rows kept (one boundary: `doSearch`, `ui/src/main.ts`) | +| `vouched: null` (no calibrated bar for the active model: the fake, or any unmeasured model) | serve as always; no verdict is offered rather than one guessed | same | same | + +The verdict rule is *lexical OR semantic*: two independent signals, because one test cannot +tell "nothing here matches" from "everything here matches". Details and history: D2. The +lexical anchor is IDF-weighted term coverage (`min_term_coverage`): each term weighs +`ln((chunks+1)/(df+1))`, and the anchor is the share of the query's own weight the vault +carries. A stopword is a measurement, never a shipped word list. The cosine bar (`min_cos`) +is the backstop, judged only on queries the lexical half leaves undecided, and thin by +construction: ask the lexical half for more, and the window collapses, because the queries it +must then rescue have the weakest semantic evidence too. The two constants are placed inside +a joint band, never tuned one at a time. They are distributional, keyed to `embed_model_id` +(M2; a swap invalidates them; the device suffix shares the reading, re-checked by +`make eval-metal`). They live in `search::BGE_BASE_EVIDENCE_BAR`, and their justification is +re-derived on every `make eval` run, never frozen into a comment (ADR-0013). + +**Width is a quality knob, not plumbing** +([GH #142](https://github.com/AlteredCraft/B2/issues/142)). Retrieval widens twice: each +façade read asks for a hit pool over its `limit`, and each signal (`search::pool_size`) pulls +5× *that* (minimum 30) before RRF fuses the two lists. Note-level `Vault::search` keeps 3×: +dedup collapses every chunk sharing a note onto that note's best one, so a pool of exactly +`limit` would under-fill `limit` distinct notes. Passage-level `Vault::search_chunks` (chat's +retrieve step) has no dedup and keeps `limit + 2`: enough to backfill the one hit a C1 torn +read can drop, and no more. Giving it 3× too is not a free tidy-up: the 5× multiplies every +hit of headroom into candidates, and RRF over a wider candidate set returns *different* +answers. At k = 60, a chunk ranked ~60th in both lists outscores one ranked first in a single +list (`2/121 > 1/61`). Width moves only on measured relevance (§5). + +**`--exclude` subtracts rows, never evidence.** `search_evidence_excluding` is the same read +minus a caller-named set of notes: the follow-up-search form an agent loop passes its +already-inspected paths to. The verdict and its signals still read the whole vault, the +remaining rows keep their fused order, and the pool is unchanged, so a heavily excluded query +may honestly under-fill. + +**The FTS tokenizer is `porter unicode61`, stemmed, and that is a measured verdict, not a +default.** Unstemmed BM25 matched surface forms only: `pedalling` found nothing in a note +that says "pedals", leaving the lexical half at rank 41 to 46 on queries the dense half +ranked first, which RRF's consensus bias turned into hybrid demotions of correct dense hits. +The A/B that settled it ([GH #157](https://github.com/AlteredCraft/B2/issues/157)): porter +improved 7 BM25-only and 3 hybrid note ranks, worsened none, and dissolved every standing +fusion demotion, while the precision probes built to vote *against* stemming (the +`universe`/`university` collision, the code-literal queries) did not move. Stemming remains a +real trade (Porter is English-only, and a vault holds code, identifiers, and proper nouns), +so the retired arm stays measurable: `Vault::rebuild_fts` swaps the tokenizer over identical +chunk rows and vectors, and `make eval-stemmer` scores the unstemmed ablation beside every +default run. + +**The tail.** Folding where a *real* query's per-hit evidence runs out is measured and stays +unshipped ([GH #206](https://github.com/AlteredCraft/B2/issues/206)): with `tail_relevant` +labels in place, the four-family prefix-cut bake-off found every admissible rule near-vacuous. +The fused order is not an evidence order, so admissible folds reach 2 to 23 of the 367 rows an +oracle fold would cut. The ruling and its numbers live in [evals.md](evals.md); the bake-off +re-arms every run. + +### Flow ③: similar, the two-stage discovery scan + +`discover.rs` (`candidates`); the entry point is `Vault::similar`. Discovery makes zero model +calls and touches no network: it is a pure read over stored vectors, so the CLI can open the +vault with the fake embedder and still serve real-model rankings. The anchor is represented +by its *stored* chunk vectors, never an `embed_query` of its text (bge's asymmetric query +prefix is the wrong side of the space). The stages: + +1. **Honest refusals.** A resource anchor is `ResourceUnsupported`; an unknown ref is + `NoteNotFound`. The grade flag is read from the recorded model id + (`db::recorded_embedder`): a fake-embedded space serves ungraded. +2. **Load the anchor.** Its stored chunk vectors (`db::note_chunk_vectors`), centroid + computed in-process. Nothing is re-embedded. +3. **Exclude the already-connected.** `graph::reachable_within(anchor, 1)`: the anchor and + its 1-hop neighbors are excluded *up front*, so they never occupy a shortlist slot. A + 2-hop note (the triadic-closure candidate: you linked A–B and B–E but never A–E) survives, + ranked purely by semantic score. +4. **Stage 1: centroid shortlist, O(notes).** Stream every note centroid + (`for_each_note_centroid`), keep `SHORTLIST_PER_RESULT = 20` per asked result, floored at + `SHORTLIST_MIN = 200`. The shortlist is a recall device, never a quality gate: on any + vault at or below 200 candidate notes, the two-stage result equals the whole-space scan + ([GH #38](https://github.com/AlteredCraft/B2/issues/38)/[#192](https://github.com/AlteredCraft/B2/issues/192)). +5. **Stage 2: exact max-sim over the shortlist only.** Every shortlisted note scores by its + single best chunk pair against the anchor (smallest L2²), and that winning chunk rides + along as the evidence passage the surface shows. Max-sim, not the centroid, decides the + score, so one strong section inside a messy note still counts (the buried gem; + [GH #192](https://github.com/AlteredCraft/B2/issues/192)). +6. **Rank, grade, cap.** Ties break by path. The z is computed only when graded (pool ≥ 12, + spread > 0, real model). `take(limit)`. The result is `CandidateNote`: path, title, + `score = −√(best-pair d²)`, evidence, optional z. The z gates nothing; it paints the band + (§3). + +`b2 similar` surfaces, `b2 link` commits, and you are the precision gate (ADR-0009). An empty +list at a nonzero `limit` means only "nothing to compare": no unlinked note has stored +vectors yet, or the space is not semantic. The CLI's two empty states say exactly that. + +**`graph_filtered_search`** is the near-neighbor of both flows that is neither: the +vector⨝graph scoped-traversal primitive, "nearest chunks whose note is within k typed hops of +an anchor" (near ∩ connected). Discovery is its complement (near ∖ connected). +`vector_only_search` is the eval harness's ablation instrument, never an adapter surface. + +### Does brute force scale to B2? + +Comfortably. A personal vault of 10k notes is roughly 50k to 100k chunks; brute-force cosine +over ~100k × 768-dim float32 vectors is single-digit to low-tens of milliseconds. We are +nowhere near the regime where an approximate index matters. If a vault ever is, int8/binary +quantization and ANN hold a standby order behind the centroid stage +([GH #106](https://github.com/AlteredCraft/B2/issues/106)). + +### The constants in one place + +Structural constants are design choices, quoted here. Distributional constants are +measurements keyed to the embedding model: named, never quoted. Their values live in the code +and their justification is re-derived on every `make eval` run (ADR-0013). + +| Constant | Value | Governs | Source | +|---|---|---|---| +| `RRF_K` | 60 | fusion weighting (qmd heritage) | `search.rs` | +| `pool_size` | 5 × hit pool, min 30 | per-signal candidate depth | `search.rs` | +| note hit pool | 3 × limit | note-view headroom (dedup + torn reads) | `vault.rs` | +| chunk hit pool | limit + 2 | passage-view headroom (torn reads only) | `vault.rs` | +| `ASK_PASSAGES` | 10 | chat's retrieve depth (flow ④ reads this flow) | `chat.rs` | +| `SHORTLIST_PER_RESULT` | 20 | discovery stage-1 width per asked result | `discover.rs` | +| `SHORTLIST_MIN` | 200 | discovery stage-1 floor | `discover.rs` | +| `EXCLUDE_HOPS` | 1 | discovery's already-connected radius | `discover.rs` | +| `STATS_MIN_POPULATION` | 12 | smallest population a z is claimed over | `discover.rs` | +| `BGE_BASE_EVIDENCE_BAR` | named, not quoted | the lexical anchor + the semantic backstop, per model | `search.rs` | + +## 5. Deferred model machinery: the reranker and query expansion + +**The reranker is a fast follow.** Slot it where qmd puts it: after RRF fusion, before final +ranking, behind a swappable seam. v1 is retrieve → fuse → return top-N, already a strong +hybrid baseline; the fast follow inserts a cross-encoder rerank over the top ~30 candidates +plus a position-aware blend. It is a pure function `(query, candidates) → scores`, so it +changes *ordering*, not the store, the schema, or the candidate set. It is store-agnostic, a +model-side seam above the index. Tracked in +[GH #28](https://github.com/AlteredCraft/B2/issues/28). + +**Scope: this reranks `b2 search`, not `b2 similar`.** The signature is the tell: it needs +query text, and `b2 similar` has none (it is passage↔passage KNN, "near ∖ connected", §3). +The discovery-side levers are distance weighting +([GH #20](https://github.com/AlteredCraft/B2/issues/20)) and the pair-scorer seam (§3); the +discovery-side precision stance is D1's (ADR-0014). + +**Gate the decision on the eval, not intuition** (ADR-0013). RRF is a strong baseline; the +reranker buys top-k precision, whose value grows with vault size and is highest when an agent +consumes top-1/top-3 with no human eye +([GH #24](https://github.com/AlteredCraft/B2/issues/24)). Vault size changes whether the +precision is worth it, not the reranker's cost. + +**And check the instrument can see the change you are gating.** Retrieval reaches at least +`chunk_candidate_pool(10) = 60` candidates per signal (150 for the note view). While a corpus +has no more chunks than that, neither signal is truncated, and a candidate-width change +prints bit-identical numbers while genuinely reordering a real vault +([GH #141](https://github.com/AlteredCraft/B2/issues/141)). Score *relevance* on the labelled +corpus, but measure a width change with `make stability` over a vault big enough for the pool +to bind. That gate has already ruled once: +[#140](https://github.com/AlteredCraft/B2/pull/140) widened the passage view as plumbing, the +eval printed bit-identical numbers, the probe found 10 of 10 top-4 lists changed, and +[#142](https://github.com/AlteredCraft/B2/issues/142) returned it. The instrument that can +say "different" is not the one that can say "better". The blindness is to *candidates*, not +to fusion: `RRF_K` re-weights the same two lists and reorders results at any corpus size. + +**Query expansion** (qmd's third model) is optional and lowest priority: the heaviest model +for the smallest, most variable win +([GH #105](https://github.com/AlteredCraft/B2/issues/105)). + +The harness itself (corpora, labels, metrics, exit gates, process rules) is +[evals.md](evals.md); read it before touching any of them. + +## 6. The AI seams: the embedder, and grounded chat + +Two seams, both enumerated by M1 (ADR-0005) and both injected by the adapters. + +### `Embedder` + +Runtime, provisioning, and the model choice are +[ADR-0020](../ADRs/0020-embeddings-inside-the-single-binary.md): `candle` + `hf-hub` compiled +into the binary, an explicit `b2 init` into a shared XDG cache, default +`BAAI/bge-base-en-v1.5` (768-dim, CLS-pooled, L2-normalized, bge's asymmetric query prefix), +the dimension read from the model's own `config.json`. Loading fails fast if the files are +absent ("run `b2 init`"), never a surprise mid-command download. + +The trait is five methods: `model_id`, `dim`, `embed` (one passage), `embed_query` +(asymmetric-ready, defaults to symmetric), and `embed_batch`. The fake (`FakeEmbedder`, the +CI default) blake3-hashes text into a vector: identical text, identical vector, +deterministically. Not semantic; a stand-in so the fast suite stays model-free. + +**Batching.** Embedding is the reindex hot path, so the write side hands whole batches of +chunks to `embed_batch` (up to `EMBED_BATCH = 16`). The real model turns a batch into one +padded forward pass instead of N single ones (a large CPU win; on macOS candle's matmuls run +on Apple's Accelerate BLAS). Right-padding plus the attention mask make each row's CLS vector +identical to embedding that text alone, so batching is a pure speedup, never a change in +result. That claim is pinned where it can actually run: `check_batch_matches_single` (cosine +> 0.9999, row for row) needs the provisioned model, so it lives in the eval harness and runs +on every `make eval` instead of sitting behind an `#[ignore]` nobody passes `--ignored` to. + +**The model swap.** The engine-side consequences are invariants (M2, ADR-0007, +[GH #40](https://github.com/AlteredCraft/B2/issues/40)): the embedding space has exactly one +recorded identity, `meta.(embed_model_id, embed_dim)`, and the compute device folds into it +(a Metal build tags the id `@metal`). On a swap, `ensure_embedding_space` drops and recreates +the vector tables empty; every note is then "not fully embedded", so the incremental skip +(§3) re-embeds the whole vault on that reindex, automatically, no `--force` needed. Opening a +vault never mutates the vector space, so changing the configured model cannot silently wipe +vectors on a read command. A stale read fails fast: `search` compares the recorded model to +the active one and returns `ModelMismatch` ("run `b2 reindex`") rather than fusing +incomparable vectors. The fake stays the CI default, so model quality never enters the fast +suite (E2). + +### `LlmProvider`: flow ④, grounded chat + +Chat is a *reader* of this index and adds nothing to it (M1): no table, no cached response, +no `meta` row, session-only history (S4). That is the deliberate contrast with the embedder: +swapping chat models never touches the index, so a provider swap is a URL or config change. + +`Vault::ask` (`chat.rs`) is five steps: + +1. **Condense** (multi-turn only). A provider call rewrites the follow-up into a standalone + query. On failure it degrades to the raw question, so this step can never break chat. +2. **Retrieve.** `search_chunks` at `ASK_PASSAGES = 10`: the §4 pipeline unchanged, BM25-only + fallback included. +3. **Assemble.** The grounded system prompt plus numbered passages. Prompt assembly is core + logic, not an adapter's. +4. **Stream.** Tokens flow up through the caller's callback, whose return value cancels at + token granularity. Sync, no runtime. +5. **Cite.** `[n]` markers resolve to `(path, excerpt)` in the returned `AnswerView`. A + hallucinated marker resolves to nothing, and the answer text is never rewritten. + +Two properties follow from the seam's shape. Cancellation is returning early from a blocking +read loop, so no B2 crate starts an async runtime (ADR-0011): a cut stream is marked +cancelled, and what already arrived is rendered, never discarded, because a truncated answer +is not an error. And model output is untrusted content like any note (E5, ADR-0016), +enforced at the render surface rather than in the core. + +## 7. Tech stack: Rust + +The single-binary goal picked the language, not the engine; SQLite and FTS5 are +language-agnostic. See ADR-0019. + +## 8. Risks and operational burden + +**Chunk vs. note granularity.** Search is chunk-level; the typed graph is note-level. +`chunks.note_path` is the join, and search hits resolve up to notes for graph operations +(§3). + +### The bill for a path-keyed graph under `[[path|title]]` links + +Keying the graph by the vault-relative path (L1, ADR-0003) makes the stored key the same +thing you authored. The items below are the trade working as designed, not defects. They must +be budgeted, tested, and watched. + +- **Write amplification on move.** The link *is* the path, so moving one note rewrites the + inbound link text in every file that points at it: an N-file write. It is bounded and + mechanical (the materialized edges name exactly which files and links to touch, Markdown + first, then the index), but moving a heavily linked note is proportional to its backlink + count, not O(1). The rewrite is transactional, so a partial move never half-updates the + vault. The index side is one cascading `UPDATE` plus re-projection of the inbound sources, + bounded by the same count. Only the exact files the graph names are touched: a + prefix-sharing `[[foo-bar]]` is never rewritten when moving `foo`. +- **Out-of-band moves are identified, not repaired, and that is the scope decision.** A + `git mv` or Finder move is, to a path-keyed index, a delete plus a create: the old path's + rows prune, the new path projects fresh, and every inbound `[[oldpath]]` becomes a surfaced + dangling edge (G5): authored text kept, `dst` NULL, healing by itself on the next pass if + the target comes back. Nothing is silently dropped, and nothing is guessed at. + Content-addressed vectors (M4) make it cheap as well as honest: the re-created note's chunk + text hashes identically, so the delete-plus-create re-embeds nothing. Two repairs are + future investigation in GH #170: a *proposed* hash-match repair (`notes.body_hash` and the + resource `content_hash` are already stored, but a proposal must stay yours to accept, W4), + and a `b2 watch` daemon observing renames live, rejected as a lifecycle and coordination + surface that shrinks the gap rather than closing it. +- **Path ownership follows the filesystem, so there is nothing to reconcile.** A path names + at most one file (the filesystem's guarantee), so a note deleted and recreated at the same + path is that path's note, a Finder-duplicated note is two notes at two paths, and + `db::upsert_note`'s `ON CONFLICT(path)` is the whole of the reconciliation. A note deleted + with no replacement is reconciled by the whole-vault pass + ([GH #31](https://github.com/AlteredCraft/B2/issues/31)): `project_vault` prunes every + `notes` row whose path the walk did not see this run (aliases, chunks, FTS, centroid, and + outgoing edges cascade; inbound links re-dangle when phase 2 re-derives edges against the + pruned resolver), *except* rows whose file was skipped as unreadable: the walk *saw* that + file, so evicting it would lie. Single-note ingest (`add`/`mv`/`write`) touches one note + and never prunes. Orphaned vectors (hashes no chunk references after that pruning) are + collected by the same pass: the only bookkeeping content-addressing adds. +- **A single unreadable file never fails the whole index.** A real vault holds the odd + non-UTF-8 or permission-denied `.md`. Projection skips it (reported as a `skipped` entry + with a short, file-level reason, surfaced by the CLI and the desktop) and indexes + everything else, rather than aborting on one file it cannot read. +- **Derived-index consistency is a permanent invariant, not a one-time build.** W5, S3, and + L1 are the tripwires. Every edit path (`b2 mv`, link delete, out-of-band reindex) has to + preserve all three, or the graph silently diverges from the source of truth. +- **Committed edges are only ever authored, never inferred** (G1). Editing the vault can + strand a connection (deleting an authored A→B link), but B2 only ever *surfaces* the + consequence. It never silently rewrites an inbound file or an edge (W4). + +(The stamped `b2id` made "two files, one identity" representable, and cost a collision +subsystem, a shadowed-copy panel, restamp notices, and a carve-out on S3. All of it went with +the stamp, ADR-0003. Nothing anomaly-shaped was ever *stored*, so nothing had to be migrated +away.) diff --git a/docs/invariants.md b/docs/invariants.md new file mode 100644 index 0000000..8127ff0 --- /dev/null +++ b/docs/invariants.md @@ -0,0 +1,328 @@ +# Invariants + +The rules that must always be true of B2. Read this before you change B2's behavior. +Find the entries for the area you are touching and keep them true. Cite entries by id (S2, D1). + +How this page works: + +- Each entry is one testable claim. If this page disagrees with any other doc, or with the + code, this page wins. Fix the other side. +- Changing an entry is a deliberate decision, never a side effect of another edit. It means + writing or superseding a record in [ADRs/](../ADRs/README.md), which holds the *why* behind + every entry. +- [data-model.md](data-model.md) defines the shapes these rules govern (the *what*). + [index-engine.md](index-engine.md) defines the machinery (the *how*). +- The product non-negotiables (local-first, zero lock-in, single binary) live here as entries + too. + +Two ideas drive the whole list. First: your vault is volatile and the index is disposable, so +you can rewrite your notes without fear. Second: build for tomorrow's model, so a better model +drops in without a redesign. + +## S. Storage: two tiers, one projection + +- **S1. Two tiers, sharply split.** The vault (Markdown files, resources, and the folder tree) + is the source of truth. `.b2/b2.sqlite` is a disposable cache. Nothing in the index is + authoritative. ([data-model.md](data-model.md), "The two storage tiers") +- **S2. The index is a pure projection: `index = projection of (the vault directory)`.** Drop + `b2.sqlite`, reindex, and you get an identical index back. Markdown is the vault's only + *authored* format: the only format whose bytes B2 may write. Resources contribute derived + rows only. Folders are never projected at all; B2 reads them live off disk. The projected + domain is the vault's *managed subtree*: a dot-prefixed name is not vault material, whether + it is a folder, a resource, or a `.md` file. Every walk skips it before deciding what it is, + and every authoring command refuses it as a destination, so B2 never creates a file it would + then never see. The file itself stays on disk, untouched, simply outside the projection. + ([data-model.md](data-model.md) §1, §10, [index-engine.md](index-engine.md) §3, + [GH #136](https://github.com/AlteredCraft/B2/issues/136)) +- **S3. Full reindex ≡ incremental update, with no exceptions.** Re-deriving one changed note + lands on exactly the state a from-scratch rebuild would produce. That includes pruning: a + whole-vault pass removes rows for files the walk no longer finds. There is no carve-out. + Identity is the path (L1), and the filesystem guarantees one file per path, so "two files + with one identity" cannot arise; a copy is just another note at another path. + ([index-engine.md](index-engine.md) §8, + [GH #170](https://github.com/AlteredCraft/B2/issues/170)) +- **S4. No durable B2-derived state outside the Markdown.** No event log, no sidecar files, no + facts that live only in the index. The scope is *B2-derived* data. Your own folder tree is + vault material, and the filesystem is authoritative for it: folders are never projected, and + the tree listing is a live walk of the disk. ([data-model.md](data-model.md), "Folders") +- **S5. A schema change is a version bump plus a rebuild, never a data migration.** + Disposability makes this free. A migration script would be evidence that S2 broke. + ([index-engine.md](index-engine.md) §3) + +## W. Writes: the vault changes only on your command + +- **W1. B2 makes no unbidden writes. Period.** Every byte B2 writes to the vault is the + mechanics of a command you ran (W3). Reading a vault (walking it, projecting it, reindexing + it) writes nothing. `reindex` runs unchanged on a read-only vault, and a git-versioned vault + shows no diff from having been indexed. ([data-model.md](data-model.md) §1) +- **W2. B2 never writes the note body, and never asks it to carry B2 syntax.** The body is + 100% your document. The one body write is the mechanical move repair: when a note moves, B2 + rewrites the *path text* inside inbound `[[oldpath|alias]]` links so they keep resolving. It + fixes a link you already wrote. It never adds one, and aliases survive verbatim. + ([data-model.md](data-model.md) §0) +- **W3. The on-command writes are a short, closed list:** + - `b2 link` appends one `b2_relations:` entry (frontmatter, never the body). + - The move repair of W2. + - The editor save (`Vault::write`): a byte-honest splice of *your own* body bytes, guarded + by a content-hash revision. + - The frontmatter save (`Vault::write_frontmatter`): the same guarded splice of *your own* + frontmatter bytes, body untouched. B2 owns no line in that block and judges none. + - Import (`Vault::import_file` / `import_path`): the handed bytes copied verbatim, then + projected. + - Create, move, and delete of notes, resources, and folders, on explicit command. +- **W4. B2 never deletes, moves, or archives your files on its own.** Consequences of your + edits (orphans, dangling links, hash-matched move candidates) are surfaced, flagged, or + proposed. They are never silently applied. ([index-engine.md](index-engine.md) §8) +- **W5. Round trips are lossless.** `parse → serialize → parse` is byte-identical outside the + one edit performed. Unknown frontmatter keys survive verbatim, in order. B2's one key is + namespaced (`b2_relations`) so it can never collide with yours; a generic `relations:` key + is *not* read. A `b2id:` line left by an older B2 is now just an unknown key: never read, + never rewritten, never removed. Deleting `.b2/` is the whole upgrade + ([GH #170](https://github.com/AlteredCraft/B2/issues/170)). + ([data-model.md](data-model.md) §1, §6) + +## L. Identity and links + +- **L1. A note's identity is its vault-relative path, and every edge is keyed by path.** Both + link homes are already written by path: a body `[[path]]`, or a frontmatter `b2_relations:` + entry. So an edge stores what you authored, resolved at projection time, with no machine id + in the file. Consequence: a rename keeps every backlink resolving *when B2 does the move*. A + B2 move rewrites the inbound path text and re-keys the moved note's rows in one transaction. + A move made outside B2 is a delete plus a create: the inbound links surface as dangling + (G5), identified rather than silently dropped. + ([data-model.md](data-model.md) §1, §3, + [GH #170](https://github.com/AlteredCraft/B2/issues/170)) +- **L2. A note's title is its filename.** The frontmatter `title:` key is recognized but + inert: round-tripped, never driving display, aliases, or search. `b2 link` therefore writes + a bare `[[path]]` with no alias. ([data-model.md](data-model.md) §1) +- **L3. Notes and resources share one identity model.** Both are keyed by the vault-relative + path, index-only, with no sidecar files ever. The one asymmetry left is the authoring + surface, not status or identity: a note has frontmatter and authored edges because Markdown + is the one format B2 may write; a resource is a peer document B2 can read and never write. + ([data-model.md](data-model.md) §10) +- **L4. The body is read strictly as ordinary Markdown.** Every body link (wikilink, Markdown + link, embed) is an untyped, *directed* `references` edge. No prose shape (a list marker, a + leading verb) is ever B2 structure. ([data-model.md](data-model.md) §2) + +## G. The typed graph + +- **G1. Every edge is authored and active.** An edge exists exactly when it is written in the + Markdown. There is no `status` column, no suggestion queue, no lifecycle, and nothing inert. + Committing a connection means appending an authored line and re-projecting, never mutating + the index in place. ([data-model.md](data-model.md) §3, §4) +- **G2. The edge set is the union of exactly two homes, deduped frontmatter-wins.** Body links + (`origin=inline`, always untyped) plus frontmatter `b2_relations:` entries + (`origin=frontmatter`, the *only* home of a verb and explanation). If the same + `(target, type)` appears in both homes, the frontmatter row is kept, because only it can + carry the explanation. A *different* verb over a body-linked target coexists (the augment + case). Nothing is ever copied between homes or auto-removed from a file. + ([data-model.md](data-model.md) §0 to §3) +- **G3. The relation vocabulary is a closed three-verb core plus a tolerated tail.** + `references` (neutral), `supports` (for), `contradicts` (against, symmetric) is the typing + palette and what queries can rely on. Any other verb is stored verbatim as an opaque tail. + The closed core is a policy B2 can relax later (promotion), never a structural assumption. + ([data-model.md](data-model.md) §2) +- **G4. Edges are directed and stored once.** Inverse labels are display-only, computed at + read time. B2 never writes a reciprocal link into the target file. + ([data-model.md](data-model.md) §2) +- **G5. A link target that resolves to nothing projects as a surfaced dangling edge, never a + dropped one.** Broken links read as broken (`dst` NULL, the authored text kept) and heal on + the next reindex once the target exists. ([data-model.md](data-model.md) §3, + [GH #12](https://github.com/AlteredCraft/B2/issues/12)) +- **G6. The stored graph is a cache; parsing is the definition of correct.** The `edges` table + exists for what parsing one file cannot serve: backlinks, typed traversal, and discovery's + exclusion. It is rebuilt from scratch on every reindex. In v1, resources are edge *targets* + only; `src` is always a note, because an edge must trace to an authored Markdown line. + ([index-engine.md](index-engine.md) §3, [data-model.md](data-model.md) §10) + +## M. The AI seams and the embedding space + +- **M1. Only enumerated AI seams: `Embedder` and `LlmProvider`.** `b2-core` is model-free and + tested against deterministic fakes; a real model drops in through its seam with no schema or + flow change. `Embedder` (text to vector) carries the index's one recorded identity (M2). + `LlmProvider` (chat: streamed, cancellable) deliberately carries none: nothing it produces + is stored, so swapping chat models never touches the index. Machinery that compensates for a + weak model (per-pair adjudication, query expansion, heavy orchestration) is deferred or off + by default. If a reranker lands, it is the next enumerated seam, not an exception. + ([index-engine.md](index-engine.md) §5, §6, + [GH #151](https://github.com/AlteredCraft/B2/issues/151)/[#153](https://github.com/AlteredCraft/B2/issues/153)) +- **M2. The embedding space has one recorded identity: `meta.(embed_model_id, embed_dim)`, and + the compute device folds into it.** A Metal build tags the id `@metal`. Any identity change + is a model swap: `search` fails fast rather than mixing spaces, `reindex` drops the vectors + and re-embeds, and `open` never mutates the vector space. + ([index-engine.md](index-engine.md) §6, + [GH #40](https://github.com/AlteredCraft/B2/issues/40)) +- **M3. One embedding space in v1.** Every vault member funnels to *text* through the same + model. Multimodal spaces and describers are documented future seams, off by default + ([GH #110](https://github.com/AlteredCraft/B2/issues/110)). + ([data-model.md](data-model.md) §10) +- **M4. Vectors live in plain tables, scored in-process; their existence is the signal; and + they are keyed by the hash of what was embedded.** The vector tables are created at embed + time, so "the tables exist" means "this vault has an embedding space". The fallbacks key on + that (BM25-only search on a projected-but-unembedded vault). `embeddings` is + content-addressed (`text_hash → vector`): the embed input is exactly the chunk's stored + text, so identical text has one vector, a moved note re-embeds nothing, and the only + invalidation rule is "a hash no chunk references is garbage", pruned by the whole-vault + pass. Centroids are the same derived data keyed by note path: refreshed by the embed pass, + dropped on a re-chunk. Model identity is not part of the key because it does not need to be: + a swap drops the whole table (M2). ([index-engine.md](index-engine.md) §3, §4, + [GH #38](https://github.com/AlteredCraft/B2/issues/38)/[#170](https://github.com/AlteredCraft/B2/issues/170)) +- **M5. Note content is never sent off your machine unbidden.** A cloud model endpoint exists + only by your explicit configuration. The default chat endpoint is local, and a chat request + carries your question *and* retrieved note passages. So the consent moment is the + configuration moment, explained in place (plain privacy copy beside the cloud-models + setting, never a later popup). ([GH #151](https://github.com/AlteredCraft/B2/issues/151)) + +## D. Surfacing and disclosure + +- **D1. Discovery answers a relative question, the default view answers a quality one, and no + anchor-local statistic ever makes a candidate unreachable.** The rules: + - "What in my vault belongs next to this note?" is relative. So `b2 similar` and the + discovery pane rank by best-passage distance, and the full ranked list stays reachable. + `limit` is a cap; it under-fills only when there are not enough scorable notes. + - An empty surface must never claim "nothing relates" from anchor-local statistics. Such a + test cannot tell *nothing is related* from *everything is related*: the same geometry read + from opposite ends. A single-domain vault went dark on 16 of 17 notes this way + ([GH #196](https://github.com/AlteredCraft/B2/issues/196)). + - What the default view *shows* is a claim of quality, and always filling to `limit` makes + that claim falsely. A pane that always finds ten trains you to distrust all ten (measured + in real-vault dogfooding, 2026-08). + - A quality signal may therefore set a default *disclosure boundary*: a fold that is a + prefix of the ranked order, with everything below it collapsed but one gesture away. A + signal that would admit rank 5 while folding rank 2 is inadmissible: row order, band, and + fold must never visibly disagree. + - Every such signal is evidence-gated. It must win the measured bake-off on the orthogonal + corpus, on the dense single-domain fixture (where a non-empty default view is absolute), + and on real vaults via `make calibrate`. "No fold at all" is an admissible winner. + - Every such signal must be continuous in population size: a threshold may move banding or + the fold, never which rows exist or can be reached. + - Strength stays a within-list grading painted from the z-score, and it gates nothing. + + The first bake-off found no admissible fold + ([GH #200](https://github.com/AlteredCraft/B2/issues/200), 2026-08-22), so the default view + is still the whole served prefix. The permission above stands for the next candidate; the + measurement retired one rule, not the axis. The harness keeps #200's structural-zero + tripwire, which re-arms on its own if a fold ever ships. + ([index-engine.md](index-engine.md) §3, GH #196/#197/#200/#202) +- **D2. A served search result is a claim of evidence, and `limit` is a quota nowhere in + B2.** The vector half always has k nearest (*nearest* is a fact about the vault, never + evidence about the query), and RRF fuses ranks, discarding the absolute signals. Left alone, + the pipeline cannot answer zero: a nonsense query would serve `limit` confident-looking + results. The rules: + - A result in the default view must trace to positive evidence: a lexical match, or semantic + closeness clearing a bar calibrated per model in the eval harness. A query the vault holds + no evidence for answers **"no matches"**, honestly empty. The nearest-by-meaning list is + never presented as matches. + - The rule over the signals is *lexical OR semantic*: two independent signals, so the test + can tell "nothing matches" from "everything matches" where one signal could not (D1's own + lesson). The engine carries both: `hybrid_search` returns the fused order untouched, plus + each hit's rank in each list and its own distance, plus the query's best cosine + ([GH #201](https://github.com/AlteredCraft/B2/issues/201)). `Vault::search_evidence` reads + the lexical half beside it and serves exactly the rows `search` does, in the same order. + - The lexical half is IDF-weighted term coverage: how much of the query's own weight the + vault carries. A word found in most chunks weighs almost nothing; a word found in none + weighs the most. So a stopword is a measurement, not a word list. Its first form was a + hard df ceiling, and that form failed the transfer check on the dense fixture (it called + `drone` and `comb` stopwords in a beekeeping vault). The lesson is kept: a constant read + off one corpus describes that corpus. The fix was to change the *rule*, not re-tune the + number. + - Any bar is a distributional constant: it is earned against labelled negative queries + before it ships, and it must pass the real-vault transfer check (`make calibrate`, process + rule 5 in [evals.md](evals.md)). A labelled relevant query the bar would cut is the + search-side tripwire, asserted at zero with no headroom. + - The verdict is three-state, and each state is a different behavior + ([GH #202](https://github.com/AlteredCraft/B2/issues/202)): evidence found → serve as + always. No evidence → the honest empty state and *none* of the rows. This is strict: no + reveal, no `--all`, no expander, because any of those would put the nearest list forward + as candidates after all. The cost is accepted because it is bounded to one query at a + time, never a whole vault. No calibrated bar for the active model → serve as always, never + "no matches"; that third state is what the fake embedder and every unmeasured model + produce, and folding it into "no evidence" would blank a dev vault. + - `b2 search --json` is therefore an object: the rows plus the verdict. This is a documented + break of the array contract, because a query-level reading has nowhere to live in a list + of rows. It keeps serving the rows at `vouched: false` where the human surfaces show none: + an agent handed rows *plus* an explicit verdict can be honest about them where a reader + given rows alone cannot. + - `search_evidence_excluding` (`b2 search --exclude`) is the same read minus a caller-named + set of notes, for agent follow-up loops. The subtraction is the caller's, never the + verdict's: the verdict and its signals still read the whole vault, and a heavily excluded + query may honestly under-fill. + - The per-hit *tail* fold (cutting where a real query's evidence runs out) is unshipped by + measurement ([GH #206](https://github.com/AlteredCraft/B2/issues/206)): the fused order is + not an evidence order, so no admissible prefix cut reaches more than 23 of the 367 filler + rows an oracle fold would cut. The tail complaint is an *ordering* problem, and its + payment is the reranker seam M1 reserves. The bake-off re-arms every run. + + ([index-engine.md](index-engine.md) §4, GH #201/#202/#206, [evals.md](evals.md)) + +## E. Engineering discipline: what keeps the above true + +- **E1. The core is deterministic.** No wall clock and no randomness inside `b2-core`. + Timestamps are passed in (the `created` params), and nothing is minted at all: identity is + the path, so the id generator is gone rather than merely injected. Clocks and log + subscribers live in the adapters. ([CLAUDE.md](../CLAUDE.md), Conventions) +- **E2. `cargo test` is fast, deterministic, and model-free; model quality never enters CI.** + Real-model work lives behind `b2 init` and the out-of-CI eval harness. `#[ignore]` is + forbidden: a hard-to-write test is a signal to re-anchor on the invariant or fix the system. + ([evals.md](evals.md)) +- **E3. The `Vault` façade is the one typed API, and every adapter is dumb.** CLI and desktop + commands are deserialize, one façade call, serialize. Logic that wants to live in an adapter + belongs behind the façade. Dependencies point one way (adapters to core, never back). Façade + operations are added on need, never pre-built. + ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md)) +- **E4. User-facing errors are generic and actionable, never leaking internals.** Full detail + goes to logs and `B2_DEBUG`, not to the terminal or the webview. + ([CLAUDE.md](../CLAUDE.md), Conventions) +- **E5. Note content is untrusted input; rendering is a trust boundary.** Authorship is not + trust: a `.md` can come from anywhere (a shared vault, a downloaded or web-clipped note), so + B2 treats rendered note content, and model output (the same class of input, M1), as hostile. + Two rules hold together: B2 HTML-escapes every value *it* interpolates into UI chrome, and + the single Markdown-to-HTML render seam (`renderMarkdown`) sanitizes its output before it + reaches the DOM. The webview CSP (`default-src 'self'`, no inline scripts) is a second, + independent layer: defense in depth, never the sole guard. The same posture governs a note's + links: the webview *is* the application, so a note's link never navigates it. A web link + (`http`, `https`, `mailto`) is an OS handoff performed host-side behind a scheme allow-list, + and every other scheme is refused. + ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md), + [GH #77](https://github.com/AlteredCraft/B2/issues/77)) + +## C. Concurrency: many readers, one builder + +- **C1. Any number of processes may hold one vault's index open at once.** Concurrent + *readers* are unrestricted, and a reader is never refused: opening an index already at the + current `schema_version` takes no write lock at all, so a running reindex cannot turn a + `search` into an error. Creating or rebuilding the projection is the one step that must be + atomic and serialized: an `open` observes a complete schema at the current version, or waits + out a bounded budget for the opener building one. Never a partial schema. The no-partial + half is absolute; the waiting half is deliberately not: past the budget, a stuck writer is + reported rather than hung on. "Complete" is checked, not assumed: a current stamp over + missing tables is stale and rebuilt from empty, because surviving rows would look up-to-date + to an incremental reindex and break S3. The same holds for the vector tables (M4). + Concurrent *writers* stay single-in-flight through the `reindex` advisory lock, which + readers never take. ([index-engine.md](index-engine.md) §3, + [GH #111](https://github.com/AlteredCraft/B2/issues/111)/[#114](https://github.com/AlteredCraft/B2/issues/114)) + +## K. Interaction: keyboard-first + +- **K1. B2 is fully operable from the keyboard; the mouse is an accelerator, never a + requirement.** Every action the mouse can take has a keyboard path: a focusable control in a + sensible tab order, or a documented shortcut. This covers the whole desktop surface: the + file tree and open/create/rename/move/delete, search and find-in-note, edit mode (⌘E) and + every in-editor chord, discovery and linking, chat (⌘J), the graph, and each menu and modal + (`Escape` dismisses, `Enter` confirms, focus is trapped while an overlay is open and + restored on close). Focus is always visible and follows platform/ARIA conventions. Three + consequences: + - A chord live in the app is B2's to document, whoever authored it. The macOS menu bar's + accelerators are declared in `b2-desktop/src/menu.rs` rather than inherited from Tauri's + default, so the reference sheet can list them and the conflict gate can see them. + - The chords are yours, not B2's. Every chord B2 dispatches is re-recordable from + Settings → Keyboard and stored as a UI preference (`localStorage`, like the theme; never + vault state, never the index). The one exception is narrow and stated per row + (`Binding.fixed`): the platform's own reflexes, like ⏎/Esc in a text field or ⏎/Space on a + button, which handing out would break what every other app does. + - A rebinding is judged before it is accepted: refused on a same-scope clash or a menu-bar + chord; advised, and allowed, when an inner surface or the editor also binds it. + + The `b2` CLI satisfies this by nature; K1 governs the GUI adapter. + ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md), + [GH #78](https://github.com/AlteredCraft/B2/issues/78)/[#119](https://github.com/AlteredCraft/B2/issues/119)/[#121](https://github.com/AlteredCraft/B2/issues/121)) From c6e460cb624a7d095e8c8e90afee5e9e3b80ead7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 05:25:57 +0000 Subject: [PATCH 2/6] =?UTF-8?q?docs:=20the=20three=20guides=20as=20Markdow?= =?UTF-8?q?n=20=E2=80=94=20quickstart,=20architecture,=20explainer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit quickstart.md replaces quickstart.html and becomes the one home of the command reference and the environment-variable table. architecture.md replaces architecture.html as the orientation tour, linking into the specs instead of restating them. search-and-similarity.md replaces the HTML explainer: same plain-language walk, glossary, and component table, re-authored per the documentation style guide. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CnZh7mztkPnvXJ4GnfrZiE --- docs/architecture.md | 175 +++++++++++++ docs/quickstart.md | 461 ++++++++++++++++++++++++++++++++++ docs/search-and-similarity.md | 348 +++++++++++++++++++++++++ 3 files changed, 984 insertions(+) create mode 100644 docs/architecture.md create mode 100644 docs/quickstart.md create mode 100644 docs/search-and-similarity.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..089af27 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,175 @@ +# Architecture + +How the B2 system is built, for anyone working on the code. Read this to orient yourself: +what the pieces are, how data flows through them, and where each rule is enforced. Then +follow the links into the two specs for the details. + +B2 is a local-first Markdown vault with an AI layer that surfaces semantically similar, +not-yet-linked notes for you to connect. This repo is the index engine plus its two adapters: +the `b2` CLI and a Tauri desktop app. A SQLite store is treated as a disposable projection of +your Markdown, real local AI sits behind two enumerated seams, and both adapters drive the +same typed `Vault` API. The *why* behind each choice is an [ADR](../ADRs/README.md); the +normative spec is [invariants.md](invariants.md), [data-model.md](data-model.md), and +[index-engine.md](index-engine.md). + +## The governing equation + +One equation shapes every decision (ADR-0002; invariants S1 to S5): + +``` +index = projection of (the vault directory) +``` + +Delete `b2.sqlite`, rebuild it from the Markdown, and it must come back identical. Exactly +one thing is durable and un-derivable: your vault, the source of truth for both knowledge and +every committed connection. There is no state anywhere outside your notes. + +- **Tier 1, the vault: source of truth.** Notes plus every committed edge, plain `.md`, fully + usable in Obsidian with no B2. Plus resource peers (PDFs, images, clippings) and the folder + tree itself. Stays pristine: B2's only writes are surgical and in frontmatter; the body is + 100% yours. +- **Tier 2, `b2.sqlite`: disposable cache.** FTS5, plain vector tables scored in-process, and + the typed graph. Holds nothing that can't be rebuilt from the vault. + +A whole vault is one portable folder: the index lives under `/.b2/`, a dot-folder both +Obsidian and B2's own vault scanner ignore. Point B2 at a folder of Markdown; that is the +entire setup. + +## The five crates + +The engine owes nothing to the two adapters above it or the two model crates beside it. The +AI-heavy crates sit behind traits defined in `b2-core` (the `Embedder` and `LlmProvider` +seams), so the engine and its fast suite never import a tensor or an HTTP client, and adding +the desktop UI added one adapter, not new architecture (ADR-0012). + +| Crate | Role | +|---|---| +| `b2-core` | The engine: turns a folder of Markdown into a queryable index and keeps it a pure function of disk. Model-free (no candle, no network) so its suite is fast and deterministic. rusqlite (bundled SQLite + FTS5), blake3, yaml. No vector extension: vectors are plain tables scored in-process (ADR-0006) | +| `b2-embed` | The real embedder: `LocalEmbedder`, candle-backed BERT (bge-base 768-dim default, bge-small 384-dim supported), pure-Rust inference. `b2 init` downloads + verifies into a shared XDG cache. All heavy ML deps live only here (ADR-0020) | +| `b2-llm` | The real chat provider: a hand-rolled sync OpenAI-compatible SSE client over `ureq` behind the `LlmProvider` seam. Ollama by default, any compatible endpoint by config. Owns the stream's quirks as tests. No async runtime anywhere (ADR-0011) | +| `b2-cli` | The `b2` binary. Holds no engine logic: parse args, inject the embedder and chat provider, call the `Vault` façade, print (human, streamed, or `--json` for agents). Funnels every error through `user_message` so nothing internal leaks | +| `b2-desktop` | The Tauri host, the GUI sibling of `b2-cli`, equally logic-free: each `#[tauri::command]` is deserialize, one `Vault` call, serialize, reusing the CLI's `--json` view types as the IPC contract. The `ui/` frontend (Vite + vanilla TS + CodeMirror 6) renders the read → discover → link → edit → chat loop | + +Ground truth: the layering is enforced by where the dependencies live. `b2-core`'s +`Cargo.toml` has no candle and no HTTP client; the whole engine suite runs on deterministic +fakes. The real models are reached only through `b2 init`, a configured chat endpoint, and +the out-of-CI eval harness. + +## The `Vault` façade + +Every engine module is called directly only by the integration tests. The `Vault` façade +(`crates/b2-core/src/vault.rs`) is the single typed entry point, and its only clients are the +two adapters. It owns the connection and the injected seams, and returns display-ready view +structs the CLI prints and the desktop reuses as its IPC contract. Façade operations are +added when a command needs them, never pre-built. The full command surface, with what each +command needs, is the [quickstart's command reference](quickstart.md#command-reference). + +## The four flows + +Each flow is specified in [index-engine.md](index-engine.md); this is the shape of each, and +where it lives. + +**Flow ①: ingest** (`ingest.rs`). `b2 reindex` composes two separately invokable passes: +model-free `project_vault` (notes, resources, chunks, FTS, edges), then `embed_vault` (fill +the missing vectors). Projection runs in two phases so link resolution never depends on file +order. Ingest writes nothing to the vault, is incremental by default, and converges on +exactly what a from-scratch rebuild would produce. A projected-but-unembedded vault is +already keyword-searchable. Details: [index-engine.md §3](index-engine.md). + +**Flow ②: search** (`search.rs`). BM25 keyword search and brute-force vector KNN run over the +same index and fuse with Reciprocal Rank Fusion, resolved from chunks up to notes (ADR-0008). +A served result is a claim of evidence (D2, ADR-0015): a query the vault holds no lexical or +semantic evidence for answers "no matches", and `--json` is an object (rows plus the +`vouched` verdict). On an unembedded vault, results are BM25-only: degraded honestly, never +an error. Details: [index-engine.md §4](index-engine.md). + +**Flow ③: similar, then link** (`discover.rs`, `note.rs`). `b2 similar ` surfaces the +nearest notes you haven't linked yet: a two-stage scan over stored vectors, no model call, no +network. The machine finds candidates; you supply the judgment and the type (ADR-0009). The +ranked list is always served; `limit` is a cap, not a promise (D1, ADR-0014). Committing is +`b2 link`, which appends one typed-link line to the source note's frontmatter and re-projects +it, or a `[[link]]` you write in the body yourself. Details: +[index-engine.md §3 and §4](index-engine.md). + +**Flow ④: grounded chat** (`chat.rs`, `Vault::ask`). `b2 ask`, `b2 chat`, and the desktop's +chat pane stream an answer grounded in your notes: condense, retrieve (flow ② at 10 +passages), assemble the grounded prompt, stream (cancellable at token granularity), cite +(`[n]` markers resolve to path + excerpt). Chat is a reader: nothing model-derived is stored, +and swapping chat models is a config change, never a reindex. Details: +[index-engine.md §6](index-engine.md). + +## The write discipline + +The vault stays yours. A parsed note keeps its raw text verbatim and records only the byte +spans of the frontmatter block; serialization returns those raw bytes. So +`parse → serialize → parse` is byte-identical: unknown keys, comments, odd whitespace, a +missing final newline all survive (W5). Against that backdrop, every byte B2 writes is the +mechanics of a command you invoked (W1, ADR-0004). The complete list of on-command writes is +[invariants.md W3](invariants.md). + +`b2 mv` never breaks the graph. A note's identity is its path, so a move is a re-key, not a +rebuild: children reference `notes(path)` with `ON UPDATE CASCADE`, so one statement carries +chunks, vectors, and edges across, and inbound link text is repaired in exactly the files the +materialized graph names. Move a note outside B2 and the bargain is the honest one: the links +dangle, visibly, until you repair them, and content-addressed vectors make the +delete-plus-create re-embed nothing. Details: [index-engine.md §8](index-engine.md). + +## The AI seams + +Every AI part sits behind a swappable trait defined in `b2-core` (ADR-0005; M1). The engine +is built and tested against deterministic fakes; a real model drops in through the identical +seam with no schema or flow change. There are exactly two seams, and they differ on the one +axis that matters: what they are allowed to store. + +- **`Embedder` carries the index's identity.** `meta` records + `(embed_model_id, embed_dim)`, device tag included; on a change, the vector tables are + dropped and a full re-embed follows on `reindex`. Opening a vault never mutates the vector + space; a stale `search` fails fast with `ModelMismatch` (M2, ADR-0007). +- **`LlmProvider` deliberately carries none.** Chat stores nothing, so swapping chat models + never touches the index. Chat config is adapter-level (flags/env on the CLI, Settings on + the desktop, resolved once in `b2_llm::LlmConfig`), never vault or index state. + +The relation vocabulary rides beside them: a closed three-verb stance core (`references`, +`supports`, `contradicts`) is your typing palette on `b2 link --type`, each verb with a +display-only inverse label. Three, not thirty, because stance is the one thing embedding +similarity cannot infer: the vectors already tell you two notes are about the same thing; +whether one backs or fights the other is what only you know. A tail verb you write by hand is +kept verbatim (ADR-0010; [data-model.md §2](data-model.md)). + +## Grounded in the tests + +Nothing here is aspirational. The suite is the executable specification, integration-first: +most tests open a real SQLite database, ingest a real vault on disk, and assert on the +resulting projection. Most share the golden vault fixture (`fixtures/golden-vault/`), copied +into a temp dir before every run; the copy plus CI's tree-clean assertion is what stands +behind "no unbidden writes". + +- The engine suite (`cargo test -p b2-core`, the bulk of the weight): one integration file + per property, named for it (`roundtrip`, `graph`, `mv`, `discover_surfacing`, …). +- The CLI end to end (`cli.rs`): every command through the spawned real binary, human and + `--json`, exit codes included. +- The adapters stay dumb: `b2-desktop`'s suite pins commands, errors, fs-watch, and the menu; + `b2-embed` pins config and provisioning; `b2-llm` pins the SSE wire shape; the `ui/` + pure-logic suite runs under node's own test runner. + +The two gates are `make check` (fast, the working loop) and `make ci` (verbatim what CI runs, +ADR-0018). Nothing is `#[ignore]`d: a check that genuinely needs the real model lives in the +eval harness, which runs on demand and therefore actually runs. Model quality never flakes CI +(ADR-0013); how it *is* measured is [evals.md](evals.md). + +## Not yet built + +What remains is tuning, scale, and packaging, tracked in +[GitHub Issues](https://github.com/AlteredCraft/B2/issues): + +- Semantic quality in CI: never. The engine suite proves plumbing on the fake embedder; the + real model is measured by the out-of-CI harness ([evals.md](evals.md)). +- A cross-encoder reranker is the likely next seam: post-fusion, it changes ordering, not the + store, gated on the eval like everything else. Query expansion sits behind it in priority. +- Resource content search: resources are inventoried and are graph targets today; chunking + and embedding them is designed, not shipped ([data-model.md §10](data-model.md)). +- Packaging and distribution: B2 ships as source today. + +Source of truth for every claim: `crates/*/src/` and the test suite under each crate's +`tests/`. The spec is this folder; the *why* is [ADRs/](../ADRs/README.md); the backlog is +GitHub Issues. diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..9965908 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,461 @@ +# Quick start + +Set up B2 and work with a vault, for anyone new to it. You will build the `b2` CLI, point it +at a folder of Markdown, then search it, surface similar notes, and link them. It takes about +ten minutes, most of it a one-time model download. The last section opens the same vault in +the desktop app. + +Your `.md` files stay plain and yours the whole way. B2's whole write surface is a hidden +`.b2/` index folder plus one line of frontmatter (a `b2_relations:` entry, and only when you +commit a typed link). Indexing writes nothing to your notes. Your prose is never rewritten. + +## Before you start + +You need a Rust toolchain ([rustup.rs](https://rustup.rs)) to build the `b2` binary. B2 ships +as source today, as one static binary. macOS and Linux are the tested platforms. For the +desktop app (its own section near the end), also install Node + npm and the Tauri CLI. Run +`make doctor` in the checkout: it checks all of this and prints the fix for anything missing. + +One thing is optional, and it is local. B2 makes no network calls and needs no account or API +key at any point: + +- **The embedding model.** A one-time download (`b2 init`) of a local model that powers + semantic search and connection discovery. Two are supported, and you can switch later. + Pick **BGE Base** (768-dim, ~440 MB), the default and the better ranker. Pick **BGE Small** + (384-dim, ~130 MB) if you want faster downloads and embedding for a modest quality drop, + worth it on a big vault. Or skip the model entirely and run keyword-only (shown below). + +## Set up + +### 1. Build the b2 CLI + +Clone the repo, build the release binary, and alias it for this shell: + +```console +$ git clone https://github.com/AlteredCraft/B2 && cd B2 +$ cargo build --release -p b2-cli +$ alias b2=./target/release/b2 +$ b2 --help +``` + +You should see the command list, starting with "B2 — explore a Markdown vault's typed graph +and search from the terminal". + +Tip: every command takes a global `-C ` (or `--vault`) to point at your vault. Or set +`B2_VAULT_PATH` once so every command finds it without the flag. Add `--json` for +machine-readable output (for agents and scripts). The examples below run from inside the +vault folder, so read-only commands default to the current directory. Commands that write +(`reindex`, `add`, `mv`, `link`) need an explicit vault (a path, `-C`, or `B2_VAULT_PATH`), +so they can't silently touch the wrong directory. + +### 2. Install the embedding model (one time) + +Semantic search and connection discovery run on a local embedding model. No account, no API +key, no network at query time. `b2 init` downloads and verifies the model into a shared cache +once. It is not re-downloaded per vault, and there is never a surprise download mid-command +later. + +```console +$ b2 init +``` + +You should see the download steps, then: `Installed 'BAAI/bge-base-en-v1.5' (768 dims). Run +\`b2 reindex\` to embed your vault.` + +`b2 init` provisions whichever model is configured: the 768-dim base model unless you said +otherwise. To take the smaller, faster one instead, set it once before running `init` (see +[Config and environment](#config-and-environment)), or pick it in the desktop app's +Settings, which offers the same two and downloads on the spot: + +```toml +# /b2/config.toml +[embedder] +model = "BAAI/bge-small-en-v1.5" # 384-dim, ~130 MB +``` + +Switching later: both front ends read that one config, so they always agree on the model. +Changing it is a model swap. The vectors are rebuilt on the next `reindex`, and until then +`search` refuses rather than mixing two embedding spaces. Your Markdown is untouched by any +of it. + +No model? You can do everything in this guide offline by prefixing commands with +`B2_EMBEDDER=fake`. Search still works on keywords (BM25), but the semantic half is off. The +CLI tells you so, and never pretends a result is semantic when it isn't. + +## Create a vault + +A vault is nothing more than a folder of Markdown files. Point B2 at notes you already have, +or start fresh. There is no import step and no format to convert into. + +### 3. Create some notes, or choose a folder you already have + +Already have one? An Obsidian vault, a docs tree, a directory of daily notes: that's a vault. +Point B2 at it and skip to step 5. Nothing is converted, moved, or rewritten. + +Starting fresh? Just write Markdown files. Frontmatter is supported, not required. B2 reads +it when it's there, preserves every key it doesn't understand, and indexes a note with none +at all just as happily. Here is one of each: + +```markdown +# ~/vault/concepts/memory.md — with frontmatter +--- +type: concept +created: 2026-07-03 +tags: [cognition] +--- +The brain encodes, stores, and retrieves information. +``` + +```markdown +# ~/vault/notes/spaced-repetition.md — without any +Spaced repetition exploits the retrieval curve of [[concepts/memory]]. + +Expanding review intervals beat massed practice for long-term recall. +``` + +That `[[concepts/memory]]` is an ordinary Obsidian-style wikilink, and B2 reads it as a +connection between the two notes. There is no B2 syntax for the body: nothing to learn here, +and nothing left behind to un-learn if you stop using B2. Connections that carry a stance +("this one supports that one") do exist, but they live in frontmatter, and you'll meet them +in step 8. + +On titles: a note's display title is its filename (`spaced-repetition`), so +`concepts/memory.md` shows up as `memory` below. A frontmatter `title:` is inert to B2, kept +verbatim for whatever tool of yours does use it. + +Prefer a command? `b2 add notes/spaced-repetition --content "…"` writes a valid note and +indexes it in one step (so you can skip the reindex in step 5 for notes you create this way). +The `.md` extension is optional; parent folders are created for you. + +### 4. Know what B2 will and won't touch + +The files stay ordinary Markdown that works in Obsidian, or anything else, with no B2 +running. Indexing (next step) changes nothing at all in them: the second note above has no +frontmatter, and after a full reindex it still has none. A note that already has frontmatter +keeps every key exactly as you wrote it: same bytes, same order, same comments. B2 knows a +note by where it sits (its path in the vault is its identity), so there is nothing to stamp +into the file. + +B2 writes to your notes in exactly one place, in frontmatter, and never touches your prose: + +- `b2_relations:`, a typed connection, appended only when you commit one (step 8, or a + click in the desktop app). + +That is the whole of B2's write surface, plus the disposable `.b2/` index folder. Links you +write yourself in the body stay ordinary Markdown: B2 reads them, never edits them, and +treats each as an untyped `references` edge (labelled `inline`). A typed relation is a +frontmatter-only thing (labelled `frontmatter`). + +### 5. Index the vault + +`b2 reindex` reads every note and builds the searchable index. Reading only; your notes are +not written to. It's how B2 catches up to files you created or edited by hand: + +```console +$ cd ~/vault +$ b2 reindex . +Indexing /Users/you/vault + embedding 2/2 · notes/spaced-repetition.md (2 chunks) +Indexed 2 notes (2 embedded) and 0 resources +``` + +Run it again after editing notes. It's incremental: unchanged notes keep their vectors, so +only what actually changed is re-embedded. Two flags help: + +- `b2 reindex --dry-run` previews how much work a run would be, touching neither your notes + nor the index. +- `b2 reindex --force` re-projects everything from scratch. It still re-embeds only what + genuinely differs: identical text always produces the identical vector, so B2 keeps the + one it has. + +`b2 status` answers "is it done?" at any point: how many notes are embedded (so semantic +ranking is live rather than keyword-only), and whether a reindex is running. + +Where it lives: the index goes in a single `.b2/` folder inside your vault. It's disposable. +Delete it and the next reindex rebuilds it identical from your Markdown. Add `.b2/` to your +`.gitignore` if the vault is a git repo. + +In the app: the desktop app does this step for you. It indexes the vault when you open it and +watches the folder for changes made outside the app, so there is no reindex to remember. The +CLI keeps it explicit on purpose: a command you run is a command that can't surprise you in a +script. + +## Work with it + +### 6. Search + +One command, hybrid ranking. It fuses keyword (BM25) and semantic (vector) matches, so you +find notes by wording and by meaning: + +```console +$ b2 search "how does forgetting work" +0.0328 spaced-repetition (notes/spaced-repetition.md) + Spaced repetition exploits the retrieval curve of [[concepts/memory]]. +0.0161 memory (concepts/memory.md) + The brain encodes, stores, and retrieves information. +``` + +Use `--limit N` to widen or narrow the result set. It's a cap, never a quota to fill. A query +your vault holds no evidence for gets a plain `no matches` rather than its nearest guesses: + +```console +$ b2 search "Fasdfadsf" +No matches. Nothing in the vault matches "Fasdfadsf". +``` + +B2 decides that from two independent readings: whether your words appear at all (weighted by +how rare each one is), and how near the closest passage genuinely is. That is how it can tell +"nothing here matches" from "everything here matches". The plain-language walk-through is +[search-and-similarity.md](search-and-similarity.md). Under `--json` the answer is an object, +`results` plus the `vouched` verdict, so an agent gets the rows and the honest reading of +them. + +### 7. Explore the graph + +Follow the connections around any note. `neighbors` is a quick list of what a note links to +and from. `explain` adds each connection's provenance, and its "why" when it has one: + +```console +$ b2 neighbors notes/spaced-repetition +→ references memory (concepts/memory.md) + +$ b2 explain concepts/memory +memory (concepts/memory.md) +Connections: + ← referenced-by spaced-repetition (notes/spaced-repetition.md) [inline] +``` + +That is the body wikilink, read back as a graph edge: untyped (`references`), sourced from +the body (`inline`), and shown from the far end under its inverse label (`referenced-by`). +The edge is stored once and directed; the inverse is display only. Step 8 adds a typed one, +and this listing grows a second row. + +You address a note by its vault-relative path, with or without the `.md`: the same two forms +a link is written in. When nothing points at a note, `explain` flags it as an orphan. +Surfaced for you to notice, never auto-changed. + +### 8. Discover connections + +This is the point of B2. `b2 similar ` surfaces the notes most semantically similar to +a given one that you haven't linked yet. It is a pure, instant read over the vectors +`reindex` already stored: no model call, no network, no cost. + +```console +$ b2 similar notes/spaced-repetition +0.7132 cramming (notes/cramming.md) + Massed practice yields only short-term recall. +0.6318 forgetting-curve (concepts/forgetting-curve.md) + Retention decays exponentially without review. +# stderr: Commit one with: b2 link notes/spaced-repetition --type +``` + +Each row is a similarity score, the note's name and path, and the passage that made it +similar. The note itself and anything already linked to it never appear. `--limit N` caps the +list (ten by default). + +Ranked, always: `--limit` is a cap, not a quota. The list is the ranked nearest, and it +under-fills only when the vault genuinely has fewer unlinked, embedded notes than you asked +for. Similarity is relative to your vault: in a single-subject vault everything is somewhat +related, and the top of the list is still the right answer, so B2 serves the ranking and +leaves the judgment to you. An empty list (at any nonzero `--limit`) means only that there +was nothing to compare, not that nothing relates. + +You are the precision gate: B2 finds the candidates; you supply the judgment and the type. +There is no review queue and nothing "inert until accepted". A connection exists only once +you author it. + +Pick the connections worth keeping and commit them. There are two ways, and they are not the +same connection: + +- Write a `[[link]]` in your note's body, in any editor. Cheap, and it reads naturally in + prose. But a body link is always untyped: B2 records it as a plain `references` edge, with + no stance and no "why". The body has no syntax for those. +- Run `b2 link` (or click Link in the desktop app). This writes a typed relation, a stance + verb and optionally a "why", into the source note's frontmatter. + +```console +$ b2 link notes/spaced-repetition notes/cramming --type contradicts --explanation "massed vs. spaced practice" +Linked notes/spaced-repetition.md —contradicts→ notes/cramming.md. Wrote the relation into the source note's frontmatter. +``` + +What changed on disk is one line, in the source note's frontmatter. The body is untouched: + +```yaml +b2_relations: + - "contradicts [[notes/cramming.md]] — massed vs. spaced practice" +``` + +`--type` takes a verb from the closed stance core: `references` (neutral), `supports` (for), +`contradicts` (against). It defaults to `references`. Three, not thirty, because stance is +the one thing embedding similarity cannot infer for you; everything else about the connection +is already in the two notes. `--explanation` records the "why", and `b2 explain` reads it +back: + +```console +$ b2 explain notes/cramming +cramming (notes/cramming.md) +Connections: + ← contradicts spaced-repetition (notes/spaced-repetition.md) [frontmatter] + why: massed vs. spaced practice +``` + +Re-run `b2 similar` afterward and a note you just linked drops off the list: it's connected +now, so the exclusion filters it out. To undo a link, delete its `b2_relations:` entry; it's +gone on the next reindex. There is nothing else to clean up, because the edge only ever lived +in your Markdown. + +### 9. Ask your vault (optional, needs a local model server) + +`b2 ask` streams an answer grounded in your own notes. B2 retrieves the most relevant +passages with the same hybrid search as step 6, hands the model only those, and the answer +cites them by `[n]`. It needs a model server: Ollama by default (`ollama serve`, then +`ollama pull llama3.2`); any OpenAI-compatible endpoint works via `--llm-url`/`--llm-model`. + +```console +$ b2 ask "why does spacing out reviews beat cramming?" +Spaced repetition exploits the retrieval curve [1]: expanding intervals counter the +exponential decay of retention [2], where massed practice yields only short-term recall. + +Sources: + [1] notes/spaced-repetition.md + [2] concepts/forgetting-curve.md +``` + +`b2 chat` is the interactive form: follow-up questions remember the conversation, Ctrl-C +stops an answer mid-stream (the partial text stands), `/exit` leaves. The desktop app has the +same thing as a chat pane (⌘J). + +A reader, not a writer: chat stores nothing. No transcript, no cache, session-only history. +Nothing about a chat is ever written to your notes or the index, and swapping chat models is +a config change, never a reindex. The default endpoint is local; a cloud endpoint exists only +if you configure one. Its key rides `B2_LLM_API_KEY`, or the macOS Keychain in the desktop +app. Never a flag, never a config file. + +## Day to day + +Edit notes in whatever you like. B2 is a layer over your files, not a place you have to live +in. When you rename or reorganize, let B2 keep the graph intact: + +```console +$ b2 mv concepts/memory concepts/human-memory +Moved concepts/memory.md → concepts/human-memory.md +Rewrote 1 inbound link(s) across 1 file(s). +``` + +A move made through B2 never breaks a backlink. It rewrites the now-stale path inside every +note that pointed at the old one (so the Markdown still reads correctly in any other editor) +and re-keys the index in the same breath. Folders move as a whole. Refactor fearlessly: move, +split, merge, rename. After a batch of hand-edits, a quick `b2 reindex` catches everything +up. Or nothing at all, if you're in the desktop app, which is watching the folder anyway. + +Rename a note outside B2 (in Finder, or with `git mv`) and the links pointing at it break, +exactly as they would with Obsidian closed. The difference is that B2 tells you: `b2 explain` +lists each one as unresolved, with the path it was written to, and it heals by itself the +moment a note exists there again. + +## The same vault, in a window + +Everything above has a desktop app counterpart. Not a viewer bolted onto the CLI, but the +other front end over the same engine: same index, same search, same discovery, same rules +about what may be written to your notes. Open the vault you just built and it's all there. + +```console +$ make doctor # checks Node, npm, the Tauri CLI, the platform toolchain +$ B2_VAULT_PATH=~/vault make app +``` + +On first launch with nothing remembered, the window opens with no vault selected. Click the +vault switcher and pick a folder. After that it reopens whatever you had open last, and +`B2_VAULT_PATH` is just a way to skip that first pick. + +What the window adds over the terminal: + +- **You never run reindex.** It indexes the vault on open, and a native fs-watch keeps up + with edits made outside the app (an external editor, a `git pull`), re-projecting and + re-embedding what changed. Projection and embedding are decoupled, so a cold vault is + browsable and keyword-searchable in seconds while the vectors stream in behind. A manual + reindex is still there in Settings → Index, with live progress and a Cancel. +- **Discovery sits next to what you're reading.** The similar-but-unlinked notes are a pane, + not a command you remember to run: always the ranked nearest, with a strength band grading + each candidate within the list where a statistic exists (a vault under a dozen candidates + is served ungraded, and says so), and a one-click typed Link that writes the same + `b2_relations:` line `b2 link` does. There is also a graph view of the connections around + the open note. +- **Grounded chat is a pane (⌘J):** the same cited, streamed answers as `b2 ask`, with Esc to + stop a stream. The model endpoint and its Keychain-held key live in Settings. +- **An editor.** CodeMirror 6 with live-preview Markdown, autosave, syntax-highlighted code + fences, `[[wikilink]]` completion, and a conflict bar if the file changed under you. A + frontmatter drawer edits the block on its own, body untouched. +- **A file tree over the real folders:** create, rename, move, delete, and drag files in from + Finder to import them (a dropped `.md` keeps its own frontmatter; a dropped PDF keeps its + bytes). +- **Settings (⌘,):** pick the embedding model (base or small, downloaded on the spot), see + whether embedding runs on CPU or the Metal GPU, and browse the whole keyboard map, where + every chord is rebindable. + +Why both? The desktop app is a second dumb adapter: each of its actions is deserialize, one +call into the same typed `Vault` façade the CLI calls, render. That is what keeps them +honest. The GUI can't grow its own idea of what "linked" means, a fix in the engine fixes +both, and the app inherits the engine's test suite instead of re-implementing search or +discovery a second time. Use the CLI for scripts, pipes, agents, or SSH; use the app to read +and write. Neither is the "real" one. + +The mental model in one line: `index = projection of (Markdown)`. Your Markdown is the only +source of truth. The index is a disposable cache you can drop and rebuild identically from +your notes. Nothing locks you in. + +## Command reference + +| Command | What it does | Model? | +|---|---|---| +| `b2 init` | Download + verify the configured embedding model (one time, per machine) | downloads it | +| `b2 reindex` | Re-project the vault; incremental. `--dry-run`, `--force`, `--cancel` (stop a run backgrounded with `&`; `b2 status` names its pid) | embeds | +| `b2 status` | Embedding coverage (is semantic ranking live?) and whether a reindex is running | no | +| `b2 search ` | Hybrid keyword + semantic search. `--limit N`; `--exclude ` subtracts already-inspected notes (for agent loops) | embeds query | +| `b2 similar ` | Surface the semantically nearest notes you haven't linked yet: the ranked list, always. `--limit N` | reads vectors | +| `b2 link ` | Commit a typed relation into the source note's frontmatter. `--type` (references/supports/contradicts), `--explanation` | re-embeds note | +| `b2 neighbors ` | List a note's links, in and out; flags ones that resolve to nothing | no | +| `b2 explain ` | Every connection with its provenance and "why"; flags orphans | no | +| `b2 add ` | Create a note (`--content`, and `--title` for a frontmatter `title:`, inert to B2) and index it | embeds | +| `b2 write ` | Replace a note's body from stdin: the scripting/agent editing surface. Frontmatter is left alone | no | +| `b2 mv ` | Move/rename a note, file, or folder and repair every inbound link | re-embeds touched | +| `b2 rm ` | Delete a note, file, or folder from the vault and disk (`-r` for a folder). Inbound links dangle and are surfaced, never rewritten | no | +| `b2 ask ` | One grounded, cited, streamed answer from your notes. `--llm-url`, `--llm-model`; `--json` is a JSONL event stream | embeds query + model server | +| `b2 chat` | Interactive grounded chat; session-only history, nothing stored. Ctrl-C stops an answer; `/exit` leaves | embeds query + model server | + +## Config and environment + +Zero-config is the happy path: everything above works with no config file. When you want to +tune, a single optional TOML in your platform's config dir configures the embedder: +`~/.config/b2/config.toml` on Linux, `~/Library/Application Support/b2/config.toml` on macOS. +Both front ends read this one file, so the CLI and the desktop app can never disagree about +which model built your vectors: + +```toml +# /b2/config.toml (all optional) +[embedder] +model = "BAAI/bge-base-en-v1.5" # default — 768-dim, ~440 MB, the better ranker +# model = "BAAI/bge-small-en-v1.5" # 384-dim, ~130 MB, faster to download and embed +``` + +Those two are the supported models, the same pair the desktop app's Settings → Embedding +picker offers (where switching also downloads the new one for you). After a switch, run +`b2 init` (if you changed it by hand) and `b2 reindex`. The vectors are rebuilt, and until +they are, `search` refuses rather than mixing embedding spaces. + +| Environment variable | Effect | +|---|---| +| `B2_VAULT_PATH` | Vault root, so commands find it without `-C`. An explicit `-C`/`--vault` overrides it. Read-only commands fall back to the current dir; commands that write (`reindex`/`add`/`mv`/`link`) require it explicitly | +| `B2_EMBEDDER=fake` | Offline mode: deterministic non-semantic embedder. Search runs keyword-only | +| `B2_LLM_URL` / `B2_LLM_MODEL` | The OpenAI-compatible chat endpoint + model for `ask`/`chat` (defaults: `http://localhost:11434/v1`, Ollama's, and `llama3.2`). The `--llm-url`/`--llm-model` flags beat the env, which beats the default | +| `B2_LLM_API_KEY` | Bearer token for a cloud chat endpoint. An env var, never a flag, because a key in a flag is a key in `ps`. The desktop stores its key in the macOS Keychain instead | +| `B2_LLM=fake` | The deterministic chat provider: `B2_EMBEDDER=fake`'s sibling for `ask`/`chat` | +| `B2_DEBUG` | Print internal error detail after the generic user-facing message | +| `B2_LOG` | Structured debug logging: JSON Lines on stderr (stdout stays pure data), ready for jq/DuckDB/pandas. Takes a tracing filter (`debug`, `b2::sqlite=debug`, `warn`). Includes per-statement SQLite timings; `B2_DEBUG` or `B2_LOG_FILE` alone implies `B2_LOG=debug` | +| `B2_LOG_FILE` | Write the structured log to this file instead of stderr (append mode, so runs accumulate into one dataset) | +| `B2_SLOW_QUERY_MS` | Slow-query threshold in milliseconds (default 100): statements at or over it log at WARN with `slow=true` | + +Honest about limits: search snippets and scores come from the index, so reindex first. +`similar` reads the vectors a prior reindex stored, so index before you discover. And under +`B2_EMBEDDER=fake` the semantic half of search and similarity is off: great for offline +exploring, not for real recall. diff --git a/docs/search-and-similarity.md b/docs/search-and-similarity.md new file mode 100644 index 0000000..7a886d4 --- /dev/null +++ b/docs/search-and-similarity.md @@ -0,0 +1,348 @@ +# Search and similarity + +How B2 decides what's related, in plain language, for everyone who uses B2. Read this to +understand what search and the related-notes panel are doing, what the strength dots mean, +and what the honest limits are. No math background assumed. Each idea gets its real name the +first time, and the machinery lives in [index-engine.md](index-engine.md) when you want it. + +Everything here runs on your machine. Finding and relating notes uses no network, no API key, +and costs nothing per note or per search. The only heavy step is the one-time `b2 reindex` +that reads your vault. Your notes stay plain Markdown files that you own; B2 writes nothing +to them unless you ask it to. + +## 1. How B2 reads a note + +Before anything can be found or related, every note gets turned into something a computer can +compare. That happens in two steps. + +**Step one: your note is cut into passages.** A long note isn't one idea, it's many. If B2 +treated a 4,000-word article as a single lump, a paragraph deep inside it about one narrow +thing would be drowned out by everything around it. So B2 cuts each note into passages of +roughly 450 tokens' worth of text (about 340 words), preferring to cut at headings, then at +paragraph breaks, and never in the middle of a code block or a table. Consecutive passages +overlap by about 15%, so an idea that straddles a boundary isn't sliced in half. + +> The term: a passage is called a **chunk**. The rules for where to cut are the **chunker**. +> Deep dive: [index-engine.md §1](index-engine.md). + +**Step two: each passage becomes a list of numbers.** B2 feeds every passage to a small +language model that runs locally on your machine. The model reads the passage and outputs 768 +numbers. That list of numbers is the passage's position on a map of meaning: passages about +similar things land near each other, passages about different things land far apart. The +numbers themselves are meaningless to a human; only the distances between them matter. + +> The term: the list of numbers is an **embedding**, or a **vector**. The model is +> `bge-base-en-v1.5`. It runs entirely offline, downloaded once by `b2 init`. "768 numbers" +> is its **dimension**. + +One pass, done once. `b2 reindex` walks your vault, cuts each note into chunks, embeds each +chunk, and files the results three ways: a keyword index (exact words), the vectors +(meaning), and your links (the graph). Everything afterwards, every search and every +related-notes panel, just reads that index back. The index is disposable: delete it, +rebuild, and you get the same thing back. + +## 2. Meaning as a map + +This is the one idea worth internalizing. Everything else in B2 is built on it. + +Picture every passage in your vault as a pin on a map. The model places the pins: passages +about espresso end up in one neighborhood, passages about volcanoes in another. B2 never +"understands" your notes. It only measures how far apart the pins are. Close pins mean the +model thinks the passages are about similar things. + +The real map has 768 directions instead of two, which nobody can picture, but nothing about +how it works depends on that. Distance is distance. + +B2 measures the closeness of two pins as a number between -1 and 1, where 1 means "pointing +the same way". Two notes at 0.79 are strongly alike; two at 0.35 have little to do with each +other. That number is the raw material for everything below. + +> The term: the closeness number is **cosine similarity**. B2 stores and compares it +> internally as a distance. You see it in the app only indirectly, as the strength dots on a +> card. + +Why raw closeness numbers are never shown to you: "0.79" only means something relative to the +rest of your vault. In a vault of tightly related research notes, 0.79 might be unremarkable; +in a scrapbook of unrelated topics it would be a standout. So B2 never grades a relationship +by its raw number. See [how the list is graded](#6-reading-the-strength-dots). + +## 3. Search: two opinions, fused + +When you type a query, B2 asks two completely different systems the same question, then +merges their answers. + +- **Opinion 1: the words.** A classic keyword search: which passages actually contain your + search words? It rewards rare words heavily. If you search "moleskin", the one note + containing it wins outright. It understands nothing; it counts. + > The term: **BM25**, the standard keyword-ranking formula, running on SQLite's full-text + > index. +- **Opinion 2: the meaning.** Your query is turned into a pin on the same map as your + passages, and B2 finds the nearest ones. It never needs to share a single word with the + note it finds; it matches ideas. + > The term: **vector search**, also called semantic search or **KNN** ("k nearest + > neighbors"). + +**Why you need both.** Each is blind where the other sees. Search "how do leaves turn light +into food" and keyword search returns nothing useful: your note says "photosynthesis" and +shares almost no words with the question. Meaning search nails it. Now search for an exact +error code or a person's surname: meaning search returns a vague neighborhood, while keyword +search lands on the one note that literally contains it. + +**The merge uses positions, not scores.** The two systems produce numbers on incomparable +scales, so B2 ignores the scores entirely and merges by rank: each list awards a note points +based on where it placed, and the points are added. A note that both systems like beats a +note only one of them loves. + +> The term: the merge is **Reciprocal Rank Fusion (RRF)**. A note at position r in a list +> scores `1 / (60 + r)`. The 60 is a deliberate damper; without it, a first place would +> dominate everything else. Searching both ways and merging is called **hybrid search**. + +### When nothing matches: search answers zero + +That takes deliberate machinery, because the pipeline above can't do it by itself. The +meaning half always has a nearest neighbor ("nearest" is a fact about your vault, not about +your query), and the merge throws away both systems' actual scores in favor of positions. So +by the time anything could ask "is any of this actually relevant?", the two numbers that +could answer are gone. Left alone, typing `Fasdfadsf` would get ten confident-looking +results. + +So B2 reads two absolute signals beside the merged order, and serves results only if either +one says the vault holds something: + +1. **Do your words appear at all, weighted by how rare they are?** Not "does anything match": + almost any query shares some word with almost any vault, and a query that matches only + through *a*, *to*, and *my* has told you nothing. Each word is weighted by how many + passages contain it, so a word in most of them counts for nearly nothing and a word in + none counts for the most there is. B2 asks what share of your query's own weight the vault + actually carries. Stopwords are therefore measured, never a shipped list of words to + ignore, which matters in a single-subject vault: no fixed list would know that *comb* is a + content word in a beekeeping vault. +2. **Or is the nearest passage genuinely near?** The plain distance, before ranks replaced + it: a backstop for the real questions your notes answer in words you didn't use. + +Two independent signals, deliberately, because one couldn't do it: a single test can't tell +"nothing here is related" from "everything here is related". That is the same lesson the +related-notes panel learned the hard way (section 5). A query that clears neither gets a +plain "no matches", and B2 shows you none of its nearest guesses. Not folded away behind a +"show anyway", not counted, just not offered. The nearest list is genuinely out of reach for +that one query. That is the accepted price of the answer being trustworthy: what B2 shows +you, it vouches for. + +Worth knowing: the bar is measured per embedding model, so a model B2 has no measurement for +yet gets no verdict at all rather than a guessed one. Search behaves exactly as it always did +there, including in offline/dev mode. And `b2 search --json` still hands an agent every row +plus the verdict, even when the answer is "no matches": a program that is told the vault +vouches for nothing can be honest about the rows anyway, where a person handed ten results +cannot. + +### Three smaller heuristics you may notice + +- **Word endings are trimmed.** The keyword index reduces "running", "runs", and "ran" to a + common root so they match each other. The term is **stemming** (B2 uses the Porter + stemmer). The cost is occasional over-merging: *universe* and *university* collapse to the + same root, which is why B2's test suite deliberately includes that trap. +- **Punctuation in your query is neutralized.** Search-syntax characters are stripped before + your words reach the keyword index, so typing a question with apostrophes and quotes can't + break it. +- **Only a working set is ranked.** Each system returns a bounded pool of candidate passages + (for a normal ten-result search, 150 per system; the passage-level view keeps 60) rather + than the entire vault. On a large vault this is what keeps search instant. It also means a + note far outside both pools can't be rescued by fusion. + +## 4. Similarity: the two-stage engine + +The related-notes panel answers a different question from search. Search asks "what matches +this query?" Similarity asks "what in my vault belongs next to this note, that I haven't +already connected?" + +Running it on a note (the **anchor**) happens in two passes, for speed: + +1. **Pass one: a fast shortlist.** Comparing every passage against every other passage would + be slow on a big vault. So B2 first gives each note a single average position (the middle + of all its passages) and uses that to shortlist a few hundred plausible notes. This pass + is about recall: it is deliberately generous, and exists only to avoid scoring the whole + vault. +2. **Pass two: the real comparison.** For each shortlisted note, B2 compares every passage of + the anchor against every passage of the candidate and keeps the single best matching + pair. That best pair is the note's score, and the passage that achieved it is shown to you + on the card as the evidence for why this note appeared. + +> The terms: the note's average position is its **centroid**. The best-pair comparison is +> **max-sim**, or best-passage scoring. Deep dive: [index-engine.md §4](index-engine.md). + +**Why the second pass matters: the buried gem.** An average is a lie about any note that +covers more than one subject. Consider a weekly journal with seven unrelated sections, one of +them a genuinely excellent account of a lava field. Its average position sits in the middle +of nowhere, near nothing. But one passage of it is an outstanding match for your volcano +note. Judge the average and you lose the gem. B2 ranks and grades this note on its best +passage, not its average, so a strong section inside a messy note can still surface, with +that section quoted on the card. This is also why the same journal doesn't get wrongly +suggested for your unrelated notes: its average may drift near them, but no single passage of +it actually matches, so it's cut. + +**Notes you've already linked are removed.** Discovery shows you what you *haven't* +connected. Anything one link away from the anchor is excluded, because you already know about +it. Notes two links away stay in, since a related-but-not-directly-linked note is exactly the +connection worth finding. + +**B2 suggests; you decide.** Nothing in this panel changes your notes. A connection exists +only when you author it, with `b2 link` or by dragging a card into your note in the app. +There is no suggestion queue and no pending state: you are the precision gate. + +## 5. The list is always served, and graded, not gated + +"What in my vault belongs next to this note?" is a relative question, and the ranked list +answers it. B2 always shows you the nearest unlinked notes. The strength dots, not an empty +panel, carry the quality signal. + +Why no statistical rule for staying quiet? Because any such rule has to read the anchor's own +candidates, and that reading is ambiguous in exactly the case that matters. "Stands out from +the background" quietly assumes most of your vault is unrelated to any given note. A vault +where everything shares one subject (a research vault, a project vault, a single-subject +zettelkasten) breaks that assumption by construction: the background is itself related, +nothing can stand out from it, and a gate reads the vault with the *most* to connect as +having nothing. An anchor-local statistic cannot tell "nothing is related" from "everything +is related", so B2 lets none claim to. (That is invariant D1; the measurements that settled +it are recorded in +[ADR-0014](../ADRs/0014-discovery-always-serves-the-ranked-prefix.md).) + +So the rule is simple: the ranked list is served, and you are the judge. The panel is empty +only when there is genuinely nothing to compare (no unlinked note with stored vectors yet), +and it says exactly that, never "nothing relates". The per-candidate statistic survives as +the input to the strength dots: a within-list grading, so a research vault reads "here are +your nearest, all middling" instead of a dark panel. + +Could a smarter "show nothing" rule ever ship? Only by earning it. Any candidate has to win a +measured bake-off: on the standard test corpus, on a deliberately single-domain one built for +exactly this failure, and on real vaults via a calibration command. "No rule at all" is +allowed to win it, and has, so far. What ships today is also what every comparable tool +ships: a ranked list, with the score as a signal rather than a verdict. + +Search is the other side of that same coin, and it goes the other way. Nothing here +contradicts section 3's "no matches": *related to this note* is a comparison with no zero +point (everything in your vault is somewhat related to everything), while *relevant to this +query* has an honest zero, and B2 can measure it in two independent ways. Where a claim can +be checked, B2 makes it; where it can't, B2 hands you the ranking and gets out of the way. + +## 6. Reading the strength dots + +Each card in the app carries three dots. They are that candidate's z-score, banded. Not a raw +similarity, and not a percentage. + +- **●●● strong match.** 2.52σ and above: in the top quartile of relationships a human has + confirmed as real on the test corpus. +- **●●○ clear match.** 1.96σ and above: where the test corpus's confirmed relationships + typically lead their lists. +- **●○○ near match.** Below 1.96σ: near in this list's own terms; worth a skeptical look. + +Because the dots and the ordering of the list are read from the same number, they can never +disagree with each other. A card higher in the list never has fewer dots than one below it. +The dots grade; they never decide what appears. The list itself is always the ranked nearest. + +The dots are relative to the note you're on. "●●●" means strong compared with *this note's +other candidates*, not "95% similar". The same pair of notes can legitimately show different +dots from each side, because each side has different competition. A known consequence: in a +tightly single-subject vault, where every candidate is close, the dots can read uniformly +modest. See the open questions below. + +> The term: a **z-score** ("σ", sigma) says how far above the typical value something sits, +> measured in units of the normal spread. In B2 it grades the dots, within one note's list, +> and decides nothing about what appears. + +## 7. Known limits and honest caveats + +- **Open question: in a tightly single-subject vault, the strength dots can read uniformly + modest.** The panel always serves the ranked list (section 5); what remains open is the + grading. The dots compare each candidate with the note's other candidates, and in a vault + where every candidate is close, that comparison compresses: genuinely strong relationships + can all paint one or two dots. The list order is unaffected and correct; only the dots lose + resolution. Measuring that compression on real vaults (`make calibrate` exists for exactly + this), and deciding what the dots should reference instead, is the named next step. +- **A relation the model simply gets wrong can't be rescued by ranking.** If the model scores + one unrelated pair unusually highly, that pair sits high in the list by definition. One + such case is a known, tracked residue in B2's own test corpus: now a mis-ordering (the real + relation appears, a few places lower than it should) rather than an absence. +- **Whether any "show nothing" rule ever ships in the related-notes panel is an open, + evidence-gated question.** Every candidate must win the measured bake-off section 5 + describes, "no rule at all" is an admissible winner, and no candidate has yet beaten it. + This is the related-notes panel only: search *does* answer "no matches", because a query + has an honest zero where a note-to-note comparison does not. +- **Quality depends on your reindex being current.** Discovery reads stored vectors and never + re-embeds, so notes edited since the last index are compared on their old text. The desktop + app re-indexes automatically on changes; the CLI does it when you run `b2 reindex`. +- **An index built with the fake embedder is meaningless.** If `B2_EMBEDDER=fake` was set + (offline/dev mode), vectors are content hashes, not semantics. B2 detects this and serves + the list ungraded rather than pretending to measure strength over noise. +- **Only Markdown is read for meaning.** Other files in your vault are tracked as resources + but have no passages or vectors yet, so they never appear as similar notes. + +## 8. Every component and knob, in one table + +What is actually in the pipeline, what each part decides, and the value it ships with. + +| Component | What it decides | Ships as | Deeper dive | +|---|---|---|---| +| Chunker | Where a note is cut into passages; prefers headings, then paragraph breaks; never splits code or tables | ~450 tokens, 15% overlap | [index-engine.md §1](index-engine.md) | +| Embedding model | Turns a passage into its position on the meaning map | `bge-base-en-v1.5`, 768 dimensions, local | [architecture.md](architecture.md) | +| Keyword index | Which passages contain your words; rare words count for more | SQLite FTS5, BM25, Porter stemming | [index-engine.md §4](index-engine.md) | +| Vector scan | Which passages are nearest in meaning | Exact comparison, in-process, no extension | [index-engine.md §4](index-engine.md) | +| Fusion | How the two rankings become one | RRF with a damping constant of 60; ties break toward the meaning signal | [index-engine.md §4](index-engine.md) | +| Candidate pools | How deep each signal looks before fusing | Per signal, at a 10-result search: 150 passages (note view), 60 (passage view) | [index-engine.md §4](index-engine.md) | +| Shortlist (pass 1) | Which notes are worth comparing properly | 20× the requested count, at least 200 notes | [index-engine.md §4](index-engine.md) | +| Best-passage score (pass 2) | How related two notes actually are, and which passage proves it | Best pair across all passages of both | [index-engine.md §4](index-engine.md) | +| Link exclusion | Which candidates are hidden as already known | Anything 1 link away from the anchor | [index-engine.md §4](index-engine.md) | +| Surfacing rule | What the related-notes panel shows | The ranked list, always; no statistical gate (D1, ADR-0014) | Section 5 | +| Strength bands | How many dots a card gets (grading only, never what appears) | ●●● ≥ 2.52σ, ●●○ ≥ 1.96σ, ●○○ below; ungraded under 12 candidates | Section 6 | +| Grounded chat | Which passages an answer may cite | Top 10 passages from the same hybrid search | [index-engine.md §6](index-engine.md) | + +None of these are guesses. Every value above was chosen by measuring it against a +hand-labelled corpus of notes and expected answers, and several were chosen by rejecting a +change that looked better on paper. That test harness, its rules, and the record of every +verdict live in [evals.md](evals.md). + +## 9. Glossary + +- **Anchor.** The note you're currently looking at: the one whose related notes are being + found. +- **BM25.** The standard formula for keyword ranking. Scores a passage on how many of your + search words it contains, weighting rare words far more heavily than common ones. Knows + nothing about meaning. +- **Centroid** ("the note's average position"). One position representing a whole note, made + by averaging all its passages. Fast to compare, but misleading for a note covering several + subjects, which is why B2 uses it only to build a shortlist, never to grade a result. +- **Chunk** ("passage"). A slice of a note, roughly 450 tokens, cut at a heading or paragraph + break. Everything B2 compares is a chunk, not a whole note. +- **Cosine similarity.** The closeness of two positions on the meaning map, from -1 to 1. 1 + means "identical direction". B2 computes it but never shows it to you raw, because its + meaning depends entirely on the vault around it. +- **Embedding** ("vector"). The list of 768 numbers a model produces for a passage: its + position on the map of meaning. Passages about similar things get nearby positions. +- **Embedding model.** The local AI model that produces embeddings. B2 ships with + `bge-base-en-v1.5`, downloaded once by `b2 init` and run on your own machine. +- **Hybrid search.** Running keyword and meaning search together and merging the results: + B2's default, because each covers the other's blind spot. +- **KNN** ("k nearest neighbors"). Finding the k closest positions to a given one. What + "search by meaning" and "related notes" both do underneath. +- **Max-sim** ("best-passage scoring"). Scoring two notes by their single best-matching pair + of passages, rather than by their averages. Lets one strong section inside a messy note + count for something. +- **Reciprocal Rank Fusion (RRF).** The method for merging two ranked lists using each item's + position rather than its score. Necessary because keyword scores and distance scores aren't + on comparable scales. +- **Stemming.** Trimming words to a common root in the keyword index so "running" matches + "run". Improves recall; occasionally over-merges (*universe* / *university*). +- **z-score** ("σ", "sigma"). How far above the typical value something sits, measured in + units of the normal spread. "2.5σ" means "well above this note's ordinary candidates". In + B2 it grades the strength dots, within one note's list, and decides nothing about what + appears. + +## 10. Deeper dives + +- [index-engine.md](index-engine.md): the whole engine spec. Chunking and indexing (§1, §3), + both retrieval flows stage by stage (§4), the seams (§6). +- [architecture.md](architecture.md): the whole system, the crates, and where each flow + lives. +- [evals.md](evals.md): how every number on this page was measured, and the verdicts. +- [invariants.md](invariants.md), [data-model.md](data-model.md): the spec the code is a + projection of. +- [quickstart.md](quickstart.md): install, index a vault, and run your first search. From de43254835dcfbf15af4cf1e68691f6626a1eb6e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 05:29:04 +0000 Subject: [PATCH 3/6] docs: the eval-suite guide moves to docs/evals.md; docs/README.md is the map The one guide to the harness (instruments, report blocks, exit gate, verdict record, process rules) now lives with the rest of the docs, its links re-pointed at the corpus files it governs. A short pointer README stays beside the corpus so anyone browsing evals/ still lands on the rules before editing. docs/README.md is the new docs home: one table naming the single home of each topic. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CnZh7mztkPnvXJ4GnfrZiE --- crates/b2-embed/evals/README.md | 444 +------------------------------ docs/README.md | 32 +++ docs/evals.md | 452 ++++++++++++++++++++++++++++++++ 3 files changed, 488 insertions(+), 440 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/evals.md diff --git a/crates/b2-embed/evals/README.md b/crates/b2-embed/evals/README.md index 217dba8..2e134c8 100644 --- a/crates/b2-embed/evals/README.md +++ b/crates/b2-embed/evals/README.md @@ -1,441 +1,5 @@ -# The eval suite — the guide +# The eval suite -How B2 measures the one thing `cargo test` cannot: whether retrieval, discovery, and grounded -chat are any *good*. A unit test can prove `reindex` is idempotent; only a human-labelled -corpus can say that "how do leaves turn light into food" should rank `photosynthesis.md` -first. This is the one guide to the whole suite: every instrument, how to run it, **how to -read what it prints**, and the process rules that bind anyone editing the corpora, the -labels, or the metrics. - -Everything here runs **out of CI, on demand** (ADR-0013): `cargo test` stays fast, -deterministic, and model-free, so model quality can never flake CI. **Decision history lives -in git and in [GitHub Issues](https://github.com/AlteredCraft/B2/issues)** — every verdict in -the [record below](#the-verdict-record) names the issue that drove it, and the commit that -shipped it is the record of what changed and why. - -> **Not the audience for this page?** [**Search & similarity — the -> explainer**](../../../docs/search-and-similarity.html) is the plain-language tour of -> everything these metrics score, written for people *using* B2 rather than measuring it. - -## The instruments at a glance - -| Command | The question it answers | Model | Deterministic | -|---|---|---|---| -| `make eval` | Is retrieval/discovery *good*? Scores both labelled corpora through the real pipeline and **asserts the exit gate** | real bge | per machine+build¹ | -| `make eval-sweep` | Would a different `ChunkConfig` be better? (the [#44](https://github.com/AlteredCraft/B2/issues/44) chunker A/B, seven variants) | real bge | per machine+build¹ | -| `make eval-stemmer` | Is `porter unicode61` still the right FTS tokenizer? (the [#157](https://github.com/AlteredCraft/B2/issues/157) ablation) | real bge | per machine+build¹ | -| `make eval-metal` | The same eval on the Apple-Silicon GPU — a *different vector space* (`@metal`, ADR-0007); compare against a CPU run, never average with one | real bge | per machine+build¹ | -| `make stability` | Did the *ranking* move — under widening candidate pools, and since the blessed baseline? Says *different*, never *better* ([#141](https://github.com/AlteredCraft/B2/issues/141)) | fake | yes | -| `make calibrate VAULT=…` | Does a corpus-derived constant survive a **real vault**? (process rule 5; [#196](https://github.com/AlteredCraft/B2/issues/196)/[#197](https://github.com/AlteredCraft/B2/issues/197)) | stored vectors (pure read); `--search` loads the model | yes, per vault | -| `make eval-chat` | Does grounded chat cite the right notes and refuse what the vault can't answer? ([#154](https://github.com/AlteredCraft/B2/issues/154)) | real bge + a chat model server | no (LLM output varies) | -| `make compare-device` | CPU vs Metal embed *throughput* (a performance A/B, not a quality one) | real bge | no | - -¹ Measured bit-reproducible run-to-run on an unchanged corpus/model/build -([#188](https://github.com/AlteredCraft/B2/issues/188) — five runs, identical rows), so -"noise floor" means corpus and label drift, not run variance. Across devices the numbers -differ: the device is part of the embedding space's identity (ADR-0007), which is why rows -record the model id and are never averaged across `@metal` and CPU. - -**Why two quality instruments.** `make eval` scores **quality** — it can say *better*. -`make stability` scores **movement** — it can only say *different*, but it sees what the -labelled corpus is structurally near-blind to: **candidate width**. The eval corpus barely -exceeds the passage view's candidate pool and never the note view's, so a pool/width change -prints (nearly) bit-identical eval numbers while genuinely reordering a real vault. The -worked example is [#140](https://github.com/AlteredCraft/B2/issues/140)/[#142](https://github.com/AlteredCraft/B2/issues/142): -the eval saw nothing, the probe saw 10 of 10 probes change, and the change was reverted. -A probe can say *different*; only labels can say *better*. **Run both.** - -## Quick start - -```console -make init # provision bge-base-en-v1.5 (one time) -make eval # ~1min warm; appends rows to results.jsonl; non-zero exit on a gate regression -make stability # model-free, seconds; drift vs the blessed baseline -make calibrate VAULT=$HOME/notes # the real-vault transfer check (any built vault) -make calibrate VAULT=$HOME/notes ARGS=--search # …plus the search evidence bar's half (loads the model) -ollama serve & make eval-chat # grounded-chat scores (or any OpenAI-compatible server) -``` - -Exit codes, everywhere in the suite: **0** = the run completed (and, for `make eval`, every -gate cleared); **2** = a quality gate failed; **1** = the run itself broke (a missing model, -a label lint fault, a server that isn't there). `make stability` never gates — any completed -measurement exits 0, because drift is a signal, not a failure. - -## When to run what - -| You are changing… | Run | Because | -|---|---|---| -| chunking (`ChunkConfig`, boundaries, overlap) | `make eval-sweep`, then `make stability` | the A/B prices the change per query; passage-level ranks are where chunking shows | -| FTS tokenizer / lexical matching | `make eval-stemmer` | isolates the tokenizer over identical chunks + vectors | -| candidate pools, `pool_size`, RRF headroom | `make stability` (+ `make eval`) | the eval is near-blind to width ([#141](https://github.com/AlteredCraft/B2/issues/141)); the probe is its instrument | -| RRF constant / fusion weighting | `make eval` + `make stability` | `RRF_K` re-weights the *same* lists, so the eval does see it | -| a corpus note or a label | the token audit (process rule 2; [#218](https://github.com/AlteredCraft/B2/issues/218)), then `make eval` | a corpus edit is a change to the instrument — own commit, audited | -| any threshold read off a score distribution (a cosine bar, a z window, a band landmark) | `make calibrate` on ≥1 real vault | process rule 5 — the constant is invalid until it transfers | -| discovery ranking / the `similar` surface | `make eval` (both corpora's per-mate blocks) | the dense fixture is the geometry the orthogonal corpus can't express | -| the grounded-chat prompt, condensation, citations | `make eval-chat` | retrieval reach is reported beside it, so a miss is attributed, not guessed | -| the embed device (CPU ↔ Metal) | `make eval-metal` vs a CPU `make eval`; `make compare-device` | a device switch is a model swap; quality and throughput are separate questions | - ---- - -## `make eval` — the scored quality gate - -One run builds **two throwaway vaults** — the orthogonal corpus (`corpus/`, 31 notes) and the -dense single-domain fixture (`corpus-dense/`, 15 beekeeping notes) — and scores everything -through the real `Vault` pipeline. Nothing touches your vault, and nothing here writes to the -repo except the gitignored `results.jsonl`. - -The run refuses to start on labels the corpus cannot honour: a **label lint** checks every -labelled path exists and every `passage` is verbatim in a relevant note (a typo'd label -otherwise scores as a permanent miss and reads as an engine regression). Two **instrument -checks** then print before any score and gate everything after them: the model id (never -compare CPU and `@metal` rows), and `batch ≡ single` embedding faithfulness. - -### Reading the report, block by block - -Rank notation, used everywhere: `✓1` = ranked first; `·3` = ranked third; `✗>10` = not in -the top K=10 (discovery reads at K=5). `[check]` lines are instrument self-verification -passing; `[FAULT]` means *distrust the surrounding numbers* — the harness and engine -disagree; `[warn]` is a gate failure or a measurement caveat; `[note]` is a skipped reading -with its reason. - -1. **The per-query table** — every positive query's note rank under `bm25` (projection only, - the lexical floor), `vec` (dense-only ablation, [#158](https://github.com/AlteredCraft/B2/issues/158)), - `hybrid` (the shipped fusion), and `chunk` (passage-labelled queries only). This paired - list is the primary evidence (process rule 1); the aggregates below it are the smoke - alarm. At n≈44, one aggregate point ≈ one query. -2. **Aggregates + semantic lift** — hit@1/hit@3/MRR per mode; `semantic lift` is hybrid - hit@1 minus BM25-only hit@1, the measured value of the AI seam. The **fusion** lines name - every query hybrid ranks *worse* than the dense signal alone would — RRF's consensus bias, - counted and named on every run instead of rediscovered by hand. -3. **Chunk rank** — the passage-labelled subset (n=20) scored at chunk level; this is where - chunking levers show, and what the sweep is judged on. -4. **Discovery (`similar`)** — **per-mate** ranks (every labelled mate scored on its own, so - a hard mate can't hide behind an anchor's easy one, [#183](https://github.com/AlteredCraft/B2/issues/183)); - the **strangers** list (unlabelled notes served on positive anchors — a smoke alarm with - names attached, deliberately ungated: the cheapest way to shrink it is to label the - stranger, and labels aren't exhaustive); the **negatives** line (loner anchors serve their - ranked nearest under always-serve, ADR-0014 — the honesty rides on the strength bands); - and the **cosine piles** (labelled-related vs everything-else served — if they separate, - the gap is a floor; if they overlap, no simple floor exists and the data says so). -5. **Discovery z calibration** ([#187](https://github.com/AlteredCraft/B2/issues/187)) — - gates nothing; re-derives, every run, the windows a z existence rule *would* need - (leader: negative-anchor leaders vs positive; member: strangers vs mates). `open` means a - constant exists *on this corpus* — a real vault is the other half of any such claim - (process rule 5); `EMPTY` means the populations invert and no constant separates them. - The negatives' leaders print with the band each would paint (`●○○`/`●●○`/`●●●`) — what an - always-served card *claims* to the human whose label says "nothing relates". A `[check]` - line confirms the harness's independent z recomputation matches the engine's. -6. **Discovery fold bake-off** ([#200](https://github.com/AlteredCraft/B2/issues/200)) — - gates nothing; prices the candidate *default-disclosure* rules (mutual-k reciprocity, and - "no fold", the incumbent) on the same served lists. Columns: `cards above` (default-view - size), `mates folded` (labelled relations hidden by default — the fold's own cost, judged - at zero), `strangers above`, `loners empty` (the claim a fold exists to make), `dark - panes` (non-loner views emptied — disqualifying on the dense bench). The swept-`k` table - re-derives the admissible window; the verdict line says which bound failed. The ruling of - record: **no fold ships** — the window is empty and no `k` transfers across vault scales. -7. **Search evidence calibration + bake-off** ([#201](https://github.com/AlteredCraft/B2/issues/201), - ADR-0015) — per labelled query, the absolute signals RRF discards (OR-sanitized BM25 hit - count and best score, dense top-1 cosine) and what the surface serves, positives and - negatives apart. The bake-off sweeps the shipped rule's shape — IDF-weighted term - **coverage** OR a **cosine** bar — over the whole coverage grid; a cell is `✗` the moment - it anchors a labelled negative (no cosine bar can rescue it). The **shipped bar** lines - are gated: positives it would cut and negatives it still serves, both asserted at zero. A - `[FAULT]` here means the engine's verdict and the harness's restatement of the same rule - drifted apart — read the engine's `LexicalEvidence`, not the wording. -8. **Search tail bake-off** ([#206](https://github.com/AlteredCraft/B2/issues/206)) — gates - nothing; prices four per-hit prefix-cut families against the `tail_relevant` keep-set, - each constraint re-derived per run, every payoff read against the **oracle ceiling** (what - a fold placed by the labels themselves would cut). The ruling of record: **no tail fold - ships** — the fused order and the evidence disagree mid-list, so the tail complaint is - *ordering* work (the reranker seam), not disclosure work. -9. **The dense report** — per-mate ranks on `similar-dense.json`, the **zero empty panes** - sweep over *every* note (asserted: a vault where everything relates may never read as - "nothing relates"), the shipped search bar replayed over every note's own **title** as a - query plus built-in nonsense (`search_transfer` — titles need no labels, so nothing here - can be relabelled to clear a number), and the tail families against whole title lists - (every served row there is a real match by geometry). -10. **The cross-bench join** — printed once both corpora are read: a tail family ships only - if some constant is admissible on the labelled corpus *and* folds nothing on the dense - fixture *and* still buys something. Admissibility is not a shipping order — a joint edge - sits at a bench's own binding row with zero headroom, which the sizing method forbids. - -A `pool_blind` warning means the run's chunk count fits inside the narrower candidate pool, -so **candidate-width changes cannot move any number in that run** ([#141](https://github.com/AlteredCraft/B2/issues/141)) -— never read an unmoved number there as "no effect"; `make stability` is the instrument. - -### The exit gate - -`make eval` exits `0` only when the default config clears **all** assertions (the gate at -the end of `run()` in [`eval.rs`](../examples/eval.rs)): - -| Assertion | Constant | Direction | When it goes red, the fix is… | -|---|---|---|---| -| hybrid note hit@1 ≥ 0.75 | `FLOOR_HIT1` | floor, below the 0.95 reading | read the per-query misses above the aggregate; argue with the notes, never the labels | -| per-mate discovery MRR@5 ≥ 0.52 | `FLOOR_MATE_MRR` | floor, below the 0.650 reading | read the per-mate lines — which mate slid, on which anchor | -| dense fixture: zero empty panes | — | absolute | an existence gate is refusing a vault whose every note relates ([#196](https://github.com/AlteredCraft/B2/issues/196)) — remove it | -| dense per-mate MRR@5 ≥ 0.32 | `FLOOR_DENSE_MATE_MRR` | floor, below the 0.467 reading | as above; relabelling toward the model's order *always* looks plausible here, so don't | -| labelled negative queries served = 0 | `MAX_NEGATIVES_SERVED` | structural zero | the evidence bar is serving a query the vault holds nothing for — a rule regression or a mislabel; both want red | -| labelled relevant queries cut = 0 | `MAX_POSITIVES_CUT` | structural zero | change the **rule**, never the constant — the `df` ceiling died exactly here | -| dense titles cut = 0, dense nonsense served = 0 | `MAX_DENSE_TITLES_CUT` | structural zero | the lexical half has gone inert on a single-subject vault — the retired ceiling's failure shape | - -**How the gates are placed is the point.** A rank floor never sits *at* its reading: a gate -pinned to today's number fails on the first legitimate corpus edit, and the cheapest way to -clear a red rank is to **edit a label** — the one habit this harness must never train -(process rule 2). Run-to-run noise is zero (see ¹ above), so each MRR floor sits roughly two -lost-from-rank-1 mates under its reading, and the headroom is for *corpus drift*. The -search-evidence rows are the deliberate exception: they sit **at** their structural zeros -with no headroom, because headroom there would read as permission to serve a nonsense query -or cut a real one ([#202](https://github.com/AlteredCraft/B2/issues/202)). They skip, with a -printed `[note]`, on a model with no calibrated bar (M2) — asserting the absence of a -verdict would fail every run on a model nobody has measured yet. - -Two rows have **retired** from the gate, deliberately: the negatives' suppression assertion -(`neg_clean == neg_n`) went with the existence gate it watched — under always-serve a loner -serves its ranked nearest, and the honesty moved to the band readout -([#197](https://github.com/AlteredCraft/B2/issues/197)); and the pass-vs-pass suppression -tripwire went when it was found to compare a `similar` call against itself — structurally -incapable of firing — with a returning existence gate caught instead by the dense -zero-empty-panes assertion and the per-mate floors -([#217](https://github.com/AlteredCraft/B2/issues/217)). - -### The A/Bs: `--sweep` and `--stemmer` - -The sweep re-chunks and re-embeds the same vault under seven `ChunkConfig` variants; the -stemmer flag rebuilds `chunks_fts` under the unstemmed `unicode61` over **identical** chunks -and vectors, so every rank move is the tokenizer's alone (its dense column doubles as an -instrument check — FTS cannot reach it, so movement there means the harness broke). - -Read the **`Δ vs default` lines, not the scoreboard** (process rule 1): at this n, every -aggregate delta is 1–2 queries, and only the named flips can be argued against the labels. -"No per-query rank moved" is itself a claim to verify against a continuous quantity (the -piles) before believing "no effect" (process rule 4 — the `prepend-heading-path` lesson). -The sweep's variant rows record no calibration blocks: a disclosure rule judged on a -non-shipped chunker would be a number about the chunker. - ---- - -## `make stability` — the rank-movement probe - -Runs on `fixtures/test-vault` (~200 notes / ~780 chunks — big enough for the candidate pools -to **bind**) with the deterministic fake embedder, so the committed baseline means the same -thing on every machine. It scores no relevance: the corpus is unlabelled, so it can only say -*different, and by how much*. - -**Section 1 — pool sensitivity.** Each probe is asked at depths 4/10/30; a cell compares the -top-4 across one widening step, position by position. `=4` = the shallow answer is an exact -prefix of the deep one (pool-invariant); `3/4` = one position changed; `n/a` = the probe -matched nothing (measured probes only enter the denominators). The summary counts probes -whose top-4 moved per step — that is what candidate width is worth on this vault. - -**Section 2 — baseline drift.** The shipped top-10 against the blessed snapshot -(`stability-baseline.json`): `=10` = exact; `kept, reordered`; or `n in, m out`. **Drift is -a signal, not a failure** — a knob change is *supposed* to move ranking. Once the change is -the intended one, `make stability-bless` accepts it (fake embedder + committed vault only — -a real-model or private-vault baseline would commit a number nobody else can reproduce). -Editing `fixtures/test-vault` or `stability.json` invalidates the baseline — re-bless in the -same commit. - -Flags: `ARGS=--verbose` prints the diverging lists; `ARGS=--model` runs real bge (magnitude -only, no baseline); `ARGS="--vault "` probes any vault — -`--vault crates/b2-embed/evals/corpus` is the control experiment where every prefix holds: -that *is* [#141](https://github.com/AlteredCraft/B2/issues/141)'s blindness, demonstrated. - ---- - -## `make calibrate` — the real-vault transfer check - -Process rule 5 made mechanical: run it against **any built vault** — no labels needed, a -pure read over stored vectors, seconds even on a large vault. This is the instrument that -retired the existence gate ([#196](https://github.com/AlteredCraft/B2/issues/196) measured -16 of 17 panes dark on a real single-domain vault) and killed the `df`-ceiling evidence rule; -any future distributional constant answers to it **before** shipping. - -Per-anchor columns: pool size `n`; the pool's cosine `min/med/max`; the leader's cosine and -z; `gate serves` — what the retired z gate would serve (`DARK` = it would empty this pane; a -simulation, the shipped surface gates nothing); `fold` — the mutual-k reciprocity fold's -default view ([#200](https://github.com/AlteredCraft/B2/issues/200)'s candidate 1, replayed); -`e-bar` — the authored-edge reference bar's fold (candidate 2 — this is the **only** -instrument that can price it, since its calibration population is the human's own committed -edges and both eval corpora are link-free); `bands` — the strength-band histogram the pane -would paint. The summary block repeats each as a vault-level reading, plus the engine-z -drift `[check]`. - -Interpretation: a replayed gate darkening panes on a vault you know is inter-related is the -[#196](https://github.com/AlteredCraft/B2/issues/196) failure reproduced; an `e-bar` at 0 or -`UNPRICEABLE` is a *reading about the vault* (no scorable authored edges), not an instrument -failure; a band histogram compressed into `●○○` on a dense vault is the open A6 thread -(band compression), not a bug. A fake-embedded vault warns and prints noise — reindex with -the real model first. - -`ARGS=--search` adds the **search-side** transfer bench ([#201](https://github.com/AlteredCraft/B2/issues/201)): -first the vault's function-word weights beside an absent word's (the lexical anchor's whole -premise, checked per vault rather than assumed of English), then the shipped evidence bar -replayed over every note's own **title** (positives by construction — nothing to relabel) -and built-in nonsense (the vault-independent negatives), the ten lowest-cosine positives -(where a bar placement bites first), and the per-hit tail families priced on the one row a -title query certifies with no label: its own note. `the bar would cut N/…` is the tripwire -direction — D2 permits zero. This is the one part of the instrument that is not a pure read -(judging the cosine half embeds each probe query, so it loads the model); it has no `--json` -form yet ([#219](https://github.com/AlteredCraft/B2/issues/219)). - -Other flags: `--limit N` (pane depth), `--leader-z/--member-z` (replay a different gate; -defaults are the retired constants, so [#196](https://github.com/AlteredCraft/B2/issues/196)'s -dark-vault reading reproduces), `--mutual-k N`, `--json` (the discovery reading as one -object, for scripting sweeps). - ---- - -## `make eval-chat` — the grounded-chat scores - -Needs a model server (`ollama serve` + `ollama pull llama3.2`, or `B2_LLM_URL` / -`B2_LLM_MODEL` at any OpenAI-compatible endpoint). Builds a throwaway vault from the -**retrieval eval's corpus** — chat is scored over notes whose retrieval behaviour is already -characterised, so a surprise is attributable — and asks the labelled questions in -[`../../b2-llm/evals/questions.json`](../../b2-llm/evals/questions.json) through the real -`Vault::ask`. Unlike `queries.json`, questions do **not** avoid the target's vocabulary: -this scores what the model does with retrieved passages, not what retrieval finds. - -Four deliberately-separable scores, because a bad answer has more than one possible author: - -| Line | Meaning | A miss is… | -|---|---|---| -| **retrieval reach** | did a labelled note make it into the passages at all? | `make eval`'s result, not chat's — the ceiling every other score is judged under | -| **citation accuracy** | did the answer cite the labelled note? (scored only over reached questions) | the headline chat number | -| **grounding rate** | did the answer cite *anything*? | an uncited answer is a refusal or general knowledge, and the prompt forbids the second | -| **refusal accuracy** | on the two deliberate negatives, did it say "I don't find that in your notes"? | a confabulation — one negative is near the corpus's coverage on purpose, because models confabulate from loosely related passages | - -Per-question verdicts: `cited the note` / `refused (correct)` are the good ends; -`retrieval missed (not a chat result)` blames the other half; `MISCITED`, -`REFUSED WITH EVIDENCE PRESENT`, and `CONFABULATED` are the model's failures, with -hallucinated `[n]` markers (markers naming no passage) counted beside them. TTFT/total -latency print per question. - -**The exit code is a liveness check, not a quality bar**: `2` only when retrieval reached -labelled notes and *no* answer cited one — a broken pipeline. Model quality is read off the -numbers; a gate that fails on every small local model would be a gate nobody runs. Known -coverage gap: every question is single-turn, so the condense step is unmeasured -([#220](https://github.com/AlteredCraft/B2/issues/220)). - ---- - -## `results.jsonl` — the run log - -Both harnesses append one JSON line per scored run to their `evals/results.jsonl` -(gitignored, machine-local — scores depend on the machine's models). Every number ever -cited in an issue traces to a row here. Conventions: - -- **Append-only**, so runs accumulate into one comparable dataset. -- **Keys are additive, never redefined.** A metric that changes meaning gets a *new* key; - the old one goes absent, so a reader of a mixed file sees a missing field rather than one - number silently meaning something narrower than it used to. (Retired keys so far: - `discovery_z.shipped`/`replay_faults` with [#197](https://github.com/AlteredCraft/B2/issues/197); - `similar_per_mate_raw`/`similar_mates_suppressed` with [#217](https://github.com/AlteredCraft/B2/issues/217).) -- **`"corpus"` tags every row** (`"orthogonal"` / `"dense"`); rows are never averaged across - corpora, and the dense row's transfer reading lives under its own `search_transfer` key - rather than overloading `search_evidence`. -- **Re-derivability**: the calibration subtrees (`discovery_z`, `discovery_fold`, - `search_evidence`, `search_tail`, `search_transfer`) record the raw per-candidate / - per-row data, so any window, fold depth, or bar can be re-derived from a row without - re-running the model. -- `pool_blind: true` marks a row no candidate-width comparison may be read from. - -A few readers: - -```console -# the latest orthogonal default-config row -jq -s '[.[] | select(.corpus == "orthogonal" and .config.label == "default")] | last' results.jsonl - -# hybrid note MRR across runs -jq -r 'select(.corpus == "orthogonal") | [(.ts | todate), .git, .config.label, .note.hybrid.mrr] | @tsv' results.jsonl - -# per-mate discovery MRR, both corpora -jq -r '[(.ts | todate), .corpus, .similar_per_mate.mrr] | @tsv' results.jsonl - -# which queries missed hybrid rank 1 in a given row -jq -r 'select(.corpus == "orthogonal") | .queries[] | select(.hybrid != 1) | [.q, (.hybrid // "miss")] | @tsv' results.jsonl -``` - -## The corpora and labels (ground truth) - -| Path | Role | -|---|---| -| [`corpus/`](corpus/) | the hand-written **orthogonal** vault (31 notes): topic clusters, six long multi-chunk notes, five unambiguous loners, the stemmer-adversarial block, the [#183](https://github.com/AlteredCraft/B2/issues/183) multi-topic family, and `week-log.md`, the journal-shaped dilution extreme ([#189](https://github.com/AlteredCraft/B2/issues/189)/[#192](https://github.com/AlteredCraft/B2/issues/192)). Its token audit *minimizes* shared vocabulary — which is exactly why it cannot express topical concentration | -| [`queries.json`](queries.json) | retrieval labels — 44 positives (20 with a chunk-level `passage`; three date-shaped, [#202](https://github.com/AlteredCraft/B2/issues/202)) + 5 **negatives** (empty `relevant` = the labelled answer is *no matches*). Seven positives carry `tail_relevant` ([#206](https://github.com/AlteredCraft/B2/issues/206)) — the per-hit keep-set, **exhaustive by label** and encoded by note, never by rank. Its `description` field is the labelling rulebook — read it before editing | -| [`similar.json`](similar.json) | discovery labels — positive anchors with expected mates; **empty `expected` = a negative anchor** (a loner whose correct answer is *nothing*). Its `description` is the loner-orthogonality rulebook | -| [`corpus-dense/`](corpus-dense/) + [`similar-dense.json`](similar-dense.json) | the **dense single-domain fixture** (15 beekeeping notes, all inter-related, no loner) — the vault-level geometry the orthogonal corpus is structurally incapable of expressing, and the bench that killed the existence gate and the `df` ceiling. Rankings-only labels; scored in its own vault and row | -| [`stability.json`](stability.json) + [`stability-baseline.json`](stability-baseline.json) | the unlabelled probe set and its blessed ranking snapshot (fake embedder over `fixtures/test-vault`) | -| [`../../b2-llm/evals/questions.json`](../../b2-llm/evals/questions.json) | the chat set — questions phrased as a person would type them, `expect` = the note(s) a correct answer must cite; empty `expect` = the only correct answer is the refusal | -| `results.jsonl` (both harnesses) | append-only run logs, gitignored | - -## The verdict record - -Every ruling this suite has produced, one line each — the issue holds the full argument and -the numbers, and the commit that shipped it is the record of the change. (This table -replaced the long-form verdict log that used to live here; `git log` on this file has it.) - -| Issue | Verdict | -|---|---| -| [#44](https://github.com/AlteredCraft/B2/issues/44) | `ChunkConfig::default()` held against a seven-variant sweep; the "winning" rows were impeached by a measured boundary-luck noise floor. Retrial is one `make eval-sweep` away | -| [#141](https://github.com/AlteredCraft/B2/issues/141)/[#142](https://github.com/AlteredCraft/B2/issues/142) | the labelled corpus is near-blind to candidate width; a 3× pool widening was reverted on the stability probe's evidence | -| [#150](https://github.com/AlteredCraft/B2/issues/150) | the z-score discovery quality floor shipped — per-anchor z over the centroid population, suppression entering the gate; **superseded by [#197](https://github.com/AlteredCraft/B2/issues/197)** | -| [#156](https://github.com/AlteredCraft/B2/issues/156) | RRF fused-score ties break on the dense signal's rank — a policy the eval decided, not walk order | -| [#157](https://github.com/AlteredCraft/B2/issues/157) | `porter unicode61` FTS stemming: 7–0 BM25 / 3–0 hybrid on the paired readout, precision probes unmoved; the unstemmed arm stays measurable via `make eval-stemmer` | -| [#183](https://github.com/AlteredCraft/B2/issues/183) | the multi-topic note family landed, and with it the **per-mate** metric — the per-anchor one saturates at 1.000 and hides every hard mate | -| [#187](https://github.com/AlteredCraft/B2/issues/187) | the member window is **empty on the shipped corpus's own numbers** — mates and strangers invert, so no constant separates them; windows are re-derived every run, never frozen in a docstring | -| [#189](https://github.com/AlteredCraft/B2/issues/189) | the journal-shaped note was built, measured, and **rejected** — the first corpus edit this harness refused (its diluted centroid tops loner anchors on content it doesn't contain) | -| [#192](https://github.com/AlteredCraft/B2/issues/192) | the floor moved to the stage-2 best-passage unit, and `week-log.md` landed — read correctly in both directions at once | -| [#182](https://github.com/AlteredCraft/B2/issues/182) | the buried gem is served, and the desktop's strength bands were re-read in the judged unit (the last surface still reading the retired one) | -| [#188](https://github.com/AlteredCraft/B2/issues/188) | discovery rank entered the exit gate; the harness measured **bit-reproducible** run-to-run, so floors are sized for corpus drift, not noise | -| [#196](https://github.com/AlteredCraft/B2/issues/196)/[#197](https://github.com/AlteredCraft/B2/issues/197) | the existence gate itself was the defect: a single-domain vault went 16/17 dark while the ranking was correct throughout. The gate retired; discovery **always serves the ranked list** (ADR-0014); `make calibrate` and the dense fixture landed *before* the fix, per the sequencing rule | -| [#200](https://github.com/AlteredCraft/B2/issues/200) | the discovery fold bake-off ran and **no fold ships**: mutual-k's admissible window is empty, and its safe depth is a different rule on every vault scale; the authored-edge bar is unpriceable on link-free corpora and stays open | -| [#201](https://github.com/AlteredCraft/B2/issues/201) | search's evidence bar was **earned** — and its first form (a hard `df` ceiling) read clean on the labelled bench and cut 3/15 answerable queries on the dense one, so the *rule* changed (IDF as a weight, not a bin), not the constant | -| [#202](https://github.com/AlteredCraft/B2/issues/202) | the verdict reached the surfaces ("no matches" is a real answer; `--json` became an object), the date-shaped block landed, and search's three structural-zero rows entered the gate | -| [#206](https://github.com/AlteredCraft/B2/issues/206) | the per-hit tail bake-off ran and **no tail fold ships**: 367 of 386 filler rows sit below the oracle fold, but the fused order misplaces the evidence, so every admissible prefix cut is near-vacuous — the tail complaint is ordering work (the reranker seam) | -| [#217](https://github.com/AlteredCraft/B2/issues/217) | the pass-vs-pass suppression tripwire compared a call against itself and could never fire; removed — the dense pane assertion and the per-mate floors are what catch a returning existence gate | - -**Standing principles from the record**: when a label and a note disagree, fix whichever one -is lying (the watercolor → throat-singing precedent); an arguable negative is replaced, not -argued; instruments land *before* the rules they price; and a reading that disqualifies a -rule is re-derived every run, never frozen into a comment. - -**The deliberately open threads**: the **phishing pair** (a real relation ranked under three -stranger pairs even in the best-passage unit — served, so ordering residue, the pair-scorer's -standing evidence); the **tail's 367 rows of headroom** (the second exhibit for a reranker, -measured on search's flow); and the **dense-vault band compression** (#197's A6 — on a vault -where every candidate is close, the within-list z compresses and the dots lose resolution; -first read on a real-embedded build of the dense fixture: 0 ●●● / 6 ●●○ / 144 ●○○). - -## Process rules - -Adopted 2026-08-10; each traces to a measured mistake or a named risk, and each is binding -on anyone editing the corpora, the labels, or the metrics. - -1. **A paired per-query win/loss list is the primary readout of any A/B; the aggregate is a - smoke alarm.** At n≈40, every aggregate point is 1–2 queries — "hit@1 +0.05" and "these - two flipped" are the same fact, but only the second can be argued with against the - labels. The sweep prints the diff (`Δ vs default`) automatically. -2. **A corpus edit is a change to the instrument, so it ships as its own commit** whose - message says what changed and why, and every edit runs the **two-direction token audit** - before it lands: no existing query's content tokens may newly land in the edited/added - note, and no new query's content tokens may split evenly toward a rival — word-boundary - and stem-prefix both (the `insomnia.md` steal and the `recover`+`mistake` near-miss are - the precedents). Run the audit, don't eyeball it ([#218](https://github.com/AlteredCraft/B2/issues/218) - tracks committing it as a script). And — since the gate reads discovery rank — **a red - gate is never an argument for editing a label**: per-mate MRR and the strangers count - both move when labels move, so the only honest response to red is to argue about the - *notes*. -3. **The same person authoring notes, queries, and fixes is a ratchet toward measuring what - the engine already does.** Mitigations in order of cheapness: rule 2's audit; sourcing - future queries from outside the corpus author's head (from note titles alone, or another - person); dogfooding on a real vault before trusting any threshold. -4. **A bit-identical or unmoved metric is a claim to verify, never proof of "no effect"** — - compare a continuous quantity (the piles) before believing a discrete one. (The - `prepend-heading-path` trace; the sweep diff prints its own reminder.) -5. **A constant derived from a corpus's score *distribution* is invalid until - transfer-checked on a real vault** (`make calibrate` is the check; adopted with - [#197](https://github.com/AlteredCraft/B2/issues/197), from a measured mistake made - twice). Rank-derived readings transfer, because the corpus's *orderings* are engineered - to be checkable; its score **distributions** are an artifact of engineered orthogonality, - so any threshold read off them — a cosine bar, a z window, a band landmark — describes - the corpus, not a vault. A distributional constant ships only with a `calibrate` reading - from at least one real vault beside the corpus numbers. +The guide to this harness lives in [docs/evals.md](../../../docs/evals.md): every +instrument, how to read its output, the exit gate, the verdict record, and the process rules. +Read it before touching the corpora, the labels, or the metrics in this directory. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..aa0a35c --- /dev/null +++ b/docs/README.md @@ -0,0 +1,32 @@ +# B2 docs + +The documentation for B2: a personal, local-first Markdown vault with an AI layer that +surfaces semantically similar notes for you to connect. Start here to find the one page that +answers your question. + +Every topic has one home. The specs are normative: the code is a projection of them, and code +comments cite them by section and by invariant id (`data-model.md §2`, `invariants.md D1`). +On any conflict, [invariants.md](invariants.md) wins and the other side gets fixed. + +## Which page do you need? + +| You want to… | Read | +|---|---| +| Set up B2 and use it (install, index, search, link, chat) | [quickstart.md](quickstart.md) | +| Understand what search and the related-notes panel do, in plain language | [search-and-similarity.md](search-and-similarity.md) | +| Orient yourself in the codebase (crates, flows, seams, tests) | [architecture.md](architecture.md) | +| Know what must always be true, cited by id (S2, D1, …) | [invariants.md](invariants.md) | +| Know what a note and a connection are (frontmatter, links, the verb core) | [data-model.md](data-model.md) | +| Know how the index is built and queried (schema, ingest, search, discovery, chat) | [index-engine.md](index-engine.md) | +| Measure retrieval and chat quality, or edit the eval corpora and labels | [evals.md](evals.md) | +| Know *why* a decision reads the way it does | [ADRs/](../ADRs/README.md) | + +## Also worth knowing + +- The backlog and planned work live in + [GitHub Issues](https://github.com/AlteredCraft/B2/issues); build history lives in git. +- The command reference and every environment variable live in + [quickstart.md](quickstart.md). +- Working in the repo? The contributor ground rules are [CLAUDE.md](../CLAUDE.md), and the + desktop crate has its own in + [crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md). diff --git a/docs/evals.md b/docs/evals.md new file mode 100644 index 0000000..d0199dd --- /dev/null +++ b/docs/evals.md @@ -0,0 +1,452 @@ +# The eval suite + +How B2 measures the one thing `cargo test` cannot: whether retrieval, discovery, and grounded +chat are any *good*. Read this before you run an instrument, read its output, or touch the +corpora, the labels, or the metrics; the [process rules](#process-rules) at the bottom bind +every such edit. + +A unit test can prove `reindex` is idempotent. Only a human-labelled corpus can say that "how +do leaves turn light into food" should rank `photosynthesis.md` first. Everything here runs +out of CI, on demand (ADR-0013): `cargo test` stays fast, deterministic, and model-free, so +model quality can never flake CI. Decision history lives in git and in +[GitHub Issues](https://github.com/AlteredCraft/B2/issues): every verdict in the +[record below](#the-verdict-record) names the issue that drove it, and the commit that +shipped it is the record of what changed and why. + +Not the audience for this page? +[search-and-similarity.md](search-and-similarity.md) is the plain-language tour of everything +these metrics score, written for people *using* B2 rather than measuring it. + +The harness code and its data live in +[`crates/b2-embed/evals/`](../crates/b2-embed/evals/); this page is the guide to all of it. + +## The instruments at a glance + +| Command | The question it answers | Model | Deterministic | +|---|---|---|---| +| `make eval` | Is retrieval/discovery *good*? Scores both labelled corpora through the real pipeline and asserts the exit gate | real bge | per machine+build¹ | +| `make eval-sweep` | Would a different `ChunkConfig` be better? (the [#44](https://github.com/AlteredCraft/B2/issues/44) chunker A/B, seven variants) | real bge | per machine+build¹ | +| `make eval-stemmer` | Is `porter unicode61` still the right FTS tokenizer? (the [#157](https://github.com/AlteredCraft/B2/issues/157) ablation) | real bge | per machine+build¹ | +| `make eval-metal` | The same eval on the Apple-Silicon GPU, a *different vector space* (`@metal`, ADR-0007). Compare against a CPU run, never average with one | real bge | per machine+build¹ | +| `make stability` | Did the *ranking* move, under widening candidate pools and since the blessed baseline? Says *different*, never *better* ([#141](https://github.com/AlteredCraft/B2/issues/141)) | fake | yes | +| `make calibrate VAULT=…` | Does a corpus-derived constant survive a real vault? (process rule 5; [#196](https://github.com/AlteredCraft/B2/issues/196)/[#197](https://github.com/AlteredCraft/B2/issues/197)) | stored vectors (pure read); `--search` loads the model | yes, per vault | +| `make eval-chat` | Does grounded chat cite the right notes and refuse what the vault can't answer? ([#154](https://github.com/AlteredCraft/B2/issues/154)) | real bge + a chat model server | no (LLM output varies) | +| `make compare-device` | CPU vs Metal embed *throughput* (a performance A/B, not a quality one) | real bge | no | + +¹ Measured bit-reproducible run-to-run on an unchanged corpus/model/build +([#188](https://github.com/AlteredCraft/B2/issues/188): five runs, identical rows). So "noise +floor" means corpus and label drift, not run variance. Across devices the numbers differ: the +device is part of the embedding space's identity (ADR-0007), which is why rows record the +model id and are never averaged across `@metal` and CPU. + +**Why two quality instruments.** `make eval` scores quality: it can say *better*. +`make stability` scores movement: it can only say *different*, but it sees what the labelled +corpus is structurally near-blind to, candidate width. The eval corpus barely exceeds the +passage view's candidate pool and never the note view's, so a pool-width change prints +(nearly) bit-identical eval numbers while genuinely reordering a real vault. The worked +example is +[#140](https://github.com/AlteredCraft/B2/issues/140)/[#142](https://github.com/AlteredCraft/B2/issues/142): +the eval saw nothing, the probe saw 10 of 10 probes change, and the change was reverted. A +probe can say *different*; only labels can say *better*. Run both. + +## Quick start + +```console +make init # provision bge-base-en-v1.5 (one time) +make eval # ~1min warm; appends rows to results.jsonl; non-zero exit on a gate regression +make stability # model-free, seconds; drift vs the blessed baseline +make calibrate VAULT=$HOME/notes # the real-vault transfer check (any built vault) +make calibrate VAULT=$HOME/notes ARGS=--search # …plus the search evidence bar's half (loads the model) +ollama serve & make eval-chat # grounded-chat scores (or any OpenAI-compatible server) +``` + +Exit codes, everywhere in the suite: **0** means the run completed (and, for `make eval`, +every gate cleared); **2** means a quality gate failed; **1** means the run itself broke (a +missing model, a label lint fault, a server that isn't there). `make stability` never gates: +any completed measurement exits 0, because drift is a signal, not a failure. + +## When to run what + +| You are changing… | Run | Because | +|---|---|---| +| chunking (`ChunkConfig`, boundaries, overlap) | `make eval-sweep`, then `make stability` | the A/B prices the change per query; passage-level ranks are where chunking shows | +| FTS tokenizer / lexical matching | `make eval-stemmer` | isolates the tokenizer over identical chunks + vectors | +| candidate pools, `pool_size`, RRF headroom | `make stability` (+ `make eval`) | the eval is near-blind to width ([#141](https://github.com/AlteredCraft/B2/issues/141)); the probe is its instrument | +| RRF constant / fusion weighting | `make eval` + `make stability` | `RRF_K` re-weights the *same* lists, so the eval does see it | +| a corpus note or a label | the token audit (process rule 2; [#218](https://github.com/AlteredCraft/B2/issues/218)), then `make eval` | a corpus edit is a change to the instrument: own commit, audited | +| any threshold read off a score distribution (a cosine bar, a z window, a band landmark) | `make calibrate` on ≥1 real vault | process rule 5: the constant is invalid until it transfers | +| discovery ranking / the `similar` surface | `make eval` (both corpora's per-mate blocks) | the dense fixture is the geometry the orthogonal corpus can't express | +| the grounded-chat prompt, condensation, citations | `make eval-chat` | retrieval reach is reported beside it, so a miss is attributed, not guessed | +| the embed device (CPU ↔ Metal) | `make eval-metal` vs a CPU `make eval`; `make compare-device` | a device switch is a model swap; quality and throughput are separate questions | + +--- + +## `make eval`: the scored quality gate + +One run builds two throwaway vaults, the orthogonal corpus +([`corpus/`](../crates/b2-embed/evals/corpus/), 31 notes) and the dense single-domain fixture +([`corpus-dense/`](../crates/b2-embed/evals/corpus-dense/), 15 beekeeping notes), and scores +everything through the real `Vault` pipeline. Nothing touches your vault, and nothing here +writes to the repo except the gitignored `results.jsonl`. + +The run refuses to start on labels the corpus cannot honor: a **label lint** checks every +labelled path exists and every `passage` is verbatim in a relevant note (a typo'd label +otherwise scores as a permanent miss and reads as an engine regression). Two **instrument +checks** then print before any score and gate everything after them: the model id (never +compare CPU and `@metal` rows), and `batch ≡ single` embedding faithfulness. + +### How to read the report, block by block + +Rank notation, used everywhere: `✓1` means ranked first; `·3` means ranked third; `✗>10` +means not in the top K=10 (discovery reads at K=5). `[check]` lines are instrument +self-verification passing. `[FAULT]` means *distrust the surrounding numbers*: the harness +and engine disagree. `[warn]` is a gate failure or a measurement caveat. `[note]` is a +skipped reading with its reason. + +1. **The per-query table.** Every positive query's note rank under `bm25` (projection only, + the lexical floor), `vec` (dense-only ablation, + [#158](https://github.com/AlteredCraft/B2/issues/158)), `hybrid` (the shipped fusion), and + `chunk` (passage-labelled queries only). This paired list is the primary evidence (process + rule 1); the aggregates below it are the smoke alarm. At n≈44, one aggregate point is + about one query. +2. **Aggregates + semantic lift.** hit@1/hit@3/MRR per mode. `semantic lift` is hybrid hit@1 + minus BM25-only hit@1: the measured value of the AI seam. The **fusion** lines name every + query hybrid ranks *worse* than the dense signal alone would (RRF's consensus bias), + counted and named on every run instead of rediscovered by hand. +3. **Chunk rank.** The passage-labelled subset (n=20) scored at chunk level. This is where + chunking levers show, and what the sweep is judged on. +4. **Discovery (`similar`).** Four readings: the **per-mate** ranks (every labelled mate + scored on its own, so a hard mate can't hide behind an anchor's easy one, + [#183](https://github.com/AlteredCraft/B2/issues/183)); the **strangers** list (unlabelled + notes served on positive anchors: a smoke alarm with names attached, deliberately ungated, + because the cheapest way to shrink it is to label the stranger, and labels aren't + exhaustive); the **negatives** line (loner anchors serve their ranked nearest under + always-serve, ADR-0014; the honesty rides on the strength bands); and the **cosine piles** + (labelled-related vs everything-else served: if they separate, the gap is a floor; if they + overlap, no simple floor exists and the data says so). +5. **Discovery z calibration** ([#187](https://github.com/AlteredCraft/B2/issues/187)). + Gates nothing. Re-derives, every run, the windows a z existence rule *would* need (leader: + negative-anchor leaders vs positive; member: strangers vs mates). `open` means a constant + exists *on this corpus*; a real vault is the other half of any such claim (process rule + 5). `EMPTY` means the populations invert and no constant separates them. The negatives' + leaders print with the band each would paint (`●○○`/`●●○`/`●●●`): what an always-served + card *claims* to the human whose label says "nothing relates". A `[check]` line confirms + the harness's independent z recomputation matches the engine's. +6. **Discovery fold bake-off** ([#200](https://github.com/AlteredCraft/B2/issues/200)). + Gates nothing. Prices the candidate default-disclosure rules (mutual-k reciprocity, and + "no fold", the incumbent) on the same served lists. Columns: `cards above` (default-view + size), `mates folded` (labelled relations hidden by default: the fold's own cost, judged + at zero), `strangers above`, `loners empty` (the claim a fold exists to make), `dark + panes` (non-loner views emptied: disqualifying on the dense bench). The swept-`k` table + re-derives the admissible window; the verdict line says which bound failed. The ruling of + record: **no fold ships**. The window is empty, and no `k` transfers across vault scales. +7. **Search evidence calibration + bake-off** + ([#201](https://github.com/AlteredCraft/B2/issues/201), ADR-0015). Per labelled query, the + absolute signals RRF discards (OR-sanitized BM25 hit count and best score, dense top-1 + cosine) and what the surface serves, positives and negatives apart. The bake-off sweeps + the shipped rule's shape (IDF-weighted term **coverage** OR a **cosine** bar) over the + whole coverage grid; a cell is `✗` the moment it anchors a labelled negative (no cosine + bar can rescue it). The **shipped bar** lines are gated: positives it would cut and + negatives it still serves, both asserted at zero. A `[FAULT]` here means the engine's + verdict and the harness's restatement of the same rule drifted apart: read the engine's + `LexicalEvidence`, not the wording. +8. **Search tail bake-off** ([#206](https://github.com/AlteredCraft/B2/issues/206)). Gates + nothing. Prices four per-hit prefix-cut families against the `tail_relevant` keep-set, + each constraint re-derived per run, every payoff read against the **oracle ceiling** (what + a fold placed by the labels themselves would cut). The ruling of record: **no tail fold + ships**. The fused order and the evidence disagree mid-list, so the tail complaint is + *ordering* work (the reranker seam), not disclosure work. +9. **The dense report.** Per-mate ranks on `similar-dense.json`; the **zero empty panes** + sweep over *every* note (asserted: a vault where everything relates may never read as + "nothing relates"); the shipped search bar replayed over every note's own **title** as a + query plus built-in nonsense (`search_transfer`: titles need no labels, so nothing here + can be relabelled to clear a number); and the tail families against whole title lists + (every served row there is a real match by geometry). +10. **The cross-bench join.** Printed once both corpora are read: a tail family ships only if + some constant is admissible on the labelled corpus *and* folds nothing on the dense + fixture *and* still buys something. Admissibility is not a shipping order: a joint edge + sits at a bench's own binding row with zero headroom, which the sizing method forbids. + +A `pool_blind` warning means the run's chunk count fits inside the narrower candidate pool, +so candidate-width changes cannot move any number in that run +([#141](https://github.com/AlteredCraft/B2/issues/141)). Never read an unmoved number there +as "no effect"; `make stability` is the instrument. + +### The exit gate + +`make eval` exits `0` only when the default config clears **all** assertions (the gate at the +end of `run()` in [`eval.rs`](../crates/b2-embed/examples/eval.rs)): + +| Assertion | Constant | Direction | When it goes red, the fix is… | +|---|---|---|---| +| hybrid note hit@1 ≥ 0.75 | `FLOOR_HIT1` | floor, below the 0.95 reading | read the per-query misses above the aggregate; argue with the notes, never the labels | +| per-mate discovery MRR@5 ≥ 0.52 | `FLOOR_MATE_MRR` | floor, below the 0.650 reading | read the per-mate lines: which mate slid, on which anchor | +| dense fixture: zero empty panes | — | absolute | an existence gate is refusing a vault whose every note relates ([#196](https://github.com/AlteredCraft/B2/issues/196)); remove it | +| dense per-mate MRR@5 ≥ 0.32 | `FLOOR_DENSE_MATE_MRR` | floor, below the 0.467 reading | as above; relabelling toward the model's order *always* looks plausible here, so don't | +| labelled negative queries served = 0 | `MAX_NEGATIVES_SERVED` | structural zero | the evidence bar is serving a query the vault holds nothing for: a rule regression or a mislabel; both want red | +| labelled relevant queries cut = 0 | `MAX_POSITIVES_CUT` | structural zero | change the **rule**, never the constant; the `df` ceiling died exactly here | +| dense titles cut = 0, dense nonsense served = 0 | `MAX_DENSE_TITLES_CUT` | structural zero | the lexical half has gone inert on a single-subject vault: the retired ceiling's failure shape | + +**How the gates are placed is the point.** A rank floor never sits *at* its reading: a gate +pinned to today's number fails on the first legitimate corpus edit, and the cheapest way to +clear a red rank is to edit a label, the one habit this harness must never train (process +rule 2). Run-to-run noise is zero (see ¹ above), so each MRR floor sits roughly two +lost-from-rank-1 mates under its reading, and the headroom is for *corpus drift*. The +search-evidence rows are the deliberate exception: they sit *at* their structural zeros with +no headroom, because headroom there would read as permission to serve a nonsense query or cut +a real one ([#202](https://github.com/AlteredCraft/B2/issues/202)). They skip, with a printed +`[note]`, on a model with no calibrated bar (M2): asserting the absence of a verdict would +fail every run on a model nobody has measured yet. + +Two rows have **retired** from the gate, deliberately. The negatives' suppression assertion +(`neg_clean == neg_n`) went with the existence gate it watched: under always-serve a loner +serves its ranked nearest, and the honesty moved to the band readout +([#197](https://github.com/AlteredCraft/B2/issues/197)). And the pass-vs-pass suppression +tripwire went when it was found to compare a `similar` call against itself, structurally +incapable of firing; a returning existence gate is caught instead by the dense +zero-empty-panes assertion and the per-mate floors +([#217](https://github.com/AlteredCraft/B2/issues/217)). + +### The A/Bs: `--sweep` and `--stemmer` + +The sweep re-chunks and re-embeds the same vault under seven `ChunkConfig` variants. The +stemmer flag rebuilds `chunks_fts` under the unstemmed `unicode61` over **identical** chunks +and vectors, so every rank move is the tokenizer's alone (its dense column doubles as an +instrument check: FTS cannot reach it, so movement there means the harness broke). + +Read the `Δ vs default` lines, not the scoreboard (process rule 1): at this n, every +aggregate delta is 1 or 2 queries, and only the named flips can be argued against the labels. +"No per-query rank moved" is itself a claim to verify against a continuous quantity (the +piles) before believing "no effect" (process rule 4: the `prepend-heading-path` lesson). The +sweep's variant rows record no calibration blocks: a disclosure rule judged on a non-shipped +chunker would be a number about the chunker. + +--- + +## `make stability`: the rank-movement probe + +Runs on `fixtures/test-vault` (~200 notes / ~780 chunks: big enough for the candidate pools +to **bind**) with the deterministic fake embedder, so the committed baseline means the same +thing on every machine. It scores no relevance: the corpus is unlabelled, so it can only say +*different, and by how much*. + +**Section 1: pool sensitivity.** Each probe is asked at depths 4/10/30; a cell compares the +top-4 across one widening step, position by position. `=4` means the shallow answer is an +exact prefix of the deep one (pool-invariant); `3/4` means one position changed; `n/a` means +the probe matched nothing (measured probes only enter the denominators). The summary counts +probes whose top-4 moved per step: that is what candidate width is worth on this vault. + +**Section 2: baseline drift.** The shipped top-10 against the blessed snapshot +(`stability-baseline.json`): `=10` (exact), `kept, reordered`, or `n in, m out`. Drift is a +signal, not a failure: a knob change is *supposed* to move ranking. Once the change is the +intended one, `make stability-bless` accepts it (fake embedder + committed vault only; a +real-model or private-vault baseline would commit a number nobody else can reproduce). +Editing `fixtures/test-vault` or `stability.json` invalidates the baseline: re-bless in the +same commit. + +Flags: `ARGS=--verbose` prints the diverging lists; `ARGS=--model` runs real bge (magnitude +only, no baseline); `ARGS="--vault "` probes any vault. +`--vault crates/b2-embed/evals/corpus` is the control experiment where every prefix holds: +that *is* [#141](https://github.com/AlteredCraft/B2/issues/141)'s blindness, demonstrated. + +--- + +## `make calibrate`: the real-vault transfer check + +Process rule 5 made mechanical. Run it against any built vault: no labels needed, a pure read +over stored vectors, seconds even on a large vault. This is the instrument that retired the +existence gate ([#196](https://github.com/AlteredCraft/B2/issues/196) measured 16 of 17 panes +dark on a real single-domain vault) and killed the `df`-ceiling evidence rule. Any future +distributional constant answers to it **before** shipping. + +Per-anchor columns: pool size `n`; the pool's cosine `min/med/max`; the leader's cosine and +z; `gate serves`, what the retired z gate would serve (`DARK` means it would empty this pane; +a simulation, since the shipped surface gates nothing); `fold`, the mutual-k reciprocity +fold's default view ([#200](https://github.com/AlteredCraft/B2/issues/200)'s candidate 1, +replayed); `e-bar`, the authored-edge reference bar's fold (candidate 2: this is the only +instrument that can price it, since its calibration population is your own committed edges +and both eval corpora are link-free); `bands`, the strength-band histogram the pane would +paint. The summary block repeats each as a vault-level reading, plus the engine-z drift +`[check]`. + +Interpretation: a replayed gate darkening panes on a vault you know is inter-related is the +[#196](https://github.com/AlteredCraft/B2/issues/196) failure reproduced. An `e-bar` at 0 or +`UNPRICEABLE` is a reading about the vault (no scorable authored edges), not an instrument +failure. A band histogram compressed into `●○○` on a dense vault is the open A6 thread (band +compression), not a bug. A fake-embedded vault warns and prints noise: reindex with the real +model first. + +`ARGS=--search` adds the search-side transfer bench +([#201](https://github.com/AlteredCraft/B2/issues/201)): first the vault's function-word +weights beside an absent word's (the lexical anchor's whole premise, checked per vault rather +than assumed of English), then the shipped evidence bar replayed over every note's own +**title** (positives by construction: nothing to relabel) and built-in nonsense (the +vault-independent negatives), the ten lowest-cosine positives (where a bar placement bites +first), and the per-hit tail families priced on the one row a title query certifies with no +label, its own note. `the bar would cut N/…` is the tripwire direction: D2 permits zero. This +is the one part of the instrument that is not a pure read (judging the cosine half embeds +each probe query, so it loads the model); it has no `--json` form yet +([#219](https://github.com/AlteredCraft/B2/issues/219)). + +Other flags: `--limit N` (pane depth); `--leader-z`/`--member-z` (replay a different gate; +defaults are the retired constants, so +[#196](https://github.com/AlteredCraft/B2/issues/196)'s dark-vault reading reproduces); +`--mutual-k N`; `--json` (the discovery reading as one object, for scripting sweeps). + +--- + +## `make eval-chat`: the grounded-chat scores + +Needs a model server (`ollama serve` + `ollama pull llama3.2`, or `B2_LLM_URL` / +`B2_LLM_MODEL` at any OpenAI-compatible endpoint). Builds a throwaway vault from the +retrieval eval's corpus (chat is scored over notes whose retrieval behavior is already +characterized, so a surprise is attributable) and asks the labelled questions in +[`questions.json`](../crates/b2-llm/evals/questions.json) through the real `Vault::ask`. +Unlike `queries.json`, questions do **not** avoid the target's vocabulary: this scores what +the model does with retrieved passages, not what retrieval finds. + +Four deliberately separable scores, because a bad answer has more than one possible author: + +| Line | Meaning | A miss is… | +|---|---|---| +| **retrieval reach** | did a labelled note make it into the passages at all? | `make eval`'s result, not chat's: the ceiling every other score is judged under | +| **citation accuracy** | did the answer cite the labelled note? (scored only over reached questions) | the headline chat number | +| **grounding rate** | did the answer cite *anything*? | an uncited answer is a refusal or general knowledge, and the prompt forbids the second | +| **refusal accuracy** | on the two deliberate negatives, did it say "I don't find that in your notes"? | a confabulation. One negative is near the corpus's coverage on purpose, because models confabulate from loosely related passages | + +Per-question verdicts: `cited the note` / `refused (correct)` are the good ends; +`retrieval missed (not a chat result)` blames the other half; `MISCITED`, +`REFUSED WITH EVIDENCE PRESENT`, and `CONFABULATED` are the model's failures, with +hallucinated `[n]` markers (markers naming no passage) counted beside them. TTFT and total +latency print per question. + +**The exit code is a liveness check, not a quality bar**: `2` only when retrieval reached +labelled notes and *no* answer cited one, a broken pipeline. Model quality is read off the +numbers; a gate that fails on every small local model would be a gate nobody runs. Known +coverage gap: every question is single-turn, so the condense step is unmeasured +([#220](https://github.com/AlteredCraft/B2/issues/220)). + +--- + +## `results.jsonl`: the run log + +Both harnesses append one JSON line per scored run to their `evals/results.jsonl` +(gitignored, machine-local: scores depend on the machine's models). Every number ever cited +in an issue traces to a row here. Conventions: + +- **Append-only**, so runs accumulate into one comparable dataset. +- **Keys are additive, never redefined.** A metric that changes meaning gets a *new* key; the + old one goes absent, so a reader of a mixed file sees a missing field rather than one + number silently meaning something narrower than it used to. (Retired keys so far: + `discovery_z.shipped`/`replay_faults` with + [#197](https://github.com/AlteredCraft/B2/issues/197); + `similar_per_mate_raw`/`similar_mates_suppressed` with + [#217](https://github.com/AlteredCraft/B2/issues/217).) +- **`"corpus"` tags every row** (`"orthogonal"` / `"dense"`). Rows are never averaged across + corpora, and the dense row's transfer reading lives under its own `search_transfer` key + rather than overloading `search_evidence`. +- **Re-derivability.** The calibration subtrees (`discovery_z`, `discovery_fold`, + `search_evidence`, `search_tail`, `search_transfer`) record the raw per-candidate / + per-row data, so any window, fold depth, or bar can be re-derived from a row without + re-running the model. +- `pool_blind: true` marks a row no candidate-width comparison may be read from. + +A few readers: + +```console +# the latest orthogonal default-config row +jq -s '[.[] | select(.corpus == "orthogonal" and .config.label == "default")] | last' results.jsonl + +# hybrid note MRR across runs +jq -r 'select(.corpus == "orthogonal") | [(.ts | todate), .git, .config.label, .note.hybrid.mrr] | @tsv' results.jsonl + +# per-mate discovery MRR, both corpora +jq -r '[(.ts | todate), .corpus, .similar_per_mate.mrr] | @tsv' results.jsonl + +# which queries missed hybrid rank 1 in a given row +jq -r 'select(.corpus == "orthogonal") | .queries[] | select(.hybrid != 1) | [.q, (.hybrid // "miss")] | @tsv' results.jsonl +``` + +## The corpora and labels (ground truth) + +| Path | Role | +|---|---| +| [`corpus/`](../crates/b2-embed/evals/corpus/) | the hand-written **orthogonal** vault (31 notes): topic clusters, six long multi-chunk notes, five unambiguous loners, the stemmer-adversarial block, the [#183](https://github.com/AlteredCraft/B2/issues/183) multi-topic family, and `week-log.md`, the journal-shaped dilution extreme ([#189](https://github.com/AlteredCraft/B2/issues/189)/[#192](https://github.com/AlteredCraft/B2/issues/192)). Its token audit *minimizes* shared vocabulary, which is exactly why it cannot express topical concentration | +| [`queries.json`](../crates/b2-embed/evals/queries.json) | retrieval labels: 44 positives (20 with a chunk-level `passage`; three date-shaped, [#202](https://github.com/AlteredCraft/B2/issues/202)) + 5 **negatives** (empty `relevant` means the labelled answer is *no matches*). Seven positives carry `tail_relevant` ([#206](https://github.com/AlteredCraft/B2/issues/206)): the per-hit keep-set, exhaustive by label and encoded by note, never by rank. Its `description` field is the labelling rulebook; read it before editing | +| [`similar.json`](../crates/b2-embed/evals/similar.json) | discovery labels: positive anchors with expected mates; empty `expected` marks a negative anchor (a loner whose correct answer is *nothing*). Its `description` is the loner-orthogonality rulebook | +| [`corpus-dense/`](../crates/b2-embed/evals/corpus-dense/) + [`similar-dense.json`](../crates/b2-embed/evals/similar-dense.json) | the **dense single-domain fixture** (15 beekeeping notes, all inter-related, no loner): the vault-level geometry the orthogonal corpus is structurally incapable of expressing, and the bench that killed the existence gate and the `df` ceiling. Rankings-only labels; scored in its own vault and row | +| [`stability.json`](../crates/b2-embed/evals/stability.json) + [`stability-baseline.json`](../crates/b2-embed/evals/stability-baseline.json) | the unlabelled probe set and its blessed ranking snapshot (fake embedder over `fixtures/test-vault`) | +| [`questions.json`](../crates/b2-llm/evals/questions.json) | the chat set: questions phrased as a person would type them, `expect` names the note(s) a correct answer must cite; empty `expect` means the only correct answer is the refusal | +| `results.jsonl` (both harnesses) | append-only run logs, gitignored | + +## The verdict record + +Every ruling this suite has produced, one line each. The issue holds the full argument and +the numbers, and the commit that shipped it is the record of the change. + +| Issue | Verdict | +|---|---| +| [#44](https://github.com/AlteredCraft/B2/issues/44) | `ChunkConfig::default()` held against a seven-variant sweep; the "winning" rows were impeached by a measured boundary-luck noise floor. Retrial is one `make eval-sweep` away | +| [#141](https://github.com/AlteredCraft/B2/issues/141)/[#142](https://github.com/AlteredCraft/B2/issues/142) | the labelled corpus is near-blind to candidate width; a 3× pool widening was reverted on the stability probe's evidence | +| [#150](https://github.com/AlteredCraft/B2/issues/150) | the z-score discovery quality floor shipped: per-anchor z over the centroid population, suppression entering the gate; **superseded by [#197](https://github.com/AlteredCraft/B2/issues/197)** | +| [#156](https://github.com/AlteredCraft/B2/issues/156) | RRF fused-score ties break on the dense signal's rank: a policy the eval decided, not walk order | +| [#157](https://github.com/AlteredCraft/B2/issues/157) | `porter unicode61` FTS stemming: 7–0 BM25 / 3–0 hybrid on the paired readout, precision probes unmoved; the unstemmed arm stays measurable via `make eval-stemmer` | +| [#183](https://github.com/AlteredCraft/B2/issues/183) | the multi-topic note family landed, and with it the **per-mate** metric: the per-anchor one saturates at 1.000 and hides every hard mate | +| [#187](https://github.com/AlteredCraft/B2/issues/187) | the member window is empty on the shipped corpus's own numbers: mates and strangers invert, so no constant separates them; windows are re-derived every run, never frozen in a docstring | +| [#189](https://github.com/AlteredCraft/B2/issues/189) | the journal-shaped note was built, measured, and **rejected**: the first corpus edit this harness refused (its diluted centroid tops loner anchors on content it doesn't contain) | +| [#192](https://github.com/AlteredCraft/B2/issues/192) | the floor moved to the stage-2 best-passage unit, and `week-log.md` landed: read correctly in both directions at once | +| [#182](https://github.com/AlteredCraft/B2/issues/182) | the buried gem is served, and the desktop's strength bands were re-read in the judged unit (the last surface still reading the retired one) | +| [#188](https://github.com/AlteredCraft/B2/issues/188) | discovery rank entered the exit gate; the harness measured bit-reproducible run-to-run, so floors are sized for corpus drift, not noise | +| [#196](https://github.com/AlteredCraft/B2/issues/196)/[#197](https://github.com/AlteredCraft/B2/issues/197) | the existence gate itself was the defect: a single-domain vault went 16/17 dark while the ranking was correct throughout. The gate retired; discovery **always serves the ranked list** (ADR-0014); `make calibrate` and the dense fixture landed *before* the fix, per the sequencing rule | +| [#200](https://github.com/AlteredCraft/B2/issues/200) | the discovery fold bake-off ran and **no fold ships**: mutual-k's admissible window is empty, and its safe depth is a different rule on every vault scale; the authored-edge bar is unpriceable on link-free corpora and stays open | +| [#201](https://github.com/AlteredCraft/B2/issues/201) | search's evidence bar was **earned**, and its first form (a hard `df` ceiling) read clean on the labelled bench and cut 3/15 answerable queries on the dense one, so the *rule* changed (IDF as a weight, not a bin), not the constant | +| [#202](https://github.com/AlteredCraft/B2/issues/202) | the verdict reached the surfaces ("no matches" is a real answer; `--json` became an object), the date-shaped block landed, and search's three structural-zero rows entered the gate | +| [#206](https://github.com/AlteredCraft/B2/issues/206) | the per-hit tail bake-off ran and **no tail fold ships**: 367 of 386 filler rows sit below the oracle fold, but the fused order misplaces the evidence, so every admissible prefix cut is near-vacuous; the tail complaint is ordering work (the reranker seam) | +| [#217](https://github.com/AlteredCraft/B2/issues/217) | the pass-vs-pass suppression tripwire compared a call against itself and could never fire; removed. The dense pane assertion and the per-mate floors are what catch a returning existence gate | + +**Standing principles from the record**: when a label and a note disagree, fix whichever one +is lying (the watercolor → throat-singing precedent); an arguable negative is replaced, not +argued; instruments land *before* the rules they price; and a reading that disqualifies a +rule is re-derived every run, never frozen into a comment. + +**The deliberately open threads**: the **phishing pair** (a real relation ranked under three +stranger pairs even in the best-passage unit; served, so ordering residue, the pair-scorer's +standing evidence); the **tail's 367 rows of headroom** (the second exhibit for a reranker, +measured on search's flow); and the **dense-vault band compression** (#197's A6: on a vault +where every candidate is close, the within-list z compresses and the dots lose resolution; +first read on a real-embedded build of the dense fixture: 0 ●●● / 6 ●●○ / 144 ●○○). + +## Process rules + +Adopted 2026-08-10. Each traces to a measured mistake or a named risk, and each is binding on +anyone editing the corpora, the labels, or the metrics. + +1. **A paired per-query win/loss list is the primary readout of any A/B; the aggregate is a + smoke alarm.** At n≈40, every aggregate point is 1 or 2 queries. "hit@1 +0.05" and "these + two flipped" are the same fact, but only the second can be argued against the labels. The + sweep prints the diff (`Δ vs default`) automatically. +2. **A corpus edit is a change to the instrument, so it ships as its own commit** whose + message says what changed and why, and every edit runs the **two-direction token audit** + before it lands: no existing query's content tokens may newly land in the edited or added + note, and no new query's content tokens may split evenly toward a rival. Word-boundary and + stem-prefix both (the `insomnia.md` steal and the `recover`+`mistake` near-miss are the + precedents). Run the audit, don't eyeball it + ([#218](https://github.com/AlteredCraft/B2/issues/218) tracks committing it as a script). + And, since the gate reads discovery rank, **a red gate is never an argument for editing a + label**: per-mate MRR and the strangers count both move when labels move, so the only + honest response to red is to argue about the *notes*. +3. **The same person authoring notes, queries, and fixes is a ratchet toward measuring what + the engine already does.** Mitigations in order of cheapness: rule 2's audit; sourcing + future queries from outside the corpus author's head (from note titles alone, or another + person); dogfooding on a real vault before trusting any threshold. +4. **A bit-identical or unmoved metric is a claim to verify, never proof of "no effect".** + Compare a continuous quantity (the piles) before believing a discrete one. (The + `prepend-heading-path` trace; the sweep diff prints its own reminder.) +5. **A constant derived from a corpus's score *distribution* is invalid until + transfer-checked on a real vault.** `make calibrate` is the check (adopted with + [#197](https://github.com/AlteredCraft/B2/issues/197), from a measured mistake made + twice). Rank-derived readings transfer, because the corpus's *orderings* are engineered to + be checkable; its score **distributions** are an artifact of engineered orthogonality, so + any threshold read off them (a cosine bar, a z window, a band landmark) describes the + corpus, not a vault. A distributional constant ships only with a `calibrate` reading from + at least one real vault beside the corpus numbers. From 258f63867487b79a2a601fc3b81175f28b3e5e49 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 05:30:42 +0000 Subject: [PATCH 4/6] =?UTF-8?q?docs:=20one=20docs/=20tree=20=E2=80=94=20de?= =?UTF-8?q?sign/=20and=20the=20HTML=20pages=20retire,=20references=20repoi?= =?UTF-8?q?nted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit design/ is gone: the specs live in docs/ now, re-authored. The six HTML pages retire in favor of the Markdown guides that replaced them. Every reference follows: the root README's docs table and quick-start links, CLAUDE.md's truth table, ADR-0001/0013 and the ADR index, Cargo.toml and Makefile comments, fixtures/README, the desktop CLAUDE.md, ui/ source comments, and eval.rs's notebook-of-record pointers. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CnZh7mztkPnvXJ4GnfrZiE --- ADRs/0001-design-docs-are-normative.md | 4 +- ...013-model-quality-is-measured-out-of-ci.md | 2 +- ADRs/README.md | 2 +- CLAUDE.md | 8 +- Cargo.toml | 8 +- Makefile | 2 +- README.md | 34 +- crates/b2-desktop/CLAUDE.md | 6 +- crates/b2-embed/examples/eval.rs | 8 +- design/data-model.md | 497 ------------ design/index-engine.md | 422 ---------- design/invariants.md | 321 -------- docs/architecture.html | 753 ------------------ docs/index.html | 210 ----- docs/indexing.html | 418 ---------- docs/quickstart.html | 566 ------------- docs/retrieval.html | 486 ----------- docs/search-and-similarity.html | 734 ----------------- fixtures/README.md | 2 +- ui/src/api.ts | 4 +- ui/src/bindings.ts | 2 +- ui/src/main.ts | 4 +- ui/src/settingstabs.ts | 2 +- ui/src/shortcuts.test.ts | 2 +- ui/src/shortcuts.ts | 2 +- ui/src/sidenav.ts | 2 +- ui/src/state.ts | 2 +- ui/src/treenav.ts | 2 +- ui/src/types.ts | 6 +- 29 files changed, 52 insertions(+), 4459 deletions(-) delete mode 100644 design/data-model.md delete mode 100644 design/index-engine.md delete mode 100644 design/invariants.md delete mode 100644 docs/architecture.html delete mode 100644 docs/index.html delete mode 100644 docs/indexing.html delete mode 100644 docs/quickstart.html delete mode 100644 docs/retrieval.html delete mode 100644 docs/search-and-similarity.html diff --git a/ADRs/0001-design-docs-are-normative.md b/ADRs/0001-design-docs-are-normative.md index 5975c40..a23165c 100644 --- a/ADRs/0001-design-docs-are-normative.md +++ b/ADRs/0001-design-docs-are-normative.md @@ -1,7 +1,7 @@ # ADR-0001 — Design docs are normative; the code is a projection of them - **Status:** Accepted · 2026-06-28 -- **Refs:** `design/invariants.md` +- **Refs:** `docs/invariants.md` ## Context @@ -10,7 +10,7 @@ redefine what the product promises. Prose that merely *describes* the code canno ## Decision -- `design/invariants.md` is the **normative register** — one testable claim per entry, cited +- `docs/invariants.md` is the **normative register** — one testable claim per entry, cited by id (S2, G2, D1…). **On conflict with any other doc or with the code, the register wins** and the other side gets fixed. - `data-model.md` is the *what*, `index-engine.md` the *how*. Code comments cite them by section. diff --git a/ADRs/0013-model-quality-is-measured-out-of-ci.md b/ADRs/0013-model-quality-is-measured-out-of-ci.md index abe902b..eaf14ed 100644 --- a/ADRs/0013-model-quality-is-measured-out-of-ci.md +++ b/ADRs/0013-model-quality-is-measured-out-of-ci.md @@ -1,7 +1,7 @@ # ADR-0013 — Model quality is measured out of CI, by a labelled harness - **Status:** Accepted · 2026-07-13 -- **Refs:** invariants E2 · `crates/b2-embed/evals/README.md` (the process rules) · GH #44, #141, #187 +- **Refs:** invariants E2 · `docs/evals.md` (the process rules) · GH #44, #141, #187 ## Context diff --git a/ADRs/README.md b/ADRs/README.md index 6a47b39..38145c5 100644 --- a/ADRs/README.md +++ b/ADRs/README.md @@ -7,7 +7,7 @@ an ADR. Feature history lives in [GitHub Issues](https://github.com/AlteredCraft **Keep them terse** — Context / Decision / Consequences, a page at most. When a decision is overturned, mark the old ADR `Superseded by ADR-NNNN` and write a new one; never edit history. -Normative detail lives in `design/invariants.md` (the register — it wins on conflict); these +Normative detail lives in `docs/invariants.md` (the register — it wins on conflict); these records say *why* an entry reads the way it does. | # | Decision | diff --git a/CLAUDE.md b/CLAUDE.md index a3ee53d..750604d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,11 +16,11 @@ constantly (`data-model.md §2`, `index-engine.md §6`, invariant ids like `S2`, | Source | Role | |---|---| -| [`design/invariants.md`](design/invariants.md) | The **invariant register** — the normative list of what must always be true, cited by id. **On conflict with any other doc, it wins.** | -| [`design/data-model.md`](design/data-model.md) | The *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary. | -| [`design/index-engine.md`](design/index-engine.md) | The *how*: the SQLite (FTS5 + in-process vector scan) projection, table DDL, data flows. | +| [`docs/invariants.md`](docs/invariants.md) | The **invariant register** — the normative list of what must always be true, cited by id. **On conflict with any other doc, it wins.** | +| [`docs/data-model.md`](docs/data-model.md) | The *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary. | +| [`docs/index-engine.md`](docs/index-engine.md) | The *how*: the SQLite (FTS5 + in-process vector scan) projection, table DDL, data flows. | | [`ADRs/`](ADRs/README.md) | **Architecture Decision Records** — why each of the above reads the way it does. Key architectural choices only, terse. Add one when a decision is expensive to reverse and its *why* isn't readable off the code; do **not** add one per feature or bug. | -| [`crates/b2-embed/evals/README.md`](crates/b2-embed/evals/README.md) | The **eval suite guide** — every instrument, how to read its output, the exit gate, the verdict record, and the **process rules**; read before touching the corpus, the labels, or the metrics. | +| [`docs/evals.md`](docs/evals.md) | The **eval suite guide** — every instrument, how to read its output, the exit gate, the verdict record, and the **process rules**; read before touching the corpus, the labels, or the metrics. | | [GitHub Issues](https://github.com/AlteredCraft/B2/issues) | Backlog and planned work. Decision history = the issue that drove a verdict + the commit that shipped it. | ## Commands diff --git a/Cargo.toml b/Cargo.toml index 8ab07d2..64fdc5d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,9 +9,9 @@ members = [ ] # B2 — local-first Markdown PKM with AI connection discovery. -# Design lives in design/; this workspace is the index engine -# (design/index-engine.md), built step 0→5 against the -# golden-vault fixtures in fixtures/golden-vault/ (design/data-model.md §8). +# Design lives in docs/; this workspace is the index engine +# (docs/index-engine.md), built step 0→5 against the +# golden-vault fixtures in fixtures/golden-vault/ (docs/data-model.md §8). # Build DEPENDENCIES optimized even in dev/debug builds, while keeping our own `b2-*` # crates at opt-level 0 so the TDD loop and `cargo test -p b2-core` stay fast. Candle's @@ -19,7 +19,7 @@ members = [ # matmul backend) is ~13× slower unoptimized, so a plain `tauri dev` / `cargo run` # reindex was painfully slow (a 16-chunk embed batch took ~35s instead of ~2.5s). That # mattered doubly for the desktop app: a reindex cancel is only observed at each embed- -# batch boundary (design/index-engine.md), so a slow batch made the +# batch boundary (docs/index-engine.md), so a slow batch made the # **Cancel** button feel stuck. The `"*"` glob optimizes every dependency (targeting # just the candle crates missed the matmul backend and left it ~4× slower); the per- # crate opt-0 overrides below exclude our own crates so their rebuilds stay instant. diff --git a/Makefile b/Makefile index 510ed5a..27c14d6 100644 --- a/Makefile +++ b/Makefile @@ -250,7 +250,7 @@ coverage-app: ui-build ## Coverage for the desktop host's own unit tests. # *sensitivity* rather than model quality, so it is machine-independent (fake embedder) and # needs no model — but it is the other half of the same harness (GH #141). # -# **crates/b2-embed/evals/README.md is the guide**: what each instrument measures, how to +# **docs/evals.md is the guide**: what each instrument measures, how to # read every block it prints, the exit gate, and the process rules that bind any edit to the # corpora, the labels, or the metrics. The recipe comments here stay operational only. diff --git a/README.md b/README.md index f09366f..6915e36 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ explained connections between them yourself. > the fake stays the CI default). **Connection discovery** ships as **`b2 similar`** (surface the > nearest *unlinked* notes — local, free, no model call) **+ `b2 link`** (you commit a typed relation > to frontmatter) — the human is the precision gate; there is no LLM in the loop. A tour -> grounded in the test suite: [docs/architecture.html](docs/architecture.html). +> grounded in the test suite: [docs/architecture.md](docs/architecture.md). > > **Grounded chat is live in the CLI** — **`b2 ask "…"`** and **`b2 chat`** answer questions *from your > notes*, streaming, with `[n]` citations back to the notes the answer came from @@ -53,12 +53,12 @@ connected yet** — so the structure of your knowledge grows as you link them, i The files stay plain Markdown on your disk, yours forever; B2 is the **intelligence layer over them, not a container around them**. Humans and AI agents are both first-class users. -Full motivation, scope, and locked decisions: **[design/invariants.md](design/invariants.md)**. +Full motivation, scope, and locked decisions: **[docs/invariants.md](docs/invariants.md)**. ## How we build it Two architectural tenets shape every decision (full text: -[design/invariants.md](design/invariants.md)): +[docs/invariants.md](docs/invariants.md)): - **A volatile vault over a disposable index.** Refactor fearlessly — move, split, merge, compress, trim orphans. The index is a pure projection of your vault (drop it, rebuild it identical); @@ -70,26 +70,26 @@ Two architectural tenets shape every decision (full text: …in service of five product non-negotiables — plain-Markdown source of truth · local-first · zero lock-in · AI-native (not bolted-on) · single binary -([design/invariants.md](design/invariants.md)). +([docs/invariants.md](docs/invariants.md)). ## The docs -### HTML guides — [alteredcraft.github.io/B2](https://alteredcraft.github.io/B2/) - -New here? Start with the **[Quick start](https://alteredcraft.github.io/B2/quickstart.html)** — set up -and work with a vault in about ten minutes. Then go deeper: -[system architecture](https://alteredcraft.github.io/B2/architecture.html) · -[indexing pipeline](https://alteredcraft.github.io/B2/indexing.html) · -[the retrieval deep dive](https://alteredcraft.github.io/B2/retrieval.html) · -[search & similarity, in plain language](https://alteredcraft.github.io/B2/search-and-similarity.html). +Everything lives in [docs/](docs/README.md) — one page per topic, and that page is the map. +New here? Start with the **[Quick start](docs/quickstart.md)** — set up and work with a vault +in about ten minutes. Then go deeper: +[architecture](docs/architecture.md) · +[search & similarity, in plain language](docs/search-and-similarity.md). | Doc | What it owns | |---|---| -| [design/invariants.md](design/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, and the source of *why*, cited by id. On conflict with any other doc, it wins. | -| [design/data-model.md](design/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. | -| [design/index-engine.md](design/index-engine.md) | How the derived index is *built* — SQLite (FTS5 + an in-process vector scan) as a disposable projection. The canonical *how*. | +| [docs/invariants.md](docs/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, and the source of *why*, cited by id. On conflict with any other doc, it wins. | +| [docs/data-model.md](docs/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. | +| [docs/index-engine.md](docs/index-engine.md) | How the derived index is *built* and queried — SQLite (FTS5 + an in-process vector scan) as a disposable projection, and the four flows over it. The canonical *how*. | +| [docs/quickstart.md](docs/quickstart.md) | Set up and use B2: the walkthrough, the command reference, config and every environment variable. | +| [docs/architecture.md](docs/architecture.md) | How the system is built: the crates, the flows, the seams, the tests. | +| [docs/search-and-similarity.md](docs/search-and-similarity.md) | What search and the related-notes panel do, in plain language, for everyone who uses B2. | | [ADRs/](ADRs/README.md) | **Architecture Decision Records** — one terse record per key architectural choice: the context, the ruling, and what it costs. The *why* behind the register's entries. | -| [crates/b2-embed/evals/README.md](crates/b2-embed/evals/README.md) | The **eval suite guide** — every instrument and how to read it, corpora, labels, the exit gate, process rules, and the record of every measured verdict. Lives beside the corpus it governs. | +| [docs/evals.md](docs/evals.md) | The **eval suite guide** — every instrument and how to read it, corpora, labels, the exit gate, process rules, and the record of every measured verdict. | Planned work and the backlog live in [GitHub Issues](https://github.com/AlteredCraft/B2/issues); shipped build history lives in git. @@ -172,4 +172,4 @@ Point B2 at a vault with `-C ` (a.k.a. `--vault`) on any command, or set ` every command finds it without the flag (an explicit `-C` wins). Read-only commands (`search`, `neighbors`, …) fall back to the current dir; commands that write (`reindex`, `add`, `mv`, `link`) require an explicit vault and refuse otherwise, so they can't silently touch the wrong place. Full walkthrough: -**[Quick start](https://alteredcraft.github.io/B2/quickstart.html)**. +**[Quick start](docs/quickstart.md)**. diff --git a/crates/b2-desktop/CLAUDE.md b/crates/b2-desktop/CLAUDE.md index 4a3772a..8f8bb89 100644 --- a/crates/b2-desktop/CLAUDE.md +++ b/crates/b2-desktop/CLAUDE.md @@ -40,7 +40,7 @@ what makes that architecture pay off: - **Inherited tests.** A thin host means the façade's existing suite already covers the behavior; this crate needs only a few per-command tests (args in → right façade call → view out). Logic here would need its own parallel tests that the CLI already has. -- **The promise stays true.** [invariants.md](../../design/invariants.md) (E3) says the GUI is "a +- **The promise stays true.** [invariants.md](../../docs/invariants.md) (E3) says the GUI is "a second dumb adapter over the same contract, inheriting every test the CLI bought." That is only true while this crate stays dumb. Thinness is not tidiness; it's the load-bearing property. @@ -108,7 +108,7 @@ add a UI concern to `b2-core`, that's the signal you're putting logic in the wro ## The keyboard contract (invariant K1) -[invariants.md](../../design/invariants.md) **K1** — *B2 is fully operable from the keyboard; the +[invariants.md](../../docs/invariants.md) **K1** — *B2 is fully operable from the keyboard; the mouse is an accelerator, never a requirement* — names this file as its elaboration home. This is it. K1 governs the **GUI**: the `b2` CLI satisfies it by nature, so everything below is about `b2-desktop` + [`ui/`](../../ui) ([#78](https://github.com/AlteredCraft/B2/issues/78)). @@ -293,7 +293,7 @@ Every new surface owes all four. They are cheap while you're building it and exp ## The rendering trust boundary (invariant E5) -[invariants.md](../../design/invariants.md) **E5** — *note content is untrusted input; rendering is a +[invariants.md](../../docs/invariants.md) **E5** — *note content is untrusted input; rendering is a trust boundary* — names this file as its elaboration home, the way K1 does above. E5 governs the **GUI**: the `b2` CLI prints text, so nothing there parses into a document ([#77](https://github.com/AlteredCraft/B2/issues/77)). diff --git a/crates/b2-embed/examples/eval.rs b/crates/b2-embed/examples/eval.rs index dbcdb8f..b9a15c3 100644 --- a/crates/b2-embed/examples/eval.rs +++ b/crates/b2-embed/examples/eval.rs @@ -8,7 +8,7 @@ //! cargo run -p b2-embed --example eval -- --stemmer # + FTS tokenizer A/B (the #157 gate) //! ``` //! -//! **`crates/b2-embed/evals/README.md` is the notebook of record** — the corpus, what the exit code +//! **`docs/evals.md` is the notebook of record** — the corpus, what the exit code //! enforces, every verdict this harness has ruled, and the process rules. Read it before //! touching the corpus, the labels, or a constant here. What this comment carries is only //! what a reader of *this file* needs: @@ -824,7 +824,7 @@ fn run() -> Result> { ); // The readout the A/B is actually judged on: at this n every aggregate // delta above is 1–2 queries, so the aggregate is a smoke alarm and the - // per-query win/loss list is the data (crates/b2-embed/evals/README.md, the + // per-query win/loss list is the data (docs/evals.md, the // process rules). print_rank_moves(&positives, &hybrid, &pass); append_result( @@ -3717,7 +3717,7 @@ fn print_tail_join(ev: &SearchEvidence, orth: &TailBench, titles: &[SearchProbe] (zero headroom — the constant placement the house sizing method forbids), each payoff \ reads against the oracle ceiling above, and a shipped constant owes process rule 5's \ real-vault reading besides (`make calibrate VAULT= ARGS=--search`, the tail block). The ruling of \ - record lives in crates/b2-embed/evals/README.md." + record lives in docs/evals.md." ); } else { println!( @@ -4151,7 +4151,7 @@ fn band_glyph(z: f64) -> &'static str { /// judged on. At this corpus's n, every aggregate delta is worth 1–2 queries, so /// "hit@1 +0.05" and "these two queries flipped, this one broke" are the same /// fact — but only the second form can be argued with, per-query, against the -/// labels (crates/b2-embed/evals/README.md, the process rules). Prints nothing but a +/// labels (docs/evals.md, the process rules). Prints nothing but a /// no-moves line when the variant reproduced the reference ranking exactly — /// which, per the same rules, is itself a claim to verify against a /// continuous quantity (the piles), never bare proof of "no effect". diff --git a/design/data-model.md b/design/data-model.md deleted file mode 100644 index 3c694a5..0000000 --- a/design/data-model.md +++ /dev/null @@ -1,497 +0,0 @@ ---- -title: "B2 — Data Model" -type: note -tags: [b2, data-model, frontmatter, typed-links, edges, resources, okf] -created: 2026-06-29 -status: active ---- - -# B2 — Data Model - -> Defines **what a note is** and **what a connection is**, as the plain-Markdown source of truth — -> engine-independent. This is the yardstick the engine measures against: the SQLite schema in -> [index-engine.md](index-engine.md) §3 is a *derived projection* of this model and must satisfy it, -> never the reverse. Companion docs: [invariants.md](invariants.md) (the normative register, cited by -> id) and [index-engine.md](index-engine.md) (the *how*). -> -> **Rationale lives in [`ADRs/`](../ADRs/README.md)** — why connections live where they do -> (ADR-0010), why B2 never authors the body (ADR-0004), why identity is the path (ADR-0003), why -> there is no suggestion queue (ADR-0009). This doc defines the shapes; the ADRs say why. - -The model has exactly **two source-of-truth objects**, both plain Markdown: - -1. **A note** — one `.md` file: YAML frontmatter + a Markdown body. -2. **A connection (edge)** — a directed link from one note to another: a plain link a human writes in - the body, or a typed relation in frontmatter `b2_relations:` (§0). - -Both are **authored**. A real vault also holds **resources** — every non-`.md` file. A resource is a -**peer vault member**, *not* a third authored object: B2 can *read* it but cannot author structure -into it, because Markdown is the only format whose bytes B2 may write. So the source *tier* is **the -whole vault directory** while the two authored objects stay note + edge. Resources are defined in -**§10**; §0–§9 are unchanged by them. - -### Two storage tiers - -1. **Markdown — source of truth for *knowledge*.** Notes + every committed edge, on your disk, fully - usable with no B2. Stays **pristine**, and the **body is 100% the human's** (W2). The enumerated - on-command writes are W3; **reading your vault writes nothing at all** (W1). -2. **Index (`b2.sqlite`) — disposable cache.** The search indexes and the keyed graph. Holds - **nothing** that can't be reconstructed from the Markdown. - -> **Two tiers, sharply split** (S1); **index = projection of (the vault directory)** (S2, ADR-0002). Drop `b2.sqlite` → re-derive → an -> identical index (S3). There is **no** durable B2-derived state outside your notes (S4). A -> **resource** (§10) contributes only *derived* rows, so the guarantee is unchanged. - -### Folders — user-authored structure, filesystem-authoritative - -The vault directory carries two kinds of authored material: the Markdown files (**content**) and the -directory tree itself (**structure**). A folder — *empty or not* — is user-authored exactly like a -note, and the **filesystem is authoritative** for it; B2 proxies the OS rather than modeling folders: -`create_dir` makes missing parents but refuses an occupied target (no `mkdir -p` idempotence — the -human asked to *create*), `move_dir` is one `rename`, `delete_dir` is `remove_dir_all`, and each -resolves its target against the *disk*, never the index, so empty folders are first-class throughout. -Folders are **never projected into the index** — they carry nothing to chunk, embed, or link — so the -tree's structure listing (`Vault::list_dirs`) is a **live fs walk** (dot-folders skipped, §1): the -tree is one-to-one with the vault's managed subtree *by construction*, in both directions. S4 scopes -to **B2-derived** data; the human's own structure is vault material, not B2 state. - ---- - -## 0. The central decision — where a connection lives - -Ruled by **[ADR-0010](../ADRs/0010-typed-graph-two-homes-closed-vocabulary.md)** and -**[ADR-0004](../ADRs/0004-markdown-is-the-only-surface-b2-writes.md)**. A connection lives in -exactly one of two homes, **by origin** — and the two homes split by *what they can say*, not just -who writes them: - -| Origin of the edge | Where it lives | SSOT | -|---|---|---| -| A plain body link | **Body** — a bare `[[path\|title]]`, a Markdown `[text](path)`, an embed — ordinary Markdown, always an untyped `references` edge | the body; B2 **reads**, never writes it | -| A typed relation (committed via `b2 link`, or human/importer-written) | **Frontmatter `b2_relations:`** — a typed-link string `- " [[path\|title]] — …"` (§2); the **only** home of a verb + explanation | frontmatter; B2's managed metadata zone | - -**`b2 link` writes frontmatter, not body.** Committing appends one typed-link string to the source -note's `b2_relations:` (Markdown first, index reconciled after). The edge then materializes as an -`origin='frontmatter'` edge derived from that Markdown — committing is the projection of an authored -line, not a bespoke index write (§3). - -> One line: **the body holds the plain links the human writes (all `references`); frontmatter -> `b2_relations:` holds every *typed* relation — verb and explanation live only there; both are -> authored Markdown, and the graph is their union** (G2). - -Each edge has exactly **one** home and B2 never copies between them — so there is nothing to keep "in -sync," only a one-way projection to rebuild. A `b2_relations:` entry may deliberately target a note the -body already links: that is the **augment** flow (§2). The overlap case — the *same* `(target, type)` -in both homes (necessarily `references`, the only type a body link can carry) — is resolved at -projection time by **frontmatter-wins** dedup: the frontmatter row is kept (it alone can carry an -explanation), the redundant body reference ignored as a duplicate, never auto-removed from the file. - ---- - -## 1. The note - -A note is one `.md` file **whose vault-relative path has no dot-prefixed segment**: YAML frontmatter, -then a Markdown body. (Any segment — an ancestor folder counts, so `notes/.templates/daily.md` is no -more a note than `.scratch.md` is.) - -```markdown ---- -type: concept # optional, defaults to `note`; OKF entity discriminator -title: "Spaced repetition" # optional, INERT — the title is the filename (L2) -description: "Why expanding intervals beat massed practice." -tags: [learning, memory] -created: 2026-06-20 -updated: 2026-06-29 -aliases: [SRS] # optional Obsidian-native extra titles -b2_relations: # B2's managed zone: typed edges (§2). origin=frontmatter - - "contradicts [[notes/cramming-works|Cramming works]] — short-term recall only" -provenance: # optional; defaults to {by: human} - by: human ---- - -Spaced repetition schedules reviews at expanding intervals… - -It builds on [[concepts/memory|Human memory]] — the forgetting curve is the mechanism. -``` - -The body link is **human-authored** (`origin=inline`) and untyped — an ordinary `references` edge; the -surrounding prose is just prose. The `b2_relations:` entry is a *typed* edge (`origin=frontmatter`). - -### Hidden means hidden — a dot-prefixed name is not vault material - -A dot-prefixed name is outside the managed subtree **whatever it is**: a folder (`.git/`, -`.obsidian/`, B2's own `.b2/`), a resource (`.DS_Store`), or a Markdown file (`.scratch.md`). The -convention is the filesystem's, not B2's, so B2 reads it the same way everywhere (S2, -[GH #136](https://github.com/AlteredCraft/B2/issues/136)): - -- **The walk skips it before it routes it** (`pathspec::is_hidden`, applied *above* the note/resource - dispatch in `collect_vault_files`), so "hidden" cannot mean one thing for a PDF and another for a - Markdown draft. A dot-prefixed `.md` is never a note: no chunks, embeddings, graph presence, - listing, search, or file-tree appearance. -- **The file itself is untouched** — skipping is not deleting (W4). That is the point: keep a scratch - draft out of B2's way without leaving the folder you work in. -- **B2 will not author one either.** Every authoring destination refuses a dot-prefixed segment: - creating a member the walk would never see is a silent fs/index desync. - -*Migration note:* rows for a previously-indexed dot-prefixed `.md` are ghost-pruned on the next -reindex and inbound links re-dangle (G5); renaming it back restores it exactly. - -### Frontmatter schema - -**Required: nothing.** A note is a `.md` file; that is the whole entry requirement. Identity is the -**vault-relative path** (L1, ADR-0003) — not a key in the file — so there is no frontmatter B2 needs -present, none it will add, and a note with no frontmatter block at all is an ordinary, fully-indexed -note. - -**Optional (B2-recognized)** - -- **`type`** — what *kind* of note this is (`note`, `concept`, `source`, `person`, `daily`, …). - Controlled-but-extensible; unknown values tolerated; the OKF entity discriminator (§5). **Defaults - to `note`** — its only consumer is display, so nothing keys on its presence and the new-note - template does not seed it (GH #80: the template stamps only what can't be reconstructed later). Not - `b2`-namespaced on purpose — a courtesy the human owns, not a key B2 machines on. -- **`title`** — **recognized but inert** (L2). A note's title **is its filename** (basename minus - `.md`); the key is parsed and round-tripped losslessly like any other, and never drives display, - link aliases, or search. -- **`description`** — one-line summary; feeds the embedding prompt and OKF export. -- **`tags`** — list of strings. -- **`created` / `updated`** — ISO-8601 date or datetime. `created` is set by B2 at creation - (`b2 add`); `updated` is the human's (or another tool's) to maintain — B2 does not stamp it. -- **`aliases`** — Obsidian-native additional titles; treated as alternate link aliases. -- **`provenance`** — *optional, opt-in* note-level authorship: `{by: human | agent:, - source?, confidence?}`. Absent ⇒ `{by: human}`. B2 neither requires nor manages it. (Edges carry no - provenance — §4.) -- **`b2_relations`** — **B2's managed zone for typed edges** (§2): a YAML list of typed-link strings, - the **only** place a relation verb and explanation live. **Namespaced** — and B2's *only* key — so it - can never collide with a user's or another tool's `relations:` key; for the same reason a generic - un-namespaced `relations:` is *not* read (just another unknown key, preserved verbatim). B2 appends - here on `b2 link` (never the body); humans and importers may write it too. - -**Unknown keys** — preserved verbatim and byte-for-byte on round-trip (W5, §6). A `b2id:` line an -older B2 left behind is exactly that: an unknown key, never read, never removed (ADR-0003). - ---- - -## 2. Authored links & typed relations - -### Bare wikilink ⇒ an untyped `references` edge - -A normal `[[path|title]]` anywhere in prose is a connection of type **`references`**, `origin=inline` -— the untyped graph Obsidian already gives you, typed and materialized. It is **directed** (A→B — the -literal fact that A's text points at B), which preserves the backlink ↔ forward-link split -(`b2 neighbors` shows it as *referenced-by* from B's side), is the information-preserving default, and -keeps the explicit symmetric verb (`contradicts`) meaningful as a deliberate choice. - -### Frontmatter `b2_relations:` ⇒ a *typed* edge (`origin=frontmatter`) - -The typed-link syntax ` [[path|title]] — explanation`, as a **quoted string** in a -`b2_relations:` list — **the one and only home of a typed relation**. Optional trailing text after an -em-dash (or `:`) is the edge's **`explanation`**. - -```yaml -b2_relations: - - "supports [[concepts/forgetting-curve|Forgetting curve]] — the schedule exploits it" - - "contradicts [[notes/cramming-works|Cramming works]]" -``` - -- **Quoted** so `[[`, `|`, and `:` are always YAML-safe; the reader accepts quoted or unquoted. -- An entry that is just a bare `[[path|title]]` (no verb) is accepted and reads as `references`. -- Humans and importers may write this block too; B2 appends to it on `b2 link` and never authors the - body. - -### Typing a body link — frontmatter *augments* the body - -A `b2_relations:` entry may target a note the body already links. It **augments** that connection: the -body keeps its plain, clickable link exactly as written, and the frontmatter carries the stance. A -different verb (`supports [[x]]` over a body `[[x]]`) adds the typed edge alongside the untyped -reference — both are real, separately-authored facts. The same verb (`references [[x]] — why`) -collapses into one edge, **frontmatter-wins** (§0/§3), so the explanation survives. This is the -intended UI affordance: select a body link, choose a verb and optionally an explanation, and B2 appends -one `b2_relations:` entry — the body is never touched. - -### Relation vocabulary — a stance core + a tolerated tail - -Small, orthogonal, stable core; expressiveness in the tail (G3, ADR-0010). - -**The core (closed set — the `b2 link` palette, and what queries can rely on):** - -| Verb | Stance | Direction | Inverse (display only) | -|---|---|---|---| -| `references` | neutral | directed | referenced-by | -| `supports` | for | directed | supported-by | -| `contradicts` | against | symmetric | contradicts | - -*Boundary notes:* `references` is both the **automatic** type of a bare link and the deliberate "see -also"; `supports` is a **directed** "A backs B"; `contradicts` is a **deliberate symmetric** "these -state opposites" — tension has no aggressor, so no direction is recorded. - -**Extensibility:** the core is stable across versions. Any other verb a human writes (`elaborates`, -`part-of`, `supersedes`, …) is **tolerated and stored verbatim, never dropped**; tooling treats tail -verbs as opaque strings (no inverse label, no special traversal). A tail verb that proves common can be -**promoted** into the core later (gaining an inverse label); demotion is just removal from the palette, -stored data untouched. - -**Typing guidance:** use a stance verb whenever the notes take a position on each other; `references` -is the honest default when they don't. - -**Conventions:** lowercase kebab-case, named from the source's perspective (`derived-from`, not -`DerivedFrom`). **Edges are directed and stored once** (G4): inbound edges are computed by scanning -`dst_path` and labelled with the inverse; the symmetric verb is its own inverse. B2 **never** writes a -reciprocal link into the target file — that would be write-amplification and would edit a note the user -didn't touch. - -### Edge identity is *derived*, so the file stays clean - -An authored edge — body **or** frontmatter — is identified by the tuple **(src path, dst path, `type`, -occurrence-index)**, all recoverable from the Markdown alone. No edge-id is ever written into the file. -A committed edge carries **no provenance at all** — it is a pristine authored line, nothing more (§4). - ---- - -## 3. The connection / edge model (derived projection) - -Every edge projects to one record. This is the shape the [index-engine.md](index-engine.md) §3 `edges` -table holds; the Markdown is the source, this is the index. - -| Field | Values | Source | -|---|---|---| -| `id` | derived | edge identity, from `(src, dst, type, occurrence_index)` | -| `src_path` | note path | the authoring note (always a note — G6) | -| `dst_path` | note path, or NULL | resolved from the authored `[[path]]` at parse time | -| `dst_resource_path` | resource path, or NULL | set instead of `dst_path` when the target is a non-`.md` file (§10) | -| `dst_path_raw` | text | the target exactly as authored — what a dangling edge keeps | -| `type` | relation verb (§2) | the `b2_relations:` verb; `references` for every body link | -| `origin` | `inline` \| `frontmatter` | which of the two homes (§0) the edge came from | -| `explanation` | free text, optional | trailing text after `—`/`:` (frontmatter entries only) | -| `caption` | text, optional | a Markdown link's text / an embed's alt text | - -- **Every edge is authored and active** (G1). `origin` records *which home it came from*. There is no - lifecycle and no `status` column: an edge exists iff it is written in the Markdown, which is exactly - what keeps `index = projection of (Markdown)` exact. -- **`src`/`dst` are vault-relative paths, resolved at parse time.** The authored `[[path]]` is - normalized against the resolver (the wikilink `+ ".md"` ladder) and stored as the path it named; - `dst_path_raw` keeps the text exactly as written. So an edge stores what the human authored, and the - graph carries no identifier the vault does not. A **B2-performed** move is therefore one transaction - — rewrite the inbound `[[path|title]]` *text*, re-key the moved note's rows, re-project the inbound - sources — and L1's "rename keeps every backlink resolving" holds. A move made **outside** B2 is a - delete plus a create, and its inbound links project as surfaced dangling edges (G5). -- **The edge set is the union of the two homes, deduped** (G2). A frontmatter entry with a *different* - verb than a body link to the same target is no duplicate — it is the augment case (§2), and both - edges project. If the *same* `(src, dst, type)` is authored in **both** homes (necessarily - `references`), projection keeps the frontmatter row — **frontmatter-wins**, because only it can carry - an explanation — never auto-editing the file. -- **A `dst` may be a resource, not a note.** A body embed/link to a non-`.md` file (`![[photo.png]]`, - `[[papers/x.pdf]]`) resolves against the `resources` table and records `dst_resource_path`; `src` is - still a note (§10). -- **A `dst` that resolves to *nothing* is a surfaced dangling edge, not a dropped one** (G5). A note is - one `.md` file (§1), so a `[[Hermes]]` naming a **folder** — or a plain typo — matches no note and no - resource: the edge is still projected with both `dst_path` and `dst_resource_path` NULL. - `b2 neighbors`/`b2 explain` (and the desktop Connections pane) present these **distinctly**, as - *unresolved* with the authored target, so a mistyped link reads as broken rather than silently - vanishing ([GH #12](https://github.com/AlteredCraft/B2/issues/12)). Resolving the target turns the - same edge into an ordinary connection on the next reindex. Folder-note resolution (Obsidian-style - `Hermes/Hermes.md`) is a possible later refinement, deliberately out of scope. - -Why this projection is **materialized rather than computed on read** — and why that does not make it a -third source of truth (G6) — is [index-engine.md](index-engine.md) §3; the standing cost is its §8. - ---- - -## 4. Committing a connection - -There is **no suggestion lifecycle, no review queue, no rejection memory, and no event log** -(ADR-0009). A connection becomes real in exactly two ways, both **authored in Markdown**: - -1. **A body link you write** — a plain `[[path|title]]`, Markdown link, or embed: an untyped - `references` edge (§2). B2 **reads** it on the next reindex; it never writes the body. -2. **`b2 link [--type ] [--explanation …]`** — B2 appends one typed-link string to - the **source note's frontmatter `b2_relations:`** (Markdown first; **never the body**), then - re-projects the note so the edge materializes as `origin='frontmatter'`. `--type` defaults to - `references`; the palette is the core vocabulary (§2). B2 writes the target as a **bare `[[path]]`** - — no `|alias`: the filename is the note's title (L2), so the path already reads as the title. - *(A human writing a body link may still add any `|alias` they like; B2 reads it and never rewrites - it.)* - -The GUI adds one further authoring *gesture* over the same model: dragging a Similar card onto a line -of the note being edited types a `[[wikilink]]` at that line's end — the untyped, body kind of link, -landing in the editor's buffer exactly as `[[` completion does. B2 still authors nothing (W1) and the -human is still the precision gate. - -**No provenance tier.** A committed edge is **pristine**: no `by`, no `confidence`, no `source` — -nothing stapled to the note beyond the ` [[path|title]]` line itself. Provenance is *decision -fuel* for a review step B2 doesn't have. (Optional **note-level** `provenance:` frontmatter remains the -human's to write — §1 — and is separate from edges.) - ---- - -## 5. OKF compatibility (export is a no-op, not a migration) - -Build *like* OKF for cheap interop; don't depend on it. The model already lines up: - -- **`type`** is the OKF entity discriminator — recognized frontmatter, defaulting to `note` (§1). -- **Resource URI** — a per-note URI is derivable from its vault-relative path under a configured base. - It is exactly as durable as the path (L1) — the same handle Obsidian, a static-site generator, and a - file manager all give you (ADR-0003). -- **`index.md`** — a vault-root manifest listing notes/types, **derivable** from the frontmatter, so - an OKF consumer has a collection entry point. Generated, never a second source of truth. - -Net: "export to OKF" is selecting and re-shaping fields that already exist. The export surface itself -(minting URIs, emitting the manifest) is **not built**; -[GH #103](https://github.com/AlteredCraft/B2/issues/103) tracks it. - ---- - -## 6. Serialization discipline - -W5's round-trip losslessness is what makes the two-tier split safe, and it is a property of the -*parser/serializer*, so it is specified here: - -- Unknown frontmatter keys are preserved **verbatim and in order**; body text, whitespace, and comment - tokens are byte-preserved. -- The **only** bytes B2 ever changes are the specific mechanical edits it is asked to make (W3), of - which exactly one touches the body: rewriting an inbound `[[oldpath|title]]` → `[[newpath|title]]` on - a move, aliases preserved verbatim. An operation the human did not invoke changes no byte at all - (W1). -- Nothing is reformatted, reordered, or normalized in passing — not YAML quoting style, not list - indentation, not line endings. - -These are the tripwires [index-engine.md](index-engine.md) §8 budgets: this doc defines them, that one -enforces them in the store. - ---- - -## 7. Rejected / deferred alternatives - -The reasoning for each rejection lives in the ADR that made the call; this is the index. - -| Rejected | Why, in one line | Recorded in | -|---|---|---| -| B2 authoring the body (a `## Relations` section) | The body is the rendered/exported document and must stay 100% the human's | ADR-0004 | -| Body typed-line syntax parsed from prose | Would make B2 an interpreter of prose *shape* — `- see [[x]] for background` becoming verb `see` (L4) | ADR-0004, ADR-0010 | -| Inline-in-body as the home for committed edges | Accepted trade: a frontmatter edge isn't guaranteed clickable in vanilla Obsidian, which can't render edge *types* regardless (§0) | ADR-0010 | -| A suggestion review layer / per-pair LLM relator | ~notes × candidates model calls, and exactly the model-compensating machinery M1 defers | ADR-0009, ADR-0005 | -| A durable event-log tier (`.b2/log/`) | It would exist to hold a queue and rejection memory B2 doesn't have; anything durable outside the notes weakens S2/S4 | ADR-0002 | -| A stamped machine identity (`b2id:`) — **removed 2026-08-13** ([GH #170](https://github.com/AlteredCraft/B2/issues/170)) | Its one real buy (out-of-band move re-binding) cost an unbidden write, a machine key in every file, a collision subsystem, and a carve-out on S3 | ADR-0003 | -| Content-hash identity | Churns the graph on a typo fix. (Hash-keying the *vector store* is the opposite case and shipped with the same change — derived data, where content-addressing is exactly right, M4) | ADR-0003, ADR-0006 | -| Stored reciprocal links | Inverse edges are derived at query time (G4); writing them back amplifies writes and edits notes the user didn't touch | ADR-0010 | -| Per-edge ULIDs in the file | Authored edge identity is derived (§2); explicit ids clutter the note for no gain | ADR-0010 | -| Edge provenance in Markdown | With no review step there is no provenance to keep (§4) | ADR-0009 | - ---- - -## 8. A golden-vault sketch (for the test harness) - -The smallest fixture that exercises the whole model — an authored typed edge and a bare reference. - -`concepts/memory.md` -```markdown ---- -type: concept -title: "Human memory" -created: 2026-06-20 ---- -The brain encodes, stores, and retrieves information… -``` - -`notes/spaced-repetition.md` -```markdown ---- -type: concept -title: "Spaced repetition" -created: 2026-06-20 -b2_relations: - - "supports [[concepts/memory|Human memory]] — applies the forgetting curve" ---- -Spaced repetition exploits the [[concepts/memory|Human memory]] retrieval curve. - -Expanding review intervals exploit the forgetting curve. -``` - -The body holds one plain link (`origin=inline`, `references`) and the `b2_relations:` entry types the -same connection with a stance (`origin=frontmatter`, `supports`): the augment shape from §2, exercising -both homes at once. Derived graph (no live model needed to assert): - -- `references`: spaced-repetition → memory (origin=inline) — from the prose wikilink. -- `supports`: spaced-repetition → memory (origin=frontmatter, explanation="applies…"). - -`b2 neighbors concepts/memory` returns spaced-repetition twice (referenced-by, supported-by); both -files round-trip byte-identical; dropping and rebuilding the index reproduces the identical graph. - ---- - -## 9. Judgment calls — resolved - -The data model is **locked**; nothing is open. Every decision and its rejected alternatives are in -[`ADRs/`](../ADRs/README.md) — chiefly ADR-0002 (two tiers), ADR-0003 (path identity), ADR-0004 -(write discipline), ADR-0009 (no suggestion queue), and ADR-0010 (the two homes, the dedup rule, and -the verb core). The normative claims are the invariant register's S, W, L, and G entries. - ---- - -## 10. Resources — the second kind of vault member - -§0–§9 define the **authored** objects. A real vault also holds **resources**: every non-`.md` file. -This section defines what a resource *is* in the model; the schema is -[index-engine.md](index-engine.md) §3. - -A resource is a **peer vault member** — not a lesser one, and not a generalized note. The single -asymmetry is **authoring surface, not status**: - -> **A note is where structure is *authored*; a resource is a peer document B2 cannot write.** Notes -> have frontmatter, authored edges, and B2's write guarantees — because Markdown is the one format -> whose bytes B2 may touch. Resources have bytes, *derivable* text and vectors, and *inbound* links. -> **Identity is not part of the asymmetry**: both are keyed by their vault-relative path (L3), so the -> note rules are the resource rules with an authoring surface added, not a second identity model. - -What the asymmetry does **not** mean: a resource is never *required* to be attached to a note. An -unlinked resource fully exists — walked, classified, in the file tree, in the index, openable. - -**Identity — path-keyed, index-only.** There is nowhere to stamp a machine id (binary bytes are not -B2's to edit; a sidecar file would be durable state outside Markdown, violating S4) and nothing one -would protect. That reasoning is now the *whole vault's* — ADR-0003 read it back onto notes. - -- **`b2 mv` on a resource** is simply *the* move: rewrite the inbound `[[path]]` / `![alt](path)` text, - move the file, re-project — on identical terms to a note (§3). -- **Placing one is not authoring one.** "B2 cannot write a resource" is about its *content*: B2 never - edits the bytes, and there is no format-specific writer. Putting a file **into** the vault on - explicit command — the desktop's drag-from-Finder import (`Vault::import_file`), the same act as - moving or deleting it (W3) — is file management, and the copy is byte-honest: B2 places exactly the - bytes it was handed and then projects them, routing on the extension exactly as the walk does. A - dropped `.md` is therefore a *note* arriving, frontmatter and all. -- **An out-of-band move degrades exactly as a note's does** — identification, not repair: the old - path's rows prune, the new path projects as a new member, and every inbound link surfaces as a - dangling edge (G5). The index keeps a **blake3 content hash** per resource, and notes carry a - `body_hash`, so the *proposed*-repair idea stays buildable on data already stored; it is recorded as - future investigation in GH #170, deliberately not built (a proposal is still the human's to accept, - W4). - -**Edges — `src` is a note in v1; `dst` may be anything** (G6). A *consequence* of the invariant, not a -status rule: every edge must trace to an authored line in Markdown (§3), and a resource has no writable -home for one. Two relief valves keep this from hardening into an expressiveness wall: **(a) today**, -the tolerated tail already authors the inverse direction from the note side -(`- "supported-by [[papers/x.pdf]]"`); **(b) if needed**, resource-sourced edges get a designed future -home — a **vault-level B2-managed relations file**, so the edge is still authored Markdown and the -invariant holds ([GH #102](https://github.com/AlteredCraft/B2/issues/102), deferred until the need is -real). - -**What is built, and what is designed.** The **inventory and graph** half ships: the walk classifies -every non-`.md` file by extension into one of six classes — `text` · `html` · `pdf` · `image` · -`media` · `binary` (the total fallback) — records `(path, class, size, mtime, content_hash)`, prunes -what the walk no longer meets, and resolves body embeds/links at them into `dst_resource_path` edges. -Classification is by extension **only**: deterministic, and a misclassification degrades rather than -mis-executes. - -The **content** half is locked design, not shipped: every class funnels to *text* — native (`text`), -extracted (`html` tag-strip, `pdf` text layer), or, for an `image`, the aggregated alt-text/captions -from the notes that embed it (a pure projection of authored Markdown) — then flows through the -**existing** bge space with zero new discipline (chunks + a per-document centroid). -[GH #108](https://github.com/AlteredCraft/B2/issues/108) tracks it, -[#109](https://github.com/AlteredCraft/B2/issues/109) the PDF extraction dependency and -[#107](https://github.com/AlteredCraft/B2/issues/107) render mechanisms. Until they land, a resource -is an inventoried, linkable, openable peer that search and discovery do not reach. Multimodal image -embedding and an LLM/OCR **Describer** are documented future seams, default-off (M3, -[GH #110](https://github.com/AlteredCraft/B2/issues/110)). - -**Why a separate object, not a `kind` column on the note.** Two tables, two contracts, zero "unless -it's a resource" clauses. Generalizing `notes` to hold resources would staple a caveat onto every -invariant, write guarantee, and frontmatter behavior in §0–§9; a distinct `resources` table isolates -the different *write* contract instead of threading it through the note rules. diff --git a/design/index-engine.md b/design/index-engine.md deleted file mode 100644 index f69feff..0000000 --- a/design/index-engine.md +++ /dev/null @@ -1,422 +0,0 @@ ---- -title: "B2 — Index Engine" -type: note -tags: [b2, index-engine, sqlite, fts5, vectors, search, discovery, chat, architecture] -created: 2026-06-29 -status: active ---- - -# B2 — Index Engine - -> **The engine design — the *how*.** Specifies the disposable SQLite index (FTS5 + an in-process -> vector scan + the typed graph) and the flows over it. Companion docs: [invariants.md](invariants.md) -> (the normative *why*, cited by id) and [data-model.md](data-model.md) (the *what*). -> -> **Rationale lives in [`ADRs/`](../ADRs/README.md)** — why SQLite and not qmd (ADR-0019), why -> plain vector tables (ADR-0006), why the graph is materialized (ADR-0010), why discovery always -> serves (ADR-0014), why a served result claims evidence (ADR-0015), why candle (ADR-0020). This -> doc specifies what the engine *is*; each section names the record that decided it. - -## 1. qmd, the reference — and what we took - -[qmd](https://github.com/tobi/qmd) is a local CLI search engine for Markdown, all on-device: SQLite + -FTS5 + `sqlite-vec`; ~900-token chunks with ~15% overlap and Markdown-aware break-point scoring; -three search modes; a pipeline of LLM query expansion → parallel retrieval → **RRF fusion** -(`Σ 1/(k+rank+1)`, k=60) → cross-encoder rerank → position-aware blend; local GGUF models via -`node-llama-cpp`; TypeScript on Node, MIT. - -**We rebuilt rather than depended — ADR-0019.** Taken wholesale: the chunking heuristic, **the RRF -formula and `RRF_K = 60`**, the position-aware blend, the asymmetric query/document prompt discipline -(each model brings its own prefix — B2 ships bge's, §6), and the JSON/`--explain` agent-output -discipline. Discarded: the npm/Node packaging and the "DB is the product" framing. - -**Chunking as adapted** (`chunk.rs`, [GH #19](https://github.com/AlteredCraft/B2/issues/19)) — four -model-free changes to qmd's heuristic: a **450**-token target (headroom under bge's 512-token -truncation), a `chars/4` proxy for token sizing (the core stays tokenizer-free, E1), an unconditional -stored `heading_path` breadcrumb, and every lever on a `ChunkConfig` (overlap 0.15, backscan 200, the -H1=100 … list=5 break weights). A forced cut is pushed past a fenced code block or Markdown table -rather than bisecting it ([GH #41](https://github.com/AlteredCraft/B2/issues/41)). Tree-sitter AST -chunking for code stays deferred ([GH #104](https://github.com/AlteredCraft/B2/issues/104)). - -## 2. Why we rebuild instead of depend on qmd - -Recorded in full as **[ADR-0019](../ADRs/0019-build-our-own-sqlite-index-engine.md)**. In one -line: qmd is a search engine and B2 is a typed graph with hybrid retrieval over it — qmd models no -typed edges, no backlinks, and nothing that repairs links on a move (G1–G6, L1), and SQLite holds all -three queryable concerns in one transactional store. - -## 3. The storage architecture (one disposable SQLite index) - -One artifact, per S1/S2 and ADR-0002: a **disposable** SQLite index holding every queryable concern -transactionally, rebuildable from the vault at any time (S3). - -> The precise DDL and build order are realized in `crates/b2-core/src/db.rs` (schema) and `ingest.rs` -> (flows). The sketch below is the orientation; the code is the buildable contract. - -``` -b2.sqlite — DISPOSABLE CACHE (= projection of the vault directory; drop & rebuild any time) -├── MIRROR OF THE VAULT (lets us diff vs. disk) -│ ├── meta(key, value) -- schema_version, embed_model_id, embed_dim -│ ├── notes(path PK, type, title, description, -- the path IS the identity (L1) -│ │ created, updated, body_hash, mtime, indexed_at) -│ ├── note_aliases(note_path, alias) -- frontmatter `aliases:` -│ └── resources(path PK, class, size, mtime, -- non-.md peers; class by extension (§10 dm) -│ content_hash, indexed_at) -│ -├── DERIVED: SEARCH -│ ├── chunks(id, note_path, seq, char_start, char_end, token_count, heading_path, text, text_hash) -│ ├── chunks_fts -- FTS5 over chunk text, `porter unicode61` (BM25) -│ ├── embeddings(text_hash PK, vector) -- CONTENT-ADDRESSED plain BLOB vectors (768-dim) -│ └── note_centroids(note_path, centroid) -- per-note centroid (discovery's coarse stage) -│ -└── DERIVED: TYPED GRAPH - └── edges(id PK, src_path, dst_path, dst_resource_path, dst_path_raw, - type, origin, explanation, caption, embed, occurrence_index) -``` - -Every table is derived from the vault; there is no third home. The two **vector** tables are created -at *embed* time rather than in the base migration — their existence *is* the "this vault has an -embedding space" signal the BM25-only fallbacks key on (M4, ADR-0006). - -*(The projection runs as two separately-invokable passes — model-free `project` (notes/resources/ -chunks/FTS/edges) then `embed` (vectors), with `reindex` their composition — so keyword search and the -graph are usable before embedding completes -([GH #15](https://github.com/AlteredCraft/B2/issues/15)). S2 is untouched: a projected-but-unembedded -index is a smaller projection, never a wrong one.)* - -Why this shape fits B2 specifically: - -- **Everything keys on the vault-relative path** (L1, ADR-0003). `notes.path` is the primary key; - `chunks.note_path`, `note_aliases.note_path`, `note_centroids.note_path` and `edges.src_path` are - `REFERENCES notes(path) ON DELETE CASCADE ON UPDATE CASCADE`, which makes a B2-performed move a - **path re-key rather than a rebuild**: `UPDATE notes SET path = …` cascades through every child in - one transaction, alongside the inbound link-text rewrite and a re-projection of the inbound sources. - `edges.dst_path` is deliberately *not* an FK — it must be allowed to be NULL, the dangling case - (G5). "Rename keeps every backlink resolving" is therefore a property of the **move operation**, not - of the key; the price is that a move made *outside* B2 is a delete plus a create (§8). -- **Vectors are content-addressed** (M4, ADR-0006), keyed by blake3 of the chunk text — which *is* the - embed input, verbatim — so a note that moves re-embeds nothing, identical text anywhere shares one - vector, and the only invalidation rule is "a hash no chunk references is garbage", pruned by the - whole-vault pass on the same derived-data lifecycle as centroids. -- **Every `edges` row derives from Markdown** (G1, G2, ADR-0010), deduped frontmatter-wins. There is - **no `status` column and no suggestion queue**: `b2 link` appends a typed-link string to the source - note's frontmatter and re-projects that note — the projection of an authored line, not an in-place - index write. -- **Hybrid retrieval and graph queries compose in one query** — "semantic-nearest chunks whose note is - within 2 typed hops of note X" is a join across `embeddings`, `chunks`, and `edges`. This is the - substrate `b2 similar` runs on. -- **Deterministic seams for tests** — a fake embedder writes to `embeddings`, so the whole pipeline is - assertable with no live model (E2). - -**Resources widen the projection without disturbing any statement above** — path-keyed peers, so -`index = projection of (the vault directory)`. `resources` is a **separate** table from `notes`, not a -`kind` column on it (two tables, two contracts, zero "unless it's a resource" clauses), and -`edges.dst_resource_path` lets a body `![[photo.png]]` or `[[papers/x.pdf]]` resolve against it while -`src` stays a note (G6). What is built is inventory + graph; resource *content* search is designed, -not shipped — [data-model.md](data-model.md) §10. Either way there is **no migration**: a schema -change is a `schema_version` bump + rebuild (S5). - -### Opening the index concurrently — many readers, one builder - -Several things open one index at once: `b2 reindex &` racing a `b2 status`, the desktop app launching -while a CLI reindex runs, the desktop host's own threads. The locked stance is **C1**, enforced in -`db::open` in three layers that each answer a different failure: - -- **The `WAL` flip is retried** ([GH #111](https://github.com/AlteredCraft/B2/issues/111)). Setting - `journal_mode = WAL` is the one statement in `open` that takes a write lock and the one - `busy_timeout` cannot cover — SQLite skips the busy handler for a write lock upgraded from an - already-open read transaction — so a second opener took an immediate `SQLITE_BUSY`. The wait is ours. -- **The schema migration is one `BEGIN IMMEDIATE` transaction, entered only when there is work** - ([GH #114](https://github.com/AlteredCraft/B2/issues/114)). Unserialized, two openers that both read - a stale `schema_version` interleave — one's `DROP TABLE` landing after the other's `CREATE`, and the - current version stamped over a half-demolished schema. `busy_timeout` was irrelevant: nothing - contended, every statement succeeded, in the wrong order. The check that decides whether to enter is - a **read**, so the common open against a current schema takes no write lock and can never be refused. -- **Completeness is checked, not assumed.** A stamp is believed only alongside the tables it vouches - for; a current stamp over missing tables is stale and rebuilt from empty. Recreating just the missing - tables would be worse than useless — an incremental reindex skips notes whose `body_hash` matches, so - they would stay empty and S3 would quietly fail. - -The vector tables are the same drop-and-rebuild shape and get the same treatment. **Not an advisory -lock file:** that would be a third concurrency mechanism guarding state the database already guards, -and the weaker one where it counts — on a network share or synced folder, `flock` quietly stops -meaning anything. The `reindex` lock ([GH #55](https://github.com/AlteredCraft/B2/issues/55)) answers -a question SQLite cannot (*is another **process** already doing this expensive work?*); it is taken by -`b2-cli` alone, never by the desktop host or by readers, and so cannot cover schema atomicity. - -### Why materialize the graph at all — vs. resolving links at runtime - -A note's *outbound* links are parseable from that one file on demand, so edge metadata is not the -reason. **Inversion and composition are** — materializing turns three things from full-vault scans -(or impossibilities) into indexed lookups: - -- **Backlinks / inversion.** "Who points at X" cannot be read from X, only from every *other* note: - O(vault) per query at runtime, one lookup here. This is also what services L1 — the edges name the - exact N inbound files to rewrite on a move instead of scanning the vault to find them (§8). -- **Typed multi-hop traversal.** "notes within 2 hops of X via `supports`" is a scan *per hop* at - runtime; over `edges` it is one SQL traversal. -- **The graph⨝vector join.** "semantic-nearest chunks whose note is within k typed hops of X" is a - single join `embeddings ⨝ chunks ⨝ edges`, not expressible as a per-note parse — a **scoped - traversal** primitive. **`b2 similar`'s candidate generation is its complement**: notes semantically - near an anchor but *not* within 1 hop, where the materialized graph supplies the exclusion. - -**G6** keeps this cheap: runtime outbound-parsing is the correctness *definition*, the `edges` table -its *cache* — one more disposable table in the same store, populated by the same parse pass that -already walks each body for chunking. The standing cost is the move-repair amplification in §8. - -### Discovery surfacing serves the ranked list — `limit` is a cap, not a promise - -The standing rule is **invariants.md D1**; the decision, the seven issues of measurement behind it, -and the retired z-gate are **[ADR-0014](../ADRs/0014-discovery-always-serves-the-ranked-prefix.md)**. -Mechanically, in this engine: - -- `similar` serves the ranked top-N whenever candidates exist. `limit` under-fills only for want of - scorable notes, and **no statistical bar truncates the list**. -- The per-candidate **z survives as a statistic that gates nothing** — computed after stage 2 on the - best-passage distances, non-increasing down the row order (strictly monotonic in the *score*; tied - scores share a z and order by the path tie-break), painted as the within-list strength band. Because - the judged z is affine in the squared best-pair distance the score negates the root of, **score - order, z order, and band are one number**: a card can never show a weaker band above a stronger one. -- Below `STATS_MIN_POPULATION` (12), in a space with no spread, or under the fake embedder, **no z - exists at all**, and a surface must *say* the list is ungraded — silence there reads as "all judged, - all scored low", the opposite of what happened. -- Candidate *generation* is unchanged and stays recall-oriented: the two-stage scan (§4) over-produces, - nothing auto-links, and the human commits every edge (W4). - -The **pair-scorer escalation** stays the named long-term seam: an anchor-local rule cannot catch a -*pair-level* miscalibration (a single stranger the model scores like a cluster-mate — the standing -`encryption ↔ phishing` residue), and a discovery-side pair-scorer would be a second model seam, -sibling of §5's reranker but distinct from it (that seam needs query text and `similar` has none). It -would still only filter what is surfaced, never author a link. Under always-serve that residue is -**ordering quality, not existence**. - -## 4. Retrieval — semantic search, fusion, and discovery - -Semantic search is **in v1** (ADR-0019) — exact, in-process, no vector extension, no ANN. - -> The step-by-step walk through flows ② and ③ — every stage anchored to the function that -> implements it — is the docs site's [retrieval deep dive](../docs/retrieval.html). This section -> stays the spec; that page is the guided walk through the code. - -- **Storage & scoring.** Vectors live in plain tables — `embeddings(text_hash, vector)` plus per-note - `note_centroids` — read with one statement and scored in-process (`embed::l2_sq`). - Content-addressing costs the scan one indexed join back to `chunks` to recover which chunk each - vector ranks for. A `vec0`-style virtual table charges a per-row shadow-table probe on every scan, - which dominates at real-vault scale; the plain-table scan does not - ([GH #38](https://github.com/AlteredCraft/B2/issues/38), ADR-0006). -- **Flow ② hybrid search** (`search.rs`). BM25 over `chunks_fts` ⊕ vector KNN, fused with Reciprocal - Rank Fusion (`Σ 1/(k+rank+1)`, `RRF_K = 60`), resolved from chunks up to notes. Raw NL queries are - sanitized into a safe FTS5 `MATCH` expression — punctuation is FTS5 syntax and would otherwise crash - the parse. On a projected-but-unembedded vault the vector half is simply absent and the same fusion - runs over the single BM25 list, so scores stay on one scale. -- **The flow can answer zero, and the evidence rides beside the order** (invariants.md **D2**; - decision and history in - **[ADR-0015](../ADRs/0015-a-served-search-result-is-a-claim-of-evidence.md)**). `hybrid_search` - returns a `Retrieval`: the same fused order (untouched — provenance is carried, never folded in), - each hit naming the lists that ranked it and its own distance, plus the dense half's best cosine. - The lexical reading is deliberately *not* in it — it costs a `count(*)` per distinct term, so - `Vault::search_evidence` reads it only where a caller wants a verdict and joins the two signals - into the query-level `QueryEvidence`. The rule over them is *lexical OR semantic*: - - **The lexical anchor is IDF-weighted coverage, not presence.** `fts5_query` ORs every term, and - the labelled phrase negatives match 68 of 70 chunks through `a`/`to`/`my` alone — one reading a - *better* best-BM25 than several positives off nothing but function words. So neither a hit count - nor a raw BM25 score is a lexical test. What is: each term weighs `ln((chunks+1)/(df+1))`, and the - anchor is the share of the query's total weight the vault carries (`min_term_coverage`). Stopwords - are a *measurement*, never a shipped word-list. - - **The cosine bar is the backstop** (`min_cos`), judged only on queries the lexical half leaves - undecided, and thin by construction: ask the lexical half for more and the window collapses, - because the queries it then has to rescue are the ones with the weakest semantic evidence too. The - two constants are placed inside a **joint** band, never tuned one at a time. - - The constants are **distributional**, so they are keyed to `embed_model_id` (M2 — a swap - invalidates them; the device suffix shares the reading and `make eval-metal` is where that - assumption is re-checked). They live in `search::BGE_BASE_EVIDENCE_BAR` and their *justification* - is re-derived on every `make eval` run — never frozen into a comment (ADR-0013). - - The verdict reaches an adapter through `Vault::search_evidence`, which serves **exactly** the rows - `search` does, in the same order — three-state, and the three states are three behaviours - (ADR-0015). `search_evidence_excluding` (`b2 search --exclude`) is the same read minus a - **caller-named** set of notes — the follow-up-search form an agent loop passes its - already-inspected paths to. The subtraction is the caller's, never the verdict's: the verdict and - its signals still read the whole vault, the remaining rows keep their fused order, and the pool is - unchanged (width moves only on measured relevance, below), so a heavily-excluded query may - honestly under-fill. **The tail** — folding where a *real* query's per-hit evidence runs out — is - measured and stays unshipped (GH #206): with `tail_relevant` labels in place, the four-family - prefix-cut bake-off found every admissible rule vacuous — the fused order is not an evidence - order, so admissible folds reach 2–23 of the 367 rows an oracle fold would cut. The ruling and - its numbers live in [crates/b2-embed/evals/README.md](../crates/b2-embed/evals/README.md); the bake-off re-arms every run. -- **Flow ③ discovery is two-stage** (`discover.rs`). An O(notes) coarse scan over centroids — the - anchor and its 1-hop graph neighbours excluded *up front*, so the already-connected never occupy a - shortlist slot — keeps a shortlist (`SHORTLIST_PER_RESULT = 20` per asked result, floored at - `SHORTLIST_MIN = 200`), then an exact max-sim rescore over only the shortlist's chunk vectors. - No model call at surface time, and nothing is re-embedded: the anchor is represented by its - *stored* chunk vectors, never an `embed_query` of its text (bge's asymmetric query prefix is the - wrong side of the space). `b2 similar` surfaces, `b2 link` commits, and the human is the precision - gate (ADR-0009). -- **Candidate width is per view, and it is a quality knob, not plumbing.** Retrieval widens twice: - each façade read asks for a hit pool over its `limit`, and each signal (`search::pool_size`) pulls - 5× *that* (minimum 30) before RRF fuses the two lists. The two reads need headroom for different - reasons, so they get different pools - ([GH #142](https://github.com/AlteredCraft/B2/issues/142)). Note-level `Vault::search` keeps **3×**: - dedup collapses every chunk sharing a note onto that note's best one, so a pool of exactly `limit` - would under-fill `limit` distinct notes. Passage-level `Vault::search_chunks` has no dedup and keeps - a **small constant** (`limit + 2`) — enough to backfill the one hit it can drop to a C1 torn read, - and no more. Giving it the 3× too is not a free tidy-up: the 5× multiplies every hit of headroom - into candidates (150 per signal against 60, at a 10-result ask), and RRF over a wider candidate set - returns *different* answers — at k = 60 a chunk ranked ~60th in **both** lists outscores one ranked - first in a single list (`2/121 > 1/61`). Width moves only on measured relevance (§5). -- **A fused-score tie is broken by the dense signal, and that is a policy.** RRF over integer ranks - lands every fused score on a discrete lattice, so bit-identical ties between mirrored rank pairs — - (1, 3) vs (3, 1) — are structural, and the eval corpus produced one - ([GH #156](https://github.com/AlteredCraft/B2/issues/156): the semantic half named the labelled - answer, BM25 named the wrong one). The secondary sort key in `rrf_fuse` is therefore the candidate's - rank in the **vector list** (absent ranks below present), with id last purely for determinism — a - photo finish is decided on the signal measured to be right there, never on projection walk order. -- **The FTS tokenizer is `porter unicode61` — stemmed, and that is a *measured verdict*, not a - default.** Unstemmed BM25 matched surface forms only — `pedalling` found nothing in a note that says - "pedals", leaving the lexical half at rank 41–46 on queries the dense half ranked first, which RRF's - consensus bias turned into hybrid demotions of correct dense hits. The A/B that settled it - ([GH #157](https://github.com/AlteredCraft/B2/issues/157), 2026-08-11): porter improved 7 BM25-only - and 3 hybrid note ranks, worsened none, and dissolved every standing fusion demotion — while the - precision probes built to vote *against* stemming (the `universe`/`university` Porter collision, the - code-literal queries) did not move. Stemming remains a real trade — Porter is English-only, and a - vault holds code, identifiers, and proper nouns — so the retired arm stays measurable: - `Vault::rebuild_fts` swaps the tokenizer over identical chunk rows and vectors (nothing re-chunks or - re-embeds), and `make eval-stemmer` scores the unstemmed ablation beside every default run. -- **Does brute force scale to B2?** Comfortably. A personal vault of 10k notes → ~50–100k chunks; - brute-force cosine over ~100k × 768-dim float32 vectors is single-digit to low-tens of milliseconds. - We are nowhere near the regime where ANN matters; if a vault ever is, int8/binary quantization and - ANN hold a standby order behind the centroid stage - ([GH #106](https://github.com/AlteredCraft/B2/issues/106)). - -## 5. Deferred model machinery — the reranker & query expansion - -**The reranker is a fast follow.** Slot it where qmd puts it: **after RRF fusion, before final -ranking**, behind a swappable seam. v1 is retrieve → fuse → return top-N, already a strong hybrid -baseline; the fast follow inserts a cross-encoder rerank over the top ~30 candidates plus a -position-aware blend. It is a pure function `(query, candidates) → scores`, so it changes *ordering*, -not the store, the schema, or the candidate set — and it is **store-agnostic**, a model-side seam -above the index. Tracked in [GH #28](https://github.com/AlteredCraft/B2/issues/28). - -**Scope — this reranks `b2 search`, not `b2 similar`.** The signature is the tell: it needs *query -text*, and `b2 similar` has none — it is passage↔passage KNN, "near ∖ connected" (§3). The -discovery-side levers are distance-weighting ([GH #20](https://github.com/AlteredCraft/B2/issues/20)) -and the pair-scorer seam (§3); the discovery-side *precision* stance is D1's (ADR-0014). - -**Gate the decision on the eval, not intuition** (ADR-0013). RRF is a strong baseline; the reranker -buys **top-k precision**, whose value *grows with vault size* and is *highest when an agent consumes -top-1/top-3 with no human eye* ([GH #24](https://github.com/AlteredCraft/B2/issues/24)). Vault size -changes whether the precision is worth it, not the reranker's cost. - -**…and check the instrument can see the change you are gating.** Retrieval reaches at least -`chunk_candidate_pool(10) = 60` candidates per signal (150 for the note view); while a corpus has no -more chunks than that, neither signal is truncated and a candidate-width change prints bit-identical -numbers while genuinely reordering a real vault -([GH #141](https://github.com/AlteredCraft/B2/issues/141)). Score *relevance* on the labelled corpus, -but measure a width change with `make stability` over a vault big enough for the pool to bind. That -gate has already ruled once: [#140](https://github.com/AlteredCraft/B2/pull/140) widened the passage -view as plumbing, the eval printed bit-identical numbers, the probe found 10 of 10 top-4 lists -changed, and [#142](https://github.com/AlteredCraft/B2/issues/142) returned it — **the instrument that -can say "different" is not the one that can say "better"**. The blindness is to *candidates*, not to -fusion: `RRF_K` re-weights the same two lists and reorders results at any corpus size. - -**Query expansion** (qmd's third model) is **optional and lowest priority** — the heaviest model for -the smallest, most variable win ([GH #105](https://github.com/AlteredCraft/B2/issues/105)). - -The harness itself — corpora, labels, metrics, exit gates, process rules — is -[crates/b2-embed/evals/README.md](../crates/b2-embed/evals/README.md); read it before touching any of them. - -## 6. The AI seams — the embedder in a single binary, and grounded chat - -Two seams, both enumerated by M1 (ADR-0005) and both injected by the adapters. - -### `Embedder` - -Runtime, provisioning, and the model choice are -**[ADR-0020](../ADRs/0020-embeddings-inside-the-single-binary.md)**: `candle` + `hf-hub` compiled -into the binary, an explicit `b2 init` into a shared XDG cache, default `BAAI/bge-base-en-v1.5` -(768-dim, CLS-pooled, L2-normalized, bge's asymmetric query prefix), the dimension read from the -model's own `config.json`. - -The engine-side consequences are invariants: the embedding space has exactly one recorded identity — -`meta.(embed_model_id, embed_dim)` — and the compute **device folds into it** (ADR-0007, [GH #40](https://github.com/AlteredCraft/B2/issues/40)). A -swap drops both vector tables and re-embeds on `reindex`; `search` **fails fast** rather than mixing -spaces; `open` never mutates the vector space (M2). The fake embedder stays the CI default, so model -quality never enters the fast suite (E2). - -### `LlmProvider` — flow ④, grounded chat - -Chat is a **reader** of this index and adds nothing to it (M1): no table, no cached response, no -`meta` row, session-only history (S4). That is the deliberate contrast with the embedder — swapping -chat models never touches the index, so a provider swap is a URL/config change. - -`Vault::ask` (`chat.rs`) is five steps: **condense** (multi-turn only — a provider call rewrites the -follow-up into a standalone query, degrading to the raw question on failure, so that step can never -break chat) → **retrieve** (`search_chunks` at `ASK_PASSAGES = 10`, the §4 pipeline unchanged, -BM25-only fallback included) → **assemble** (the grounded system prompt + numbered passages — prompt -assembly is core logic, not an adapter's) → **stream** (tokens up through the caller's callback, whose -return value cancels at token granularity — sync, no runtime) → **cite** (`[n]` markers resolve to -`(path, excerpt)` in the returned `AnswerView`; a hallucinated marker resolves to nothing and the -answer text is **never rewritten**). - -Two properties follow from the seam's shape. Cancellation is returning early from a blocking read -loop, so **no B2 crate starts an async runtime** (ADR-0011) — a cut stream is marked cancelled and -what already arrived is rendered, never discarded, because a truncated answer is not an error. And -model output is untrusted content like any note (E5, ADR-0016), enforced at the render surface rather -than in the core. - -## 7. Tech-stack implications — resolved: Rust - -The **single-binary goal** picked the language, not the engine; SQLite and FTS5 are -language-agnostic. See ADR-0019. - -## 8. Risks & operational burden - -**Chunk vs. note granularity for the graph.** Search is chunk-level; the typed graph is note-level. -`chunks.note_path` is the join, and search hits resolve up to notes for graph operations (§3). - -### The bill for a path-keyed graph under `[[path|title]]` links - -Keying the graph by the vault-relative path (L1, ADR-0003) makes the stored key *the same thing the -human authored*. These are *the trade working as designed*, not defects; they must be budgeted, -tested, and watched. - -- **Write amplification on move.** The link *is* the path, so moving one note rewrites the inbound link - text in **every** file that points at it — an N-file write. It is bounded and mechanical (the - materialized edges name exactly which files and links to touch, Markdown-first then index), but - moving a heavily-linked note is proportional to its backlink count, not O(1). Keep the rewrite - transactional so a partial move never half-updates the vault. The index side is one cascading - `UPDATE` plus the re-projection of the inbound sources, bounded by the same count. -- **Out-of-band moves are identified, not repaired — and that is the scope decision.** A `git mv` or - Finder move is, to a path-keyed index, a delete plus a create: the old path's rows prune, the new - path projects fresh, and every inbound `[[oldpath]]` becomes a **surfaced dangling edge** (G5) — - authored text kept, `dst` NULL, healing by itself on the next pass if the target comes back. Nothing - is silently dropped and nothing is guessed at. **Content-addressed vectors (M4) make it cheap as - well as honest**: the re-created note's chunk text hashes identically, so the delete-plus-create - re-embeds nothing. Two repairs are future investigation in GH #170 — a *proposed* hash-match repair - (`notes.body_hash` and the resource `content_hash` are already stored, but a proposal must stay the - human's to accept, W4), and a `b2 watch` daemon observing renames live, rejected as a lifecycle and - coordination surface that shrinks the gap rather than closing it. -- **Path ownership follows the filesystem, so there is nothing to reconcile.** A path names at most - one file — the filesystem's guarantee, not B2's — so a note deleted and recreated at the same path is - that path's note, a Finder-duplicated note is two notes at two paths, and `db::upsert_note`'s - `ON CONFLICT(path)` is the whole of the reconciliation. A note file *deleted with no replacement* is - reconciled by the whole-vault pass ([GH #31](https://github.com/AlteredCraft/B2/issues/31)): - `project_vault` prunes every `notes` row whose path the walk did not see this run (aliases, chunks, - FTS, centroid, outgoing edges cascade; inbound links re-dangle when phase 2 re-derives edges against - the pruned resolver), **except** rows whose file was skipped as unreadable — the walk *saw* that - file, so evicting it would lie. Single-note ingest (`add`/`mv`/`write`) touches one note and never - prunes. Orphaned vectors — hashes no chunk references after that pruning — are collected by the same - pass, the only bookkeeping content-addressing adds. -- **A single unreadable file never fails the whole index.** A real vault holds the odd non-UTF-8 or - permission-denied `.md`; projection **skips** it (reported as a `skipped` entry carrying a short, - file-level reason, surfaced by the CLI and the desktop) and indexes everything else, rather than - aborting on one file it cannot read. -- **Derived-index consistency is a permanent invariant, not a one-time build.** W5, S3, and L1 are the - tripwires; every edit path (`b2 mv`, link delete, out-of-band reindex) has to preserve all three or - the graph silently diverges from the source of truth. -- **Committed edges are only ever authored, never inferred** (G1). Editing the vault can strand a - connection — deleting an authored `A→B` link — but B2 only ever *surfaces* the consequence, never - silently rewrites an inbound file or an edge (W4). - -*(The stamped `b2id` made "two files, one identity" representable and cost a collision subsystem, a -shadowed-copy panel, restamp notices, and a carve-out on S3. All of it went with the stamp — -ADR-0003. Nothing anomaly-shaped was ever *stored*, so nothing had to be migrated away.)* diff --git a/design/invariants.md b/design/invariants.md deleted file mode 100644 index 44c27cf..0000000 --- a/design/invariants.md +++ /dev/null @@ -1,321 +0,0 @@ ---- -title: "B2 — Invariants" -type: note -tags: [b2, invariants, architecture, canonical] -created: 2026-07-22 -status: active ---- - -# B2 — Invariants - -> The normative register of what must always be true of B2. Each entry is one testable claim; the -> linked doc holds the elaboration. This page is the top of the design set — the *why* — with the -> *what* in [data-model.md](data-model.md) and the *how* in [index-engine.md](index-engine.md). -> Product non-negotiables (local-first, zero lock-in, single-binary) are captured as invariants here. -> -> **On conflict, this page wins and the other doc gets fixed.** Changing this page is a deliberate -> decision, never a drive-by edit. Cite entries by id (S2, G2, …). -> -> The *why* behind an entry — the context it was ruled in, and what the ruling costs — lives in -> [`ADRs/`](../ADRs/README.md). Changing an entry here means writing or superseding an ADR there. - -The register is the two design tenets — *a volatile vault over a disposable index* and *build for -tomorrow's model* — made mechanical. - -## S — Storage: two tiers, one projection - -- **S1 — Two tiers, sharply split.** The vault (Markdown + resources + the directory tree) is the - source of truth; `.b2/b2.sqlite` is a disposable cache. Nothing in the index is authoritative. - ([data-model.md](data-model.md) "Two storage tiers") -- **S2 — The index is a pure projection: `index = projection of (the vault directory)`.** Drop - `b2.sqlite`, reindex, get an identical index. **Markdown is the vault's sole authored subset** — - the only format whose bytes B2 may write; resources contribute derived rows only, and folders are - never projected at all (read live off disk). The projected *domain* is the vault's **managed - subtree**: a dot-prefixed name is not vault material of any kind — folder, resource, or `.md` alike - — so it is skipped by every walk before routing, and refused as an authoring destination (B2 never - creates a member it would then never see). Such files stay on disk untouched, simply outside the - projection. ([data-model.md](data-model.md) §1, §10, - [index-engine.md](index-engine.md) §3, [GH #136](https://github.com/AlteredCraft/B2/issues/136)) -- **S3 — `full-reindex ≡ incremental-update`, unconditionally.** Re-deriving one changed note - converges on exactly the state a from-scratch rebuild would produce — including pruning rows for - deleted files on a whole-vault pass. There is no carve-out: with identity **path-keyed** (L1), the - filesystem itself guarantees one member per path, so the "two files presenting one identity" state - that once needed one (GH #81) cannot arise — a copy is simply another note at another path. - ([index-engine.md](index-engine.md) §8, [GH #170](https://github.com/AlteredCraft/B2/issues/170)) -- **S4 — No durable B2-derived state outside the Markdown.** No event log, no sidecar files, no - index-only authored facts. Scope: *B2-derived* data — the human's own directory tree is vault - material, for which the **filesystem is authoritative** (folders are never projected; the tree - listing is a live fs walk). ([data-model.md](data-model.md) "Folders") -- **S5 — Schema change = version bump + rebuild, never a data migration.** Disposability is what - makes this free; a migration script would be evidence S2 broke. - ([index-engine.md](index-engine.md) §3) - -## W — Write discipline: the vault changes only on your command - -- **W1 — B2 makes no unbidden writes. Period.** Every byte B2 writes to the vault is the mechanics of - an operation the human explicitly invoked (W3). Reading a vault — walking it, projecting it, - reindexing it — writes **nothing**, so `reindex` runs unchanged on a read-only vault and a - git-versioned vault shows no diff from having been indexed. This is the claim the `b2id` stamp - used to hold an asterisk over; removing the stamp (GH #170) is what let the asterisk go. - ([data-model.md](data-model.md) §1) -- **W2 — B2 never authors the body, and never asks it to carry B2 syntax.** The body is 100% the - human's document. The lone body write is the mechanical move-repair: rewriting an inbound - `[[oldpath|alias]]`'s *path text* when its target moves — fixing a link the human already wrote, - never adding one, aliases preserved verbatim. ([data-model.md](data-model.md) §0) -- **W3 — The on-command writes are enumerated and minimal:** append one `b2_relations:` entry on - `b2 link` (frontmatter, never the body); the move-repair of W2; the editor save (`Vault::write` — a - byte-honest splice of the *human's own* body bytes, guarded by a content-hash revision); the - frontmatter save (`Vault::write_frontmatter` — the same-guard splice of the *human's own* - frontmatter bytes, body untouched, and otherwise unjudged: B2 owns no line in that block); import - (`Vault::import_file`/`import_path` — the handed bytes copied verbatim, then projected); and - create/move/delete of notes, resources, and folders on explicit command. -- **W4 — B2 never deletes, moves, or archives vault files of its own accord.** Consequences of human - edits (orphans, dangling links, hash-matched move candidates) are *surfaced*, flagged, or proposed — - never silently applied. ([index-engine.md](index-engine.md) §8) -- **W5 — Round-trip losslessness.** `parse → serialize → parse` is byte-identical outside the specific - edit performed; unknown frontmatter keys survive verbatim, in order. B2's one key is namespaced - (`b2_relations`) so it can never collide; a generic `relations:` key is *not* read. A `b2id:` line - left by an older B2 is now exactly an unknown key — never read, never rewritten, never removed; - nothing needs migrating, and deleting `.b2/` is the whole upgrade (GH #170). - ([data-model.md](data-model.md) §1, §6) - -## L — Identity & links - -- **L1 — A note's identity is its vault-relative path, and the graph keys every edge by path** - (GH #170). Both link homes are *already* written by path — a body `[[path]]`, a frontmatter - `b2_relations:` entry — so an edge stores what the human authored, resolved at projection time, - with no machine id in the file and nothing to key on that the vault does not already carry. - Consequence: **rename keeps every backlink resolving *when B2 does the move*** — a move rewrites - the inbound path *text* and re-keys the moved note's rows in one transaction. A move made - **outside** B2 is a delete plus a create: the inbound links surface as dangling (G5) — identified, - never silently dropped — the durability a path handle has in Obsidian, one notch better in that B2 - says so. ([data-model.md](data-model.md) §1, §3) -- **L2 — A note's title is its filename.** The frontmatter `title:` key is recognized but inert — - round-tripped, never driving display, aliases, or search. `b2 link` therefore writes a bare - `[[path]]`, no alias. ([data-model.md](data-model.md) §1) -- **L3 — Notes and resources share one identity model: the vault-relative path, index-only, with no - sidecar files ever.** Since GH #170 the remaining asymmetry is **authoring surface alone**, not - status and no longer identity: a note has frontmatter and authored edges because Markdown is the - one format whose bytes B2 may write; a resource is a peer document B2 can read and never write. - ([data-model.md](data-model.md) §10) -- **L4 — The body is read strictly as ordinary Markdown.** Every body link — wikilink, Markdown link, - embed — is an untyped, **directed** `references` edge; no prose shape (list marker, leading verb) - is ever B2 structure. ([data-model.md](data-model.md) §2) - -## G — The typed graph - -- **G1 — Every edge is authored and active.** An edge exists iff it is written in the Markdown; there - is no `status` column, no suggestion queue, no lifecycle, and nothing inert. Committing is - appending an authored line and re-projecting, never an index mutation. - ([data-model.md](data-model.md) §3, §4) -- **G2 — The edge set is the union of exactly two homes, deduped frontmatter-wins.** Body links - (`origin=inline`, always untyped) ∪ frontmatter `b2_relations:` (`origin=frontmatter`, the **sole** - home of a verb + explanation). Same `(target, type)` in both homes keeps the frontmatter row (it - alone carries the explanation); a *different* verb over a body-linked target coexists (the augment - case). Nothing is ever copied between homes or auto-removed from a file. - ([data-model.md](data-model.md) §0–§3) -- **G3 — The relation vocabulary is a closed three-verb stance core plus a tolerated tail.** - `references` (neutral) / `supports` (for) / `contradicts` (against, symmetric) is the typing - palette and what queries rely on; any other verb is stored verbatim as an opaque tail. The closed - core is a *policy we can relax* (promotion path), never a structural assumption. - ([data-model.md](data-model.md) §2) -- **G4 — Edges are directed and stored once.** Inverse labels are display-only, computed at read - time; B2 never writes a reciprocal link into the target file. ([data-model.md](data-model.md) §2) -- **G5 — An unresolvable link target projects as a surfaced dangling edge, never a dropped one.** - Broken links read as broken (`dst` NULL, authored text kept) and heal on the next reindex once the - target exists. ([data-model.md](data-model.md) §3, GH #12) -- **G6 — The materialized graph is a cache; runtime parsing is the correctness definition.** The - `edges` table exists for what parsing can't serve — backlinks, typed traversal, the discovery - exclusion — and is rebuilt from scratch on every reindex. In v1 resources are edge *targets* only; - `src` is always a note, because an edge must trace to an authored Markdown line. - ([index-engine.md](index-engine.md) §3, [data-model.md](data-model.md) §10) - -## M — The AI seams & the embedding space - -- **M1 — Only enumerated AI seams: `Embedder` and `LlmProvider`.** `b2-core` is model-free and - tested against deterministic fakes; a real model drops in through its seam with **no schema or - flow change**. `Embedder` (text → vector) carries the index's one recorded identity (M2); - `LlmProvider` (chat — streamed, cooperatively cancellable) deliberately carries **none**: nothing - it produces is stored, so swapping chat models never touches the index. Model-compensating - machinery (per-pair adjudication, query expansion, heavy orchestration) is deferred or off by - default — the Bitter-Lesson tenet. A reranker, if it lands, is the next enumerated seam, not an - exception. ([index-engine.md](index-engine.md) §5–§6, GH #151/#153) -- **M2 — The embedding space has one recorded identity: `meta.(embed_model_id, embed_dim)` — and the - compute device folds into it** (a Metal build tags the id `@metal`). Any identity change is a model - swap: `search` **fails fast** rather than mixing spaces, `reindex` drops and re-embeds, and `open` - **never** mutates the vector space. ([index-engine.md](index-engine.md) §6, GH #40) -- **M3 — One embedding space in v1.** Every vault member funnels to *text* through the same model; - multimodal spaces and describers are documented future seams, default-off - ([GH #110](https://github.com/AlteredCraft/B2/issues/110)). - ([data-model.md](data-model.md) §10) -- **M4 — Vectors live in plain tables, scored in-process; their existence *is* the signal; and they - are keyed by the hash of what was embedded.** The vector tables are created at embed time, so - "tables exist" = "this vault has an embedding space" — the fallbacks (BM25-only search on a - projected-but-unembedded vault) key on it. `embeddings` is **content-addressed** - (`text_hash → vector`, GH #170): the embed input is exactly the chunk's stored text, so identical - text has one vector, a renamed or moved note re-embeds nothing, and the only invalidation rule is - "a hash no chunk references is garbage" — pruned by the whole-vault pass. Centroids are the same - derived data keyed by note path — refreshed by the embed pass, dropped on re-chunk. Model identity - is not part of the key because it need not be: a swap drops the whole table (M2). ([index-engine.md](index-engine.md) §3–§4, GH #38) -- **M5 — Note content is never sent off-machine unbidden.** A cloud model endpoint exists only by - explicit user configuration: the default chat configuration is a local endpoint, and a chat - request carries the question *and* retrieved note passages — so the consent moment is the - configuration moment, informed in place (plain-language privacy copy beside the Cloud-models - setting, never a later popup). (GH #151) - -## D — Surfacing & disclosure - -- **D1 — Discovery ranking answers a relative question; the default view answers a quality one; and - no anchor-local statistic ever makes a candidate unreachable.** "What in my vault belongs next to - this note?" is relative, so `b2 similar` and the discovery pane rank by best-passage distance and - the full ranked list stays reachable — `limit` is a cap that under-fills only for want of scorable - notes, and **an empty surface never asserts "nothing relates" from anchor-local statistics**: such - a test cannot distinguish *nothing is related* from *everything is related* (the same geometry - from opposite ends: a single-domain vault went dark on 16 of 17 notes, GH #196). Reachable is not - vouched for, though: **what the default view shows is a claim of quality, and filling to `limit` - regardless is a false one** — always-serve was GH #197's safe interim ruling, and real-vault - dogfooding (2026-08) measured its cost: a pane that always finds ten trains distrust of all ten. - A quality signal may therefore set the **default disclosure boundary** — a fold that is a *prefix* - of the ranked order (a signal that would admit rank 5 while folding rank 2 is inadmissible: row - order, band, and fold must never visibly disagree), with everything below it collapsed but one - gesture away, so a misjudged fold costs a keystroke where the retired gate cost the feature. Every - such signal is **evidence-gated** (it must win the measured bake-off on the orthogonal corpus, the - dense single-domain fixture — where a non-empty default view is absolute — and real vaults via - `make calibrate`, with "no fold at all" an admissible winner) and **continuous in population - size**: a threshold may move banding or the fold, never which rows exist or can be reached. - Strength stays a within-list grading painted from the z, which gates nothing. **The first - bake-off found no admissible fold** (GH #200, 2026-08-22): mutual-kNN reciprocity's window is - empty — on the orthogonal corpus the depth that stops hiding labelled mates is past the depth - that still empties a loner's view, while the dense fixture carries no loner and so supplies only - the lower bound and the absolute; and the two corpora's safe depths are the same *fraction* of - their candidate pools rather than the same constant — so the default view is still the whole - served prefix. The permission above stands unchanged for the next candidate; what the measurement - retires is one rule, not the axis. With no fold there were no discovery-side surfaces or gate - rows for GH #202 to land either: its exit-gate moves are all search's, and the harness keeps only - #200's **structural-zero tripwire**, which re-arms on its own if a fold ever ships. - ([index-engine.md](index-engine.md) §3, GH #196/#197/#200/#202; the - generation side's recall posture and 1-hop exclusion are unchanged) -- **D2 — A served search result is a claim of evidence, and `limit` is a quota nowhere in B2.** - Flow ②'s vector half always has k nearest — *nearest* is a fact about the vault, never evidence - about the query — and RRF fuses ranks, discarding the absolute signals that could tell the - difference, so the pipeline as first shipped could not answer zero: a nonsense query served - `limit` confident-looking results, the same false claim D1 names. The rule: a result in the - default view must trace to positive evidence — a lexical match, or semantic proximity clearing a - bar calibrated per model in the harness — and a query the vault holds no evidence for answers - **"no matches"**, honestly empty, the nearest-by-meaning list never presented as matches. The - clause that once said "at most folded behind D1's disclosure boundary" is struck: **#200 built no - such boundary**, and #202 ruled the human surface *strict* rather than folded (below). D1's - guards apply unchanged: any bar is a distributional constant (process rule 5's `make calibrate` - transfer check), earned against labelled negative queries in the eval corpus before it ships — - and **a labelled relevant query the bar would cut is the search-side tripwire, asserted at zero - with no headroom**. - **The engine now answers it** (GH #201, 2026-08-22): `hybrid_search` carries the discarded - signals beside the untouched fused order — per hit its rank in each list and its own distance, - per query the best cosine; the lexical reading is read beside it where a verdict is wanted - (`Vault::search_evidence`) — and the rule over them is *lexical OR - semantic*, two independent signals so the test can tell "nothing matches" from "everything - matches" where a one-signal one could not (D1's own reason, GH #196). The lexical half is - **IDF-weighted term coverage**: how much of the query's own weight the vault carries, a word in - most chunks weighing ~nothing and a word in none weighing the most — so a stopword is a - measurement, not a word-list, and a query sharing only a *function* word with the vault carries - almost none of its own weight and so is not anchored. Its first form was a hard - df ceiling and that form **failed process rule 5 on the dense fixture** (a 1.5-chunk ceiling - called `drone` and `comb` stopwords in a beekeeping vault, cutting 3 of 15 queries naming notes - the vault holds) — the register keeps that, because it is the same lesson twice: a constant read - off one corpus's distribution describes that corpus, and the fix was to change the *rule* rather - than re-tune the number. The verdict reaches an adapter through `Vault::search_evidence`, which - serves exactly the rows `search` does in the same order. The per-hit **tail** fold is unshipped - by **measurement** now, not by missing labels (GH #206, 2026-08-25): `tail_relevant` made the - labels exhaustive per query, the four-family prefix-cut bake-off ran on both corpora and a real - vault, and no admissible family reaches more than 23 of the **367** filler rows an oracle fold - would cut — the fused order is not an evidence order (filler outranks keep rows it loses to on - every absolute signal), and D1's prefix requirement rightly forbids a fold from re-sorting it. - So the tail complaint is an **ordering** problem, its payment the reranker seam M1 reserves, and - the bake-off re-arms every run - ([crates/b2-embed/evals/README.md](../crates/b2-embed/evals/README.md), the #206 verdict). - **The surfaces carry it now** (GH #202, 2026-08-22), and the verdict is **three-state, each state - a different behavior**: evidence found → serve as always; **no evidence → the honest empty state - and none of the rows** (*strict* — no reveal, no `--all`, no expander: any of those would put the - nearest list forward as candidates after all, which is the claim this invariant says a served row - makes; the nearest list becomes unreachable from the human surface, and that cost is accepted - because it is bounded to one query at a time, never a whole vault, which is what separates it - from the failure GH #196 measured); and **no calibrated bar for the active model → serve as - always, never "no matches"**, since that third state is what the fake embedder and every model - until the harness measures one produce (M2), and folding it into "no evidence" would blank a dev - vault. `b2 search --json` is consequently an **object** — the rows plus the verdict, a documented - break of the array contract, because a query-level reading has nowhere to live in a list of rows - — and it keeps serving the rows at `vouched: false` where the human surfaces show none: an agent - handed rows *plus* an explicit verdict can be honest about them where a reader given rows - alone cannot. - ([index-engine.md](index-engine.md) §4, GH #201/#202, - [crates/b2-embed/evals/README.md](../crates/b2-embed/evals/README.md) process rules) - -## E — Engineering discipline (what keeps the above true) - -- **E1 — The core is deterministic.** No wall-clock and no randomness inside `b2-core`; timestamps - are injected (`created` params) and nothing is minted at all — since GH #170 identity is the path, - so the id generator that was the core's other randomness source is gone rather than merely - injected. Clocks and log subscribers live in the adapters. - ([CLAUDE.md](../CLAUDE.md) Conventions) -- **E2 — `cargo test` is fast, deterministic, and model-free; model quality never enters CI.** - Real-model work lives behind `b2 init` / the out-of-CI eval harness. `#[ignore]` is forbidden — a - hard-to-write test is a signal to re-anchor on the invariant or fix the system. - ([crates/b2-embed/evals/README.md](../crates/b2-embed/evals/README.md), the harness under `crates/b2-embed/evals/`) -- **E3 — The `Vault` façade is the one typed API, and every adapter is dumb.** CLI and desktop - commands are deserialize → one façade call → serialize; logic that wants to live in an adapter - belongs behind the façade. Dependencies point one way (adapters → core, never back); façade ops are - added on need, never pre-built. ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md)) -- **E4 — User-facing errors are generic and actionable, never leaking internals.** Full detail goes - to logs / `B2_DEBUG`, not to the terminal or webview. ([CLAUDE.md](../CLAUDE.md) Conventions) -- **E5 — Note content is untrusted input; rendering is a trust boundary.** Authorship is not trust: a - `.md` can come from anyone (a shared vault, a downloaded or web-clipped note), so B2 treats rendered - note content — and model output, which is the same class of input (M1) — as hostile. Two rules hold - together: B2 HTML-escapes every value *it* interpolates into UI chrome, and the **single** - Markdown→HTML render seam (`renderMarkdown`) sanitizes its output before it reaches the DOM, so no - note can inject executable markup at any call site. The webview CSP (`default-src 'self'`, no inline - scripts) is a second, independent layer — defense-in-depth, never the sole guard. The same posture - governs a note's **links**: the webview *is* the application, so a note's link never navigates it — - a web link (`http`, `https`, `mailto`) is an **OS handoff** performed host-side behind a scheme - allow-list, and every other scheme is refused rather than handed to an OS that would launch whatever - app claims it. - ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md), GH #77) - -## C — Concurrency: many readers, one builder - -- **C1 — Any number of processes may hold one vault's index open at once.** The index is a - disposable projection, so concurrent *readers* are unrestricted and **a reader is never - refused** — opening an index already at the current `schema_version` takes no write lock at - all, so a running reindex cannot turn a `search` into an error. Creating and rebuilding that - projection is the one step that must be **atomic and serialized**: an `open` observes a complete - schema at the current `schema_version`, or waits out a bounded budget for the opener building one — - **never a partial schema**. The no-partial half is absolute; the waiting half is deliberately not — - past the budget a stuck writer is reported rather than hung on. "Complete" is checked, not assumed: - a current stamp over missing tables is stale and rebuilt from empty, since surviving rows would look - up-to-date to an incremental reindex and break S3. The same holds for the index's *other* - drop-and-rebuild, the vector tables (M4). Concurrent *writers* stay single-in-flight by the - `reindex` advisory lock — which readers never take, and so cannot cover this. - ([index-engine.md](index-engine.md) §3, GH #111, #114) - -## K — Interaction: keyboard-first - -- **K1 — B2 is fully operable from the keyboard; the mouse is an accelerator, never a requirement.** - Every action the mouse can take has a keyboard path — a focusable control in a sensible tab order, or - a documented shortcut — across the whole desktop surface: the file tree and open/create/rename/move/ - delete, search and find-in-note, edit mode (⌘E) and every in-editor chord, discovery and linking, - chat (⌘J), the graph, and each menu/modal (`Escape` dismisses, `Enter` confirms, focus is trapped - while an overlay is open and restored on close). Focus is always visible and follows platform/ARIA - conventions. Three corollaries: - - **A chord live in the app is B2's to document, whoever authored it.** The macOS menu bar's - accelerators are *declared* rather than inherited from Tauri's default (`b2-desktop/src/menu.rs`), - so the reference sheet can list them and the conflict gate can see them — a chord nothing - enumerates cannot be found, and the app cannot warn about landing on it. - - **The chords are the user's, not B2's.** Every chord B2 dispatches is re-recordable from - Settings → Keyboard and stored as a UI preference (`localStorage`, like the theme — never vault - state, never the index). The exception is narrow and stated per row (`Binding.fixed`): the - platform's own reflexes — ⏎/Esc in a text field, a dialog's default button, ⏎/Space on a - `