diff --git a/docs/plans/tag-properties.md b/docs/plans/tag-properties.md new file mode 100644 index 00000000..066b80c6 --- /dev/null +++ b/docs/plans/tag-properties.md @@ -0,0 +1,276 @@ +# Tag Properties — structured data, harvested by the pipeline + +## Status + +Plan, drafted 2026-08-08; manual-entry, lifecycle, and extraction-honesty +semantics settled in a same-day design talk-through. Direction settled in +conversation after analyzing Logseq 2.0's DB release; no implementation yet. Supersedes the "templates & +structured atoms" framing from the Discord feature analysis — this doc is the +deliberate answer to that demand, not a transcription of it. + +## Thesis + +Logseq 2.0 rebuilt itself around manual structured data: a tag carries typed +properties, and tagging a node hands the *human* a form to fill. Atomic already +has the two things Logseq spent years building — a canonical SQLite store and a +tag system — plus the one thing Logseq doesn't have: a pipeline that already +reads every atom with an LLM. So we invert the fill direction: + +**A tag declares the properties its atoms tend to have. The pipeline fills +them.** Tag a transcript `#meeting` (usually automatically) and `date`, +`attendees`, `decisions` get extracted in the same breath — typed, queryable, +and provenance-tracked. The human never sees a form. + +The compounding loop this closes, entirely on existing features: auto-tagging +assigns the tag → extraction fills the properties → the tag's wiki prompt +(shipped v1.44) writes the running summary → chat answers property questions +deterministically instead of hoping semantic search lands on the right chunk. + +## Design principles (settled — revisit deliberately, not casually) + +1. **Structure is harvested, not imposed.** Declarations are lightweight hints + that steer extraction. Nothing validates, nothing blocks a save, no atom is + ever "malformed". +2. **The tag is the schema carrier.** No separate "type" entity. This is the + third entry in the per-tag config seam (`autotag_description`, wiki + prompts) and reuses its storage, routes, and context-menu patterns. + Declarations inherit down the existing tag tree — a child tag carries its + ancestors' declarations. No multi-parent `Extends`; the tree we have is the + inheritance we support. +3. **Manual beats extracted, always.** User-authored frontmatter and + panel-entered values are manual intent; extraction never overwrites them. + Same rule re-tag established for tags. +4. **The pipeline never edits the user's text.** Extracted values live in the + index table only. User-authored frontmatter stays in the content, + untouched. Export materializes *both* as frontmatter so the markdown ZIP + remains complete — the mistake to avoid is Logseq's lossy markdown export. +5. **The feature costs nothing until used.** An atom whose tags declare no + properties adds zero LLM calls and zero UI. Cost scales with adoption, not + with corpus size. +6. **Small type system.** `text`, `number`, `date`, `boolean`, and multi-value + variants of text (stored as JSON arrays). No cardinality apparatus, no + constrained-choice UI, no per-tag visibility matrix. Atom-references are + explicitly deferred (see Open questions). +7. **Extraction is honest or it is silent.** Every generated schema field is + nullable and the prompt instructs the model to fill only what the source + states with high confidence — omit the rest. An absent value is an + affordance (the panel invites the human to fill it); a guessed value is a + trust bug. And some fields are categorically not the model's to fill — + judgment fields like `priority` or `status`, where the failure mode isn't + "the text doesn't state it" but "the text *implies* it": each declaration + carries `auto_extract` (default on), and opted-out fields are excluded + from the generated schema entirely — never shown to the model, not + instructed-to-skip. Same invitation-only philosophy as auto-tag targets, + one level down. + +## Data model + +Two tables per data database (SQLite V26, Postgres 027 — follow the V25/026 +conventions exactly, including the transactional-DDL lesson from the +adversarial review of PR #228): + +``` +tag_property_defs + id TEXT PRIMARY KEY + tag_id TEXT NOT NULL REFERENCES tags(id) ON DELETE CASCADE + name TEXT NOT NULL -- snake_case key, unique per tag + value_type TEXT NOT NULL -- text | number | date | boolean | text_list + description TEXT NOT NULL DEFAULT '' -- doubles as the extraction hint + auto_extract BOOLEAN NOT NULL DEFAULT TRUE -- principle 7: off = manual-only field + sort_order INTEGER NOT NULL DEFAULT 0 + UNIQUE(tag_id, name) + +atom_properties + atom_id TEXT NOT NULL REFERENCES atoms(id) ON DELETE CASCADE + name TEXT NOT NULL + value TEXT NOT NULL -- canonical string; JSON array for text_list + value_type TEXT NOT NULL + source TEXT NOT NULL -- manual | extracted + tag_id TEXT -- def-bearing tag that produced it (extracted only) + updated_at TEXT NOT NULL + PRIMARY KEY (atom_id, name) +``` + +One row per (atom, name): when a manual value exists, extraction skips that +key entirely (principle 3 enforced at the write, not at read time). `tag_id` +on extracted rows keeps the audit trail and lets a tag's backfill re-extract +only its own keys. + +Keying on `(atom_id, name)` rather than `(atom_id, tag_id, name)` is a +semantic choice, not a shortcut: properties describe the **atom**; tags +contribute vocabulary. An atom tagged `#meeting` + `#project-alpha` gets the +union of both declaration sets, and two tags declaring the same name (with +the same type) converge on one field — shared names are shared vocabulary, +the way a column name means the same thing across joined tables, and either +tag's queries see the value. Same name with *different* types is a +user-fixable smell: first declaration wins, a debug log records the +collision, no UI ceremony in v1. Postgres adds `db_id` to both tables with the same fencing +the tags queries carry — and the same two-db fencing test shape +(`pg_tag_wiki_prompts_fenced_by_db_id` is the model). + +`Tag` model stays lean (the tag tree ships hundreds of rows); defs are fetched +on demand like wiki prompts. `AtomWithTags` responses gain a `properties` +array — values are small, and the reader panel needs them without a second +round trip. + +## Extraction flow + +Anchor: the tag-extraction step the pipeline already runs +(`embedding.rs` — `extract_tags_from_content` via +`call_structured::` in `extraction.rs`, which already feeds +`autotag_description` into the prompt via `get_tag_tree_for_llm`). + +After tag assignment resolves (auto or manual), collect the atom's assigned +tags' declarations — walking up the tag tree for inherited defs — and if any +exist, make **one** additional structured call: + +- The JSON schema is *generated* from the merged declarations (name → type, + description → field docs) — but only from fields with `auto_extract` on; + manual-only fields never enter the schema. Every field is nullable, and + the prompt instructs: fill only what the source states with high + confidence, omit everything else (principle 7). Two tags declaring the + same name merge to one field; first-writer wins on conflicting types, and + a debug log records the collision. +- Model: the tagging model (`provider_config.llm_model()`), same tier and + temperature philosophy as tagging — this is utility extraction, not + agentic work. +- Writes: upsert extracted rows, skipping any key with a manual row. Emit a + `PropertiesExtracted` event through the existing callback → broadcast + bridge so the reader panel refreshes live. +- Failure: log-and-leave-empty, exactly like tag extraction. Never blocks the + pipeline. + +Manual tag add/remove outside the pipeline (tag chip UI) triggers the same +pass for the affected atom. Content edits re-extract on the normal pipeline +run; extracted rows are cheap to overwrite (manual rows are not touched). + +## Manual entry: the panel first, frontmatter for power users + +The reader's properties panel is the **primary** manual path, and it is more +than an editor for existing values — declared-but-empty properties render as +affordances. An atom tagged `#meeting` whose transcript contained no date +shows `date: —` with a date picker; every declared key (including manual-only +ones, and those inherited through the tag tree) appears as a typed empty slot. +That empty slot does double duty: manual-entry invitation *and* an honest +signal about what the source actually contains (principle 7 guarantees +extraction left it absent rather than guessing). Nothing is required, nothing +blocks — the form is scaffolding, not a gate. + +Panel rules: + +- **Editing an extracted value converts it to manual.** The user's touch is + manual intent; the row's provenance flips and re-extraction never claws it + back — the per-value analog of manual tags surviving re-tag. Provenance is + shown subtly (extracted values marked the way auto-applied tags are + distinguished from manual ones). +- **Ad-hoc keys are allowed** ("+ add property"): undeclared properties are + pure manual data — extraction ignores them, queries and export see them. + Frontmatter can contain any key, so the panel must too; the two manual + paths stay equal in expressive power. + +Frontmatter remains the power-user and portability path. On atom save, a +leading `---` block is parsed for **flat** `key: value` pairs into manual rows +(typed by simple inference: number, ISO date, true/false, `[a, b]` lists; +everything else text). No YAML dependency and no nested structures — the +import path's frontmatter handling (`lib.rs` Obsidian import) shows the +precedent and the restraint. The chunker excludes the frontmatter block from +embeddings (`chunking.rs`). Panel edits write manual rows — they never +rewrite the user's text (principle 4). Export (markdown ZIP, per-atom) +materializes the merged property set as frontmatter. + +## Lifecycle: untagging and declaration edits + +Extracted values are a **projection** of (content × current declarations); +manual values are assertions. Every lifecycle rule follows from that split: + +- **Untagging re-projects.** Removing `#meeting` drops the extracted values + its schema produced (`tag_id` audit column) unless a remaining tag still + declares the key. Manual values — including edit-to-own conversions — + survive unconditionally: removing a tag doesn't make the date less true. + The dominant untag case with an auto-tagger is mis-tag correction, and the + extracted values are products of the same mistake; keeping them would + leave machine-authored ghost data. Nothing is lost that matters: + re-tagging re-extracts in one pipeline pass. +- **Deleting a declaration** from a tag sweeps that key's extracted values + across the tag's atoms the same way; manual values survive as ad-hoc + properties. + +## Backfill + +Per-tag action mirroring `retag_all_atoms` / `claim_all_for_retagging_sync`: +"Extract properties" on a tag sweeps its atoms (including descendants' +atoms? — no: the tag's own atom set, descendants opt in from their own tag, +keeping cost visible), re-running extraction for that tag's keys only. +Surfaced next to "Re-tag" in whatever UI hosts it, and in the tag Properties +modal. Progress via the existing event pattern. + +## Phases + +**Phase 1 — declarations, extraction, display (shippable alone).** +Migrations; storage trait + both impls; defs CRUD on the per-tag config seam +(`GET/PUT /api/tags/{id}/properties`, modeled byte-for-byte on wiki-prompts); +pipeline extraction pass; frontmatter parse + chunker exclusion; reader +properties panel (declared-but-empty slots with typed inputs, edit-to-own, +ad-hoc keys); tag context-menu "Properties…" modal (name / type / +description / auto-extract toggle rows, mirroring the wiki-prompt modal's +load-gate lessons); untag/declaration-edit re-projection; per-tag backfill. +Export materialization. + +**Phase 2 — the query surface (where the value compounds).** +Property predicates in search (`routes` + search UI filter row); an agent tool +in `chat_tools.rs` (`query_atoms_by_properties`: tag + simple predicates — +equals / contains / before / after / is-set), scope-gated like its siblings. +This is what turns chat from "semantically similar chunks" into "what's +overdue" answered from the index. List-view sort by property rides along if +cheap. + +**Phase 3 — templates as capture rituals (separate doc when reached).** +A template body attached to a tag: creating an atom *from* the tag pre-fills +the shape (`{{date}}`, `{{title}}` placeholders only). Validated by Logseq's +"apply template to tags" — but deferred until phases 1–2 prove the extraction +loop, because templates without the query surface are just snippets. + +## Cost model + +Zero additional calls for atoms with no def-bearing tags. One structured call +per pipeline run otherwise — the same cost class as the tagging call it rides +beside. A 500-atom backfill on a tag ≈ 500 tagging-tier calls, batched and +progress-reported like re-tag. No embedding cost anywhere (properties are not +embedded; they are the *deterministic* complement to embeddings). + +## Testing + +The PR #228 review taught the pattern this plan inherits: + +- Precedence pinned end-to-end: a wiremock test asserting the extraction + request's generated schema contains the declared fields, and that a manual + row survives a re-extraction that returns a different value. Each test must + fail under the specific mutation it guards. +- Both backends for every storage behavior; PG fencing test for `db_id`; the + PG suites now actually run in CI (test.yml — verified in #228), so new + PG-gated tests execute. +- Migration crash-safety: transactional DDL, literal version stamps. +- Frontmatter parser: property-based tests over the flat grammar; malformed + frontmatter degrades to "it's just content", never an error. + +## Open questions (deferred, not blocking Phase 1) + +- **Atom-reference type**: `attendees` pointing at person atoms would light up + the graph, but resolution (name → atom) is fuzzy and belongs with the trust + layer work. Ship `text` / `text_list` first; references are additive later. +- **Card/list display**: should grid cards surface a property chip (e.g. due + date)? Decide after the reader panel exists and real usage shows which + properties earn ambient display. +- **Wiki integration**: feeding a properties digest into wiki generation + (deterministic tables in articles). Natural, but rides on Phase 2's query + layer. +- **Confidence scores**: the trust-layer theme wants tagger-confidence judged + by a second model; extracted properties would join that scheme when it + exists. Schema reserves nothing for it — `source` can grow values later. + +## Out of scope (by principle, not by laziness) + +Validation and required fields; cardinality rules; constrained choices; +multi-parent inheritance; property-triggered automation (repeating tasks +etc. — the automations-vision doc owns triggers); an expression language; +nested frontmatter; writing extracted values into content.