feat(seeding): add --populate-summaries to rebuild summaries only

[m2ux]

Why

Repairing summaries previously meant a full re-seed: documents reloaded, re-chunked, concepts re-extracted. The stage cache stores the document overview and the concept extraction under one key and only counts as a hit when both are present, so there was no way to invalidate a summary without paying for concept extraction (1–2 min/doc) as well.

On the live database that mattered: 58,547 of 60,083 concepts (97%) had an empty summary, plus 8 catalog fallbacks and 17 categories still on the generated description. Concept summaries feed 20% of concept-search scoring.

What

A standalone mode on the seeding script that fills in missing summaries against an existing database — no --filesdir, no document loading:

# Report what's missing — no LLM calls, no writes
npx tsx hybrid_fast_seed.ts --populate-summaries --dry-run

# Fill the gaps (all three tables by default)
RUST_LOG=error npx tsx hybrid_fast_seed.ts --populate-summaries

# Narrow it down, or try a few rows first
npx tsx hybrid_fast_seed.ts --populate-summaries=concepts --summary-max-items 50

Table	Regenerated from	Missing means
catalog.summary	existing chunks in reading order → overview → re-enriched with the row's concept/category names → vector re-embedded	empty, Document overview (N pages), or a failure marker
concepts.summary	concept name	empty
categories.summary	category name	empty, or identical to the generated description

Also: --force-summaries, --dry-run, --summary-batch-size, --summary-flush-size, --summary-max-items, --summary-model.

Notable decisions

Merge-insert, not drop-and-recreate. Rows are rebuilt field-by-field from the live schema and written with mergeInsert('id').whenMatchedUpdateAll(), so unknown and future columns pass through and the Arrow schema is never rewritten. Batches flush every 250 summaries, so an interrupt keeps what it wrote and a re-run continues. (Batched CASE WHEN updates were the first design — Lance's SQL planner rejects CASE.)

Deletes scripts/populate_summaries.ts. It did the job this replaces, but rewrote the concepts table from a hard-coded field list that predated the current schema: it dropped catalog_titles and adjacent_ids, and renamed related_ids to related_concept_ids. All three are read at query time by lancedb-concept-repository.ts, so running it silently destroyed the concept graph.

Chunk ordering bug fixed. Chunks are ordered for summarisation by a stable sort on page number only. Chunk ids are hash-based and carry no sequence, and EPUBs store every chunk as page 1 — using ids as a tiebreak shuffles the document. Before the fix three of eight documents started mid-book ("based on static models, as shown in the…"); after, they start at the title page or ToC.

One prompt, one implementation. generateDocumentOverview moved into summary_generator.ts; the seeder's callOpenRouterChat now delegates to it. No behaviour change for seeding (rate limiting stays opt-in, since seeding generates overviews across parallel workers).

Verification

• 31 new tests; full suite 1525 passed, 3 files skipped. Build typecheck clean.
• Ran the backfill against a copy of the live database with stubbed LLM calls: row counts and Arrow schemas identical on all three tables, 30 concept summaries changed with zero drift in any other column, all 8 catalog rows re-summarised with vectors refreshed and metadata intact.
• Write path benchmarked on a copy of the real 60k-row concepts table: ~110 ms per merge, negligible disk churn.
• --dry-run against the live DB reported 8/327 catalog, 58,547/60,083 concepts, 17/794 categories.

Note for reviewers

src/concepts/summary_generator.ts carries an in-flight one-line change from the working tree (DEFAULT_MODEL → x-ai/grok-4.6) that could not be split out of the file. The related src/config.ts model rename, its test, and a .gitignore edit were deliberately left out of this branch.

🤖 Generated with Claude Code