Map: a from-scratch design for paperscale embed

[charitarthchugh]

Destination

A locked, implementation-ready design for a from-scratch paperscale embed: one
document committed to the repo holding both the design and the implementation plan,
synthesized from every decision this map makes. No production code ships from this map —
the map is done when nothing is left to decide.

Notes

Domain. paperscale embed reads the results/*.jsonl that OCR runs produce and turns
each document's text into vectors against an external, OpenAI-compatible /v1/embeddings
server. Vocabulary lives in CONTEXT.md.

Skills every session should consult: /grilling and /domain-modeling by default;
/research for the research ticket; /prototype for the prototype ticket. This repo has a
graphify knowledge graph — run graphify query "<question>" before reading source.

Prior art. Branch feat/embed holds an abandoned 4,869-line implementation. It is
reference only. Do not read it by default — the point of starting over is to avoid
inheriting its assumptions, and reading it pollutes the session's context with a design we
already rejected. Consult it only when a specific ticket names it.

Standing decisions

Fixed while charting. Not open for re-litigation inside a ticket; a ticket that needs one
of these reversed should say so and stop.

1. From scratch. New branch off main. Nothing is salvaged as code.

2. Pinned model families: Qwen3-Embedding and Nemotron-3-Embed. Both are 32,768-token
   context. "Long context" therefore means 32K, not "the document always fits" — at
   ~3.6 chars/token that is roughly 45 dense pages. Overflow is therefore the minority
   path, not the common one — measured against the 49-Document smoke corpus it is 3 of 49
   (~6%), stable across chars/token ratios of 3.0–4.0. It is real and large when it happens
   (the tail runs to 89 pages / 218k chars), which is what makes Chunking strategy when a document exceeds the discovered limit #24's greedy packer and
   Pooling: one document vector from N chunk vectors #25's token weighting worth their complexity. Amended: the original text asserted
   overflow was "routine, not exceptional"; the only measurement available contradicts that,
   on a small sample.

3. Uniform output shape, always. Every Document emits Chunk vectors plus one pooled
   Document vector, and both Sinks carry both. Amended after
   #36: paperscale serves two
   Consumers, and they read different artifacts. RAG reads the Chunk vectors; the
   Document vector is at most a coarse-to-fine filter. Classification reads the Document
   vectors — a per-Document label needs one fixed-width feature row per Document, and there is
   no query to reduce against, so the reduction cannot be deferred to read time. Neither
   artifact is "the convenience", and Write the design and implementation document #31 must say so per Consumer rather than ranking them.
   This is also why zero of How comparable pipelines are actually implemented #36's fourteen comparable pipelines produce a Document vector:
   every one of them is a retrieval pipeline, so the unanimity was an artifact of a homogeneous
   sample and the divergence is a different job, not a trade. Splitting the two artifacts
   across the two Sinks was considered and rejected — it would put coarse-to-fine RAG across a
   cross-Sink join, leave The LanceDB schema and its namespacing #27's per-Document provenance homeless, and cost the property Is MRL truncation offered, and how does an adapter express validity? #34 and
   Pooling: one document vector from N chunk vectors #25 earned deliberately, that one Sink alone is enough to recompute and verify the
   Document vector. The common case is n_chunks == 1, where the Document vector is a
   bit-exact copy of the sole Chunk vector; that duplication is the price of a Sink that is
   complete on its own.

4. --embed-model selects an adapter, mirroring the existing --ocr-model registry in
   src/paperscale/models/__init__.py. The rule is ask the server for everything it can be
   trusted on; the adapter carries only what the server cannot be trusted for. Revised
   after #23 — the original "hardcode
   nothing but the model id" is not implementable. The residue is explicitly three facts:

   1. MRL validity, as a range [min_dim, native_dim] — never a list. Restated after
      #34: neither pinned family
      publishes an enumeration of valid dimensions, and native_dim doubles as the assertion
      that the server has the right model loaded.
   2. The instruction convention and its literal strings, for both query and document sides.
   3. The model's card context length. Restated after
      #37: the validated context length
      is a rule, not a constant the Adapter carries alone. The budget defaults to
      min(card, server_max_model_len). --context-length overrides it, bounded by the server
      and not by the card: above server_max_model_len it is rejected, because that is a
      guaranteed hard failure; above the card it is allowed with an explicit warning, because
      that is a quality risk the operator may knowingly take. A default
      vllm serve advertises 262144 for Nemotron and 40960 for Qwen3-8B against a documented 32768,
      so the server cannot be trusted as an upper bound; but an operator serving with
      --max-model-len 8192 makes the card wrong as a lower one. Neither source is safe alone.
      The Adapter's concrete contents, and the Chunk budget as numbers #37 also found a third number on both cards — the length each actually exercises (Qwen3
      8192 in its own examples, Nemotron 4096 for its published evaluation numbers) — recorded and
      deliberately not used, since quality evaluation belongs to the Consumer.

   The output dimension is obtained by probe — one cheap request — never by asking, on every
   engine surveyed. Everything else is asked of the server and recorded.

5. Two sinks: per-document .npz files and LanceDB. No Qdrant. No safetensors.

6. Document name mirrors the markdown export: Source-File lexically normalized
   (os.path.normpath), then leading / stripped, then remaining .. and empty components
   removed, extension appended (case.pdf -> case.pdf.npz). Append replaced "extension
   replaced" after #41: replacement was
   chosen to mirror the markdown export, and Markdown export silently overwrites when two sources differ only by extension #32 showed that is exactly where the export is wrong.
   Normalization added after
   #28: the export's sanitizer drops
   .. instead of resolving it and keeps ., so /a/b/../c.pdf and /a/b/c.pdf derive the same
   name. Resume's correctness rests entirely on name stability, so embed cannot inherit that. A digest of the Source-File string
   (never of the text) exists for every document as a permanent backup identity and as the name
   itself when no usable path exists. It is not a collision tiebreak — What happens when two Documents derive the same name #41 removed that third
   job: a tiebroken name cannot be made stable across Invocations, which Resume requires, so a
   collision surviving the rules above is fatal at startup instead.
   Corrected in #22 —
   this decision originally said "relative to the input directory", but embed reads JSONL and
   never sees a PDF input directory.

7. No content-change detection. Resume asks one question: "do I know this name?"
   Re-OCR-ing a corpus is therefore the user's problem — the docs must carry the warning
   that the code no longer does.

8. Full stats + logs TUI panel, built on the existing RichReporter / Phase in
   src/paperscale/tui.py.

9. The adapter applies the document-side instruction, and records it in every output.
   Revised after #23 — the map
   generalized from Qwen3 and assumed the convention was query-side only. It is not: Nemotron-3-Embed
   requires passage: on documents. Applying it is document-side work and always was in scope;
   only the query side stays out. Recording is therefore load-bearing rather than defensive — a
   consumer that does not know documents were prefixed will build mismatched queries. Sharp edge:
   TEI accepts --default-prompt and /info does not report it, so a server can silently
   double-prefix.

10. vLLM is the only supported engine. Narrowed from "vLLM and SGLang" after
    #33 showed the two pins do not
    compose — SGLang does not serve Nemotron-3-Embed. Rather than carry engine validity per
    adapter, both pinned model families keep working by dropping the second engine. TEI, Ollama
    and NIM were already out: TEI and Ollama truncate silently by default and their
    OpenAI-compatible routes cannot be made truncation-safe, and NIM cannot report max input
    tokens over HTTP. Silent truncation is unacceptable because stored offsets would describe
    text that was never embedded.

    What this costs, knowingly: SGLang's GET /v1/loads reported GPU-busy prefill throughput
    directly and needed no flag. vLLM has no single equivalent — but its /metrics is mounted
    unconditionally and src/paperscale/vllm_stats.py already parses it, so the panel is fed
    either way. What is lost is the busy-time ratio, not the numbers.

Decisions so far

• The document identity rule: one name, three jobs
  — the name mirrors the markdown export (lstrip("/"), .. stripped, extension replaced), the
  run label appears only when one invocation embeds more than one run, and a digest of the
  Source-File string exists for every document as backup identity and as the last-resort name.
  Corrected standing decision 6; Source-File is knowingly left unnormalized. Rules 3 and 5 amended
  by What happens when two Documents derive the same name #41 — the extension is appended, not replaced, and the digest is not a tiebreak. The
  tiebreak-vs-fatal contradiction Clear the stale text from the closed record #39 flagged is closed.

• The stats + logs panel — variant A:
  the shape the repo already renders. Groups run / server / issues, one document bar, no
  verdict string (it did not survive 80 columns) and no chunk bar (the chunk total is unknowable
  mid-run). Carries three prerequisite tui.py changes into Write the design and implementation document #31. Prototype on branch
  prototype/embed-panel.

• What an embedding server will tell you, and what it never will
  — full four-engine inventory on branch research/embedding-server-discovery. Dimension is a
  probe, never an ask. The residue is three facts, not two: the model's validated context
  length is undiscoverable and every engine over-reports it. TEI and Ollama truncate silently
  by default and their /v1/embeddings route cannot be made safe. Nemotron requires passage:
  on documents, so the instruction convention is not query-side-only. Contradicts standing
  decisions 4 and 9; both raised rather than reversed.

• What SGLang exposes for embeddings, and whether it truncates silently
  — findings on branch research/sglang-embedding-surface. SGLang errors on oversized input by
  default and, uniquely among the five engines surveyed, the setting is askable (/server_info),
  so truncation safety becomes a startup assertion rather than an assumption — but there is no
  per-request truncate field, so What an embedding server will tell you, and what it never will #23's "send the flag on every request" has no SGLang
  implementation. GET /v1/loads needs no flag and gives GPU-busy prefill tok/s directly, which
  vLLM cannot; conversely SGLang's /metrics is off by default. Engine detection is via
  /v1/models -> data[0].owned_by, and error body shape differs (vLLM nests under error, SGLang
  is flat). Decisive: SGLang does not serve Nemotron-3-Embed — standing decisions 2 and 10 do
  not compose.

• Chunking strategy when a document exceeds the discovered limit
  — greedy page packing, no overlap, token counts asked of vLLM POST /v1/tokenize, boundaries
  recorded as character offsets and page spans. The JSONL's pdf_page_numbers spans already
  tile the text, so page-respecting chunks cost no offset math; because splitting a string can
  only increase its BPE count, packing by summed per-page counts can never overflow and needs no
  re-verification. The common case is one tokenize call per document. A page that alone exceeds the
  budget is cut at its last newline, else hard — text is never dropped. Hands Pooling: one document vector from N chunk vectors #25 the fact that
  chunks are unequal, so mean pooling is length-biased.

• Is MRL truncation offered, and how does an adapter express validity?
  — yes: --embed-dim, default 768, applied client-side. Both pinned families publish a
  range (32→native), not a list, so validity is two adapter constants native_dim / min_dim;
  the ticket's premise that they differ in kind was wrong. Server-side dimensions works on vLLM
  but 400s on a default launch, since neither family declares is_matryoshka — and the two routes
  are provably identical anyway, so the choice was operational. Slice and re-normalize each chunk
  vector before pooling, so the document vector is reproducible from the stored chunk vectors.
  Records stored_dim and native_dim; native_dim also stops a run against the wrong model.
  Restated standing decision 4's residue item 1.

• Pooling: one document vector from N chunk vectors
  — token-weighted mean of the stored chunk vectors, re-normalized; n_chunks == 1 short-circuits
  to a copy so the identity case is bit-exact, not approximate. Weighting by token_count makes the
  document vector approximately invariant to where Chunking strategy when a document exceeds the discovered limit #24 happened to cut. Every document gets one: a
  sentinel for diluted long documents would need a threshold no measurement here could justify, and
  that judgement belongs to the consumer, which has the chunk vectors. Two of the ticket's bullets
  were already closed by Chunking strategy when a document exceeds the discovered limit #24 and Is MRL truncation offered, and how does an adapter express validity? #34.

• The .npz file: arrays, dtypes, and provenance without a header
  — provenance splits by scope: one Invocation manifest at <out>/paperscale-embed.json for the
  seven facts that never vary, a per-Document sidecar .json for the four that do, and the .npz
  holds eight arrays and no metadata. A dict inside an .npz is not merely inelegant, it is
  unreadable without allow_pickle=True. No chunk text (UTF-32 costs 4x, and UTF-8 would put
  character offsets and byte offsets in one file), no compression (~10%), int32 throughout,
  chunk_index/n_chunks dropped as derived. Pins the path-digest as sha256(...)[:16], which The document identity rule: one name, three jobs #22
  left open and The LanceDB schema and its namespacing #27 must match. A second Invocation compares the manifest's invariants and stops on
  disagreement. Sidecar first then .npz, both via rename, so the .npz implies the sidecar.
  Added Invocation to CONTEXT.md.

• The LanceDB schema and its namespacing
  — two tables (documents, chunks), one pair per database, Runs separated by a run_label
  column rather than a table name, and the seven invariant facts in write-once table
  metadata. That immutability is what makes the wrong-model check an assertion rather than a
  comment; the fixed-size-list column enforces Is MRL truncation offered, and how does an adapter express validity? #34's width check for free. Table-per-label was
  rejected on an obstacle, not taste: table names allow only [A-Za-z0-9._-] and run labels are
  unvalidated, so it needs a second name-mangling rule. Upsert via merge_insert, and on chunks
  a scoped when_not_matched_by_source_delete — measured: a plain upsert leaves phantom Chunks
  behind when a re-embed yields fewer of them. BTree index on document_name, no vector index
  (IVF_PQ is lossy and the recall trade is the Consumer's). Corrected the ticket: Lance is
  versioned, so overwrite does not lose data, and its list_versions() replaces The .npz file: arrays, dtypes, and provenance without a header #26's invocation
  log. Namespaces exist in 0.37.1 but create_table cannot use them.

• How resume knows a document is done
  — resume derives its state from the outputs; no manifest, no flag files. os.walk over 20,000
  Documents costs 29 ms against a manifest read's 3 ms, so the manifest's only advantage was 26
  milliseconds, and a second source of truth can desynchronise. A Document is done when every
  enabled Sink holds it, which turns The LanceDB schema and its namespacing #27's double-write conflict into a self-healing gap. Empty
  Documents are recorded as empty outputs — zero-length arrays in the .npz, a NULL vector in
  LanceDB, which search skips. --no-resume re-embeds and overwrites but deletes nothing,
  diverging from the OCR side, because an embed output is a deliverable rather than scratch. layout
  joins the manifest's invariant block as an eighth fact. Accepts The .npz file: arrays, dtypes, and provenance without a header #26's write ordering, revises
  standing decision 6, and drafts the standing-decision-7 user warning.

• Concurrency, batching, and what the panel measures
  — the Document is the unit of work, already forced by The .npz file: arrays, dtypes, and provenance without a header #26/The LanceDB schema and its namespacing #27/How resume knows a document is done #28; a request is bounded by a
  token budget, never a count of Chunks, because Chunking strategy when a document exceeds the discovered limit #24's greedy packing makes Chunk sizes vary by
  orders of magnitude and Chunking strategy when a document exceeds the discovered limit #24 already knows every count; concurrency is a small fixed default with
  vllm:num_requests_waiting as the tuning instrument, since What an embedding server will tell you, and what it never will #23 found vLLM publishes no batch
  ceiling to derive one from. Requests mix Documents and therefore split on terminal failure,
  re-issuing one at a time so a single poison Document cannot fail forty. Retry taxonomy mirrors
  try_single_page_with_backoff. The panel needs no new plumbing — but it must read prompt_tps,
  never gen_tps, which is structurally zero for a prefill-only workload. empty moves from
  issues to run (How resume knows a document is done #28 made it an outcome, not a problem); the freed slot shows retrying. Adds
  the end-of-run report: non-zero exit on any failure, and <out>/paperscale-embed-failures.txt.
  Every constant is marked unmeasured — no live embedding server existed to measure against.
  Amended after How comparable pipelines are actually implemented #36: the fixed default stays, but the panel stops being merely passive — it
  emits an advisory when vllm:num_requests_waiting stays above zero for a sustained window
  (~60 s), naming the flag: "queue depth sustained; --concurrency 64 may be too high for this
  server". Advisory only — no control loop, so nothing can oscillate.

• The CLI surface: flags, defaults, and how embed installs
  — a subcommand shaped like paperscale evaluate, hyphenated throughout; .npz by default with
  --lancedb PATH as its own opt-in and --no-npz to disable the file sink; run labels enforced
  against [A-Za-z0-9._-]+ in embed only, since _parse_runs is shared with evaluate where
  the constraint was never needed; one embed extra following tui, imports inside the handler.
  Concurrency, batching, and what the panel measures #30's unmeasured defaults now have an in-repo anchor: --pplx-concurrency 64 and
  _MAX_TOKENS_PER_CHUNK = 32_000 are a prefill-only vLLM workload on the same hardware, and 32,000
  is almost exactly one full Chunk — the floor Concurrency, batching, and what the panel measures #30 derived. --embed-model is required where
  --ocr-model is not, because vectors are meaningless across models where text is merely different.
  Also records the flags deliberately not added, each answered by a closed ticket.

• How comparable pipelines are actually implemented
  — fourteen comparable pipelines read at pinned commits, from source; findings on branch
  research/comparable-pipeline-implementations (3b95a7a, local only). No decision contradicted;
  one challenged. LangChain independently implements Pooling: one document vector from N chunk vectors #25's exact algorithm, short-circuit included;
  olmOCR independently digests path strings not content, as The document identity rule: one name, three jobs #22 does. Against that: vLLM's own
  long-text embedding never re-normalizes its cross-chunk mean, Vespa truncates MRL-style without
  re-normalizing, unstructured-ingest and ColBERT both ship the partial-write bug The .npz file: arrays, dtypes, and provenance without a header #26's rename
  ordering prevents, ColBERT's resume carries a literal # TODO: Verify config matches, and three
  targets declare a provenance slot they never fill. Zero of fourteen produce a document vector —
  but all fourteen are retrieval pipelines, and paperscale also serves a classification
  Consumer that needs one fixed-width row per Document with no query to reduce against. The
  unanimity was an artifact of a homogeneous sample; standing decision 3 stands, amended to name
  both Consumers. Refines
  Concurrency, batching, and what the panel measures #30: vLLM fans an input array into N independent engine requests, so batching is HTTP-only and
  concurrency is what fills the engine. Challenge resolved: Vespa's feed client adapts concurrency by
  measuring throughput/inflight, which Concurrency, batching, and what the panel measures #30's "no published ceiling" reasoning did not answer.
  Settled as fixed default plus a panel advisory, not a control loop — see Concurrency, batching, and what the panel measures #30's amendment.

• The Adapter's concrete contents, and the Chunk budget as numbers
  — validated_context_length is a rule: it defaults to min(card, server), and
  --context-length overrides it bounded by the server, not the card — rejected above
  server_max_model_len, allowed with an explicit warning above the card. The card is authoritative
  about the model, the server about the deployment, neither about both; and only the server's number
  is a correctness boundary, the card's being a quality claim. Both cards carry a third number — the length they actually exercise (Qwen3
  8192, Nemotron 4096 for its published eval numbers) — recorded, not used: at 4096 the smoke corpus
  goes from 52 Chunks to 97, which lands hardest on the classification Consumer, and quality
  evaluation is out of scope. SAFETY_MARGIN falls from an implied 868 to 64 and is
  re-justified — Chunking strategy when a document exceeds the discovered limit #24's subadditivity proof already covers packing and the Instruction, so the
  margin covers only a /v1/tokenize vs /v1/embeddings special-token disagreement. embed
  becomes its own package, src/paperscale/embed/, holding the whole subcommand and mirroring
  evaluation/; one Adapter per model size (native_dim differs per size); no default model.
  Instruction becomes two plain strings — document_instruction ("" for Qwen3, empty-string
  never null) and query_instruction (a template for Qwen3, copied and never applied) — plus the
  published 1–5% retrieval cost of the Consumer omitting it. Collides with The CLI surface: flags, defaults, and how embed installs #35 twice:
  --context-length against its "must not be operator-settable", and its --request-tokens 32000
  now sits 704 tokens below one Chunk rather than on the floor.

• The panel's server group: who produces its rows
  — Concurrency, batching, and what the panel measures #30's "no new plumbing" was wrong in a bigger way than the ticket framed: the mismatch is in
  push_vllm_stats's inputs, not its row names. Of Concurrency, batching, and what the panel measures #30's four server rows only tok/s is
  reachable from (stats, poller). So embed gets its own push function in
  src/paperscale/embed/, inheriting format_rate, Rates, and the fixed-row-set discipline —
  set_stat can add and overwrite but never remove, so every branch must write every row.
  The vllm group is renamed server rather than joined by one: 6 lines, and one concept
  keeps one name. I recommended joining, to leave OCR untouched; the operator overruled on
  consistency and was right — a panel title is not a row, so criterion 4 was never in tension.
  in-flight is <client outstanding>/<vllm:num_requests_waiting>, counting /v1/embeddings
  only — Chunking strategy when a document exceeds the discovered limit #24 proved /v1/tokenize never enters the engine scheduler, so it cannot cause the
  queue it would be compared against. model leaves the panel for the header: at 80 columns
  in-flight (9) leaves values 11 cells against running's 13, and both pinned ids are 23-31
  characters. That also drops server to three rows, which — verified by running _layout_budget
  — removes an 18-row height cliff that dropped in-flight, the saturation signal, first.

• Clear the stale text from the closed record
  — six corrections posted on the issues carrying the stale text, no decision changed. Concurrency, batching, and what the panel measures #30's batching
  rationale, How comparable pipelines are actually implemented #36's retracted "Chunk vectors are primary", The stats + logs panel #29's doubled empty, a scope note on What SGLang exposes for embeddings, and whether it truncates silently #33,
  and cli.py:66 on The LanceDB schema and its namespacing #27/How resume knows a document is done #28/The CLI surface: flags, defaults, and how embed installs #35. Item 3 was not stale text. The document identity rule: one name, three jobs #22 calls a derived-name collision a
  tiebreak and, seven lines later, fatal; nothing ever resolved it and The .npz file: arrays, dtypes, and provenance without a header #26 simply picked fatal.
  Recorded on The document identity rule: one name, three jobs #22 and flagged on the The document identity rule: one name, three jobs #22 line above — it needs its own ticket, because settling it
  is a decision.

• The four handoffs #30 never picked up
  — LanceDB writes: add() for Documents Resume knows are new, merge_insert only to replace,
  which removes the O(N²/B) term The LanceDB schema and its namespacing #27's unconditional read-modify-write imposed and leaves batch size
  bounded by crash-loss alone. Batch 64, anchored to --concurrency 64, not a flag; accepted cost
  is ~1,562 fragments per 100k Documents, mitigated by compaction (unverified). /v1/tokenize:
  its own concurrency bound (it never reaches the GPU), the shared retry taxonomy, a failed Document
  rather than a failed Invocation, and no panel row. encoding_format: base64 — measured 3.9x
  smaller than float-JSON at both widths, compounding with Is MRL truncation offered, and how does an adapter express validity? #34's native-width responses (vLLM support
  unverified; float is the fallback and the two are mathematically identical). Retries mirror
  evaluation/pplx.py, not the OCR path — three axes, bounded delay, raises instead of
  sys.exit(1); --max-request-retries 8 bounds the bad-response axis; full jitter added on the
  connection axis, which no precedent has, because one server restart fails all 64 in flight from a
  single cause. --request-tokens floor is enforced by raising: the effective budget is
  max(flag, chunk_budget) with a log line, because The Adapter's concrete contents, and the Chunk budget as numbers #37's arithmetic puts the 32,000 default below
  the floor for both pinned families — rejecting would reject the default configuration.

• What happens when two Documents derive the same name
  — the collision is prevented, not handled. The Document name appends the source extension
  (case.pdf.npz), so the only class that occurs in practice cannot arise; combined with standing
  decision 6's normalization, what reaches the fatal branch is a leading-slash or tarball-collapse
  collision only. Those are fatal at startup, before any GPU work, listing both raw Source-File
  values. The digest is not a tiebreak — How resume knows a document is done #28 derives Resume from the outputs, so a tiebroken name
  must be stable across Invocations, and neither scheme is: suffixing the loser depends on iteration
  order, suffixing a whole colliding set silently reverts when the set changes. The document identity rule: one name, three jobs #22's contradiction
  turned out not to be two decisions in tension: DuplicateSourceFileError fires on the raw
  Source-File ("the join key is ambiguous"), so it never described a derived-name collision at all.
  Measured before deciding: the live corpus is 39,905 files, all .pdf, zero collisions.
  #32 is deliberately left alone — it
  gains a guard and keeps its names — and the resulting divergence is filed as
  #42.

Not yet specified

Empty. The last three items — the CLI surface, run-label path-safety, and how embed installs —
graduated into #35 once #30 closed, and
that ticket is now resolved. Nothing is left to decide. What remains is
#36, which checks the finished design
against how comparable pipelines are actually implemented, and
#31, which writes it down.

Out of scope

• Query-side embedding — ruled out; the document-side convention is recorded in the
  output instead so the consumer can construct matching queries (standing decision 9).
• Retrieval, reranking, RAG — paperscale's job ends when vectors exist and are
  identified.
• Embedding-quality evaluation — measured downstream, by the project that consumes the
  vectors. It is a second destination and cannot be designed against a format not yet locked.
• Qdrant, or any second vector store — LanceDB only.
• SGLang as a serving engine — ruled out by the narrowing of standing decision 10. It is
  technically the better-instrumented engine for this workload, but it does not serve
  Nemotron-3-Embed, and keeping both model families was worth more than keeping both engines. The
  #33 findings stay on record if that trade is ever revisited.
• TEI, Ollama and NIM as serving engines — ruled out by standing decision 10. TEI and Ollama
  truncate silently and cannot be made safe over /v1/embeddings; NIM will not report max input
  tokens over HTTP. The #23 inventory covering them stays on record in case the ruling is ever
  revisited.
• safetensors as an output format — replaced by .npz.
• Fixing the markdown export's extension collision
  (#32) — surfaced while resolving
  The document identity rule: one name, three jobs #22. It is a pre-existing bug in the OCR-side export on main, not part of designing embed.
  Filed separately.