Feat/mxbai embeddings and hardening

[ulielitay]

Summary

Makes the embedding model selectable from a registry, where picking a model auto-derives everything downstream (vector dimension, container memory limits, query/passage prompts, the pre-baked image layer), and upgrades the default to a stronger model. Also adds the lint/type gate the repo was missing, hardens sitemap XML parsing, and adds parity guards against the duplicated-code drift the codebase already warned about in comments.

Embedding model

config/models.yaml is the single source of truth. make configure MODEL=<name> resolves the selection into .env and re-renders db/init/01_schema.sql from a template so vector(N) always matches the model.

Model	dim	ingestion / mcp memory
mixedbread-ai/mxbai-embed-large-v1 (new default)	1024	2g / 2g
intfloat/multilingual-e5-large	1024	2g / 2g
BAAI/bge-base-en-v1.5	768	1500m / 1g
BAAI/bge-small-en-v1.5 (previous default, used by CI)	384	1500m / 1g

Still FastEmbed / ONNX / CPU — no torch, no GPU.

Bug found while implementing

The previous code (and ADR-001) assumed FastEmbed's passage_embed()/query_embed() apply the BGE passage:/query: prefixes. They don't — in FastEmbed 0.8 both delegate straight to embed() for every non-multitask model (bge, mxbai, e5), so queries were being embedded with no instruction prefix at all. Prompts are now applied explicitly from the registry (mxbai/bge instruct the query only; e5 prefixes both sides). The chunker's tokenizer also follows the embedding model now, so chunk sizing matches the model that actually embeds the text.

Other changes

• Lint/type gate: ruff + mypy, make lint / make typecheck, new CI job. mypy quarantines a documented pre-existing typing backlog so the gate enforces types on new/changed code without a risky refactor. Coverage added to make test.
• CI cost: CI runs make configure MODEL=BAAI/bge-small-en-v1.5 so the suite doesn't pull 1.2GB on every run — which also continuously exercises the configure path.
• Security: sitemap XML now parsed with defusedxml (forbids DTD/entity expansion/XXE); these documents come from untrusted upstream sources.

Other changes

• Lint/type gate: ruff + mypy, make lint / make typecheck, new CI job. mypy quarantines a documented pre-existing typing backlog so the gate enforces types on new/changed code without a risky refactor. Coverage added to make test.
• CI cost: CI runs make configure MODEL=BAAI/bge-small-en-v1.5 so the suite doesn't pull 1.2GB on every run — which also continuously exercises the configure path.
• Security: sitemap XML now parsed with defusedxml (forbids DTD/entity expansion/XXE); these documents come from untrusted upstream sources.
• Parity guards: tests assert the committed schema equals the rendered template, both services' defaults equal the registry default, and the hand-duplicated SSRF _addr_is_private helper stays byte-identical across services.
• Housekeeping: LICENSE added; small B904/B905 fixes surfaced by the new lint gate.
• Docs: README/runbook corrected (they still described the model as fixed), new ADR-004, ADR-001 given an inline supersede note in its existing style, and the false prefix claim in IMPLEMENTATION_PLAN.md corrected.

⚠️ Upgrading an existing deployment

The default dimension changes 384 → 1024, so stored vectors are invalid and the column width changes. Existing deployments need a rebuild + re-embed:

make configure                          # or MODEL=<your choice>
docker compose build ingestion mcp-server   # model is baked in at build time
docker compose down -v db && docker compose up -d db
make up && make sync

Same-dimension model swaps can use make reindex instead. Full procedure: Runbook → switch the embedding model.

Verification

• make lint (ruff) clean; make typecheck (mypy) clean across both packages + scripts.
• ingestion suite 250 passed, 44 skipped; mcp-server 39 passed, 4 skipped (skips are DB-gated).
• Parity suite: 9 passed.
• make configure round-trip verified: selecting bge-small yields dim 384 + 1500m/1g in .env and vector(384) in the schema; the default yields 1024 + 2g/2g and vector(1024).
• make configure round-trip verified: selecting bge-small yields dim 384 + 1500m/1g in .env and vector(384) in the schema; the default yields 1024 + 2g/2g and vector(1024).
• docker compose --profile full config confirms both services resolve to the same model with the derived memory limits and prompts.
• XXE smoke: an entity-expansion sitemap is rejected (EntitiesForbidden) while a normal sitemap still parses.

One pre-existing test (test_startup_fails_fast_on_unreachable_db) fails locally under WSL2 — it expects a fast ECONNREFUSED from 127.0.0.1:1, which this sandbox hangs instead. Unrelated to these changes; it should pass in CI.

Not done deliberately: MCP host_origin_protection is left disabled — enabling it risks breaking Traefik-proxied access, which is out of scope here.