fix: give each query lexeme its own BM25 candidate budget

[nuemaan]

Summary

• The FTS prefilter from perf: bound classic BM25 candidates with Postgres FTS #252 bounded candidates with one global ts_rank_cd ordering. That ordering scores term density inside a chunk and ignores corpus-wide rarity, while BM25 weights rare terms heavily, so the chunk BM25 ranks first can be the first one truncated.
• Each query lexeme now draws from its own share of the budget through a lateral join, so a lexeme matching few chunks always contributes them. _MIN_CANDIDATES_PER_LEXEME keeps many-lexeme queries from dividing the budget into slivers.
• Adds a warning when the candidate pool saturates. The debug line reported candidates == limit whether the corpus held exactly that many or far more, so silent truncation was indistinguishable from a healthy query.
• Classic route only. Map-nav default path untouched. No API, worker, deployment, or migration impact.
• Resolves Classic BM25 FTS prefilter can truncate the top BM25 match in large namespaces #278, follows up on Optimize BM25 retrieval candidate loading with Postgres FTS prefilter #195 and perf: bound classic BM25 candidates with Postgres FTS #252.

Measured on 5001 chunks where 5000 densely repeat a common term and one holds a rare term, querying data zebra:

ts_rank_cd position of the rare chunk:   5001 of 5001
rank_rows_by_bm25 position:                 1 of 5001

candidate pool, global ordering, limit 2000:  2000 rows, rare chunk absent
candidate pool, per-lexeme budget:            1001 rows, rare chunk present

The per-lexeme pool is smaller and still keeps the match BM25 wants. Cost is one GIN probe per lexeme rather than one overall, and _MAX_FTS_QUERY_TOKENS already caps that at 50.

Verification

• uv run pytest apps/api/tests/unit/test_bm25_channel_tsquery.py apps/api/tests/contract/test_bm25_fts_prefilter_contract.py — 16 passed against real Postgres 16
• uv run pytest packages/shared-python/shared/tests/test_retrieval_search_channels.py — 4 passed
• uvx ruff check packages/shared-python apps/api/tests — clean
• Reverted the channel change and re-ran the new regression test to confirm it fails without the fix: AssertionError: assert 'common-2' == 'rare-zebra'

test_content_channel_uses_bounded_or_fts_after_scope_filters asserted on the literal LIMIT :fts_candidate_limit as a position marker for the candidate bound. That clause is now LIMIT lb.per_lexeme_limit, so the assertions were repointed. The intent from #244, that scope filters land ahead of any candidate bound so exclusions cannot consume the budget, is unchanged and still asserted.

Not tested: production-scale corpora, and lateral-probe cost at the 50 lexeme ceiling on a large namespace. Both need a dataset I do not have locally.

Deployment Notes

• No new environment variables. RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT keeps its meaning as the total budget, now spent per lexeme.
• No database migrations.
• Backwards compatible. The full-scan fallback is untouched, so rolling back is a code-only revert.

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change

[nuemaan]

Heads up that CI here is sitting at action_required rather than a pass or fail, so the required checks never ran. Fork PRs need a maintainer to approve the workflow run, and a fresh push does not clear it. That is what mergeable_state: blocked reflects, not a failing build.

Rebased onto current main just now. No conflicts, channels.py was untouched upstream. Re-ran locally against Postgres 16 after the rebase: 16 passed on the api unit and contract tests, 4 passed on the shared channel tests, ruff clean.

Worth separating the two things regardless of what happens to this PR: the truncation described in #278 is live on main today. If you would rather fix it differently, or fold it into other retrieval work, that is fine by me. The repro in the issue stands on its own.