fix(semantic-search): align AI Search catalog retrieval

[aryasaatvik]

Summary

Keep Cloudflare AI Search authoritative for semantic tool discovery while making the indexed catalog resilient to integration changes, stale provider state, and individual upload failures.

Flow

tools.search(query, namespace)
  -> AI Search native hybrid retrieval + reranking
  -> stable path and namespace validation
  -> best chunk per tool
  -> paged tool results

reindex batch
  -> manifest + integration context
  -> enriched AI Search document
  -> per-item upload result
  -> local recovery/status ledger

Changes

• Add integration name, purpose, URL, and a document-versioned fingerprint to indexed tool documents.
• Lower the native AI Search retrieval/reranking threshold for short catalog queries without adding lexical fallback or client-side reranking.
• Treat the local upload ledger as recovery/status state rather than query visibility; AI Search metadata remains the query authority.
• Preserve integration-prefix filters and canonical path deduplication.
• Keep replaced remote items until their replacement completes, and isolate per-item reindex failures so one tool cannot abort a page.

Validation

• bun run --cwd packages/plugins/semantic-search test -- src/sdk/ai-search.test.ts — 18 passed.
• bunx vitest run --exclude src/sdk/store-fts-d1.test.ts — 92 passed.
• bun run --cwd packages/plugins/semantic-search typecheck — passed.
• bunx oxfmt --check <six changed files> — passed.
• bunx oxlint -c .oxlintrc.jsonc <six changed files> --deny-warnings — passed.

The package-wide test run still has the known 10 better-sqlite3 D1 FTS failures caused by the local Node ABI mismatch (137 vs 147). The repository-wide typecheck also reaches the changed package successfully but stops on the existing SVG module-declaration errors in plugin-toolkits.

Follow-up

Deployment, reindex execution, and live MCP validation are intentionally separate from this reviewable source change.

[greptile-apps]

Greptile Summary

The PR keeps Cloudflare AI Search authoritative for discovery while strengthening indexed document metadata and replacement recovery.

• Adds versioned integration enrichment to tool-search documents and fingerprints.
• Isolates per-item lookup and upload failures while reporting failed counts.
• Persists all deferred replacement deletions and performs cleanup only after replacement rows are recorded.
• Lowers native retrieval and reranking thresholds and removes local-ledger result gating.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/plugins/semantic-search/src/sdk/ai-search.ts	Reworks AI Search retrieval authority, per-item failure isolation, and replacement cleanup ordering; the previously reported defects are no longer established at current HEAD.
packages/plugins/semantic-search/src/sdk/ai-search.test.ts	Adds coverage for optional enrichment failure, persistence-safe replacement, outdated-item replacement, and preservation of every failed deletion.
packages/plugins/semantic-search/src/sdk/documents.ts	Adds document-versioned fingerprints and optional integration metadata to indexed tool documents.
packages/plugins/semantic-search/src/sdk/collections.ts	Extends recovery rows with multiple pending deletion IDs while preserving legacy-row decoding.
packages/plugins/semantic-search/src/api/group.ts	Extends the reindex response schema with an optional per-item failure count.
packages/plugins/semantic-search/src/sdk/tool-search-backend.ts	Extends the refresh-result contract with the optional failure count.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Q[Tool search request] --> S[Cloudflare AI Search]
  S --> V[Validate namespace and path]
  V --> D[Deduplicate by canonical tool path]
  D --> P[Return paged results]
  R[Reindex request] --> M[Load manifests and optional integration context]
  M --> U[Upload changed documents independently]
  U --> L[Persist replacement rows]
  L --> C[Delete deferred provider items]
  C --> F[Persist failed deletions for retry]

Loading

_{Reviews (4): Last reviewed commit: "fix(semantic-search): retain all deferre..." | Re-trigger Greptile}