feat(mcp): host an MCP server for Unstract API deployments

[hari-kuriakose]

What

Hosts an MCP server for the Unstract platform, so coding agents can run document
extraction as a tool call instead of hand-rolling HTTP requests against an API
deployment.

Modelled on how the mfbt backend hosts its MCP server (Zipstack/mfbt,
backend/app/routers/mcp_http.py): a hand-rolled JSON-RPC 2.0 endpoint mounted
in the existing app, with a tool registry behind it. Same pattern, adapted from
FastAPI to Django/DRF.

How it mirrors mfbt

mfbt	This PR
JSON-RPC endpoint in the existing FastAPI app	JSON-RPC endpoint in the existing Django app
Scoped to a project via URL	Scoped to an API deployment via URL
Bearer API key + key-in-URL fallback	Same
MCP_TOOLS name→callable dict	MCPToolRegistry, declarative + JSON schema
Hosted in-process, not a separate service	Same

mfbt also ships a stdio server (mcp_server.py, FastMCP). That is not the
hosted path, so it is not what this replicates.

Endpoint

An MCP session is scoped to one API deployment and mirrors that deployment's
REST URL:

POST /deployment/api/<org_name>/<api_name>/     # REST (existing)
POST /mcp/<org_name>/<api_name>/                # MCP  (new)

Auth is the deployment's existing API key — same key, same management UI.
There is no second credential to mint or revoke.

claude mcp add --transport http unstract \
  https://<host>/mcp/<org_name>/<api_name>/ \
  --header "Authorization: Bearer <api_key>"

A /<api_key> path variant exists for MCP clients that cannot set headers
(mfbt carries the same workaround).

Tools

Tool	Purpose
readMeFirst	Orientation guide, built from the live deployment
getApiInfo	Deployment name, description, workflow, active state
extractDocument	Run extraction over S3 pre-signed URLs. Consumes quota
getExecutionStatus	Poll for a pending extraction's result

Design decisions

• Auth delegates to DeploymentHelper — the same validation the REST
  endpoint uses, so the two surfaces cannot drift on who is allowed in.
  Org scoping falls out of this: a valid key for org A cannot reach org B's
  deployment (covered by the wrong org case).
• Execution delegates to ExecutionRequestSerializer — URL validation
  (S3-only, HTTPS-only) and the file-count cap live there. Reimplementing them
  would let the MCP surface silently diverge from REST on what input is safe.
• All auth failures answer identically (401, no detail) so the endpoint
  cannot be used to enumerate deployment names.
• Tool errors are JSON-RPC results with isError: true, not protocol
  errors — clients treat protocol errors as unrecoverable transport faults,
  whereas an agent-fixable problem should be readable and retryable.

Tests

34 passing. Protocol/dispatch and tool logic in the unit tier; the auth
boundary in the integration tier. Registers a new mcp-server-auth critical
path in tests/critical_paths.yaml.

Writing the happy-path tests caught three real bugs, now fixed:

1. Tool descriptions said documents could be any reachable URL — the serializer
   accepts only S3 pre-signed URLs, so an agent following the description
   would have failed every call.
2. The tags schema advertised an unbounded array; the serializer caps it at 1.
3. Documents were downloaded before the rate-limit slot was taken, so a
   call about to be rejected still pulled every document over the network.

Verified locally against Postgres: full backend unit tier 306 passed, no
regressions. The 35 integration failures in
workflow_manager/execution/ are pre-existing in my sandbox (no Redis) — the
identical 35 fail on the base commit.

Not implemented / notes for review

• OAuth 2.1 + dynamic client registration is deliberately out of scope.
  mfbt implements it for the claude.ai one-click browser connector; bearer auth
  covers Claude Code and API clients. Adding it later is additive — discovery
  endpoints alongside this router, no transport change.
• No live-deployment smoke test. The happy path is covered by mocks at the
  DeploymentHelper boundary, and a full MCP session (initialize → tools/list →
  tools/call) was exercised end-to-end against a real DB during development, but
  nothing has run a real extraction through this path. Worth one manual check
  against a live deployment before merge.
• Cloud mounting unverified. Mounted in backend/base_urls.py, the only
  ROOT_URLCONF in this repo. cloud_base_urls.py exists in the cloud overlay
  but not here — someone with cloud context should confirm the endpoint mounts
  there too.
• Protocol version is pinned to 2024-11-05 and does not negotiate against
  the client's requested version (mfbt does the same). Fine for current clients.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6895d9f3-aa66-4aca-8750-3a51f4457c96

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch worktree-mcp-server

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}