Define the MCP tool, resource, and experience contract

[aidanbruneel]

Question to answer

What smallest stable public tool/resource contract lets agents obtain useful IMAGE text/audio interpretations and continue with image-specific questions across supported hosts, without exposing internal pipeline topology or committing the API to provider-specific attachment objects? Is there any possibility for Monarch experiences to be included as well?

Decision or next step this enables

Before implementing the gateway, document and approve the first version of IMAGE’s public MCP contract: accepted inputs, returned results, errors and partial-result warnings, experience-handle lifecycle, follow-up behaviour, resource retrieval, client-facing annotations, and data-retention rules.

Current understanding and key unknowns

Current hypothesis:

• Start with interpret_image and ask_about_image, plus experience/App resources.
• Keep internal preprocessors, handlers, routes, renderer IDs, and raw rendering payloads private.
• Return explicit owned handles because MCP transport state is not application state.

Unknown:

• Required first-release user journeys and locales.
• Whether audio is required for the first production increment or may follow text.
• Which image-source variants are truly portable across target hosts.
• Whether follow-up should retain server-side conversation turns or accept sufficient explicit context on every call.

Proposed approach

1. Write concrete journeys for direct upload, connector-sourced image, chart/screenshot, text-only result, segmented audio, failure, expiry, cancellation, and follow-up.
2. Run tool-description/tool-choice evaluations with representative agent prompts, including privacy-sensitive and irrelevant prompts.
3. Draft JSON Schema 2020-12 contracts with examples and evolution rules.
4. Map every public field to IMAGE behavior or an adapter responsibility.
5. Review with accessibility users/experts, pipeline maintainers, security, and operators.
6. Record non-goals and a compatibility policy for additive versus breaking changes.

Completion criteria

• Approved user journeys and beneficiaries are documented.
• Approved schemas exist for both tools, experience manifest, warnings/errors, and artifact descriptors.
• Ownership, expiry, idempotency, retry, cancellation, and partial-result semantics are explicit.
• Host-specific attachment metadata is isolated behind adapters and absent from the canonical domain contract.
• Tool-choice evaluations show the tool is selected for intended prompts and not for clearly unrelated or unsafe automatic transfers.

Evidence and related work

• Prototype tool: https://github.com/Shared-Reality-Lab/IMAGE-server/blob/mcp-endpoint/orchestrator/src/mcp.ts
• IMAGE request pipeline: https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/orchestrator

Checks

• I kept this Investigation focused on one bounded question with a clear stopping condition.
• I will track resulting production implementation as separate Bugs, Features, or Tasks.
• I removed secrets, credentials, personal information, and confidential data.