Skip to content

test(llm): add a cross-provider conformance suite #926

Description

@devlikebear

Part of #919. Phase 1 of the provider modernization roadmap. LP-007.

Prerequisites: #920, #921, #922, #923, #925 — the suite encodes the behavior those establish.

Summary

Add one table-driven suite that every provider client must pass, so capability gaps surface as failing tests instead of accumulating silently.

Problem

Provider tests are currently per-file and per-provider: anthropic_test.go, openai_compat_client_test.go, gemini_native_test.go, claude_code_cli_test.go, antigravity_cli_test.go. Each covers its own client on its own terms, and nothing asserts that the set is uniform.

That is how the Anthropic gaps in this epic went unnoticed: gemini-native learned to round-trip reasoning signatures (gemini_native_convert.go:161) while anthropic.go did not, and no test could observe the divergence. The same shape applies to cache reporting, tool choice, and structured output.

The goal is not to force every provider to support everything — CLI-backed providers legitimately cannot do some of it. The goal is that support is declared and verified, so an unsupported capability is a documented Unsupported rather than a silent no-op like reasoning_effort on Anthropic.

Scope

  • Define a capability matrix: which providers support tool calling, streaming, reasoning effort, thinking round-trip, prompt caching, cache usage reporting, JSON schema response format, service tier, session resume, multimodal input.
  • Add a shared table-driven suite that runs the same scenarios against every registered provider, using recorded fixtures or stub transports — no network in the default make test path.
  • Assert that a capability declared unsupported produces a diagnostic when requested, never a silent no-op.
  • Cover the round trips that matter for multi-turn correctness: tool call → tool result → follow-up, and reasoning block → follow-up.
  • Assert cache and token usage fields are populated from each provider's own response shape.
  • Extend the //go:build integration live tests, following TestAntigravityCLILive (internal/llm/antigravity_integration_test.go), to cover anthropic and openai; keep them opt-in and skipping when unconfigured.
  • Document how to add a provider and what the suite requires of it.
  • Keep the suite Windows-clean — do not add to the exclusion lists in scripts/windows_test.sh.

Acceptance Criteria

  • Every provider client runs the same scenario table.
  • Adding a provider without declaring its capabilities fails the build or the suite.
  • Requesting an undeclared capability produces an observable diagnostic in a test.
  • The default make test path makes no network calls.
  • Live tests skip cleanly with no credentials and pass with them.
  • The suite runs on Windows.

Validation

Out of Scope

  • Benchmarking or model quality evaluation.
  • Testing provider-side behavior; this tests the TARS conversion layer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestgoPull requests that update go codemlp-domain-platformMLP: Domain-agnostic agent platform

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions