You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of #919. Phase 1 of the provider modernization roadmap. LP-007.
Prerequisites:#920, #921, #922, #923, #925 — the suite encodes the behavior those establish.
Summary
Add one table-driven suite that every provider client must pass, so capability gaps surface as failing tests instead of accumulating silently.
Problem
Provider tests are currently per-file and per-provider: anthropic_test.go, openai_compat_client_test.go, gemini_native_test.go, claude_code_cli_test.go, antigravity_cli_test.go. Each covers its own client on its own terms, and nothing asserts that the set is uniform.
That is how the Anthropic gaps in this epic went unnoticed: gemini-native learned to round-trip reasoning signatures (gemini_native_convert.go:161) while anthropic.go did not, and no test could observe the divergence. The same shape applies to cache reporting, tool choice, and structured output.
The goal is not to force every provider to support everything — CLI-backed providers legitimately cannot do some of it. The goal is that support is declared and verified, so an unsupported capability is a documented Unsupported rather than a silent no-op like reasoning_effort on Anthropic.
Scope
Define a capability matrix: which providers support tool calling, streaming, reasoning effort, thinking round-trip, prompt caching, cache usage reporting, JSON schema response format, service tier, session resume, multimodal input.
Add a shared table-driven suite that runs the same scenarios against every registered provider, using recorded fixtures or stub transports — no network in the default make test path.
Assert that a capability declared unsupported produces a diagnostic when requested, never a silent no-op.
Cover the round trips that matter for multi-turn correctness: tool call → tool result → follow-up, and reasoning block → follow-up.
Assert cache and token usage fields are populated from each provider's own response shape.
Extend the //go:build integration live tests, following TestAntigravityCLILive (internal/llm/antigravity_integration_test.go), to cover anthropic and openai; keep them opt-in and skipping when unconfigured.
Document how to add a provider and what the suite requires of it.
Keep the suite Windows-clean — do not add to the exclusion lists in scripts/windows_test.sh.
Acceptance Criteria
Every provider client runs the same scenario table.
Adding a provider without declaring its capabilities fails the build or the suite.
Requesting an undeclared capability produces an observable diagnostic in a test.
The default make test path makes no network calls.
Live tests skip cleanly with no credentials and pass with them.
Part of #919. Phase 1 of the provider modernization roadmap. LP-007.
Prerequisites: #920, #921, #922, #923, #925 — the suite encodes the behavior those establish.
Summary
Add one table-driven suite that every provider client must pass, so capability gaps surface as failing tests instead of accumulating silently.
Problem
Provider tests are currently per-file and per-provider:
anthropic_test.go,openai_compat_client_test.go,gemini_native_test.go,claude_code_cli_test.go,antigravity_cli_test.go. Each covers its own client on its own terms, and nothing asserts that the set is uniform.That is how the Anthropic gaps in this epic went unnoticed:
gemini-nativelearned to round-trip reasoning signatures (gemini_native_convert.go:161) whileanthropic.godid not, and no test could observe the divergence. The same shape applies to cache reporting, tool choice, and structured output.The goal is not to force every provider to support everything — CLI-backed providers legitimately cannot do some of it. The goal is that support is declared and verified, so an unsupported capability is a documented
Unsupportedrather than a silent no-op likereasoning_efforton Anthropic.Scope
make testpath.//go:build integrationlive tests, followingTestAntigravityCLILive(internal/llm/antigravity_integration_test.go), to cover anthropic and openai; keep them opt-in and skipping when unconfigured.scripts/windows_test.sh.Acceptance Criteria
make testpath makes no network calls.Validation
make testmake test-racemake windows-testmake lint-diffOut of Scope