Skip to content

feat(llm): make output limits and provider beta features tier-configurable #923

Description

@devlikebear

Part of #919. Phase 0 of the provider modernization roadmap. LP-004.

Prerequisite: #922 (LP-003) — the thinking budget must fit under whatever max_tokens becomes configurable here.

Summary

Add max_tokens and provider beta feature flags to the tier binding, and plumb them through the resolver and router so frontier-model capabilities are reachable from config.

Problem

Output is pinned at 4096 with no way to raise it. The Anthropic client defaults to 4096 when unset:

// internal/llm/provider.go:373
if config.MaxTokens <= 0 {
    config.MaxTokens = 4096
}

and nothing ever sets it. LLMTierBinding (internal/config/types.go:141) and ResolvedLLMTier (internal/config/llm_resolve.go:22) have no max_tokens field, and helpers_llm_router.go:34 builds ProviderOptions without one — so MaxTokens is structurally always 0. Every tier is capped at 4096 output tokens regardless of model.

This also collides with LP-003: a thinking budget must be strictly below max_tokens, so any meaningful budget is unreachable while the cap is 4096.

Beta features are unreachable. anthropic-beta is one hardcoded constant:

// internal/llm/anthropic.go:16
const anthropicPromptCachingBeta = "prompt-caching-2024-07-31"

sent unconditionally at anthropic.go:67. Prompt caching went GA long ago, so this occupies the only slot with a header that no longer does anything — while 1M context, interleaved thinking, and extended cache TTL all need that slot.

It is also sent to third-party Anthropic-compatible gateways (config/default.yaml:21 points minimax at one), which may reject or mishandle an unknown beta.

Scope

  • Add max_tokens to LLMTierBinding and ResolvedLLMTier; pass it through helpers_llm_router.go.
  • Replace the hardcoded beta constant with a list assembled per request.
  • Add a tier-level opt-in for provider beta features, named provider-agnostically so other providers can adopt the mechanism.
  • Drop the GA'd prompt-caching beta from the default set.
  • Choose per-model output defaults instead of a blanket 4096 when a tier omits max_tokens; keep 4096 as the last-resort fallback for unknown models.
  • Validate thinking_budget < max_tokens at resolve time with a loud error, not at request time.
  • Document the new fields in config/tars.config.example.yaml.
  • Surface the fields in the console tier editor and the onboarding wizard review step.

Acceptance Criteria

  • A tier can produce a response longer than 4096 tokens.
  • A tier with no max_tokens still works, using a sane per-model default.
  • thinking_budget >= max_tokens fails at config load with a message naming both values.
  • Beta flags are opt-in per tier and absent from the request when unset.
  • A kind: anthropic gateway tier can be configured to send no beta header at all.

Validation

  • Resolver unit tests for present/absent/invalid max_tokens
  • Unit test for the thinking_budget vs max_tokens guard
  • Request-builder tests asserting the assembled beta header for: default, opt-in, and explicitly empty
  • Round-trip test through YAML → env override → resolved tier
  • make test
  • make lint-diff

Out of Scope

  • Context-window budgeting — that is LP-006, which lands on the same tier binding and should be sequenced after this.
  • Per-role output limits. Tier granularity is enough for now.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions