Skip to content

fix(llm-gateway): say a gated model is unavailable, not an auth failure - #2

Draft
evanmarshall wants to merge 1 commit into
masterfrom
ito/upstream-pr-93611-e1cb5336
Draft

fix(llm-gateway): say a gated model is unavailable, not an auth failure#2
evanmarshall wants to merge 1 commit into
masterfrom
ito/upstream-pr-93611-e1cb5336

Conversation

@evanmarshall

Copy link
Copy Markdown

This is an Ito QA mirror of PostHog#93611.

Original author: @PostHog[bot]
Pinned upstream head: e1cb533

This draft is for QA and build-environment experimentation and will not be merged.


Problem

  • Picking a preview model (DeepSeek V4 Flash, GLM 5.3 Flash, Kimi K3) ends the whole task with a red "Failed to authenticate" banner, after a sandbox has already launched. There is no stated next step.
  • The gateway denies the model in its auth dependency with a 403, but that denial reads as three different wrong things downstream:
Signal the gateway sends What Desktop concludes What is actually true
code: "model_gate" ignored — the classifier matches only the free-tier wording a permission denial
message ends in (rate_limit) retryable permanent until the flag is on
model listed with allowed: true safe to offer in the picker every request with it 403s
  • The (rate_limit) suffix is a documented compat shim for the free-tier gate. The rollout gate copied it, and a rollout flag never clears on a retry.

Changes

  • A blocked preview model now says so: "This model isn't available — it's in preview and isn't turned on for your account yet. Pick another model to keep going." No payment prompt, because no payment unlocks it.
  • The gated models stop appearing in the picker at all. The gateway's model listing drops any model whose access flag the caller does not hold, and fails closed on a flag-evaluation outage, matching what enforcement does.
  • The gateway's rollout denial carries a machine-readable reason, and no longer wears the free-tier shim:
-"message": "Model '<id>' is not available. Choose another model. (rate_limit)",
+"message": "Model '<id>' is not available for your account. Choose another model.",
 "code": "model_gate",
+"reason": "model_not_available",
  • code: "model_gate" stays on that body, so builds that read only the code keep reaching a model prompt rather than session teardown.
  • Mechanical: the classifier matches the structured fields by regex because the ACP layer delivers the error body as text; a new model_unavailable cause threads through the limit copy, the failure title, and the analytics cause union.
Why drop the model from the listing instead of marking it allowed: false

allowed: false already exists for the free-tier gate, and a marked model renders as a locked upsell row. That is right when a payment method unlocks it. A rollout flag has no user action behind it, so the row would be a dead end.

How did you test this code?

Automated only — the agent did not run the desktop app or reproduce the banner.

  • services/llm-gateway: full suite passes. New cases pin that the rollout denial carries reason and no (rate_limit) suffix, and that a flag-gated model is listed only when its flag returns true (off and unavailable both drop it).
  • @posthog/shared, @posthog/core, @posthog/ui: suites pass. New cases pin classification from the structured code/reason, and that the wrapped "Internal error: … 403" body is not treated as a fatal session error.
  • @posthog/agent fails 312 tests on this branch and the same 312 on master — pre-existing, unrelated.
  • Typecheck passes for shared, core, ui, agent. ruff check and ruff format clean.

Automatic notifications

  • Publish to changelog?

Docs update

None — error-body shape and in-app copy.

🤖 Agent context

Autonomy: Fully autonomous

  • Written by PostHog Desktop from an inbox report. No repo skills were loaded through the skill tool; the copy, comment, PR-description, and simplification conventions were applied from AGENTS.md directly.
  • The report proposed reusing the existing model_gate cause on the client. That was rejected once the copy behind it turned out to be a billing upsell: routing a rollout gate there would tell users to add a payment method for a model no payment unlocks. Hence the separate model_unavailable cause, with code: "model_gate" retained on the wire for older builds.
  • Public artifact: nothing here carries session material. The model ids, flag keys, and error bodies all come from this repository.

Created with PostHog Desktop from this inbox report.

A model behind a rollout flag was denied with the free-tier gate's wording and
its "(rate_limit)" compat suffix, so Desktop could not tell the denial apart
from a generic failure and offered a payment method for something no payment
unlocks.

- The flag denial now carries reason="model_not_available" and drops the
  "(rate_limit)" suffix, which only ever belonged to the free-tier shim.
- The Desktop classifier reads the structured code and reason instead of
  matching free-tier wording, and routes a rollout gate to its own prompt.
- The model listing drops models whose access flag the caller does not hold,
  so a picker no longer offers a model every request will reject.

Generated-By: PostHog Desktop
Task-Id: 68fea8bf-f61a-4194-a8ef-b022767ecb29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant