fix(runtime): classify provider capacity errors

[CxHsin]

Summary

Fixes #3341

Provider resource-exhausted failures now retain a stable provider_capacity classification, use bounded retry metadata, and receive capacity-specific Desktop error and recovery guidance instead of Unknown error and direct-retry advice.

Verification

• npm install succeeded and installed @ai-sdk/code-mode@1.0.23.
• npm run build passed for all workspaces.
• Runtime provider classification tests passed: 11/11.
• Desktop provider capacity presentation tests passed: 2/2.
• Full npm test was attempted; it remains blocked by unrelated Windows environment failures including symlink permissions (EPERM), SQLite locks (EBUSY), missing Rive CLI, and workspace timeouts.

AI use

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: Codex implemented the runtime classification, retry mapping, Desktop copy/recovery behavior, and regression tests. A Generated-by: Codex trailer is present in the commit.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — provider capacity failures now have stable classification and wait/switch-model guidance.
• No

[Astro-Han]

Thanks for this — classifying on the provider's structured code/type rather than on message text is the right foundation, and matching the existing CONTEXT_OVERFLOW_PROVIDER_CODES shape means this slots in without inventing a new mechanism. The copy and recovery-hint plumbing through to the desktop is complete and the tests assert the classification and the retry metadata rather than restating the implementation.

No [P0]/[P1]. One [P2] that is really a question about the problem definition, and two [P3]s.

The [P2]: resource-exhausted / resource_exhausted is not a single meaning across providers. In gRPC and the Google API error model, RESOURCE_EXHAUSTED (status 8) is the standard code for quota exhaustion — per-minute, per-day, or per-project — not for "the server is busy right now". The user-facing copy this PR routes it to is 模型服务暂时满载，请等待几分钟或切换模型后重试 / Wait a few minutes or switch models, and that advice is actively wrong for a daily quota: waiting a few minutes will not help, and the correct action is different.

The 429 branch above catches the common case, since a Gemini quota error usually carries HTTP 429 and returns RateLimit before reaching this check. So this is not a blanket misclassification. But that also means the capacity branch is reached precisely when the status code is absent or non-429 — which is the case where the code alone is carrying the whole meaning, and where it is most ambiguous.

Could you say which providers you observed emitting these two codes, and with which meaning? If the set is narrow and "server at capacity" is what they actually mean, then pinning that in the comment next to PROVIDER_CAPACITY_CODES resolves it and I have no further concern. If it turns out the same code also arrives for quota exhaustion, the classification is fine but the recovery copy needs to not promise that waiting works.

I want to be explicit that I am asking rather than asserting: I have not seen your traces, and you may well have picked these two spellings from concrete provider payloads that mean exactly what you say.

Review assisted by AI (Claude Opus 5). Findings were verified against the files at this head; the reviewer is accountable for them.

[Astro-Han]

[P2] See the review body. Short version: in the gRPC / Google API error model RESOURCE_EXHAUSTED is the standard code for quota exhaustion, not "server temporarily busy". This branch is only reached when the error did not already classify as RateLimit via 429, i.e. exactly when the code is carrying the meaning by itself.

If these two spellings came from concrete payloads that genuinely mean "at capacity", a one-line note here naming the providers would settle it permanently — the next person to add a code to this set will need the same reasoning, and right now the comment says what the code means but not who sends it or how you established that.

[Astro-Han]

[P3] This treats an absent Retry-After as more retryable than a malformed one, which inverts the usual information ordering.

parseRetryAfterMs returns undefined when neither header is present, and null when a header is present but unusable (unparseable, <= 0, or beyond MAX_SAFE_TIMER_DELAY_MS). So: no header at all → retryable: true and the caller backs off on its own; Retry-After: 0 or a garbage value → retryable: false and the turn does not retry at all.

A provider that sends a broken header ends up strictly worse off than one that sends none, even though the underlying condition is identically transient. The RateLimit branch below collapses both to non-retryable, which is defensible there because the server's own window is the whole point — but capacity is retried with local backoff, so the malformed case has a sensible fallback available and does not use it.

Not a blocker: it fails closed, and the user can retry by hand.

[Astro-Han]

[P3] Object.assign(capacity, { responseHeaders: ... }) mutates capacity in place rather than deriving a new error, so after this line the object asserted on above no longer has the shape it was asserted with. It happens to be harmless in the current order, but it makes the two assertions look independent when they are not — if anyone later reorders them or adds a case in between, the earlier assert.deepEqual(providerRetryMetadata(capacity), { retryable: true }) starts failing for a reason that has nothing to do with the code under test.

{ ...capacity, responseHeaders: ... } would not work here since these are Error instances, but building a second error the same way you built the first would.

Worth saying that the test is otherwise well-shaped: it pins both code spellings, covers the top-level code carrier as well as the nested data.error.code one, and asserts retryAfterMs rather than only the boolean.

[Astro-Han]

[P3] The new class is collapsed back to provider_unavailable here, so everything downstream of ProviderRetryReason loses the distinction this PR just introduced — including the retry banner copy, which will say 模型服务暂时不可用 / Model service temporarily unavailable rather than anything about capacity.

That may well be deliberate: ProviderRetryReason is a narrower vocabulary than ModelFailureKind and widening it touches the retry banner in both locales. If so, it is worth one line of comment here saying the narrowing is intentional, because as written it reads like the case was added only to satisfy the switch.

(Credit where due — this is the one thing the other reviewer on our side and I independently landed on, so it does stand out to a reader.)

[CxHsin]

Thanks for calling this out. The narrowing was not intentional; it did lose the user-visible distinction in the live retry banner.\n\nFixed in commit d473c5e: ProviderRetryReason now includes provider_capacity, the Runtime mapping and Runtime Host decoder preserve it, the compatibility epoch was bumped to 30 for the wire-contract change, and both locales have capacity-specific retry copy. Focused Runtime, Runtime Host protocol, and UI projection tests cover the full path.

[Astro-Han]

Heads up — this is currently conflicting with main, so I can't review or merge it as-is. Your CI is fully green, so it really is just the base that's behind.

One thing worth knowing: #3397 landed on 2026-08-22 and added ASF license headers across ~2685 files, so the rebase will touch more than you'd expect, and any file you add now needs a header (npm run write:asf-headers).

Ping me once it's rebased and I'll pick it up.

[CxHsin]

@Astro-Han PR #3365 已完成 rebase，当前基于最新 main（f19eede03），远端 head 为 4f00034。冲突已解决，构建和相关功能测试已通过；当前 PR 状态为 MERGEABLE，剩余 BLOCKED 原因为 REVIEW_REQUIRED，请继续处理。

[Astro-Han]

[P2] The capacity branch sits ahead of the partial-output and successful-tool guards, so a Turn that already produced work is told to retry — and retrying can re-run tool side effects.

if (lower === 'provider_capacity') return { action: 'retry',        label: copy.capacity };
if (input.partialOutputRetained)   return { action: 'continue',     label: copy.partial };
if (input.toolActivityCount > 0)   return { action: 'inspect_tool', label: copy.toolRecord };

Only the errored-tool guard runs before capacity; the two guards that protect retained output and successful tool activity run after it.

That combination is reachable. A Turn completes an assistant/tool step, the next model request comes back resource-exhausted, and all 10 physical attempts for the current step fail before producing new output. Auto-retry only asks whether this attempt has observable output (ai-sdk-backend.ts:2530-2602), so the Turn can terminate with errorClass=provider_capacity while partialOutputRetained=true or toolActivityCount>0. Calling the presentation function directly on this head:

• capacity + partialOutputRetained=true → retry / Wait a few minutes or switch models before retrying
• capacity + toolActivityCount=1 → the same retry prompt

Why the consequence is worse than a mislabel: the user follows that advice and clicks Regenerate, and session-manager.ts:4846-4903 resubmits the original Turn's user content — so tool side effects that already succeeded can happen a second time. "Continue" and "inspect tool" exist precisely to keep completed work from being redone.

The existing suite says as much: session-status-presentation.test.ts:60-80 explicitly treats these two as higher-priority prompts. The new capacity tests only cover the 0-output / 0-tool case, so the ordering regression passes unnoticed.

Minimal fix: move the capacity branch below the partial-output and successful-tool guards and above the generic output-free fallback, and add a regression for capacity with retained output or tool activity.

Independent line, bound to 4f000347. Verified at the gate 2026-08-23 13:12 UTC.

[CxHsin]

已在 8ebbb02 修复：partialOutputRetained 和 toolActivityCount 现在先于 provider_capacity 处理，分别引导 continue / inspect_tool；新增这两个 mixed-state 回归断言。已通过 Desktop recovery 测试。

[Astro-Han]

[P2] Weak wrapper evidence outranks the exact capacity code, so the classification this PR is built on is lost whenever a transport layer wraps the error.

if (text.includes('abort')) return 'Abort';
if (statusCode === '402' || code === '402') return 'ProviderBilling';
if (statusCode === '429' || code === '429') return 'RateLimit';
if (statusCode === '401' || statusCode === '403' || ...) return 'Auth';
if (PROVIDER_CAPACITY_CODES.has(normalizedCode) || structuredCodes.some(...)) return 'ProviderCapacity';

Free-text abort matching and the generic numeric fallbacks all run before the precise structured code. Probed directly on this head:

• {statusCode: 429, data: {error: {code: 'resource-exhausted'}}} → RateLimit, and with no Retry-After present, retryable=false
• {message: 'request aborted because model at capacity', data: {error: {code: 'resource-exhausted'}}} → Abort

The same verified xAI code loses its capacity copy and its bounded local backoff purely because an SDK or gateway added an outer transport status or a descriptive phrase. The 429 case is the sharper one: it does not merely relabel, it can flip retryable to false — turning a condition this PR wants to back off from into one that does not retry at all.

The existing tests only exercise payloads with no outer status and no abort wording, so the precedence problem is invisible from inside them.

Why reordering is safe here specifically: this head already excludes the underscore form resource_exhausted, whose meaning is broader (in Google/gRPC it commonly means quota). Promoting the exact hyphenated code above weak wrapper evidence therefore does not re-blur the Google/gRPC quota case that exclusion was added to avoid.

Minimal fix: match the exact resource-exhausted structured/top-level code before free-text abort and the generic numeric fallbacks, and add a 429-wrapper regression. This is also the precedence #2521 already adopts — structured provider identifiers ahead of ambiguous transport facts — so the two changes agree on the principle.

Independent line, bound to 4f000347. Verified at the gate 2026-08-23 13:12 UTC.

[CxHsin]

已在 8ebbb02 修复：保留显式 RetryError abort，但 provider 的精确 resource-exhausted 结构化证据现在优先于 free-text abort 和 402/429/401/403 fallback。新增 abort-text 与 429 混合载荷回归断言，并验证 capacity 仍可重试。

[Astro-Han]

Independent review of 4f000347786612c0b1b85af27ffa970bd85d07e4. Two [P2]s, both inline, both about ordering rather than logic — every individual rule here is right; two of them are in the wrong place relative to their neighbours.

• fix(runtime): classify provider capacity errors #3365 (comment) — the capacity branch runs before the partial-output and successful-tool guards, so a Turn that already did work is told to retry, and retrying can re-run tool side effects.
• fix(runtime): classify provider capacity errors #3365 (comment) — free-text abort and the generic 402/429/401/403 fallbacks outrank the exact resource-exhausted code, so a wrapped error loses its capacity classification (and in the 429 case, its retryability).

The bigger question is not in this diff: duplicate authority with #2521

Inside this branch there is no second classifier — it plugs into the existing classification, retry, wire, and Desktop presentation chain. Of the 18 files, 7 are tests and the other 11 are real core-type / runtime / strict-wire / Desktop boundaries. Deleting files is not the simplification available here.

The actual duplication is with #2521, which is open on the same baseline f19eede0:

• 13 of 18 files overlap.
• Both raise the compatibility epoch from 39 to 40.
• git merge-tree produces 7 content conflicts — classifier, model adapter, Desktop presentation, Runtime Host protocol/handshake.
• fix(runtime): distinguish usage limits from auth errors #2521 has already established ProviderFailureResult as a single provider-failure authority.

That is not a third finding against this diff, but it is a merge-order and design gate: two implementations of the same concept should not both land. If #2521 is the intended authority, the low-entropy path is for this PR to rebase and contribute ProviderCapacity into that one taxonomy. If this one lands first, #2521 has to rebase and absorb the concept. Either order works — doing neither is the failure mode.

Candidates raised and withdrawn

• "Classify the generic resource_exhausted as capacity too" — in Google/gRPC that commonly means quota; restricting to the xAI hyphenated form is evidence-based. Withdrawn.
• "Fold the capacity retry reason back into provider_unavailable to avoid the epoch bump" — that discards the stable semantics this PR exists to establish, at the exact moment the user can see it. Withdrawn.
• "18 files means the change is too broad" — 7 tests plus 11 genuine contract boundaries. Withdrawn; the overlap to remove is with fix(runtime): distinguish usage limits from auth errors #2521, not within this diff.

State of the head

• Fresh REST reports mergeable=false / dirty against origin/main=2e5fe9c5; local merge-tree reproduces the conflict in packages/runtime-host/src/__tests__/protocol.test.ts. The 01:42 UTC comment saying MERGEABLE is stale.
• check-runs on this head: 0. The only workflow run (32609206478) is terminal action_required with 0 jobs — the tests never started. Not green; never run.
• All three review endpoints were read separately. The only substantive prior review is bound to the older commit 2a8a0b9e, and its findings (underscore quota, malformed Retry-After, test-object reuse, lost retry reason) are addressed on this head. Neither P2 above is a restatement of those.
• Local verification: clean worktree, npm ci fine, Desktop build:with-deps fine, 150/150 relevant tests pass. Those 150 do not cover either mixed-state path above — which is the point.

Blind line: provisional judgment sealed before any existing review was read. Reviewed at 2026-08-23 13:12 UTC. No overall verdict offered.

[Astro-Han]

Both [P2]s are fixed on the new head 8ebbb02953a69057814a3969eda1b582577463da. I re-derived them rather than taking the rebase on trust — the ordering is now correct in both places.

Presentation — capacity moved below the two guards it was jumping:

if (input.partialOutputRetained) return { action: 'continue',     label: copy.partial };
if (input.toolActivityCount > 0) return { action: 'inspect_tool', label: copy.toolRecord };
if (lower === 'provider_capacity') return { action: 'retry',      label: copy.capacity };

A Turn that already retained output or ran a tool successfully now gets continue / inspect tool instead of retry, so the path that could re-run completed tool side effects is closed.

Classification — the exact structured code now precedes the weak wrapper evidence:

if (PROVIDER_CAPACITY_CODES.has(normalizedCode) || structuredCodes.some(...)) return 'ProviderCapacity';
...
if (text.includes('abort')) return 'Abort';
if (statusCode === '429' || code === '429') return 'RateLimit';

{statusCode: 429, data: {error: {code: 'resource-exhausted'}}} now classifies as ProviderCapacity rather than RateLimit, which also stops it from losing retryable when no Retry-After is present. Context-overflow structured evidence was promoted alongside it, which is consistent — structured provider identifiers ahead of ambiguous transport facts, applied uniformly rather than only where it was reported.

Two things still open, neither of them mine to close:

1. The fix(runtime): distinguish usage limits from auth errors #2521 overlap stands and is unaffected by a rebase: 13 of 18 files in common, 7 content conflicts, and both PRs raising the compatibility epoch. That remains a merge-order decision — whichever lands first, the other must absorb the concept rather than duplicate it.
2. This head arrived at action_required with zero jobs, same as the last one. I have approved the run so it can report. Not green yet — never run yet.

I will re-check once CI reports.

[CxHsin]

To clarify the integration recommendation for #2521 and this PR:

These are not merely adjacent changes. #2521 establishes a broader provider-failure authority (ProviderFailureResult) across the same runtime classification, runtime-host protocol, Desktop presentation, and test surfaces that #3365 changes for ProviderCapacity. Both also evolve compatibility/protocol behavior.

I recommend choosing one authority before merging:

• If fix(runtime): distinguish usage limits from auth errors #2521 is intended to land, rebase fix(runtime): classify provider capacity errors #3365 onto it and express ProviderCapacity through ProviderFailureResult.
• If fix(runtime): distinguish usage limits from auth errors #2521 is not landing first, merge fix(runtime): classify provider capacity errors #3365, then rebase fix(runtime): distinguish usage limits from auth errors #2521 and have it absorb ProviderCapacity, the capacity-specific recovery behavior, and the protocol epoch change.

Landing both independently would leave two classification authorities and makes divergent retry/recovery behavior and protocol conflicts likely. This is an architectural merge-order decision rather than a routine conflict to resolve after both land.

[CxHsin]

Fixed in c001f6b.

durableProviderErrorClass() now preserves structured ProviderCapacity evidence before HTTP fallback classification, matching the existing ContextLength treatment. Added durable-diagnostic regression coverage for resource-exhausted wrapped by both HTTP 429 and 503. The targeted runtime classification suite passes (12/12).

[Astro-Han]

Approving c001f6b85adf6d4268b91e414b0d1a9d9036c1b2. Required test is completed / success bound to that exact SHA. No P0–P3.

Re-review at the current head. Both earlier [P2]s were re-derived here rather than accepted as fixed on the strength of the "Fixed in c001f6b" note.

Ordering fix, verified in place. In classifyProviderFacts, the PROVIDER_CAPACITY_CODES test against structuredCodes now sits above text.includes('abort'), above the 429 branch and above the 5xx branch. That is the whole point of the finding: a transport layer wrapping the error used to let weak textual evidence outrank the exact capacity code the PR is built on. It no longer can. providerFailureDiagnostic also retains ProviderCapacity rather than widening it back to RateLimit/ProviderUnavailable.

Guard ordering fix, verified in place. In session-status-presentation.ts, partialOutputRetained and toolActivityCount are handled before the capacity branch, so provider_capacity only reaches a Turn that retained no output and ran no tools. That matters beyond presentation: the old order could invite a retry on a Turn whose tool side effects had already happened.

One deliberate non-change worth recording. resource_exhausted with an underscore is still not treated as capacity. That is correct rather than an oversight — in the gRPC/Google error model RESOURCE_EXHAUSTED is the standard code for quota exhaustion, not "server temporarily busy", and folding it in would recreate the misclassification this PR exists to remove.

Coverage now spans abort text, a 429 wrapper and a 503 wrapper; focused suites 217/217.

Disclosure, because it changes what this approval is worth: this is an AI review. Under CONTRIBUTING.md §Review it does not count as the required independent human review. It means the code has been checked, not that the gate is open — merge still needs a committer other than the author to give LGTM and to decide.

[Astro-Han]

Hi — thanks for c001f6b85; the durableProviderErrorClass() change reads right.

Heads-up that this branch now conflicts with current main and can't be merged as-is. I tested a rebase locally in a throwaway worktree (your branch was untouched); it stops on:

• packages/runtime-host/src/protocol/index.ts
• packages/runtime-host/src/__tests__/protocol.test.ts

Worth knowing what that first conflict is, because it isn't mechanical: it's RUNTIME_HOST_COMPATIBILITY_EPOCH. Every protocol-changing PR increments that single counter and prepends a line to the changelog comment above it, so any two protocol PRs in flight at once collide there by construction. main has since moved the epoch past the value your branch sets.

Resolving it means re-deriving your epoch bump on top of whatever main is at now, and keeping your changelog line alongside the ones that landed meanwhile — i.e. it's a deliberate "is my compatibility claim still accurate relative to the current epoch?" decision, which is exactly why it isn't auto-mergeable.

git fetch upstream && git rebase upstream/main
# resolve, then
git push --force-with-lease

Once the branch is conflict-free and CI is green on the new head, I'll pick the review back up.

---

AI-assisted maintenance note, not a review. It does not count as the required human review under CONTRIBUTING.md §Review.

[CxHsin]

Rebased and resolved in 569f7b066.

The branch is now based on current apache/maka:main (efddab2fb) and is conflict-free. Since main already uses compatibility epoch 41, the provider-capacity retry protocol change is now recorded as epoch 42, while retaining the epoch 41 and 40 changelog entries. The protocol assertion was updated accordingly.

Verification after a clean workspace build:

• npm run build passed
• Runtime Host protocol tests: 43/43 passed
• Runtime provider classification tests: 12/12 passed

The rebased head was pushed with --force-with-lease; please rerun CI and resume review.

[Astro-Han]

Approving at exact head 569f7b066dd8a9b2a8ab5963dfb484fb21f81772.

This head is the rebase of the previously-approved c001f6b85. I re-anchored rather than carrying the old approval forward: comparing the two ranges, the first two commits are byte-identical and the third differs only in how the protocol-epoch conflict was resolved — the test assertion moves from > 39 to > 41 and merges with main's compaction-epoch test. That is the expected resolution for this file, and it changes nothing about the reviewed behaviour.

CI is green on this exact head. Worth noting for the record that the run only happened after the fork workflow was approved — before that, this head had zero check-runs, so the earlier state was "never ran", not "passed".

No live [P0]–[P2] at this head.