fix(sdk): harden agent-loop error recovery, retry, and pause semantics#2
Open
matej21 wants to merge 2 commits into
Open
fix(sdk): harden agent-loop error recovery, retry, and pause semantics#2matej21 wants to merge 2 commits into
matej21 wants to merge 2 commits into
Conversation
Closes three related defects in the agent loop's error/pause/retry paths — the same resume_from_error ↔ infer family previously patched case-by-case in 1cb6a25, b2cfd17, c2220d6 and 245b2b1: - Retryable LLM errors that exhaust withLLMRetry no longer spin an unbounded resume_from_error ↔ infer loop. Error-resume cycles back off exponentially (5s base, 60s cap, configurable via errorResumeBackoff) using new AgentState bookkeeping (consecutiveInferenceFailures / lastInferenceFailureAt); onError hooks (parent mailbox notification) fire once per error episode instead of once per cycle. A session restart resets the episode. - afterInference:'retry' no longer strands the agent or corrupts history. Dequeue-token consumption is deferred until the turn commits, and a new inference_retried event rolls back the staged turn so the retry re-collects the same messages instead of double-appending tool results (Anthropic rejects duplicate tool_result blocks). Consuming after the commit also closes the crash window that dropped a consumed-but-uncommitted message on restart: the gap now re-delivers (at-least-once) instead of losing the message. - A pause raised mid-turn is no longer clobbered. The tool_completed / tool_failed / inference_completed reducers preserve 'paused', the tool_exec loop stops at the first pause instead of running the remaining tools, afterToolCall pause still commits the executed tool's result (no side-effect re-execution on resume), and agent_resumed routes back to tool_exec while tool calls are pending. Adds TestSession.pauseAgent/resumeAgent helpers and an error-recovery.integration.test.ts suite covering all three paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2Dk23As1wEZ14mie22hGA
The agent-tree and agent-detail projections duplicated the status recompute logic that the parent commit fixed in the core reducer, so a mid-turn pause was still clobbered in the client-side view (and resume went to 'pending' even with tool calls pending) while the server kept the agent paused / in tool_exec. With agent_state_changed never emitted, nothing corrected the drift. - Extract the transition rules into projections/agent-status.ts and use it in both projections: preserve 'paused' on inference_completed and tool_completed/tool_failed, resume into tool_exec when tool calls are pending, and handle the new inference_retried rollback. - Reset 'errored' agents on session_restarted (core already did; the projections only reset 'inferring'). - Add a conformance test that replays identical event sequences through the sdk core reducer and both projections and fails on any status or pending-tool-call divergence. The sdk cannot import the shared helper (project-reference cycle), so the test is what keeps the two copies in lockstep. It imports sdk from source (not dist) and lives outside shared/src so it runs under bun test without joining the tsc project. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2Dk23As1wEZ14mie22hGA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the three high-severity findings from the agent-SDK architecture review (CORR-1..3), plus the two related mediums they subsume (CORR-6 crash-window message loss, CORR-10 lost manual pause).
Context
Git history shows this is one bug family patched case-by-case:
1cb6a25introduced preserve-for-retry token semantics, thenb2cfd17,c2220d6and245b2b1each fixed one instance of the resultingresume_from_error ↔ inferloop (empty-response, aborted, non-retryable). This PR closes the family with three invariants instead of a fifth spot patch.Invariant 1 — error-resume cycles back off (CORR-1)
A retryable LLM error (rate limit / server error / network / timeout) that outlives
withLLMRetry's 5 attempts keeps the dequeue token unconsumed, sodecide()re-enteredinferimmediately: an unbounded, backoff-free spin that pinned the loop and sent the parent one mailbox error message per cycle during a provider outage.AgentStatenow tracksconsecutiveInferenceFailures/lastInferenceFailureAt(reducers:inference_failedincrements,inference_completedresets,session_restartedresets the episode).continue()'sresume_from_errorbranch waits out an exponential backoff (5s base, 60s cap, per-agenterrorResumeBackoffconfig) and yields instead of spinning; a timer re-schedules processing when the backoff elapses.onErrorhooks (parent notification) fire once per error episode, not once per cycle.Invariant 2 — tokens are consumed only when the turn commits (CORR-2, CORR-6)
markConsumedran beforeexecuteAfterInference, so a plugin returning{action:'retry'}recursed intorunInferencewith the mailbox message already consumed: a pure-message turn stranded the agent ininferring(→idle, unrecoverable), a tool-result turn double-appendedpendingMessagesand produced duplicatetool_resultblocks that Anthropic rejects.inference_retriedevent rolls back the staged turn (clearspendingMessages, like theinference_failedrollback from1cb6a25) so the retry re-collects the same unconsumed messages — nothing lost, nothing duplicated.session_restartedwipes stagedpendingMessages, so consume-first lost the message).Invariant 3 — pause is never clobbered mid-turn (CORR-3, CORR-10)
The
tool_execloop ran all pending tools without re-checking status, and thetool_completed/tool_failed/inference_completedreducers recomputed status unconditionally — a pause from a tool hook (or a manualpauseAgent()landing mid-inference) was silently overwritten, the paused tool re-executed its side effects, and the history could end with an unpairedtool_use.paused; thetool_execloop stops at the first pause (remaining tools stay pending and run after resume).afterToolCallpause still emits the executed tool's result event — side effects already ran, so the committedtool_usegets itstool_resultand nothing re-executes on resume. (beforeToolCallpause keeps gate semantics: tool never started, runs after resume.)agent_resumedroutes back totool_execwhile tool calls are pending — resuming intopendingwould have run inference with unresolved tool calls.Tests
New
error-recovery.integration.test.ts(6 tests): outage → backoff timing between failure/resume events, single parent notification, message survives and is delivered exactly once;afterInferenceretry for pure-message and tool-result turns (no stranding, no duplicate tool results in history or LLM request);beforeToolCall/afterToolCallpause (nothing runs while paused, exactly-once execution across resume); manual pause during in-flight inference (pause survives the commit). Also addsTestSession.pauseAgent/resumeAgent.bun run ts:buildclean, full sdk suite 1642 pass / 0 fail.🤖 Generated with Claude Code
https://claude.ai/code/session_01X2Dk23As1wEZ14mie22hGA