fix(sandbox): handle every Daytona lifecycle state + recover dead containers mid-request - #2
Merged
Merged
Conversation
…e raw bucket URL The Copy link / Open URL was the game's physical R2 path, games/<clerkUserId>/<threadId>/current/index.html — every shared link carried the owner's Clerk user id to whole classrooms. A new public /play/[threadId] route resolves the owner from the lessonplay_game_version index and streams current/index.html through, so the bucket layout (and the user id) stays server-side. The tRPC router now hands the UI a same-origin sharePath, the header builds the absolute URL from the page's own origin at copy time, and the agent's publishedUrl — what the model pastes into chat after a publish — points at /play too, derived from the request origin so dev and preview deploys hand out links on their own host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the container dies mid-request Production incident (trace 65e76ce8): a thread reopened while Daytona was archiving its sandbox got a handle in state "archiving", which the old four-state walk in getOrCreateSandbox silently returned as "existing-other" without starting it. Every tool call then failed with "bad request: failed to resolve container IP after 3 attempts: no IP address found. Is the Sandbox started?". Three guardrails: - src/server/sandbox/lifecycle.ts (new, env-free): planSandboxStep maps ALL ~23 SDK states to an explicit step — use / start / wait-started / wait / fail — so mid-transition states (archiving, stopping, pausing, ...) are polled until they settle instead of being used, unknown future states wait rather than pass, and broken states (error, destroyed) throw something readable up front. - daytona.ts: ensureSandboxStarted walks any observed state to STARTED with a deadline and a start-attempt cap; getOrCreateSandbox now reports the raw initial state, and prepare logs it, so the next novel state shows up in one log line instead of needing to be inferred. - withRecoveredSandbox: every sandbox tool (bash/read/write/edit/validate/ publish) retries once via a route-supplied recoverSandbox when the container dies under a call (auto-stop or archive racing a request). Recovery re-runs the full prepareLessonSandbox — not a bare restart — because the s3fs mount does not survive the container; prepare is already idempotent. Covered by unit tests in src/server/sandbox/__tests__/lifecycle.test.ts, including the exact production error message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incident
Production trace
65e76ce8(found via runtime logs):A thread was reopened while Daytona was archiving its sandbox (auto-stop after 30 min idle → auto-archive after 60 min stopped; moving the ~3 GiB filesystem takes minutes).
getOrCreateSandboxhandled only 4 of the SDK's ~23 states (STARTED,STOPPED,ARCHIVED,STARTING);archivingfell through toexisting-otherand the sandbox was returned without being started — so every tool call in the run failed with the container-IP error. The code literally didn't know the sandbox was being archived.Guardrails
src/server/sandbox/lifecycle.ts(new, env-free) —planSandboxStepmaps every state to an explicit step:use/start/wait-started/wait/fail. Mid-transition states (archiving,stopping,pausing, …) are polled until they settle; unknown future states wait rather than silently pass; broken states (error,destroyed, …) fail fast with a readable message.daytona.ts—ensureSandboxStartedwalks any observed state toSTARTED, with a 180 s settle deadline, a 120 s start timeout (archived restores are slow), and a start-attempt cap.getOrCreateSandboxnow returns the raw initial state andpreparelogs it (sandbox.acquired { status, state }), so the next novel state is diagnosable from one log line.withRecoveredSandbox— all six sandbox tools (bash/read/write/edit/validate/publish) retry once via a route-suppliedrecoverSandboxwhen the container dies under a call (auto-stop/archive racing a request, or the prepare promise itself rejecting that way). Recovery re-runs the fullprepareLessonSandbox— not a barestart()— because the s3fs R2 mount doesn't survive the container; prepare is already idempotent. Omitted in tests, so the tool set stays unit-testable.Testing
pnpm typecheck✅pnpm test✅ — 67 passed, including newlifecycle.test.tscovering the state map, the unreachable-error matcher (with the exact production error string), and the recover-once semantics.pnpm sandbox:smokecovers the happy path, and the archiving window is hard to reproduce on demand.🤖 Generated with Claude Code