Skip to content

Make RemoveWatcher converge after lost HTTP responses #18

Description

@sweetcornna

Problem

PR #15 makes physical watcher cleanup single-owner, but RemoveWatcher is not replay-safe end to end. If removal succeeds and the HTTP response is lost, a retry may return success while the record remains retired, then later return not_found after reap/finalization. The client currently makes only one cleanup request, so an ambiguous transport failure can leave a watcher until later lifecycle cleanup.

The no-wake path also needs an explicit result for a sandbox whose E2B kill deadline has expired but whose physical teardown has not completed.

Required semantics

Define removal as a goal-state operation.

  • For an authorized sandbox and well-formed watcher ID, sequential and concurrent retries converge to success while the watcher is active, retired, already reaped, or already absent.
  • If the daemon removed the watcher but the response was lost, retrying succeeds without a second physical side effect.
  • If the first request never arrived, retrying performs the removal.
  • Any tombstone or deduplication state is bounded; the design must not retain watcher IDs forever.
  • The console uses bounded retry/backoff for ambiguous transient transport outcomes.
  • Removal remains no-wake for frozen, explicitly paused, stopped, and logically dead sandboxes, including the interval after the E2B kill deadline but before physical teardown.
  • When retirement applies, the next legitimate wake reaps it before user work.

Acceptance tests

  • response lost after the server acted;
  • first request never arrived;
  • sequential and concurrent duplicate removal;
  • repeated removal while retired;
  • retry after reap/finalization;
  • logically dead before physical teardown;
  • frozen/paused/stopped paths remain cold;
  • physical watcher count and table capacity return to baseline.

Non-goals

Follow-up to #15.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions