Skip to content

fix(storage): a non-convergent onboarding intent permanently bricks the runtime-policy store #3566

Description

Problem

Crash recovery treats a connection-onboarding intent that cannot be replayed as fatal, and the failure is permanent: the intent journal is never cleared on that path, so every subsequent open of the interactive runtime-policy store fails the same way. The app cannot start against that storage root again without manually deleting runtime-policy-onboarding.json.

Mechanism (line refs at main@e05308be7):

  • openInteractiveRuntimePolicyStoresForWrite runs coordinator.recoverForWrite() before handing out any writer (packages/storage/src/runtime-policy-stores.ts:192).
  • recoverForWriterecoverConnectionOnboarding (coordinator.ts:209, :1317) replays the journaled intent through applyConnectionOnboardingprepareOnboardingUpsert.
  • When the catalog has drifted so that the slug the intent resolves to is occupied by a connection with a different connectionId, the upsert throws invalid_document — "Onboarding intent conflicts with the connection id" (connection-catalog-document.ts:534) — which recovery wraps as commit_outcome_unknown — "Connection onboarding recovery did not converge" (coordinator.ts:1330) and rethrows.
  • Nothing clears the intent on this path, so the next open replays the same intent into the same conflict. Verified experimentally: three successive opens with fresh root leases all fail identically.

Reproduction (verified against real stores)

  1. Onboard a connection so an intent is journaled, and hard-kill the process between writeConnectionOnboardingIntent and clearConnectionOnboardingIntent (the crash window the journal exists for).
  2. Out of band, delete and re-create the connection at the same slug (new connectionId) — e.g. by another build, a restored backup, or manual edit of connection-catalog.json.
  3. Reopen the store: commit_outcome_unknown / invalid_document: Onboarding intent conflicts with the connection id, permanently.

Reaching the state requires a hard kill plus out-of-band catalog drift, so it is rare — but the blast radius (the store never opens again) is disproportionate to the fault (one stale journal entry describing a row that provably no longer exists).

Why fail-closed is the wrong shape here

Fail-closed is right for I/O errors and undecodable documents: retrying may converge and throwing protects unknown state. This case is different — the conflict proves the intent is obsolete: the connection it named is gone, and its slug now belongs to a different identity. Replaying it can never become correct, so refusing to open the store protects nothing; it converts a completed-or-abandoned onboarding into a bricked installation.

Suggested direction

In recoverConnectionOnboarding, treat the id/slug-conflict (invalid_document from the upsert's identity cross-check) as an obsolete intent: log it, clearConnectionOnboardingIntent, and continue opening. Keep the current fail-closed behavior for every other error class (I/O failures, undecodable journal, vault errors), where retry can still converge. A regression test can hand-write the drifted catalog + stale journal and assert the store opens and the journal is cleared.

Provenance

Found during adversarial review of #3467 (which adds one more input to the same pre-existing throw sites but does not change the recovery semantics); reproduced there through a path untouched by that PR. Details in that PR's review-record comment.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions