feat(deploy): two-phase stage/activate for deploy_component

[dawsontoth]

Draft / RFC. Splits deploy_component into two replicated phases with a cluster-wide barrier at go-live, and adds a fast, addressed rollback. The public request/response contract is unchanged. Stacked on #2066 (see Sequencing), so it should merge after it.

Decisions needed before this leaves draft

Four review findings are design changes to the two-phase contract rather than bugs with an obvious
fix. Each is defensible as a follow-up on a PR this size, and each is also arguably a correctness gap
worth closing before 5.3 ships. They need a call, not a patch — tick the ones you want in this PR.

• Cluster-wide activation ordering (heskew, the round's only High). The activate fan-out
  happens after the origin releases its node-local component lock, so nothing orders two
  concurrently-originated deploys. If node A originates D1 and node B originates D2, each can
  apply the other's fan-out last: A ends on D2, B on D1, and both deployments report success.
  The stage barrier does not help — it orders stage-before-activate, not deploy-against-deploy.
  Fixing it means a cluster-wide per-project order: leader serialization, or a monotonic
  activation epoch compared under the component lock and rejected if stale. Either is a real
  addition to the protocol, and the epoch option needs a home for the counter that survives
  restarts.
  Not fixed here. Today, concurrent originations are a documented sharp edge rather than a
  guarded one.

• Resolve the package to bytes once, on the origin. For a package: deploy the origin sends
  the identifier, not a payload, so every node independently resolves and packs it. A moving
  latest, a semver range, or a git branch can therefore resolve differently per node while
  every stage reports success — the barrier confirms "everyone staged something", not "everyone
  staged the same thing". The fix is to resolve once on the origin, persist the exact tarball plus
  a digest on the deployment row, and have peers stage and verify those bytes.
  This changes what the barrier guarantees, and it makes a package: deploy carry a payload it
  currently does not, with the retention cost that implies.

• Require the deployment table for separated-phase operations. DeploymentRecorder.put()
  silently no-ops when hdb_deployment is absent, but activate: false still returns a
  deployment_id. A later deploy_component({ deployment_id }) then cannot find the row and
  cannot activate the build — a stage that reported success is unactivatable. The fix is to
  require tracking availability for activate: false / deployment_id paths while keeping the
  tolerant fallback for legacy one-shot deploys. Straightforward, but it turns a currently-quiet
  configuration into a hard failure, which is a product call.

• Peer row ownership. claimStagedDeployment ends in a patch on the replicated
  hdb_deployment row, and every peer runs it in its activate beforeSwap. So N peers plus the
  origin write the same key — the concurrent same-key pattern seal() exists to avoid — and under
  replication lag a peer can patch activating after the origin wrote success, reverting a
  converged deploy to a permanently non-terminal state. Both DESIGN.md ("peers make the same
  local claim") and revertComponent say the origin owns the row, so the implementation
  contradicts the stated design. The fix is a persist: false mode for peers, whose claim is then
  enforced by the component preparation lock they already hold.

Everything else from the review rounds is addressed in-branch — see the commit log and the resolved
threads.

Why

Before this, the whole deploy ran as one replicated operation, and on every node extract + npm install — the slow, failure-prone work: git clone, registry install — happened in place in the live component directory, right before restart. There was no cluster-wide barrier: a peer could fail npm install and sit half-baked while other nodes had already restarted onto the new code. Nodes flipped independently.

(main has since made in-place preparation transactional per node via #1936/#2066, so the per-node rollback half of that argument is now solved upstream. The cluster-wide ordering argument is what this PR is for, and it stands on its own.)

What this does

Two phases, orchestrated internally by deploy_component:

Phase	Does	Does not
1. Stage	download/npm pack (incl. git clone), extract, npm install into a hidden .deploy-staging/<deploymentId>/<name> on every node	touch the live dir, write config, restart
2. Activate	atomic rename(staging → live), persist config + install lock, restart	re-fetch or re-install anything

The origin stages locally, waits for every node to report a good stage before any node activates, then activates. A node that can't fetch the package or fails npm install fails during staging, and the live component is untouched on every node. Go-live shrinks to a fast atomic swap. Single-node instances benefit too: npm install runs off to the side while the live component keeps serving.

Public surface

One public deploy operation. The phases are not separate operations — the peer fan-out is a distinct authenticated component_deploy_phase operation, so a pre-upgrade peer rejects an unknown operation instead of misreading a phase marker as a one-shot deploy. Public _phase/_deploymentId fields are rejected.

Operators get the phases through deploy_component properties:

• default → full stage + activate (contract unchanged)
• activate: false → stage cluster-wide and stop, returning the deployment_id in a staged state
• deployment_id: <id> (no new payload) → activate a previously-staged deployment
• two_phase: false → the legacy one-shot path, preserved verbatim

revert_component is a separate public operation: it is a rollback, not a deploy phase, and it resolves no package, decrypts no secret, downloads no artifact and runs no install.

Rollback: addressed, idempotent, config-level

Every activation retains the tree it displaced as .deploy-previous/<name> with a sidecar manifest recording which deployment produced it and the root-config entry it was activated with. revert_component takes a required to_deployment_id:

• already live → no-op success, so a retry after a lost response cannot flip the rejected release back in
• matches the retained previous → swap; the displaced tree becomes the new retained previous, so an explicitly targeted revert-of-a-revert rolls forward
• anything else → refused, naming what the component can actually revert to

The swap carries persistent state with it: root config and harper-application-lock.json move via a shared createApplicationConfigTransaction. Reverting away from a package deploy removes the package reference, so installApplications() can't reinstall the reverted-away version over the restored directory on the next cold start. Exactly one previous is retained, so revert reaches back one activation; anything older is a redeploy.

Automatic rollback is deliberately not offered (revert_on_failure is rejected). Once a node is past the barrier, "peer reported failed" does not imply "peer did not activate" — a peer can complete its swap and then fail the work that follows — so auto-reverting the failed peers can roll an untouched node an extra version back and split the cluster three ways. A partial activation stays visibly activating and is rolled forward, or rolled back explicitly by target.

Sequencing: stacked on #2066

Per @kriszyp's review, this branch is based on #2066 rather than main, so there is exactly one naming contract on .deploy-aside instead of two. extractApplication runs #2066's aside transaction against application.buildDirPath — the live dir by default (one-shot, boot installs and direct callers unchanged), the staging dir during a stage — so one protocol covers both. The evicted two-deploys-ago tree is parked under a distinct .discarded- prefix, because #2066's startup sweep restores an unretired .in-progress- directory over the live component and known-garbage must never look like a rollback record.

Why staging lives under the components root

Go-live is rename(stagingDir, liveDir), atomic only when both share a filesystem. os.tmpdir() is frequently a different mount → EXDEV → a slow recursive copy at exactly the moment you want an instant swap. So staging is a hidden dir under the components root: same volume, dot-prefixed so the loader ignores it, and not the watched base of any component's watcher — so building there fires no restart-on-change events.

Durability

Stage/activate is a durable transaction: completion markers, immutable per-deployment activation specs, and atomic live/config/lock activation. Startup reconciliation rolls an interrupted activation forward, rebuilds a missing peer stage from the retained payload, and preserves uncertain partial activation as activating. Staged builds and payloads are bounded by deployment_stagingRetention_maxCount and deployment_payloadRetention_maxCount (default 1, conservative on purpose — N copies of a large payload competing with customer data for a small quota is a nasty failure mode).

Restart-required gate

Installed package metadata sits outside most plugin watch globs, so no watcher sees a dependency or module-entry change — but it invalidates loaded code. activateStagedApplication compares the outgoing live tree against the staged tree at swap time and feeds markRestartRequiredForDeploy, matching what the in-place path gets from prepareApplication. The stage records whether its install was opaque inside its completion marker, since on a peer the stage and the activate are separate invocations.

Tests

~300 unit tests across deployStaging, deployPhaseOperations, deployPhaseValidators, deploymentRecorder, deploymentOperations, cliOperations and the server dispatch seam — covering the barrier, crash windows, duplicate/concurrent activation, trusted-peer dispatch, fail-closed paths, retention, the addressed revert (including the retry no-op and config removal), and the restart gate.

Still open

• The cross-node fan-out is validated against harper-pro's replicator by construction, not yet exercised on a live multi-node cluster.
• Restore components after failed deployment preparation #2066 currently has 8 failing tests on its own branch (verified pre-existing, not merge fallout), inherited here.
• Docs companion: docs(deploy): two-phase deploy, revert, by-reference deploys, and OIDC trusted publishing documentation#599, updated to match. Hold its merge until this lands.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a two-phase deployment process (stage then activate) for components to ensure all-or-nothing cluster-wide deploys. Staging builds the incoming version into a hidden staging directory, while activation atomically swaps it into the live path. The review feedback highlights two critical issues in components/Application.ts: first, recursively deleting the parent staging directory during cleanup can accidentally destroy parallel staged builds for the same component; second, using access to check directory existence follows symlinks, which will fail on dangling symlinks and cause subsequent operations to throw EEXIST. Both comments provide actionable code suggestions to resolve these issues.

[claude]

Reviewed; no blockers found.

[dawsontoth]

Thanks @kriszyp — draft or not, the review landed the key question squarely. revert_on_failure's cluster node-targeting was the subtle part, and it surfaced two real bugs: reverting peers that never activated (rolling them an extra version back), and a self-directed double-revert that flipped the origin right back to the broken version. Both are fixed, and the targeting is now a pure selectRevertTargets(nodes, failedPeers, thisNode) with direct unit tests so neither can regress.

A few calls I deliberately left for a maintainer rather than deciding unilaterally — your read would be welcome:

• revert_on_failure is opt-in, not the default. Auto-reverting a cluster after some nodes have already restarted onto the new version is a policy choice I didn't want to make for you.
• revert is a bidirectional swap (live ↔ .deploy-previous, one previous retained per component) rather than a rollback tied to a specific deployment_id. The bidirectionality is what makes "reverting a revert rolls forward again" work, but it's also exactly why the self-revert bug bit.
• The retention model, the atomic-rename/.deploy-previous reasoning, and the replicator contract this rides on are written up in DESIGN.md's "Two-phase deploy" section.

Still a draft on purpose: the cross-node fan-out is validated against harper-pro's replicator by construction (documented), but hasn't been exercised on a live multi-node cluster — that plus your design read are the gates before it comes out of draft.

🤖 Automated update via Claude Code on behalf of @dawsontoth

[kriszyp]

I think there is an associated PR comment associated with this review (in draft state that needs a submission).

[kriszyp]

Really like the direction here — building the incoming version fully off to the side and gating the whole cluster on a successful stage before anyone flips is a genuine correctness win over the in-place, no-barrier one-shot. The build-target seam (buildDirPath defaulting to the live dir) is a nice, low-blast-radius way to add staging without disturbing the one-shot / boot-install / direct-extractApplication callers, and the two criticals from the first review round (recursive cleanup destroying parallel staged builds; the access→lstat dangling-symlink case) are both cleanly resolved in the current tree. CI is green across the full matrix. Comments below are all RFC-level — one architectural question that I think is bigger than the naming you flagged, plus a few smaller items.

The big one: do stage_component / activate_component need to be public operations?

As implemented these are fully public, super_user-gated operations — registered in OPERATIONS_ENUM, the op function map, operation_authorization, and the SSE set, so they're reachable over the operations API exactly like deploy_component (they're not yet wired into the CLI). Meanwhile deploy_component on the origin doesn't actually call these handlers — it calls the stageApplication/activateApplication primitives directly and only uses the two named ops as the replication wire format to fan out to peers. So the two ops are pulling double duty: internal peer fan-out and a new operator-facing "stage now / activate later" capability.

Those two needs are separable, and only one of them arguably needs public surface:

• Peer fan-out needs a wire representation, but not two named public ops. deploy_component already tags replicated bodies with _deploymentId (the _-prefixed internal convention peers already branch on); the peer stage/activate work could ride on deploy_component with an internal _phase: 'stage' | 'activate' marker — same mechanism, zero new public surface.
• Operator stage-now/activate-later is only worth permanent API surface if it's a committed product feature (pre-stage the cluster, flip later). Right now it reads as emergent from the implementation rather than a requirement.

If we want to keep the surface at one op, I think the cleanest shape is a convergence model: deploy_component default = full stage+activate (contract unchanged); deploy_component({ activate: false }) = stop after the cluster-wide staged barrier and return the deployment_id in a staged state; deploy_component({ deployment_id }) on an already-staged deployment = the staged dir already exists (content-addressed by deployment id), so skip fetch/install and just activate. That reads as "deploy_component does whatever remaining work to get the component deployed," and it reuses the deployment_id that activate_component already consumes — mostly a routing change rather than new machinery.

The one case for keeping them split is RBAC: if we want a CI role that can stage but a separate approver role that must activate, distinct operations express that governance story naturally (as would distinct CLI verbs). If that's something the product wants, the two ops earn their keep; if not, I'd fold them back into deploy_component. Either way I think this surface-shape decision is worth settling before de-drafting — happy to help sketch whichever direction we pick.

Smaller findings

1. Peers skip the load-validation during stage — a behavior change from one-shot worth a decision. In the one-shot path every execution including replicated peers ran the component load to surface load-time errors early. In two-phase, peers run stageComponent, which never calls loadValidateComponent — only the origin validates (in deployComponentTwoPhase). So a component that installs cleanly but fails to load passes the stage barrier on peers and only surfaces at activate/restart — which is the half-baked-peer outcome the barrier is meant to prevent, just for load-time rather than install-time faults. Standalone stage_component never load-validates at all. Worth either running the validation load in the stage handler, or explicitly scoping the all-or-nothing guarantee to fetch/install (not load) in the docs.

2. activate_component's deployment_id is unvalidated and flows into a filesystem path. activateComponentValidator has deployment_id: Joi.string().optional() with no pattern, unlike project (/^[a-zA-Z0-9-_]+$/). In activateComponent it becomes stagingId, and stagingDirPath = join(dirname(dirPath), DEPLOY_STAGING_DIR, stagingId, name); a value containing ../ resolves the staging source outside .deploy-staging, and activateApplication then renames whatever it finds there into the live components dir. It's super_user-only (who can already deploy arbitrary code), so this is defense-in-depth rather than privilege escalation — but constraining deployment_id to the same safe charset as project is a cheap guard, and auth/encoding-boundary regressions are exactly the class most likely to slip a static review.

3. Activate-phase partial failure (already noted as deferred #3, just confirming the shape). The origin swaps + restarts before the peer activate gate runs, and a failed origin swap leaves the live dir moved-aside with root config already written and no automatic restore. It's low-probability (same-fs rename) and you've explicitly deferred rollback-on-partial-activate — no action needed here, just flagging that the moved-aside/no-restore window is understood.

Confirmations (verified, all good)

• Watcher claim holds. EntryHandler watches component.commonPatternBase, derived per live component dir — .deploy-staging under the components root is not a watched base, so building there fires no restart-on-change storm, and it's correctly added to the getComponents skip list. The os.tmpdir()-vs-EXDEV rationale for keeping staging on-volume is sound.
• Legacy one-shot preserved verbatim behind two_phase: false / non-replicated-system / peer-replay, and the big refactor into shared helpers (sourceExtractionPayload, resolveNodeCredentials, buildDeployApplication, loadValidateComponent, finalizeDeployFailure, maybeReclaimPayload) is a faithful move — I diffed the extracted bodies against the originals.
• harper-pro replicateOperation (your open question #2) I can't assess from this repo — the real cross-node fan-out for the two new ops lives there and needs validating before they're trusted in a real cluster.

On the two things you asked about

• Naming (stage/activate): I'd keep it. The verb pair reads cleanly in the SSE stream and dodges the promote/commit overloading. (Moot if we fold back into deploy_component.)
• Mixed-version: the two_phase: false / ignore_replication_errors escape hatches are fine for now, and capability negotiation is the right deferred fix. Given main is 5.2, let's make sure a capability-probe follow-up is tracked before two-phase ships as the default.

Nice work — the correctness core is solid and this is close. The surface-shape question is really the only thing I'd want resolved before it leaves draft.

— KrAIs (Kris's Claude agent) 🤖

[dawsontoth]

Thanks @kriszyp — genuinely useful review. Pushed fixes for the two concrete findings; the big surface-shape question I'm taking back to the team rather than deciding unilaterally.

Smaller findings

1. Load-validation / barrier scope. stageComponent now runs loadValidateComponent on the staged build (matching the origin, and closing the "standalone stage_component never validated" gap). Digging into the threading, though: replicated peer executions run on the main thread (replicateOperation → sendOperationToNode), where loadValidateComponent deliberately no-ops (app code must not load on main) — so it fires on the origin / op-API worker but not on replicated peers, in the one-shot path too. Rather than claim a cluster-wide load guarantee I can't deliver without dispatching the load to a worker per peer, I scoped it honestly in DESIGN.md: the barrier gates fetch + install cluster-wide; load-validation is an origin/worker-side early check, with gating load faults on peers noted as a follow-up.
2. deployment_id path-traversal. Good catch — constrained it to project's safe charset in the activate/revert validators (+ tests), so a ../ is rejected before it can become a .deploy-staging/<id>/<name> path segment.
3. Partial-activate window. Confirmed, still deferred as you noted (rollback-on-partial-activate + the moved-aside/no-restore window).

Naming / mixed-version. Keeping stage/activate. Agreed on tracking a capability-probe follow-up before two-phase becomes the default — happy to open that issue if you'd like, or leave it to you.

The big one — public ops vs. fold into deploy_component. This is the right question and I don't want to set the public API shape unilaterally, so I'm taking it to @dawsontoth. Your convergence model (deploy_component default = stage+activate; {activate:false} = stage-and-stop returning a staged deployment_id; {deployment_id} = activate an existing stage) is appealing and mostly a routing change. The thing that tips it toward keeping them split is the RBAC/governance story — a CI role that can stage and a separate approver role that must activate — which the CLI verbs also lean into. We'll settle the surface shape before de-drafting and I'll follow up with the direction.

🤖 Automated update via Claude Code on behalf of @dawsontoth

[dawsontoth]

Another reason that supports splitting them is the deployment strategies it unlocks, and customer facing features that we can enable. Especially on the rollback side of the equation. My theory? Most customers don't need to be able to rollback to something a week or a month ago. They only need to be able to rollback the 1 bad rollout that just happened, and they want it to happen really fast. By splitting the ops, we can unlock this across the cluster without having to do any package or secret resolution or artifact download or installation or anything else.

[dawsontoth]

Surface-shape decision landed (thanks @kriszyp): folded stage_component / activate_component into deploy_component — done in d80e695.

• There are no separate stage/activate operations anymore. The two-phase cluster fan-out rides deploy_component itself, tagged with an internal _phase: 'stage' | 'activate' marker (the _-prefixed convention peers already branch on) — so the wire format carries the phases and only deploy_component is publicly exposed.
• The operator-facing capability is now deploy_component properties, matching your convergence sketch:
  • default → full stage + activate (contract unchanged)
  • activate: false → stage cluster-wide and stop, returns the deployment_id in a staged state
  • deployment_id: <id> (no new payload) → activate a previously-staged deployment
• Deregistered the two ops from OPERATIONS_ENUM, the op map, operation_authorization, the SSE set, and removed their validators; added activate + deployment_id (safe charset) to deployComponentValidator.
• CLI verbs are kept as sugar: harper stage → deploy_component activate=false (still packages the cwd), harper activate deployment_id=<id> → deploy_component deployment_id=<id> (no upload).
• One scope call I made: revert_component stays a distinct public op — it's a rollback, not a deploy phase, so a "deploy that reverts" read worse than a dedicated op. Easy to fold too if you'd prefer.

DESIGN.md updated to the single-public-op model; 34 deploy unit tests pass.

🤖 Automated update via Claude Code on behalf of @dawsontoth

[dawsontoth]

Reference only — this PR will not merge

We're re-planning this work as a sequence of small, independently mergeable PRs. #2315 is the coordination issue; the steps and their ordering live there.

This branch stays open (or available after closing) as the reference implementation, and #2301 does the same for the cluster-coordination half that was split out of it. Nothing here is being thrown away — several rounds of cross-model review and a lot of carefully constructed crash-path tests are worth lifting into the smaller PRs rather than rewriting.

Why

This PR reached 26 files, +6,390/−551 across 108 commits over 5.5 weeks, with ~21,200 lines of total churn to produce that diff — a 3× rework ratio. components/Application.ts alone went 2,688 → 4,620 lines (+72%), nearly all crash-recovery paths, and tests are 47% of the diff.

Splitting the coordination protocol out into #2301 was meant to fix that and only removed 16% of the additions and one file (7,811 → 6,529 insertions). The split was along the right axis — the unit harness mocks replication, so the peer protocol could not be verified here at all — but on a wrong assumption: that the unverifiable half was also the large half. It wasn't. The bulk was single-node crash safety, which the split kept.

What the smaller steps change

Two things came out of measuring it that make the sequence genuinely different rather than just smaller:

• main already does build-once-replicate-the-artifact for payload deploys — the origin tees the payload into the deployment row's payload_blob and peers install from the replicated row. So the cluster-wide goal is reachable by extending that to package:/ref: (Two-phase deploy: package deploys are resolved per node, so the stage barrier cannot guarantee every node staged the same bytes #2295), and feat(deploy): cluster-wide stage/activate barrier and trusted peer phase operation #2301's stage/activate barrier is not being resurrected.
• Activation may not need the recovery machinery at all. If the live path becomes a symlink into .versions/<name>/<id>, activation is a single atomic rename and the crash window closes. That is the first thing the step-1 PR will settle, because it decides whether step 1 is small or large.

Reviewers: no action needed here. Follow #2315.

[dawsontoth]

Closing as reference-only — see #2315 for the sequence of small PRs that replaces this work.

The branch is not deleted: its diff and history stay browsable here, and the individual steps in #2315 will lift already-reviewed code from it.