feat(deploy): two-phase stage/activate for deploy_component

DESIGN.md

@@ -193,6 +193,35 @@ A package-manager timeout must not release this lock while npm descendants are s
 
 Boot's `harper-application-lock.json` records an application configuration only after preparation fulfills. Recording at queue time would make a failed install look complete and suppress its retry on the next boot.
 
+## Component deploys build off to the side and swap atomically
+
+`deploy_component` prepares the candidate under `.deploy-staging/<deployment UUID>/<project>`, runs
+extract + `npm install` there, validates that it loads _where that check runs at all_ (it is a no-op on
+the main thread, which is where the operations API deploys — see "Staged deploy" below), and only then
+renames it into the live path —
+committing root config and `harper-application-lock.json` in the same compensating transaction. The
+live component keeps serving through the slow, failure-prone work (git clone, registry install), and
+go-live is one atomic rename. A fetch or install failure leaves the running component untouched
+instead of half-replaced in place.
+
+This is **per node**. The operation replicates as a whole, exactly as the single-phase deploy always
+did, and each node performs its own staged build. There is deliberately no cluster-wide barrier here:
+nothing orders activation across nodes, so two deploys originated concurrently on different nodes can
+still end with different versions live. That ordering guarantee — a leader or a monotonic activation
+epoch — is tracked in #2294, and byte-identical package resolution across nodes in #2295. Both are
+protocol additions that need real multi-node verification, so they are deliberately not attempted
+here.
+
+The deployment row carries the activation specification (package/install settings, routing, credential
+references, `force`). It is not a coordination channel between nodes; it exists so the deploy is
+observable, so the payload has a durable home, and — the load-bearing part — so startup
+reconciliation can reconcile root config for an activation that was interrupted mid-swap. A row
+without `activation_spec` cannot be recovered, which is why it is written before the build starts.
+
+Startup reconciliation is the recovery record: it settles interrupted extractions, finishes or undoes
+interrupted reverts, rolls an interrupted activation forward, and fails a component closed rather than
+loading a live tree whose durable configuration disagrees with it.
+
 ## Peer-side deploy_component payload read: retryable blob stalls and `Readable.from()` cancellation
 
 `readPayloadBlobWithRetry` (`components/deploymentRecorder.ts`) wraps the peer's read of a replicated `hdb_deployment` row's `payload_blob` so a transient 503 `BlobReadError` (`BLOB_UNAVAILABLE_STATUS`, `resources/blob.ts`) — content bytes not arriving within `blobReadTimeout`, e.g. a parked blob send on the origin — retries instead of failing the whole deploy. Two non-obvious constraints shaped the design:
@@ -475,6 +504,121 @@ this fix doesn't attempt to solve. `deploy_component`/`package_component` still
 declared entry points (`jsResource`/`graphqlSchema`) survived extraction — a truncation from some
 other future cause would still report success silently; that's a deferred, separate fix.
 
+## Staged deploy: build aside, then swap (`components/Application.ts`, `components/operations.js`)
+
+`deploy_component` builds the incoming version — download/`npm pack` (incl. a git clone), extract,
+`npm install` — into a hidden staging directory, then atomically renames it into the live component
+path.
+
+**The pre-swap load check does not run everywhere, and that is not new.** `loadValidateComponent`
+returns immediately on the main thread, and the operations API executes `deploy_component` there — so
+on a node whose deploy runs on the main thread, a candidate that installs cleanly but throws at load
+still reaches the live path. It runs where the deploy executes on a worker (e.g. the op-API worker).
+The staged build bounds the _install_, not the load. The live component keeps serving throughout, and a fetch or install
+failure leaves it untouched rather than half-replaced in place. The request/response contract is
+unchanged; only the SSE phase names differ (`stage`/`activate` vs the old `prepare`/`replicate`).
+
+Each node does this for itself: the operation replicates as a whole, as it always did. There is no
+cluster-wide barrier and no private peer operation — ordering activation across nodes (#2294) and
+guaranteeing every node staged the same bytes (#2295) are protocol additions that need real
+multi-node verification, and are deliberately out of scope here.
+
+**Why staging lives under the components root.** Go-live is `rename(stagingDir, liveDir)`, atomic only
+when both share a filesystem. `os.tmpdir()` is frequently a different mount → `EXDEV` → a slow
+recursive copy at exactly the moment an instant swap is wanted. So staging is a hidden directory under
+the components root: same volume, dot-prefixed so the loader ignores it, and not the watched base of
+any component's watcher, so building there fires no restart-on-change events.
+
+**Reversibility: retained previous + `revert_component`.** `activateStagedApplication` does not discard
+the tree its swap displaced — it retains it as `.deploy-previous/<name>`, evicting the older one so
+exactly one previous is kept per component (a bounded one extra copy). `revert_component` swaps the live
+directory with that retained previous via three same-filesystem renames through a hidden holding path,
+cluster-wide, and fetches nothing: no package resolution, no secret decryption, no artifact download, no
+install. That is the point — the rollback operators actually need is the bad rollout that just happened,
+and every node already has those bytes.
+
+**It is addressed, not toggled.** `to_deployment_id` is required and names the version the caller expects
+live afterwards. If that version is already live the call is a no-op success; if it matches the retained
+previous, the swap happens and the displaced tree becomes the new retained previous. Anything else is
+refused, naming what the component can actually revert to. A bare "swap to the other one" toggle is
+unsafe under ordinary request retries — a caller that loses the response and retries would flip the
+rejected release back in — which is why the target is mandatory. Reaching an older version is a redeploy,
+not a revert.
+
+**The swap carries persistent state with it.** Each retained tree has a sidecar manifest recording the
+deployment that produced it and the root-config entry it was activated with, and revert applies that
+entry to both the root config and `harper-application-lock.json`
+(`createApplicationConfigTransaction`, shared with activation). A null entry means REMOVE it: reverting
+away from a `package` deploy has to drop the package reference, or `installApplications()` would
+reinstall the reverted-away version over the restored directory on the next cold start and silently undo
+the rollback. A config-write failure compensates by swapping the directory back. The previous copy and
+its manifest are per-node (each node retains its own outgoing tree during its own activate), so a
+replicated revert has a local rollback source on every node.
+
+**One contract on `.deploy-aside`.** The directory has a single protocol, shared with the in-place
+extraction transaction: an `.in-progress-<ts>-<pid>-<uuid>` directory with no matching
+`.retired-<...>` marker is a ROLLBACK RECORD, and `recoverInterruptedComponentExtractions` restores the
+newest such directory OVER the live component path at startup. Anything else there is residue and is
+swept. A tree that is already known to be garbage when it is parked — the evicted two-deploys-ago
+retained-previous — therefore carries a distinct `.discarded-` prefix, because a crash between parking
+and sweeping would otherwise make startup recovery resurrect an ancient version over the current one.
+The three startup passes read disjoint directories — `.deploy-aside` for an interrupted in-place
+preparation, `.deploy-previous` for an interrupted revert, and `.deploy-staging`/`.deploy-activating`
+for an interrupted two-phase activation — and the two directory repairs run before the staged
+reconciliation so it decides roll-forward against a settled live directory.
+
+**An interrupted revert self-heals.** The three-way swap compensates in process (each rename is undone
+if a later one fails, so an exceptional I/O error leaves the component serving what it was serving),
+but compensation cannot cover the process dying between renames — and after the first rename the
+component has NO live directory. So the holding path is named `.reverting-<component>-<uuid>`, and
+`recoverInterruptedReverts` restores it at startup into whichever slot is empty: the live path when the
+swap had not yet placed the reverted-to version (the revert is undone and can be retried), or the
+retained-previous path when the swap completed and only the retain step was lost. With both slots
+occupied the swap finished and the holding tree is residue, so it is discarded.
+
+**Staged-build retention.** A successful deploy consumes its staged build immediately (activation
+renames it live), so nothing normally accumulates. What does accumulate is the residue of deploys that
+never reached activation — a failed install, a crash between stage and swap — each leaving
+`.deploy-staging/<deploymentId>/<name>` behind. `stageApplication` bounds this: after a successful
+stage it evicts the oldest not-yet-activated staged builds for that component beyond
+`deployment_stagingRetention_maxCount` (default 5, `pruneStagedBuilds`), always keeping the just-staged
+one and the newest N−1 by mtime. Eviction is best-effort (`allSettled`, trace-logged) but awaited so
+the count is settled when the stage returns. Retention is deliberately count-only and automatic:
+`hdb_deployment` rows stay as the audit trail (payload blobs already self-reclaim by size,
+`deployment_payloadRetention_maxSize`), and no `delete_deployment` op was added — eviction-on-stage
+keeps the surface at zero new operations. Nothing addresses a staged directory by id any more, so
+eviction has no user-visible consequence beyond reclaiming disk; activating a previously staged
+deployment moves to #2301 with the rest of the coordination protocol.
+
+**Payload retention.** Two independent bounds apply to the tarballs stored in `hdb_deployment`'s
+`payload_blob`, and they answer different questions:
+
+- `deployment_payloadRetention_maxSize` (default 10 MiB) — reclaims _this_ deploy's payload right after
+  it succeeds, if the tarball was large. Bounds the size of any single retained payload.
+- `deployment_payloadRetention_maxCount` (default **1**, `pruneProjectPayloads`) — keeps at most N
+  stored payloads _per project_, newest first, dropping the `payload_blob` of the rest after a
+  successful deploy. Bounds how many payloads accumulate over time, which is what actually caps disk.
+
+Only rows that still hold a payload count toward `maxCount`, so the cap reads literally as "at most N
+stored payloads per project." Rows are never deleted — pruning nulls the blob and nothing else, so the audit
+trail and `get_deployment` stay intact; only `get_deployment_payload` stops working for pruned
+deployments (`payload_blob_present: false`). Automatic pruning does NOT append `payload_dropped` to the
+rows it prunes: `event_log` is append-only and adding to it is a read-copy-write, which would lose a
+concurrent writer's entry, so `payload_blob_present: false` is what records the drop there. The
+deploying operation still emits `payload_dropped` on its own progress stream (and so into its own
+row's `event_log`), and the explicit `delete_deployment_payload` does append it to the row it targets —
+that one is a single operator-driven write with no concurrent writer to lose. A
+non-terminal deployment is counted but never dropped: its blob may still be the replication channel
+peers are installing from (the same guard `delete_deployment_payload` uses). Pruning is best-effort and
+off the deploy's critical path — a prune failure is logged, never fatal — and is skipped entirely when a
+peer failed, since the older payloads are the retry artifact in that case.
+
+The default of 1 is deliberately conservative rather than a redeploy-window convenience: instances on
+small quotas (free tier is 5 GB total) must not have N copies of a large app payload quietly competing
+with the customer's own data. Operators who want a wider redeploy-by-reference window raise it
+explicitly; 0 retains none. Note that an explicit `delete_deployment_payload` (harper#1893) forfeits
+redeployability for that deployment the same way an automatic prune does.
+
 ## RocksDB backup/restore: the restore lock + marker protocol (`dataLayer/restoreMarker.ts`, `dataLayer/rocksdbBackup.ts`)
 
 The `restore_backup` operation restores a user database on a live server by closing it across all

bin/cliOperations.ts

@@ -23,7 +23,30 @@ import { initConfig, getConfigPath } from '../config/configUtils.ts';
 // rather than being restated per caller (it also keeps `components/` off the CLI's import graph).
 import { deriveGitSecretName, directoryProjectName, normalizeGitHost } from '../utility/componentNames.ts';
 
-const OP_ALIASES = { deploy: 'deploy_component', package: 'package_component' };
+// Plain name aliases. `revert` is deliberately NOT here — it lives in OP_VERB_PROPS below so it can
+// carry the `_cliVerb` marker its missing-target guard keys on, and buildRequest checks OP_ALIASES
+// first, so an entry here would shadow that.
+const OP_ALIASES = {
+	deploy: 'deploy_component',
+	package: 'package_component',
+};
+
+const OP_VERB_PROPS: Record<string, Record<string, unknown>> = {
+	// `harper revert` uploads nothing: the version it activates is already on disk. `_cliVerb` is a
+	// CLI-internal marker (stripped before the request is sent) that drives the missing-target guard below.
+	revert: { operation: 'revert_component', _cliVerb: 'revert' },
+};
+
+// The operation has no notion of which verb invoked it, so requirements that belong to the verb are
+// enforced here. Returns an error message, or null when the request is fine.
+function verbRequirementError(req: any): string | null {
+	// revert_component requires its target so a retry can't toggle the rejected release back in. Caught
+	// here too, so the CLI names the flag instead of surfacing a raw validation error.
+	if (req._cliVerb === 'revert' && !req.to_deployment_id) {
+		return '`harper revert` requires the deployment you want live again — usage: harper revert project=<name> to_deployment_id=<id> (list_deployments reports the id)';
+	}
+	return null;
+}
 
 // Shown for any local-instance connection failure (missing pid, missing/stale domain
 // socket, or a refused/ENOENT connect against it) — they're all the same user-facing
@@ -36,7 +59,7 @@ const LOCAL_NOT_RUNNING_MESSAGE = 'Harper is not running. Use `harperdb run` (or
 // deploy completes. Add an operation here only after wiring its server-side
 // SSE_PROGRESS_OPERATIONS entry — otherwise the server returns the buffered JSON path and
 // the SSE parser sees no events.
-const SSE_OPERATIONS = new Set(['deploy_component']);
+const SSE_OPERATIONS = new Set(['deploy_component', 'revert_component']);
 
 // The fields that decide *where* an operation connects and *as whom* — see transportContext().
 const CONNECTION_FIELDS = ['target', 'auth_username', 'auth_password', 'rejectUnauthorized'];
@@ -304,6 +327,7 @@ export {
 	buildRequest,
 	redactCredentials,
 	refreshExpiredOperationToken,
+	verbRequirementError,
 	transportContext,
 	resolveGitTarget,
 	resolveCredentialHost,
@@ -570,7 +594,14 @@ export function prepareDeployByRef(req: any): void {
 	process.stderr.write(`Deploying "${req.project}" by reference: ${req.package}\n`);
 }
 
+// `harper revert` uploads nothing, so it has no packaging step — but it still needs the CWD project
+// default every other deploy-family verb gets, or it fails server-side on a missing `project`.
+const prepareRevert = async (req) => {
+	req.project ||= directoryProjectName(process.cwd());
+};
+
 const PREPARE_OPERATION: any = {
+	revert_component: prepareRevert,
 	deploy_component: async (req) => {
 		if (req.package) {
 			return;
@@ -619,6 +650,9 @@ function buildRequest(): any {
 	for (const arg of process.argv.slice(2)) {
 		if (OP_ALIASES.hasOwnProperty(arg)) {
 			req.operation = OP_ALIASES[arg];
+		} else if (OP_VERB_PROPS.hasOwnProperty(arg)) {
+			// Sugar verb (stage/activate) → deploy_component + preset props (e.g. activate:false).
+			Object.assign(req, OP_VERB_PROPS[arg]);
 		} else if (arg.includes('=')) {
 			let [first, ...rest] = arg.split('=');
 			let restStr: any = rest.join('=');
@@ -871,13 +905,22 @@ export async function resolveRequestOptions(req: any): Promise<{ options: any; t
 async function cliOperations(req: any, skipResponseLog = false) {
 	require('dotenv').config();
 
+	// Enforce CLI-verb requirements (e.g. `harper revert` needs a to_deployment_id) before connecting or
+	// packaging, so a mistake fails fast instead of building + uploading a fresh deploy from the CWD.
+	const verbError = verbRequirementError(req);
+	if (verbError) {
+		console.error(verbError);
+		process.exit(1);
+	}
+
 	// Resolve target/auth inside the try so a credential or connection error (e.g. an incomplete
 	// `auth_username=`/`auth_password=` pair, which resolveRequestOptions throws on) is mapped to the
 	// same console.error + process.exit(1) as every other failure below, rather than escaping as an
 	// unhandled rejection. `target` is declared out here so the catch can still reference it.
 	let options: any, target: any;
 	try {
 		({ options, target } = await resolveRequestOptions(req));
+		delete req._cliVerb;
 		await PREPARE_OPERATION[req.operation]?.(req);
 		// Streaming deploy (multipart upload + SSE progress) only works against >= 5.1 servers.
 		// When deploying to a remote target, probe its version first and downgrade to the
@@ -913,7 +956,7 @@ async function cliOperations(req: any, skipResponseLog = false) {
 		// One renderer owns the (future) upload bar and the SSE event rendering for a
 		// multipart deploy. Created here so the upload-stream tap and the SSE consumer
 		// below share the same instance.
-		const renderer = req._multipart ? new DeployRenderer({ uploadTotal: req._uploadSizeEstimate ?? 0 }) : null;
+		const renderer = useSse ? new DeployRenderer({ uploadTotal: req._uploadSizeEstimate ?? 0 }) : null;
 		let body;
 		if (req._multipart) {
 			// Create the package stream here — after the renderer exists — so we can pass