diff --git a/DESIGN.md b/DESIGN.md index f30892f91d..97f3344c98 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -193,6 +193,35 @@ A package-manager timeout must not release this lock while npm descendants are s Boot's `harper-application-lock.json` records an application configuration only after preparation fulfills. Recording at queue time would make a failed install look complete and suppress its retry on the next boot. +## Component deploys build off to the side and swap atomically + +`deploy_component` prepares the candidate under `.deploy-staging//`, runs +extract + `npm install` there, validates that it loads _where that check runs at all_ (it is a no-op on +the main thread, which is where the operations API deploys — see "Staged deploy" below), and only then +renames it into the live path — +committing root config and `harper-application-lock.json` in the same compensating transaction. The +live component keeps serving through the slow, failure-prone work (git clone, registry install), and +go-live is one atomic rename. A fetch or install failure leaves the running component untouched +instead of half-replaced in place. + +This is **per node**. The operation replicates as a whole, exactly as the single-phase deploy always +did, and each node performs its own staged build. There is deliberately no cluster-wide barrier here: +nothing orders activation across nodes, so two deploys originated concurrently on different nodes can +still end with different versions live. That ordering guarantee — a leader or a monotonic activation +epoch — is tracked in #2294, and byte-identical package resolution across nodes in #2295. Both are +protocol additions that need real multi-node verification, so they are deliberately not attempted +here. + +The deployment row carries the activation specification (package/install settings, routing, credential +references, `force`). It is not a coordination channel between nodes; it exists so the deploy is +observable, so the payload has a durable home, and — the load-bearing part — so startup +reconciliation can reconcile root config for an activation that was interrupted mid-swap. A row +without `activation_spec` cannot be recovered, which is why it is written before the build starts. + +Startup reconciliation is the recovery record: it settles interrupted extractions, finishes or undoes +interrupted reverts, rolls an interrupted activation forward, and fails a component closed rather than +loading a live tree whose durable configuration disagrees with it. + ## Peer-side deploy_component payload read: retryable blob stalls and `Readable.from()` cancellation `readPayloadBlobWithRetry` (`components/deploymentRecorder.ts`) wraps the peer's read of a replicated `hdb_deployment` row's `payload_blob` so a transient 503 `BlobReadError` (`BLOB_UNAVAILABLE_STATUS`, `resources/blob.ts`) — content bytes not arriving within `blobReadTimeout`, e.g. a parked blob send on the origin — retries instead of failing the whole deploy. Two non-obvious constraints shaped the design: @@ -475,6 +504,121 @@ this fix doesn't attempt to solve. `deploy_component`/`package_component` still declared entry points (`jsResource`/`graphqlSchema`) survived extraction — a truncation from some other future cause would still report success silently; that's a deferred, separate fix. +## Staged deploy: build aside, then swap (`components/Application.ts`, `components/operations.js`) + +`deploy_component` builds the incoming version — download/`npm pack` (incl. a git clone), extract, +`npm install` — into a hidden staging directory, then atomically renames it into the live component +path. + +**The pre-swap load check does not run everywhere, and that is not new.** `loadValidateComponent` +returns immediately on the main thread, and the operations API executes `deploy_component` there — so +on a node whose deploy runs on the main thread, a candidate that installs cleanly but throws at load +still reaches the live path. It runs where the deploy executes on a worker (e.g. the op-API worker). +The staged build bounds the _install_, not the load. The live component keeps serving throughout, and a fetch or install +failure leaves it untouched rather than half-replaced in place. The request/response contract is +unchanged; only the SSE phase names differ (`stage`/`activate` vs the old `prepare`/`replicate`). + +Each node does this for itself: the operation replicates as a whole, as it always did. There is no +cluster-wide barrier and no private peer operation — ordering activation across nodes (#2294) and +guaranteeing every node staged the same bytes (#2295) are protocol additions that need real +multi-node verification, and are deliberately out of scope here. + +**Why staging lives under the components root.** Go-live is `rename(stagingDir, liveDir)`, atomic only +when both share a filesystem. `os.tmpdir()` is frequently a different mount → `EXDEV` → a slow +recursive copy at exactly the moment an instant swap is wanted. So staging is a hidden directory under +the components root: same volume, dot-prefixed so the loader ignores it, and not the watched base of +any component's watcher, so building there fires no restart-on-change events. + +**Reversibility: retained previous + `revert_component`.** `activateStagedApplication` does not discard +the tree its swap displaced — it retains it as `.deploy-previous/`, evicting the older one so +exactly one previous is kept per component (a bounded one extra copy). `revert_component` swaps the live +directory with that retained previous via three same-filesystem renames through a hidden holding path, +cluster-wide, and fetches nothing: no package resolution, no secret decryption, no artifact download, no +install. That is the point — the rollback operators actually need is the bad rollout that just happened, +and every node already has those bytes. + +**It is addressed, not toggled.** `to_deployment_id` is required and names the version the caller expects +live afterwards. If that version is already live the call is a no-op success; if it matches the retained +previous, the swap happens and the displaced tree becomes the new retained previous. Anything else is +refused, naming what the component can actually revert to. A bare "swap to the other one" toggle is +unsafe under ordinary request retries — a caller that loses the response and retries would flip the +rejected release back in — which is why the target is mandatory. Reaching an older version is a redeploy, +not a revert. + +**The swap carries persistent state with it.** Each retained tree has a sidecar manifest recording the +deployment that produced it and the root-config entry it was activated with, and revert applies that +entry to both the root config and `harper-application-lock.json` +(`createApplicationConfigTransaction`, shared with activation). A null entry means REMOVE it: reverting +away from a `package` deploy has to drop the package reference, or `installApplications()` would +reinstall the reverted-away version over the restored directory on the next cold start and silently undo +the rollback. A config-write failure compensates by swapping the directory back. The previous copy and +its manifest are per-node (each node retains its own outgoing tree during its own activate), so a +replicated revert has a local rollback source on every node. + +**One contract on `.deploy-aside`.** The directory has a single protocol, shared with the in-place +extraction transaction: an `.in-progress---` directory with no matching +`.retired-<...>` marker is a ROLLBACK RECORD, and `recoverInterruptedComponentExtractions` restores the +newest such directory OVER the live component path at startup. Anything else there is residue and is +swept. A tree that is already known to be garbage when it is parked — the evicted two-deploys-ago +retained-previous — therefore carries a distinct `.discarded-` prefix, because a crash between parking +and sweeping would otherwise make startup recovery resurrect an ancient version over the current one. +The three startup passes read disjoint directories — `.deploy-aside` for an interrupted in-place +preparation, `.deploy-previous` for an interrupted revert, and `.deploy-staging`/`.deploy-activating` +for an interrupted two-phase activation — and the two directory repairs run before the staged +reconciliation so it decides roll-forward against a settled live directory. + +**An interrupted revert self-heals.** The three-way swap compensates in process (each rename is undone +if a later one fails, so an exceptional I/O error leaves the component serving what it was serving), +but compensation cannot cover the process dying between renames — and after the first rename the +component has NO live directory. So the holding path is named `.reverting--`, and +`recoverInterruptedReverts` restores it at startup into whichever slot is empty: the live path when the +swap had not yet placed the reverted-to version (the revert is undone and can be retried), or the +retained-previous path when the swap completed and only the retain step was lost. With both slots +occupied the swap finished and the holding tree is residue, so it is discarded. + +**Staged-build retention.** A successful deploy consumes its staged build immediately (activation +renames it live), so nothing normally accumulates. What does accumulate is the residue of deploys that +never reached activation — a failed install, a crash between stage and swap — each leaving +`.deploy-staging//` behind. `stageApplication` bounds this: after a successful +stage it evicts the oldest not-yet-activated staged builds for that component beyond +`deployment_stagingRetention_maxCount` (default 5, `pruneStagedBuilds`), always keeping the just-staged +one and the newest N−1 by mtime. Eviction is best-effort (`allSettled`, trace-logged) but awaited so +the count is settled when the stage returns. Retention is deliberately count-only and automatic: +`hdb_deployment` rows stay as the audit trail (payload blobs already self-reclaim by size, +`deployment_payloadRetention_maxSize`), and no `delete_deployment` op was added — eviction-on-stage +keeps the surface at zero new operations. Nothing addresses a staged directory by id any more, so +eviction has no user-visible consequence beyond reclaiming disk; activating a previously staged +deployment moves to #2301 with the rest of the coordination protocol. + +**Payload retention.** Two independent bounds apply to the tarballs stored in `hdb_deployment`'s +`payload_blob`, and they answer different questions: + +- `deployment_payloadRetention_maxSize` (default 10 MiB) — reclaims _this_ deploy's payload right after + it succeeds, if the tarball was large. Bounds the size of any single retained payload. +- `deployment_payloadRetention_maxCount` (default **1**, `pruneProjectPayloads`) — keeps at most N + stored payloads _per project_, newest first, dropping the `payload_blob` of the rest after a + successful deploy. Bounds how many payloads accumulate over time, which is what actually caps disk. + +Only rows that still hold a payload count toward `maxCount`, so the cap reads literally as "at most N +stored payloads per project." Rows are never deleted — pruning nulls the blob and nothing else, so the audit +trail and `get_deployment` stay intact; only `get_deployment_payload` stops working for pruned +deployments (`payload_blob_present: false`). Automatic pruning does NOT append `payload_dropped` to the +rows it prunes: `event_log` is append-only and adding to it is a read-copy-write, which would lose a +concurrent writer's entry, so `payload_blob_present: false` is what records the drop there. The +deploying operation still emits `payload_dropped` on its own progress stream (and so into its own +row's `event_log`), and the explicit `delete_deployment_payload` does append it to the row it targets — +that one is a single operator-driven write with no concurrent writer to lose. A +non-terminal deployment is counted but never dropped: its blob may still be the replication channel +peers are installing from (the same guard `delete_deployment_payload` uses). Pruning is best-effort and +off the deploy's critical path — a prune failure is logged, never fatal — and is skipped entirely when a +peer failed, since the older payloads are the retry artifact in that case. + +The default of 1 is deliberately conservative rather than a redeploy-window convenience: instances on +small quotas (free tier is 5 GB total) must not have N copies of a large app payload quietly competing +with the customer's own data. Operators who want a wider redeploy-by-reference window raise it +explicitly; 0 retains none. Note that an explicit `delete_deployment_payload` (harper#1893) forfeits +redeployability for that deployment the same way an automatic prune does. + ## RocksDB backup/restore: the restore lock + marker protocol (`dataLayer/restoreMarker.ts`, `dataLayer/rocksdbBackup.ts`) The `restore_backup` operation restores a user database on a live server by closing it across all diff --git a/bin/cliOperations.ts b/bin/cliOperations.ts index 438d00694d..24fe77d268 100644 --- a/bin/cliOperations.ts +++ b/bin/cliOperations.ts @@ -23,7 +23,30 @@ import { initConfig, getConfigPath } from '../config/configUtils.ts'; // rather than being restated per caller (it also keeps `components/` off the CLI's import graph). import { deriveGitSecretName, directoryProjectName, normalizeGitHost } from '../utility/componentNames.ts'; -const OP_ALIASES = { deploy: 'deploy_component', package: 'package_component' }; +// Plain name aliases. `revert` is deliberately NOT here — it lives in OP_VERB_PROPS below so it can +// carry the `_cliVerb` marker its missing-target guard keys on, and buildRequest checks OP_ALIASES +// first, so an entry here would shadow that. +const OP_ALIASES = { + deploy: 'deploy_component', + package: 'package_component', +}; + +const OP_VERB_PROPS: Record> = { + // `harper revert` uploads nothing: the version it activates is already on disk. `_cliVerb` is a + // CLI-internal marker (stripped before the request is sent) that drives the missing-target guard below. + revert: { operation: 'revert_component', _cliVerb: 'revert' }, +}; + +// The operation has no notion of which verb invoked it, so requirements that belong to the verb are +// enforced here. Returns an error message, or null when the request is fine. +function verbRequirementError(req: any): string | null { + // revert_component requires its target so a retry can't toggle the rejected release back in. Caught + // here too, so the CLI names the flag instead of surfacing a raw validation error. + if (req._cliVerb === 'revert' && !req.to_deployment_id) { + return '`harper revert` requires the deployment you want live again — usage: harper revert project= to_deployment_id= (list_deployments reports the id)'; + } + return null; +} // Shown for any local-instance connection failure (missing pid, missing/stale domain // socket, or a refused/ENOENT connect against it) — they're all the same user-facing @@ -36,7 +59,7 @@ const LOCAL_NOT_RUNNING_MESSAGE = 'Harper is not running. Use `harperdb run` (or // deploy completes. Add an operation here only after wiring its server-side // SSE_PROGRESS_OPERATIONS entry — otherwise the server returns the buffered JSON path and // the SSE parser sees no events. -const SSE_OPERATIONS = new Set(['deploy_component']); +const SSE_OPERATIONS = new Set(['deploy_component', 'revert_component']); // The fields that decide *where* an operation connects and *as whom* — see transportContext(). const CONNECTION_FIELDS = ['target', 'auth_username', 'auth_password', 'rejectUnauthorized']; @@ -304,6 +327,7 @@ export { buildRequest, redactCredentials, refreshExpiredOperationToken, + verbRequirementError, transportContext, resolveGitTarget, resolveCredentialHost, @@ -570,7 +594,14 @@ export function prepareDeployByRef(req: any): void { process.stderr.write(`Deploying "${req.project}" by reference: ${req.package}\n`); } +// `harper revert` uploads nothing, so it has no packaging step — but it still needs the CWD project +// default every other deploy-family verb gets, or it fails server-side on a missing `project`. +const prepareRevert = async (req) => { + req.project ||= directoryProjectName(process.cwd()); +}; + const PREPARE_OPERATION: any = { + revert_component: prepareRevert, deploy_component: async (req) => { if (req.package) { return; @@ -619,6 +650,9 @@ function buildRequest(): any { for (const arg of process.argv.slice(2)) { if (OP_ALIASES.hasOwnProperty(arg)) { req.operation = OP_ALIASES[arg]; + } else if (OP_VERB_PROPS.hasOwnProperty(arg)) { + // Sugar verb (stage/activate) → deploy_component + preset props (e.g. activate:false). + Object.assign(req, OP_VERB_PROPS[arg]); } else if (arg.includes('=')) { let [first, ...rest] = arg.split('='); let restStr: any = rest.join('='); @@ -871,6 +905,14 @@ export async function resolveRequestOptions(req: any): Promise<{ options: any; t async function cliOperations(req: any, skipResponseLog = false) { require('dotenv').config(); + // Enforce CLI-verb requirements (e.g. `harper revert` needs a to_deployment_id) before connecting or + // packaging, so a mistake fails fast instead of building + uploading a fresh deploy from the CWD. + const verbError = verbRequirementError(req); + if (verbError) { + console.error(verbError); + process.exit(1); + } + // Resolve target/auth inside the try so a credential or connection error (e.g. an incomplete // `auth_username=`/`auth_password=` pair, which resolveRequestOptions throws on) is mapped to the // same console.error + process.exit(1) as every other failure below, rather than escaping as an @@ -878,6 +920,7 @@ async function cliOperations(req: any, skipResponseLog = false) { let options: any, target: any; try { ({ options, target } = await resolveRequestOptions(req)); + delete req._cliVerb; await PREPARE_OPERATION[req.operation]?.(req); // Streaming deploy (multipart upload + SSE progress) only works against >= 5.1 servers. // When deploying to a remote target, probe its version first and downgrade to the @@ -913,7 +956,7 @@ async function cliOperations(req: any, skipResponseLog = false) { // One renderer owns the (future) upload bar and the SSE event rendering for a // multipart deploy. Created here so the upload-stream tap and the SSE consumer // below share the same instance. - const renderer = req._multipart ? new DeployRenderer({ uploadTotal: req._uploadSizeEstimate ?? 0 }) : null; + const renderer = useSse ? new DeployRenderer({ uploadTotal: req._uploadSizeEstimate ?? 0 }) : null; let body; if (req._multipart) { // Create the package stream here — after the renderer exists — so we can pass diff --git a/components/Application.ts b/components/Application.ts index 8a483fe643..0d17561b4b 100644 --- a/components/Application.ts +++ b/components/Application.ts @@ -1,6 +1,15 @@ import { type Logger } from '../utility/logging/logger.ts'; -import { getConfigObj, getConfigValue, getConfigPath } from '../config/configUtils.ts'; +import { + addConfig, + deleteConfigFromFile, + getConfigObj, + getConfigValue, + getConfigPath, + readConfigFile, +} from '../config/configUtils.ts'; import { CONFIG_PARAMS } from '../utility/hdbTerms.ts'; +import { ClientError } from '../utility/errors/hdbError.ts'; +import { HTTP_STATUS_CODES } from '../utility/errors/commonErrors.ts'; import logger, { errorForLog } from '../utility/logging/harper_logger.ts'; import { broadcastDeployStart, broadcastDeployEnd } from './deployLifecycle.ts'; import { ComponentPreparationLockTimeoutError, withComponentPreparationLock } from './componentPreparationLock.ts'; @@ -15,7 +24,7 @@ import { import { getSecretDecryptor } from '../resources/secretDecryptor.ts'; import { ENV_ENCRYPTED_PREFIX } from '../utility/envFile.ts'; -import { basename, dirname, extname, join } from 'node:path'; +import { basename, dirname, extname, isAbsolute, join, relative } from 'node:path'; import { access, chmod, @@ -25,9 +34,10 @@ import { mkdtemp, readdir, readFile, + readlink, rename, - rmdir, rm, + rmdir, stat, symlink, writeFile, @@ -477,10 +487,33 @@ async function runNpmPack( } // Hidden directory under the components root holding component versions renamed aside -// during a deploy swap (see extractApplication). The leading dot keeps +// during a deploy swap (see activateStagedApplication). The leading dot keeps // loadComponentDirectories from loading its contents as components. export const ASIDE_STAGING_DIR = '.deploy-aside'; const IN_PROGRESS_ASIDE_PREFIX = '.in-progress-'; +// Parked and known-disposable at park time (an evicted two-deploys-ago retained-previous). NEVER a +// recovery candidate — see discardDirAside for why the distinction has to be in the name. +export const DISCARDED_ASIDE_PREFIX = '.discarded-'; +// The holding path a revert's three-way swap parks the outgoing live tree in. Recoverable: if the +// process dies mid-swap, recoverInterruptedReverts puts it back. See revertApplication. +const REVERTING_PREFIX = '.reverting-'; +// Records the manifest state a revert recovery is working toward. Written before recovery mutates +// anything and removed only once the directory, manifest and config are all durable, so a recovery +// interrupted part-way is resumable rather than losing its own evidence. See recoverInterruptedReverts. +const REVERT_RECOVERY_SUFFIX = '.recovering.json'; +// Distinguishes the no-live restore branch's marker from a swap's holding-bound one; there is no +// holding directory in that branch, so the marker needs its own stable name. +const RESTORE_MARKER_INFIX = '.restoring'; + +// The revert-intent marker is named after the holding directory it describes. A fixed per-component +// path would be reused by every future revert of that component, so an orphan left by one attempt +// could be trusted by a later unrelated one. +function revertRecoveryMarkerFor(holdingPath: string): string { + return `${holdingPath}${REVERT_RECOVERY_SUFFIX}`; +} +// Splits `-` off a revert holding directory name, anchored on the UUID so a +// component name containing dashes is preserved intact. +const REVERTING_NAME_PATTERN = /^(.+)-[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; const RETIRED_ASIDE_PREFIX = '.retired-'; const PRIOR_ABSENT_RECORD_SUFFIX = '-prior-absent'; const DEFAULT_COMMAND_TIMEOUT_MS = 60 * 60 * 1000; @@ -491,6 +524,825 @@ const COMPONENT_RECOVERY_LOCK_PURPOSE = 'component-recovery'; const MAX_GIT_EXTRACTION_COMMANDS = 4; const MAX_INSTALL_COMMANDS = 2; +// Hidden directory under the components root where the INCOMING version of a component is +// fully built (extracted + `npm install`) before it goes live — the counterpart to +// ASIDE_STAGING_DIR, which holds the OUTGOING version. Two-phase deploy stages here first +// (the stage phase), then activateStagedApplication renames the staged copy into the live +// component path in one atomic step (the activate phase). +// +// It lives UNDER the components root on purpose, even though the bytes are "temporary": +// - Same filesystem as the live path, so the go-live rename() is atomic. An os.tmpdir() +// location is frequently a different mount (tmpfs / separate volume); a cross-device +// rename throws EXDEV and degrades to a slow recursive copy — reintroducing the very +// downtime window the staged build exists to remove. +// - The leading dot keeps loadComponentDirectories (componentLoader) from loading it as a +// phantom component, and it is not the watched base of any component's file watcher +// (those are rooted at each live component dir), so building here triggers no +// restart-on-change storm and needs no deploy:start watcher suppression. +export const DEPLOY_STAGING_DIR = '.deploy-staging'; +export const DEPLOY_ACTIVATION_DIR = '.deploy-activating'; +const STAGED_COMPLETE_MARKER = '.complete'; +const ACTIVATION_BACKUP_PREFIX = '.previous-'; +const ACTIVATION_NEW_PREFIX = '.new-'; +const DEPLOYMENT_ID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +// Hidden directory under the components root that RETAINS the immediately-previous live version of a +// component (`.deploy-previous/`) after an activate swap, so it can be swapped back by +// revert_component. Exactly one previous version is kept per component — each activate evicts the +// older one — so the retention cost is bounded at one extra copy. Same filesystem as the live path +// (atomic swap on revert), leading-dot-hidden (loader/watchers ignore it). This is what turns the +// deploy swap into something reversible: a customer can activate, run their own health checks, and +// revert if unhappy; and a partially-failed activate can be swapped back cluster-wide. +export const DEPLOY_PREVIOUS_DIR = '.deploy-previous'; + +// Max not-yet-activated staged builds kept per component before the oldest are evicted on the next +// stage. A full deploy consumes its staged build immediately (activate renames it live), so this only +// bounds the residue of stages that never reached activation. Configurable via +// deployment_stagingRetention_maxCount. +export const DEFAULT_STAGING_RETENTION_MAX_COUNT = 5; + +export function getStagingRetentionMaxCount(): number { + const configured = getConfigValue(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT); + // Only a number or numeric string is a valid count; reject everything else (unset, boolean, array, + // blank) and fall back to the default — mirroring getPayloadRetentionMaxSize's defensive coercion. + if (typeof configured !== 'number' && typeof configured !== 'string') return DEFAULT_STAGING_RETENTION_MAX_COUNT; + if (typeof configured === 'string' && configured.trim() === '') return DEFAULT_STAGING_RETENTION_MAX_COUNT; + const parsed = Number(configured); + return Number.isFinite(parsed) && parsed >= 1 ? Math.floor(parsed) : DEFAULT_STAGING_RETENTION_MAX_COUNT; +} + +/** + * Prune the oldest not-yet-activated staged builds for a component, keeping at most `maxCount` (newest + * by mtime) and ALWAYS retaining the just-built one (`keepStagingId`). Staged builds live at + * `.deploy-staging//`, and each stagingId parent holds exactly one component's build, + * so an evicted build's whole parent directory is removed. Entirely best-effort: any error (a racing + * concurrent stage, a busy dir) is logged at trace and never fails the stage that triggered it. + */ +async function pruneStagedBuilds(componentName: string, keepStagingId: string, maxCount: number): Promise { + try { + if (!(maxCount >= 1)) return; + const componentsRoot = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); + if (!componentsRoot) return; + const stagingRoot = join(componentsRoot, DEPLOY_STAGING_DIR); + let parents: import('node:fs').Dirent[]; + try { + parents = await readdir(stagingRoot, { withFileTypes: true }); + } catch (err) { + if ((err as any).code === 'ENOENT') return; // nothing staged yet + throw err; + } + // Under the component lock, so two stages of the same component cannot each enumerate before the + // other evicts and race their decisions. A lock we cannot get lands in the catch below, which is + // right for a disk bound: skipping a prune costs space, not correctness. + await withComponentPreparationLock(join(componentsRoot, componentName), async () => { + // Collect this component's staged builds: // that still exist. + const builds: Array<{ stagingId: string; parentPath: string; mtime: number }> = []; + for (const parent of parents) { + if (!parent.isDirectory()) continue; + const parentPath = join(stagingRoot, parent.name); + try { + const st = await stat(join(parentPath, componentName)); + builds.push({ stagingId: parent.name, parentPath, mtime: st.mtimeMs }); + } catch (err) { + if ((err as any).code !== 'ENOENT') throw err; // parent holds a different component; skip + } + } + // Keep the build just made, plus anything strictly NEWER than it, plus the newest of the rest up + // to the count. Filling from "the others" alone would privilege the current build + // unconditionally: a stage delayed behind slow peers finishes with an older mtime than a + // sibling that started later and already returned, and that newer sibling's tree would be + // evicted. Only *strictly* newer builds are protected here, unlike the row side, which protects + // ties too. mtime granularity ties every build staged in the same tick, so protecting ties on + // disk would stop this bound converging at all — four rapid stages under a bound of two would + // keep all four. The residual: two stages tying to the millisecond can have the second evict + // the first's tree while its row stays `staged`, so activating that id later fails with "no + // valid component tree" rather than serving nothing. A clear failure, and the reconcile pass + // settles such a row. The same applies above `maxCount` genuinely-concurrent deploys of ONE + // component: builds serialize on the component lock but validation runs after it is released, + // so the oldest queued candidate can be evicted before it activates. Ownership is deliberately + // not tracked to close that: a registry entry leaked by a deploy that dies between stage and + // activate would pin a tree forever, defeating the disk bound this exists to enforce, which is + // worse than a loud failure on the 6th simultaneous deploy of the same component. + // Sorting breaks ties by stagingId so concurrent prunes choose the same + // victims instead of each deleting the other's. + const current = builds.find((build) => build.stagingId === keepStagingId); + const others = builds + .filter((build) => build.stagingId !== keepStagingId) + .sort((a, b) => b.mtime - a.mtime || a.stagingId.localeCompare(b.stagingId)); + const protectedBuilds = current ? others.filter((build) => build.mtime > current.mtime) : []; + const budget = Math.max(0, maxCount - (current ? 1 : 0) - protectedBuilds.length); + // Awaited (best-effort via allSettled) so the retention count is settled by the time the stage + // returns. + const evictions = others + .filter((build) => !protectedBuilds.includes(build)) + .slice(budget) + .map((build) => + rm(build.parentPath, { recursive: true, force: true, maxRetries: 3, retryDelay: 100 }).catch((err) => + logger.trace?.(`Deferred prune of staged ${componentName} build ${build.stagingId}: ${err.message}`) + ) + ); + await Promise.allSettled(evictions); + }); + } catch (err) { + logger.trace?.(`Staged-build prune for ${componentName} skipped: ${(err as Error).message}`); + } +} + +/** + * Park `targetDirPath` in this component's `.deploy-aside` directory under a name that marks it + * KNOWN-DISPOSABLE at the moment it is parked, and sweep it best-effort. Returns false when there + * was nothing there to park. + * + * Renaming aside — instead of clearing in place — is immune to the race where a still-running worker + * keeps writing into the directory (e.g. a live Next.js app writing into `.next/cache`): an in-place + * recursive rm races that writer and fails with ENOTEMPTY, whereas the rename is atomic and the old + * worker harmlessly keeps writing into the renamed inode until it exits on restart. + * + * The `.discarded-` prefix is load-bearing, and is the whole reason this exists alongside the + * extraction transaction's `.in-progress-` asides. `.deploy-aside` has exactly ONE contract: an + * `.in-progress-` directory with no matching `.retired-` marker is + * a ROLLBACK RECORD, and `recoverInterruptedComponentExtractions` restores the newest such directory + * OVER the live component path at startup. A tree that is already known to be garbage when it is + * parked — the evicted two-deploys-ago retained-previous below — must therefore never carry that + * prefix, or a crash between parking and sweeping would make startup recovery resurrect an ancient + * version over the current one. `.discarded-` says "never restore this", and cleanupExtractionPaths + * already removes anything that is not an unretired `.in-progress-`. + */ +export async function discardDirAside( + targetDirPath: string, + componentName: string, + asideStagingDir: string = extractionStagingDirectory(targetDirPath) +): Promise { + try { + // lstat, not access(F_OK): access follows symlinks, so a DANGLING symlink at the path (left by a + // prior `file:`-directory deploy whose target was removed) would report ENOENT and be skipped + // here — then a later mkdir fails EEXIST because the dead link still occupies the path. lstat + // sees the link itself, so we park it like any other occupant. + await lstat(targetDirPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return false; // nothing there to park + throw err; + } + await ensureExtractionStagingDirectory(asideStagingDir); + const discardedPath = join(asideStagingDir, `${DISCARDED_ASIDE_PREFIX}${Date.now()}-${process.pid}-${randomUUID()}`); + await rename(targetDirPath, discardedPath); + // Best-effort: the outgoing worker may still hold files open in the renamed copy, so a failure here + // is expected in that case and swept by the next deploy or the next startup recovery pass. + void cleanupExtractionPaths( + { name: componentName, dirPath: targetDirPath, logger }, + asideStagingDir, + new Set([discardedPath]) + ).catch((err) => logger.trace?.(`Deferred cleanup of discarded ${componentName} directory: ${errorMessage(err)}`)); + return true; +} + +/** + * The retained-previous manifest for a component: which deployment produced the tree now sitting in + * `.deploy-previous/`, which produced the tree now LIVE, and the root-config entry each one was + * activated with. + * + * This is what makes `revert_component` addressable rather than a blind toggle: the caller names the + * deployment it expects to end up live, so a retried request whose + * response was lost is a no-op instead of flipping the rejected release back in. It is also what lets + * a revert restore persistent state, not just the directory: `application_config` is the root-config + * entry (and install-lock entry) that belongs with each tree, so reverting away from a `package` + * deploy removes that package reference instead of leaving `installApplications()` free to reinstall + * the reverted-away version over the restored directory on the next cold start. + * + * `application_config: null` means "that version had no root-config entry" (a payload deploy) and is + * therefore an instruction to DELETE the entry on revert, not to leave it alone. + */ +type RetainedVersion = { + deployment_id: string | null; + application_config: ApplicationConfig | null; +}; + +type RetainedPreviousManifest = { + previous: RetainedVersion; + live: RetainedVersion; + // Revert recovery markers only, which share this shape: set once the revert's persistent-state commit + // has returned. Recovery needs that as a fact rather than inferring it from the manifest, which is + // written after the commit and so disagrees with it across a crash in between. + persisted?: boolean; +}; + +// Absolute path of the retained-previous copy for a component's live directory, and of the sidecar +// manifest describing it. The manifest is a sibling FILE rather than something inside the retained +// tree, so the tree stays a byte-for-byte copy of what was live. +function previousDirPathFor(liveDirPath: string): string { + return join(dirname(liveDirPath), DEPLOY_PREVIOUS_DIR, basename(liveDirPath)); +} + +function previousManifestPathFor(liveDirPath: string): string { + return `${previousDirPathFor(liveDirPath)}.json`; +} + +async function readRetainedPreviousManifest(liveDirPath: string): Promise { + try { + const parsed = JSON.parse(await readFile(previousManifestPathFor(liveDirPath), 'utf8')); + if (!parsed?.previous || !parsed?.live) return undefined; + return parsed as RetainedPreviousManifest; + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return undefined; + throw err; + } +} + +async function readRevertRecoveryMarker(markerPath: string): Promise { + try { + const parsed = JSON.parse(await readFile(markerPath, 'utf8')); + if (!parsed?.previous || !parsed?.live) return undefined; + return parsed as RetainedPreviousManifest; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined; + throw error; + } +} + +async function writeJsonAtomically(targetPath: string, value: unknown): Promise { + const tempPath = `${targetPath}.${process.pid}.${randomUUID()}.tmp`; + await mkdir(dirname(targetPath), { recursive: true }); + await writeFile(tempPath, JSON.stringify(value, null, 2), { mode: 0o600 }); + await rename(tempPath, targetPath); +} + +async function writeRetainedPreviousManifest(liveDirPath: string, manifest: RetainedPreviousManifest): Promise { + await writeJsonAtomically(previousManifestPathFor(liveDirPath), manifest); +} + +/** + * Retain the tree this activation displaced as the component's rollback source + * (`.deploy-previous/`), recording which deployment produced it and the root-config entry it + * was activated with. Called by activateStagedApplication after a committed swap, in place of simply + * deleting the displaced tree. + * + * Exactly one previous version is kept per component — this evicts the older one — so retention costs + * a bounded one extra copy per component rather than growing without limit. + * + * Best-effort by design: a component that cannot retain its previous version is still successfully + * deployed, it just isn't revertable. Failing the deploy here would be strictly worse. + */ +async function retainActivatedPrevious( + application: Application, + displacedPath: string | undefined, + deploymentId: string, + activatedConfig: ApplicationConfig | undefined, + outgoing: RetainedVersion +): Promise { + const liveDirPath = application.dirPath; + const previousPath = previousDirPathFor(liveDirPath); + // Whether the displaced tree actually reached the retained slot. If it did, the slot holds exactly what + // a manifest would describe, so a later failure must still record it — otherwise the tree sits in place + // with no manifest and `getRevertTarget` reports nothing, permanently. + let displacedMoved = false; + try { + // The manifest goes FIRST, by being removed. getRevertTarget only checks that a retained tree + // exists, so a manifest that outlives its tree is worse than no manifest: it names a deployment id + // against bytes that are no longer the ones it describes, and an addressed revert would then either + // restore the wrong tree or report "already live" and do nothing. Clearing it means every + // intermediate state below reads as "not revertable", which is what the failure path promises. + await rm(previousManifestPathFor(liveDirPath), { force: true }); + // Evict the older retained-previous (two deploys ago) before renaming this one into its place, so + // the rename never races an incomplete recursive delete (ENOTEMPTY). Also covers the first-deploy + // case, where any retained tree left over from a dropped-and-redeployed component is stale. + await discardDirAside(previousPath, application.name); + if (displacedPath) { + await mkdir(dirname(previousPath), { recursive: true }); + await rename(displacedPath, previousPath); + displacedMoved = true; + } + // Written unconditionally: even a first-ever deploy that retained nothing has to record which + // deployment is now live, or the NEXT activation cannot name the tree it displaces and the + // component stays unrevertable forever. + await writeRetainedPreviousManifest(liveDirPath, { + previous: displacedPath ? outgoing : { deployment_id: null, application_config: null }, + live: { deployment_id: deploymentId, application_config: activatedConfig ?? null }, + }); + } catch (err) { + // Best-effort by design: the deploy succeeded, so failing it here would be worse. + // + // What the manifest may say depends on what is in the retained slot. If a stale tree still occupies + // it, any manifest naming the displaced release would name it against the WRONG bytes, so the + // manifest goes. If the slot is empty, recording the intended state is safe — getRevertTarget + // requires a tree, so it still reads as not revertable — and it is the only thing that keeps the + // parked backup addressable, letting recovery name the release it holds instead of "unknown". + const slotOccupied = !!(await statIfPresent(previousPath)) && !displacedMoved; + if (slotOccupied) { + await rm(previousManifestPathFor(liveDirPath), { force: true }).catch(() => {}); + } else { + await writeRetainedPreviousManifest(liveDirPath, { + previous: displacedPath ? outgoing : { deployment_id: null, application_config: null }, + live: { deployment_id: deploymentId, application_config: activatedConfig ?? null }, + }).catch(async () => { + await rm(previousManifestPathFor(liveDirPath), { force: true }).catch(() => {}); + }); + } + logger.warn( + `Deployed ${application.name}, but could not retain its previous version for revert:`, + errorForLog(err as Error) + ); + } +} + +/** + * Repair a revert that died between renames. The holding directory (`.reverting--`, a + * sibling of the retained previous) only exists while a swap is in flight, so finding one at startup + * means the process was killed mid-swap and the component may have no live directory at all. + * + * Recovery restores the holding tree to whichever slot is empty: the live path when the swap had not + * yet placed the reverted-to version (so the component comes back on the version it was serving), or + * the retained-previous path when the swap DID complete and only the retain step was lost. If both + * slots are occupied the swap finished and the holding tree is residue, so it is discarded. + * + * Best-effort per component: one component's unrecoverable state must not stop the rest from loading. + */ +export async function recoverInterruptedReverts(componentsRootDirPath: string): Promise> { + const previousRoot = join(componentsRootDirPath, DEPLOY_PREVIOUS_DIR); + const failures = new Map(); + let entries; + try { + entries = await readdir(previousRoot, { withFileTypes: true }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return failures; + throw error; + } + // A crash between the final rename and the marker removal leaves a marker whose holding directory is + // gone. Nothing would revisit it — the swap loop below is keyed on finding a holding directory — so + // sweep those first rather than leave them to be trusted by a future attempt. + const entryNames = new Set(entries.map((entry) => entry.name)); + for (const entry of entries) { + if (entry.isDirectory() || !entry.name.endsWith(REVERT_RECOVERY_SUFFIX)) continue; + const holdingName = entry.name.slice(0, -REVERT_RECOVERY_SUFFIX.length); + // The no-live restore branch has no holding directory by design, so absence is not orphanhood for + // its marker. It is recovered on its own terms below. + if (holdingName.endsWith(RESTORE_MARKER_INFIX)) continue; + if (entryNames.has(holdingName)) continue; + const orphanName = holdingName.startsWith(REVERTING_PREFIX) + ? REVERTING_NAME_PATTERN.exec(holdingName.slice(REVERTING_PREFIX.length))?.[1] + : undefined; + if (!orphanName || !safeComponentName(orphanName)) { + await rm(join(previousRoot, entry.name), { force: true }).catch(() => {}); + continue; + } + // Decided under the component lock, then re-read: a revert writes this marker immediately before + // renaming live into the holding directory, so an unlocked sweep can catch that window and delete + // the marker of a revert that is still running — leaving its crash unrecoverable. + await withComponentPreparationLock(join(componentsRootDirPath, orphanName), async () => { + const stillOrphaned = await lstat(join(previousRoot, holdingName)).then( + () => false, + (err) => (err as NodeJS.ErrnoException).code === 'ENOENT' + ); + if (stillOrphaned) await rm(join(previousRoot, entry.name), { force: true }).catch(() => {}); + }).catch((error) => { + logger.warn(`Could not settle a stray revert marker for ${orphanName}:`, errorForLog(error as Error)); + }); + } + // The no-live restore branch: `previous` was renamed straight to live because there was nothing to + // displace. Its marker is the only evidence, and rolling forward is the only sound repair — undoing + // would leave the component with no live tree at all. + for (const entry of entries) { + if (entry.isDirectory() || !entry.name.endsWith(REVERT_RECOVERY_SUFFIX)) continue; + const markerBase = entry.name.slice(0, -REVERT_RECOVERY_SUFFIX.length); + if (!markerBase.endsWith(RESTORE_MARKER_INFIX)) continue; + const componentName = markerBase.slice(0, -RESTORE_MARKER_INFIX.length); + const markerPath = join(previousRoot, entry.name); + if (!safeComponentName(componentName)) { + await rm(markerPath, { force: true }).catch(() => {}); + continue; + } + const liveDirPath = join(componentsRootDirPath, componentName); + try { + await withComponentPreparationLock(liveDirPath, async () => { + const intended = await readRevertRecoveryMarker(markerPath); + if (!intended) { + await rm(markerPath, { force: true }); + return; + } + const previousPath = previousDirPathFor(liveDirPath); + const liveExists = await liveComponentPresent(liveDirPath); + const previousExists = await liveComponentPresent(previousPath); + if (!liveExists && !previousExists) { + await rm(markerPath, { force: true }); + throw new Error( + `Interrupted restore of ${componentName} has neither a live nor a retained tree; ` + + `its bytes are gone and it must be redeployed` + ); + } + if (liveExists && previousExists) { + // The restore finished and something else has since retained a version. Residue only. + await rm(markerPath, { force: true }); + return; + } + if (!liveExists) { + await mkdir(dirname(liveDirPath), { recursive: true }); + await rename(previousPath, liveDirPath); + } + // Idempotent on a repeat pass, and ordered config-then-manifest so the marker outlives both. + const configTransaction = await createApplicationConfigTransaction( + componentName, + intended.live.application_config + ); + await configTransaction.commit(); + await writeRetainedPreviousManifest(liveDirPath, { + previous: { deployment_id: null, application_config: null }, + live: intended.live, + }); + await rm(markerPath, { force: true }); + logger.warn( + `Completed an interrupted restore of ${componentName}: the retained version is live and its ` + + `configuration has been reconciled. It cannot be reverted again until its next deploy` + ); + }); + } catch (error) { + const recoveryError = error instanceof Error ? error : new Error(String(error)); + failures.set(componentName, recoveryError); + logger.error(`Could not recover the interrupted ${componentName} restore:`, errorForLog(recoveryError)); + } + } + for (const entry of entries) { + // A holding tree can itself be a symlink: a `file:` directory deploy makes the live path a symlink, + // and the revert renames that live path here. Gating on isDirectory() alone skipped those entries, + // so such a component was never recovered — it stayed with no live tree across every restart. + if (!entry.isDirectory() && !entry.isSymbolicLink()) continue; + if (!entry.name.startsWith(REVERTING_PREFIX)) continue; + const holding = join(previousRoot, entry.name); + // `.reverting--`. The component name may itself contain dashes, so match the + // trailing UUID explicitly rather than cutting at the last dash — which would leave most of the + // UUID glued to the name and recover into the wrong (or a nonexistent) component directory. + const componentName = REVERTING_NAME_PATTERN.exec(entry.name.slice(REVERTING_PREFIX.length))?.[1]; + if (!componentName || !safeComponentName(componentName)) { + await rm(holding, { recursive: true, force: true }).catch(() => {}); + continue; + } + const liveDirPath = join(componentsRootDirPath, componentName); + try { + await withComponentPreparationLock(liveDirPath, async () => { + const liveExists = await liveComponentPresent(liveDirPath); + if (!liveExists) { + // Two different crash shapes land here, and they need opposite repairs. Either the swap never + // placed the reverted-to version (undo: the holding tree goes back to live), or it did and + // compensation got as far as moving live away before failing (roll forward: the reverted-to + // tree in `previous` goes to live, because config and the manifest already describe it). + // The marker distinguishes them: if the on-disk manifest already matches its intended `live`, + // the persistent side committed and undoing the directories would contradict it. + const marker = await readRevertRecoveryMarker(revertRecoveryMarkerFor(holding)); + const manifest = await readRetainedPreviousManifest(liveDirPath); + // `persisted` is the direct signal, written by the revert the moment its commit returned. The + // manifest comparison stays as a fallback for markers written before that field existed. + const persistedAlreadyExchanged = + !!marker && + (marker.persisted === true || + (!!manifest && + manifest.live?.deployment_id === marker.live?.deployment_id && + manifest.previous?.deployment_id === marker.previous?.deployment_id)); + if (persistedAlreadyExchanged && (await liveComponentPresent(previousDirPathFor(liveDirPath)))) { + await mkdir(dirname(liveDirPath), { recursive: true }); + await rename(previousDirPathFor(liveDirPath), liveDirPath); + await rename(holding, previousDirPathFor(liveDirPath)); + await rm(revertRecoveryMarkerFor(holding), { force: true }); + logger.warn( + `Completed an interrupted ${componentName} revert whose compensation had already begun; ` + + `the reverted-to version is live and matches its persisted configuration` + ); + return; + } + await mkdir(dirname(liveDirPath), { recursive: true }); + await rename(holding, liveDirPath); + await rm(revertRecoveryMarkerFor(holding), { force: true }); + logger.warn( + `Restored the live ${componentName} component directory after an interrupted revert; ` + + `the revert did not take effect and can be retried` + ); + return; + } + const previousPath = previousDirPathFor(liveDirPath); + const previousExists = await liveComponentPresent(previousPath); + if (!previousExists) { + // The reverted-to version is live and only the retain step was lost, so finish it. The + // marker is the source of truth over the manifest, because an earlier pass may already have + // exchanged the manifest and exchanging it twice flips it back. + const recoveryMarkerPath = revertRecoveryMarkerFor(holding); + let intended = await readRevertRecoveryMarker(recoveryMarkerPath); + if (!intended) { + const staleManifest = await readRetainedPreviousManifest(liveDirPath); + if (staleManifest) { + intended = { previous: staleManifest.live, live: staleManifest.previous }; + await writeJsonAtomically(recoveryMarkerPath, intended); + } + } + if (intended) { + // Config first, then manifest, then the directory — so the holding tree survives until + // everything else is durable. Each step is idempotent on a repeat pass. + const configTransaction = await createApplicationConfigTransaction( + componentName, + intended.live.application_config + ); + await configTransaction.commit(); + await writeRetainedPreviousManifest(liveDirPath, intended); + } + await mkdir(dirname(previousPath), { recursive: true }); + await rename(holding, previousPath); + await rm(recoveryMarkerPath, { force: true }); + logger.warn( + `Completed an interrupted ${componentName} revert: the reverted-to version is live, the ` + + `version it displaced is retained again, and its configuration has been reconciled` + ); + return; + } + // Both slots occupied: the swap completed, so this is residue. The marker goes with it — + // leaving it behind keeps durable crash evidence that no longer matches the disk. + await rm(holding, { recursive: true, force: true }); + await rm(revertRecoveryMarkerFor(holding), { force: true }).catch(() => {}); + }); + } catch (error) { + const recoveryError = error instanceof Error ? error : new Error(String(error)); + failures.set(componentName, recoveryError); + logger.error(`Could not recover the interrupted ${componentName} revert:`, errorForLog(recoveryError)); + } + } + return failures; +} + +/** + * Remove a component's retained previous version and its manifest. Both live under the components root + * rather than inside the component directory, so dropping the component does not reach them — and a + * surviving retained tree lets `revert_component` resurrect a dropped component, re-adding its + * root-config and application-lock entries. + */ +export async function discardRetainedPrevious(componentDirPath: string): Promise { + await rm(previousManifestPathFor(componentDirPath), { force: true }); + // Parked in the COMPONENT's aside. Deriving it from the retained path would create + // `.deploy-previous/.deploy-aside`, which startup recovery never sweeps and which then makes the + // `rmdir(previousRoot)` below fail ENOTEMPTY — stranding the tree forever. + await discardDirAside( + previousDirPathFor(componentDirPath), + basename(componentDirPath), + extractionStagingDirectory(componentDirPath) + ); + const previousRoot = join(dirname(componentDirPath), DEPLOY_PREVIOUS_DIR); + for (const entry of await readdir(previousRoot, { withFileTypes: true }).catch(() => [])) { + // Any in-flight revert artifact for this component is meaningless once it is dropped. + if (entry.name.startsWith(`${REVERTING_PREFIX}${basename(componentDirPath)}-`)) { + await rm(join(previousRoot, entry.name), { recursive: true, force: true }).catch(() => {}); + } + } + await rmdir(previousRoot).catch(() => {}); +} + +/** What a component can currently be reverted to, for reporting and for revert targeting. */ +export async function getRevertTarget( + componentDirPath: string +): Promise<{ live: RetainedVersion; previous: RetainedVersion } | undefined> { + const manifest = await readRetainedPreviousManifest(componentDirPath); + if (!manifest) return undefined; + try { + await lstat(previousDirPathFor(componentDirPath)); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return undefined; // manifest without a tree + throw err; + } + return { live: manifest.live, previous: manifest.previous }; +} + +/** + * Swap a component's live version with its retained previous version (`.deploy-previous/`), + * atomically, then let watchers restart onto it. This is what backs `revert_component`: a customer can + * deploy, run their own health checks, and swap back fast — with no package resolution, artifact + * download or install, because the bytes are already on disk. + * + * ADDRESSED, NOT TOGGLED. The caller passes the deployment id it + * expects to be live when this returns: + * - already live → a no-op success, so an ordinary request retry whose first response was lost + * cannot swap the rejected release back in. + * - matches the retained previous → swap, and the displaced tree becomes the new retained previous + * (so a deliberate, explicitly-targeted revert-of-a-revert still rolls forward). + * - anything else → rejected, naming what this component can actually be reverted to. + * + * Returns the root-config entry the newly-live tree was originally activated with, so the caller can + * restore persistent config/install-lock state as part of the rollback, plus whether a swap happened. + * + * This method should only be called from the main thread. + */ +export async function revertApplication( + application: Application, + toDeploymentId: string, + hooks: { + commitPersistentState?: (config: ApplicationConfig | null) => Promise; + rollbackPersistentState?: () => Promise; + } = {} +): Promise<{ swapped: boolean; activatedConfig: ApplicationConfig | null; fromDeploymentId: string | null }> { + const liveDirPath = application.dirPath; + return withComponentPreparationLock(liveDirPath, async () => { + const target = await getRevertTarget(liveDirPath); + if (!target) { + // A caller asking for something the node cannot supply, not a node failure: 409, so the operations + // API does not report an unsatisfiable revert as a 500 the client is invited to retry. + throw new ClientError( + `Cannot revert ${application.name}: no previous version is retained. A component must have been ` + + `deployed over a prior version (which activation retains as .deploy-previous) to be reverted.`, + HTTP_STATUS_CODES.CONFLICT + ); + } + if (target.live.deployment_id === toDeploymentId) { + // Already there. Idempotent so a retry is safe. + return { swapped: false, activatedConfig: target.live.application_config, fromDeploymentId: null }; + } + if (target.previous.deployment_id !== toDeploymentId) { + throw new ClientError( + `Cannot revert ${application.name} to deployment '${toDeploymentId}': it is neither the live version ` + + `('${target.live.deployment_id ?? 'unknown'}') nor the retained previous version ` + + `('${target.previous.deployment_id ?? 'unknown'}'). Only the immediately-previous version is retained ` + + `on disk; redeploy the version you want with deploy_component instead.`, + HTTP_STATUS_CODES.CONFLICT + ); + } + + const previousPath = previousDirPathFor(liveDirPath); + const deployLifecycleId = await broadcastDeployStart(application.name); + try { + let liveExists = true; + try { + await lstat(liveDirPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') liveExists = false; + else throw err; + } + await mkdir(dirname(previousPath), { recursive: true }); + if (liveExists) { + // Three-way atomic swap through a hidden holding path: live → holding, previous → live, + // holding(old live) → previous. The only window where `dirPath` is absent is between the + // first two renames, and deploy:start suppresses watchers across it (same as activation). + // + // Each step is compensated, because a failure here is the one that hurts most: after the + // first rename the component has NO live directory, so an uncompensated I/O error (disk + // full, a permission change, an unexpected fault on previousPath) would leave the component + // unservable with its bytes stranded under the holding path. The holding name is also + // recoverable by name at startup (recoverInterruptedReverts), which covers the case + // compensation cannot: the process dying between renames. + const holding = join(dirname(previousPath), `${REVERTING_PREFIX}${basename(liveDirPath)}-${randomUUID()}`); + // Written before anything moves: a crash between the renames and the durable writes below + // leaves the holding tree AND this marker, which is what lets recoverInterruptedReverts finish + // the job. Without it, recovery would re-exchange an already-exchanged manifest. + const recoveryMarkerPath = revertRecoveryMarkerFor(holding); + await writeJsonAtomically(recoveryMarkerPath, { + previous: target.live, + live: target.previous, + }); + await rename(liveDirPath, holding); + try { + await rename(previousPath, liveDirPath); + } catch (swapError) { + // Nothing is live: put the outgoing tree straight back and fail with the component intact. + await rename(holding, liveDirPath).catch((restoreError) => { + throw new AggregateError( + [swapError, restoreError], + `Failed to revert ${application.name} and could not restore its live directory from ` + + `${holding}; the component has no live version until that directory is restored` + ); + }); + throw swapError; + } + // Persistent state commits here, inside the lock and while the holding tree still exists. + // Committing it after the lock released let a queued activation swap and commit in between, + // leaving that activation's bytes live under this revert's config. + // + // Everything from the commit to the final rename shares one undo path. A failure after the + // commit — a manifest write, or the retain rename — has to take config and the application + // lock back too, or the directories end up describing one release while the persisted state + // names the other, and a cold start reinstalls over the live bytes. + // ATTEMPTED, not committed. The config transaction marks itself started before writing root + // config and can reject between that write and the application-lock write, so a rejected + // commit may still have changed persisted state. Its rollback is a no-op when nothing was + // written, which is what makes calling it after any attempt the safe choice. + let persistAttempted = false; + let manifestWritten = false; + const undoRevert = async (cause: unknown, detail: string): Promise => { + const undoErrors: unknown[] = []; + if (persistAttempted) { + try { + await hooks.rollbackPersistentState?.(); + } catch (rollbackError) { + // Persisted state may still name the reverted-to release and we could not take it back. + // Moving the directories now would contradict it, so leave the holding tree and the + // marker untouched: that is exactly the shape startup recovery rolls forward from. + throw new AggregateError( + [cause, rollbackError], + `Reverted ${application.name} but could not ${detail}, and could not roll its ` + + `configuration back. The directories and ${recoveryMarkerPath} are left in place for ` + + `startup recovery to finish the revert` + ); + } + } + if (manifestWritten) { + // The manifest currently claims the exchange happened. Put it back before the directories + // move, so no window reports the reverted-to version as live over the old bytes. + await writeRetainedPreviousManifest(liveDirPath, { + previous: target.previous, + live: target.live, + }).catch((manifestError) => undoErrors.push(manifestError)); + } + try { + await rename(liveDirPath, previousPath); + await rename(holding, liveDirPath); + application.useLiveBuildDir(); + await rm(recoveryMarkerPath, { force: true }); + } catch (restoreError) { + undoErrors.push(restoreError); + } + if (undoErrors.length) { + throw new AggregateError( + [cause, ...undoErrors], + `Reverted ${application.name} but could not ${detail}, and could not fully undo it; ` + + `${holding} may still hold the previously-live tree` + ); + } + throw cause; + }; + try { + persistAttempted = true; + await hooks.commitPersistentState?.(target.previous.application_config); + // Recorded as a fact, not inferred later from the manifest. The manifest is written AFTER + // this commit, so a crash in between leaves persisted state exchanged while the manifest + // still reads pre-revert — and recovery, comparing the two, would undo directories that + // config already describes, leaving old code running under new configuration. + await writeJsonAtomically(recoveryMarkerPath, { + previous: target.live, + live: target.previous, + persisted: true, + }); + application.useLiveBuildDir(); + await writeRetainedPreviousManifest(liveDirPath, { + previous: target.live, + live: target.previous, + }); + manifestWritten = true; + } catch (persistError) { + await undoRevert(persistError, 'persist its configuration'); + } + // Consumes the recovery evidence, so it goes last. + try { + await rename(holding, previousPath); + } catch (retainError) { + await undoRevert(retainError, 'retain the displaced version'); + } + await rm(recoveryMarkerPath, { force: true }); + } else { + // No live version to preserve; restore the previous into place. Nothing becomes the new + // retained previous, so the component can't be reverted again until its next deploy. + // + // This branch gets the same intent marker and undo as the swap above: without them a config + // failure after the rename left the retained slot empty, the persisted state naming an absent + // version, and no `.reverting-*` artifact for recovery to find. + const restoreMarkerPath = revertRecoveryMarkerFor(`${previousPath}${RESTORE_MARKER_INFIX}`); + await writeJsonAtomically(restoreMarkerPath, { + previous: { deployment_id: null, application_config: null }, + live: target.previous, + }); + await rename(previousPath, liveDirPath); + try { + await hooks.commitPersistentState?.(target.previous.application_config); + application.useLiveBuildDir(); + await writeRetainedPreviousManifest(liveDirPath, { + previous: { deployment_id: null, application_config: null }, + live: target.previous, + }); + } catch (persistError) { + try { + await hooks.rollbackPersistentState?.(); + } catch (rollbackError) { + // As in the swap branch: persisted state may name the restored release, so undoing the + // rename would contradict it. Leave the marker for startup recovery to roll forward. + throw new AggregateError( + [persistError, rollbackError], + `Restored ${application.name} from its retained version but could not persist its ` + + `configuration or roll that back. ${restoreMarkerPath} is left in place for startup ` + + `recovery to finish the restore` + ); + } + const undoErrors: unknown[] = []; + await rename(liveDirPath, previousPath).catch((restoreError) => undoErrors.push(restoreError)); + await rm(restoreMarkerPath, { force: true }).catch(() => {}); + if (undoErrors.length) { + throw new AggregateError( + [persistError, ...undoErrors], + `Restored ${application.name} from its retained version but could not persist configuration ` + + `or undo the restore` + ); + } + throw persistError; + } + await rm(restoreMarkerPath, { force: true }); + } + return { + swapped: true, + activatedConfig: target.previous.application_config, + fromDeploymentId: target.live.deployment_id, + }; + } finally { + broadcastDeployEnd(application.name, deployLifecycleId); + } + }); +} type ExtractionTransaction = { commit(): Promise; rollback(): Promise; @@ -598,7 +1450,10 @@ function canonicalizeJSON(value: any): any { * * Only one of `application.payload` or `application.package` should be specified; otherwise, an error is thrown. * - * Writes the application to the configured components root directory using the `application.name` and overwrites any existing directory. + * Writes the application into `application.buildDirPath`, overwriting any existing directory there. + * By default that is the live component directory (`application.dirPath`); during a two-phase deploy + * `stageApplication` points it at the hidden staging directory instead, so the live path is never + * touched until `activateStagedApplication` swaps the staged copy into place. * * This method may be called from any Harper thread. Same-component calls are serialized across * threads by the preparation lock below. @@ -634,8 +1489,10 @@ export async function extractApplication( tarball = Readable.from(payload as Buffer); } } else { - // Given a package, there are a a couple options - const parentDirPath = dirname(application.dirPath); + // Given a package, there are a a couple options. The tarball is packed next to the build + // target (the staging dir during a two-phase deploy) so it lands on the same filesystem and + // is swept with the staging area rather than littering the live components root. + const parentDirPath = dirname(application.buildDirPath); // If the package identifier is a file path we need to check if its a tarball or a directory if (application.packageIdentifier.startsWith('file:')) { @@ -645,8 +1502,19 @@ export async function extractApplication( const stats = await stat(packagePath); if (stats.isDirectory()) { - // If its a directory, symlink - await symlink(packagePath, application.dirPath, 'dir'); + // If its a directory, symlink. Anything already at the build target has to go first, or + // symlink() throws EEXIST — which on the in-place path means a redeploy of a + // directory-package component. + // + // Parked aside with an atomic rename, NOT removed in place. This returns early, before + // the extraction transaction below, so an in-place recursive rm here would delete the + // LIVE component tree outright: no aside, no rollback record, nothing for startup + // recovery to restore, and it races a still-running worker writing into the directory it + // is deleting (the ENOTEMPTY/EPERM hazard documented on discardDirAside). The rename is + // atomic and the sweep is best-effort afterwards. + await discardDirAside(application.buildDirPath, application.name); + await mkdir(dirname(application.buildDirPath), { recursive: true }); + await symlink(packagePath, application.buildDirPath, 'dir'); // And return early since we're done; no extraction needed return; } @@ -739,16 +1607,32 @@ export async function extractApplication( // leading dot keeps loadComponentDirectories from picking it up as a phantom // component, and the per-component path means a sibling component never collides // with (or sweeps) another's aside. - const asideStagingDir = extractionStagingDirectory(application.dirPath); + // The directory this extraction replaces. Defaults to the live component path — so the one-shot + // deploy, boot-time installs, and every direct extractApplication caller behave exactly as before — + // but points at `.deploy-staging//` while a two-phase stage is building. Everything + // below is expressed against this target rather than `application.dirPath`, which is what lets ONE + // transaction protocol cover both an in-place replacement and a staged build. + const buildDirPath = application.buildDirPath; + const buildingInPlace = buildDirPath === application.dirPath; + // What the rollback/cleanup helpers act on: `dirPath` here is the extraction TARGET, which is not + // necessarily the live component path. + const extractionContext: ExtractionContext = { + name: application.name, + dirPath: buildDirPath, + logger: application.logger, + }; + const asideStagingDir = extractionStagingDirectory(buildDirPath); const transactionPaths = new Set(); let asidePath: string | undefined; let recoveryRecordPath: string; try { await ensureExtractionStagingDirectory(asideStagingDir); - await recoverOrCleanupStaleExtractionPaths(application, asideStagingDir); + await recoverOrCleanupStaleExtractionPaths(extractionContext, asideStagingDir); let componentExists = true; try { - await lstat(application.dirPath); + // lstat, not access: access follows symlinks, so a dangling symlink at the target reports ENOENT + // and the mkdir below then fails EEXIST because the dead link still occupies the path. + await lstat(buildDirPath); } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; componentExists = false; @@ -756,7 +1640,7 @@ export async function extractApplication( if (componentExists) { await ensureExtractionStagingDirectory(asideStagingDir); asidePath = join(asideStagingDir, `${IN_PROGRESS_ASIDE_PREFIX}${Date.now()}-${process.pid}-${randomUUID()}`); - await rename(application.dirPath, asidePath); + await rename(buildDirPath, asidePath); transactionPaths.add(asidePath); recoveryRecordPath = asidePath; } else { @@ -768,18 +1652,23 @@ export async function extractApplication( await writeFile(recoveryRecordPath, '', { flag: 'wx', mode: 0o600 }); transactionPaths.add(recoveryRecordPath); } - if (asidePath) application.isNewComponent = false; + // A non-null aside means something already occupied the extraction target, so an IN-PLACE build + // is replacing an already-active component rather than deploying a new one (harper#1806). Only + // meaningful when the target IS the live directory: during a two-phase stage the target is a + // fresh staging dir whose prior existence says nothing about the live component, so + // isNewComponent is left for activateStagedApplication to read off the live path at swap time. + if (asidePath && buildingInPlace) application.isNewComponent = false; try { - await mkdir(application.dirPath, { recursive: true }); - await pipeline(tarball, gunzip(), extract(application.dirPath)); + await mkdir(buildDirPath, { recursive: true }); + await pipeline(tarball, gunzip(), extract(buildDirPath)); - const extracted = await readdir(application.dirPath, { withFileTypes: true }); + const extracted = await readdir(buildDirPath, { withFileTypes: true }); if (extracted.length === 1 && extracted[0].isDirectory()) { - const topLevelDirPath = join(application.dirPath, extracted[0].name); + const topLevelDirPath = join(buildDirPath, extracted[0].name); if (process.platform === 'win32') { for (const childName of await readdir(topLevelDirPath)) { - await rename(join(topLevelDirPath, childName), join(application.dirPath, childName)); + await rename(join(topLevelDirPath, childName), join(buildDirPath, childName)); } await rmdir(topLevelDirPath); } else { @@ -787,14 +1676,14 @@ export async function extractApplication( const tempDirPath = join(asideStagingDir, `.normalize-${process.pid}-${Date.now()}-${randomUUID()}`); transactionPaths.add(tempDirPath); await rename(topLevelDirPath, tempDirPath); - await rmdir(application.dirPath); - await rename(tempDirPath, application.dirPath); + await rmdir(buildDirPath); + await rename(tempDirPath, buildDirPath); transactionPaths.delete(tempDirPath); } } } catch (error) { try { - await rollbackExtractedDirectory(application, asideStagingDir, asidePath, transactionPaths, false); + await rollbackExtractedDirectory(extractionContext, asideStagingDir, asidePath, transactionPaths, false); } catch (rollbackError) { throw new AggregateError( [error, rollbackError], @@ -820,11 +1709,11 @@ export async function extractApplication( const retiredMarkerPath = await retireExtractionAside(recoveryRecordPath); transactionPaths.add(retiredMarkerPath); settled = true; - await cleanupExtractionPaths(application, asideStagingDir, transactionPaths); + await cleanupExtractionPaths(extractionContext, asideStagingDir, transactionPaths); }, async rollback() { if (settled) return; - await rollbackExtractedDirectory(application, asideStagingDir, asidePath, transactionPaths, true); + await rollbackExtractedDirectory(extractionContext, asideStagingDir, asidePath, transactionPaths, true); settled = true; }, }; @@ -1446,19 +2335,24 @@ async function rollbackExtractedDirectory( } /** - * Install an application to its relative `application.dirPath` using either a + * Install an application into `application.buildDirPath` using either a * configured `application.install` command, a derived package manager from the * application's `package.json#devEngines`, or falling back to the default * package manager, `npm`. * - * Will return early if `node_modules` already exists within the `application.dirPath` + * `buildDirPath` is the live component directory by default, or the hidden staging directory + * during a two-phase deploy (see stageApplication) — so `npm install`, the slowest and most + * failure-prone step, runs against the staged copy and never leaves the live path half-installed. + * + * Will return early if `node_modules` already exists within the build directory. * * This method may be called from any Harper thread as part of a serialized preparation. */ export async function installApplication(application: Application) { + const buildDirPath = application.buildDirPath; let packageJSON: any; try { - packageJSON = JSON.parse(await readFile(join(application.dirPath, 'package.json'), 'utf8')); + packageJSON = JSON.parse(await readFile(join(buildDirPath, 'package.json'), 'utf8')); } catch (err) { if (err.code !== 'ENOENT') throw err; // If no package.json, nothing to install @@ -1467,10 +2361,14 @@ export async function installApplication(application: Application) { } try { // Does node_modules exist? - await access(join(application.dirPath, 'node_modules'), constants.F_OK); + // buildDirPath, not dirPath: during a two-phase stage the candidate being installed is the staging + // tree, and the live directory's node_modules says nothing about it. + await access(join(buildDirPath, 'node_modules'), constants.F_OK); application.logger.info( `Application ${application.name} already has node_modules; skipping install and treating the runtime as opaque for redeploy comparison` ); + // The installed tree came from the payload rather than from an install we ran, so its contents + // can't be compared against a fresh install to decide whether the runtime changed. application.installationIsOpaque = true; return; } catch (err) { @@ -1488,7 +2386,7 @@ export async function installApplication(application: Application) { application.name, command, args, - application.dirPath, + buildDirPath, application.install?.timeout, customOnLine, application.npmUserconfigPath @@ -1550,7 +2448,7 @@ export async function installApplication(application: Application) { application.name, (application.packageManagerPrefix ? application.packageManagerPrefix + ' ' : '') + packageManager.name, application.install?.allowInstallScripts ? ['install'] : ['install', '--ignore-scripts'], // All of `npm`, `yarn`, and `pnpm` support the `install` command. If we need to configure options here we may have to use some other defaults though - application.dirPath, + buildDirPath, application.install?.timeout, pmOnLine, application.npmUserconfigPath @@ -1608,7 +2506,7 @@ export async function installApplication(application: Application) { application.name, (application.packageManagerPrefix ? application.packageManagerPrefix + ' ' : '') + 'npm', npmInstallArgs, - application.dirPath, + buildDirPath, application.install?.timeout, npmOnLine, application.npmUserconfigPath @@ -1651,6 +2549,11 @@ interface ApplicationOptions { // Deploy credentials already resolved to literal tokens, of any kind; partitioned by the // constructor into the npm and git halves, which are injected by entirely different mechanisms. credentials?: ResolvedCredential[]; + // Stable identifier for the hidden staging directory this deploy builds into, so a two-phase + // deploy's activate phase can reconstruct the same staging path the prior stage phase + // built (both derive it from the deployment id). Defaults to a random UUID for + // callers that stage and activate against one in-memory Application instance. + stagingId?: string; } export class Application { @@ -1672,6 +2575,11 @@ export class Application { // Path to the per-deploy `.npmrc`, set by writeTransientNpmrc() during prepareApplication and // passed to the spawn calls; undefined when no registry credentials were provided. npmUserconfigPath?: string; + // Stable id for this deploy's staging directory (see ApplicationOptions.stagingId). + stagingId: string; + // When set, extract/install build here instead of the live `dirPath`. stageApplication() points + // it at `stagingDirPath`; activateStagedApplication() clears it after swapping the staged copy live. + #buildDirPath?: string; #npmrcTempDir?: string; #gitCredentialSession?: GitCredentialSession; // Existing components rely on their runtime-equivalence checks; only a first deploy restarts unconditionally. @@ -1679,7 +2587,15 @@ export class Application { packageMetadataChanged: boolean = false; installationIsOpaque: boolean = false; - constructor({ name, payload, packageIdentifier, install, onInstallLine, credentials }: ApplicationOptions) { + constructor({ + name, + payload, + packageIdentifier, + install, + onInstallLine, + credentials, + stagingId, + }: ApplicationOptions) { this.name = name; this.payload = payload; this.packageIdentifier = packageIdentifier && derivePackageIdentifier(packageIdentifier); @@ -1702,10 +2618,50 @@ export class Application { const componentsRoot = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); if (!componentsRoot) throw new Error('componentsRoot is not configured'); this.dirPath = join(componentsRoot, name); + this.stagingId = stagingId ?? randomUUID(); this.logger = logger.loggerWithTag(name); this.packageManagerPrefix = getConfigValue(CONFIG_PARAMS.APPLICATIONS_PACKAGEMANAGERPREFIX); } + // Directory where extract/install currently build. Defaults to the live component directory + // (`dirPath`) — the legacy in-place path, still used by boot-time installApplications() — and is + // repointed at the hidden staging directory for the duration of a two-phase deploy. + get buildDirPath(): string { + return this.#buildDirPath ?? this.dirPath; + } + + // Hidden, per-deploy staging directory the incoming version is built into before it goes live: + // `/.deploy-staging//`. Deterministic from (stagingId, component + // name) so the activate phase can find what the stage phase built. Sits under the components + // root (dirname(dirPath)) so the go-live rename() into `dirPath` stays on one filesystem and is + // therefore atomic. Two properties fall out of putting the deployment id ABOVE the component name: + // - the leaf directory's basename IS the component name, so the pre-go-live validation load + // (componentLoader keys the ApplicationScope + status off basename) sees the real name, not a + // UUID; and + // - each deployment gets its OWN parent (.deploy-staging/), so a parallel or queued + // deploy of the same component never shares a directory — cleanup can't sweep a sibling. + // See DEPLOY_STAGING_DIR. + get stagingDirPath(): string { + return join(dirname(this.dirPath), DEPLOY_STAGING_DIR, this.stagingId, this.name); + } + + // The retained-previous copy this component would revert to (`.deploy-previous/`). See + // DEPLOY_PREVIOUS_DIR / activateStagedApplication / revertApplication. + get previousDirPath(): string { + return previousDirPathFor(this.dirPath); + } + + // Route extract/install into the staging directory. Called by stageApplication(). + useStagingBuildDir(): void { + this.#buildDirPath = this.stagingDirPath; + } + + // Restore the live component directory as the build target. Called by activateStagedApplication() + // once the staged copy has been swapped into place. + useLiveBuildDir(): void { + this.#buildDirPath = undefined; + } + // Write the transient `.npmrc` into a fresh 0700 temp dir (file mode 0600) and record its path // so the deploy's npm spawns authenticate against the private registry. No-op without registry // credentials. @@ -1908,6 +2864,827 @@ export async function prepareApplication(application: Application) { } } +export function stagedApplicationPath(componentDirPath: string, deploymentId: string): string { + if (!DEPLOYMENT_ID_PATTERN.test(deploymentId)) throw new Error(`Invalid deployment id '${deploymentId}'`); + return join(dirname(componentDirPath), DEPLOY_STAGING_DIR, deploymentId, basename(componentDirPath)); +} + +async function ensureSecureDirectory(directory: string, create: boolean, description: string): Promise { + let directoryStat; + try { + directoryStat = await lstat(directory); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + if (!create) return false; + await mkdir(directory, { recursive: true, mode: 0o700 }); + directoryStat = await lstat(directory); + } + if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink()) { + throw new Error(`${description} is not a directory: ${directory}`); + } + return true; +} + +async function secureStagingDeploymentDirectory( + componentDirPath: string, + deploymentId: string, + create: boolean +): Promise { + const stagingDirPath = stagedApplicationPath(componentDirPath, deploymentId); + const deploymentDirPath = dirname(stagingDirPath); + const stagingRoot = dirname(deploymentDirPath); + if (!(await ensureSecureDirectory(stagingRoot, create, 'Component deploy staging path'))) return undefined; + if (!(await ensureSecureDirectory(deploymentDirPath, create, 'Component deploy staging path'))) return undefined; + return deploymentDirPath; +} + +/** Build and install a candidate under .deploy-staging without mutating the live component tree. */ +export async function stageApplication(application: Application, deploymentId: string): Promise { + const liveDirPath = application.dirPath; + const stagingDirPath = stagedApplicationPath(liveDirPath, deploymentId); + application.stagingId = deploymentId; + application.useStagingBuildDir(); + try { + await withComponentPreparationLock(liveDirPath, async () => { + const deploymentDirPath = await secureStagingDeploymentDirectory(liveDirPath, deploymentId, true); + await rm(stagingDirPath, { recursive: true, force: true }); + await rm(join(deploymentDirPath!, STAGED_COMPLETE_MARKER), { force: true }); + try { + await application.writeTransientNpmrc(); + try { + await application.startGitCredentialSession(); + await extractApplication(application); + } finally { + await application.cleanupGitCredentialSession(); + } + await installApplication(application); + await writeFile( + join(deploymentDirPath!, STAGED_COMPLETE_MARKER), + JSON.stringify({ installationIsOpaque: application.installationIsOpaque }), + { flag: 'wx', mode: 0o600 } + ); + } catch (error) { + await rm(stagingDirPath, { recursive: true, force: true }).catch(() => {}); + await rm(join(deploymentDirPath!, STAGED_COMPLETE_MARKER), { force: true }).catch(() => {}); + throw error; + } finally { + await application.cleanupTransientNpmrc(); + } + }); + } finally { + application.useLiveBuildDir(); + } + await pruneStagedBuilds(application.name, deploymentId, getStagingRetentionMaxCount()); + return stagingDirPath; +} + +function activationStagingDirectory(componentDirPath: string): string { + return join(dirname(componentDirPath), DEPLOY_ACTIVATION_DIR, basename(componentDirPath)); +} + +async function activationArtifacts(componentDirPath: string, deploymentId: string): Promise { + const directory = activationStagingDirectory(componentDirPath); + let entries; + try { + entries = await readdir(directory, { withFileTypes: true }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return []; + throw error; + } + return entries + .filter( + (entry) => + entry.name.startsWith(`${ACTIVATION_BACKUP_PREFIX}${deploymentId}`) || + entry.name === `${ACTIVATION_NEW_PREFIX}${deploymentId}` + ) + .map((entry) => join(directory, entry.name)); +} + +/** + * `lstat`/`stat` where only genuine absence reads as absent. Collapsing every error to "not there" + * makes a transient EACCES/EIO/EMFILE look like a missing tree, and the staged-candidate branch + * deletes what it believes is unusable — so anything other than ENOENT has to propagate. + */ +async function statIfPresent( + path: string, + followSymlinks = false +): Promise> | undefined> { + try { + return await (followSymlinks ? stat(path) : lstat(path)); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined; + throw error; + } +} + +export async function hasCompleteStagedApplication(stagingDirPath: string): Promise { + const deploymentDirPath = dirname(stagingDirPath); + const [stagingRootStat, deploymentStat, stagedStat, stagedTargetStat, markerStat] = await Promise.all([ + statIfPresent(dirname(deploymentDirPath)), + statIfPresent(deploymentDirPath), + statIfPresent(stagingDirPath), + statIfPresent(stagingDirPath, true), + statIfPresent(join(deploymentDirPath, STAGED_COMPLETE_MARKER)), + ]); + return ( + !!stagingRootStat?.isDirectory() && + !stagingRootStat.isSymbolicLink() && + !!deploymentStat?.isDirectory() && + !deploymentStat.isSymbolicLink() && + !!stagedStat && + (stagedStat.isDirectory() || stagedStat.isSymbolicLink()) && + !!stagedTargetStat?.isDirectory() && + !!markerStat?.isFile() && + !markerStat.isSymbolicLink() + ); +} + +/** + * What the stage recorded about its own install. Falls back to "opaque" when the marker carries no + * usable content: an unknown install has to be treated as one whose result can't be compared, so the + * restart gate errs toward requiring a restart rather than silently skipping one. + */ +async function readStagedCompletion(stagingDirPath: string): Promise<{ installationIsOpaque: boolean }> { + try { + const raw = await readFile(join(dirname(stagingDirPath), STAGED_COMPLETE_MARKER), 'utf8'); + if (!raw.trim()) return { installationIsOpaque: true }; + return { installationIsOpaque: JSON.parse(raw)?.installationIsOpaque !== false }; + } catch { + return { installationIsOpaque: true }; + } +} + +/** + * Re-point dependency links that the activation swap invalidated. + * + * `npm install` runs against the STAGING directory, and activation then renames that directory to the + * live path. Any dependency npm materialized as a link with an ABSOLUTE target inside the staging + * directory therefore dangles the moment the rename happens — the path it names no longer exists. + * + * This is the normal case for a `file:` dependency on Windows, where npm creates a directory JUNCTION + * and junctions are always absolute. On Linux npm writes a relative symlink (`../vendor/probe`), which + * survives the move untouched, which is why this only ever bites on Windows — the failure there is a + * bare `Cannot find module ''` at component load, well after a deploy that reported success. + * + * Each such link is recreated pointing at the same relative location under the LIVE directory. Links + * whose targets are relative, or absolute but outside the staging tree (a dependency deliberately + * linked elsewhere on the machine), are left exactly as they are. + */ +async function repointStagedDependencyLinks( + treeDirPath: string, + futureLiveDirPath: string, + currentTargetRootPath: string = treeDirPath +): Promise { + const nodeModulesPath = join(treeDirPath, 'node_modules'); + // The walk must stay inside the component's own node_modules. `readdir` FOLLOWS symlinks, so a staged + // payload shipping `node_modules` (or a `node_modules/@scope`) as a link to somewhere else on the + // machine would otherwise have its target's contents enumerated — and a link in there whose target + // happened to point into staging would be removed and recreated, writing outside the component tree. + // Requiring a real directory at each level we descend keeps every candidate under the real root. + const nodeModulesStat = await statIfPresent(nodeModulesPath); + if (!nodeModulesStat?.isDirectory() || nodeModulesStat.isSymbolicLink()) return 0; + // Package links live at `node_modules/` or `node_modules/@scope/`, and npm nests a further + // `node_modules` under a package whenever hoisting is blocked by a version conflict — routinely so + // under workspaces. A link nested that way dangles after the swap exactly like a top-level one, so the + // walk has to follow real nested trees rather than stopping at the first two levels. + const candidates: string[] = []; + const collect = async (directoryPath: string): Promise => { + let entries; + try { + entries = await readdir(directoryPath, { withFileTypes: true }); + } catch (error) { + // Absence is benign; anything else is not. Swallowing an EIO/EACCES here would make the walk + // find nothing to repoint and let the activation swap in a tree whose absolute links still + // address the staging path — the dangling state this repair exists to prevent. + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + return; + } + for (const entry of entries) { + if (entry.name.startsWith('.')) continue; + const entryPath = join(directoryPath, entry.name); + if (entry.name.startsWith('@')) { + // Only descend into a REAL directory: a symlinked scope directory belongs to whatever it points + // at, not to this component (see the note on nodeModulesStat above). + const scopeStat = await statIfPresent(entryPath); + if (scopeStat?.isDirectory() && !scopeStat.isSymbolicLink()) await collect(entryPath); + continue; + } + candidates.push(entryPath); + const packageStat = await statIfPresent(entryPath); + if (!packageStat?.isDirectory() || packageStat.isSymbolicLink()) continue; + const nestedPath = join(entryPath, 'node_modules'); + const nestedStat = await statIfPresent(nestedPath); + if (nestedStat?.isDirectory() && !nestedStat.isSymbolicLink()) await collect(nestedPath); + } + }; + await collect(nodeModulesPath); + + let repointed = 0; + for (const linkPath of candidates) { + const linkStat = await statIfPresent(linkPath); + if (!linkStat?.isSymbolicLink()) continue; + let target; + try { + target = await readlink(linkPath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + continue; + } + if (!isAbsolute(target)) continue; + // Belt-and-braces containment: never mutate a path that is not under the real node_modules root. + const withinNodeModules = relative(nodeModulesPath, linkPath); + if (withinNodeModules.startsWith('..') || isAbsolute(withinNodeModules)) continue; + // Windows returns junction targets from readlink in the `\\?\C:\…` extended-length form. Compared + // against a plain root, `relative` sees two different roots and hands back an absolute path, so the + // containment test below rejects every junction and silently skips repointing — exactly the links + // that dangle after the swap. Strip the prefix so both sides are in the same form. + const withinStaging = relative(currentTargetRootPath, stripExtendedLengthPrefix(target)); + // `..` or an absolute result means the target is outside the staging tree — not ours to touch. + if (!withinStaging || withinStaging.startsWith('..') || isAbsolute(withinStaging)) continue; + // NOT best-effort. This link is only being touched because activation is about to invalidate its + // target, so a failure here means the component goes live with a dangling dependency — which the + // pre-swap load validation cannot catch, because the link was perfectly valid in staging. Throwing + // keeps the old release live and returns an error, instead of reporting a successful deploy of a + // component that cannot resolve its dependencies. + await rm(linkPath, { force: true }); + await symlink(join(futureLiveDirPath, withinStaging), linkPath, process.platform === 'win32' ? 'junction' : 'dir'); + repointed++; + } + return repointed; +} + +/** Atomically replace the live component and compensate if persistent activation work fails. */ +export async function activateStagedApplication( + application: Application, + deploymentId: string, + hooks: { + beforeSwap?: () => Promise; + beforeCommit?: () => Promise; + onRollback?: () => Promise; + /** + * The immutable activation specification this deployment is being activated with. Used only to + * derive the root-config entry recorded alongside the retained previous version, so a later + * `revert_component` can restore persistent config/install-lock state and not just the directory. + * Omit it and the swap still happens — the component just isn't revertable afterwards. + */ + activationSpec?: Record; + } = {} +): Promise { + const stagingDirPath = stagedApplicationPath(application.dirPath, deploymentId); + await withComponentPreparationLock(application.dirPath, async () => { + await secureStagingDeploymentDirectory(application.dirPath, deploymentId, false); + if (!(await hasCompleteStagedApplication(stagingDirPath))) { + const stagedStat = await lstat(stagingDirPath).catch(() => undefined); + if (!stagedStat) { + throw new Error(`Cannot activate ${application.name}: deployment '${deploymentId}' has no staged build`); + } + throw new Error(`Cannot activate ${application.name}: staged build is incomplete`); + } + + const activationDir = activationStagingDirectory(application.dirPath); + await ensureSecureDirectory(dirname(activationDir), true, 'Component activation staging path'); + await ensureSecureDirectory(activationDir, true, 'Component activation staging path'); + // Who is live right now, so the tree this activation displaces stays addressable for revert. Read + // before the swap, since the swap is what makes it "previous". + // NOT caught: the reader already maps a missing manifest to undefined, so anything it throws is a + // real read failure (EACCES, EIO, corrupt JSON). Treating those as "no manifest" would record the + // outgoing release as `deployment_id: null` and let retention overwrite the addressable previous + // version with an unaddressable one — losing revert silently. Failing here is before the swap, so + // the live component keeps serving. + const outgoing: RetainedVersion = (await readRetainedPreviousManifest(application.dirPath))?.live ?? { + // No manifest yet: either a first-ever deploy, or a component last activated by a Harper that + // predates retention. Record the config it is running so a revert can still restore that, even + // though its deployment id is unknowable and so cannot be a revert target. + deployment_id: null, + application_config: readConfigFile()?.[application.name] ?? null, + }; + const deployLifecycleId = await broadcastDeployStart(application.name); + let backupPath: string | undefined; + let newMarkerPath: string | undefined; + let swapped = false; + let repointedLinks = 0; + // Attempted, not counted: the repoint mutates links as it walks, so a throw partway leaves some + // already aimed at the live path while the assignment below never happens. Keying compensation on + // the returned count skipped the undo for exactly that case, and a retry of this deployment id then + // validated the staged tree against whatever release is live. + let repointAttempted = false; + try { + await hooks.beforeSwap?.(); + const existingArtifacts = await activationArtifacts(application.dirPath, deploymentId); + backupPath = existingArtifacts.find((candidate) => basename(candidate).startsWith(ACTIVATION_BACKUP_PREFIX)); + newMarkerPath = existingArtifacts.find((candidate) => basename(candidate).startsWith(ACTIVATION_NEW_PREFIX)); + if (backupPath || newMarkerPath) { + // Resuming an attempt that already moved the displaced release aside (or recorded that there + // was none). Anything at the live path now therefore arrived AFTER that move, so it is not the + // tree this activation displaced: boot-time installApplications() recreates a package component + // from root config whenever it finds the path missing, and it runs before this recovery. Left + // in place it makes the rename below fail ENOTEMPTY — deterministically, on every boot, because + // installation recreates it again each time. Parked rather than deleted; the release this + // activation actually displaced is the one already held in the backup. + if (await statIfPresent(application.dirPath)) { + logger.warn( + `Parking an unexpected ${application.name} directory that appeared after this activation ` + + `moved the live tree aside; the displaced release is already retained` + ); + await discardDirAside(application.dirPath, application.name); + } + } else { + try { + await lstat(application.dirPath); + application.isNewComponent = false; + // Installed package metadata sits outside most plugin watch globs, so no file watcher sees a + // dependency or module-entry change — but it does invalidate already-loaded code. The one-shot + // path compares it across its in-place install (prepareApplication); the two-phase path has to + // compare the outgoing live tree against the staged one, which is only possible here, while + // both still exist. Feeds markRestartRequiredForDeploy (harper#674). + application.packageMetadataChanged = installedRuntimeChanged( + await readInstalledPackageMetadata(application.dirPath), + await readInstalledPackageMetadata(stagingDirPath), + (await readStagedCompletion(stagingDirPath)).installationIsOpaque + ); + backupPath = join( + activationDir, + `${ACTIVATION_BACKUP_PREFIX}${deploymentId}-${Date.now()}-${process.pid}-${randomUUID()}` + ); + await rename(application.dirPath, backupPath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + application.isNewComponent = true; + newMarkerPath = join(activationDir, `${ACTIVATION_NEW_PREFIX}${deploymentId}`); + try { + await writeFile(newMarkerPath, '', { flag: 'wx', mode: 0o600 }); + } catch (markerError) { + if ((markerError as NodeJS.ErrnoException).code !== 'EEXIST') throw markerError; + const markerStat = await lstat(newMarkerPath); + if (!markerStat.isFile() || markerStat.isSymbolicLink()) { + throw new Error(`Component activation marker is not a regular file: ${newMarkerPath}`); + } + } + } + } + // Re-point dependency links BEFORE the swap and inside the transaction. npm installed against the + // staging path, so any link it created with an absolute target inside staging is about to become + // dangling; rewriting them to their future live targets now means a failure lands in the catch + // below, which restores the previous release rather than leaving a live component that cannot + // resolve its dependencies. Done pre-swap for the compensation, not post-swap for convenience. + repointAttempted = true; + repointedLinks = await repointStagedDependencyLinks(stagingDirPath, application.dirPath); + if (repointedLinks) { + logger.debug?.( + `Re-pointed ${repointedLinks} dependency link(s) in ${application.name} from the staging path to the live path` + ); + } + await rename(stagingDirPath, application.dirPath); + swapped = true; + await hooks.beforeCommit?.(); + } catch (error) { + const rollbackErrors: unknown[] = []; + if (swapped) { + try { + await rename(application.dirPath, stagingDirPath); + } catch (rollbackError) { + rollbackErrors.push(rollbackError); + } + } + if (repointAttempted) { + // The candidate is going back to staging, so its links have to point at staging again. Left + // aimed at the live path they would resolve against whatever release is live, so a retry of + // this same deployment id would validate the staged tree against the wrong bytes. + try { + // The candidate now sits at the staging path (moved back above, or never swapped), while its + // links still point at the live path: walk it there, and aim them back at staging. + await repointStagedDependencyLinks(stagingDirPath, stagingDirPath, application.dirPath); + } catch (rollbackError) { + rollbackErrors.push(rollbackError); + } + } + if (backupPath) { + try { + await rename(backupPath, application.dirPath); + } catch (rollbackError) { + rollbackErrors.push(rollbackError); + } + } + if (newMarkerPath) + await rm(newMarkerPath, { force: true }).catch((rollbackError) => rollbackErrors.push(rollbackError)); + try { + await hooks.onRollback?.(); + } catch (rollbackError) { + rollbackErrors.push(rollbackError); + } + if (rollbackErrors.length) { + throw new AggregateError( + [error, ...rollbackErrors], + `Failed to activate and fully restore ${application.name}` + ); + } + throw error; + } finally { + broadcastDeployEnd(application.name, deployLifecycleId); + } + // The displaced tree is the component's rollback source now, not garbage: retain it as + // `.deploy-previous/` with a manifest recording which deployment produced it. Best-effort — + // a component that can't retain its previous version is still deployed, just not revertable. + await retainActivatedPrevious( + application, + backupPath, + deploymentId, + hooks.activationSpec ? applicationConfigFromActivationSpec(hooks.activationSpec) : undefined, + outgoing + ); + if (newMarkerPath) await rm(newMarkerPath, { force: true }); + await rmdir(activationDir).catch((error) => { + if (!['ENOENT', 'ENOTEMPTY'].includes((error as NodeJS.ErrnoException).code ?? '')) throw error; + }); + }); + await rm(dirname(stagingDirPath), { recursive: true, force: true }).catch((error) => + logger.warn(`Failed to remove committed deploy staging for ${application.name}:`, errorForLog(error)) + ); +} + +export async function discardStagedApplication(componentDirPath: string, deploymentId: string): Promise { + const stagingDirPath = stagedApplicationPath(componentDirPath, deploymentId); + await withComponentPreparationLock(componentDirPath, async () => { + if (!(await secureStagingDeploymentDirectory(componentDirPath, deploymentId, false))) return; + await rm(dirname(stagingDirPath), { recursive: true, force: true }); + }); +} + +export async function discardProjectStagedApplications(componentDirPath: string): Promise { + const stagingRoot = join(dirname(componentDirPath), DEPLOY_STAGING_DIR); + if (!(await ensureSecureDirectory(stagingRoot, false, 'Component deploy staging path'))) return; + for (const entry of await readdir(stagingRoot, { withFileTypes: true })) { + if (!entry.isDirectory() || !DEPLOYMENT_ID_PATTERN.test(entry.name)) continue; + const deploymentDirPath = join(stagingRoot, entry.name); + if (existsSync(join(deploymentDirPath, basename(componentDirPath)))) { + await rm(deploymentDirPath, { recursive: true, force: true }); + } + } +} + +export async function discardProjectActivationArtifacts(componentDirPath: string): Promise { + const activationRoot = join(dirname(componentDirPath), DEPLOY_ACTIVATION_DIR); + if (!(await ensureSecureDirectory(activationRoot, false, 'Component activation staging path'))) return; + await rm(activationStagingDirectory(componentDirPath), { recursive: true, force: true }); + await rmdir(activationRoot).catch((error) => { + if (!['ENOENT', 'ENOTEMPTY'].includes((error as NodeJS.ErrnoException).code ?? '')) throw error; + }); +} + +type DeploymentLookup = (deploymentId: string) => Promise | undefined>; + +function safeComponentName(name: unknown): name is string { + return typeof name === 'string' && /^[a-zA-Z0-9_-]+$/.test(name); +} + +function activationArtifactDeploymentId(name: string): string | undefined { + const prefix = name.startsWith(ACTIVATION_BACKUP_PREFIX) + ? ACTIVATION_BACKUP_PREFIX + : name.startsWith(ACTIVATION_NEW_PREFIX) + ? ACTIVATION_NEW_PREFIX + : undefined; + if (!prefix) return undefined; + const deploymentId = name.slice(prefix.length, prefix.length + 36); + return DEPLOYMENT_ID_PATTERN.test(deploymentId) ? deploymentId : undefined; +} + +/** `\\?\C:\x` → `C:\x`. Windows readlink reports junctions in the extended-length form. */ +function stripExtendedLengthPrefix(target: string): string { + return target.startsWith('\\\\?\\') ? target.slice(4) : target; +} + +/** + * Whether a usable live component occupies `livePath`. A `file:` directory deploy is materialized as a + * symlink by design, so requiring a real directory would report a perfectly good live component as + * missing — and recovery would then treat its displaced release as residue. + */ +async function liveComponentPresent(livePath: string): Promise { + const linkStat = await statIfPresent(livePath); + if (!linkStat) return false; + if (linkStat.isSymbolicLink()) return !!(await statIfPresent(livePath, true))?.isDirectory(); + return linkStat.isDirectory(); +} + +/** + * Finish the retention half of an interrupted activation. The tree the swap displaced is still parked + * as `.deploy-activating//.previous--…`, and clearing it as residue is exactly what makes + * a recovered deploy unrevertable — the crash shape here is "swapped and committed, but died before + * retaining". Runs the same protocol a normal activation does, so the manifest describes the tree. + */ +async function retainRecoveredActivation( + componentDirPath: string, + componentName: string, + deploymentId: string, + activationSpec: Record | undefined +): Promise { + const backupPath = (await activationArtifacts(componentDirPath, deploymentId)).find((candidate) => + basename(candidate).startsWith(ACTIVATION_BACKUP_PREFIX) + ); + if (!backupPath) return; + // Which side names the displaced release depends on how far the original activation got. An + // untouched pre-activation manifest still has it as `live`; one written by the failed-retain path + // above already describes the intended end state, so there it is `previous`. `live` matching the + // deployment being recovered is what tells the two apart — without this the recovered manifest + // retains the right bytes under a null id, and revert_component cannot address them. + const manifest = await readRetainedPreviousManifest(componentDirPath); + const displaced = manifest?.live?.deployment_id === deploymentId ? manifest?.previous : manifest?.live; + const outgoing: RetainedVersion = displaced ?? { deployment_id: null, application_config: null }; + await retainActivatedPrevious( + new Application({ name: componentName }), + backupPath, + deploymentId, + activationSpec ? applicationConfigFromActivationSpec(activationSpec) : undefined, + outgoing + ); +} + +/** + * The project a staged deployment belongs to, read from `//`. Used to + * attribute a reconciliation failure when the deployment row itself could not be read. + */ +async function stagedDeploymentProjectName(deploymentPath: string): Promise { + const entries = await readdir(deploymentPath, { withFileTypes: true }).catch(() => []); + const projects = entries.filter((entry) => entry.isDirectory() && safeComponentName(entry.name)); + return projects.length === 1 ? projects[0].name : undefined; +} + +async function removeActivationArtifacts(componentDirPath: string, deploymentId: string): Promise { + for (const artifact of await activationArtifacts(componentDirPath, deploymentId)) { + await rm(artifact, { recursive: true, force: true }); + } + await rmdir(activationStagingDirectory(componentDirPath)).catch(() => {}); +} + +export async function reconcileStagedApplicationArtifacts( + componentsRootDirPath: string, + getDeployment: DeploymentLookup, + persistActivation: (row: Record) => Promise, + settleStagedDeployment?: (deploymentId: string, reason: string) => Promise +): Promise<{ + recovered: string[]; + removed: string[]; + errors: Map; + failedProjects: Map; +}> { + const recovered = new Set(); + const removed: string[] = []; + const errors = new Map(); + // Same failures as `errors`, keyed by COMPONENT rather than deployment id. An activation whose + // persistent work could not be completed leaves the live tree and its durable configuration + // disagreeing, so the caller has to be able to keep that specific component from loading — which it + // cannot do from a deployment id. + const failedProjects = new Map(); + const stagingRoot = join(componentsRootDirPath, DEPLOY_STAGING_DIR); + let deploymentEntries = []; + try { + if (await ensureSecureDirectory(stagingRoot, false, 'Component deploy staging path')) { + deploymentEntries = await readdir(stagingRoot, { withFileTypes: true }); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + + for (const entry of deploymentEntries) { + const deploymentPath = join(stagingRoot, entry.name); + if (!entry.isDirectory() || !DEPLOYMENT_ID_PATTERN.test(entry.name)) { + await rm(deploymentPath, { recursive: true, force: true }); + removed.push(entry.name); + continue; + } + // Declared outside the try so the catch can attribute a reconciliation failure to its component. + let row: Record | undefined; + // Same reason: durable evidence that an activation began, which the catch has to treat exactly like + // an `activating` row when deciding whether to fail the component closed. + let activationBegan = false; + try { + row = await getDeployment(entry.name); + if (!row || !safeComponentName(row.project)) { + await rm(deploymentPath, { recursive: true, force: true }); + removed.push(entry.name); + continue; + } + const componentDirPath = join(componentsRootDirPath, row.project); + // Local activation evidence is consulted BEFORE any status-based cleanup, because the row is the + // less reliable of the two. A crash between the swap and the status/config commit can leave the + // row `loading`, already terminal (a replicated origin write), or absent entirely when tracking + // is unavailable — while the candidate is live on disk. Deleting staging state on the strength of + // that status destroys the only evidence, and the artifact sweep then neither persists config nor + // fails the component closed: swapped-in code loads under the previous release's configuration. + // Read under the lock so an in-flight activation cannot create artifacts between probe and act. + await withComponentPreparationLock(componentDirPath, async () => { + activationBegan = (await activationArtifacts(componentDirPath, entry.name)).length > 0; + }); + if (!activationBegan && !['staged', 'activating'].includes(row.status)) { + let shouldRemove = false; + await withComponentPreparationLock(componentDirPath, async () => { + row = await getDeployment(entry.name); + shouldRemove = !row || !safeComponentName(row.project) || !['staged', 'activating'].includes(row.status); + if (!shouldRemove) return; + // Re-read the evidence under this lock too: an activation may have started since the probe + // above, and its artifacts outrank the row. + if ((await activationArtifacts(componentDirPath, entry.name)).length > 0) { + activationBegan = true; + shouldRemove = false; + return; + } + // A row still `pending`/`staging` once its staging directory is going away cannot make + // progress — the process that owned it is gone. Payload retention only reclaims rows that + // reached a terminal status, so leaving it in flight pins its tarball on every node forever. + if (row?.status === 'pending' || row?.status === 'staging') { + await settleStagedDeployment?.(entry.name, 'the deploy did not survive a restart'); + } + await rm(deploymentPath, { recursive: true, force: true }); + }); + if (shouldRemove) { + removed.push(entry.name); + continue; + } + } + const stagedPath = stagedApplicationPath(componentDirPath, entry.name); + // Local evidence outranks the replicated row's status. A peer swaps WITHOUT writing the row — + // the origin owns it — so a peer that died between its swap and its config commit has a + // `staged` row and no staged leaf. Read as a broken candidate that settled the row `failed` + // (over the origin's own row) and left new code live under the previous release's config. An + // activation artifact for this deployment is proof the swap began, so it is an interrupted + // activation and belongs in the roll-forward path below. + if (row.status === 'staged' && !activationBegan) { + if (!(await hasCompleteStagedApplication(stagedPath))) { + let discarded = false; + await withComponentPreparationLock(componentDirPath, async () => { + // Re-read BOTH signals under the lock. The probes above are an unlocked fast path, and an + // activation can create its backup and rename the staged tree live in between — settling on + // the stale read would patch the origin-owned row `failed` and delete a live activation's + // deployment directory. + if ((await activationArtifacts(componentDirPath, entry.name)).length > 0) { + activationBegan = true; + return; + } + if (await hasCompleteStagedApplication(stagedPath)) return; + // Only a not-yet-activated candidate is broken; the live tree and its persisted config are + // consistent. Failing the component closed here would take a healthy component offline and + // keep it offline, because nothing else sweeps a staging directory whose subtree is missing. + logger.warn( + `Discarding staged deployment '${entry.name}' for '${row.project}': no valid component tree. ` + + `The live component is unaffected.` + ); + await settleStagedDeployment?.(entry.name, 'its staged component tree was incomplete'); + await rm(deploymentPath, { recursive: true, force: true }); + discarded = true; + }); + if (discarded) removed.push(entry.name); + } + // An activation that appeared under the lock belongs in the roll-forward path below. + if (!activationBegan) continue; + } + // Reached for `activating` rows and for ANY row status backed by activation evidence. + if (await hasCompleteStagedApplication(stagedPath)) { + await activateStagedApplication(new Application({ name: row.project }), entry.name, { + beforeCommit: () => persistActivation(row), + // A recovered roll-forward retains its displaced tree the same as a normal activation, so a + // deploy that crashed mid-swap is still revertable once the node is back up. + activationSpec: row.activation_spec, + }); + } else { + let ownedByActivation = false; + await withComponentPreparationLock(componentDirPath, async () => { + // Re-read under the lock for the same reason as the `staged` branch above: if a complete + // candidate is here now, an in-flight activation owns these artifacts and will settle them. + if (await hasCompleteStagedApplication(stagedPath)) { + ownedByActivation = true; + return; + } + if (!(await liveComponentPresent(componentDirPath))) { + throw new Error(`Interrupted activation '${entry.name}' has neither a staged nor live component tree`); + } + await persistActivation(row); + // The swap already happened, so the displaced tree is the rollback source now. + await retainRecoveredActivation(componentDirPath, row.project, entry.name, row.activation_spec); + }); + if (ownedByActivation) continue; + } + await removeActivationArtifacts(componentDirPath, entry.name); + recovered.add(entry.name); + } catch (error) { + const reconcileError = error instanceof Error ? error : new Error(String(error)); + errors.set(entry.name, reconcileError); + // Fail the component closed for an interrupted activation, where the live tree and its durable + // configuration can disagree. A confirmed `staged` failure leaves live state consistent, so it + // does not. + if ((row?.status === 'activating' || activationBegan) && safeComponentName(row?.project)) { + // `activationBegan` counts the same as an `activating` row: a swap that already happened is an + // interrupted activation whatever the replicated status says, so a rejecting persistence step + // must not leave the swapped candidate loading under the previous release's configuration. + failedProjects.set(row!.project, reconcileError); + } else if (!row) { + // The row is what says whether this was an interrupted activation, so an unreadable row is + // the one case we cannot rule that out. The project name does not depend on the row — it is + // the directory under the deployment id — so attribute from disk rather than fail open and + // load a component whose live tree may disagree with its durable config. + const project = await stagedDeploymentProjectName(deploymentPath); + if (project) failedProjects.set(project, reconcileError); + } + } + } + + const activationRoot = join(componentsRootDirPath, DEPLOY_ACTIVATION_DIR); + if (await ensureSecureDirectory(activationRoot, false, 'Component activation staging path')) { + for (const projectEntry of await readdir(activationRoot, { withFileTypes: true })) { + const projectPath = join(activationRoot, projectEntry.name); + if (!projectEntry.isDirectory() || !safeComponentName(projectEntry.name)) { + await rm(projectPath, { recursive: true, force: true }); + continue; + } + const livePath = join(componentsRootDirPath, projectEntry.name); + // This sweep renames a backup back over the live path, so it must hold the lock every other + // mutator of that path holds. A reload cycle can retry reconciliation while an activation is + // mid-swap, and an unlocked sweep would restore the backup underneath it — after which the + // in-flight rename fails ENOTEMPTY and its own compensation fails ENOENT. Waiting out a live + // owner is what makes the decisions below sound: they are read from settled state, not a + // half-finished swap. + await withComponentPreparationLock(livePath, async () => { + for (const artifact of await readdir(projectPath, { withFileTypes: true })) { + const deploymentId = activationArtifactDeploymentId(artifact.name); + const artifactPath = join(projectPath, artifact.name); + if (!deploymentId) { + await rm(artifactPath, { recursive: true, force: true }); + continue; + } + let row: Record | undefined; + try { + row = await getDeployment(deploymentId); + } catch (lookupError) { + // Cannot tell absent from unreadable, so destroy nothing and fail the component closed. + const reconcileError = lookupError instanceof Error ? lookupError : new Error(String(lookupError)); + errors.set(deploymentId, reconcileError); + failedProjects.set(projectEntry.name, reconcileError); + continue; + } + const liveUsable = await liveComponentPresent(livePath); + if (row?.status === 'activating' && row.project === projectEntry.name && liveUsable) { + try { + await persistActivation(row); + if (artifact.name.startsWith(ACTIVATION_BACKUP_PREFIX)) { + // The displaced tree, not residue: retaining it is what keeps a recovered deploy + // revertable. The retain consumes the artifact, so there is nothing left to remove. + await retainRecoveredActivation(livePath, projectEntry.name, deploymentId, row.activation_spec); + } else { + await rm(artifactPath, { recursive: true, force: true }); + } + recovered.add(deploymentId); + } catch (error) { + const reconcileError = error instanceof Error ? error : new Error(String(error)); + errors.set(deploymentId, reconcileError); + failedProjects.set(projectEntry.name, reconcileError); + } + } else if (artifact.name.startsWith(ACTIVATION_BACKUP_PREFIX) && !liveUsable) { + // `!liveUsable` rather than `!liveStat`: a dangling symlink at the live path is an occupant + // that cannot serve, and rename replaces it. Keying on lstat left the component pointing + // at nothing with its recoverable backup sitting right there. + await rename(artifactPath, livePath); + } else if ( + artifact.name.startsWith(ACTIVATION_BACKUP_PREFIX) && + row && + liveUsable && + (await readRetainedPreviousManifest(livePath))?.live?.deployment_id === deploymentId + ) { + // The activation finished — settled row, good live tree — but a backup is still parked, + // which only happens when `retainActivatedPrevious` failed inside its best-effort catch. + // That backup is the sole remaining copy of the release this deploy displaced, so deleting + // it as residue is what makes the deploy permanently unrevertable. Finish the retention. + // + // Gated on the manifest naming THIS deployment as live. A settled row proves the + // activation ended, not that it ended as the current release: an artifact left by an older + // deployment would otherwise overwrite a valid retained previous with stale bytes and + // write a manifest naming a release that is no longer live. + try { + await retainRecoveredActivation(livePath, projectEntry.name, deploymentId, row.activation_spec); + } catch (error) { + const reconcileError = error instanceof Error ? error : new Error(String(error)); + errors.set(deploymentId, reconcileError); + } + } else if (artifact.name.startsWith(ACTIVATION_BACKUP_PREFIX)) { + // A backup artifact still here means the activation never finished retaining it, so this is + // the only copy of the tree it displaced. It is deleted only when the state positively says + // it is residue. An absent row does not: the lookup returns undefined both for a row + // retention reclaimed and for a deployment table that was never provisioned. Neither does + // an unusable live path, which is how a crash mid-swap looks. + logger.warn( + `Keeping displaced component tree '${artifact.name}' for '${projectEntry.name}': ` + + `its deployment row or live tree could not be confirmed, so it is not provably disposable` + ); + } else { + await rm(artifactPath, { recursive: true, force: true }); + } + } + }); + await rmdir(projectPath).catch(() => {}); + } + await rmdir(activationRoot).catch(() => {}); + } + await rmdir(stagingRoot).catch(() => {}); + return { recovered: [...recovered], removed, errors, failedProjects }; +} + /** * Install all applications specified in the root config. * @@ -2028,6 +3805,31 @@ export async function installApplications() { // in-memory state rather than a stale snapshot silently clobbering a sibling's just-written change. const applicationLockWriteQueues = new Map>(); +const PERSISTENT_STATE_LOCK_PURPOSE = 'application-persistent-state'; + +function persistentStateLockPath(): string { + return join(getConfigValue(CONFIG_PARAMS.ROOTPATH), 'harper-application-lock.json'); +} + +/** + * Serialize a root-config + application-lock read-modify-write across every isolate and process. The + * snapshot and both files have to sit inside one critical section: the entries are per-project but the + * files are shared, so an unsynchronized read-modify-write drops a sibling project's entry. + * + * Never call this while already holding it — the file lock is not reentrant. The `*Unlocked` variants + * exist for callers that are already inside it. + */ +export async function withPersistentStateLock(operation: () => Promise): Promise { + return withComponentPreparationLock(persistentStateLockPath(), operation, { + purpose: PERSISTENT_STATE_LOCK_PURPOSE, + timeoutMs: 30_000, + // Without this, a ticket left behind by a terminated worker still carries this process's pid and + // instance nonce, so it reads as live and is never reclaimed — wedging every later activation, + // revert and drop on a holder that no longer exists. + isOwnerAlive: (owner) => owner.pid !== process.pid || isThreadRunning(owner.threadId), + }); +} + async function persistApplicationLock( harperApplicationLockPath: string, harperApplicationLock: { applications: Record } @@ -2044,6 +3846,136 @@ async function persistApplicationLock( await next; } +function applicationConfigFromActivationSpec( + spec: Record | null | undefined +): ApplicationConfig | undefined { + // A row with no activation spec says nothing about config, so config is a no-op rather than a throw. + // Recovery reads `row.activation_spec` from rows it did not write — a row from before the spec was + // recorded, or a hand-edited one — and throwing there would fail that component closed on every boot + // with no way out but repairing the row by hand. + if (!spec?.package) return undefined; + const applicationConfig: ApplicationConfig = { package: spec.package }; + if (spec.install_command !== null || spec.install_timeout !== null || spec.install_allow_scripts !== null) { + applicationConfig.install = {}; + if (spec.install_command !== null) applicationConfig.install.command = spec.install_command; + if (spec.install_timeout !== null) applicationConfig.install.timeout = spec.install_timeout; + if (spec.install_allow_scripts !== null) applicationConfig.install.allowInstallScripts = spec.install_allow_scripts; + } + if (spec.urlPath !== null) applicationConfig.urlPath = spec.urlPath; + if (spec.host !== null) applicationConfig.host = spec.host; + if (spec.credentials?.length) applicationConfig.credentials = spec.credentials; + return applicationConfig; +} + +async function readApplicationLock(lockPath: string): Promise<{ applications: Record }> { + try { + const lock = JSON.parse(await readFile(lockPath, 'utf8')); + if (!lock.applications || typeof lock.applications !== 'object') lock.applications = {}; + return lock; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return { applications: {} }; + throw error; + } +} + +/** Persist a runtime activation into the boot-time application lock, or remove it during compensation. */ +export async function updateApplicationLockEntry( + name: string, + applicationConfig: ApplicationConfig | undefined +): Promise { + return withPersistentStateLock(() => updateApplicationLockEntryUnlocked(name, applicationConfig)); +} + +async function updateApplicationLockEntryUnlocked( + name: string, + applicationConfig: ApplicationConfig | undefined +): Promise { + const lockPath = join(getConfigValue(CONFIG_PARAMS.ROOTPATH), 'harper-application-lock.json'); + const previous = applicationLockWriteQueues.get(lockPath) ?? Promise.resolve(); + const next = previous + .catch(() => {}) + .then(async () => { + const lock = await readApplicationLock(lockPath); + if (applicationConfig === undefined) delete lock.applications[name]; + else lock.applications[name] = applicationConfig; + const tempPath = `${lockPath}.${process.pid}.${randomUUID()}.tmp`; + await writeFile(tempPath, JSON.stringify(lock, null, 2), 'utf8'); + await rename(tempPath, lockPath); + }); + applicationLockWriteQueues.set(lockPath, next); + await next; +} + +async function getApplicationLockEntryUnlocked(name: string): Promise { + const lockPath = join(getConfigValue(CONFIG_PARAMS.ROOTPATH), 'harper-application-lock.json'); + const previous = applicationLockWriteQueues.get(lockPath) ?? Promise.resolve(); + let entry: ApplicationConfig | undefined; + const read = previous + .catch(() => {}) + .then(async () => { + entry = (await readApplicationLock(lockPath)).applications[name]; + }); + applicationLockWriteQueues.set(lockPath, read); + await read; + return entry; +} + +/** + * Move a component's persisted root-config entry and boot-time install-lock entry to `nextConfig` as + * one reversible step, snapshotting the current values at commit time so `rollback()` restores exactly + * what was there. + * + * `nextConfig: null` is an instruction to REMOVE the entry, not to leave it alone — the case that + * matters when reverting away from a `package` deploy to a payload one, where a stale `package:` entry + * would let installApplications() reinstall the reverted-away version on the next cold start. + * `undefined` means "this caller has nothing to say about config" and the transaction is a no-op. + */ +export async function createApplicationConfigTransaction( + project: string, + nextConfig: ApplicationConfig | null | undefined +): Promise<{ commit(): Promise; rollback(): Promise }> { + if (nextConfig === undefined) return { commit: async () => {}, rollback: async () => {} }; + let previousConfig: ApplicationConfig | undefined; + let previousLockConfig: ApplicationConfig | undefined; + let commitStarted = false; + return { + async commit() { + if (commitStarted) return; + // Snapshot and both writes inside one critical section, so a concurrent activation of a different + // project cannot land between the read and the write and lose one of the two entries. + await withPersistentStateLock(async () => { + // Re-checked inside the critical section: the guard above is only a fast path, so two + // concurrent commits on this transaction would both pass it and repeat the read-modify-write. + if (commitStarted) return; + previousConfig = readConfigFile()?.[project]; + previousLockConfig = await getApplicationLockEntryUnlocked(project); + commitStarted = true; + if (nextConfig === null) deleteConfigFromFile([project]); + else await addConfig(project, nextConfig); + await updateApplicationLockEntryUnlocked(project, nextConfig ?? undefined); + }); + }, + async rollback() { + if (!commitStarted) return; + await withPersistentStateLock(async () => { + if (previousConfig === undefined) deleteConfigFromFile([project]); + else await addConfig(project, previousConfig); + await updateApplicationLockEntryUnlocked(project, previousLockConfig); + commitStarted = false; + }); + }, + }; +} + +export async function createApplicationActivationTransaction( + project: string, + spec: Record +): Promise<{ commit(): Promise; rollback(): Promise }> { + // A payload deploy has no package reference to persist, so activating from it says nothing about + // root config and must leave whatever is there alone (undefined, not null). + return createApplicationConfigTransaction(project, applicationConfigFromActivationSpec(spec)); +} + /** * Keep the boot-time application lock honest while a reinstall is in flight. Factored as an * explicit production seam so the failure transition can be tested without replacing module diff --git a/components/componentLoader.ts b/components/componentLoader.ts index 250b5f3d64..a5c5d73b85 100644 --- a/components/componentLoader.ts +++ b/components/componentLoader.ts @@ -53,9 +53,14 @@ import { materializeGlobalSecrets, processComponentEnv } from './componentSecret import { PluginModule } from './PluginModule.ts'; import { getEnvBuiltInComponents, + createApplicationActivationTransaction, + reconcileStagedApplicationArtifacts, recoverInterruptedComponentExtraction, recoverInterruptedComponentExtractions, + recoverInterruptedReverts, } from './Application.ts'; +import { getDeploymentRow, markDeploymentTerminal } from './deploymentRecorder.ts'; +import { hostname } from 'node:os'; import { ComponentPreparationLockTimeoutError } from './componentPreparationLock.ts'; import { pathToFileURL } from 'node:url'; @@ -163,15 +168,120 @@ export async function loadComponentDirectories( if (loadedResources) resources = loadedResources; if (loadedPluginModules) loadedComponents = loadedPluginModules; const cycleResources = resources; - let failedRecoveries = new Map(); + // Order matters: both directory repairs must settle the live path before the staged reconciliation + // decides whether to roll an activation forward. The three passes read disjoint directories + // (`.deploy-aside`, `.deploy-previous`, `.deploy-staging`/`.deploy-activating`), so none can claim + // another's artifacts — see DESIGN.md, "One contract on .deploy-aside". + let failedRecoveries: Map; try { failedRecoveries = await recoverInterruptedComponentExtractions(CF_ROUTES_DIR); } catch (error) { - const recoveryError = error instanceof Error ? error : new Error(String(error)); - harperLogger.warn( - 'Loading existing filesystem components without deploy recovery because staging could not be inspected:', - errorForLog(recoveryError) + // The scan itself failed, so we cannot tell WHICH components are mid-extraction and cannot fail just + // those closed. A missing staging root is not this case — it returns no work — so reaching here means + // the root is unreadable or is not a directory, and any component may hold a half-extracted tree + // whose rollback record we could not read. Abort startup, the same way both scans below do. + harperLogger.error( + 'Could not inspect component deploy staging for interrupted extractions:', + errorForLog(error as Error) ); + throw error; + } + if (isMainThread) { + // A revert that died between its renames can leave a component with no live directory at all, with + // the bytes parked under `.deploy-previous/.reverting-*`. Repaired before anything loads, and + // before the staged reconciliation below, for the same reason extraction recovery runs first: the + // roll-forward decision should see a settled live directory. + try { + for (const [component, error] of await recoverInterruptedReverts(CF_ROUTES_DIR)) { + if (!failedRecoveries.has(component)) failedRecoveries.set(component, error); + } + } catch (error) { + // The scan itself failed, so we cannot tell WHICH components are mid-revert and cannot fail just + // those closed. An interrupted revert can have the reverted-to bytes live with config not yet + // committed, so loading everything over unreconciled state is not sound — abort startup instead, + // the same way an unreadable staged-artifact scan does below. + harperLogger.error( + 'Could not inspect retained component versions for interrupted reverts:', + errorForLog(error as Error) + ); + throw error; + } + } + // Every main-thread load cycle, not just the first. A reload cycle (loadComponentDirectories + + // restartWorkers) runs long after startup, and an activation that failed at runtime AND failed to + // compensate leaves exactly the inconsistent staged/live/backup state this pass exists to settle. + // Skipping it there would load the candidate against old or partial configuration until a cold + // restart. Idempotent and cheap once settled: the scan is a readdir of a directory this pass empties. + if (isMainThread) { + try { + const settleDiscardedDeployment = async (deploymentId: string, reason: string) => { + await markDeploymentTerminal(deploymentId, 'failed', new Error(reason)); + }; + const reconciliation = await reconcileStagedApplicationArtifacts( + CF_ROUTES_DIR, + getDeploymentRow, + async (row) => { + const transaction = await createApplicationActivationTransaction(row.project, row.activation_spec); + try { + await transaction.commit(); + } catch (error) { + try { + await transaction.rollback(); + } catch (rollbackError) { + throw new AggregateError( + [error, rollbackError], + `Could not persist or roll back the recovered component activation for '${row.project}'` + ); + } + throw error; + } + }, + settleDiscardedDeployment + ); + for (const deploymentId of reconciliation.recovered) { + // The row is created by the ORIGIN only, so this node may be settling a deployment it did + // not originate. Settling one it did own is the whole story — the activation it started is + // now complete — while a peer stamping the origin's row would report on nodes it cannot see. + // A row left `activating` is not stranded forever either way: the next deploy of the project + // settles it. + let settled = false; + try { + const row = await getDeploymentRow(deploymentId); + if (row?.origin_node === hostname()) { + await markDeploymentTerminal(deploymentId, 'success'); + settled = true; + } + } catch (error) { + // Observability only; a component that is live and reconciled must not fail to load + // because its row could not be updated. + harperLogger.warn( + `Could not settle the deployment row for recovered activation '${deploymentId}':`, + errorForLog(error as Error) + ); + } + harperLogger.warn( + `Rolled forward interrupted component activation '${deploymentId}' on this node` + + (settled ? '' : `; its deployment row still reads activating`) + ); + } + for (const [deploymentId, error] of reconciliation.errors) { + harperLogger.error(`Could not reconcile staged component deployment '${deploymentId}':`, errorForLog(error)); + } + // FAIL CLOSED. An activation we could not finish leaves the live tree and its durable + // configuration disagreeing — the swapped-in code with the previous release's root config, or no + // live directory at all after an interrupted rename — while the deployment row still claims the + // activation is incomplete. Loading that is worse than not loading it: the component serves + // requests whose behavior nobody can predict from either the code or the config, and the next + // cold start may reinstall over it. So the affected component does not load. + for (const [project, error] of reconciliation.failedProjects) { + if (!failedRecoveries.has(project)) failedRecoveries.set(project, error); + } + } catch (error) { + // The scan itself failed, so we cannot tell WHICH components are affected and cannot fail just + // those closed. Abort startup rather than load every component over possibly-unreconciled state. + harperLogger.error('Could not inspect staged component deployments during startup:', errorForLog(error as Error)); + throw error; + } } // Materialize hdb_secret global-tier rows into process.env and snapshot the scoped tier before // any application loads (root components — including the Pro custody registration — have @@ -371,6 +481,10 @@ let errorReporter; export function setErrorReporter(reporter) { errorReporter = reporter; } +/** So a caller that installs a reporter can put the previous one back when it is done with it. */ +export function getErrorReporter() { + return errorReporter; +} let compName: string; export const getComponentName = () => compName; diff --git a/components/deploymentOperations.ts b/components/deploymentOperations.ts index adf4af5170..3fffc32294 100644 --- a/components/deploymentOperations.ts +++ b/components/deploymentOperations.ts @@ -101,8 +101,8 @@ export async function handleGetDeployment(req: GetRequest): Promise { // SSE content-negotiated branch — when serverHandlers.js detects // `Accept: text/event-stream` it attaches a ProgressEmitter as req.progress and wraps // our return as the operation's final SSE event. We replay event_log on connect, then - // tail the deployment's live emitter (if it's still running on this node) until it - // reaches a terminal status. The final return value becomes the SSE `done` event. + // tail the deployment's live emitter (if it's still running on this node) until its + // recorder finishes. The final return value becomes the SSE `done` event. if (req.progress && typeof (req.progress as any).emit === 'function') { const sse = req.progress; const liveEmitter = getActiveEmitter(req.deployment_id); @@ -114,22 +114,28 @@ export async function handleGetDeployment(req: GetRequest): Promise { let lastReplayedTs = 0; let resolveLive: (() => void) | null = null; let liveDone = false; + // Held out here so the terminal-event path can stop it immediately. Otherwise it survives up to one + // more interval after the request is already settled. + let pollTimer: ReturnType | undefined; const liveBuffer: Array<{ t: number; event: { event: string; data: unknown } }> = []; + // A terminal signal — either an explicit success/error event from the lifecycle, or the recorder's + // `_recorder_finished` sentinel emitted before it unsubscribes. + const isTerminalEvent = (e: { event: string; data: unknown }) => + e.event === '_recorder_finished' || + e.event === 'error' || + (e.event === 'phase' && + e.data && + typeof e.data === 'object' && + (e.data as { phase?: string }).phase === 'success'); + const settleLive = () => { + if (liveDone) return; + liveDone = true; + if (pollTimer) clearInterval(pollTimer); + resolveLive?.(); + }; const forwardLive = (e: { event: string; data: unknown }) => { sse.emit(e.event, e.data); - // A terminal signal — either explicit success/error event from the lifecycle, or - // the recorder's `_recorder_finished` sentinel emitted before it unsubscribes. - const isTerminalEvent = - e.event === '_recorder_finished' || - e.event === 'error' || - (e.event === 'phase' && - e.data && - typeof e.data === 'object' && - (e.data as { phase?: string }).phase === 'success'); - if (isTerminalEvent && !liveDone) { - liveDone = true; - resolveLive?.(); - } + if (isTerminalEvent(e)) settleLive(); }; const unsubscribe = liveEmitter ? liveEmitter.subscribe((event) => { @@ -150,31 +156,41 @@ export async function handleGetDeployment(req: GetRequest): Promise { if (typeof entry.t === 'number') lastReplayedTs = Math.max(lastReplayedTs, entry.t); } - if (liveEmitter && !TERMINAL_STATUSES.has(row.status)) { + // A full two-phase deploy checkpoints `staged` before immediately entering activation. + // The live emitter, not that transient row status, is authoritative for whether this + // origin is still running the deployment. + if (liveEmitter) { await new Promise((resolve) => { resolveLive = resolve; - // Flush anything that arrived during replay, filtering to events the replay missed. + // Flush anything that arrived during replay, filtering to events the replay missed. A + // deduplicated event must still be inspected for terminal-ness: dropping it outright would + // lose the only signal that the deployment finished, and the request would then hang until + // the fallback timer happened to notice the emitter was gone — or forever, if it had not. for (const buffered of liveBuffer) { if (buffered.t > lastReplayedTs) forwardLive(buffered.event); + else if (isTerminalEvent(buffered.event)) settleLive(); } if (liveDone) resolve(); // Safety net — if the in-memory emitter is dropped (recorder finished or // the process recycled) before signaling, poll the row's status as a // fallback so the client never hangs indefinitely. - const pollTimer = setInterval(async () => { + pollTimer = setInterval(() => { if (liveDone) { clearInterval(pollTimer); return; } const live = getActiveEmitter(req.deployment_id); - if (!live || live !== liveEmitter) { - clearInterval(pollTimer); - const latest = await table.get(req.deployment_id); - if (latest && TERMINAL_STATUSES.has(latest.status) && !liveDone) { - liveDone = true; - resolve(); - } - } + if (live && live === liveEmitter) return; + // The emitter this request was tailing is gone, so there is no local work left to + // follow: `staged` is a valid resting result and `activating` records uncertain + // cluster completion. Settle unconditionally — including when the row has been + // reclaimed — because clearing the timer without resolving hangs the request + // forever. Nothing is read here on purpose: an await inside a timer callback can + // reject outside the promise being awaited, which is an unhandled rejection AND + // leaves this promise unsettled. The row is re-read below for the final payload. + clearInterval(pollTimer); + liveDone = true; + resolve(); }, 500); }); } diff --git a/components/deploymentRecorder.ts b/components/deploymentRecorder.ts index 47759da942..39a3a8350c 100644 --- a/components/deploymentRecorder.ts +++ b/components/deploymentRecorder.ts @@ -6,8 +6,8 @@ // payload_blob (with sha256 + size), and writes the terminal status at the end. // Subscribes to a ProgressEmitter so phase transitions and install lines land in // event_log as they happen — making the deploy observable by Studio polling -// get_deployment without an attached CLI. The persisted payload_blob will also serve -// as the rollback source when that operation lands. +// get_deployment without an attached CLI. The persisted payload_blob also serves as +// the replication source for peer staging. import { randomUUID } from 'node:crypto'; import { createHash } from 'node:crypto'; @@ -50,8 +50,15 @@ type DeploymentStatus = | 'pending' | 'extracting' | 'installing' + | 'staging' + // Reserved for the coordination protocol (#2301), which is the only producer of a resting `staged` + // row. Nothing in this tree leaves a deployment there. + | 'staged' | 'loading' | 'replicating' + | 'activating' + // Retained for compatibility with deployment rows written by earlier preview builds. + | 'reverting' | 'restarting' | 'success' | 'failed' @@ -64,10 +71,10 @@ interface CreateOptions { restart_mode?: 'immediate' | 'rolling' | null; rollback_of?: string | null; // Deploy credentials in reference form (`{ registry, secret, scope? }` / `{ host, secret, - // username? }`) — never a literal token. Kept so a rollback can re-resolve the credential from - // hdb_secret without the operator re-supplying it. Null when the deploy used no credentials or a - // no-custody transient token. + // username? }`) — never a literal token. Peers and later activation resolve the reference from + // hdb_secret. Null when the deploy used no credentials or a no-custody transient token. credentials?: CredentialReference[] | null; + activation_spec?: Record | null; emitter?: ProgressEmitter; // Open-transaction budget (ms) for the payload-ingest write, below. Mirrors the // `deployment_timeout` operation parameter already used for the peer-side row/blob waits @@ -113,6 +120,10 @@ export class DeploymentRecorder { private dirty = false; private sealed = false; private flushSuppressed = false; + // Set only on the degraded (no-table) ingest path, which buffers the whole upload in memory. The + // original source is exhausted by then, so this is the ONLY replayable copy — and with no row for + // peers to read a blob from, it is what has to travel in the replicated operation. + private bufferedPayload?: Buffer; private readonly ingestTimeoutMs?: number; private constructor(deploymentId: string, initial: Record, ingestTimeoutMs?: number) { @@ -142,6 +153,7 @@ export class DeploymentRecorder { user: options.user ?? null, rollback_of: options.rollback_of ?? null, credentials: options.credentials ?? null, + activation_spec: options.activation_spec ?? null, error: null, }; const recorder = new DeploymentRecorder(deploymentId, record, options.ingestTimeoutMs); @@ -227,6 +239,14 @@ export class DeploymentRecorder { * In-memory sources (a Buffer, or the legacy base64-in-JSON/CBOR body) are already * materialized by the time they reach us, so they take the simpler buffer path. */ + /** + * A replayable copy of the ingested payload, present only when ingest buffered it in memory — + * which is exactly the no-table case, where there is no row for peers to read the bytes from. + */ + get replayablePayload(): Buffer | undefined { + return this.bufferedPayload; + } + async ingestPayload(source: Readable | Buffer | string): Promise { this.flushSuppressed = true; try { @@ -255,6 +275,7 @@ export class DeploymentRecorder { // already holds. No cap — a large base64-in-JSON body is the caller's choice. if (Buffer.isBuffer(source) || typeof source === 'string') { const buffer = Buffer.isBuffer(source) ? source : Buffer.from(source, 'base64'); + this.bufferedPayload = buffer; hash.update(buffer); this.record.payload_blob = createBlob(buffer, { type: 'application/gzip' }); this.record.payload_hash = hash.digest('hex'); @@ -293,6 +314,7 @@ export class DeploymentRecorder { chunks.push(buf); } const buffer = Buffer.concat(chunks); + this.bufferedPayload = buffer; hash.update(buffer); this.record.payload_blob = createBlob(buffer, { type: 'application/gzip' }); this.record.payload_hash = hash.digest('hex'); @@ -465,17 +487,24 @@ export class DeploymentRecorder { this.sealed = true; } - async finish(status: 'success' | 'failed' | 'rolled_back', error?: unknown): Promise { + async checkpoint(status: DeploymentStatus, phase: string): Promise { if (this.finished) return; - // Send a terminal sentinel through the emitter (if any) BEFORE we unsubscribe and - // remove it from the registry, so any SSE tail subscribers can resolve their wait - // even on a code path that doesn't emit an explicit `error` event. + while (this.pendingPut) await this.pendingPut; + this.record.status = status; + this.record.phase = phase; + this.record.completed_at = null; + await this.put(); + } + + async finish(status: 'success' | 'failed' | 'rolled_back' | 'staged' | 'activating', error?: unknown): Promise { + if (this.finished) return; + // Keep the emitter registered until the terminal row write lands. An SSE tailer may + // otherwise observe the sentinel, re-read the preceding staged checkpoint, and + // incorrectly finish with a stale result. const emitter = activeEmitters.get(this.deploymentId); - emitter?.emit('_recorder_finished', { status }); this.finished = true; this.unsubscribe?.(); this.unsubscribe = null; - activeEmitters.delete(this.deploymentId); // Drain the ENTIRE coalesced-flush chain before mutating + persisting the terminal // state. Just awaiting `this.pendingPut` once isn't enough: its `.finally` may // re-schedule another put (when `dirty` was set during the in-flight put), and @@ -490,7 +519,7 @@ export class DeploymentRecorder { } } this.record.status = status; - this.record.completed_at = Date.now(); + this.record.completed_at = status === 'activating' ? null : Date.now(); if (error) { const e = error as { message?: string; code?: string | number; stack?: string }; this.record.error = { @@ -499,7 +528,15 @@ export class DeploymentRecorder { phase: this.record.phase, }; } - await this.put(); + try { + await this.put(); + } finally { + // Emit after persistence but before removing the registry entry so subscribers + // always have a durable terminal row to re-read. The sentinel also releases + // tailers on failure paths that did not emit an explicit error event. + emitter?.emit('_recorder_finished', { status }); + activeEmitters.delete(this.deploymentId); + } } get row(): Record { @@ -517,6 +554,162 @@ export class DeploymentRecorder { } } +/** + * Point-read a deployment row by id, or undefined when the row (or the table) is absent. + * + * Distinct from `awaitDeploymentRow`, which polls for a row to arrive by replication AND to carry a + * `payload_blob`: this is for reading a row the caller already owns locally — e.g. recovering a staged + * deployment's `package_identifier` when it is activated later by `deployment_id`. A row whose payload + * has been reclaimed by retention is a valid result here, which is exactly what awaitDeploymentRow + * would refuse to return. + */ +export async function getDeploymentRow(deploymentId: string): Promise | undefined> { + if (!deploymentId) return undefined; + const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; + if (!table) return undefined; + return table.get(deploymentId); +} + +/** + * Best-effort status update for an existing deployment row by id, used when a later operation + * finishes a deployment the current process didn't record with a live DeploymentRecorder — e.g. + * a build that an earlier stage left in the + * `staged` state. No-op when the row (or the table) is absent; observability only, so callers treat a + * failure here as non-fatal. + */ +export async function markDeploymentTerminal( + deploymentId: string, + status: 'success' | 'failed' | 'rolled_back' | 'staged' | 'activating', + error?: unknown +): Promise { + const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; + if (!table) return; + const row = await table.get(deploymentId); + if (!row) return; + // Patch (partial update) rather than mutating the row: table.get() returns a read-only record, so + // `row.status = …` throws "Cannot assign to read only property". A patch also touches only these two + // fields, so it can't truncate the rest of the row the way a spread-and-put might. + const update: Record = { + status, + completed_at: status === 'activating' ? null : Date.now(), + }; + if (error !== undefined) { + update.error = { + message: error instanceof Error ? error.message : String(error), + }; + } else { + // Cleared, not left alone: a recovery or a successful activate retry moves a row that already + // carries a failure, and keeping that object would report success and an error at once. + update.error = null; + } + await table.patch(deploymentId, update); +} + +export async function recordDeploymentPeers(deploymentId: string, results: unknown): Promise { + if (!Array.isArray(results) || results.length === 0) return; + const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; + if (!table) return; + const row = await table.get(deploymentId); + if (!row) return; + const peers = Array.isArray(row.peer_results) + ? row.peer_results.map((entry: unknown) => ({ ...(entry as object) })) + : []; + for (const result of results) { + const normalized = normalizePeerResult(result); + const index = normalized.node ? peers.findIndex((entry: any) => entry.node === normalized.node) : -1; + if (index >= 0) peers[index] = normalized; + else peers.push(normalized); + } + await table.patch(deploymentId, { peer_results: peers }); +} + +/** + * Whether deployment tracking is provisioned on this node. + * + * `DeploymentRecorder.put()` is deliberately tolerant — a deploy still works with no `hdb_deployment` + * table, because tracking is observability. This reports whether a row will actually exist, which the + * deploy path needs to decide whether peers can read the payload from the row or must be sent it. + */ +export function isDeploymentTrackingAvailable(): boolean { + return !!(databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; +} + +// Deployment statuses that are settled — a non-terminal deployment's payload_blob may still be the +// replication channel peers are installing from, so retention must never yank it mid-flight. Mirrors +// deploymentOperations.ts's guard for the explicit delete_deployment_payload operation. +const TERMINAL_STATUSES = new Set(['success', 'failed', 'rolled_back']); + +export async function invalidateProjectStagedDeployments(project: string): Promise { + const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; + if (!table) return []; + const invalidated: string[] = []; + for await (const row of table.search([{ attribute: 'project', value: project }])) { + // `staging` counts too: a deploy interrupted mid-stage leaves the row there, and nothing else + // settles it — payload retention only reclaims terminal rows, so the tarball would be pinned and + // `get_deployment` would never converge for a component that has since been dropped. + if (!['staging', 'staged', 'activating'].includes(row?.status)) continue; + await table.patch(row.deployment_id, { + status: 'failed', + completed_at: Date.now(), + error: { message: 'Staged build invalidated because the component was dropped', phase: row.phase }, + }); + invalidated.push(row.deployment_id); + } + return invalidated; +} + +/** + * Count-based payload retention: keep at most `maxCount` stored payload tarballs for a project, + * newest first, dropping the payload_blob of the rest. Rows are always retained — only the tarball + * bytes are reclaimed, so the audit trail (metadata + event_log) stays intact and `get_deployment` + * still reports the deployment; only `get_deployment_payload` stops being available for the pruned + * ones (`payload_blob_present: false`). + * + * Complements the size-based drop (`deployment_payloadRetention_maxSize`, which reclaims a single + * oversized payload right after its own deploy): this bounds how many payloads accumulate per project + * over time, which is what actually caps disk. Only rows that still HOLD a payload count toward + * `maxCount`, so the cap is literally "at most N stored payloads per project". + * + * Best-effort and observability-only — callers must not fail a deploy because pruning failed. + * Returns the total bytes reclaimed. + */ +export async function pruneProjectPayloads(project: string, maxCount: number): Promise { + if (!project) return 0; + if (!Number.isFinite(maxCount) || maxCount < 0) return 0; + const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; + if (!table) return 0; + + const withPayload: Array> = []; + for await (const row of table.search([{ attribute: 'project', value: project }])) { + if (row?.payload_blob != null) withPayload.push(row); + } + // Newest first by started_at; ties broken by deployment_id so the ordering (and therefore which + // payloads survive) is stable across runs — same tiebreak as handleListDeployments. + withPayload.sort( + (a, b) => (b.started_at ?? 0) - (a.started_at ?? 0) || String(a.deployment_id).localeCompare(b.deployment_id) + ); + + let freed = 0; + for (const row of withPayload.slice(maxCount)) { + // An in-flight deployment still counted toward maxCount above (its payload occupies disk) but + // must not be dropped — skip it and let a later prune reclaim it once it settles. + if (!TERMINAL_STATUSES.has(row.status)) continue; + const size = typeof row.payload_size === 'number' ? row.payload_size : 0; + // Patch only the field this prune owns. Writing the whole record back would rewrite fields a + // concurrent deploy just changed — reverting a status, dropping an error — while reclaiming a + // tarball. The drop is not appended to `event_log` either: that would be a read-copy-write of an + // append-only list, so a concurrent writer's entry would be lost. `payload_blob_present: false` + // already reports the outcome on the row, and the reclaim is logged here for the audit trail. + await table.patch(row.deployment_id, { payload_blob: null }); + logger.info?.( + `Reclaimed ${size} bytes of deployment payload for '${project}' (${row.deployment_id}): ` + + `beyond deployment_payloadRetention_maxCount=${maxCount}` + ); + freed += size; + } + return freed; +} + // Default peer-wait budget for the hdb_deployment row to replicate. A deploy is a rare, // heavyweight, user-initiated operation, and the `system`-table replication channel can be // backlogged behind unrelated writes when several deploys land in succession, so the row can @@ -569,7 +762,12 @@ export function ingestTransactionTimeoutMs(deploymentTimeout: unknown): number { */ export async function awaitDeploymentRow( deploymentId: string, - options: { timeoutMs?: number; pollIntervalMs?: number; initialPollIntervalMs?: number } = {} + options: { + timeoutMs?: number; + pollIntervalMs?: number; + initialPollIntervalMs?: number; + requirePayload?: boolean; + } = {} ): Promise> { // Coerce defensively: the deploy operation's `deployment_timeout` reaches us via the // operation body, and the Joi validator's coerced number is discarded by validateBySchema @@ -584,6 +782,7 @@ export async function awaitDeploymentRow( // human-noticeable latency, then back off exponentially up to maxIntervalMs for the // rare case where the row is genuinely still replicating. let intervalMs = options.initialPollIntervalMs ?? 5; + const requirePayload = options.requirePayload !== false; const table = (databases as any).system?.[terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME]; if (!table) { throw new Error( @@ -600,7 +799,7 @@ export async function awaitDeploymentRow( try { const row = await table.get(deploymentId); if (row) { - if (row.payload_blob != null) return row; + if (!requirePayload || row.payload_blob != null) return row; // Row replicated but its payload_blob write hasn't landed yet — replication is // alive, just mid-flight. Remember this so the timeout message points at the // payload write rather than a dead channel. @@ -709,7 +908,7 @@ async function* readPayloadBlobChunks( } } -function normalizePeerResult(raw: unknown): Record { +export function normalizePeerResult(raw: unknown): Record { if (!raw || typeof raw !== 'object') { // Replication layer returned a primitive — preserve as a stringified marker so the // audit row at least records that something came back from a peer. @@ -744,10 +943,16 @@ function startStatusFor(phase: string | undefined): DeploymentStatus | null { return 'extracting'; case 'install': return 'installing'; + case 'stage': + return 'staging'; case 'load': return 'loading'; case 'replicate': return 'replicating'; + case 'activate': + return 'activating'; + case 'revert': + return 'reverting'; case 'restart': return 'restarting'; default: diff --git a/components/operations.js b/components/operations.js index 4d95977d2d..0f11755997 100644 --- a/components/operations.js +++ b/components/operations.js @@ -7,6 +7,7 @@ const fg = require('fast-glob'); const normalize = require('normalize-path'); const validator = require('./operationsValidation.js'); const log = require('../utility/logging/harper_logger.ts'); +const { randomUUID } = require('node:crypto'); const hdbTerms = require('../utility/hdbTerms.ts'); const env = require('../utility/environment/environmentManager.ts'); const configUtils = require('../config/configUtils.ts'); @@ -29,17 +30,40 @@ const { streamPackagedDirectory, } = require('../components/packageComponent.ts'); const { Resources } = require('../resources/Resources.ts'); -const { Application, prepareApplication, ASIDE_STAGING_DIR, dropComponentDirectory } = require('./Application.ts'); +const { + Application, + prepareApplication, + stageApplication, + revertApplication, + activateStagedApplication, + discardProjectStagedApplications, + discardProjectActivationArtifacts, + updateApplicationLockEntry, + withPersistentStateLock, + createApplicationActivationTransaction, + createApplicationConfigTransaction, + getRevertTarget, + dropComponentDirectory, + discardRetainedPrevious, + ASIDE_STAGING_DIR, + DEPLOY_STAGING_DIR, + DEPLOY_ACTIVATION_DIR, + DEPLOY_PREVIOUS_DIR, +} = require('./Application.ts'); const { COMPONENT_PREPARATION_LOCK_DIR, withComponentPreparationLock } = require('./componentPreparationLock.ts'); const { server } = require('../server/Server.ts'); const { DeploymentRecorder, awaitDeploymentRow, + isDeploymentTrackingAvailable, + invalidateProjectStagedDeployments, + pruneProjectPayloads, readPayloadBlobWithRetry, coerceTimeoutMs, DEFAULT_AWAIT_ROW_TIMEOUT_MS, } = require('./deploymentRecorder.ts'); const { ProgressEmitter } = require('../server/serverHelpers/progressEmitter.ts'); +const { isOperationAuthorizationBypassed } = require('../server/serverHelpers/operationAuthorizationState.ts'); const DROP_COMPONENT_LOCK_TIMEOUT_MS = 5 * 60 * 1000; @@ -324,14 +348,29 @@ async function dropCustomFunctionProject(req) { try { const projectDir = path.join(cfDir, project); const stagingDir = path.join(cfDir, ASIDE_STAGING_DIR, project); + // Nothing live AND nothing parked aside means there is genuinely no such component: let stat throw + // the ENOENT so the caller gets "no such project" rather than a silent success. if (!(await fs.pathExists(projectDir)) && !(await fs.pathExists(stagingDir))) await fs.stat(projectDir); + // Invalidate staged deployments BEFORE the directory goes away, so an activate racing this drop + // cannot swap a staged build back into a project that is being deleted. + await withComponentPreparationLock( projectDir, async () => { + // Retires any interrupted-extraction aside for this component and renames the live directory + // aside before removing it, so a concurrent startup recovery can never restore a tree over a + // component that was just dropped. + // Inside the lock: an invalidation done before acquiring it leaves a window where a + // concurrent stage completes and is then activated over the just-dropped component. + await invalidateProjectStagedDeployments(project); + await discardProjectStagedApplications(projectDir); + await discardProjectActivationArtifacts(projectDir); + await discardRetainedPrevious(projectDir); await dropComponentDirectory(projectDir, project, log); }, componentDropLockOptions(project) ); + await updateApplicationLockEntry(project, undefined); const response = await server.replication.replicateOperation(req); response.message = `Successfully deleted project: ${project}`; return response; @@ -432,82 +471,110 @@ async function packageComponent(req) { } /** - * Can deploy a component in multiple ways. If a 'package' is provided all it will do is write that package to - * harperdb-config, when HDB is restarted the package will be installed in hdb/nodeModules. If a base64 encoded string is passed it - * will write string to a temp tar file and extract that file into the deployed project in hdb/components. + * Deploy a component. Front door for the deploy family: derives the project name, validates, ingests + * any credential token into the secrets store (so it lives as a replicated reference, not embedded), + * then stages the build and swaps it in. + * + * * @param req - * @returns {Promise} + * @returns {Promise} */ async function deployComponent(req) { + normalizeRequestBooleans(req); if (req.project) { req.project = canonicalProjectName(req.project); } else if (req.package) { req.project = projectNameFromPackage(req.package); } - const validation = validator.deployComponentValidator(req); + const isReplicatedExecution = isTrustedReplicatedOperation(req); + let requestToValidate = req; + if (isReplicatedExecution) { + if (req._phase !== undefined) { + throw new ServerError( + `Unsupported legacy component deployment phase '${req._phase}'; upgrade the originating node before deploying` + ); + } + const { _deploymentId, ...publicRequest } = req; + requestToValidate = publicRequest; + } + const validation = validator.deployComponentValidator(requestToValidate); if (validation) { throw handleHDBError(validation, validation.message, HTTP_STATUS_CODES.BAD_REQUEST); } - // Ingest any provided credential token into the secrets store so the credential lives as // replicated ciphertext (reference, not embed); already-reference entries pass through, and with // no custody a literal token stays as a transient, this-node-only fallback (#1158). Peers // re-running a replicated deploy already carry references and never re-ingest. - const { ingestCredentials, resolveCredentials } = require('./secretOperations.ts'); + const { ingestCredentials } = require('./secretOperations.ts'); req.credentials = await ingestCredentials(req, req.credentials, req.project); // References are safe to persist (config + deployment row) and replicate; a no-custody literal - // token is not — it is used only for this node's install below, then stripped before replication. + // token is not — it is used only for this node's install, then stripped before replication. const credentialReferences = (req.credentials ?? []).filter((entry) => entry && entry.secret !== undefined); - // Write to root config if the request contains a package identifier - if (req.package) { - // Check if trying to overwrite a core component (requires force) - // Lazy-load to avoid circular dependency with componentLoader - const { TRUSTED_RESOURCE_PLUGINS } = require('./componentLoader.ts'); - if (TRUSTED_RESOURCE_PLUGINS[req.project] && !req.force) { - throw handleHDBError( - new Error(), - `Cannot deploy component with name '${req.project}': this is a protected core component name. Use force: true to overwrite.`, - HTTP_STATUS_CODES.CONFLICT - ); - } + return deployComponentOneShot(req, credentialReferences, isReplicatedExecution); +} - const applicationConfig = { package: req.package }; - // Avoid writing an empty `install:` block - if (req.install_command || req.install_timeout || req.install_allow_scripts !== undefined) { - applicationConfig.install = { - command: req.install_command, - timeout: req.install_timeout, - allowInstallScripts: req.install_allow_scripts, - }; - } - if (req.urlPath !== undefined) applicationConfig.urlPath = req.urlPath; - if (req.host !== undefined) applicationConfig.host = req.host; - // Persist credential references (never tokens) so every cold install of this component — - // reboot, new peer, rollback — re-resolves the credential from the store. - if (credentialReferences.length) applicationConfig.credentials = credentialReferences; - await configUtils.addConfig(req.project, applicationConfig); +/** + * Mark a restart as needed when this deploy changed code the running process can't pick up on its own. + * Setter only — it does not restart; it makes get_status report restartRequired:true and lets the REST + * route-miss path surface the actionable "needs a restart" 404 (harper#674). + * + * Two triggers: + * - `isNewComponent`: a genuinely-new, never-loaded component can't serve its routes until Harper + * restarts. Scoped to new components (harper#1806) because an existing component's own file watcher + * independently requests a restart when a redeploy actually needs one, so a redeploy stays quiet. + * - `packageMetadataChanged`: installed package metadata sits outside most plugin watch globs, so no + * watcher sees a dependency or module-entry change — but it does invalidate already-loaded code. + * Compared across the swap by the deploy path that performed it. + * + * Runs per node: each node checks its own directory state, which can differ across the cluster (a + * component can be new on a peer that never had it and a redeploy on the origin). The one-shot path has + * extractApplication/prepareApplication set both flags in place; the staged path has + * activateStagedApplication set them at swap time, comparing the pre-swap live tree against the staged + * one (staging is always fresh, so extraction never sees the live dir). + */ +function markRestartRequiredForDeploy(application) { + if (application.isNewComponent || application.packageMetadataChanged) { + const { requestRestart } = require('./requestRestart.ts'); + requestRestart(); } +} - // Create a hdb_deployment row up front so the deploy is observable and auditable - // even if the CLI disconnects. The row also holds the payload in a Blob attribute, - // which doubles as the source for peer replication and (later) rollback. - // - // Only the origin node records — peers receiving a replicated deploy_component skip - // recording so we don't accumulate one row per node for the same deploy. The row - // reaches peers via the table's standard replication; the peer-side branch below - // reads payload_blob back from there. - const isReplicatedExecution = typeof req._deploymentId === 'string'; - // An SSE-bound caller already attached a ProgressEmitter (created in the server - // handler so it can also drive the response stream). Reuse it; otherwise spin up a - // fresh emitter so the recorder still gets phase events for non-SSE deploys. +/** + * Legacy one-shot deploy: extract + `npm install` in place on the origin, then replicate the whole + * deploy_component operation to peers, which each do the same. Preserved verbatim (behavior-for- + * behavior) as the fallback path for `two_phase: false` and for peers replaying a one-shot deploy. + * The wrapper has already derived the project name, validated, and ingested credentials. + */ +async function deployComponentOneShot(req, credentialReferences, isReplicatedExecution) { + const { resolveCredentials } = require('./secretOperations.ts'); + + // Before ANY work: the rejection should not come after a credential has been ingested into the + // secrets store and a durable deployment row created for a deploy that was never allowed. This used + // to sit inside the root-config write, which the staged path replaced with the activation + // transaction. Package deploys only, exactly as before — a payload deploy has always been allowed + // to use a core name. + if (req.package) assertNotProtectedCoreComponent(req.project, req.force); + + // Create a hdb_deployment row up front so the deploy is observable and auditable even if the CLI + // disconnects. The row also holds the payload in a Blob attribute, which doubles as the source for + // peer replication and (later) rollback. Only the origin node records — peers replaying the + // replicated deploy skip recording so we don't accumulate one row per node for the same deploy. + // An SSE-bound caller already attached a ProgressEmitter (created in the server handler so it can + // also drive the response stream). Reuse it; otherwise spin up a fresh emitter so the recorder + // still gets phase events for non-SSE deploys. const emitter = isReplicatedExecution ? null : (req.progress ?? new ProgressEmitter()); if (emitter && !req.progress) req.progress = emitter; + // Built before the recorder so the row can carry it. Startup reconciliation reads + // `row.activation_spec` to reconcile root config after an interrupted activation, so a row without + // it cannot be recovered. + const activationSpec = activationSpecFromRequest(req, credentialReferences); const recorder = isReplicatedExecution ? null : await DeploymentRecorder.create({ project: req.project, + activation_spec: activationSpec, package_identifier: req.package ?? null, user: req.hdb_user?.username, restart_mode: req.restart === 'rolling' ? 'rolling' : req.restart ? 'immediate' : null, @@ -523,177 +590,91 @@ async function deployComponent(req) { const emit = (event, data) => emitter?.emit(event, data); - // The new payload-via-replicated-row path depends on the `system` database actually - // being replicated on this node. If the cluster is configured with a narrower - // REPLICATION_DATABASES list that excludes `system`, peers won't see the - // hdb_deployment row and falling back to sending req.payload through the operation - // body is the only viable path. + // The payload-via-replicated-row path depends on `system` actually replicating on this node. const systemReplicated = isSystemDatabaseReplicated(); - let extractionPayload = req.payload; - // Bounded ring buffer of install stdout/stderr so a non-SSE caller sees the tail - // in the thrown error. SSE callers still stream every line live. + // Bounded ring buffer of install stdout/stderr so a non-SSE caller sees the tail in the thrown + // error. SSE callers still stream every line live. const installCapture = createInstallCapture(); try { - // On the origin, tee the tarball (Buffer or Readable from the multipart parser) - // through a hash-and-size tap into the row's payload_blob, then re-source extraction - // from the persisted blob. When `system` replicates, the blob becomes the channel - // peers read from; when it doesn't, the blob stays local for audit and rollback. - if (recorder && req.payload != null) { - await recorder.ingestPayload(req.payload); - extractionPayload = recorder.row.payload_blob.stream(); - } else if (isReplicatedExecution && req.payload == null && !req.package) { - // Peer received a replicated deploy without a payload — read the tarball from - // the replicated hdb_deployment row's payload_blob. Blob.stream() blocks on - // in-flight BLOB_CHUNK writes until the chunks land. If the row never arrives - // within the timeout, peer records a failure and origin sees it in peer_results. - // The wait budget defaults to 120s but is overridable per-deploy via - // `deployment_timeout` (ms) for clusters where the system-table channel is - // heavily backlogged (harper-pro#402). - const payloadTimeoutMs = coerceTimeoutMs(req.deployment_timeout, DEFAULT_AWAIT_ROW_TIMEOUT_MS); - // One deadline covers both phases (row wait + blob content) so a slow row doesn't - // double the peer's total worst-case wait — whatever's left of payloadTimeoutMs after - // the row arrives is what the blob retry gets. - const payloadDeadline = Date.now() + payloadTimeoutMs; - const row = await awaitDeploymentRow(req._deploymentId, { timeoutMs: payloadTimeoutMs }); - // Blob content can stall independently of the row itself arriving (e.g. a - // parked/declined blob send on the origin, harper-pro#403) — the header lands but - // content bytes don't, and stream() gives up with a retryable 503 after - // blobReadTimeout of no progress. Retry with backoff, bounded by the remaining - // budget, so a transient stall doesn't fail the whole deploy (harper-pro incident, - // 2026-07-16). - extractionPayload = readPayloadBlobWithRetry(() => row.payload_blob.stream(), { - timeoutMs: Math.max(0, payloadDeadline - Date.now()), - }); - } - - // Resolve credential references into concrete tokens for this node's npm pack/install - // (a no-custody literal-token fallback passes through unchanged). On a peer running a - // replicated deploy, the referenced hdb_secret row may arrive just behind the deploy op, so - // allow a bounded grace period (same budget as the payload-row wait) for it to replicate in. - let credentialsWaitMs = 0; - if (isReplicatedExecution) { - credentialsWaitMs = coerceTimeoutMs(req.deployment_timeout, DEFAULT_AWAIT_ROW_TIMEOUT_MS); - } - const resolvedCredentials = await resolveCredentials(req.credentials, req.project, { - waitMs: credentialsWaitMs, - }); - - const application = new Application({ - name: req.project, - payload: extractionPayload, - packageIdentifier: req.package, - install: { - command: req.install_command, - timeout: req.install_timeout, - allowInstallScripts: req.install_allow_scripts, - }, - // Tee each install line into both the capture buffer (for the thrown-error - // fallback) and the SSE channel (when a caller is streaming). Peers have no - // emitter, so their install output goes to the local logger and the buffer only. - onInstallLine: (manager, stream, line) => { - installCapture.push(manager, stream, line); - if (emitter) emit('install', { manager, stream, line }); - }, - // Deploy credentials (already resolved above), used here for this node's npm pack/install: - // registry entries via a transient .npmrc, git-host entries via the in-memory credential - // socket the clone spawn talks to. - credentials: resolvedCredentials, + const extractionPayload = await sourceExtractionPayload({ req, recorder, isReplicatedExecution }); + const resolvedCredentials = await resolveNodeCredentials({ req, resolveCredentials, isReplicatedExecution }); + const application = buildDeployApplication({ + req, + extractionPayload, + resolvedCredentials, + installCapture, + emitter, + emit, }); // Reduce req.credentials to references only (never a token) before it can reach an error/log // path or replication: references are what peers resolve from their own replicated hdb_secret // copy; a no-custody literal token is dropped entirely (peers fall back to their fabric-injected - // NPM_CONFIG_USERCONFIG, as before). This also fixes the prior success-only strip that leaked a - // literal token on a prepare/load failure. + // NPM_CONFIG_USERCONFIG, as before). if (credentialReferences.length) req.credentials = credentialReferences; else delete req.credentials; - emit('phase', { phase: 'prepare', status: 'start' }); - await prepareApplication(application); - emit('phase', { phase: 'prepare', status: 'done' }); - - // now we attempt to actually load the component in case there is - // an error we can immediately detect and report, but app code should not run on the main thread - if (!isMainThread && !process.env.HARPER_SAFE_MODE) { - const pseudoResources = new Resources(); - pseudoResources.isWorker = true; - - const componentLoader = require('./componentLoader.ts').default || require('./componentLoader.ts'); - const { trackScopeClose } = require('./scopeShutdown.ts'); - let lastError; - componentLoader.setErrorReporter((error) => (lastError = error)); - emit('phase', { phase: 'load', status: 'start' }); - // This load exists only to surface load-time errors early; the Scopes it creates are - // throwaway. They are collected (instead of registered for worker-shutdown auto-close) so we - // can close them here once validation completes — otherwise each deploy leaks the Scope's - // deploy-lifecycle listeners on this worker, eventually tripping MaxListenersExceededWarning - // (#1462). - const validationScopes = new Set(); - // Process-wide `server.*` registrations (registerOperation, setMcpQuotaHandler) are not owned by - // a Scope, so a candidate's top-level registration during this throwaway load would otherwise - // outlive it and pollute the live worker on a failed/rolled-back deploy. The guard makes those - // registration methods no-op for the duration of the load. - const { runWithDeployValidationGuard } = require('../server/serverHelpers/deployValidationState.ts'); - const validation = runWithDeployValidationGuard(async () => { - try { - await componentLoader.loadComponent(application.dirPath, pseudoResources, undefined, { - collectScopes: validationScopes, - }); - } finally { - const closeResults = await Promise.allSettled(Array.from(validationScopes, (scope) => scope.close())); - for (const result of closeResults) { - if (result.status === 'rejected') log.warn('Failed to close a deploy-validation Scope', result.reason); - } - } - }); - // Track the load+close so a concurrent worker shutdown waits for these scopes to finish - // disposing — a plugin may start a native runtime in handleApplication — before realExit. - trackScopeClose(validation); - await validation; - emit('phase', { phase: 'load', status: 'done' }); + // Build off to the side, then swap. The live component keeps serving through the slow, + // failure-prone work — git clone, registry install — and go-live is one atomic rename, so a fetch + // or install failure leaves the running component untouched instead of half-replaced in place. + // This is per node: the operation replicates as a whole and each node does its own staged build. + // There is deliberately no cluster-wide barrier here; ordering activation across nodes is #2294. + const deploymentId = recorder?.deploymentId ?? req._deploymentId ?? randomUUID(); + emit('phase', { phase: 'stage', status: 'start' }); + const stagingDirPath = await stageApplication(application, deploymentId); + emit('phase', { phase: 'stage', status: 'done' }); + + // Validate the STAGED tree, so a component that installs cleanly but throws at load never reaches + // the live path (throwaway scopes; see loadValidateComponent). + await loadValidateComponent({ dirPath: stagingDirPath, emit }); + + // Root config and `harper-application-lock.json` commit inside the swap, so the directory and the + // durable state describing it move together and compensate together. + const configTransaction = await createApplicationActivationTransaction(req.project, activationSpec); + emit('phase', { phase: 'activate', status: 'start' }); + await activateStagedApplication(application, deploymentId, { + beforeCommit: () => configTransaction.commit(), + onRollback: () => configTransaction.rollback(), + activationSpec, + }); + emit('phase', { phase: 'activate', status: 'done' }); - if (lastError) throw lastError; - } const rollingRestart = req.restart === 'rolling'; // if doing a rolling restart set restart to false so that other nodes don't also restart. req.restart = rollingRestart ? false : req.restart; - // ProgressEmitter holds function listeners that can't survive the replication - // channel's serialization; strip it unconditionally. + // ProgressEmitter holds function listeners that can't survive the replication channel's + // serialization; strip it unconditionally. delete req.progress; - // req.credentials was already deleted immediately after the Application ctor (above) so the - // token never reaches the replication channel or a peer's operation log; peers authenticate - // against the private registry via their own fabric-injected NPM_CONFIG_USERCONFIG on reinstall. - if (systemReplicated && recorder) { - // The hdb_deployment row + payload_blob will reach peers via table replication, - // so peers can look up the payload by deployment_id. Drop req.payload to keep - // the operation body small (the operations channel has frame-size limits the - // blob-replication channel doesn't share). _deploymentId is the handoff that - // lets peers find the replicated row. + if (systemReplicated && recorder && !isDeploymentTrackingAvailable() && recorder.replayablePayload) { + // No row exists for peers to read the blob from, and `req.payload` is an EXHAUSTED source by now + // — ingest drained it. Replace it with the buffered copy so the bytes actually travel in the + // replicated operation; keeping the spent stream would send peers an EOF after this node was + // already live. + req.payload = recorder.replayablePayload; + } else if (systemReplicated && recorder && isDeploymentTrackingAvailable()) { + // The hdb_deployment row + payload_blob reach peers via table replication, so peers look up + // the payload by deployment_id. Drop req.payload to keep the operation body small. + // + // Gated on the table actually existing. `recorder` is truthy either way — its writes no-op + // without the table — so stripping the payload on that alone left peers with a `_deploymentId`, + // no bytes, and no row to resolve, after this node was already live. With no row to read from, + // the payload has to ride along in the replicated operation, which is what the one-shot path did + // before deployment tracking existed. delete req.payload; } - // As each peer settles, update the origin row so observers polling get_deployment - // see per-peer progress in real time rather than only at the aggregate end. - // replicateOperation in harper-pro accepts an optional onPeerResult callback that - // fires per peer; callers without the callback (older replicator) fall back to - // the aggregate response.replicated below. const onPeerResult = recorder ? (result) => { recorder.recordPeer(result); emit('peer', result); } : undefined; - // Seal the recorder before the replicate phase so the row's terminal write (finish()) - // isn't part of the tight put burst that can commit out of order on a peer and revert - // it (harperdb/harper#1170). onPeerResult/peer_results accumulate in memory and land in - // finish()'s single write; live SSE 'peer' events still fire below. + // Seal before the replicate phase so the row's terminal write (finish()) isn't part of the tight + // put burst that can commit out of order on a peer and revert it (harperdb/harper#1170). recorder?.seal(); emit('phase', { phase: 'replicate', status: 'start' }); let response = await server.replication.replicateOperation(req, { onPeerResult }); emit('phase', { phase: 'replicate', status: 'done' }); if (recorder && response?.replicated) { - // Fallback path for replicators that don't honor onPeerResult: re-record the - // aggregate. recordPeer's upsert-by-node-name semantics make this idempotent - // when the per-peer callback already fired for these. recorder.recordPeers(response.replicated); } if (req.restart === true) { @@ -723,31 +704,25 @@ async function deployComponent(req) { // deploy checks its own local isNewComponent, since directory state (and therefore // whether the component was already active) can differ per node. // - // An existing component's watched files are handled by Scope/EntryHandler. Package - // metadata is deliberately outside most plugin globs, so compare it across the atomic - // swap as well: a dependency or module-entry change also invalidates loaded code. - if (application.isNewComponent || application.packageMetadataChanged) { - const { requestRestart } = require('./requestRestart.ts'); - requestRestart(); - } + // An existing, already-active component being redeployed does NOT force a restart here: some + // updates (e.g. static files only) may not need one at all, and when one genuinely is needed, + // that component's already-running file watcher (Scope/EntryHandler, see deployLifecycle.ts) + // independently detects the post-deploy file changes and requests the restart itself. Package + // metadata is the exception the helper handles — it sits outside most plugin globs, so no + // watcher sees a dependency or module-entry change. + markRestartRequiredForDeploy(application); response.message = `Successfully deployed: ${application.name}`; } - // Replication failures don't reject replicateOperation — they surface as 'failed' - // entries in peer_results. By default, treat any failed peer as an overall deploy - // failure so the operation returns a non-2xx status (and the CLI a non-zero exit - // code). The component is already deployed — and, if requested, restarted — on this - // origin node; the failure signals that one or more peers did not receive it. Pass - // ignore_replication_errors: true for best-effort deploys to partially-available clusters. + // Replication failures don't reject replicateOperation — they surface as 'failed' entries in + // peer_results. By default, treat any failed peer as an overall deploy failure so the operation + // returns a non-2xx status. Pass ignore_replication_errors: true for best-effort deploys. if (recorder && !req.ignore_replication_errors) { const failedPeers = recorder.getFailedPeers(); if (failedPeers.length > 0) { - const detail = failedPeers - .map((peer) => `${peer.node ?? 'unknown'} (${peer.error?.message ?? 'unknown error'})`) - .join(', '); throw new ServerError( `Component '${application.name}' was deployed on the origin node but failed to replicate to ` + - `${failedPeers.length} of ${recorder.row.peer_results.length} peer node(s): ${detail}. ` + + `${failedPeers.length} of ${recorder.row.peer_results.length} peer node(s): ${describePeers(failedPeers)}. ` + `See deployment ${recorder.deploymentId} (get_deployment) for details, or pass ` + `ignore_replication_errors: true to treat replication failures as non-fatal.` ); @@ -756,69 +731,444 @@ async function deployComponent(req) { if (recorder) { response.deployment_id = recorder.deploymentId; - // Reclaim the payload tarball for large deploys: every peer has now installed from - // the blob (replicateOperation resolved) and the origin no longer needs it. Dropping - // the reference before finish() folds the null into the single terminal write, which - // unlinks the file locally and replicates the null so peers drop their copies too. - // Metadata (size, hash, event_log) is retained for the audit trail. Two guards keep - // the tarball when it's still the artifact you'd debug or retry with: failed deploys - // don't reach this branch, and a deploy that reached here only because - // ignore_replication_errors masked failed peers keeps its payload for those peers. - const payloadSize = recorder.row.payload_size; - const retentionMaxSize = getPayloadRetentionMaxSize(); - if (typeof payloadSize === 'number' && payloadSize > retentionMaxSize && recorder.getFailedPeers().length === 0) { - const freed = recorder.dropPayload(); - if (freed > 0) emit('payload_dropped', { payload_size: freed, max_size: retentionMaxSize }); - } + maybeReclaimPayload(recorder, emit); emit('phase', { phase: 'success', status: 'done' }); await recorder.finish('success'); + // After finish(), so this deploy's row is terminal and counts as the newest retained payload. + schedulePayloadRetentionPrune(recorder, req.project, emit); } return response; } catch (err) { - // Pack phase, install output tail, and deployment_id into http_resp_msg so the - // Fastify error handler forwards them verbatim (it does when http_resp_msg is an - // object). Non-SSE callers see structured failure detail; SSE callers already - // got the same data live via emit('error', ...) below. - const capture = installCapture.snapshot(); - const phase = recorder?.row.phase; - const baseMessage = err?.message ?? String(err); - const structured = { error: baseMessage }; - if (phase) structured.phase = phase; - if (capture.lines.length > 0) structured.install_output = capture; - if (recorder?.deploymentId) structured.deployment_id = recorder.deploymentId; - // Surface failed peer outcomes so callers see which nodes the deploy did not reach - // without a second get_deployment round-trip. Populated for replication failures (the - // throw above) and any other failure that occurred after peers reported. Carried on - // both the structured non-SSE body and the SSE 'error' event below so the two transports - // stay symmetric (the CLI uses SSE for deploy_component). - const failedPeers = recorder?.getFailedPeers() ?? []; - if (failedPeers.length > 0) structured.failed_peers = failedPeers; - - // Wrap as a ServerError so the Fastify error handler picks a 500 by default; preserve - // an upstream statusCode (e.g. a ClientError from payload validation) if present. - const outErr = new ServerError(baseMessage, err?.statusCode); - outErr.http_resp_msg = structured; - - emit('error', { - message: baseMessage, - code: outErr?.statusCode ?? err?.code, - phase, - install_output: capture.lines.length > 0 ? capture : undefined, - deployment_id: recorder?.deploymentId, - failed_peers: failedPeers.length > 0 ? failedPeers : undefined, + throw await finalizeDeployFailure({ err, recorder, installCapture, emit }); + } +} + +// Every boolean in the deploy_component/revert_component schemas. Joi coerces a string `"false"`, but +// validateBySchema discards `result.value`, so the raw string reaches the handler and reads as truthy. +// `install_allow_scripts` is the one that matters most: a caller explicitly disabling lifecycle scripts +// over multipart/form would otherwise run third-party install code with credentials available. +const REQUEST_BOOLEAN_FIELDS = [ + 'activate', + 'two_phase', + 'ignore_replication_errors', + 'force', + 'restart', + 'install_allow_scripts', +]; + +function normalizeRequestBooleans(req) { + for (const field of REQUEST_BOOLEAN_FIELDS) { + const value = req[field]; + if (typeof value !== 'string') continue; + const lowered = value.trim().toLowerCase(); + if (lowered === 'true') req[field] = true; + else if (lowered === 'false') req[field] = false; + } +} + +function activationSpecFromRequest(req, credentialReferences) { + return { + project: req.project, + package: req.package ?? null, + install_command: req.install_command ?? null, + install_timeout: req.install_timeout ?? null, + install_allow_scripts: req.install_allow_scripts ?? null, + urlPath: req.urlPath ?? null, + host: req.host ?? null, + credentials: credentialReferences.length ? credentialReferences : null, + force: req.force === true, + }; +} + +function describePeerFailures(failed) { + return failed + .map((peer) => `${peer.node ?? 'unknown'} (${peer.error?.message ?? peer.reason ?? 'unknown error'})`) + .join(', '); +} + +function isTrustedReplicatedOperation(req) { + const user = req.hdb_user; + return ( + isOperationAuthorizationBypassed() && + req.replicated === false && + !!user && + !!(user.name || user.replicates || user.subscribers) + ); +} + +/** + * revert_component — put a component's retained previous version back in service, cluster-wide. + * + * This is the fast-rollback half of the deploy story, and it is deliberately a separate public + * operation rather than a deploy phase: it fetches nothing, resolves no package or secret, downloads + * no artifact and runs no install. The bytes it activates are already on disk on every node — the tree + * each node's last activation displaced and retained as `.deploy-previous/` — so the whole + * cluster can go back to the version it was serving a minute ago at the cost of one atomic rename per + * node. It exists for the case operators actually hit: the one bad rollout that just happened. + * + * It is NOT a general "go to any past version" operation. Exactly one previous version is retained per + * component, so this reaches back exactly one activation. Returning to an older version is a redeploy + * (`deploy_component`), not a revert. + * + * `to_deployment_id` is required and names the deployment the caller expects to be live afterwards, + * which is what makes the operation idempotent under ordinary request retries: if + * that version is already live the call succeeds without touching anything, instead of toggling the + * rejected release back in. Targeting the retained previous swaps, and the displaced tree becomes the + * new retained previous — so an explicitly-targeted revert-of-a-revert still rolls forward. + * + * The swap is paired with persistent state. `revertApplication` reports the root-config entry the + * newly-live tree was originally activated with, and this applies it to both the root config and the + * boot-time install lock. Without that, reverting away from a `package` deploy would leave the config + * still naming the reverted-away package, and `installApplications()` would quietly reinstall it over + * the restored directory on the next cold start — undoing the rollback. + */ +async function revertComponent(req) { + normalizeRequestBooleans(req); + if (req.project) req.project = path.parse(req.project).name; + const validation = validator.revertComponentValidator(req); + if (validation) throw handleHDBError(validation, validation.message, HTTP_STATUS_CODES.BAD_REQUEST); + assertNotProtectedCoreComponent(req.project, req.force); + + const isReplicatedExecution = isTrustedReplicatedOperation(req); + const emitter = isReplicatedExecution ? null : (req.progress ?? new ProgressEmitter()); + const emit = emitter ? (event, data) => emitter.emit(event, data) : () => {}; + delete req.progress; + + const application = new Application({ name: req.project }); + // Read the retained-previous state before anything is created or moved: it supplies the deployment + // this rollback takes out of service (recorded as `rollback_of`, which the recorder can only accept + // at create time), and it lets an unrevertable request fail before a deployment row exists. + const componentsRoot = configUtils.getConfigPath(hdbTerms.CONFIG_PARAMS.COMPONENTSROOT); + const revertTarget = await getRevertTarget(path.join(componentsRoot, req.project)); + // The origin records an auditable rollback row; a peer replaying the revert does not (the origin + // owns the row, exactly as in the deploy fan-out). + const recorder = isReplicatedExecution + ? null + : await DeploymentRecorder.create({ + project: req.project, + user: req.hdb_user?.username, + restart_mode: req.restart === 'rolling' ? 'rolling' : req.restart ? 'immediate' : null, + rollback_of: revertTarget?.live?.deployment_id ?? null, + emitter, + }); + try { + emit('phase', { phase: 'revert', status: 'start' }); + // The config transaction commits inside revertApplication's component lock, between the directory + // swap and the manifest write, so the directory, the manifest and root config/application-lock + // share one fence. Committing it out here — after the lock released — let a queued activation swap + // and commit in the gap, leaving that activation's bytes live under this revert's config. + // revertApplication compensates the swap if this throws; rolling the partial config write back is + // this transaction's job, since commit() makes two persistent writes and the first can land alone. + let configTransaction; + const result = await revertApplication(application, req.to_deployment_id, { + commitPersistentState: async (activatedConfig) => { + configTransaction = await createApplicationConfigTransaction(req.project, activatedConfig); + await configTransaction.commit(); + }, + // Called for any failure after the commit landed, not just a failing commit: the manifest write + // and the retain rename come after it, and undoing only the directories would leave root config + // and the application lock naming the release that is no longer live. + rollbackPersistentState: async () => { + await configTransaction?.rollback(); + }, }); - // Record the terminal failure, but never let a finish() write error (full disk, lock, - // dropped system table) mask the actual deploy failure — outErr carries the phase, - // install output, and failed_peers the caller needs. - if (recorder) { - try { - await recorder.finish('failed', err); - } catch (finishErr) { - log.warn('Failed to record deployment failure row', finishErr); + emit('phase', { phase: 'revert', status: 'done' }); + + // Fan out to peers. The operation is idempotent and target-addressed, so a peer that already + // holds the target live converges to the same place without swapping — which is what makes a + // straight re-run of the same operation the right replication primitive here. + let replication; + if (req.replicated !== false && !isReplicatedExecution) { + replication = await server.replication.replicateOperation( + { + operation: hdbTerms.OPERATIONS_ENUM.REVERT_COMPONENT, + project: req.project, + to_deployment_id: req.to_deployment_id, + restart: req.restart === true, + ...(req.force ? { force: req.force } : {}), + ...(req.deployment_timeout !== undefined ? { deployment_timeout: req.deployment_timeout } : {}), + }, + { + onPeerResult: (peerResult) => { + recorder?.recordPeer(peerResult); + emit('peer', peerResult); + }, + } + ); + if (replication?.replicated) recorder?.recordPeers(replication.replicated); + const failed = recorder?.getFailedPeers() ?? []; + if (failed.length && !req.ignore_replication_errors) { + throw new ServerError( + `Component '${req.project}' reverted on this node but failed on ${failed.length} peer node(s): ` + + `${describePeerFailures(failed)}. The cluster is split across versions; retry the revert (it is a ` + + `no-op on the nodes that already reverted) or roll forward with deploy_component.` + ); + } + } + + const restart = await restartRevertedComponent(req, emit); + emit('phase', { phase: 'success', status: 'done' }); + await recorder?.finish('rolled_back'); + return { + message: result.swapped + ? `Reverted component: ${req.project} to deployment ${req.to_deployment_id}${restart.restartMessage}` + : `Component ${req.project} is already running deployment ${req.to_deployment_id}; nothing to revert`, + project: req.project, + reverted: result.swapped, + to_deployment_id: req.to_deployment_id, + ...(result.fromDeploymentId ? { from_deployment_id: result.fromDeploymentId } : {}), + ...(recorder ? { deployment_id: recorder.deploymentId } : {}), + ...(replication?.replicated ? { replicated: replication.replicated } : {}), + ...(restart.restartJobId ? { restartJobId: restart.restartJobId } : {}), + }; + } catch (error) { + const message = error?.message ?? String(error); + emit('error', { message, code: error?.statusCode ?? error?.code }); + await recorder + ?.finish('failed', error) + .catch((finishError) => log.warn('Failed to record component revert failure', finishError)); + throw new ServerError(message, error?.statusCode); + } +} + +/** Restart after a revert, mirroring the deploy path's restart handling. */ +async function restartRevertedComponent(req, emit) { + if (req.restart === true) { + emit('phase', { phase: 'restart', status: 'start' }); + manageThreads.restartWorkers('http'); + emit('phase', { phase: 'restart', status: 'done' }); + return { restartMessage: ', restarting Harper' }; + } + if (req.restart === 'rolling') { + const serverUtilities = require('../server/serverHelpers/serverUtilities.ts'); + emit('phase', { phase: 'restart', status: 'start' }); + const jobResponse = await serverUtilities.executeJob({ + operation: 'restart_service', + service: 'http', + replicated: true, + }); + emit('phase', { phase: 'restart', status: 'done' }); + return { restartMessage: ', restarting Harper', restartJobId: jobResponse.job_id }; + } + return { restartMessage: '' }; +} + +function assertNotProtectedCoreComponent(project, force) { + // Lazily required: componentLoader requires this module back. + const { TRUSTED_RESOURCE_PLUGINS } = require('./componentLoader.ts'); + if (TRUSTED_RESOURCE_PLUGINS[project] && !force) { + throw handleHDBError( + new Error(), + `Cannot deploy component with name '${project}': this is a protected core component name. Use force: true to overwrite.`, + HTTP_STATUS_CODES.CONFLICT + ); + } +} + +// Resolve the tarball to extract from. On the origin, tee req.payload into the row's blob (the +// channel peers read from) and re-source extraction from the persisted blob. On a peer replaying a +// deploy without a payload, read the tarball from the replicated row's blob (bounded wait). +async function sourceExtractionPayload({ req, recorder, isReplicatedExecution }) { + if (recorder && req.payload != null) { + await recorder.ingestPayload(req.payload); + return recorder.row.payload_blob.stream(); + } + if (isReplicatedExecution && req.payload == null && !req.package) { + // Blob.stream() blocks on in-flight BLOB_CHUNK writes until the chunks land. If the row never + // arrives within the timeout, the peer records a failure and the origin sees it in peer_results. + // The wait budget defaults to 120s but is overridable per-deploy via `deployment_timeout` (ms) + // for clusters where the system-table channel is heavily backlogged (harper-pro#402). + const payloadTimeoutMs = coerceTimeoutMs(req.deployment_timeout, DEFAULT_AWAIT_ROW_TIMEOUT_MS); + // One deadline covers both phases (row wait + blob content) so a slow row doesn't double the + // peer's worst-case wait — whatever's left after the row arrives is what the blob retry gets. + const payloadDeadline = Date.now() + payloadTimeoutMs; + const row = await awaitDeploymentRow(req._deploymentId, { timeoutMs: payloadTimeoutMs }); + // Blob content can stall independently of the row arriving (a parked/declined blob send on the + // origin, harper-pro#403): the header lands but content bytes don't, and stream() gives up with a + // retryable 503 after blobReadTimeout of no progress. Retry with backoff, bounded by the remaining + // budget, so a transient stall doesn't fail the whole deploy (harper-pro incident, 2026-07-16). + return readPayloadBlobWithRetry(() => row.payload_blob.stream(), { + timeoutMs: Math.max(0, payloadDeadline - Date.now()), + }); + } + return req.payload; +} + +// Resolve credential references into concrete tokens for this node's npm pack/install. On a peer, the +// referenced hdb_secret row may arrive just behind the deploy op, so allow a bounded grace period. +async function resolveNodeCredentials({ req, resolveCredentials, isReplicatedExecution }) { + let credentialsWaitMs = 0; + if (isReplicatedExecution) { + // Same budget as the payload-row wait, via the shared coercion helper (harper#1838 dedup). + credentialsWaitMs = coerceTimeoutMs(req.deployment_timeout, DEFAULT_AWAIT_ROW_TIMEOUT_MS); + } + return resolveCredentials(req.credentials, req.project, { waitMs: credentialsWaitMs }); +} + +// Construct the Application for a deploy, teeing each install line into the capture buffer (for the +// thrown-error tail) and the SSE channel (when a caller is streaming). +function buildDeployApplication({ + req, + extractionPayload, + resolvedCredentials, + stagingId, + installCapture, + emitter, + emit, +}) { + return new Application({ + name: req.project, + payload: extractionPayload, + packageIdentifier: req.package, + install: { + command: req.install_command, + timeout: req.install_timeout, + allowInstallScripts: req.install_allow_scripts, + }, + onInstallLine: (manager, stream, line) => { + installCapture.push(manager, stream, line); + if (emitter) emit('install', { manager, stream, line }); + }, + credentials: resolvedCredentials, + stagingId, + }); +} + +// Load a component directory to surface load-time errors early (throwaway scopes). No-op on the main +// thread or in safe mode. It loads the STAGED directory, before go-live, where it runs at all — see +// loads the live directory after in-place prepare. +// `componentLoader.setErrorReporter` is module-global, so two deploys validating concurrently on the +// same worker cross-attribute their load failures: B installs its reporter while A is loading, A's +// error lands in B, and A then activates broken bytes while B rejects a good candidate. Validation is +// serialized here — it is already the slow path — and the previous reporter is restored, so the global +// is only ever owned by one in-flight validation. +let validationChain = Promise.resolve(); +async function loadValidateComponent(args) { + const run = validationChain.then( + () => loadValidateComponentExclusive(args), + () => loadValidateComponentExclusive(args) + ); + validationChain = run.then( + () => {}, + () => {} + ); + return run; +} + +async function loadValidateComponentExclusive({ dirPath, emit }) { + if (isMainThread || process.env.HARPER_SAFE_MODE) return; + const pseudoResources = new Resources(); + pseudoResources.isWorker = true; + + const componentLoader = require('./componentLoader.ts').default || require('./componentLoader.ts'); + const { trackScopeClose } = require('./scopeShutdown.ts'); + let lastError; + const priorErrorReporter = componentLoader.getErrorReporter?.(); + componentLoader.setErrorReporter((error) => (lastError = error)); + emit('phase', { phase: 'load', status: 'start' }); + // The Scopes this load creates are throwaway. Collect them (instead of registering for + // worker-shutdown auto-close) so we can close them here once validation completes — otherwise each + // deploy leaks the Scope's deploy-lifecycle listeners on this worker (#1462). + const validationScopes = new Set(); + // Process-wide `server.*` registrations (registerOperation, setMcpQuotaHandler) are not owned by + // a Scope, so a candidate's top-level registration during this throwaway load would otherwise + // outlive it and pollute the live worker on a failed/rolled-back deploy. The guard makes those + // registration methods no-op for the duration of the load. + const { runWithDeployValidationGuard } = require('../server/serverHelpers/deployValidationState.ts'); + const validation = runWithDeployValidationGuard(async () => { + try { + await componentLoader.loadComponent(dirPath, pseudoResources, undefined, { collectScopes: validationScopes }); + } finally { + const closeResults = await Promise.allSettled(Array.from(validationScopes, (scope) => scope.close())); + for (const result of closeResults) { + if (result.status === 'rejected') log.warn('Failed to close a deploy-validation Scope', result.reason); } } - throw outErr; + }); + // Track the load+close so a concurrent worker shutdown waits for these scopes to finish disposing. + trackScopeClose(validation); + try { + await validation; + } finally { + componentLoader.setErrorReporter(priorErrorReporter); } + emit('phase', { phase: 'load', status: 'done' }); + if (lastError) throw lastError; +} + +// Render failed peer outcomes as "node (error)" for an operator-facing error message. +function describePeers(failedPeers) { + return failedPeers.map((peer) => `${peer.node ?? 'unknown'} (${peer.error?.message ?? 'unknown error'})`).join(', '); +} + +// Reclaim the payload tarball for large deploys once every peer has installed from the blob. Dropping +// the reference before finish() folds the null into the single terminal write, which unlinks the file +// locally and replicates the null so peers drop their copies too. Metadata is retained. Kept when a +// peer failed (still the retry artifact) or the payload is small. +function maybeReclaimPayload(recorder, emit) { + const payloadSize = recorder.row.payload_size; + const retentionMaxSize = getPayloadRetentionMaxSize(); + if (typeof payloadSize === 'number' && payloadSize > retentionMaxSize && recorder.getFailedPeers().length === 0) { + const freed = recorder.dropPayload(); + if (freed > 0) emit('payload_dropped', { payload_size: freed, max_size: retentionMaxSize }); + } +} + +/** + * Count-based payload retention (deployment_payloadRetention_maxCount): after a successful deploy, keep + * only the newest N stored payloads for this project and drop the rest. Where the size-based reclaim + * above only ever considers THIS deploy's payload, this bounds the total that accumulates per project. + * + * Deliberately best-effort and never awaited into the deploy's critical path: retention is disk + * hygiene, so a prune failure is logged and the deploy still succeeds. Skipped when a peer failed — + * the older payloads are still the retry/rollback artifacts in that case. + */ +function schedulePayloadRetentionPrune(recorder, project, emit) { + if (recorder.getFailedPeers().length > 0) return; + const maxCount = getPayloadRetentionMaxCount(); + pruneProjectPayloads(project, maxCount) + .then((freed) => { + if (freed > 0) emit('payload_dropped', { payload_size: freed, max_count: maxCount }); + }) + .catch((err) => log.warn(`Failed to prune retained deployment payloads for '${project}'`, err)); +} + +// Build the structured failure (phase, install-output tail, deployment_id, failed peers) that the +// Fastify error handler forwards verbatim, emit the matching SSE 'error' event so the two transports +// stay symmetric, and record the terminal failure row. Returns the ServerError to throw. +async function finalizeDeployFailure({ err, recorder, installCapture, emit }) { + const capture = installCapture.snapshot(); + const phase = recorder?.row.phase; + const baseMessage = err?.message ?? String(err); + const structured = { error: baseMessage }; + if (phase) structured.phase = phase; + if (capture.lines.length > 0) structured.install_output = capture; + if (recorder?.deploymentId) structured.deployment_id = recorder.deploymentId; + const failedPeers = recorder?.getFailedPeers() ?? []; + if (failedPeers.length > 0) structured.failed_peers = failedPeers; + + // Wrap as a ServerError so the Fastify error handler picks a 500 by default; preserve an upstream + // statusCode (e.g. a ClientError from payload validation) if present. + const outErr = new ServerError(baseMessage, err?.statusCode); + outErr.http_resp_msg = structured; + + emit('error', { + message: baseMessage, + code: outErr?.statusCode ?? err?.code, + phase, + install_output: capture.lines.length > 0 ? capture : undefined, + deployment_id: recorder?.deploymentId, + failed_peers: failedPeers.length > 0 ? failedPeers : undefined, + }); + // Record the terminal failure, but never let a finish() write error mask the actual deploy failure. + if (recorder) { + try { + await recorder.finish('failed', err); + } catch (finishErr) { + log.warn('Failed to record deployment failure row', finishErr); + } + } + return outErr; } // Ring buffer of install stdout/stderr lines, capped by both line count and bytes so @@ -893,6 +1243,9 @@ async function getComponents() { if ( itemName === 'node_modules' || itemName === ASIDE_STAGING_DIR || + itemName === DEPLOY_STAGING_DIR || + itemName === DEPLOY_ACTIVATION_DIR || + itemName === DEPLOY_PREVIOUS_DIR || itemName === COMPONENT_PREPARATION_LOCK_DIR ) continue; @@ -985,6 +1338,26 @@ const DEFAULT_COMPONENT_FILE_MAX_SIZE = 5 * 1024 * 1024; // 5 MB // Configurable via deployment_payloadRetention_maxSize; set it very high to retain all payloads. const DEFAULT_PAYLOAD_RETENTION_MAX_SIZE = 10 * 1024 * 1024; // 10 MiB +// How many stored payload tarballs to keep per project (newest first); older ones have their +// payload_blob dropped after a successful deploy. Default 1 — retain only the current version's +// payload. Conservative on purpose: instances on small quotas (free tier is 5GB total) must not have +// N copies of a large app payload quietly competing with the customer's own data. Raise it to widen +// the redeploy-by-reference window; 0 retains none. Configurable via +// deployment_payloadRetention_maxCount. +const DEFAULT_PAYLOAD_RETENTION_MAX_COUNT = 1; + +function getPayloadRetentionMaxCount() { + const configured = configUtils.getConfigValue(hdbTerms.CONFIG_PARAMS.DEPLOYMENT_PAYLOADRETENTION_MAXCOUNT); + // Same input discipline as getPayloadRetentionMaxSize: only a number or numeric string is a valid + // count. Anything else (unset, boolean, array, blank string) would coerce to 0 or 1 and silently + // change retention, so fall back to the default instead. + if (typeof configured !== 'number' && typeof configured !== 'string') return DEFAULT_PAYLOAD_RETENTION_MAX_COUNT; + if (typeof configured === 'string' && configured.trim() === '') return DEFAULT_PAYLOAD_RETENTION_MAX_COUNT; + const parsed = Number(configured); + if (!Number.isFinite(parsed) || parsed < 0) return DEFAULT_PAYLOAD_RETENTION_MAX_COUNT; + return Math.floor(parsed); +} + function getPayloadRetentionMaxSize() { const configured = configUtils.getConfigValue(hdbTerms.CONFIG_PARAMS.DEPLOYMENT_PAYLOADRETENTION_MAXSIZE); // Only a number or a numeric string is a valid threshold. Reject everything else (unset, @@ -1202,7 +1575,25 @@ async function dropComponent(req) { } if (!file) { + // Invalidate staged deployments before the directory goes away, so an activate racing this + // drop cannot swap a staged build back into a component that is being dropped. + await invalidateProjectStagedDeployments(project); + await discardProjectStagedApplications(componentPath); + await discardProjectActivationArtifacts(componentPath); + await discardRetainedPrevious(componentPath); + // Retires any interrupted-extraction aside and renames the live tree aside before removing + // it, so startup recovery can never restore a tree over a dropped component. await dropComponentDirectory(componentPath, project, log); + // The directory goes FIRST and the config entry only after, because applications in the + // components root load by directory scan — the root-config entry is what CONSTRAINS where one + // is served (host/urlPath), not what makes it load. Dropping config first leaves a window + // where the tree is still discoverable with its mount gone, so a crash there resurrects the + // component served on every host. Losing the operator's isolation is worse than a drop that + // has to be re-run, which is what this ordering leaves instead: tree gone, entry present, + // re-runnable — and the same residual `main` has today. Both writes are one transaction, + // config before the application lock. + const dropTransaction = await createApplicationConfigTransaction(project, null); + await dropTransaction.commit(); } else if (await fs.pathExists(pathToComponent)) { await fs.remove(pathToComponent); } @@ -1216,7 +1607,12 @@ async function dropComponent(req) { await fs.writeFile(packageJsonPath, JSON.stringify(packageJson, null, 2), 'utf8'); } - configUtils.deleteConfigFromFile([project]); + if (file) { + // Under the persistent-state lock: an unlocked delete can be clobbered by a concurrent + // activation writing back a document that still contains this project. A full drop has + // already removed its config above. + await withPersistentStateLock(async () => configUtils.deleteConfigFromFile([project])); + } }, componentDropLockOptions(project) ); @@ -1237,6 +1633,7 @@ exports.addComponent = addComponent; exports.dropCustomFunctionProject = dropCustomFunctionProject; exports.packageComponent = packageComponent; exports.deployComponent = deployComponent; +exports.revertComponent = revertComponent; exports.getComponents = getComponents; exports.getComponentFile = getComponentFile; exports.setComponentFile = setComponentFile; diff --git a/components/operationsValidation.js b/components/operationsValidation.js index aa3b95c6b1..4516d267db 100644 --- a/components/operationsValidation.js +++ b/components/operationsValidation.js @@ -14,6 +14,7 @@ const { ENV_ENCRYPTED_PREFIX } = require('../utility/envFile.ts'); // with the CLI (utility/componentNames.ts): `harper deploy setup=true` resolves a project name and a // credential host client-side, and has to reject exactly what these schemas would. const { PROJECT_NAME_PATTERN: PROJECT_FILE_NAME_REGEX, GIT_HOST_PATTERN } = require('../utility/componentNames.ts'); +const DEPLOYMENT_ID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; // dotenv's accepted key character set. Restricting keys to this prevents a crafted key (e.g. one // containing `=` or a newline) from injecting extra assignments into a .env file. @@ -31,6 +32,7 @@ module.exports = { dropCustomFunctionProjectValidator, packageComponentValidator, deployComponentValidator, + revertComponentValidator, setComponentFileValidator, getComponentFileValidator, dropComponentFileValidator, @@ -446,6 +448,63 @@ const GIT_CREDENTIAL_ENTRY = Joi.object({ .xor('token', 'secret') .unknown(false); +// The kind-heterogeneous deploy credentials array, shared by deploy_component and its two-phase +// peer fan-out (component_deploy_phase) so both stay in lockstep. An entry's +// kind is implied by its identifying key (`registry` → npm registry auth, otherwise git host auth), +// dispatched here so a malformed entry reports what is actually wrong with it rather than a generic +// "no alternative matched". +const CREDENTIALS_ARRAY_SCHEMA = Joi.array() + .items( + Joi.alternatives().conditional('.registry', { + is: Joi.exist(), + then: REGISTRY_CREDENTIAL_ENTRY, + otherwise: GIT_CREDENTIAL_ENTRY, + }) + ) + .optional(); + +// A component's URL mount path. Rejects `..` so a deploy can't mount a component outside its intended +// path. Shared across deploy_component and the two-phase sub-operations that persist component config. +const URL_PATH_SCHEMA = Joi.string() + .min(1) + .custom((value, helpers) => { + if (value.includes('..')) return helpers.error('any.invalid'); + // A component mount has no relative base and WHATWG clients strip '.' segments before + // sending the request, so a dot-segment mount would simply be unreachable. + if (value.split('/').includes('.')) return helpers.error('string.dotSegment'); + return value; + }) + .optional() + .messages({ + 'any.invalid': '{#label} must not contain ".."', + 'string.dotSegment': '{#label} must not contain "." path segments', + }); + +// Virtual hostname the component is served on. Like `urlPath`, this is deployment routing and belongs +// on the root-config entry, not in the component's own config.yaml. `hostname()` rejects a value +// carrying a port or path, which would never match the router's host compare. IPv6 literals are +// accepted in their bare form only — the router unwraps the brackets it finds in a Host header, so a +// bracketed value here would never match. Shared for the same reason as URL_PATH_SCHEMA. +const HOST_SCHEMA = Joi.string() + .custom((value, helpers) => { + if (value.startsWith('[') || value.endsWith(']')) return helpers.error('string.bracketedHost'); + return value; + }) + .custom((value, helpers) => { + if (!HOSTNAME_SCHEMA.validate(value).error) return value; + // Accept a bare IPv6 literal, which `hostname()` rejects but the router can match. + return IPV6_SCHEMA.validate(value).error ? helpers.error('string.hostname') : value; + }) + .optional() + .messages({ 'string.bracketedHost': '{#label} must not be bracketed; use the bare IPv6 literal' }); + +// `registryAuth` was the credentials field's name on the 5.2 dev line. Rejected rather than ignored: +// validation allows unknown keys, so a caller still sending it would otherwise get a deploy that +// silently installs with no credentials. Shared so every deploy-family op rejects it identically. +const FORBIDDEN_REGISTRY_AUTH = Joi.any().forbidden().messages({ + 'any.unknown': `'registryAuth' has been renamed to 'credentials'`, +}); + /** * Validate deployComponent requests. * @param req @@ -465,69 +524,68 @@ function deployComponentValidator(req) { deployment_timeout: Joi.number().min(0).optional(), force: Joi.boolean().optional(), ignore_replication_errors: Joi.boolean().optional(), - urlPath: Joi.string() - .min(1) - .custom((value, helpers) => { - if (value.includes('..')) return helpers.error('any.invalid'); - // A component mount has no relative base and WHATWG clients strip '.' segments before - // sending the request, so a dot-segment mount would simply be unreachable. - if (value.split('/').includes('.')) return helpers.error('string.dotSegment'); - return value; - }) - .optional() - .messages({ - 'any.invalid': '{#label} must not contain ".."', - 'string.dotSegment': '{#label} must not contain "." path segments', - }), - // Virtual hostname the component is served on. Like `urlPath`, this is deployment routing and - // belongs on the root-config entry, not in the component's own config.yaml. `hostname()` - // rejects a value carrying a port or path, which would never match the router's host compare. - // IPv6 literals are accepted in their bare form only — the router unwraps the brackets it - // finds in a Host header, so a bracketed value here would never match. - host: Joi.string() - .custom((value, helpers) => { - if (value.startsWith('[') || value.endsWith(']')) return helpers.error('string.bracketedHost'); - return value; - }) - .custom((value, helpers) => { - if (!HOSTNAME_SCHEMA.validate(value).error) return value; - // Accept a bare IPv6 literal, which `hostname()` rejects but the router can match. - return IPV6_SCHEMA.validate(value).error ? helpers.error('string.hostname') : value; - }) - .optional() - .messages({ 'string.bracketedHost': '{#label} must not be bracketed; use the bare IPv6 literal' }), - // Deploy credentials. The array is kind-heterogeneous: an entry's kind is implied by its - // identifying key rather than a separate discriminator field, so a new kind is added as - // another item alternative here without reshaping the field. Today: npm registry auth - // (`registry`) and git host auth for a git-reference package (`host`, #1792). - // - // Every kind supplies its credential exactly one of two ways: - // - `token`: a literal token, used only for this node's install and never persisted or - // replicated (stripped from req before replicateOperation). - // - `secret`: the name of an hdb_secret row (#1550); the token is resolved by decrypting - // that row on this node at deploy time, so the credential lives in the secrets store - // (reference, not embed) instead of travelling in the operation body. - // Dispatched on the presence of `registry` rather than tried as alternatives, so a malformed - // entry reports what is actually wrong with it (a newline in the token, an invalid scope) rather - // than a generic "no alternative matched". - credentials: Joi.array() - .items( - Joi.alternatives().conditional('.registry', { - is: Joi.exist(), - then: REGISTRY_CREDENTIAL_ENTRY, - otherwise: GIT_CREDENTIAL_ENTRY, - }) - ) - .optional(), - // `registryAuth` was this field's name on the 5.2 dev line before it grew to carry other - // credential kinds. Rejected rather than ignored: validation allows unknown keys, so a caller - // still sending it would otherwise get a deploy that silently installs with no credentials. - registryAuth: Joi.any().forbidden().messages({ - 'any.unknown': `'registryAuth' has been renamed to 'credentials'`, + // Automatic rollback is deliberately NOT offered. A node that reports `failed` may still have + // completed its swap and then failed the work that follows, so auto-reverting "the nodes that + // failed" can roll an untouched node an extra version back. Roll back explicitly, by target, with + // revert_component. See DESIGN.md, "Partial activation". + revert_on_failure: Joi.any().forbidden().messages({ + 'any.unknown': `'revert_on_failure' is not supported; roll back explicitly with revert_component`, }), + _deploymentId: Joi.any().forbidden(), + // Removed with the coordination protocol (see #2301). Forbidden rather than dropped, because the + // schema allows unknown keys: a caller still sending the unreleased contract would otherwise have + // `activate: false` silently ignored and get a full deploy — the opposite of the intent. + activate: Joi.any().forbidden().messages({ + 'any.unknown': `'activate' is not supported; staged deploys are tracked in HarperFast/harper#2301`, + }), + two_phase: Joi.any().forbidden().messages({ + 'any.unknown': `'two_phase' is not supported; every deploy now stages and swaps on each node`, + }), + deployment_id: Joi.any().forbidden().messages({ + 'any.unknown': `'deployment_id' is not supported; activating a previously staged deployment is tracked in HarperFast/harper#2301`, + }), + _phase: Joi.any().forbidden(), + urlPath: URL_PATH_SCHEMA, + host: HOST_SCHEMA, + // Deploy credentials. Each entry is npm registry auth (`registry`) or git host auth (`host`, + // #1792), and supplies its secret either as a literal `token` (used only for this node's + // install, never persisted/replicated) or a `secret` reference to an hdb_secret row (#1550). + credentials: CREDENTIALS_ARRAY_SCHEMA, + registryAuth: FORBIDDEN_REGISTRY_AUTH, }) .with('urlPath', 'package') .with('host', 'package'); return validator.validateBySchema(req, deployProjSchema); } + +/** + * Validate revert_component requests — swap a component's live version back to its retained previous + * version. There are no build inputs: nothing is fetched, resolved or installed, because the bytes + * being reverted to are already on disk. Just the project, the version being targeted, an optional + * restart, and the replication controls. + * @param req + * @returns {*} + */ +function revertComponentValidator(req) { + const revertSchema = Joi.object({ + project: Joi.string() + .pattern(PROJECT_FILE_NAME_REGEX) + .required() + .messages({ 'string.pattern.base': HDB_ERROR_MSGS.BAD_PROJECT_NAME }), + // The deployment the caller expects to be live once this returns. REQUIRED, and the whole reason + // revert is safe to retry: a revert that names its target is a no-op when that version is already + // live, where a bare "swap to the other one" toggle would flip the rejected release back in if a + // caller retried after losing the first response. + to_deployment_id: Joi.string().pattern(DEPLOYMENT_ID_REGEX).required().messages({ + 'string.pattern.base': `'to_deployment_id' must be a UUID`, + 'any.required': `'to_deployment_id' is required: name the deployment you expect to be live after the revert (list_deployments reports it, and deploy_component returns it)`, + }), + restart: Joi.alternatives().try(Joi.boolean(), Joi.string().valid('rolling')).optional(), + deployment_timeout: Joi.number().min(0).optional(), + ignore_replication_errors: Joi.boolean().optional(), + force: Joi.boolean().optional(), + }); + + return validator.validateBySchema(req, revertSchema); +} diff --git a/config/configUtils.ts b/config/configUtils.ts index 846d1918ed..6dd200b7d8 100644 --- a/config/configUtils.ts +++ b/config/configUtils.ts @@ -1263,7 +1263,7 @@ export async function addConfig(topLevelElement, values) { atomicWriteFile(getConfigFilePath(), String(configDoc)); } -export function deleteConfigFromFile(param: string) { +export function deleteConfigFromFile(param: Array) { const configFilePath = getConfigFilePath(hdbUtils.getPropsFilePath()); const configDoc = parseYamlDoc(configFilePath); configDoc.deleteIn(param); diff --git a/integrationTests/deploy/deploy-tracking-events.test.ts b/integrationTests/deploy/deploy-tracking-events.test.ts index 0d80f2f600..7e86b14044 100644 --- a/integrationTests/deploy/deploy-tracking-events.test.ts +++ b/integrationTests/deploy/deploy-tracking-events.test.ts @@ -132,9 +132,10 @@ suite('Deployment tracking — events + SSE', (ctx: ContextWithHarper) => { ok(Array.isArray(row.event_log), 'event_log should be an array'); ok(row.event_log.length >= 2, `expected at least 2 events, got ${row.event_log.length}`); const phases = row.event_log.filter((e: any) => e.event === 'phase').map((e: any) => e.data?.phase); - // We emit prepare → (load) → replicate → success in the lifecycle. Verify the spine. - ok(phases.includes('prepare'), `event_log should include a prepare phase: ${phases.join(',')}`); - ok(phases.includes('replicate'), `event_log should include a replicate phase: ${phases.join(',')}`); + // A two-phase deploy emits stage → (load) → activate → success. Verify the spine (load is only + // emitted off the main thread, so it isn't asserted here). + ok(phases.includes('stage'), `event_log should include a stage phase: ${phases.join(',')}`); + ok(phases.includes('activate'), `event_log should include an activate phase: ${phases.join(',')}`); }); test('get_deployment with Accept: text/event-stream replays event_log and closes cleanly', async () => { @@ -205,7 +206,7 @@ suite('Deployment tracking — events + SSE', (ctx: ContextWithHarper) => { // Poll list_deployments until we see the in-flight row. The recorder row is created // after the multipart body is fully received (which is fast for a tiny fixture). - const TERMINAL = new Set(['success', 'failed', 'rolled_back']); + const TERMINAL = new Set(['success', 'failed', 'rolled_back', 'staged']); let inFlightId: string | null = null; for (let i = 0; i < 15 && !inFlightId; i++) { await sleep(300); diff --git a/integrationTests/deploy/deploy-tracking-peer-branch.test.ts b/integrationTests/deploy/deploy-tracking-peer-branch.test.ts index 9964d05432..49fe7fa5af 100644 --- a/integrationTests/deploy/deploy-tracking-peer-branch.test.ts +++ b/integrationTests/deploy/deploy-tracking-peer-branch.test.ts @@ -1,36 +1,23 @@ /** - * Deployment tracking — peer-side branch. + * Deployment tracking — peer-operation authorization boundary. * - * In a real multi-node deploy, the origin strips `req.payload` before `replicateOperation` - * and the peer reads the tarball from the replicated `hdb_deployment.payload_blob` row - * attribute instead. This test exercises that **peer-side branch** in isolation on a - * single node by: + * In a real multi-node deploy, the origin replicates the whole `deploy_component` + * operation and the peer reads the tarball from the replicated + * `hdb_deployment.payload_blob` row. The authorization-bypass context that admits that + * operation exists only around trusted replication dispatch, so an HTTP caller must not + * be able to impersonate a peer with the legacy `_deploymentId` marker. This test: * * 1. Doing a normal deploy to populate an `hdb_deployment` row with a `payload_blob`. - * 2. Submitting a second `deploy_component` operation with `_deploymentId` set to that - * row's id and **no** `payload` field — the same shape origin produces for peers. - * 3. Asserting the deploy completes successfully — meaning the peer-side branch in - * `deployComponent` found the row, streamed `payload_blob`, and ran prepare/install/load - * from the blob bytes. + * 2. Submitting a second public `deploy_component` operation with `_deploymentId` set. + * 3. Asserting validation rejects the caller-controlled internal marker. * - * The true 3-node test (verifies BLOB_CHUNK replication actually delivers the row to - * peers, and that `peer_results` is populated) lives in harper-pro, where the actual - * `replicateOperation` is implemented. This OSS test only verifies the handler wiring. + * The true 3-node test (including the trusted peer operation, BLOB_CHUNK delivery, and + * `peer_results`) lives in harper-pro, where `replicateOperation` is implemented. */ import { suite, test, before, after } from 'node:test'; import { ok, strictEqual } from 'node:assert'; import { join } from 'node:path'; -import { - existsSync, - mkdtempSync, - mkdirSync, - readdirSync, - readFileSync, - rmSync, - statSync, - truncateSync, - writeFileSync, -} from 'node:fs'; +import { existsSync, mkdtempSync, mkdirSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { randomFillSync } from 'node:crypto'; import { setTimeout as sleep } from 'node:timers/promises'; @@ -52,15 +39,6 @@ function filesUnder(directory: string): string[] { }); } -async function waitForMarkedAside(asidePath: string): Promise { - const deadline = Date.now() + 10_000; - while (Date.now() < deadline) { - if (existsSync(asidePath) && readdirSync(asidePath).some((entry) => entry.startsWith('.in-progress-'))) return; - await sleep(5); - } - throw new Error('Timed out waiting for peer extraction to move the previous component aside'); -} - function postMultipart( url: URL, contentType: string, @@ -116,10 +94,9 @@ async function callOperation( return { status: res.status, body: parsed, rawText: text }; } -suite('Deployment tracking — peer-side branch', (ctx: ContextWithHarper) => { +suite('Deployment tracking — peer-operation authorization boundary', (ctx: ContextWithHarper) => { let fixtureDir: string; let seedDeploymentId: string; - let seedPayloadBlobPath: string; before(async () => { await startHarper(ctx, { config: { storage: { blobReadTimeout: 2000 } }, env: {} }); @@ -171,85 +148,18 @@ suite('Deployment tracking — peer-side branch', (ctx: ContextWithHarper) => { .sort((left, right) => right.size - left.size); const [payloadBlobFile] = blobFiles; ok(payloadBlobFile?.size > 1024 * 1024, 'expected the retained deployment payload to be file-backed'); - seedPayloadBlobPath = payloadBlobFile.path; }); - // On Bun the deploy hangs after extraction when reading a Web ReadableStream from a - // file-backed blob inside the same Harper process — same code passes on Node v22/v24 - // across Linux and Windows. Skipping for now; the harper-pro 3-node cluster test - // (HarperFast/harper-pro#221) covers the same code path end-to-end with real replication. - const skipOnBun = process.env.HARPER_RUNTIME === 'bun'; - test( - 'peer-side branch: deploy_component with _deploymentId + no payload uses the row blob', - { skip: skipOnBun }, - async () => { - // Simulate the operation shape origin produces for peers via `replicateOperation`: - // `_deploymentId` set, no `payload`, no multipart. The handler should detect this is a - // replicated execution and source the tarball from the row's payload_blob. - const response = await callOperation(ctx, { - operation: 'deploy_component', - project: PEER_PROJECT, - restart: false, - _deploymentId: seedDeploymentId, - }); - strictEqual(response.status, 200, `peer-side deploy should succeed; got ${response.status}: ${response.rawText}`); - - // Confirm the component was actually written on disk (peer code path ran extraction - // from the row's payload_blob and not from a missing req.payload). - const fetched = await fetch(`${ctx.harper.operationsAPIURL}/${PEER_PROJECT}/`, { - headers: { - Authorization: - 'Basic ' + Buffer.from(`${ctx.harper.admin.username}:${ctx.harper.admin.password}`).toString('base64'), - }, - }); - // The component exposes nothing routable, but a 404 from the component (vs. a 503/connection - // failure) confirms it loaded — Harper only routes to deployed component names. - ok( - fetched.status === 404 || fetched.status === 200, - `expected component to be reachable (any 200/404 from the loaded component), got ${fetched.status}` - ); - } - ); - - // This test intentionally corrupts the seed deployment blob and must remain the suite's final deployment test. - test( - 'peer-side branch: a payload blob failure after the swap restores the previous tree', - { skip: skipOnBun }, - async () => { - ok(seedPayloadBlobPath, 'seed payload blob path should be recorded before the failure test'); - const componentPath = join(ctx.harper.dataRootDir, 'components', PEER_PROJECT); - const asidePath = join(ctx.harper.dataRootDir, 'components', '.deploy-aside', PEER_PROJECT); - const oldOnlyPath = join(componentPath, 'old-only.txt'); - writeFileSync(oldOnlyPath, 'previous bytes\n'); - truncateSync(seedPayloadBlobPath, 128 * 1024); - - const responsePromise = callOperation(ctx, { - operation: 'deploy_component', - project: PEER_PROJECT, - restart: false, - _deploymentId: seedDeploymentId, - deployment_timeout: 5000, - }); - const [asideResult, responseResult] = await Promise.allSettled([waitForMarkedAside(asidePath), responsePromise]); - if (responseResult.status === 'rejected') throw responseResult.reason; - if (asideResult.status === 'rejected') { - throw new Error(`${asideResult.reason}; deploy response: ${responseResult.value.rawText}`); - } - const response = responseResult.value; - - strictEqual(response.status, 500, `peer deploy should fail after payload truncation: ${response.rawText}`); - ok( - /blob|payload|incomplete|stall/i.test(response.rawText), - `peer failure should report payload delivery, got: ${response.rawText}` - ); - strictEqual(readFileSync(oldOnlyPath, 'utf8'), 'previous bytes\n'); - strictEqual(readFileSync(join(componentPath, 'web', 'index.html'), 'utf8'), '

Hello, Peer Branch!

'); - strictEqual(existsSync(asidePath), false, 'rollback should remove the component deploy staging directory'); - } - ); - - // Note: the bogus-_deploymentId-id timeout case isn't covered here because the - // awaitDeploymentRow 120s default would balloon test time. The timeout path (and the - // per-deploy deployment_timeout override) is exercised by the unit tests for - // awaitDeploymentRow directly. + test('public deploy_component rejects the legacy _deploymentId peer marker', async () => { + const response = await callOperation(ctx, { + operation: 'deploy_component', + project: PEER_PROJECT, + restart: false, + _deploymentId: seedDeploymentId, + }); + strictEqual(response.status, 400, `internal marker should be rejected; got: ${response.rawText}`); + strictEqual(response.body.error, "'_deploymentId' is not allowed"); + }); + // Peer behavior is the replicated `deploy_component` operation itself, so there is no private + // peer entry point to drive here; the three-node harper-pro suite covers real replication. }); diff --git a/integrationTests/deploy/deploy-tracking.test.ts b/integrationTests/deploy/deploy-tracking.test.ts index 57bc623c40..255e6d28b2 100644 --- a/integrationTests/deploy/deploy-tracking.test.ts +++ b/integrationTests/deploy/deploy-tracking.test.ts @@ -196,7 +196,9 @@ suite('Deployment tracking', (ctx: ContextWithHarper) => { // match what the install command actually printed; an empty array would pass the // pre-fix code, so assert on the line content too. const body = JSON.parse(response.body); - strictEqual(body.phase, 'prepare'); + // The install runs during the stage phase of a two-phase deploy, so a failed install is + // recorded against 'stage' (was 'prepare' in the one-shot flow). + strictEqual(body.phase, 'stage'); ok(Array.isArray(body.install_output?.lines) && body.install_output.lines.length > 0); ok( body.install_output.lines.some( diff --git a/json/systemSchema.json b/json/systemSchema.json index bd29bb501c..e986a1bf51 100644 --- a/json/systemSchema.json +++ b/json/systemSchema.json @@ -424,6 +424,9 @@ { "attribute": "rollback_of" }, + { + "attribute": "activation_spec" + }, { "attribute": "error" }, diff --git a/server/serverHelpers/serverHandlers.js b/server/serverHelpers/serverHandlers.js index 6da3d60d11..d175e33daa 100644 --- a/server/serverHelpers/serverHandlers.js +++ b/server/serverHelpers/serverHandlers.js @@ -33,6 +33,7 @@ const { ProgressEmitter, createSSEResponseStream } = require('./progressEmitter. // the client disconnects (the emitter's abort signal), so subscribers see new lines live. const SSE_PROGRESS_OPERATIONS = new Set([ terms.OPERATIONS_ENUM.DEPLOY_COMPONENT, + terms.OPERATIONS_ENUM.REVERT_COMPONENT, terms.OPERATIONS_ENUM.GET_DEPLOYMENT, terms.OPERATIONS_ENUM.READ_LOG, ]); diff --git a/server/serverHelpers/serverUtilities.ts b/server/serverHelpers/serverUtilities.ts index 8b4c95d9cf..7ee13eb542 100644 --- a/server/serverHelpers/serverUtilities.ts +++ b/server/serverHelpers/serverUtilities.ts @@ -361,6 +361,9 @@ _assignPackageExport('operation', operation); export function operation(operation: OperationRequestBody, context: Context, authorize: boolean) { operation.hdb_user = context?.user; const bypassAuth = !authorize; + // Deliberately NOT stamped onto `operation` as `bypass_auth`: that name is client-supplied input + // which serverHandlers strips before dispatch, and nothing reads it. The bypass travels in + // operationAuthorizationState's async context, which is what isTrustedReplicatedOperation checks. return runWithOperationAuthorizationBypass(bypassAuth, () => { const operation_function = chooseOperation(operation, bypassAuth); return processLocalTransaction({ body: operation }, operation_function); @@ -602,6 +605,10 @@ function initializeOperationFunctionMap(): Map { }); }); + // The CLI still negotiates with pre-5.1 servers (`targetSupportsStreamingDeploy` → + // `_legacyDeploy` → CBOR body). Removing the staged-deploy probe took this whole block with it, + // which left those still-live branches uncovered — a later request-format change could silently + // break deploying to an older Harper. describe('deploy_component cross-version compatibility', () => { const target = 'https://example.com:9925/'; let originalPackageDirectory; @@ -1631,6 +1635,36 @@ describe('cliOperations', () => { }); }); +describe('harper revert CLI verb', () => { + const { buildRequest, verbRequirementError } = cliOperationsModule; + let savedArgv; + beforeEach(() => { + savedArgv = process.argv; + }); + afterEach(() => { + process.argv = savedArgv; + }); + + it('`revert` maps to revert_component and carries the verb marker', () => { + // The marker has to survive buildRequest for the guard below to fire at all. `revert` deliberately + // lives in OP_VERB_PROPS rather than OP_ALIASES: buildRequest checks the alias table FIRST, so an + // alias entry would set the operation and never attach `_cliVerb`. + process.argv = ['node', 'harper', 'revert', 'project=my_app', 'to_deployment_id=abc-123']; + const req = buildRequest(); + assert.strictEqual(req.operation, 'revert_component'); + assert.strictEqual(req.to_deployment_id, 'abc-123'); + assert.strictEqual(verbRequirementError(req), null); + }); + + it('`revert` WITHOUT a to_deployment_id is rejected before anything is sent', () => { + // A revert with no target would be a blind toggle, which is unsafe to retry. + process.argv = ['node', 'harper', 'revert', 'project=my_app']; + const req = buildRequest(); + assert.strictEqual(req.operation, 'revert_component'); + assert.match(verbRequirementError(req), /to_deployment_id/); + }); +}); + describe('deploy by reference (by_ref)', () => { const { prepareDeployByRef, resolveGitTarget, resolveCredentialHost, deriveGitSecretName } = cliOperationsModule; const GITHUB_ENV = ['GITHUB_REPOSITORY', 'GITHUB_SHA', 'GITHUB_REF', 'GITHUB_EVENT_PATH']; diff --git a/unitTests/components/deployOperations.test.js b/unitTests/components/deployOperations.test.js new file mode 100644 index 0000000000..cc5e268cd3 --- /dev/null +++ b/unitTests/components/deployOperations.test.js @@ -0,0 +1,457 @@ +'use strict'; + +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const { existsSync } = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const zlib = require('node:zlib'); +const { Readable } = require('node:stream'); +const tarfs = require('tar-fs'); + +const testUtils = require('../testUtils.js'); +testUtils.preTestPrep(); + +const { DEPLOY_STAGING_DIR, DEPLOY_ACTIVATION_DIR } = require('#src/components/Application'); +const operations = require('#src/components/operations'); +const { restartNeeded, resetRestartNeeded } = require('#src/components/requestRestart'); +const { server } = require('#src/server/Server'); +const { databases } = require('#src/resources/databases'); +const { SYSTEM_TABLE_NAMES, CONFIG_PARAMS } = require('#src/utility/hdbTerms'); +const { getConfigPath, readConfigFile } = require('#src/config/configUtils'); +const environment = require('#src/utility/environment/environmentManager'); +const { runWithOperationAuthorizationBypass } = require('#src/server/serverHelpers/operationAuthorizationState'); + +const COMPONENTS_ROOT = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); +const DEPLOYMENT_TABLE = SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME; + +function packDirectory(directory) { + return new Promise((resolve, reject) => { + const chunks = []; + tarfs + .pack(directory) + .pipe(zlib.createGzip()) + .on('data', (chunk) => chunks.push(chunk)) + .on('end', () => resolve(Buffer.concat(chunks))) + .on('error', reject); + }); +} + +async function makePayload(marker, version = marker, withNodeModules = true) { + const source = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-phase-op-')); + await fs.writeFile(path.join(source, 'package.json'), JSON.stringify({ name: 'phase-op', version })); + await fs.writeFile(path.join(source, 'index.js'), `module.exports = ${JSON.stringify(marker)};\n`); + if (withNodeModules) await fs.mkdir(path.join(source, 'node_modules'), { recursive: true }); + const payload = await packDirectory(source); + await fs.rm(source, { recursive: true, force: true }); + return payload; +} + +describe('deploy_component staged deploy', function () { + this.timeout(30_000); + const rows = new Map(); + let priorTable; + let priorReplicate; + let priorSafeMode; + let sequence = 0; + const names = []; + + before(async () => { + priorSafeMode = process.env.HARPER_SAFE_MODE; + process.env.HARPER_SAFE_MODE = 'true'; + // The first component operation completes lazy server initialization, which replaces + // databases.system. Run it before installing the table seam used by these tests. + // + // One-shot on purpose: the separated phases now require deployment tracking, and the whole point + // of this warmup is that it runs BEFORE the table seam exists. `after` removes the component + // directory along with every other fixture name. + await operations.deployComponent({ + project: name(), + payload: await makePayload('warmup'), + restart: false, + }); + if (!databases.system) databases.system = {}; + priorTable = databases.system[DEPLOYMENT_TABLE]; + }); + + beforeEach(() => { + resetRestartNeeded(); + rows.clear(); + databases.system[DEPLOYMENT_TABLE] = { + async get(id) { + return rows.get(id); + }, + async put(row) { + rows.set(row.deployment_id, { ...row }); + }, + async patch(id, partial) { + const row = rows.get(id); + if (row) rows.set(id, { ...row, ...partial }); + }, + async *search(conditions = []) { + for (const row of rows.values()) { + if (conditions.every((condition) => row[condition.attribute] === condition.value)) yield row; + } + }, + }; + priorReplicate = server.replication.replicateOperation; + server.replication.replicateOperation = async () => ({ replicated: [] }); + }); + + afterEach(() => { + server.replication.replicateOperation = priorReplicate; + resetRestartNeeded(); + }); + + after(async () => { + if (priorSafeMode === undefined) delete process.env.HARPER_SAFE_MODE; + else process.env.HARPER_SAFE_MODE = priorSafeMode; + if (priorTable === undefined) delete databases.system[DEPLOYMENT_TABLE]; + else databases.system[DEPLOYMENT_TABLE] = priorTable; + for (const name of names) await fs.rm(path.join(COMPONENTS_ROOT, name), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, '.deploy-aside'), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, '.deploy-previous'), { recursive: true, force: true }); + }); + + function name() { + const value = `phase_op_${process.pid}_${sequence++}`; + names.push(value); + return value; + } + + it('replicates the payload itself when there is no deployment table to carry it', async () => { + // The legacy single-phase path is the documented escape hatch, so it has to actually be safe. It + // normally strips `req.payload` because peers read the bytes from the row's payload_blob — but with + // no table there is no row, so stripping left peers holding a `_deploymentId`, no bytes and nothing + // to resolve, after this node was already live. Asserting on the REPLICATED REQUEST, not just on + // local activation: replication is stubbed here, so a local-only assertion proves nothing about peers. + const project = name(); + const priorTable = databases.system[DEPLOYMENT_TABLE]; + const priorReplicate = server.replication.replicateOperation; + const replicated = []; + server.replication.replicateOperation = async (op) => { + replicated.push(op); + return { replicated: [] }; + }; + delete databases.system[DEPLOYMENT_TABLE]; + try { + // A real Readable, not a reusable Buffer: ingest DRAINS the source, so a Buffer would hide the + // actual failure (peers receiving an exhausted stream / EOF). + const bytes = await makePayload('untracked-oneshot'); + await operations.deployComponent({ + project, + payload: Readable.from([bytes]), + restart: false, + }); + assert.match( + await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), + /untracked-oneshot/, + 'the component goes live locally' + ); + assert.strictEqual(replicated.length, 1, 'the deploy is replicated to peers'); + // Assert the BYTES, not merely that the property exists — presence of a spent stream is exactly + // the bug this covers. + const sent = replicated[0].payload; + assert.strictEqual(Buffer.isBuffer(sent), true, `peers must receive replayable bytes, got ${typeof sent}`); + assert.strictEqual(Buffer.compare(sent, bytes), 0, 'and the bytes must be the payload that was uploaded'); + } finally { + server.replication.replicateOperation = priorReplicate; + databases.system[DEPLOYMENT_TABLE] = priorTable; + } + }); + + it('normalizes a string install_allow_scripts rather than reading it as truthy', async () => { + // Joi coerces it, but validateBySchema discards `result.value`, so the raw string reaches the + // handler. `install_allow_scripts: 'false'` would then read as truthy and run package lifecycle + // scripts for a caller that explicitly disabled them — over multipart/form, where every value + // arrives as a string. + const project = name(); + const result = await operations.deployComponent({ + project, + payload: await makePayload('1.0.0'), + install_allow_scripts: 'false', + restart: false, + }); + + assert.ok(result, 'the deploy succeeds'); + assert.strictEqual( + rows.get(result.deployment_id).activation_spec.install_allow_scripts, + false, + 'and the activation spec records a real boolean, not the string' + ); + }); + + it('uses a no-custody literal registry token for the origin install without recording it', async () => { + const project = name(); + const token = 'transient-origin-token'; + const installCommand = + `node -e "const fs=require('fs');` + + `const value=fs.readFileSync(process.env.npm_config_userconfig||process.env.NPM_CONFIG_USERCONFIG,'utf8');` + + `if(!value.includes('//registry.example.com/:_authToken='))process.exit(7);` + + `fs.writeFileSync('credential-seen','yes')"`; + const result = await operations.deployComponent({ + project, + payload: await makePayload('credential-origin', '6.0.0', false), + install_command: installCommand, + credentials: [{ registry: 'https://registry.example.com', token }], + restart: false, + }); + + // The install ran in the staging directory; the marker it wrote travels with the tree through the + // swap, so it lands at the live path. + assert.strictEqual( + await fs.readFile(path.join(COMPONENTS_ROOT, project, 'credential-seen'), 'utf8'), + 'yes', + 'the install saw the credential, and its output was swapped in with the tree' + ); + assert.strictEqual(rows.get(result.deployment_id).activation_spec.credentials, null); + assert.doesNotMatch(JSON.stringify(rows.get(result.deployment_id)), new RegExp(token)); + }); + + it('stages and activates a full deploy before reporting success', async () => { + const project = name(); + const result = await operations.deployComponent({ + project, + payload: await makePayload('full-deploy', '6.0.0'), + }); + + assert.match(result.message, /Successfully deployed/); + assert.strictEqual(rows.get(result.deployment_id).status, 'success'); + assert.match(await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), /full-deploy/); + assert.strictEqual(existsSync(path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR, result.deployment_id)), false); + assert.strictEqual(restartNeeded(), true, 'a new component activated without restart requires one'); + }); + + // revert_component + + it('reverts the cluster to a named previous deployment and fans the target out to peers', async () => { + const project = name(); + const first = await operations.deployComponent({ project, payload: await makePayload('rev-v1', '1.0.0') }); + const second = await operations.deployComponent({ project, payload: await makePayload('rev-v2', '2.0.0') }); + const fanout = []; + server.replication.replicateOperation = async (op) => { + fanout.push(op); + return { replicated: [] }; + }; + + const result = await operations.revertComponent({ project, to_deployment_id: first.deployment_id }); + + assert.strictEqual(result.reverted, true); + assert.strictEqual(result.to_deployment_id, first.deployment_id); + assert.strictEqual(result.from_deployment_id, second.deployment_id); + assert.match(await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), /rev-v1/); + assert.strictEqual(rows.get(result.deployment_id).status, 'rolled_back'); + assert.strictEqual( + rows.get(result.deployment_id).rollback_of, + second.deployment_id, + 'the audit row records which deployment the rollback took out of service' + ); + assert.strictEqual(fanout.length, 1, 'peers get the revert'); + assert.strictEqual(fanout[0].operation, 'revert_component'); + assert.strictEqual( + fanout[0].to_deployment_id, + first.deployment_id, + 'peers are told WHICH version to end on, so the fan-out is idempotent per node' + ); + }); + + it('is a no-op when the requested deployment is already live, so a retry is safe', async () => { + const project = name(); + const first = await operations.deployComponent({ project, payload: await makePayload('retry-v1', '1.0.0') }); + await operations.deployComponent({ project, payload: await makePayload('retry-v2', '2.0.0') }); + await operations.revertComponent({ project, to_deployment_id: first.deployment_id }); + + // The caller lost the first response and retried the identical request. + const retry = await operations.revertComponent({ project, to_deployment_id: first.deployment_id }); + + assert.strictEqual(retry.reverted, false); + assert.match(retry.message, /already running/); + assert.match( + await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), + /retry-v1/, + 'a retried revert must not toggle the rejected version back in' + ); + }); + + it('rejects a revert with no target, and one whose target is not retained', async () => { + const project = name(); + await operations.deployComponent({ project, payload: await makePayload('target-v1', '1.0.0') }); + await operations.deployComponent({ project, payload: await makePayload('target-v2', '2.0.0') }); + + await assert.rejects( + () => operations.revertComponent({ project }), + /to_deployment_id/, + 'the target is mandatory — that is what makes a retry safe' + ); + await assert.rejects( + () => operations.revertComponent({ project, to_deployment_id: '00000000-0000-4000-8000-000000000000' }), + /neither the live version/, + 'only the immediately-previous version is retained' + ); + assert.match( + await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), + /target-v2/, + 'a refused revert changes nothing' + ); + }); + + it('takes the package reference out of root config when reverting away from a package deploy', async () => { + // Without this, installApplications() would reinstall the reverted-away package over the restored + // directory on the next cold start and silently undo the rollback. + const project = name(); + const packaged = await operations.deployComponent({ + project, + payload: await makePayload('cfg-packaged', '1.0.0'), + }); + // Stamp root config as a package deploy would have, then activate a payload version over it. + const { addConfig } = require('#src/config/configUtils'); + await addConfig(project, { package: 'some-pkg@1.0.0' }); + const plain = await operations.deployComponent({ project, payload: await makePayload('cfg-plain', '2.0.0') }); + assert.ok(plain.deployment_id); + + await operations.revertComponent({ project, to_deployment_id: packaged.deployment_id }); + + const entry = readConfigFile()?.[project]; + assert.strictEqual( + entry?.package, + undefined, + 'the reverted-to version had no package reference, so the stale one must be gone' + ); + }); + + it('reclaims an oversized payload only after a full two-phase activation succeeds', async () => { + const project = name(); + const priorMaxSize = environment.get(CONFIG_PARAMS.DEPLOYMENT_PAYLOADRETENTION_MAXSIZE); + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_PAYLOADRETENTION_MAXSIZE, 1); + try { + const result = await operations.deployComponent({ + project, + payload: await makePayload('reclaimed-full-deploy', '6.0.0'), + }); + + const row = rows.get(result.deployment_id); + assert.strictEqual(row.status, 'success'); + assert.strictEqual(row.payload_blob, null); + assert.ok(row.event_log.some((event) => event.event === 'payload_dropped')); + } finally { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_PAYLOADRETENTION_MAXSIZE, priorMaxSize); + } + }); + + it('accepts the legacy deployment row marker only from a trusted replicated operation', async () => { + const project = name(); + const payload = await makePayload('trusted-one-shot', '6.0.0'); + const result = await runWithOperationAuthorizationBypass(true, () => + operations.deployComponent({ + project, + payload, + _deploymentId: '41faded8-6cf5-4a2a-95f8-863e7ea498fa', + replicated: false, + hdb_user: { name: 'cluster-peer' }, + }) + ); + + assert.match(result.message, /Successfully deployed/); + assert.match(await fs.readFile(path.join(COMPONENTS_ROOT, project, 'index.js'), 'utf8'), /trusted-one-shot/); + }); + + it('never leaves a component discoverable with its routing config already gone', async () => { + // Applications in the components root load by DIRECTORY SCAN; the root-config entry is what + // constrains where one is served (host/urlPath). So the state to make unreachable is tree-present + // with entry-gone — a crash there resurrects the component on every host, silently dropping the + // isolation the operator configured. The tree is parked first and the entry removed only after, + // which leaves the opposite (and merely re-runnable) partial state instead. + const project = name(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-drop-crash-')); + const componentsRoot = path.join(configRoot, 'components'); + const configPath = path.join(configRoot, 'harper-config.yaml'); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + const priorComponentsRoot = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + environment.setProperty(CONFIG_PARAMS.COMPONENTSROOT, componentsRoot); + await fs.writeFile(configPath, `rootPath: ${JSON.stringify(configRoot)}\n`); + try { + await fs.mkdir(path.join(componentsRoot, project), { recursive: true }); + await fs.writeFile(path.join(componentsRoot, project, 'index.js'), "module.exports = 'live';\n"); + await fs.appendFile(configPath, `${project}:\n package: some-package@1.0.0\n`); + // Make every aside-based teardown step fail: `.deploy-aside` occupied by a file is rejected as + // "not a directory", which is the closest deterministic stand-in for dying mid-teardown. + await fs.writeFile(path.join(componentsRoot, '.deploy-aside'), 'not a directory\n'); + + await assert.rejects(() => operations.dropComponent({ project })); + + const treePresent = existsSync(path.join(componentsRoot, project)); + const entryPresent = (await fs.readFile(configPath, 'utf8')).includes(project); + assert.strictEqual(treePresent, true, 'teardown failed, so the live tree is still on disk'); + assert.strictEqual( + entryPresent, + true, + 'and its routing entry survived with it — a discoverable tree must never outlive its mount' + ); + } finally { + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + environment.setProperty(CONFIG_PARAMS.COMPONENTSROOT, priorComponentsRoot); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); + + it('drop_component removes leftover recovery artifacts and the root-config entry', async () => { + const project = name(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-phase-drop-')); + const componentsRoot = path.join(configRoot, 'components'); + const configPath = path.join(configRoot, 'harper-config.yaml'); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + const priorComponentsRoot = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + environment.setProperty(CONFIG_PARAMS.COMPONENTSROOT, componentsRoot); + await fs.writeFile(configPath, `rootPath: ${JSON.stringify(configRoot)}\n`); + try { + const staged = await operations.deployComponent({ + project, + payload: await makePayload('drop-stage', '6.0.0'), + restart: false, + }); + const activationPath = path.join(componentsRoot, DEPLOY_ACTIVATION_DIR, project, 'interrupted'); + await fs.mkdir(activationPath, { recursive: true }); + await fs.writeFile( + path.join(configRoot, 'harper-application-lock.json'), + JSON.stringify({ applications: { [project]: { package: 'stale-package' } } }) + ); + // A root-config entry is what makes a dropped component come back: installApplications() reads + // it on the next boot and reinstalls the package. Removing it and the application-lock entry as + // two separate writes meant a crash or a failed second write left this behind. + await fs.appendFile(configPath, `${project}:\n package: stale-package\n`); + + await operations.dropComponent({ project }); + + const deploymentStagePath = path.join(componentsRoot, DEPLOY_STAGING_DIR, staged.deployment_id); + assert.strictEqual( + existsSync(deploymentStagePath), + false, + `staged deployment directory still contains: ${await fs.readdir(deploymentStagePath).catch(() => [])}` + ); + assert.strictEqual(existsSync(path.join(componentsRoot, DEPLOY_ACTIVATION_DIR, project)), false); + const applicationLock = JSON.parse(await fs.readFile(path.join(configRoot, 'harper-application-lock.json'))); + assert.strictEqual(applicationLock.applications[project], undefined); + assert.strictEqual( + (await fs.readFile(configPath, 'utf8')).includes(project), + false, + 'the root-config entry is removed in the same step, so the next boot cannot reinstall the drop' + ); + } finally { + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + environment.setProperty(CONFIG_PARAMS.COMPONENTSROOT, priorComponentsRoot); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); +}); diff --git a/unitTests/components/deployStaging.test.js b/unitTests/components/deployStaging.test.js new file mode 100644 index 0000000000..2e23e7e50a --- /dev/null +++ b/unitTests/components/deployStaging.test.js @@ -0,0 +1,2097 @@ +'use strict'; + +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const { existsSync } = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { randomUUID } = require('node:crypto'); +const zlib = require('node:zlib'); +const tarfs = require('tar-fs'); + +const testUtils = require('../testUtils.js'); +testUtils.preTestPrep(); + +const { + Application, + stageApplication, + activateStagedApplication, + discardStagedApplication, + discardProjectActivationArtifacts, + reconcileStagedApplicationArtifacts, + createApplicationActivationTransaction, + revertApplication, + getRevertTarget, + discardDirAside, + DISCARDED_ASIDE_PREFIX, + getStagingRetentionMaxCount, + DEFAULT_STAGING_RETENTION_MAX_COUNT, + recoverInterruptedReverts, + createApplicationConfigTransaction, + extractApplication, + stagedApplicationPath, + DEPLOY_STAGING_DIR, + DEPLOY_ACTIVATION_DIR, + DEPLOY_PREVIOUS_DIR, + ASIDE_STAGING_DIR, +} = require('#src/components/Application'); +const { getConfigPath, readConfigFile } = require('#src/config/configUtils'); +const { withComponentPreparationLock } = require('#src/components/componentPreparationLock'); +const { CONFIG_PARAMS } = require('#src/utility/hdbTerms'); +const environment = require('#src/utility/environment/environmentManager'); + +const COMPONENTS_ROOT = getConfigPath(CONFIG_PARAMS.COMPONENTSROOT); + +function packDirectory(dir) { + return new Promise((resolve, reject) => { + const chunks = []; + tarfs + .pack(dir) + .pipe(zlib.createGzip()) + .on('data', (chunk) => chunks.push(chunk)) + .on('end', () => resolve(Buffer.concat(chunks))) + .on('error', reject); + }); +} + +async function makeComponentPayload(marker, version = '1.0.0') { + const source = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-stage-src-')); + await fs.writeFile(path.join(source, 'package.json'), JSON.stringify({ name: 'stage-fixture', version })); + await fs.writeFile(path.join(source, 'index.js'), `module.exports = ${JSON.stringify(marker)};\n`); + await fs.mkdir(path.join(source, 'node_modules'), { recursive: true }); + const payload = await packDirectory(source); + await fs.rm(source, { recursive: true, force: true }); + return payload; +} + +async function readMarker(directory) { + return fs.readFile(path.join(directory, 'index.js'), 'utf8'); +} + +describe('two-phase component directory transaction', function () { + this.timeout(30_000); + let sequence = 0; + + before(async () => fs.mkdir(COMPONENTS_ROOT, { recursive: true })); + + function fixtureName() { + return `stage_test_${process.pid}_${sequence++}`; + } + + async function cleanup(name) { + await fs.rm(path.join(COMPONENTS_ROOT, name), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, ASIDE_STAGING_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR), { recursive: true, force: true }); + await fs.rm(path.join(COMPONENTS_ROOT, `${DEPLOY_PREVIOUS_DIR}`), { recursive: true, force: true }); + } + + it('builds a staged tree without touching the live component', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + + const stagedPath = await stageApplication(application, deploymentId); + + assert.strictEqual(stagedPath, stagedApplicationPath(application.dirPath, deploymentId)); + assert.match(await readMarker(stagedPath), /candidate/); + assert.strictEqual(existsSync(application.dirPath), false); + await cleanup(name); + }); + + it('atomically activates a staged tree and consumes only that deployment', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const siblingId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + application.payload = await makeComponentPayload('sibling'); + await stageApplication(application, siblingId); + + await activateStagedApplication(application, deploymentId); + + assert.match(await readMarker(application.dirPath), /candidate/); + assert.strictEqual(existsSync(stagedApplicationPath(application.dirPath, deploymentId)), false); + assert.strictEqual(existsSync(stagedApplicationPath(application.dirPath, siblingId)), true); + await cleanup(name); + }); + + it('rejects a candidate whose staging build did not complete', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('incomplete') }); + const stagedPath = await stageApplication(application, deploymentId); + await fs.rm(path.join(path.dirname(stagedPath), '.complete')); + + await assert.rejects(activateStagedApplication(application, deploymentId), /staged build is incomplete/); + + assert.strictEqual(existsSync(application.dirPath), false); + assert.match(await readMarker(stagedPath), /incomplete/); + await cleanup(name); + }); + + it('resumes a new-component activation after its durable marker was already created', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('resumed') }); + await stageApplication(application, deploymentId); + const activationPath = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationPath, { recursive: true }); + await fs.writeFile(path.join(activationPath, `.new-${deploymentId}`), '', { mode: 0o600 }); + + await activateStagedApplication(application, deploymentId); + + assert.match(await readMarker(application.dirPath), /resumed/); + assert.strictEqual(existsSync(activationPath), false); + await cleanup(name); + }); + + it('does not report activation failure when only committed-stage cleanup is denied', async function () { + if (process.platform === 'win32' || process.getuid?.() === 0) this.skip(); + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('cleanup-deferred') }); + await stageApplication(application, deploymentId); + const stagingRoot = path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR); + await fs.chmod(stagingRoot, 0o500); + try { + await activateStagedApplication(application, deploymentId); + } finally { + await fs.chmod(stagingRoot, 0o700); + } + + assert.match(await readMarker(application.dirPath), /cleanup-deferred/); + assert.strictEqual(existsSync(path.join(stagingRoot, deploymentId)), true, 'cleanup remains retryable garbage'); + await cleanup(name); + }); + + it('restores live and preserves the staged tree when persistent activation work fails', async () => { + const name = fixtureName(); + const livePath = path.join(COMPONENTS_ROOT, name); + const deploymentId = randomUUID(); + await fs.mkdir(livePath, { recursive: true }); + await fs.writeFile(path.join(livePath, 'index.js'), 'module.exports = "live";\n'); + const application = new Application({ name, payload: await makeComponentPayload('candidate', '2.0.0') }); + await stageApplication(application, deploymentId); + + await assert.rejects( + activateStagedApplication(application, deploymentId, { + beforeCommit: async () => { + throw new Error('config write failed'); + }, + }), + /config write failed/ + ); + + assert.match(await readMarker(livePath), /live/); + assert.match(await readMarker(stagedApplicationPath(livePath, deploymentId)), /candidate/); + await cleanup(name); + }); + + it('serializes duplicate activation without ever losing the live directory', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + + const outcomes = await Promise.allSettled([ + activateStagedApplication(application, deploymentId), + activateStagedApplication(application, deploymentId), + ]); + + assert.strictEqual(outcomes.filter((outcome) => outcome.status === 'fulfilled').length, 1); + assert.strictEqual(outcomes.filter((outcome) => outcome.status === 'rejected').length, 1); + assert.match(await readMarker(application.dirPath), /candidate/); + await cleanup(name); + }); + + it('snapshots config at commit time so a queued rollback preserves the preceding winner', async () => { + const name = fixtureName(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-activation-config-')); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + await fs.writeFile(path.join(configRoot, 'harper-config.yaml'), `rootPath: ${JSON.stringify(configRoot)}\n`); + const spec = (version, urlPath) => ({ + project: name, + package: `example@${version}`, + install_command: null, + install_timeout: null, + install_allow_scripts: null, + urlPath, + host: null, + credentials: null, + }); + const first = await createApplicationActivationTransaction(name, spec('1.0.0', '/first')); + const second = await createApplicationActivationTransaction(name, spec('2.0.0', '/second')); + try { + await first.commit(); + await second.commit(); + await second.rollback(); + + assert.deepStrictEqual(readConfigFile()[name], { package: 'example@1.0.0', urlPath: '/first' }); + const lock = JSON.parse( + await fs.readFile(path.join(configRoot, 'harper-application-lock.json'), { encoding: 'utf8' }) + ); + assert.deepStrictEqual(lock.applications[name], { package: 'example@1.0.0', urlPath: '/first' }); + } finally { + await second.rollback(); + await first.rollback(); + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); + + it('discard removes only the selected staged tree and leaves live unchanged', async () => { + const name = fixtureName(); + const livePath = path.join(COMPONENTS_ROOT, name); + const deploymentId = randomUUID(); + await fs.mkdir(livePath, { recursive: true }); + await fs.writeFile(path.join(livePath, 'index.js'), 'module.exports = "live";\n'); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + + await discardStagedApplication(livePath, deploymentId); + + assert.strictEqual(existsSync(stagedApplicationPath(livePath, deploymentId)), false); + assert.match(await readMarker(livePath), /live/); + await cleanup(name); + }); + + it('drop cleanup removes only the selected component activation artifacts', async () => { + const name = fixtureName(); + const sibling = fixtureName(); + const activationRoot = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR); + await fs.mkdir(path.join(activationRoot, name, 'old'), { recursive: true }); + await fs.mkdir(path.join(activationRoot, sibling, 'keep'), { recursive: true }); + + await discardProjectActivationArtifacts(path.join(COMPONENTS_ROOT, name)); + + assert.strictEqual(existsSync(path.join(activationRoot, name)), false); + assert.strictEqual(existsSync(path.join(activationRoot, sibling, 'keep')), true); + await cleanup(name); + }); + + it('startup reconciliation removes terminal stages and preserves valid staged rows', async () => { + const name = fixtureName(); + const keptId = randomUUID(); + const removedId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('kept') }); + await stageApplication(application, keptId); + application.payload = await makeComponentPayload('removed'); + await stageApplication(application, removedId); + const rows = new Map([ + [keptId, { deployment_id: keptId, project: name, status: 'staged' }], + [removedId, { deployment_id: removedId, project: name, status: 'failed' }], + ]); + + const result = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => rows.get(id), + async () => {} + ); + + assert.strictEqual(existsSync(stagedApplicationPath(application.dirPath, keptId)), true); + assert.strictEqual(existsSync(stagedApplicationPath(application.dirPath, removedId)), false); + assert.deepStrictEqual(result.removed, [removedId]); + await cleanup(name); + }); + + it('startup reconciliation rolls an activating staged tree forward', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const livePath = path.join(COMPONENTS_ROOT, name); + await fs.mkdir(livePath, { recursive: true }); + await fs.writeFile(path.join(livePath, 'index.js'), 'module.exports = "old";\n'); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + const activationPath = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationPath, { recursive: true }); + await fs.rename(livePath, path.join(activationPath, `.previous-${deploymentId}-crash`)); + const row = { + deployment_id: deploymentId, + project: name, + status: 'activating', + activation_spec: { project: name }, + }; + const persisted = []; + + const result = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? row : undefined), + async (value) => persisted.push(value.deployment_id) + ); + + assert.match(await readMarker(livePath), /candidate/); + assert.deepStrictEqual(persisted, [deploymentId]); + assert.deepStrictEqual(result.recovered, [deploymentId]); + assert.strictEqual(existsSync(activationPath), false); + await cleanup(name); + }); + + it('startup reconciliation finishes an activation whose candidate is already live', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const livePath = path.join(COMPONENTS_ROOT, name); + const activationPath = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(livePath, { recursive: true }); + await fs.writeFile(path.join(livePath, 'index.js'), 'module.exports = "candidate";\n'); + const backupDir = path.join(activationPath, `.previous-${deploymentId}-crash`); + await fs.mkdir(backupDir, { recursive: true }); + await fs.writeFile(path.join(backupDir, 'index.js'), 'module.exports = "displaced";\n'); + const row = { + deployment_id: deploymentId, + project: name, + status: 'activating', + activation_spec: { project: name }, + }; + let persisted = 0; + + const result = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? row : undefined), + async () => persisted++ + ); + + assert.strictEqual(persisted, 1); + assert.deepStrictEqual(result.recovered, [deploymentId]); + assert.match(await readMarker(livePath), /candidate/); + assert.strictEqual(existsSync(activationPath), false); + // This crash shape died after the swap but before retention, so the tree under `.previous-*` is the + // only copy of what the activation displaced. Clearing it as residue would leave the recovered + // deploy permanently unrevertable, which is the whole point of retaining it. + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + assert.match(await readMarker(previousPath), /displaced/, 'the displaced tree is retained, not deleted'); + const target = await getRevertTarget(livePath); + assert.strictEqual(target.live.deployment_id, deploymentId, 'and the manifest names the recovered deployment'); + await cleanup(name); + }); + + it('rejects a non-UUID before it can become a filesystem path segment', () => { + assert.throws(() => stagedApplicationPath(path.join(COMPONENTS_ROOT, fixtureName()), '../../escape'), /Invalid/); + }); + + it('rejects a symlinked staging root without touching its target', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-stage-outside-')); + await fs.writeFile(path.join(outside, 'sentinel'), 'keep'); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR), { recursive: true, force: true }); + await fs.symlink(outside, path.join(COMPONENTS_ROOT, DEPLOY_STAGING_DIR), 'dir'); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + + await assert.rejects(stageApplication(application, deploymentId), /staging path is not a directory/); + assert.strictEqual(await fs.readFile(path.join(outside, 'sentinel'), 'utf8'), 'keep'); + + await cleanup(name); + await fs.rm(outside, { recursive: true, force: true }); + }); + + it('activates a completed local-directory package symlink with secure staging containers', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const packageDirectory = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-stage-package-')); + await fs.writeFile(path.join(packageDirectory, 'marker.txt'), 'directory-package'); + const application = new Application({ name, packageIdentifier: packageDirectory }); + + const stagedPath = await stageApplication(application, deploymentId); + assert.strictEqual((await fs.lstat(stagedPath)).isSymbolicLink(), true); + await activateStagedApplication(application, deploymentId); + + assert.strictEqual((await fs.lstat(application.dirPath)).isSymbolicLink(), true); + assert.strictEqual(await fs.readFile(path.join(application.dirPath, 'marker.txt'), 'utf8'), 'directory-package'); + + await cleanup(name); + await fs.rm(packageDirectory, { recursive: true, force: true }); + }); + // Retained previous + addressed revert + + it('retains the tree an activation displaced, addressed by the deployment that produced it', async () => { + const name = fixtureName(); + const firstId = randomUUID(); + const secondId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + await stageApplication(application, firstId); + await activateStagedApplication(application, firstId, { activationSpec: { package: null } }); + application.payload = await makeComponentPayload('v2'); + await stageApplication(application, secondId); + await activateStagedApplication(application, secondId, { activationSpec: { package: null } }); + + assert.match(await readMarker(application.dirPath), /v2/); + const retained = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + assert.match(await readMarker(retained), /v1/); + + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, secondId); + assert.strictEqual(target.previous.deployment_id, firstId); + await cleanup(name); + }); + + it('a first-ever deploy retains nothing, so there is no version to revert to', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('only') }); + await stageApplication(application, deploymentId); + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + + assert.strictEqual(await getRevertTarget(application.dirPath), undefined); + await assert.rejects( + () => revertApplication(application, randomUUID()), + (error) => { + assert.match(error.message, /no previous version is retained/); + assert.strictEqual(error.statusCode, 409, "an unsatisfiable revert is the caller's problem, not a 500"); + return true; + }, + 'a component deployed once cannot be reverted' + ); + await cleanup(name); + }); + + it('reverts to the named previous deployment and exchanges the retained roles', async () => { + const name = fixtureName(); + const firstId = randomUUID(); + const secondId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + await stageApplication(application, firstId); + await activateStagedApplication(application, firstId, { activationSpec: { package: null } }); + application.payload = await makeComponentPayload('v2'); + await stageApplication(application, secondId); + await activateStagedApplication(application, secondId, { activationSpec: { package: null } }); + + const result = await revertApplication(application, firstId); + + assert.strictEqual(result.swapped, true); + assert.strictEqual(result.fromDeploymentId, secondId); + assert.match(await readMarker(application.dirPath), /v1/, 'live is the reverted-to version'); + assert.match( + await readMarker(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), + /v2/, + 'the displaced version becomes the new retained previous' + ); + + // Explicitly targeting the other direction rolls forward again. + const forward = await revertApplication(application, secondId); + assert.strictEqual(forward.swapped, true); + assert.match(await readMarker(application.dirPath), /v2/); + await cleanup(name); + }); + + it('is a no-op when the named deployment is already live, so a retry cannot toggle it back', async () => { + const name = fixtureName(); + const firstId = randomUUID(); + const secondId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + await stageApplication(application, firstId); + await activateStagedApplication(application, firstId, { activationSpec: { package: null } }); + application.payload = await makeComponentPayload('v2'); + await stageApplication(application, secondId); + await activateStagedApplication(application, secondId, { activationSpec: { package: null } }); + await revertApplication(application, firstId); + + // The delivery of the first response is lost and the caller retries the identical request. A + // bidirectional toggle would put the rejected v2 back live; an addressed revert must not. + const retry = await revertApplication(application, firstId); + + assert.strictEqual(retry.swapped, false, 'a repeated revert to the live version does nothing'); + assert.match(await readMarker(application.dirPath), /v1/, 'still on the reverted-to version'); + await cleanup(name); + }); + + it('refuses a deployment that is neither live nor the retained previous', async () => { + const name = fixtureName(); + const firstId = randomUUID(); + const secondId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + await stageApplication(application, firstId); + await activateStagedApplication(application, firstId, { activationSpec: { package: null } }); + application.payload = await makeComponentPayload('v2'); + await stageApplication(application, secondId); + await activateStagedApplication(application, secondId, { activationSpec: { package: null } }); + + await assert.rejects( + () => revertApplication(application, randomUUID()), + (error) => { + assert.match(error.message, /neither the live version .* nor the retained previous version/s); + assert.strictEqual(error.statusCode, 409, "asking for a version nobody kept is the caller's problem"); + return true; + }, + 'only one previous version is retained, so anything else is a redeploy' + ); + assert.match(await readMarker(application.dirPath), /v2/, 'a refused revert changes nothing'); + await cleanup(name); + }); + + it('retains exactly one previous version across three activations', async () => { + const name = fixtureName(); + const ids = [randomUUID(), randomUUID(), randomUUID()]; + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + for (const [index, id] of ids.entries()) { + if (index > 0) application.payload = await makeComponentPayload(`v${index + 1}`); + await stageApplication(application, id); + await activateStagedApplication(application, id, { activationSpec: { package: null } }); + } + + assert.match(await readMarker(application.dirPath), /v3/); + assert.match(await readMarker(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), /v2/); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.previous.deployment_id, ids[1], 'v1 is evicted; only v2 stays revertable'); + await cleanup(name); + }); + + it('parks the evicted retained-previous where startup recovery will never restore it', async () => { + // The single `.deploy-aside` contract: an `.in-progress-` directory with no `.retired-` marker is + // a rollback record that recoverInterruptedComponentExtractions restores OVER the live component. + // An evicted two-deploys-ago tree is known garbage when parked, so it must not carry that prefix, + // or a crash before the sweep would resurrect an ancient version over the current one. + const name = fixtureName(); + const ids = [randomUUID(), randomUUID(), randomUUID()]; + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + for (const [index, id] of ids.entries()) { + if (index > 0) application.payload = await makeComponentPayload(`v${index + 1}`); + await stageApplication(application, id); + await activateStagedApplication(application, id, { activationSpec: { package: null } }); + } + + assert.match(await readMarker(application.dirPath), /v3/, 'the newest activation is live'); + assert.match(await readMarker(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), /v2/); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.previous.deployment_id, ids[1], 'only the immediately-previous version is retained'); + await cleanup(name); + }); + + it('reports the config each retained version was activated with, so a revert can restore it', async () => { + // This is what makes a revert durable across a cold restart: reverting away from a `package` + // deploy has to take the package reference out of root config too, or installApplications() + // reinstalls the reverted-away version over the restored directory on the next boot. + const name = fixtureName(); + const packagedId = randomUUID(); + const payloadId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('packaged') }); + await stageApplication(application, packagedId); + await activateStagedApplication(application, packagedId, { + activationSpec: { + package: 'stage-fixture@1.0.0', + install_command: null, + install_timeout: null, + install_allow_scripts: null, + urlPath: null, + host: null, + }, + }); + application.payload = await makeComponentPayload('plain'); + await stageApplication(application, payloadId); + await activateStagedApplication(application, payloadId, { activationSpec: { package: null } }); + + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.previous.application_config.package, 'stage-fixture@1.0.0'); + const back = await revertApplication(application, packagedId); + assert.strictEqual(back.activatedConfig.package, 'stage-fixture@1.0.0'); + await cleanup(name); + }); + + it('moves a dangling symlink at the live path aside instead of failing EEXIST', async () => { + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + const missingTarget = path.join(os.tmpdir(), `harper-missing-${randomUUID()}`); + await fs.mkdir(COMPONENTS_ROOT, { recursive: true }); + await fs.symlink(missingTarget, application.dirPath, 'dir'); + + await stageApplication(application, deploymentId); + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + + assert.strictEqual((await fs.lstat(application.dirPath)).isSymbolicLink(), false); + assert.match(await readMarker(application.dirPath), /candidate/); + await cleanup(name); + }); + it('extracts in place over a DANGLING symlink at the component path instead of failing EEXIST', async () => { + // access(F_OK) follows symlinks, so a dead link at the target reports ENOENT and would be treated + // as "nothing here" — then mkdir fails EEXIST because the link still occupies the path. lstat sees + // the link itself. Left by a prior `file:`-directory deploy whose target was removed. + const name = fixtureName(); + const application = new Application({ name, payload: await makeComponentPayload('over-dead-link') }); + await fs.mkdir(COMPONENTS_ROOT, { recursive: true }); + await fs.symlink(path.join(os.tmpdir(), `harper-missing-${randomUUID()}`), application.dirPath, 'dir'); + + await extractApplication(application); + + assert.strictEqual((await fs.lstat(application.dirPath)).isSymbolicLink(), false); + assert.match(await readMarker(application.dirPath), /over-dead-link/); + await cleanup(name); + }); + // Restart gate: package metadata compared across the swap (harper#674) + // + // main's in-place prepareApplication compares installed package metadata before extraction against + // after install; the two-phase path never touches the live directory until the swap, so the + // equivalent comparison is outgoing-live vs staged, taken inside activateStagedApplication. These + // assert the flag it sets, which operations.js feeds into markRestartRequiredForDeploy. + + async function activateFrom(name, marker, version, { withNodeModules = true, dependencies } = {}) { + const source = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-gate-src-')); + const manifest = { name: 'stage-fixture', version }; + if (dependencies) manifest.dependencies = dependencies; + await fs.writeFile(path.join(source, 'package.json'), JSON.stringify(manifest)); + await fs.writeFile(path.join(source, 'index.js'), `module.exports = ${JSON.stringify(marker)};\n`); + if (withNodeModules) await fs.mkdir(path.join(source, 'node_modules'), { recursive: true }); + const payload = await packDirectory(source); + await fs.rm(source, { recursive: true, force: true }); + + const application = new Application({ name, payload }); + const deploymentId = randomUUID(); + await stageApplication(application, deploymentId); + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + return application; + } + + it('does not flag a metadata change when a redeploy is identical and comparable', async () => { + // The harper#1806 guarantee: a redeploy of an already-loaded component stays quiet, because that + // component's own file watcher requests a restart if the changed files actually need one. It holds + // only for a COMPARABLE install — no bundled node_modules to make the result opaque. + const name = fixtureName(); + await activateFrom(name, 'quiet', '1.0.0', { withNodeModules: false }); + const second = await activateFrom(name, 'quiet', '1.0.0', { withNodeModules: false }); + + assert.strictEqual(second.isNewComponent, false, 'the second activation is a redeploy'); + assert.strictEqual(second.packageMetadataChanged, false, 'identical metadata across the swap must stay quiet'); + await cleanup(name); + }); + + it('flags a metadata change the watchers cannot see', async () => { + const name = fixtureName(); + await activateFrom(name, 'changed', '1.0.0', { withNodeModules: false }); + const second = await activateFrom(name, 'changed', '2.0.0', { withNodeModules: false }); + + assert.strictEqual(second.packageMetadataChanged, true, 'a changed package.json version invalidates loaded code'); + await cleanup(name); + }); + + it('treats a bundled node_modules redeploy as opaque, since its install cannot be compared', async () => { + const name = fixtureName(); + await activateFrom(name, 'opaque', '1.0.0'); + const second = await activateFrom(name, 'opaque', '1.0.0'); + + assert.strictEqual(second.packageMetadataChanged, true, 'a skipped install leaves nothing to compare'); + await cleanup(name); + }); + + it('treats installable dependencies with no lockfile as opaque', async () => { + const name = fixtureName(); + await activateFrom(name, 'nolock', '1.0.0'); + const second = await activateFrom(name, 'nolock', '1.0.0', { dependencies: { 'some-dep': '1.0.0' } }); + + assert.strictEqual(second.packageMetadataChanged, true, 'dependencies with no lockfile are not reproducible'); + await cleanup(name); + }); + + it('does not flag a metadata change on a first-ever deploy', async () => { + const name = fixtureName(); + const first = await activateFrom(name, 'brand-new', '1.0.0', { withNodeModules: false }); + + assert.strictEqual(first.isNewComponent, true); + assert.strictEqual(first.packageMetadataChanged, false, 'nothing to compare against; isNewComponent carries it'); + await cleanup(name); + }); + // Interrupted revert: compensation and startup recovery + + async function twoActivations(name) { + const first = randomUUID(); + const second = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('v1') }); + await stageApplication(application, first); + await activateStagedApplication(application, first, { activationSpec: { package: null } }); + application.payload = await makeComponentPayload('v2'); + await stageApplication(application, second); + await activateStagedApplication(application, second, { activationSpec: { package: null } }); + return { application, first, second }; + } + + // NOT covered here: an in-process failure of the middle rename. Once `rename(live, holding)` has run, + // the live path is gone, so `rename(previous, live)` has no existing target to conflict with and + // succeeds for every filesystem state reachable from a test (a file, a dangling symlink, a directory + // all rename cleanly onto a free path). Inducing it would need a fault-injection seam in production + // code. The compensation is still there — it is what turns an exceptional I/O error into "the + // component keeps serving what it was serving" — but the durable half below is what actually covers + // the case that matters: a process that dies mid-swap, which compensation inherently cannot handle. + + it('startup recovery restores the live tree from a revert that died before the swap', async () => { + const name = fixtureName(); + const { application } = await twoActivations(name); + // Simulate a crash right after `rename(live, holding)`: live is gone, holding holds its tree. + const holding = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, `.reverting-${name}-${randomUUID()}`); + await fs.rename(application.dirPath, holding); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.match(await readMarker(application.dirPath), /v2/, 'the interrupted revert is undone'); + assert.strictEqual(existsSync(holding), false); + await cleanup(name); + }); + + it('startup recovery re-retains the displaced tree when only the retain step was lost', async () => { + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + // Simulate a crash after `rename(previous, live)` but before `rename(holding, previous)`: the + // reverted-to version is live, and the displaced tree is still parked. + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + const holding = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, `.reverting-${name}-${randomUUID()}`); + await fs.rm(application.dirPath, { recursive: true, force: true }); + await fs.rename(previousPath, application.dirPath); + await fs.mkdir(holding, { recursive: true }); + await fs.writeFile(path.join(holding, 'index.js'), "module.exports = 'displaced';\n"); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.match(await readMarker(application.dirPath), /v1/, 'the reverted-to version stays live'); + assert.match(await readMarker(previousPath), /displaced/, 'the displaced tree is retained again'); + assert.strictEqual(existsSync(holding), false); + + // The directories are only half the state. The manifest was written before the swap, so completing + // the recovery has to exchange its roles too — otherwise it names the retained tree as live and the + // live tree as retained, and the retry below matches its target against the reversed `previous` + // entry and swaps the successful revert straight back out. + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, first, 'the manifest names the reverted-to version as live'); + assert.strictEqual(target.previous.deployment_id, second, 'and the displaced version as the retained one'); + + // Idempotency has to survive the recovery: re-issuing the same addressed revert changes nothing. + const retry = await revertApplication(application, first); + assert.strictEqual(retry.swapped, false, 'a retry after recovery is a no-op, not a swap back'); + assert.match(await readMarker(application.dirPath), /v1/, 'still on the reverted-to version'); + await cleanup(name); + }); + + it('startup recovery discards a holding tree left by a revert that had already completed', async () => { + const name = fixtureName(); + const { application } = await twoActivations(name); + const holding = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, `.reverting-${name}-${randomUUID()}`); + await fs.mkdir(holding, { recursive: true }); + await fs.writeFile(path.join(holding, 'index.js'), "module.exports = 'residue';\n"); + + await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(existsSync(holding), false, 'both slots were occupied, so the holding tree is residue'); + assert.match(await readMarker(application.dirPath), /v2/, 'live is untouched'); + await cleanup(name); + }); + it('redeploys a local-directory package in place without destroying the live tree', async () => { + // This path returns early, BEFORE the extraction transaction, so clearing the build target with an + // in-place recursive rm would delete the LIVE tree outright: no aside, no rollback record, nothing + // for startup recovery, and it races a worker still writing into the directory being removed. It + // must be parked with an atomic rename instead. + const name = fixtureName(); + const packageDirectory = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-dirpkg-')); + await fs.writeFile(path.join(packageDirectory, 'marker.txt'), 'v2'); + const application = new Application({ name, packageIdentifier: `file:${packageDirectory}` }); + + // A live tree is already there (the component was deployed before, by any means). + await fs.mkdir(application.dirPath, { recursive: true }); + await fs.writeFile(path.join(application.dirPath, 'marker.txt'), 'v1'); + + await extractApplication(application); + + assert.strictEqual((await fs.lstat(application.dirPath)).isSymbolicLink(), true, 'the new version is linked'); + assert.strictEqual(await fs.readFile(path.join(application.dirPath, 'marker.txt'), 'utf8'), 'v2'); + await cleanup(name); + await fs.rm(packageDirectory, { recursive: true, force: true }); + }); + it('rolls both persistent writes back, so a failed commit cannot leave config half-applied', async () => { + // commit() makes TWO persistent writes — root config, then the boot-time application lock — and + // arms rollback (`commitStarted`) BEFORE either. That is what lets a caller whose commit threw + // partway undo the write that did land: without it, compensating by swapping the directories back + // still left root config naming the version just rolled away from, for a cold start to act on. + // + // A genuinely partial commit is not injectable from a test: the lock is READ (getApplicationLockEntry) + // before the config write, so any corruption that would break the lock write breaks that read first + // and fails before mutating anything. What is verifiable is the property the compensation depends + // on — that rollback restores the exact pre-commit state of both writes. + const name = fixtureName(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-cfg-rollback-')); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + await fs.writeFile(path.join(configRoot, 'harper-config.yaml'), `rootPath: ${JSON.stringify(configRoot)}\n`); + const lockPath = path.join(configRoot, 'harper-application-lock.json'); + const readLock = async () => JSON.parse(await fs.readFile(lockPath, { encoding: 'utf8' })).applications[name]; + try { + // The version that is live now, and that a revert would be rolling back to. + const original = await createApplicationConfigTransaction(name, { package: 'example@1.0.0' }); + await original.commit(); + assert.deepStrictEqual(readConfigFile()[name], { package: 'example@1.0.0' }); + assert.deepStrictEqual(await readLock(), { package: 'example@1.0.0' }); + + const reverting = await createApplicationConfigTransaction(name, { package: 'example@2.0.0' }); + await reverting.commit(); + assert.deepStrictEqual(readConfigFile()[name], { package: 'example@2.0.0' }, 'both writes moved forward'); + assert.deepStrictEqual(await readLock(), { package: 'example@2.0.0' }); + + await reverting.rollback(); + + assert.deepStrictEqual( + readConfigFile()[name], + { package: 'example@1.0.0' }, + 'rollback restores the root config the commit replaced' + ); + assert.deepStrictEqual(await readLock(), { package: 'example@1.0.0' }, 'and the application-lock entry'); + + // A transaction that never committed must not touch anything on rollback, so compensation on an + // early failure cannot clobber a live config. + const untouched = await createApplicationConfigTransaction(name, { package: 'example@3.0.0' }); + await untouched.rollback(); + assert.deepStrictEqual(readConfigFile()[name], { package: 'example@1.0.0' }, 'no-op rollback changes nothing'); + } finally { + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); + + it('reports the component whose activation persistence failed, so it can be kept from loading', async () => { + // Startup reconciliation used to only LOG a failure to finish an `activating` deployment, then carry + // on into normal component loading — serving a swapped-in tree whose durable configuration still + // described the previous release. The caller can only fail that component closed if it is told which + // component failed, which a deployment id alone does not give it. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + await fs.mkdir(application.dirPath, { recursive: true }); + + const row = { + deployment_id: deploymentId, + project: name, + status: 'activating', + activation_spec: { package: null }, + }; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => row, + async () => { + throw new Error('simulated activation-persistence failure'); + } + ); + + assert.strictEqual(reconciliation.errors.has(deploymentId), true, 'the failure is reported by deployment'); + assert.strictEqual( + reconciliation.failedProjects.get(name)?.message, + 'simulated activation-persistence failure', + 'and attributed to the component, so the loader can fail it closed' + ); + assert.strictEqual( + reconciliation.recovered.includes(deploymentId), + false, + 'a failed reconciliation is not "recovered"' + ); + await cleanup(name); + }); + it('re-points a dependency link whose absolute target the activation swap invalidated', async () => { + // npm installs against the STAGING directory, and activation renames that directory to the live + // path — so any dependency npm materialized with an ABSOLUTE target inside staging dangles the + // instant the rename lands. That is the normal shape of a `file:` dependency on Windows, where npm + // creates a junction and junctions are always absolute; on Linux it writes a relative symlink that + // survives the move. The Windows symptom is a bare `Cannot find module ''` at component load, + // well after a deploy that reported success. + // + // Absolute links exist on every platform, so the behavior is reproducible here regardless of OS. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('with-dep') }); + const stagedPath = await stageApplication(application, deploymentId); + + // Stand in for what `npm install --force` leaves behind for `file:vendor/probe`: a real vendored + // directory, plus an ABSOLUTE link to it from node_modules. + await fs.mkdir(path.join(stagedPath, 'vendor', 'probe'), { recursive: true }); + await fs.writeFile(path.join(stagedPath, 'vendor', 'probe', 'index.js'), "module.exports = 'probe';\n"); + await fs.mkdir(path.join(stagedPath, 'node_modules'), { recursive: true }); + await fs.symlink(path.join(stagedPath, 'vendor', 'probe'), path.join(stagedPath, 'node_modules', 'probe'), 'dir'); + // A scoped package, and a link deliberately pointing OUTSIDE staging, which must be left alone. + const external = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-external-dep-')); + await fs.writeFile(path.join(external, 'index.js'), "module.exports = 'external';\n"); + await fs.mkdir(path.join(stagedPath, 'node_modules', '@scope'), { recursive: true }); + await fs.symlink( + path.join(stagedPath, 'vendor', 'probe'), + path.join(stagedPath, 'node_modules', '@scope', 'scoped'), + 'dir' + ); + await fs.symlink(external, path.join(stagedPath, 'node_modules', 'external'), 'dir'); + + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + + // The dependency has to be resolvable from the LIVE tree, which is the whole point. + assert.strictEqual( + await fs.readFile(path.join(application.dirPath, 'node_modules', 'probe', 'index.js'), 'utf8'), + "module.exports = 'probe';\n", + 'the staged-path link was re-pointed at the live path' + ); + assert.strictEqual( + await fs.readFile(path.join(application.dirPath, 'node_modules', '@scope', 'scoped', 'index.js'), 'utf8'), + "module.exports = 'probe';\n", + 'scoped packages are re-pointed too' + ); + // Untouched: an absolute link to somewhere outside the staging tree is a deliberate choice. + assert.strictEqual( + await fs.readlink(path.join(application.dirPath, 'node_modules', 'external')), + external, + 'a link outside staging is left exactly as it was' + ); + + await cleanup(name); + await fs.rm(external, { recursive: true, force: true }); + }); + it('leaves external contents untouched when node_modules/@scope is a symlink', async () => { + // `readdir` follows symlinks, so a staged payload shipping `node_modules/@scope` as a link to + // somewhere else on the machine would otherwise have that directory's children enumerated as + // candidates — and a link in there whose target happened to point into staging would be removed and + // recreated, writing outside the component tree. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('scoped') }); + const stagedPath = await stageApplication(application, deploymentId); + + // An external directory holding a link that DOES point into staging — the dangerous shape. + const external = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-external-scope-')); + await fs.mkdir(path.join(stagedPath, 'vendor', 'probe'), { recursive: true }); + await fs.writeFile(path.join(stagedPath, 'vendor', 'probe', 'index.js'), "module.exports = 'probe';\n"); + await fs.symlink(path.join(stagedPath, 'vendor', 'probe'), path.join(external, 'bait'), 'dir'); + const externalBaitTarget = await fs.readlink(path.join(external, 'bait')); + + // node_modules/@scope is a LINK to that external directory. + await fs.mkdir(path.join(stagedPath, 'node_modules'), { recursive: true }); + await fs.symlink(external, path.join(stagedPath, 'node_modules', '@scope'), 'dir'); + + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + + assert.strictEqual( + await fs.readlink(path.join(external, 'bait')), + externalBaitTarget, + 'a link inside a symlinked scope directory is never rewritten' + ); + assert.strictEqual( + (await fs.lstat(path.join(application.dirPath, 'node_modules', '@scope'))).isSymbolicLink(), + true, + 'the scope link itself is left as a link' + ); + await cleanup(name); + await fs.rm(external, { recursive: true, force: true }); + }); + + it('keeps the previous release live when a dependency link cannot be repaired', async () => { + // The repair is not best-effort: the link is only being touched because activation is about to + // invalidate its target, so a failure means the component would go live unable to resolve a + // dependency — which pre-swap load validation cannot catch, since the link was valid in staging. + const name = fixtureName(); + const first = randomUUID(); + const second = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('live-v1') }); + await stageApplication(application, first); + await activateStagedApplication(application, first, { activationSpec: { package: null } }); + + application.payload = await makeComponentPayload('candidate-v2'); + const stagedPath = await stageApplication(application, second); + await fs.mkdir(path.join(stagedPath, 'vendor', 'probe'), { recursive: true }); + await fs.mkdir(path.join(stagedPath, 'node_modules'), { recursive: true }); + await fs.symlink(path.join(stagedPath, 'vendor', 'probe'), path.join(stagedPath, 'node_modules', 'probe'), 'dir'); + // Make recreating the link impossible: replace its parent with a read-only directory so the unlink + // and re-symlink cannot succeed. + await fs.chmod(path.join(stagedPath, 'node_modules'), 0o500); + + try { + await assert.rejects( + () => activateStagedApplication(application, second, { activationSpec: { package: null } }), + 'an unrepairable dependency link must fail the activation' + ); + assert.match( + await readMarker(application.dirPath), + /live-v1/, + 'the previous release is still live — the swap was compensated' + ); + } finally { + await fs.chmod(path.join(stagedPath, 'node_modules'), 0o700).catch(() => {}); + } + await cleanup(name); + }); + + it('resumes revert recovery when a first pass fails before the tree is moved back', async () => { + // The holding directory is the only durable evidence that recovery is unfinished, so it must not be + // consumed until the manifest and config writes are durable. A recovery marker carries the intended + // end state across restarts, and on re-entry it — not the manifest — is the source of truth, because + // an earlier attempt may already have exchanged the manifest and exchanging it twice flips it back. + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + const manifestPath = `${previousPath}.json`; + const holding = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, `.reverting-${name}-${randomUUID()}`); + // The marker is named after the holding directory it describes, so an orphan from one attempt can + // never be trusted by a later, unrelated revert of the same component. + const markerPath = `${holding}.recovering.json`; + + // Crash state: the reverted-to version is live, the displaced tree is still parked. + const staleManifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')); + await fs.rm(application.dirPath, { recursive: true, force: true }); + await fs.rename(previousPath, application.dirPath); + await fs.mkdir(holding, { recursive: true }); + await fs.writeFile(path.join(holding, 'index.js'), "module.exports = 'displaced';\n"); + // Stand in for a first pass that recorded its intent and then died: the marker is present. + await fs.writeFile( + markerPath, + JSON.stringify({ previous: staleManifest.live, live: staleManifest.previous }, null, 2) + ); + // Make the manifest WRITE fail (its read still succeeds, and the marker is what recovery reads + // anyway) by putting a directory where the manifest file belongs. + await fs.rm(manifestPath, { recursive: true, force: true }); + await fs.mkdir(manifestPath, { recursive: true }); + + const firstPass = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(firstPass.has(name), true, 'the failed pass is reported, so the component is failed closed'); + assert.strictEqual(existsSync(holding), true, 'the holding tree survives, so a later pass can still finish'); + assert.strictEqual(existsSync(markerPath), true, 'and the recovery marker survives with it'); + + // Clear the injected fault and run recovery again, as the next process start would. + await fs.rm(manifestPath, { recursive: true, force: true }); + const secondPass = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(secondPass.size, 0, 'the second pass completes'); + assert.match(await readMarker(application.dirPath), /v1/, 'the reverted-to version is live'); + assert.match(await readMarker(previousPath), /displaced/, 'the displaced tree is retained again'); + assert.strictEqual(existsSync(holding), false, 'the holding tree is consumed only once everything is durable'); + assert.strictEqual(existsSync(markerPath), false, 'and the marker is cleared'); + // The marker, not the twice-read manifest, decided the end state — so roles are exchanged once. + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, first); + assert.strictEqual(target.previous.deployment_id, second); + await cleanup(name); + }); + it('commits config inside the swap, so a revert cannot land config after a later activation', async () => { + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + const order = []; + + const result = await revertApplication(application, first, { + commitPersistentState: async () => { + // Observed from inside the lock: the reverted-to tree is already live and the displaced tree is + // still parked, which is what makes this the only safe point to persist config. + order.push('commit'); + assert.match(await readMarker(application.dirPath), /v1/, 'reverted-to version is live at commit time'); + assert.strictEqual( + existsSync(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), + false, + 'the displaced tree is still in the holding path, not yet retained' + ); + }, + }); + + assert.deepStrictEqual(order, ['commit'], 'the hook ran exactly once'); + assert.strictEqual(result.swapped, true); + assert.strictEqual(result.fromDeploymentId, second); + assert.match(await readMarker(application.dirPath), /v1/); + assert.match(await readMarker(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), /v2/); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, first); + await cleanup(name); + }); + + it('undoes the swap and leaves nothing revertable-looking when persisting config fails', async () => { + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + + await assert.rejects( + () => + revertApplication(application, first, { + commitPersistentState: async () => { + throw new Error('simulated config failure'); + }, + }), + /simulated config failure/ + ); + + assert.match(await readMarker(application.dirPath), /v2/, 'the pre-revert version is live again'); + assert.match(await readMarker(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), /v1/); + const stranded = (await fs.readdir(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR))).filter((entry) => + entry.startsWith('.reverting-') + ); + assert.deepStrictEqual(stranded, [], 'no holding tree or marker is left behind'); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, second, 'the manifest still describes the un-reverted state'); + await cleanup(name); + }); + + it('sweeps a revert marker whose holding directory is gone', async () => { + // A crash between the final rename and the marker removal orphans the marker. Nothing revisits it — + // recovery is keyed on finding a holding directory — so it must be swept, or a later unrelated + // revert of the same component could be resumed against it. + const name = fixtureName(); + const { application, first } = await twoActivations(name); + const orphan = path.join( + COMPONENTS_ROOT, + DEPLOY_PREVIOUS_DIR, + `.reverting-${name}-${randomUUID()}.recovering.json` + ); + await fs.writeFile(orphan, JSON.stringify({ previous: { deployment_id: 'x' }, live: { deployment_id: 'y' } })); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.strictEqual(existsSync(orphan), false, 'the orphaned marker is swept'); + // The sweep must not have disturbed the component's actual revert state. + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.previous.deployment_id, first, 'the real retained-previous entry is untouched'); + await cleanup(name); + }); + + it('re-points a nested node_modules dependency link, not just top-level ones', async () => { + // npm nests a link under `node_modules//node_modules/` whenever hoisting is blocked by a + // version conflict, and under workspaces routinely. A nested link dangles after the swap exactly + // like a top-level one, and the hard-fail above cannot help if it is never enumerated. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('nested') }); + const stagedPath = await stageApplication(application, deploymentId); + + await fs.mkdir(path.join(stagedPath, 'vendor', 'deep'), { recursive: true }); + await fs.writeFile(path.join(stagedPath, 'vendor', 'deep', 'index.js'), "module.exports = 'deep';\n"); + const outerPackage = path.join(stagedPath, 'node_modules', 'outer', 'node_modules'); + await fs.mkdir(outerPackage, { recursive: true }); + await fs.symlink(path.join(stagedPath, 'vendor', 'deep'), path.join(outerPackage, 'deep'), 'dir'); + // A scoped package nested one level further down, to prove the recursion handles both shapes. + await fs.mkdir(path.join(outerPackage, '@inner'), { recursive: true }); + await fs.symlink(path.join(stagedPath, 'vendor', 'deep'), path.join(outerPackage, '@inner', 'scoped'), 'dir'); + + await activateStagedApplication(application, deploymentId, { activationSpec: { package: null } }); + + assert.strictEqual( + await fs.readFile( + path.join(application.dirPath, 'node_modules', 'outer', 'node_modules', 'deep', 'index.js'), + 'utf8' + ), + "module.exports = 'deep';\n", + 'the nested link resolves from the live tree' + ); + assert.strictEqual( + await fs.readFile( + path.join(application.dirPath, 'node_modules', 'outer', 'node_modules', '@inner', 'scoped', 'index.js'), + 'utf8' + ), + "module.exports = 'deep';\n", + 'a nested scoped link resolves too' + ); + await cleanup(name); + }); + + it('parks a disposable tree under the discarded prefix, never as a recovery candidate', async () => { + // Asserted directly rather than after an activation: discardDirAside sweeps fire-and-forget, so an + // after-the-fact directory listing passes vacuously whenever the sweep wins the race and cannot + // distinguish correct `.discarded-` parking from a regression that parked as `.in-progress-`. + const name = fixtureName(); + const target = path.join(COMPONENTS_ROOT, `${name}-disposable`); + await fs.mkdir(target, { recursive: true }); + await fs.writeFile(path.join(target, 'index.js'), "module.exports = 'disposable';\n"); + const asideDir = path.join(COMPONENTS_ROOT, ASIDE_STAGING_DIR, `${name}-disposable`); + + const parkedSomething = await discardDirAside(target, name); + + assert.strictEqual(parkedSomething, true); + assert.strictEqual(existsSync(target), false, 'the tree is moved out of the way'); + // Read before the detached sweep can remove it; if it already has, there is nothing to misclassify. + const entries = await fs.readdir(asideDir).catch(() => []); + for (const entry of entries) { + assert.strictEqual( + entry.startsWith(DISCARDED_ASIDE_PREFIX), + true, + `parked entry ${entry} must carry the discarded prefix, not a recovery-candidate prefix` + ); + } + assert.strictEqual(await discardDirAside(target, name), false, 'nothing to park the second time'); + await fs.rm(path.join(COMPONENTS_ROOT, ASIDE_STAGING_DIR), { recursive: true, force: true }); + }); + + it('does not evict a staged build newer than the one being staged', async () => { + // The filesystem half of the retention inversion: a stage delayed behind slow peers finishes with + // an older mtime than a sibling that started later and already returned. Filling the retention + // budget from "the others" alone always reserved the current build and evicted that newer + // sibling's tree, so the deployment the operator was just told about lost its staged bytes. + const name = fixtureName(); + const priorMax = environment.get(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT); + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, 1); + try { + const application = new Application({ name, payload: await makeComponentPayload('slow') }); + const slowId = randomUUID(); + await stageApplication(application, slowId); + application.payload = await makeComponentPayload('newer'); + const newerId = randomUUID(); + await stageApplication(application, newerId); + + // Make the newer build unambiguously newer on disk, then re-run the slow stage's prune by + // staging it again — which is what a delayed origin resuming its own stage does. + const newerPath = stagedApplicationPath(application.dirPath, newerId); + const future = new Date(Date.now() + 60_000); + await fs.utimes(newerPath, future, future); + application.payload = await makeComponentPayload('slow'); + await stageApplication(application, slowId); + + assert.strictEqual( + existsSync(newerPath), + true, + 'the newer sibling survives a prune driven by an older, still-finishing stage' + ); + assert.strictEqual( + existsSync(stagedApplicationPath(application.dirPath, slowId)), + true, + 'and the build being staged is kept too' + ); + } finally { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, priorMax); + } + await cleanup(name); + }); + + it('evicts staged builds beyond the retention count and always keeps the newest', async () => { + const name = fixtureName(); + const priorMax = environment.get(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT); + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, 2); + const ids = []; + try { + const application = new Application({ name, payload: await makeComponentPayload('retained') }); + for (let index = 0; index < 4; index++) { + const deploymentId = randomUUID(); + application.payload = await makeComponentPayload(`retained-${index}`); + await stageApplication(application, deploymentId); + ids.push(deploymentId); + } + const surviving = ids.filter((id) => existsSync(stagedApplicationPath(application.dirPath, id))); + assert.strictEqual(surviving.length, 2, `expected exactly 2 staged builds to survive, got ${surviving.length}`); + assert.strictEqual(surviving.includes(ids.at(-1)), true, 'the just-staged build is never evicted'); + } finally { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, priorMax); + } + await cleanup(name); + }); + + it('falls back to the default staging retention count for unusable configured values', () => { + const prior = environment.get(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT); + try { + for (const value of [undefined, '', ' ', true, [], {}, 'abc', 0, -1]) { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, value); + assert.strictEqual( + getStagingRetentionMaxCount(), + DEFAULT_STAGING_RETENTION_MAX_COUNT, + `${JSON.stringify(value)} must fall back to the default rather than coerce` + ); + } + for (const [value, expected] of [ + ['3', 3], + [3, 3], + [2.7, 2], + ]) { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, value); + assert.strictEqual(getStagingRetentionMaxCount(), expected); + } + } finally { + environment.setProperty(CONFIG_PARAMS.DEPLOYMENT_STAGINGRETENTION_MAXCOUNT, prior); + } + }); + it('leaves a rolled-back candidate re-activatable, with its links aimed at staging again', async () => { + // Re-pointing before the swap mutates the staged candidate, so a rollback has to undo it. Left + // aimed at the live path the links would resolve against whatever release is live, and a retry of + // the same deployment id would validate the staged tree against the wrong bytes. + const name = fixtureName(); + const first = randomUUID(); + const second = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('live-v1') }); + await stageApplication(application, first); + await activateStagedApplication(application, first, { activationSpec: { package: null } }); + + application.payload = await makeComponentPayload('candidate-v2'); + const stagedPath = await stageApplication(application, second); + await fs.mkdir(path.join(stagedPath, 'vendor', 'probe'), { recursive: true }); + await fs.writeFile(path.join(stagedPath, 'vendor', 'probe', 'index.js'), "module.exports = 'probe';\n"); + await fs.mkdir(path.join(stagedPath, 'node_modules'), { recursive: true }); + await fs.symlink(path.join(stagedPath, 'vendor', 'probe'), path.join(stagedPath, 'node_modules', 'probe'), 'dir'); + + // Fail after the links were rewritten and the swap landed, so the rollback path runs in full. + await assert.rejects(() => + activateStagedApplication(application, second, { + activationSpec: { package: null }, + beforeCommit: async () => { + throw new Error('simulated persistent-work failure'); + }, + }) + ); + + assert.match(await readMarker(application.dirPath), /live-v1/, 'the previous release is live again'); + // The candidate is back in staging and its dependency resolves from there, not from the live path. + assert.strictEqual( + await fs.readFile(path.join(stagedPath, 'node_modules', 'probe', 'index.js'), 'utf8'), + "module.exports = 'probe';\n", + 'the rolled-back candidate resolves its dependency from staging' + ); + + // And it can still be activated on a retry. + await activateStagedApplication(application, second, { activationSpec: { package: null } }); + assert.match(await readMarker(application.dirPath), /candidate-v2/); + assert.strictEqual( + await fs.readFile(path.join(application.dirPath, 'node_modules', 'probe', 'index.js'), 'utf8'), + "module.exports = 'probe';\n", + 'the retry re-points the links at the live path' + ); + await cleanup(name); + }); + it('discards a broken staged candidate without failing the healthy live component closed', async () => { + // Only a not-yet-activated candidate is broken here; the live tree and its persisted config are + // consistent, so the fail-closed rationale that applies to an interrupted activation does not. + // Failing this component would also be permanent: nothing else sweeps a staging directory whose + // component subtree is missing. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('live') }); + await stageApplication(application, deploymentId); + await fs.mkdir(application.dirPath, { recursive: true }); + await fs.writeFile(path.join(application.dirPath, 'index.js'), "module.exports = 'live';\n"); + // Break the candidate: remove its component subtree but leave the deployment directory behind. + await fs.rm(stagedApplicationPath(application.dirPath, deploymentId), { recursive: true, force: true }); + + const settled = []; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => ({ + deployment_id: deploymentId, + project: name, + status: 'staged', + activation_spec: { package: null }, + }), + async () => { + throw new Error('activation persistence must not be reached for a staged row'); + }, + async (id) => settled.push(id) + ); + + assert.strictEqual(reconciliation.failedProjects.has(name), false, 'the live component is not failed closed'); + assert.deepStrictEqual( + settled, + [deploymentId], + 'the unusable candidate row is settled so its payload can be reclaimed' + ); + assert.strictEqual(reconciliation.removed.includes(deploymentId), true, 'and its residue is removed'); + assert.match(await readMarker(application.dirPath), /live/, 'the live component is untouched'); + await cleanup(name); + }); + + it('does not fail a component closed when settling a broken staged candidate itself fails', async () => { + // The staged branch can still throw — an I/O error on the completeness probe, or on settling or + // removing the residue. None of those say the live tree disagrees with its config, so none should + // keep the component from loading; only an interrupted activation does. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('live') }); + await stageApplication(application, deploymentId); + await fs.mkdir(application.dirPath, { recursive: true }); + await fs.writeFile(path.join(application.dirPath, 'index.js'), "module.exports = 'live';\n"); + await fs.rm(stagedApplicationPath(application.dirPath, deploymentId), { recursive: true, force: true }); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => ({ + deployment_id: deploymentId, + project: name, + status: 'staged', + activation_spec: { package: null }, + }), + async () => {}, + async () => { + throw new Error('simulated settle failure'); + } + ); + + assert.strictEqual(reconciliation.errors.has(deploymentId), true, 'the failure is still reported'); + assert.strictEqual( + reconciliation.failedProjects.has(name), + false, + 'but it is not attributed to the component, so the healthy live tree still loads' + ); + await cleanup(name); + }); + it('rolls config back when a failure lands after the config commit, not just during it', async () => { + // The commit is followed by the manifest write and the retain rename. A failure in either used to + // undo only the directories, leaving root config and the application lock naming the reverted-to + // release while the original bytes were live again — which a cold start would then reinstall over. + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + let committed = 0; + let rolledBack = 0; + + // Fail the retain rename by making the retained-previous slot unwritable at that moment: the + // manifest write has already succeeded, so this is strictly post-commit. + const previousRoot = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR); + await assert.rejects(() => + revertApplication(application, first, { + commitPersistentState: async () => { + committed++; + await fs.chmod(previousRoot, 0o500); + }, + rollbackPersistentState: async () => { + rolledBack++; + await fs.chmod(previousRoot, 0o700); + }, + }) + ); + await fs.chmod(previousRoot, 0o700).catch(() => {}); + + assert.strictEqual(committed, 1, 'the commit ran'); + assert.strictEqual(rolledBack, 1, 'and a post-commit failure rolled it back'); + assert.match(await readMarker(application.dirPath), /v2/, 'the pre-revert version is live again'); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, second, 'the manifest was put back too, so live/manifest agree'); + assert.strictEqual(target.previous.deployment_id, first); + await cleanup(name); + }); + + it('rolls persistent state back when the commit itself rejects, not only when a later step does', async () => { + // `createApplicationConfigTransaction.commit()` marks itself started before writing root config and + // can reject between that write and the application-lock write. Gating compensation on "the commit + // resolved" therefore skipped the rollback for a commit that had already changed persisted state, + // and the directories were restored while config still named the reverted-to release — which a cold + // start would reinstall over. + const name = fixtureName(); + const { application, second } = await twoActivations(name); + let rolledBack = 0; + const revertTo = (await getRevertTarget(application.dirPath)).previous.deployment_id; + + await assert.rejects( + () => + revertApplication(application, revertTo, { + commitPersistentState: async () => { + throw new Error('application lock write failed after root config changed'); + }, + rollbackPersistentState: async () => { + rolledBack++; + }, + }), + /application lock write failed/ + ); + + assert.strictEqual(rolledBack, 1, 'a rejected commit is still an attempted commit, so it is rolled back'); + assert.match(await readMarker(application.dirPath), /v2/, 'and the pre-revert version is live again'); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, second); + await cleanup(name); + }); + + it('leaves the holding tree and marker in place when the persistent rollback itself fails', async () => { + // Persisted state may name the reverted-to release with no way to take it back. Undoing the + // directories then would contradict it, so compensation stops and leaves exactly the shape startup + // recovery rolls forward from — rather than restoring the old bytes and consuming the evidence. + const name = fixtureName(); + const { application } = await twoActivations(name); + const previousRoot = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR); + const revertTo = (await getRevertTarget(application.dirPath)).previous.deployment_id; + + await assert.rejects( + () => + revertApplication(application, revertTo, { + commitPersistentState: async () => { + throw new Error('commit failed'); + }, + rollbackPersistentState: async () => { + throw new Error('rollback failed too'); + }, + }), + /left in place for startup recovery/ + ); + + const holdings = (await fs.readdir(previousRoot)).filter( + // The marker shares the holding directory's prefix, so exclude it or it counts as a second tree. + (entry) => entry.startsWith(`.reverting-${name}-`) && !entry.endsWith('.recovering.json') + ); + assert.strictEqual(holdings.length, 1, 'the holding tree still holds the previously-live bytes'); + assert.strictEqual( + existsSync(path.join(previousRoot, `${holdings[0]}.recovering.json`)), + true, + 'and its marker survives, which is what recovery keys on' + ); + // Live holds the reverted-to bytes and the retained slot is empty: precisely the state recovery + // reads as "the revert happened, only the retain step was lost". + assert.match(await readMarker(application.dirPath), /v1/, 'the reverted-to version is left live'); + assert.strictEqual( + existsSync(path.join(previousRoot, name)), + false, + 'and the retained slot is empty, because the holding tree has not been moved into it yet' + ); + + // Recovery finishes what compensation could not. + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + assert.strictEqual(failures.size, 0); + assert.match(await readMarker(application.dirPath), /v1/, 'the reverted-to version stays live'); + assert.match(await readMarker(path.join(previousRoot, name)), /v2/, 'and the displaced tree is retained'); + await cleanup(name); + }); + + it('rolls forward an interrupted revert whose compensation had already moved live away', async () => { + // Crash shape: config and the manifest committed, then compensation renamed live away before it + // could put the holding tree back. Undoing the directories here would contradict the persisted + // state, so recovery has to finish the revert instead — decided by comparing marker to manifest. + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + const holding = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, `.reverting-${name}-${randomUUID()}`); + + // Build that exact state by hand: live absent, previous = reverted-to tree, holding = old live, + // manifest already exchanged, and a marker recording the same intent. + const staleManifest = JSON.parse(await fs.readFile(`${previousPath}.json`, 'utf8')); + const intended = { previous: staleManifest.live, live: staleManifest.previous }; + await fs.rename(application.dirPath, holding); + await fs.writeFile(`${holding}.recovering.json`, JSON.stringify(intended, null, 2)); + await fs.writeFile(`${previousPath}.json`, JSON.stringify(intended, null, 2)); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.match(await readMarker(application.dirPath), /v1/, 'the reverted-to version is live, matching config'); + assert.match(await readMarker(previousPath), /v2/, 'the displaced tree is retained'); + assert.strictEqual(existsSync(holding), false); + assert.strictEqual(existsSync(`${holding}.recovering.json`), false); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual(target.live.deployment_id, first); + assert.strictEqual(target.previous.deployment_id, second); + await cleanup(name); + }); + it('serializes root-config and application-lock writes on a lock outside this isolate', async () => { + // The in-isolate write queue only orders writers within one isolate, but peer phases run in worker + // threads and different projects take different component locks — so an unsynchronized + // read-modify-write of the shared lock file drops a sibling project's entry. The transaction now + // takes a filesystem lock keyed on the lock file itself, which is what makes it cross-isolate. + // + // Holding that same lock from outside proves the critical section exists: a commit cannot proceed + // while it is held. A same-isolate concurrency test could NOT demonstrate this — the in-isolate + // queue masks the interleaving, which is exactly why the file lock was needed. + const name = fixtureName(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-persist-lock-')); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + await fs.writeFile(path.join(configRoot, 'harper-config.yaml'), `rootPath: ${JSON.stringify(configRoot)}\n`); + const lockPath = path.join(configRoot, 'harper-application-lock.json'); + try { + const transaction = await createApplicationConfigTransaction(name, { package: 'example@1.0.0' }); + let committed = false; + let releaseHold; + const holdReleased = new Promise((resolve) => (releaseHold = resolve)); + + // The holder waits on an external signal, and the commit is started OUTSIDE it. Awaiting the + // commit from inside would deadlock: the lock is not reentrant, so the holder could not release + // until the commit finished and the commit could not start until the holder released. + // Signalled from inside the callback rather than slept on: a fixed delay does not prove the + // filesystem lock was actually acquired, so on a loaded runner the commit could win the race. + let holderEntered; + const entered = new Promise((resolve) => (holderEntered = resolve)); + const holder = withComponentPreparationLock(lockPath, () => { + holderEntered(); + return holdReleased; + }); + await entered; + const commitPromise = transaction.commit().then(() => (committed = true)); + await new Promise((resolve) => setTimeout(resolve, 200)); + assert.strictEqual(committed, false, 'the commit must wait while the persistent-state lock is held'); + + releaseHold(); + await holder; + await commitPromise; + assert.strictEqual(committed, true, 'and proceed once it is released'); + assert.deepStrictEqual(readConfigFile()[name], { package: 'example@1.0.0' }); + const lock = JSON.parse(await fs.readFile(lockPath, 'utf8')); + assert.deepStrictEqual(lock.applications[name], { package: 'example@1.0.0' }); + } finally { + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); + + it("keeps a sibling project's application-lock entry across a second project's transaction", async () => { + const first = fixtureName(); + const second = fixtureName(); + const configRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-persist-sibling-')); + const priorRootEnv = process.env.ROOTPATH; + const priorRootConfig = getConfigPath(CONFIG_PARAMS.ROOTPATH); + process.env.ROOTPATH = configRoot; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, configRoot); + await fs.writeFile(path.join(configRoot, 'harper-config.yaml'), `rootPath: ${JSON.stringify(configRoot)}\n`); + try { + const a = await createApplicationConfigTransaction(first, { package: 'a@1.0.0' }); + const b = await createApplicationConfigTransaction(second, { package: 'b@1.0.0' }); + await Promise.all([a.commit(), b.commit()]); + + const lock = JSON.parse(await fs.readFile(path.join(configRoot, 'harper-application-lock.json'), 'utf8')); + assert.deepStrictEqual(lock.applications[first], { package: 'a@1.0.0' }, 'the first entry survives'); + assert.deepStrictEqual(lock.applications[second], { package: 'b@1.0.0' }, 'and so does the second'); + } finally { + if (priorRootEnv === undefined) delete process.env.ROOTPATH; + else process.env.ROOTPATH = priorRootEnv; + environment.setProperty(CONFIG_PARAMS.ROOTPATH, priorRootConfig); + await fs.rm(configRoot, { recursive: true, force: true }); + } + }); + + it('settles a crash-stranded in-flight row so its payload stops being retained', async () => { + // A node killed mid-stage leaves a `pending`/`staging` row. Reconciliation removes its staging + // directory, but payload retention only reclaims rows that reached a terminal status — so leaving + // the row in flight pins its tarball on the origin and every peer indefinitely. + const name = fixtureName(); + const strandedId = randomUUID(); + const terminalId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('stranded') }); + await stageApplication(application, strandedId); + application.payload = await makeComponentPayload('terminal'); + await stageApplication(application, terminalId); + const rows = new Map([ + [strandedId, { deployment_id: strandedId, project: name, status: 'staging' }], + // An already-terminal row is swept the same way but must NOT be re-settled: patching it would + // rewrite a successful deploy's status to failed. + [terminalId, { deployment_id: terminalId, project: name, status: 'success' }], + ]); + + const settled = []; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => rows.get(id), + async () => {}, + async (id, reason) => settled.push([id, reason]) + ); + + assert.deepStrictEqual( + settled, + [[strandedId, 'the deploy did not survive a restart']], + 'only the in-flight row is settled, and with a reason naming the crash' + ); + assert.strictEqual(reconciliation.removed.includes(strandedId), true, 'its staging residue is removed'); + assert.strictEqual(reconciliation.removed.includes(terminalId), true, "and so is the terminal row's"); + await cleanup(name); + }); + + it('rolls a crashed activation forward even after boot-time installation recreated the live directory', async () => { + // installApplications() runs BEFORE this recovery and recreates a package component from root + // config whenever the live path is missing — which is exactly the state an activation that died + // between its two renames leaves behind. Resuming that attempt reuses the existing backup, so + // nothing moves the recreated tree away and `rename(staging, live)` used to fail ENOTEMPTY. Being + // recreated on every boot, that made the component permanently unloadable. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('displaced') }); + await stageApplication(application, deploymentId); + await fs.mkdir(application.dirPath, { recursive: true }); + await fs.writeFile(path.join(application.dirPath, 'index.js'), "module.exports = 'displaced';\n"); + application.payload = await makeComponentPayload('candidate'); + const secondId = randomUUID(); + await stageApplication(application, secondId); + + // The crash shape: live already moved aside under this deployment's backup, staging not yet moved. + const activationDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationDir, { recursive: true }); + await fs.rename(application.dirPath, path.join(activationDir, `.previous-${secondId}-1-1-${randomUUID()}`)); + // Then boot-time installation recreates the live directory from root config. + await fs.mkdir(application.dirPath, { recursive: true }); + await fs.writeFile(path.join(application.dirPath, 'index.js'), "module.exports = 'reinstalled';\n"); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === secondId ? { deployment_id: secondId, project: name, status: 'activating' } : undefined), + async () => {} + ); + + assert.deepStrictEqual([...reconciliation.errors.keys()], [], 'the roll-forward succeeds'); + assert.strictEqual(reconciliation.failedProjects.has(name), false, 'so the component is not failed closed'); + assert.match(await readMarker(application.dirPath), /candidate/, 'the staged candidate is live'); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + assert.match( + await readMarker(previousPath), + /displaced/, + 'and the release the activation actually displaced is retained, not the reinstalled tree' + ); + await cleanup(name); + }); + + it('does not discard a staged candidate when the completeness probe fails for a reason other than absence', async () => { + // The probe used to map every error to "absent", so a transient EACCES/EIO/EMFILE at startup made a + // complete release look broken and the staged branch deleted it. Only genuine absence may authorize + // destroying a candidate. A self-referencing symlink gives a deterministic ELOOP for this, with no + // dependence on permissions or platform. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('staged') }); + await stageApplication(application, deploymentId); + const stagedPath = stagedApplicationPath(application.dirPath, deploymentId); + await fs.rm(stagedPath, { recursive: true, force: true }); + await fs.symlink(stagedPath, stagedPath); + + const settled = []; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => ({ deployment_id: deploymentId, project: name, status: 'staged' }), + async () => {}, + async (id, reason) => settled.push([id, reason]) + ); + + assert.deepStrictEqual(settled, [], 'the row is not settled on an inconclusive probe'); + assert.strictEqual(reconciliation.removed.includes(deploymentId), false, 'and its tree is not deleted'); + assert.strictEqual(reconciliation.errors.has(deploymentId), true, 'the failure is reported instead'); + assert.strictEqual( + reconciliation.failedProjects.has(name), + false, + 'a staged candidate never blocks the live component, which is unaffected either way' + ); + await fs.rm(stagedPath, { force: true }); + await cleanup(name); + }); + + it('rolls a peer activation forward from local evidence when its row still reads staged', async () => { + // A peer swaps WITHOUT writing the deployment row — the origin owns it — so a peer that died + // between its swap and its config commit leaves a `staged` row with no staged leaf. That read as a + // broken candidate: the row was settled `failed` (replicating over the origin's own row) and the + // config was never persisted, so the peer ran new code under the previous release's configuration. + // The activation artifact is local proof the swap began and has to outrank the replicated status. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + + // The peer's post-swap state: candidate live, staged leaf gone, backup parked, row still `staged`. + const stagedPath = stagedApplicationPath(application.dirPath, deploymentId); + const activationDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationDir, { recursive: true }); + const backupPath = path.join(activationDir, `.previous-${deploymentId}-1-1-${randomUUID()}`); + await fs.mkdir(backupPath, { recursive: true }); + await fs.writeFile(path.join(backupPath, 'index.js'), "module.exports = 'displaced';\n"); + await fs.rename(stagedPath, application.dirPath); + + const settled = []; + let persisted = 0; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? { deployment_id: id, project: name, status: 'staged' } : undefined), + async () => persisted++, + async (id, reason) => settled.push([id, reason]) + ); + + assert.strictEqual(persisted, 1, 'the interrupted activation persists its configuration'); + assert.deepStrictEqual(settled, [], 'and the row is NOT settled failed over the origin-owned row'); + assert.strictEqual(reconciliation.failedProjects.has(name), false); + assert.match(await readMarker(application.dirPath), /candidate/, 'the swapped candidate stays live'); + await cleanup(name); + }); + + it('fails a component closed when the peer-evidence roll-forward cannot persist its configuration', async () => { + // The roll-forward path now accepts a `staged` row plus an activation artifact, so its FAILURE has to + // fail closed the same way an `activating` row does. Keying attribution on the row status alone let a + // rejecting persistence step (disk full, bad activation spec) record an error and still load the + // swapped candidate under the previous release's durable configuration. + const name = fixtureName(); + const deploymentId = randomUUID(); + const application = new Application({ name, payload: await makeComponentPayload('candidate') }); + await stageApplication(application, deploymentId); + const stagedPath = stagedApplicationPath(application.dirPath, deploymentId); + const activationDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationDir, { recursive: true }); + await fs.mkdir(path.join(activationDir, `.previous-${deploymentId}-1-1-${randomUUID()}`), { recursive: true }); + await fs.rename(stagedPath, application.dirPath); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? { deployment_id: id, project: name, status: 'staged' } : undefined), + async () => { + throw new Error('activation spec is unusable'); + } + ); + + assert.strictEqual(reconciliation.errors.has(deploymentId), true, 'the failure is reported'); + assert.strictEqual( + reconciliation.failedProjects.has(name), + true, + 'and the component does not load with a swapped tree under unreconciled config' + ); + await cleanup(name); + }); + + it('rolls forward an interrupted no-live restore instead of sweeping away its marker', async () => { + // The no-live revert branch renames `previous` straight to live because there is nothing to + // displace, so it has no holding directory — and its marker is the only evidence. Absence of a + // holding directory therefore must not read as an orphaned marker: sweeping it leaves the + // component with no live tree at all and no record that a restore was in progress. + const name = fixtureName(); + const { application, first } = await twoActivations(name); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + const manifest = JSON.parse(await fs.readFile(`${previousPath}.json`, 'utf8')); + + // Crash shape: marker written, live already gone, the retained tree not yet moved into place. + await fs.rm(application.dirPath, { recursive: true, force: true }); + const restoreMarker = `${previousPath}.restoring.recovering.json`; + await fs.writeFile( + restoreMarker, + JSON.stringify({ previous: { deployment_id: null, application_config: null }, live: manifest.previous }, null, 2) + ); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.match(await readMarker(application.dirPath), /v1/, 'the retained version is live'); + assert.strictEqual(existsSync(previousPath), false, 'and is no longer parked under retention'); + assert.strictEqual(existsSync(restoreMarker), false, 'the marker is cleared only after everything committed'); + const restored = JSON.parse(await fs.readFile(`${previousPath}.json`, 'utf8')); + assert.strictEqual(restored.live.deployment_id, first, 'the manifest names the version the marker named'); + assert.strictEqual(restored.previous.deployment_id, null, 'with nothing retained behind it'); + assert.strictEqual( + await getRevertTarget(application.dirPath), + undefined, + 'so the component reports as not revertable until its next deploy' + ); + await cleanup(name); + }); + + it('keeps a parked backup it cannot prove belongs to the live release, rather than retaining or deleting it', async () => { + // A settled row plus a good live tree proves the activation ended — not that it ended as the CURRENT + // release. With no manifest naming this deployment as live, the artifact could be left over from an + // older deployment, and retaining it would overwrite a valid retained previous with stale bytes. + // Deleting it is equally wrong: it may be the only copy of what that deploy displaced. So it stays. + const name = fixtureName(); + const deploymentId = randomUUID(); + const livePath = path.join(COMPONENTS_ROOT, name); + await fs.mkdir(livePath, { recursive: true }); + await fs.writeFile(path.join(livePath, 'index.js'), "module.exports = 'live';\n"); + const activationProjectDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationProjectDir, { recursive: true }); + const backupPath = path.join(activationProjectDir, `.previous-${deploymentId}-1-1-${randomUUID()}`); + await fs.mkdir(backupPath, { recursive: true }); + await fs.writeFile(path.join(backupPath, 'index.js'), "module.exports = 'displaced';\n"); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? { deployment_id: id, project: name, status: 'success' } : undefined), + async () => { + throw new Error('a settled row must not be re-persisted'); + } + ); + + assert.deepStrictEqual([...reconciliation.errors.keys()], []); + assert.strictEqual(existsSync(backupPath), true, 'the unprovable backup is left in place'); + assert.strictEqual( + existsSync(path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name)), + false, + 'and nothing stale is promoted into the retained slot' + ); + assert.match(await readMarker(livePath), /live/, 'the live tree is untouched'); + await fs.rm(path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name), { recursive: true, force: true }); + await cleanup(name); + }); + + it('keeps the displaced release ADDRESSABLE when retention failed, not just present', async () => { + // Retaining the bytes is only half the job: a manifest that records the displaced release as an + // unknown deployment leaves revert_component with nothing to target, so the recovered deploy is + // still effectively unrevertable. The failed-retain path therefore records the intended manifest + // when the retained slot is empty, and recovery reads the side of it that names the displaced tree. + const name = fixtureName(); + const { application, first, second } = await twoActivations(name); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + + // Rebuild the failed-retain shape by hand: the displaced tree (v1) parked as an activation backup, + // the retained slot empty, and the manifest describing the intended end state. + const activationProjectDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationProjectDir, { recursive: true }); + const backupPath = path.join(activationProjectDir, `.previous-${second}-1-1-${randomUUID()}`); + await fs.rename(previousPath, backupPath); + await fs.writeFile( + `${previousPath}.json`, + JSON.stringify({ + previous: { deployment_id: first, application_config: null }, + live: { deployment_id: second, application_config: null }, + }) + ); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === second ? { deployment_id: id, project: name, status: 'success' } : undefined), + async () => {} + ); + + assert.deepStrictEqual([...reconciliation.errors.keys()], []); + assert.match(await readMarker(previousPath), /v1/, 'the displaced bytes are retained'); + const target = await getRevertTarget(application.dirPath); + assert.strictEqual( + target.previous.deployment_id, + first, + 'and the manifest names which deployment they are, so revert_component can address it' + ); + assert.strictEqual(target.live.deployment_id, second); + await cleanup(name); + }); + + it('recovers a revert whose holding tree is itself a symlink', async () => { + // A `file:` directory deploy makes the live path a symlink, and the revert renames that live path + // into the holding slot — so the holding entry is a symlink, not a directory. Gating the recovery + // loop on isDirectory() skipped those entries entirely: the component was left with no live tree + // and recovered on no restart, ever. + const name = fixtureName(); + const linkTarget = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-revert-dirpkg-')); + await fs.writeFile(path.join(linkTarget, 'index.js'), "module.exports = 'symlinked-live';\n"); + const livePath = path.join(COMPONENTS_ROOT, name); + const previousRoot = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR); + await fs.mkdir(previousRoot, { recursive: true }); + + // Crash shape: live (a symlink) already renamed into the holding slot, nothing put back yet. + const holding = path.join(previousRoot, `.reverting-${name}-${randomUUID()}`); + await fs.symlink(linkTarget, holding); + + const failures = await recoverInterruptedReverts(COMPONENTS_ROOT); + + assert.strictEqual(failures.size, 0); + assert.match( + await readMarker(livePath), + /symlinked-live/, + 'the symlinked tree is restored to the live path rather than skipped forever' + ); + assert.strictEqual(existsSync(holding), false, 'and the holding slot is emptied'); + + await fs.rm(livePath, { force: true }); + await fs.rm(linkTarget, { recursive: true, force: true }); + await cleanup(name); + }); + + it('treats a live directory-package symlink as a live component, not a missing one', async () => { + // A `file:` directory deploy is materialized as a symlink by design. Requiring a real directory + // reported this shape as "neither staged nor live", and the artifact sweep then deleted the + // `.previous-*` backup — the only copy of the displaced release — while the component stayed + // failed closed. This is the exact post-swap crash state: symlink already renamed live, the + // displaced tree still parked, persistence not yet finished. + const name = fixtureName(); + const deploymentId = randomUUID(); + const livePath = path.join(COMPONENTS_ROOT, name); + const linkTarget = await fs.mkdtemp(path.join(os.tmpdir(), 'harper-dirpkg-')); + await fs.writeFile(path.join(linkTarget, 'index.js'), "module.exports = 'directory-package';\n"); + await fs.symlink(linkTarget, livePath); + const activationProjectDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationProjectDir, { recursive: true }); + const backupPath = path.join(activationProjectDir, `.previous-${deploymentId}-1-1-${randomUUID()}`); + await fs.mkdir(backupPath, { recursive: true }); + await fs.writeFile(path.join(backupPath, 'index.js'), "module.exports = 'displaced';\n"); + + let persisted = 0; + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async (id) => (id === deploymentId ? { deployment_id: id, project: name, status: 'activating' } : undefined), + async () => persisted++ + ); + + assert.strictEqual(persisted, 1, 'the activation is finished rather than reported unrecoverable'); + assert.strictEqual(reconciliation.failedProjects.has(name), false, 'so the component is not failed closed'); + assert.match(await readMarker(livePath), /directory-package/, 'the symlinked live component is untouched'); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + assert.match(await readMarker(previousPath), /displaced/, 'and its displaced release is retained, not deleted'); + + await fs.rm(livePath, { force: true }); + await fs.rm(linkTarget, { recursive: true, force: true }); + await cleanup(name); + }); + + it('fails a component closed when an activation-artifact lookup throws, destroying nothing', async () => { + // A failed lookup cannot distinguish "no such deployment" from "the table is unreadable", so it may + // not authorize cleanup: the artifact could be the displaced release, and the live tree could be + // mid-activation and disagreeing with its durable config. Block the component and touch nothing. + const name = fixtureName(); + const deploymentId = randomUUID(); + await twoActivations(name); + const previousPath = path.join(COMPONENTS_ROOT, DEPLOY_PREVIOUS_DIR, name); + const activationProjectDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationProjectDir, { recursive: true }); + const artifactPath = path.join(activationProjectDir, `.previous-${deploymentId}-1-1-${randomUUID()}`); + await fs.mkdir(artifactPath, { recursive: true }); + + const reconciliation = await reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => { + throw new Error('deployment table unavailable'); + }, + async () => {} + ); + + assert.strictEqual(reconciliation.failedProjects.has(name), true, 'the component does not load'); + assert.strictEqual(reconciliation.errors.has(deploymentId), true, 'and the failure is reported'); + assert.strictEqual(existsSync(artifactPath), true, 'the artifact is left alone'); + assert.match(await readMarker(previousPath), /v1/, 'and the retained previous release is untouched'); + await fs.rm(activationProjectDir, { recursive: true, force: true }); + await cleanup(name); + }); + + it('waits for the component lock before restoring an activation backup over the live path', async () => { + // The sweep renames a backup back over the live path. Run unlocked, a reload-cycle retry could do + // that inside an activation's swap window — live momentarily absent, backup present — after which + // the in-flight rename fails ENOTEMPTY and its own compensation fails ENOENT. So the sweep has to + // hold the same lock every other mutator of that path holds. + const name = fixtureName(); + const deploymentId = randomUUID(); + const livePath = path.join(COMPONENTS_ROOT, name); + const activationProjectDir = path.join(COMPONENTS_ROOT, DEPLOY_ACTIVATION_DIR, name); + await fs.mkdir(activationProjectDir, { recursive: true }); + const backupPath = path.join(activationProjectDir, `.previous-${deploymentId}-1-1-${randomUUID()}`); + await fs.mkdir(backupPath, { recursive: true }); + await fs.writeFile(path.join(backupPath, 'index.js'), "module.exports = 'backup';\n"); + // The swap window this models: no live directory, only the backup. + assert.strictEqual(existsSync(livePath), false); + + let releaseHold; + const holdReleased = new Promise((resolve) => (releaseHold = resolve)); + // Reconcile is started OUTSIDE the holder — awaiting it from inside would deadlock, since the lock + // is not reentrant. Entry is signalled from inside the callback, not slept on, so the assertions + // below cannot pass merely because the sweep lost a timing race. + let holderEntered; + const entered = new Promise((resolve) => (holderEntered = resolve)); + const holder = withComponentPreparationLock(livePath, () => { + holderEntered(); + return holdReleased; + }); + await entered; + + let reconciled = false; + const reconcilePromise = reconcileStagedApplicationArtifacts( + COMPONENTS_ROOT, + async () => undefined, + async () => {} + ).then((result) => { + reconciled = true; + return result; + }); + await new Promise((resolve) => setTimeout(resolve, 300)); + + assert.strictEqual(reconciled, false, 'the sweep must not finish while the component lock is held'); + assert.strictEqual(existsSync(livePath), false, 'and must not restore the backup underneath the lock holder'); + assert.strictEqual(existsSync(backupPath), true, 'leaving the backup for after the lock is released'); + + releaseHold(); + await holder; + await reconcilePromise; + + assert.strictEqual(existsSync(backupPath), false, 'once released, the sweep settles the artifact'); + assert.match(await readMarker(livePath), /backup/, 'restoring the backup over the absent live path'); + await cleanup(name); + }); + + // Same contract the extraction scan owes its caller: componentLoader fails startup closed on a + // rejection, so an absent retention root (nothing ever retained) must stay distinguishable from one + // we could not read, where a component may be mid-revert with no live directory at all. + it('resolves empty when the retention root is absent but rejects when it is unreadable', async () => { + const scanRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'revert-scan-')); + try { + assert.strictEqual((await recoverInterruptedReverts(scanRoot)).size, 0); + + await fs.writeFile(path.join(scanRoot, DEPLOY_PREVIOUS_DIR), 'not a directory\n'); + await assert.rejects(() => recoverInterruptedReverts(scanRoot), { code: 'ENOTDIR' }); + } finally { + await fs.rm(scanRoot, { recursive: true, force: true }); + } + }); +}); diff --git a/unitTests/components/deployValidators.test.js b/unitTests/components/deployValidators.test.js new file mode 100644 index 0000000000..dd0f426950 --- /dev/null +++ b/unitTests/components/deployValidators.test.js @@ -0,0 +1,34 @@ +'use strict'; + +const assert = require('node:assert'); +const validator = require('#js/components/operationsValidation'); + +const invalid = (result) => assert.ok(result, 'expected a validation error'); + +describe('deployComponentValidator', () => { + it('rejects retry-unsafe automatic rollback', () => { + invalid(validator.deployComponentValidator({ project: 'my_app', revert_on_failure: true })); + }); + + it('rejects the removed staged-deploy contract fields instead of ignoring them', () => { + // The schema allows unknown keys, so dropping these from it would have left a caller still sending + // the (never-released) staged contract silently getting a full deploy — `activate: false` ignored, + // the opposite of the intent. They fail fast and name the follow-on instead. + invalid(validator.deployComponentValidator({ project: 'my_app', activate: false })); + invalid(validator.deployComponentValidator({ project: 'my_app', two_phase: true })); + invalid(validator.deployComponentValidator({ project: 'my_app', two_phase: false })); + invalid( + validator.deployComponentValidator({ project: 'my_app', deployment_id: '41faded8-6cf5-4a2a-95f8-863e7ea498fa' }) + ); + invalid(validator.deployComponentValidator({ project: 'my_app', _phase: 'stage' })); + }); + + it('rejects the caller-supplied internal deployment marker', () => { + invalid(validator.deployComponentValidator({ project: 'my_app', _deploymentId: 'x' })); + }); + + it('preserves routing validation', () => { + invalid(validator.deployComponentValidator({ project: 'my_app', package: 'npm:x', urlPath: '/a/./b' })); + invalid(validator.deployComponentValidator({ project: 'my_app', package: 'npm:x', urlPath: '/a/../b' })); + }); +}); diff --git a/unitTests/components/deploymentOperations.test.js b/unitTests/components/deploymentOperations.test.js index 5748039f9c..e2b26bc1ca 100644 --- a/unitTests/components/deploymentOperations.test.js +++ b/unitTests/components/deploymentOperations.test.js @@ -14,7 +14,13 @@ const { Readable } = require('node:stream'); const testUtils = require('../testUtils.js'); testUtils.preTestPrep(); -const { handleGetDeploymentPayload, handleDeleteDeploymentPayload } = require('#src/components/deploymentOperations'); +const { + handleGetDeployment, + handleGetDeploymentPayload, + handleDeleteDeploymentPayload, +} = require('#src/components/deploymentOperations'); +const { DeploymentRecorder } = require('#src/components/deploymentRecorder'); +const { ProgressEmitter } = require('#src/server/serverHelpers/progressEmitter'); const { databases } = require('#src/resources/databases'); const terms = require('#src/utility/hdbTerms'); @@ -64,6 +70,33 @@ async function collect(stream) { return Buffer.concat(chunks); } +describe('handleGetDeployment SSE tail', () => { + let installed; + beforeEach(() => { + installed = installMockDeploymentTable(); + }); + afterEach(() => installed.restore()); + + it('keeps tailing a transient staged checkpoint while the origin recorder is active', async () => { + const lifecycle = new ProgressEmitter(); + const recorder = await DeploymentRecorder.create({ project: 'app', emitter: lifecycle }); + await recorder.checkpoint('staged', 'staged'); + let settled = false; + const resultPromise = handleGetDeployment({ + deployment_id: recorder.deploymentId, + progress: new ProgressEmitter(), + }).then((result) => { + settled = true; + return result; + }); + await new Promise((resolve) => setImmediate(resolve)); + assert.strictEqual(settled, false, 'staged is only terminal after its live recorder is gone'); + + await recorder.finish('success'); + assert.strictEqual((await resultPromise).status, 'success'); + }); +}); + describe('handleGetDeploymentPayload', () => { let installed; beforeEach(() => { @@ -138,13 +171,15 @@ describe('handleDeleteDeploymentPayload', () => { }); }); - it('409s on a non-terminal deployment (blob may still be replicating to peers)', async () => { - installed.mock.rows.set('d1', { deployment_id: 'd1', status: 'pending', payload_blob: mockBlob(Buffer.from('x')) }); - await assert.rejects(handleDeleteDeploymentPayload({ deployment_id: 'd1' }), (err) => { - assert.match(err.message, /not in a terminal state/); - assert.strictEqual(err.statusCode, 409); - return true; - }); + it('409s on pending and staged deployments whose blobs may still be needed by peers', async () => { + for (const status of ['pending', 'staged']) { + installed.mock.rows.set('d1', { deployment_id: 'd1', status, payload_blob: mockBlob(Buffer.from('x')) }); + await assert.rejects(handleDeleteDeploymentPayload({ deployment_id: 'd1' }), (err) => { + assert.match(err.message, /not in a terminal state/); + assert.strictEqual(err.statusCode, 409); + return true; + }); + } assert.strictEqual(installed.mock.puts.length, 0, 'must not write the row'); }); diff --git a/unitTests/components/deploymentRecorder.test.js b/unitTests/components/deploymentRecorder.test.js index 134068ec1f..b63f01bd19 100644 --- a/unitTests/components/deploymentRecorder.test.js +++ b/unitTests/components/deploymentRecorder.test.js @@ -18,6 +18,11 @@ const { DeploymentRecorder, awaitDeploymentRow, readPayloadBlobWithRetry, + markDeploymentTerminal, + recordDeploymentPeers, + invalidateProjectStagedDeployments, + pruneProjectPayloads, + getDeploymentRow, ingestTransactionTimeoutMs, DEFAULT_INGEST_TRANSACTION_TIMEOUT_MS, } = require('#src/components/deploymentRecorder'); @@ -29,17 +34,35 @@ const { waitFor } = require('../waitFor.js'); const DEPLOYMENT_TABLE = terms.SYSTEM_TABLE_NAMES.DEPLOYMENT_TABLE_NAME; -// Lightweight mock: keeps a Map of rows, exposes get(id) and put(row). -function installMockDeploymentTable() { +// Lightweight mock: keeps a Map of rows, exposes get(id), put(row), and patch(id, partial). With +// `freezeGet`, get() returns a FROZEN shallow copy to mirror the real table (table.get() yields a +// read-only record) so a caller that tries to mutate the fetched row — as the pre-fix +// markDeploymentTerminal did — throws here just as it would in production; the default returns the +// stored object so awaitDeploymentRow callers keep the real row (and its payload_blob). patch() +// applies a partial update to the stored row. +function installMockDeploymentTable({ freezeGet = false } = {}) { const rows = new Map(); const mock = { rows, async get(id) { - return rows.get(id); + const row = rows.get(id); + if (!row) return undefined; + return freezeGet ? Object.freeze({ ...row }) : row; }, async put(row) { rows.set(row.deployment_id, row); }, + async patch(id, partial) { + const existing = rows.get(id); + if (existing) rows.set(id, { ...existing, ...partial }); + }, + // Minimal equality-only search over the stored rows, matching how the real table yields rows for + // `[{attribute, value}]` conditions. Enough for pruneProjectPayloads (filters by project). + async *search(conditions = []) { + for (const row of rows.values()) { + if (conditions.every((c) => row[c.attribute] === c.value)) yield row; + } + }, }; if (!databases.system) databases.system = {}; const prior = databases.system[DEPLOYMENT_TABLE]; @@ -370,6 +393,12 @@ describe('awaitDeploymentRow', () => { assert.ok(result.payload_blob); }); + it('can wait for package deployment metadata without requiring a payload blob', async () => { + const row = { deployment_id: 'package-row', package_identifier: 'npm:example', payload_blob: null }; + installed.mock.rows.set(row.deployment_id, row); + assert.strictEqual(await awaitDeploymentRow(row.deployment_id, { timeoutMs: 0, requirePayload: false }), row); + }); + it('rejects with a "did not replicate" timeout when the row never arrives within timeoutMs', async () => { await assert.rejects( () => awaitDeploymentRow('never-arrives', { timeoutMs: 100, pollIntervalMs: 25 }), @@ -779,3 +808,255 @@ describe('readPayloadBlobWithRetry', () => { ); }); }); + +describe('markDeploymentTerminal', () => { + let installed; + beforeEach(() => { + // freezeGet mirrors the real table's read-only get() so a regression back to mutating the fetched + // row (row.status = …) throws here instead of silently passing against a mutable mock. + installed = installMockDeploymentTable({ freezeGet: true }); + }); + afterEach(() => installed.restore()); + + it('flips an existing row to the terminal status via patch (not a read-only mutation)', async () => { + // This is what deploy_component({ deployment_id }) does after taking a stage-and-stop build live. + // The row from table.get() is read-only; markDeploymentTerminal must patch rather than assign onto + // it (assigning threw "Cannot assign to read only property 'status'" and silently lost the update). + installed.mock.rows.set('dep-1', { deployment_id: 'dep-1', status: 'staged', completed_at: null }); + await markDeploymentTerminal('dep-1', 'success'); + const row = installed.mock.rows.get('dep-1'); + assert.strictEqual(row.status, 'success', 'the staged row was flipped to success'); + assert.strictEqual(typeof row.completed_at, 'number', 'completed_at was stamped'); + }); + + it('preserves an uncertain partial activation as non-terminal with its error', async () => { + installed.mock.rows.set('dep-activating', { + deployment_id: 'dep-activating', + status: 'staged', + completed_at: 100, + }); + + await markDeploymentTerminal('dep-activating', 'activating', new Error('peer did not acknowledge')); + + const row = installed.mock.rows.get('dep-activating'); + assert.strictEqual(row.status, 'activating'); + assert.strictEqual(row.completed_at, null); + assert.strictEqual(row.error.message, 'peer did not acknowledge'); + }); + + it('is a no-op when the row is absent (best-effort, never throws)', async () => { + await markDeploymentTerminal('missing', 'success'); + assert.strictEqual(installed.mock.rows.has('missing'), false, 'no row is fabricated for an unknown id'); + }); +}); + +describe('getDeploymentRow', () => { + let installed; + beforeEach(() => { + installed = installMockDeploymentTable(); + }); + afterEach(() => installed.restore()); + + it('returns the row, including its package_identifier', async () => { + // deploy_component({deployment_id}) reads this to recover the package identifier of a deployment + // that was staged as a `package` deploy — an activate-by-id request carries no `package` of its own. + installed.mock.rows.set('dep-1', { + deployment_id: 'dep-1', + project: 'my-app', + package_identifier: 'npm:@my-org/my-app@1.2.3', + status: 'staged', + }); + const row = await getDeploymentRow('dep-1'); + assert.strictEqual(row?.package_identifier, 'npm:@my-org/my-app@1.2.3'); + }); + + it('returns a row whose payload has already been reclaimed', async () => { + // The point of this helper over awaitDeploymentRow: that one polls until the row carries a + // payload_blob, so it would never return a deployment whose payload retention already dropped — + // which is exactly the row an activate-by-id still needs to read. + installed.mock.rows.set('reclaimed', { + deployment_id: 'reclaimed', + package_identifier: 'npm:pkg@1', + payload_blob: null, + status: 'success', + }); + const row = await getDeploymentRow('reclaimed'); + assert.strictEqual(row?.package_identifier, 'npm:pkg@1', 'a payload-less row is still returned'); + }); + + it('returns undefined for an unknown id or a blank id', async () => { + assert.strictEqual(await getDeploymentRow('nope'), undefined); + assert.strictEqual(await getDeploymentRow(''), undefined); + }); + + it('returns undefined when the deployment table is not provisioned', async () => { + installed.restore(); + const prior = databases.system?.[DEPLOYMENT_TABLE]; + if (databases.system) delete databases.system[DEPLOYMENT_TABLE]; + try { + assert.strictEqual(await getDeploymentRow('dep-1'), undefined, 'a missing table is not an error'); + } finally { + if (databases.system && prior !== undefined) databases.system[DEPLOYMENT_TABLE] = prior; + installed = installMockDeploymentTable(); + } + }); +}); + +describe('staged deployment state', () => { + let installed; + beforeEach(() => { + installed = installMockDeploymentTable({ freezeGet: true }); + }); + afterEach(() => installed.restore()); + + it('invalidates staged and activating rows when their component is dropped', async () => { + for (const status of ['staged', 'activating', 'success']) { + installed.mock.rows.set(status, { deployment_id: status, project: 'app', status }); + } + + const invalidated = await invalidateProjectStagedDeployments('app'); + + assert.deepStrictEqual(invalidated.sort(), ['activating', 'staged']); + assert.strictEqual(installed.mock.rows.get('staged').status, 'failed'); + assert.strictEqual(installed.mock.rows.get('activating').status, 'failed'); + assert.strictEqual(installed.mock.rows.get('success').status, 'success'); + }); + + it('persists normalized peer outcomes on an existing staged row', async () => { + installed.mock.rows.set('dep', { + deployment_id: 'dep', + status: 'activating', + peer_results: [{ node: 'peer-a', status: 'success' }], + }); + + await recordDeploymentPeers('dep', [ + { node: 'peer-a', status: 'failed', reason: 'offline' }, + { node: 'peer-b', status: 'success' }, + ]); + + const peers = installed.mock.rows.get('dep').peer_results; + assert.strictEqual(peers.length, 2); + assert.strictEqual(peers.find((peer) => peer.node === 'peer-a').error.message, 'offline'); + assert.strictEqual(peers.find((peer) => peer.node === 'peer-b').status, 'success'); + }); +}); + +describe('pruneProjectPayloads (deployment_payloadRetention_maxCount)', () => { + let installed; + beforeEach(() => { + installed = installMockDeploymentTable(); + }); + afterEach(() => installed.restore()); + + // A stored payload is modelled as a non-null payload_blob plus a payload_size, matching the row shape + // the recorder writes. started_at drives the newest-first ordering. + function seed(id, { project = 'app', startedAt, status = 'success', size = 100, payload = true }) { + installed.mock.rows.set(id, { + deployment_id: id, + project, + status, + started_at: startedAt, + payload_size: size, + payload_blob: payload ? { marker: id } : null, + event_log: [], + }); + } + const hasPayload = (id) => installed.mock.rows.get(id).payload_blob != null; + + it('keeps only the newest payload at the default count of 1, dropping older ones', async () => { + seed('d1', { startedAt: 100 }); + seed('d2', { startedAt: 200 }); + seed('d3', { startedAt: 300 }); + + const freed = await pruneProjectPayloads('app', 1); + + assert.strictEqual(hasPayload('d3'), true, 'the newest payload is retained'); + assert.strictEqual(hasPayload('d2'), false, 'older payloads are dropped'); + assert.strictEqual(hasPayload('d1'), false, 'older payloads are dropped'); + assert.strictEqual(freed, 200, 'reports the bytes reclaimed from the two dropped payloads'); + }); + + it('retains rows and their metadata — only the tarball bytes are reclaimed', async () => { + seed('keep', { startedAt: 200 }); + seed('pruned', { startedAt: 100 }); + + await pruneProjectPayloads('app', 1); + + const row = installed.mock.rows.get('pruned'); + assert.ok(row, 'the pruned deployment row still exists (audit trail preserved)'); + assert.strictEqual(row.status, 'success', 'status is untouched'); + assert.strictEqual(row.payload_size, 100, 'payload_size is retained as metadata'); + assert.strictEqual(row.payload_blob, null, 'the tarball is reclaimed'); + // The drop is deliberately NOT appended to event_log: that would be a read-copy-write of an + // append-only list, so a concurrent writer's entry would be lost to reclaim a tarball. The null + // blob already reports the outcome on the row, and the reclaim is logged. + assert.deepStrictEqual( + row.event_log.filter((e) => e.event === 'payload_dropped'), + [], + 'and nothing is appended to the audit list' + ); + }); + + it('honors a higher count, keeping the N newest', async () => { + seed('d1', { startedAt: 100 }); + seed('d2', { startedAt: 200 }); + seed('d3', { startedAt: 300 }); + + await pruneProjectPayloads('app', 2); + + assert.deepStrictEqual( + ['d1', 'd2', 'd3'].map(hasPayload), + [false, true, true], + 'the two newest are kept, the oldest dropped' + ); + }); + + it('never drops staged or activating deployments whose blobs may still be the replication channel', async () => { + seed('newest', { startedAt: 300 }); + seed('in-flight', { startedAt: 200, status: 'activating' }); + seed('staged', { startedAt: 150, status: 'staged' }); + seed('old', { startedAt: 100 }); + + await pruneProjectPayloads('app', 1); + + assert.strictEqual(hasPayload('newest'), true, 'newest retained'); + assert.strictEqual(hasPayload('in-flight'), true, 'the in-flight deployment keeps its payload'); + assert.strictEqual(hasPayload('staged'), true, 'the staged deployment keeps its payload for later activation'); + assert.strictEqual(hasPayload('old'), false, 'the settled older one is still dropped'); + }); + + it('scopes pruning to the given project', async () => { + seed('a-new', { project: 'app-a', startedAt: 200 }); + seed('a-old', { project: 'app-a', startedAt: 100 }); + seed('b-old', { project: 'app-b', startedAt: 100 }); + + await pruneProjectPayloads('app-a', 1); + + assert.strictEqual(hasPayload('a-old'), false, "the other project's older payload is dropped"); + assert.strictEqual(hasPayload('b-old'), true, 'a different project is untouched'); + }); + + it('counts only rows that still hold a payload, so the cap is "at most N stored"', async () => { + // d3 is newest but already reclaimed (e.g. by the size-based drop), so it occupies no slot and the + // newest payload that DOES exist fills the single retained slot. + seed('d3', { startedAt: 300, payload: false }); + seed('d2', { startedAt: 200 }); + seed('d1', { startedAt: 100 }); + + await pruneProjectPayloads('app', 1); + + assert.strictEqual(hasPayload('d2'), true, 'the newest surviving payload is retained'); + assert.strictEqual(hasPayload('d1'), false, 'the older one is dropped'); + }); + + it('drops every payload at a count of 0, and is safe with no rows / bad input', async () => { + seed('only', { startedAt: 100 }); + assert.strictEqual(await pruneProjectPayloads('app', 0), 100, 'count 0 retains nothing'); + assert.strictEqual(hasPayload('only'), false); + + assert.strictEqual(await pruneProjectPayloads('no-such-project', 1), 0, 'unknown project frees nothing'); + assert.strictEqual(await pruneProjectPayloads('', 1), 0, 'a blank project is a no-op'); + assert.strictEqual(await pruneProjectPayloads('app', -1), 0, 'a negative count is rejected, not treated as 0'); + assert.strictEqual(await pruneProjectPayloads('app', NaN), 0, 'a non-finite count is rejected'); + }); +}); diff --git a/unitTests/components/extractApplicationSwap.test.js b/unitTests/components/extractApplicationSwap.test.js index 4081f81822..ee566638df 100644 --- a/unitTests/components/extractApplicationSwap.test.js +++ b/unitTests/components/extractApplicationSwap.test.js @@ -872,4 +872,21 @@ describe('extractApplication directory swap', () => { await fs.rm(componentsRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); await fs.rm(sourceDir, { recursive: true, force: true }); }); + + // componentLoader fails startup closed when this scan rejects, so the two outcomes have to stay + // distinguishable: an absent staging root is a fresh install with no work, while an unreadable one + // means some component may hold a half-extracted tree we could not see. + it('resolves empty when the staging root is absent but rejects when it is unreadable', async () => { + const componentsRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'extract-swap-scan-')); + + assert.strictEqual((await recoverInterruptedComponentExtractions(componentsRoot)).size, 0); + + await fs.writeFile(path.join(componentsRoot, '.deploy-aside'), 'not a directory\n'); + await assert.rejects( + () => recoverInterruptedComponentExtractions(componentsRoot), + /staging path is not a directory/ + ); + + await fs.rm(componentsRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + }); }); diff --git a/unitTests/server/fastifyRoutes/operations.test.js b/unitTests/server/fastifyRoutes/operations.test.js index c325ac6c40..944e66d0f5 100644 --- a/unitTests/server/fastifyRoutes/operations.test.js +++ b/unitTests/server/fastifyRoutes/operations.test.js @@ -151,6 +151,7 @@ describe('Test custom functions operations', () => { await fs.ensureFile(path.join(CF_DIR_ROOT, 'my-cool-component', '.hidden')); await fs.ensureFile(path.join(CF_DIR_ROOT, 'my-cool-component', 'utils', 'utils.js')); await fs.outputFile(path.join(CF_DIR_ROOT, 'my-other-component', 'config.yaml'), test_yaml_string); + await fs.ensureFile(path.join(CF_DIR_ROOT, '.deploy-activating', 'my-cool-component', '.new-deployment')); sandbox.stub(configUtils, 'getConfiguration').returns({ 'my-other-component': { package: '@my-org/my-other-component', @@ -182,6 +183,7 @@ describe('Test custom functions operations', () => { expect(otherComponent.urlPath).to.equal('/other'); expect(otherComponent.host).to.equal('other.example.com'); expect(otherComponent.loadComponent).to.equal('if-installed'); + expect(result.entries.find((e) => e.name === '.deploy-activating')).to.be.undefined; }); it('Test getComponents includes status information when component status exists', async () => { @@ -521,9 +523,13 @@ describe('Test custom functions operations', () => { // Mock addConfig to prevent actual file writes const addConfigStub = sandbox.stub(configUtils, 'addConfig').resolves(); - // Mock prepareApplication to prevent actual installation - const prepareApplicationStub = sandbox.stub(); - operations.__set__('prepareApplication', prepareApplicationStub); + // Mock the two-phase build/swap primitives to prevent actual install + filesystem work. + const stageApplicationStub = sandbox.stub().resolves('/tmp/staging'); + const activateApplicationStub = sandbox + .stub() + .callsFake(async (_application, _deploymentId, hooks) => hooks?.beforeCommit?.()); + operations.__set__('stageApplication', stageApplicationStub); + operations.__set__('activateStagedApplication', activateApplicationStub); // This should work - user components can be overwritten without force await operations.deployComponent({ @@ -531,13 +537,14 @@ describe('Test custom functions operations', () => { package: '@org/new-package', }); - // Verify addConfig was called + // Verify addConfig was called (at activation, once the bits are staged) expect(addConfigStub.calledOnce).to.be.true; expect(addConfigStub.firstCall.args[0]).to.equal('existing-component'); expect(addConfigStub.firstCall.args[1].package).to.equal('@org/new-package'); - // Verify prepareApplication was called - expect(prepareApplicationStub.calledOnce).to.be.true; + // Verify the component was staged then activated + expect(stageApplicationStub.calledOnce).to.be.true; + expect(activateApplicationStub.calledOnce).to.be.true; }); it('Test deployComponent allows deploying new component without force flag', async () => { @@ -547,9 +554,13 @@ describe('Test custom functions operations', () => { // Mock addConfig to prevent actual file writes const addConfigStub = sandbox.stub(configUtils, 'addConfig').resolves(); - // Mock prepareApplication to prevent actual installation - const prepareApplicationStub = sandbox.stub(); - operations.__set__('prepareApplication', prepareApplicationStub); + // Mock the two-phase build/swap primitives to prevent actual install + filesystem work. + const stageApplicationStub = sandbox.stub().resolves('/tmp/staging'); + const activateApplicationStub = sandbox + .stub() + .callsFake(async (_application, _deploymentId, hooks) => hooks?.beforeCommit?.()); + operations.__set__('stageApplication', stageApplicationStub); + operations.__set__('activateStagedApplication', activateApplicationStub); // This should work fine - no component exists yet await operations.deployComponent({ @@ -561,8 +572,9 @@ describe('Test custom functions operations', () => { expect(addConfigStub.calledOnce).to.be.true; expect(addConfigStub.firstCall.args[0]).to.equal('new-component'); - // Verify prepareApplication was called - expect(prepareApplicationStub.calledOnce).to.be.true; + // Verify the component was staged then activated + expect(stageApplicationStub.calledOnce).to.be.true; + expect(activateApplicationStub.calledOnce).to.be.true; }); it('Test deployComponent prevents overwriting core component without force flag', async () => { @@ -593,9 +605,13 @@ describe('Test custom functions operations', () => { // Mock addConfig to prevent actual file writes const addConfigStub = sandbox.stub(configUtils, 'addConfig').resolves(); - // Mock prepareApplication to prevent actual installation - const prepareApplicationStub = sandbox.stub(); - operations.__set__('prepareApplication', prepareApplicationStub); + // Mock the two-phase build/swap primitives to prevent actual install + filesystem work. + const stageApplicationStub = sandbox.stub().resolves('/tmp/staging'); + const activateApplicationStub = sandbox + .stub() + .callsFake(async (_application, _deploymentId, hooks) => hooks?.beforeCommit?.()); + operations.__set__('stageApplication', stageApplicationStub); + operations.__set__('activateStagedApplication', activateApplicationStub); // This should NOT throw an error because force is true await operations.deployComponent({ @@ -609,8 +625,9 @@ describe('Test custom functions operations', () => { expect(addConfigStub.firstCall.args[0]).to.equal('graphql'); expect(addConfigStub.firstCall.args[1].package).to.equal('@org/override-package'); - // Verify prepareApplication was called - expect(prepareApplicationStub.calledOnce).to.be.true; + // Verify the component was staged then activated + expect(stageApplicationStub.calledOnce).to.be.true; + expect(activateApplicationStub.calledOnce).to.be.true; }); it('Test deployComponent prevents overwriting multiple core component names', async () => { diff --git a/unitTests/server/serverHelpers/serverUtilities.test.js b/unitTests/server/serverHelpers/serverUtilities.test.js index 5a6a4b4007..194729a75f 100644 --- a/unitTests/server/serverHelpers/serverUtilities.test.js +++ b/unitTests/server/serverHelpers/serverUtilities.test.js @@ -751,9 +751,11 @@ describe('Test serverUtilities.js module ', () => { describe('registerOperation permission seam', function () { const { server } = require('#src/server/Server'); const op_auth = require('#src/utility/operation_authorization'); + const { isOperationAuthorizationBypassed } = require('#src/server/serverHelpers/operationAuthorizationState'); const { validateOperations } = require('#src/utility/operationPermissions'); const SU_OP = 'test_registered_su_op'; const OPEN_OP = 'test_registered_open_op'; + const AUTH_STATE_OP = 'test_internal_authorization_state'; // A non-super_user request JSON for a given op, optionally carrying an `operations` grant. const nonSuRequest = (op, operations) => ({ @@ -800,7 +802,15 @@ describe('Test serverUtilities.js module ', () => { // Keep the process-global registries clean — these test-only ops shouldn't leak into other // suites. registerOperation touches three globals (the op-function map plus verifyPerms' // requiredPermissions and the grantable-ops set), so undo all three, not just the map. - for (const op of [SU_OP, OPEN_OP, 'test_name_pinning_op', 'shared_op_a', 'shared_op_b', 'dyn_grantable_op']) { + for (const op of [ + SU_OP, + OPEN_OP, + AUTH_STATE_OP, + 'test_name_pinning_op', + 'shared_op_a', + 'shared_op_b', + 'dyn_grantable_op', + ]) { serverUtilities.OPERATION_FUNCTION_MAP.delete(op); op_auth.unregisterOperationPermission(op); } @@ -864,6 +874,19 @@ describe('Test serverUtilities.js module ', () => { it('allows a non-super_user whose role grants the op via the operations allowlist (SU-bypass)', function () { assert.equal(op_auth.verifyPerms(nonSuRequest(SU_OP, [SU_OP]), SU_OP), null); }); + + it('exposes trusted authorization bypass through server.operation without leaking it afterward', async function () { + server.registerOperation({ + name: AUTH_STATE_OP, + execute: async () => ({ bypassed: isOperationAuthorizationBypassed() }), + requiresSuperUser: true, + }); + + const result = await server.operation({ operation: AUTH_STATE_OP }, { user: { name: 'cluster-peer' } }, false); + + assert.deepEqual(result, { bypassed: true }); + assert.equal(isOperationAuthorizationBypassed(), false); + }); }); // #1809 — process-wide server.* registrations must not leak from a throwaway deploy-validation load. diff --git a/utility/hdbTerms.ts b/utility/hdbTerms.ts index 163f6557b6..e83587715b 100644 --- a/utility/hdbTerms.ts +++ b/utility/hdbTerms.ts @@ -324,6 +324,8 @@ export const OPERATIONS_ENUM = { DEPLOY_CUSTOM_FUNCTION_PROJECT: 'deploy_custom_function_project', PACKAGE_COMPONENT: 'package_component', DEPLOY_COMPONENT: 'deploy_component', + // A public operation rather than a deploy phase: it fetches, resolves and installs nothing. + REVERT_COMPONENT: 'revert_component', READ_TRANSACTION_LOG: 'read_transaction_log', DELETE_TRANSACTION_LOGS_BEFORE: 'delete_transaction_logs_before', INSTALL_NODE_MODULES: 'install_node_modules', @@ -591,6 +593,11 @@ export const CONFIG_PARAMS = { OPERATIONSAPI_NETWORK_MAXREQUESTBODYSIZE: 'operationsApi_network_maxRequestBodySize', OPERATIONSAPI_COMPONENTFILE_MAXSIZE: 'operationsApi_componentFile_maxSize', DEPLOYMENT_PAYLOADRETENTION_MAXSIZE: 'deployment_payloadRetention_maxSize', + // Bounds retained payload tarballs per project. Only the blob is dropped — rows are always kept, so + // the audit trail survives. + DEPLOYMENT_PAYLOADRETENTION_MAXCOUNT: 'deployment_payloadRetention_maxCount', + // Bounds not-yet-activated staged builds per component. + DEPLOYMENT_STAGINGRETENTION_MAXCOUNT: 'deployment_stagingRetention_maxCount', OPERATIONSAPI_TLS: 'operationsApi_tls', OPERATIONSAPI_TLS_CERTIFICATE: 'operationsApi_tls_certificate', OPERATIONSAPI_TLS_PRIVATEKEY: 'operationsApi_tls_privateKey', diff --git a/utility/operation_authorization.ts b/utility/operation_authorization.ts index 617b4eb1c1..538030ffcf 100644 --- a/utility/operation_authorization.ts +++ b/utility/operation_authorization.ts @@ -313,6 +313,10 @@ requiredPermissions.set(functionsOperations.addComponent.name, new (permission a requiredPermissions.set(functionsOperations.dropCustomFunctionProject.name, new (permission as any)(true, [])); requiredPermissions.set(functionsOperations.packageComponent.name, new (permission as any)(true, [])); requiredPermissions.set(functionsOperations.deployComponent.name, new (permission as any)(true, [])); +requiredPermissions.set( + functionsOperations.revertComponent.name, + new (permission as any)(true, [], terms.OPERATIONS_ENUM.REVERT_COMPONENT) +); requiredPermissions.set( deploymentOperations.handleListDeployments.name, new (permission as any)(true, [], terms.OPERATIONS_ENUM.LIST_DEPLOYMENTS)