Split out of #1849, which adds two-phase (stage → activate) component deploys.
For a package: deploy the origin replicates the package identifier, not bytes. Confirmed in that
branch: peers wait on the deployment row with requirePayload: !spec.package
(components/operations.js), so a package deploy deliberately has no shared payload and every node
independently resolves and packs the reference.
The failure
A mutable reference can resolve differently per node:
- a moving tag such as
latest
- a semver range
- a git branch
Each node then stages a component and reports success, so the barrier passes and activation
proceeds — while the nodes are running different code. The barrier confirms "everyone staged
something", not "everyone staged the same thing".
This is silent: every stage succeeds, the deployment rows all look healthy, and nothing compares what
was produced.
DESIGN.md is currently accurate but narrow — it scopes the barrier's guarantee to "fetch + install,
not load" and makes no claim about byte equality. So this is a missing guarantee rather than a
contradiction of documented intent.
What a fix requires
Resolve once, on the origin:
- Origin resolves and packs the package, as it already does for its own stage.
- Persist the exact tarball plus a digest on the deployment row.
- Peers stage those bytes and verify the digest instead of resolving independently.
Two consequences worth deciding deliberately:
- It changes what the barrier guarantees — from "each node installed something" to "every node
installed these bytes".
- It makes a
package: deploy carry a payload it currently does not, with the storage and payload
retention cost that implies (deployment_payloadRetention_maxCount then applies to package
deploys too).
Notes
Split out of #1849, which adds two-phase (stage → activate) component deploys.
For a
package:deploy the origin replicates the package identifier, not bytes. Confirmed in thatbranch: peers wait on the deployment row with
requirePayload: !spec.package(
components/operations.js), so a package deploy deliberately has no shared payload and every nodeindependently resolves and packs the reference.
The failure
A mutable reference can resolve differently per node:
latestEach node then stages a component and reports success, so the barrier passes and activation
proceeds — while the nodes are running different code. The barrier confirms "everyone staged
something", not "everyone staged the same thing".
This is silent: every stage succeeds, the deployment rows all look healthy, and nothing compares what
was produced.
DESIGN.md is currently accurate but narrow — it scopes the barrier's guarantee to "fetch + install,
not load" and makes no claim about byte equality. So this is a missing guarantee rather than a
contradiction of documented intent.
What a fix requires
Resolve once, on the origin:
Two consequences worth deciding deliberately:
installed these bytes".
package:deploy carry a payload it currently does not, with the storage and payloadretention cost that implies (
deployment_payloadRetention_maxCountthen applies to packagedeploys too).
Notes
that is what keeps this contained relative to Two-phase deploy: concurrent activations from different origins can leave the cluster on different versions with both reporting success #2294.