Skip to content

fix(upgrade): stage runtime work before cutover - #78

Merged
fuller-stack-dev merged 1 commit into
openclaw:mainfrom
fuller-stack-dev:fix/stage-upgrade-cutover
Aug 14, 2026
Merged

fix(upgrade): stage runtime work before cutover#78
fuller-stack-dev merged 1 commit into
openclaw:mainfrom
fuller-stack-dev:fix/stage-upgrade-cutover

Conversation

@fuller-stack-dev

Copy link
Copy Markdown
Member

Summary

  • stage target runtime installation privately while the current Gateway remains healthy
  • run snapshot preflight and immutable runtime rollback backup before service quiescence
  • keep complete snapshot capture, staged runtime publication, finalization, restart verification, and rollback inside the cold cutover
  • fail preparation before cutover without changing the environment, binding, secrets, service policy, snapshot history, or published runtime

This intentionally does not change delayed stop acknowledgement, desired-versus-observed start state, health convergence, or false-rollback behavior. Those remain in the separate convergence owner lane.

Canonical transaction boundary

Pre-cutover now contains target resolution/staging, complete snapshot preflight, and runtime rollback backup. Cutover begins immediately before source service quiescence, then captures the complete rollback snapshot, atomically publishes the staged runtime, performs config repair/finalization, restores the requested service policy, and verifies health/RPC readiness. Any failure after publication retains the existing fail-closed rollback path.

Finalization stays cold because it can mutate OpenClaw config, SQLite state, and plugins. Snapshot capture stays after quiescence for point-in-time consistency.

Availability evidence

Same deterministic fixture: 1.5s target preparation delay, 1.5s finalization delay, continuous health polling.

Untouched canonical main at 614f6cb4b9c40d70342aecb1af01286feff613fb:

  • service stopped: 3.948s
  • target preparation: 4.382-5.957s
  • finalization: 6.218-7.777s
  • target ready: 7.831s
  • stop-to-ready: 3.883s
  • result: red; source Gateway was unavailable during target preparation

This head at 739cac55cb1610177652093c2ce785e5d9371a17:

  • target preparation: 5.021-6.522s
  • service stopped: 8.047s
  • finalization: 8.966-10.541s
  • target ready: 10.580s
  • stop-to-ready: 2.533s
  • result: green; source health remained continuous through preparation

The deterministic cutover interval drops by 1.350s (~35%). Absolute host setup time varies; the contract asserts phase ordering and continuous health rather than a flaky wall-clock threshold.

Related work

PR #60 overlaps some upgrade/runtime files but combines broader gateway lifecycle and runtime changes. This PR keeps the transaction-ordering fix isolated and does not absorb the separate convergence workstream.

Validation

  • cargo fmt --all -- --check
  • cargo test --test upgrade_availability_tests -- --nocapture
  • cargo test --test upgrade_command_tests -- --test-threads=1 (46 passed)
  • cargo test --test runtime_command_tests -- --test-threads=1 (46 passed, including current-main source-extension coverage)
  • cargo test --lib -- --test-threads=1 (296 passed)
  • cargo clippy --all-targets -- -D warnings -A clippy::question_mark -A clippy::large_enum_variant -A clippy::too_many_arguments
  • source-blind behavior validation: 6/6 contract clauses passed, no blockers
  • autoreview: clean, no accepted/actionable findings; TruffleHog clean

@fuller-stack-dev
fuller-stack-dev merged commit 381a5ed into openclaw:main Aug 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant