Skip to content

fix: harden gateway restart, cleanup, and child temp dirs - #103

Merged
hannesrudolph merged 7 commits into
openclaw:mainfrom
hannesrudolph:fix/gateway-lifecycle-cleanup-tmpdir
Aug 21, 2026
Merged

fix: harden gateway restart, cleanup, and child temp dirs#103
hannesrudolph merged 7 commits into
openclaw:mainfrom
hannesrudolph:fix/gateway-lifecycle-cleanup-tmpdir

Conversation

@hannesrudolph

@hannesrudolph hannesrudolph commented Aug 21, 2026

Copy link
Copy Markdown
Member

Related: #60

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

ocm service restart directly replaces a supervised gateway even when OpenClaw advertises restart-handoff support, so eligible active sessions and subagents can be interrupted without receiving OpenClaw's recovery marker. A restart issued from inside the target gateway can also wait on its own replacement.

Environment removal and service uninstall currently reconcile the complete supervisor registry. If sibling runtime metadata or inherited process environment has drifted, deleting one environment can rewrite or restart unrelated gateways. Exited children can leave descendants or an unreaped process-group leader behind, and persisted macOS per-login TMPDIR values can become invalid before a later gateway spawn.

Why This Change Was Made

Use OpenClaw restart-handoff protocol v1 for normal service restarts, return immediately for a self-restart, retain a warned direct-restart fallback for legacy bindings, and expose --force as the explicit recovery bypass.

Make uninstall, guarded destroy, direct removal, and prune reconcile only the affected environment. Clean the complete child process group with TERM, bounded polling, KILL fallback, and an unconditional leader wait. Stop persisting TMPDIR; each child spawn receives a private daemon-scoped mode-0700 directory derived from the daemon's current temp base, with a safe fallback when that base is unavailable.

Local companion packaging and named Tailscale Service ingress changes are intentionally outside this PR.

User Impact

Gateway restarts preserve OpenClaw's eligible recovery behavior instead of blindly interrupting active work. Removing or stopping one environment leaves sibling gateway definitions and PIDs unchanged despite unrelated drift. Gateway descendants are cleaned up reliably, and future child spawns no longer inherit stale temporary-directory paths after login, reboot, or daemon replacement.

Legacy gateways remain restartable through the existing direct supervisor path and receive an explicit warning; operators can deliberately select that path with --force.

Evidence

  • Isolated split branch based directly on current main; it contains no companion-packaging or Tailscale ingress changes.
  • cargo fmt --all -- --check: passed.
  • git diff --check origin/main..HEAD: passed.
  • cargo test --locked --test daemon_runtime_tests --test env_destroy_tests --test service_command_tests --test upgrade_command_tests: 126 passed, 0 failed.
  • Hosted CI passed on macOS, Ubuntu, Windows, formatting, and the Rust 1.88 minimum lane.
  • The original umbrella head also passed 183 focused tests and its complete hosted CI matrix after rebasing.
  • No OCM binary was installed locally, and no resident daemon or managed gateway was restarted.

@clawsweeper

clawsweeper Bot commented Aug 21, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 21, 2026
@clawsweeper

clawsweeper Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 21, 2026, 3:57 PM ET / 19:57 UTC.

ClawSweeper review

What this changes

The PR makes gateway restarts recovery-aware, scopes cleanup to the selected environment, reaps child process groups, and creates fresh private temporary directories for supervised children.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

The re-review resolves the prior activated-shell blocker: readiness is skipped only for the target gateway identity, while an activated shell still waits for replacement and health. No new actionable patch defect was found; this member-authored PR should remain open for normal maintainer review.

Priority: P1
Reviewed head: 7f924cb8f3e9beed5c40509b5619b905ef1fdc22

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The focused patch resolves the prior blocker and has relevant regression coverage with no remaining discrete correctness finding.
Proof confidence 🌊 off-meta tidepool Not applicable: The author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the branch does provide focused source-level integration coverage.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the branch does provide focused source-level integration coverage.
Evidence reviewed 6 items Activated-shell fix: The self-restart exemption now requires both the matching active environment and the gateway service-kind marker, preventing an ordinary activated shell from skipping readiness.
Readiness remains on the normal CLI path: The service wrapper applies gateway readiness unless the dual gateway-origin condition is true.
Activated-shell regression coverage: The daemon integration test sets only OCM_ACTIVE_ENV before restart and asserts gatewayReady and gatewayState after replacement.
Findings None None.
Security None None.

Live Verification

Command: cargo run --locked -- service restart --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

  • FAIL expect_output: --force

How this fits together

OCM manages OpenClaw gateway processes for named environments. Lifecycle commands update the supervisor, which replaces gateway children and returns readiness information to operators.

flowchart LR
  Operator[Operator lifecycle command] --> Service[OCM service lifecycle]
  Service --> Mode{Restart mode}
  Mode -->|Protocol handoff| Gateway[OpenClaw gateway]
  Mode -->|Legacy or forced| Supervisor[OCM supervisor]
  Gateway --> Child[Replacement gateway child]
  Supervisor --> Child
  Child --> Ready[Readiness result]
Loading

Before merge

  • Resolve merge risk (P1) - Protocol-v1 restarts now replace an active gateway immediately; eligible work depends on OpenClaw's recovery contract, while legacy and forced paths intentionally retain direct replacement.
  • Resolve merge risk (P1) - Targeted cleanup preserves sibling child specifications during unrelated registry drift, changing when operators observe reconciliation of that drift.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test delta production/docs +507/-50; tests +465/-21 The 13-file lifecycle change includes substantial focused regression coverage across three test files.

Merge-risk options

Maintainer options:

  1. Merge with the fallback paths intact (recommended)
    After required checks complete, accept the protocol-v1 restart behavior while preserving the tested legacy direct restart and explicit force recovery path.

Technical review

Best possible solution:

Land the recovery-aware restart path with its legacy fallback and explicit force escape hatch, while retaining the activated-shell and process-cleanup regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes: the checked-in daemon integration test starts a local supervisor, restarts with only OCM_ACTIVE_ENV set, and requires gateway readiness after replacement; the self-restart fixture covers the gateway-marker branch.

Is this the best way to solve the issue?

Yes: reusing the existing protocol-v1 gateway marker is the narrowest fix because normal activation removes that marker, so only an actual supervised gateway bypasses its own readiness wait.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 768714ae3491.

Labels

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the branch does provide focused source-level integration coverage.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P1: Gateway restart and cleanup behavior can interrupt active operator workflows when it regresses.
  • merge-risk: 🚨 compatibility: The new default restart path must continue to work for legacy bindings and ordinary activated shells.
  • merge-risk: 🚨 availability: Gateway replacement and process-group cleanup can affect service availability if their lifecycle assumptions fail.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a repository MEMBER, so the external-contributor real-behavior-proof gate does not apply; the branch does provide focused source-level integration coverage.

Evidence

What I checked:

  • Activated-shell fix: The self-restart exemption now requires both the matching active environment and the gateway service-kind marker, preventing an ordinary activated shell from skipping readiness. (src/service/manage.rs:74, 7f924cb8f3e9)
  • Readiness remains on the normal CLI path: The service wrapper applies gateway readiness unless the dual gateway-origin condition is true. (src/service/mod.rs:231, 7f924cb8f3e9)
  • Activated-shell regression coverage: The daemon integration test sets only OCM_ACTIVE_ENV before restart and asserts gatewayReady and gatewayState after replacement. (tests/daemon_runtime_tests.rs:1508, 7f924cb8f3e9)
  • Gateway identity boundary: Current main applies OPENCLAW_SERVICE_KIND=gateway only for protocol-v1 external supervision, while the ordinary supervision hint removes that marker. (src/infra/shell.rs:121, 768714ae3491)
  • Process cleanup implementation: Stopped and exited children now receive group-wide TERM/KILL handling followed by an unconditional leader wait; focused Unix tests cover remaining descendants and leader reaping. (src/supervisor/mod.rs:1881, 7f924cb8f3e9)
  • Feature provenance: The current restart-handoff surface traces to the existing external-handoff implementation, with later async-supervisor maintenance in the same area. (src/supervisor/openclaw_handoff.rs:139, 9d4716fde323)

Likely related people:

  • Shakker: Git history identifies this contributor with the external restart-handoff implementation and the current protocol-v1 identity boundary. (role: feature introducer; confidence: high; commits: 9d4716fde323; files: src/supervisor/openclaw_handoff.rs, src/infra/shell.rs)
  • Jason (Json): Recent main history includes asynchronous supervisor transition maintenance near the lifecycle behavior this PR changes. (role: recent adjacent contributor; confidence: medium; commits: 5d8ef1184f34; files: src/supervisor/mod.rs, src/service/manage.rs)
  • Hannes Rudolph: This PR and prior merged service-identity work both touch the gateway supervision boundary. (role: recent area contributor; confidence: high; commits: 51aa84f4d574, 7f924cb8f3e9; files: src/service/manage.rs, src/service/mod.rs)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-08-21T19:11:16.314Z sha 1f4969b :: needs changes before merge. :: [P1] Keep readiness checks for restarts from an activated shell
  • reviewed 2026-08-21T19:20:46.226Z sha 1f4969b :: needs changes before merge. :: [P1] Keep readiness checks for restarts from an activated shell

@hannesrudolph

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

I accepted the activated-shell finding and pushed 7f924cb.

Change selected

The self-restart exemption now requires both:

  • OCM_ACTIVE_ENV matching the target environment; and
  • OPENCLAW_SERVICE_KIND=gateway.

I used the existing service-kind marker instead of adding another OCM-only variable. OCM_ACTIVE_ENV is deliberately exported by ocm use, while OPENCLAW_SERVICE_KIND=gateway is applied only to a protocol-v1 supervised gateway child. Activated shells therefore continue through replacement observation and readiness verification, while a real gateway-originated restart still returns immediately and avoids waiting on itself.

Coverage

  • The true self-restart fixture now carries the gateway service-kind marker and still proves the no-wait handoff path.
  • The readiness integration test now performs restart with only OCM_ACTIVE_ENV set, reproducing an activated shell and requiring gatewayReady: true after replacement.
  • Full focused branch validation: 126 passed, 0 failed across daemon runtime, environment destroy, service command, and upgrade command suites.
  • Formatting and git diff --check passed.

I did not decline any part of the actionable review advice; the only implementation choice was to reuse the existing protocol-v1 gateway identity instead of creating a new marker.

@clawsweeper

clawsweeper Bot commented Aug 21, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 21, 2026
@hannesrudolph
hannesrudolph force-pushed the fix/gateway-lifecycle-cleanup-tmpdir branch from 7f924cb to 1c1a177 Compare August 21, 2026 22:03
@clawsweeper

clawsweeper Bot commented Aug 21, 2026

Copy link
Copy Markdown

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix: harden gateway restart, cleanup, and child temp dirs This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@hannesrudolph
hannesrudolph merged commit 9cc226e into openclaw:main Aug 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant