This repository was archived by the owner on Jul 24, 2026. It is now read-only.
feat(restart): convoy restart — the SAFE restart (stop the host, agents survive, re-adopt) - #107
Draft
myobie wants to merge 2 commits into
Draft
feat(restart): convoy restart — the SAFE restart (stop the host, agents survive, re-adopt)#107myobie wants to merge 2 commits into
convoy restart — the SAFE restart (stop the host, agents survive, re-adopt)#107myobie wants to merge 2 commits into
Conversation
…e the FULL fleet after an outage Parking-recovery for the 2026-07-22 incident: a supervisor bring-up after a mass outage brought back only part of the fleet; the rest stayed PARKED from a prior supervisor's give-up and had to be hand-launched. Root cause: `strategy.status=flapping` + the fast-fail counter PERSIST to a session's tags (the on-disk supervision contract), so they outlive the supervisor that wrote them. An outage drives the cap to its limit → the agents park → and a fresh `convoy up`, reading those stale tags, hits classify's `isFlapping → skip` and never relaunches them. Fix: a foreground `convoy up` is a DELIBERATE bring-up — the operator gesture that says "restore the fleet" — so at startup it clears the park AND zeroes the fast-fail counter for permanent members (regardless of prior fail count), giving each a fresh cap budget. The cap still re-accrues tick-to-tick WITHIN this supervisor's watch (the real crash-loop protection). The `--once` shepherd cron does NOT un-park: it runs every few minutes, so un-parking there would relaunch a genuinely broken agent every tick — parking must stay durable for it. - flapping-cap.ts: `clearParkForFreshSupervisor` (pure) — the reset decision, unit-tested. - up.ts: the FRESH-SUPERVISOR UN-PARK startup pass (foreground only), which clears the on-disk park (removes the status tag — updateTags MERGES) and seeds the in-memory classifier state. - Tests: pure cases for the reset + a process-level proof (a parked, gone-but-recorded agent is UN-PARKED and RELAUNCHED by a fresh foreground up, while `--once` leaves it parked). Part 2 of 4 of the decoupling-hardening task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gbfntB6cu21sL4YBp21LF
…ents survive, re-adopt)
Part 3 of 4 of the decoupling-hardening task. `convoy restart` exists so nobody reaches for
`convoy down` + `convoy up` to restart a live network — `down` KILLS every agent (it is the only
teardown), which is the mass-outage footgun that started the 2026-07-22 incident.
`restart` instead STOPS the running host PROCESS with SIGTERM (agents keep running — the Nomad
decoupling: up's handler just sets `stop`, and its exit path leaves every session up), waits for it
to release the host lock (so a fresh up never refuses or double-hosts), then becomes a fresh
`convoy up` that RE-ADOPTS the still-running agents (reconcile skips live ones) and, via the
fresh-supervisor un-park, restores any parked members. With no host running it simply starts one.
- up.ts: `restart()` + `RestartOptions`.
- cli.ts: `cmdRestart` + dispatch + help line ("use this, never `down` + `up`").
- command-table.ts: the `restart` entry (keeps completions + dispatch in parity).
- Test: a process-level proof — a real agent supervised by host A survives a real `convoy restart`
at the SAME pid while the restart process takes over as the new host; plus the no-host-running case.
Stacks on #106 (the fresh-supervisor un-park) since restart's re-adopt reuses that up() path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gbfntB6cu21sL4YBp21LF
myobie
marked this pull request as draft
July 22, 2026 00:36
Collaborator
Author
|
On hold per CoS: parts 3 & 4 (convoy hardening) are paused pending Nathan's morning call on swap-to-st2-and-retire-convoy vs keep-hardening. This was already built + green before the hold landed; marking draft so it's not requesting review. Reopen/merge only if the decision is to keep hardening convoy. (Stacks on #106.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Part 3 of 4 of the decoupling-hardening task.
convoy restartexists so nobody reaches forconvoy down+convoy upto restart a live network —downkills every agent (it is the only teardown), which is the mass-outage footgun that started the 2026-07-22 incident.restart:SIGTERM— agents keep running (the Nomad decoupling: up's handler just setsstop, and its exit path leaves every session up; proven by test(up): the DECISIVE decoupling test — killingconvoy upmust not kill its agents #105).convoy upthat RE-ADOPTS the still-running agents (reconcile skips live ones) and, via the fresh-supervisor un-park (fix(up): a fresh foregroundconvoy upUN-PARKS its members — restore the FULL fleet after an outage #106), restores any parked members.With no host running, it simply starts one.
Changes
up.ts:restart()+RestartOptions.cli.ts:cmdRestart+ dispatch + a help line ("use this, neverdown+up").command-table.ts: therestartentry — keeps completions + dispatch in parity (the nix parity check).convoy restartat the same pid while the restart process takes over as the new host; plus the no-host-running case.Full suite green.
🤖 Generated with Claude Code
https://claude.ai/code/session_014gbfntB6cu21sL4YBp21LF