Skip to content

Fix generation-safe launchd service restarts - #1059

Merged
dcellison merged 1 commit into
mainfrom
fix/generation-safe-service-restart
Aug 22, 2026
Merged

Fix generation-safe launchd service restarts#1059
dcellison merged 1 commit into
mainfrom
fix/generation-safe-service-restart

Conversation

@dcellison

Copy link
Copy Markdown
Owner

Summary

  • treat each macOS launchd wrapper and its descendants as one service generation
  • refuse to modify the installation until the verified prior generation exits, with bounded and group-scoped escalation
  • prevent a replacement launcher from adopting a listener owned by a stale generation
  • make the Python service drain when launchctl kickstart -k removes only its wrapper
  • require /health readiness from the newly registered launchd generation before reporting install success

Root cause

The launchd wrapper started Python in the background, then inferred the real Python PID from the process listening on the HTTP port. During shutdown, launchctl bootout could unregister the wrapper before the draining Python process exited. A replacement wrapper could then observe the old listener, adopt its PID, and report a stale generation as the new service while the actual replacement failed on held resources such as Qdrant or the HTTP port.

Safety properties

  • launchd's reported wrapper PID is accepted for stop escalation only when it is also the process-group leader
  • stop escalation targets only that verified process group
  • recorded generation IDs must be numeric and greater than 1
  • listener discovery accepts only a PID in the current wrapper's process group
  • launcher shutdown and predecessor waits are bounded
  • health readiness must identify the exact newly registered launchd generation
  • a failed stop aborts before installation changes begin

Validation

  • make check
  • make typecheck
  • .venv/bin/python -m pytest -q
    • 6,121 passed, 1 skipped
  • generated launcher validated with Bash syntax checking
  • behavioral launcher tests cover pre-bind termination and refusal to adopt a foreign listener

Installed qualification after merge

  1. Run make install and confirm it reports service readiness, not merely launchd registration.
  2. Confirm make install-status reports the service loaded and all Workshop authority checks remain healthy.
  3. Send a normal Kai request and confirm the configured backend responds.
  4. Restart with sudo launchctl kickstart -k system/com.syrinx.kai.
  5. Confirm the browser reconnects, Kai responds after restart, and no interrupted-generation message appears for a new request.
  6. Verify only one live Kai service generation remains.

Closes #1058

@dcellison
dcellison merged commit 332c5ce into main Aug 22, 2026
2 checks passed
@dcellison
dcellison deleted the fix/generation-safe-service-restart branch August 22, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Install/restart can overlap with a draining Kai process

2 participants