Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@

## Unreleased

### Live daemon registry recovery

- `pty recover-live --metadata <snapshot.json> <name>` lets the original live
daemon rebind an accidentally unlinked pathname socket and republish its
pid/metadata sidecars without restarting the daemon, its child, or existing
clients. Recovery validates PID, generation, OS process-start identity, and
launch identity; it refuses conflicting registry owners.
- Recovery uses a short-lived authenticated request watched by the daemon; it
sends no process signal. Supporting daemons explicitly stamp
`recoveryProtocol: 1` and `daemonStartToken`; an unsupported snapshot fails
closed with transcript/existing-attachment guidance.

### Read-only session listing

- `listSessions()` and `pty list` are now strictly observational: they no
Expand Down Expand Up @@ -45,9 +57,12 @@

### Storage format

`<name>.json` gains optional `generation` and `daemonPid` lifecycle fields.
`<name>.json` gains optional `generation`, `daemonPid`, `recoveryProtocol`, and
`daemonStartToken` lifecycle fields.
The generation is an opaque cleanup-ownership token; the daemon PID lets
`pty rm` wait for deferred shutdown after the child has exited.
`pty rm` wait for deferred shutdown after the child has exited. The recovery
fields prove the daemon can safely handle a live rebind request and bind the
snapshot to the same OS process start.

### Restartable launch parity and bounded fleet listing

Expand Down
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,22 @@ sessions — see [Auto-running gc](#auto-running-gc). `pty list` is strictly
observational and never removes registry state. `keep=true` and
`strategy=permanent` are exempt from the gc sweep.

If a live daemon's socket, pid, and metadata path are accidentally unlinked,
do not rerun its launch command: that can create a second writer while the
original child is still alive. A metadata snapshot captured from a supporting
daemon can ask that exact process to rebind its listener without restarting:

```sh
pty recover-live --metadata ./backups/myserver.json myserver
```

Recovery validates the daemon PID, generation, process-start identity, name,
and launch definition. It refuses snapshots from older daemons, leaves a
foreign replacement socket untouched, and preserves existing attached clients.
An already-running daemon that predates this protocol cannot be upgraded
in-place; keep it alive and use an existing attachment or provider transcript
fallback.

### Events

Sessions automatically log terminal events — bell, title changes, desktop notifications (OSC 9/99/777), focus requests, and cursor visibility transitions — plus metadata mutations: `display_name_change` on rename, `tags_change` on tag updates, and any `user.*` events published via `pty emit`. Everything goes into per-session JSONL files.
Expand Down
5 changes: 4 additions & 1 deletion completions/pty.bash
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ _pty() {
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
commands="run attach a exec peek send events list ls stats restart kill rm remove gc tag tag-multi emit rename up down test remote-serve"
commands="run attach a exec peek send events list ls stats restart recover-live kill rm remove gc tag tag-multi emit rename up down test remote-serve"

if [[ ${COMP_CWORD} -eq 1 ]]; then
if [[ "${cur}" == -* ]]; then
Expand Down Expand Up @@ -74,6 +74,9 @@ _pty() {
COMPREPLY=($(compgen -W "${names}" -- "${cur}"))
fi
;;
recover-live)
COMPREPLY=($(compgen -W "--metadata --timeout-ms" -- "${cur}"))
;;
kill)
if [[ "${cur}" == -* ]]; then
COMPREPLY=($(compgen -W "" -- "${cur}"))
Expand Down
3 changes: 3 additions & 0 deletions completions/pty.fish
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ complete -c pty -n __pty_needs_command -a list -d 'List sessions'
complete -c pty -n __pty_needs_command -a ls -d 'List sessions'
complete -c pty -n __pty_needs_command -a stats -d 'Live CPU / memory / PIDs'
complete -c pty -n __pty_needs_command -a restart -d 'SIGTERM + respawn'
complete -c pty -n __pty_needs_command -a recover-live -d 'Rebind a stranded live daemon without restarting it'
complete -c pty -n __pty_needs_command -a kill -d 'SIGTERM a running session'
complete -c pty -n __pty_needs_command -a rm -d 'Remove exited metadata'
complete -c pty -n __pty_needs_command -a remove -d 'Remove exited metadata'
Expand Down Expand Up @@ -113,6 +114,8 @@ complete -c pty -n '__pty_using_command stats' -a '(__pty_sessions)' -d 'Session
complete -c pty -n '__pty_using_command restart' -l yes -s y -d 'Skip confirmation'
complete -c pty -n '__pty_using_command restart' -l force -d 'Attach after restart even from inside another pty'
complete -c pty -n '__pty_using_command restart' -a '(__pty_sessions)' -d 'Session'
complete -c pty -n '__pty_using_command recover-live' -l metadata -d 'Captured live metadata snapshot'
complete -c pty -n '__pty_using_command recover-live' -l timeout-ms -d 'Recovery timeout in milliseconds'
complete -c pty -n '__pty_using_command kill' -a '(__pty_sessions)' -d 'Session'
complete -c pty -n '__pty_using_command rm remove' -a '(__pty_sessions)' -d 'Session'
complete -c pty -n '__pty_using_command gc' -l dry-run -s n -d 'Preview without changing anything'
Expand Down
6 changes: 6 additions & 0 deletions completions/pty.zsh
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ _pty() {
'ls:Alias for list'
'stats:Live CPU / memory / PIDs'
'restart:SIGTERM + respawn'
'recover-live:Rebind a stranded live daemon without restarting it'
'kill:SIGTERM a running session'
'rm:Remove exited metadata'
'remove:Alias for rm'
Expand Down Expand Up @@ -128,6 +129,11 @@ _pty() {
'--force[Attach after restart even from inside another pty]' \
'1:session:_pty_sessions'
;;
recover-live)
_arguments \
'--metadata[Captured live metadata snapshot]' \
'--timeout-ms[Recovery timeout in milliseconds]'
;;
kill)
_arguments \
'1:session:_pty_sessions'
Expand Down
8 changes: 8 additions & 0 deletions docs/disk-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ For non-Node tools that want to read pty's state without paying Node startup. Th
| `<name>.sock` | daemon IPC socket (Unix) | 2 |
| `<name>.pid` | daemon pid (decimal) | 2 |
| `<name>.lock` | creation-race lock | 2 |
| `<name>.recover-request.<nonce>` | authenticated one-shot live-daemon recovery request | 2 |
| `theme` | last-selected TUI theme | 2 |
| `gc.log` | stdout/stderr of `pty gc` when run by launchd/cron (only present after auto-running gc is installed) | 2 |
| `<name>.json.tmp.<pid>.<rand>` | atomic-write tmp — readers MUST ignore | n/a |
Expand All @@ -35,6 +36,8 @@ Pretty-printed JSON. Source of truth: `SessionMetadata` in `src/sessions.ts`.
{
generation?: string; // opaque daemon generation; guards cleanup ownership
daemonPid?: number; // daemon owning this generation, retained after child exit
recoveryProtocol?: 1; // daemon supports fail-closed live registry recovery
daemonStartToken?: string; // OS process-start identity, when available
command: string; // resolved binary path
args: string[];
displayCommand: string; // command as the user typed it
Expand All @@ -60,6 +63,11 @@ Pretty-printed JSON. Source of truth: `SessionMetadata` in `src/sessions.ts`.
removes files still owned by its generation, and `pty rm` waits for that
daemon to finish deferred shutdown before it reports success. Readers should
treat the generation token as opaque.
- `recoveryProtocol` and `daemonStartToken` are written together by daemons
that can safely handle `pty recover-live`. The CLI refuses recovery unless a
captured snapshot contains both fields and the live process still has the
same start token. Recovery uses a short-lived request file watched by that
daemon and sends no process signal.
- Reserved tag keys (`ptyfile*`, `strategy`, anything starting with `:`) are pty/tool-internal; hidden from `pty list` unless `--tags`.
- User-facing tags that drive pty behavior but are visible by default:
- `strategy=permanent` — `pty gc` respawns the session when its daemon exits (the historic supervisor's role; now stateless and run on a cron).
Expand Down
162 changes: 162 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import * as fs from "node:fs";
import * as os from "node:os";
import * as path from "node:path";
import * as net from "node:net";
import * as readline from "node:readline/promises";
import { spawnSync, execFileSync } from "node:child_process";
import { randomBytes } from "node:crypto";
Expand All @@ -27,9 +28,13 @@ import {
allRefs,
readMetadata,
readSessionPid,
readProcessStartToken,
writeMetadata,
atomicWriteFileSync,
getSessionDir,
getSocketPath,
createLiveRecoveryRequest,
removeLiveRecoveryRequest,
DEFAULT_SESSION_DIR,
type SessionInfo,
type SessionMetadata,
Expand Down Expand Up @@ -259,6 +264,18 @@ Examples:
pty restart myserver
pty restart -y myserver`,

"recover-live": `Usage: pty recover-live --metadata <snapshot.json> [--timeout-ms <ms>] <name>

Ask the original live daemon to republish a lost pathname socket and registry.
The daemon and its child are not restarted. The snapshot must have been
captured while the same daemon generation was live.

This command fails closed unless the snapshot proves that the daemon supports
the recovery protocol; older daemons are never poked or restarted.

Examples:
pty recover-live --metadata ./myserver.json myserver`,

kill: `Usage: pty kill <ref>

SIGTERM a running session's daemon. Metadata is kept — restart or \`pty rm\` it later.
Expand Down Expand Up @@ -465,6 +482,7 @@ Modify:
pty emit <ref> user.<type> [...] Same, targeting a specific session

Lifecycle:
pty recover-live --metadata <file> <name> Rebind a stranded live daemon without restarting it
pty restart <ref> SIGTERM + respawn using stored metadata (prompts if running)
pty restart -y <ref> Same, no prompt
pty kill <ref> SIGTERM a running session's daemon
Expand Down Expand Up @@ -1267,6 +1285,35 @@ async function main(): Promise<void> {
break;
}

case "recover-live": {
let metadataPath: string | null = null;
let recoveryName: string | null = null;
let timeoutMs = 5000;
for (let ai = 1; ai < args.length; ai++) {
const a = args[ai];
if (a === "--metadata" && ai + 1 < args.length) {
metadataPath = args[++ai];
} else if (a === "--timeout-ms" && ai + 1 < args.length) {
timeoutMs = Number(args[++ai]);
} else if (!recoveryName) {
recoveryName = a;
} else {
console.error(`pty recover-live: unexpected argument "${a}"`);
process.exit(1);
}
}
if (!metadataPath || !recoveryName) {
console.error("Usage: pty recover-live --metadata <snapshot.json> [--timeout-ms <ms>] <name>");
process.exit(1);
}
if (!Number.isFinite(timeoutMs) || timeoutMs < 100 || timeoutMs > 60_000) {
console.error("pty recover-live: --timeout-ms must be between 100 and 60000");
process.exit(1);
}
await cmdRecoverLive(recoveryName, metadataPath, timeoutMs);
break;
}

case "kill": {
if (args.length < 2) {
console.error("Usage: pty kill <name>");
Expand Down Expand Up @@ -2474,6 +2521,121 @@ function renameUsage(): void {
console.error(COMMAND_HELP.rename);
}

function probeLocalSocket(socketPath: string, timeoutMs = 250): Promise<boolean> {
return new Promise((resolve) => {
const socket = net.createConnection(socketPath);
const timer = setTimeout(() => {
socket.destroy();
resolve(false);
}, timeoutMs);
socket.on("connect", () => {
clearTimeout(timer);
socket.destroy();
resolve(true);
});
socket.on("error", () => {
clearTimeout(timer);
resolve(false);
});
});
}

async function cmdRecoverLive(
name: string,
metadataPath: string,
timeoutMs: number,
): Promise<void> {
validateName(name);

let snapshot: SessionMetadata;
try {
const parsed: unknown = JSON.parse(fs.readFileSync(metadataPath, "utf-8"));
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
throw new Error("snapshot must be a JSON object");
}
snapshot = parsed as SessionMetadata;
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
throw new Error(`Cannot read recovery snapshot "${metadataPath}": ${message}`);
}

// This marker is written only after the supporting daemon has installed its
// request watcher. Its absence is a hard stop: old daemons cannot be upgraded
// in place, and recovery must never infer permission to relaunch them.
if (snapshot.recoveryProtocol !== 1) {
throw new Error(
"Snapshot predates live recovery support; refusing recovery. " +
"Keep the process alive and use an already-attached client or transcript fallback.",
);
}
if (
!Number.isInteger(snapshot.daemonPid) ||
Number(snapshot.daemonPid) <= 1 ||
typeof snapshot.generation !== "string" ||
snapshot.generation.length === 0
) {
throw new Error("Snapshot must contain a daemonPid and generation");
}
if (
typeof snapshot.daemonStartToken !== "string" ||
snapshot.daemonStartToken.length === 0
) {
throw new Error("Snapshot lacks a stable daemonStartToken; refusing recovery");
}

const daemonPid = Number(snapshot.daemonPid);
const observedStartToken = readProcessStartToken(daemonPid);
if (observedStartToken !== snapshot.daemonStartToken) {
throw new Error(
`Daemon PID ${daemonPid} no longer has the captured process-start identity`,
);
}
try {
process.kill(daemonPid, 0);
} catch {
throw new Error(`Daemon PID ${daemonPid} is not alive`);
}

const nonce = randomBytes(16).toString("hex");
createLiveRecoveryRequest({
protocol: 1,
name,
nonce,
createdAt: new Date().toISOString(),
expectedPid: daemonPid,
expectedGeneration: snapshot.generation,
expectedStartToken: snapshot.daemonStartToken,
snapshot,
});

try {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const metadata = readMetadata(name);
const pid = readSessionPid(name);
if (
metadata?.generation === snapshot.generation &&
metadata.daemonPid === daemonPid &&
metadata.daemonStartToken === snapshot.daemonStartToken &&
pid === daemonPid &&
await probeLocalSocket(getSocketPath(name))
) {
console.log(
`Session "${name}" recovered in daemon PID ${daemonPid} ` +
`(generation ${snapshot.generation}).`,
);
return;
}
await new Promise((resolve) => setTimeout(resolve, 50));
}
throw new Error(
`Live daemon ${daemonPid} did not republish session "${name}" within ${timeoutMs}ms`,
);
} finally {
removeLiveRecoveryRequest(name, nonce);
}
}

async function cmdRename(rawArgs: string[]): Promise<void> {
const insideSession = !!process.env.PTY_SESSION;

Expand Down
8 changes: 8 additions & 0 deletions src/completions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,14 @@ const COMMANDS: readonly CommandSpec[] = [
{ name: "force", desc: "Attach after restart even from inside another pty" },
],
},
{
name: "recover-live",
desc: "Rebind a stranded live daemon without restarting it",
flags: [
{ name: "metadata", desc: "Captured live metadata snapshot" },
{ name: "timeout-ms", desc: "Recovery timeout in milliseconds" },
],
},
{
name: "kill",
desc: "SIGTERM a running session",
Expand Down
Loading
Loading