st2 up spawns the Node-based pty CLI once per supervisor tick: 23.5 spawns/s burn 1.32 of 32 cores in interpreter startup

[schickling-assistant]

st2 up <cell> --interval 1 observes runtime state by spawning the pty CLI once per tick. pty is a Node script, so every tick pays a full Node process startup to answer a question that is, underneath, a readdir of the pty state directory.

Measurement

32-core Linux host, 150 s window, 2026-08-27. Every spawn classified by executed program (argv[0] resolved through interpreters).

pty CLI spawns	23.5 / sec
CPU per invocation	~55 ms
Continuous cost	1.32 of 32 cores

Parents: 36 concurrent st2 up …/cell --interval 1 processes.
Argv shapes: pty list --json 14.1/s, pty run -d --force --id … 8.1/s. (Two further arms, pty --root list --json 1.8/s and pty --root stats --json 1.1/s, are not emitted by the st2 up supervisor loop: every --root st2 constructs is an st2 subcommand flag, never a pty one — the only --root argv builders in the tree are reconcile.rs:363 and eval_run.rs:2461, both st2 ding --root. They are excluded from the claim above. The likely source is a wrapper: the agent-policy pty shim runs pty --root … list --json --tags as a pre-flight lookup on every pty send.)

pty resolves to a Node script (…-pty-0.1.0/lib/pty/bin/pty, shebang …-nodejs-24.18.1/bin/node).

Decomposed on an 8-core arm64 laptop, mean wall ms per call, 40 records in PTY_ROOT:

true (fork/exec floor)                  2.7
sh stub: same readdir, no Node          9.2
node -e 0 (interpreter floor)          51.8
pty list --json (what st2 runs)       121.3

So the split is roughly: ~3 ms fork/exec, ~49 ms Node interpreter, ~70 ms pty's module graph, ~6 ms actual observation work. "Nearly all Node startup" is right, but it is two components, and the larger one is the module graph.

Mechanism — one supervisor tick

main.rs:3417   up()                 -> up_loop(root, host, runner, interval)
run.rs:2674    loop {               -> once per --interval (or on catalog change)
run.rs:2682      reconcile_pass(..) -> run.rs:1770  runner.list_sessions()
run.rs:1019      SystemRunner       -> self.pty.list_sessions()  +  self.exec.list()
run.rs:749       PtyCli             -> Command::new("pty").args(["list","--json"])   <-- node starts

The spec-file entry point (main.rs:3315 → up_spec_fleet → up_loop_specs, run.rs:2399 → run.rs:2099) converges on the same PtyCli::list_entries_at call.

Steady state is 1 Node process per supervisor per tick. Upper bound per pass is higher: MAX_PRESENTATION_PATCHES_PER_PASS = 8 (run.rs:48) allows up to 8 more pty metadata patch spawns (run.rs:719), plus one pty run per launch/restart.

36 supervisors × 1 Hz × ~55 ms ≈ 2 cores' worth of interpreter startup before any of them does anything.

RCA

• PtyCli is defined with bin: "pty" and shells out for every operation — run.rs:360, run.rs:370.
• The read path that runs every tick: PtyCli::list_entries_at → Command::new(&self.bin).args(["list", "--json"]), run.rs:746–run.rs:762. Output is parsed back out of JSON into PtyListEntry.
• There is no pty daemon to query instead. pty's listSessions (sessions.ts:895) is pure filesystem observation: readdirSync of the session dir, readPid + isProcessAlive, readMetadata of the .json records, plus opportunistic socket reachability probes for entries whose pid isn't provably alive. The sockets in server.ts:652 are per-session control sockets, not a registry. So "talk to the daemon over its socket" is not an available path — the state st2 wants is on disk, and the Node process exists only to read it and re-serialize it as JSON.
• st2 already has the native version of this, for the other backend. ExecBackend::list (exec_backend.rs:195) does exactly this observation in Rust — fs::read_dir over the state dir, per-id generation observation, no subprocess. SystemRunner::list_sessions (run.rs:1019) calls both; only the pty half spawns anything.

Options

1. Read $PTY_ROOT natively in Rust, mirroring ExecBackend::list. Removes the per-tick process entirely; the cost falls to the ~6 ms the observation actually is. Cost: it makes pty's on-disk layout a contract between the two repos, and duplicates the running/exited/vanished classification and the socket-probe fallback (sessions.ts:895–1010), which can drift.
2. Long-lived pty query process — one Node startup per supervisor instead of one per tick, over a request/response pipe. One implementation, no format duplication. Cost: st2 gains a child to supervise, and a hung query is worse than a slow one — the current 2 s PTY_LIST_TIMEOUT (run.rs:46) has no equivalent once the process is persistent.
3. Lazy imports in pty's CLI. Ships fastest and is a one-repo change, but recovers only the module-graph share (~60%); the ~40% interpreter floor remains — roughly 0.9 of the 1.32 cores at the measured rate.
4. Raise the default interval. A mitigation, not a fix: it trades reconcile latency for CPU linearly and leaves the per-call cost untouched. Not recommended.

Recommendation: (1), with (3) as an independent, cheap win in pty. The precedent is already in-tree — ExecBackend::list proves the observation is a read_dir — and it is the only option that removes the cost rather than dividing it. The drift risk is real but bounded and testable: the classification is ~100 lines, and st2 already depends on a JSON contract with pty — today it is the CLI's list --json output (PtyListEntry, run.rs:377), which would become the on-disk <name>.json records instead.

Scope

The pty run -d --force --id … arm (8.1/s) is not steady-state supervision — 36 supervisors at 1 Hz cannot produce it unless spawns are failing and retrying, and run.rs:816 amplifies one failed spawn into up to 7 Node starts (SPAWN_ATTEMPTS = 4, with a pty rm between attempts). Why spawns are churning is being investigated separately; this issue is about the per-invocation cost, which applies to that arm too.

Reproduction

https://github.com/schickling-repros/2026-08-st2-pty-cli-node-startup

Runs N concurrent 1 Hz pty list --json loops against a throwaway PTY_ROOT and reports cores burned, with a non-Node control at the same rate and worker count. No st2 build or live fleet needed; nothing outlives the run.

arm                          workers  interval      cpu_s    cores
pty list --json @1Hz               1       1.0       1.07     0.09
pty list --json @1Hz               4       1.0       3.97     0.33
pty list --json @1Hz               8       1.0       8.35     0.70
CONTROL stub-list @1Hz             8       1.0       1.25     0.10
CONTROL pty list @0.1Hz            8      10.0       1.81     0.15

Linear in supervisor count (~0.087 cores each at --interval 1 on that host); the identical loop running a non-Node lister costs 7× less, which isolates the Node process rather than the fork or the filesystem work as the cost.

Posted on behalf of @schickling

field	value
agent_identity	unknown
session	unknown
agent_persona	unknown
agent_supervisor	unavailable
agent_tool	unknown
agent_tool_version	unknown
agent_runtime	unknown
tooling_profile	dotfiles@586b3a0-dirty