fix: make an always-on agent reachable, and stop reporting liveness it hasn't earned

[schickling-assistant]

Fixes the two defects blocking a reachable always-on agent. Two independent commits, reviewable and revertable separately.

Both issues turned out to be misdiagnosed in their write-ups. Neither original framing survived reproduction. That is the most important thing in this PR, so it is up front.

---

Defect 1 — #101: the sidecar was armed the whole time

What the issue said

st ding starts, writes a single session_start event, and then never arms.

What actually happens

The sidecar arms fine. The watcher arms, the startup scan runs, and delivery works. What is broken is that a message can be held indefinitely, silently, while the sidecar keeps advertising the agent as available.

Three things I checked before touching code:

1. session_start is not written by smalltalk at all. Nothing in this repo emits it — that log is convoy/pty sidecar telemetry. "Only one session_start record" is not evidence that ding did nothing; ding simply does not write there. It was a red herring.
2. The target session "<id>" not yet registered line does not gate delivery. I ran a ding against a target with no pidfile at all: it delivered normally. The session-watch only controls exit, and its startup grace suppresses that. Also a red herring.
3. The real block is in guardedDeliver. On the no-input branch, a changing pane frame returns held unconditionally. forceCap is computed at the top of the function but only ever consulted on the un-submitted-input branch — so this hold has no upper bound.

assessPane compares two full pty peek --plain frames taken 300 ms apart with byte equality. Any difference means "mid-turn". A working agent's pane changes constantly, so every poke is held for as long as the agent keeps working — and the hold logs only under ST_DING_DEBUG, which production does not set. Meanwhile startStatusRefresh runs on a timer completely decoupled from delivery, bumping the status mtime the entire time.

Net effect matches the report exactly: process alive, message sits in the inbox, status reads available, stderr completely silent.

Reproduced

Isolated temp root, stub pty on PATH, never against the live bus:

boot ding on an empty inbox -> send one message -> keep the stub pane's frame changing

  pty send calls:     0
  message:            still in inbox
  process:            alive
  status mtime:       bumped every tick
  stderr (no debug):  (nothing)

Flip the stub pane to static and the message is delivered within one retry — so there is no deeper state bug; the hold does recover.

I also sampled real panes on this host read-only. Two of three were byte-changing at that instant; the Claude Code status line carries an elapsed clock (⏱14h31m) and a cost counter. But an idle pane sampled 8× in a row was static every time.

Correction to the issue's claim

The issue says an agent that "boots, drains, and goes idle is unreachable for the rest of its life." That over-states it. An idle pane is byte-static and a held message lands within ~20 s of the pane going idle. The accurate claim is narrower and still a blocker: an agent is unreachable for exactly as long as its pane keeps changing — i.e. the whole time it is working. For an always-on agent on long tasks that is effectively unbounded, and it is unbounded by construction since the cap is never consulted on this branch.

The fix, and what it deliberately does not do

The hold decision itself is correct and unchanged. Submitting into an active Claude Code turn seeds CC's queued-input replay bug, and there is an explicit regression test in this repo (hold cap with a still-changing (mid-turn) frame → keeps HOLDING, never force-submits) forbidding a force-submit. That test still passes. Adding a hard time ceiling that force-submits would reintroduce precisely the bug the hold exists to prevent, so I did not.

What was wrong is that the sidecar lied about it. So past the hold cap it now:

• warns once, loudly, on stderr — production had no signal for this state
• suspends the status heartbeat, so the mtime freezes and readers derive staleness through the existing path
• resumes, with a log line, once no message remains undeliverable

⚠️ That last bullet originally read "on the next successful delivery", and that was a blocker — see the follow-up section below. Clearing only on delivery made the stall permanent whenever the agent drained its own inbox, which is the documented common path. Fixed in 440b1f4.

On the issue's (a)/(b)/(c) options

All three are premised on the sidecar failing to arm, which I disproved. Reframed: the violated invariant is not "unarmed writes available" but "armed-but-not-delivering writes available".

• (a) arm-or-exit — does not apply. There is no arm failure to exit on, and exiting a healthy sidecar because its target is busy would be worse than the bug.
• (b) gate the status write — chosen, moved from armed state to delivery state, which is what actually predicts whether mail gets surfaced. It also reuses the death-coupling this codebase already relies on (ding exits → touches cease → reads dead) rather than inventing a second liveness mechanism.
• (c) liveness from the provider — a larger change and not obviously better here: the provider being up is exactly what was true and misleading in the incident. The provider was alive; its mail was not being delivered.

---

Defect 2 — #102: there is already a window; it is the wrong question

What the issue said

st agents reports the raw status value with no freshness window.

What actually happens

st agents does apply a window. It reads through readIdentityStatus → readState, which applies STATUS_STALE_MS and collapses to unknown. Verified:

status backdated  3 minutes  ->  st agents: available
status backdated 20 minutes  ->  st agents: unknown

So this is a threshold mismatch, not a missing check — which matters, because it changes what the correct fix is.

The two windows answer different questions:

	question	sized for	value
STATUS_STALE_MS	do we still trust this value?	slowest writer — MCP's 5-min refresh	15 min
convoy's window	is this agent live right now?	ding's 30 s heartbeat	~2 min

Both are defensible. They were never the same question, and st agents at 3 minutes was answering the first one correctly.

This makes the obvious fix a trap. Tightening STATUS_STALE_MS to ~120 s to match convoy would flap every MCP-refreshed agent into unknown between refreshes.

The fix — (c), plus (a)'s annotation

• STATUS_LIVENESS_MS (2 min) named alongside the existing STATUS_STALE_MS — the reader half of the R=30 s/T≈120 s contract the ding already documents in common.ts
• readIdentityLiveness(), one reader returning both verdicts plus the recorded value and mtime, exported from the package index so consumers inherit a definition instead of each inventing one. That is the part that actually stops the divergence
• st agents states the age of the value: available (3m ago), and unknown (was busy, 22m ago) past the trust window

The roster reports a fact, not a verdict — this is deliberate and I got it wrong first. My initial commit rendered available (stale 3m). But st agents enumerates a mixed population and the writers don't share a cadence: a ding-backed agent is touched every 30 s, an MCP-backed one only every 5 min. Judged against a single 2-minute window, a perfectly healthy MCP-backed agent reads stale for three of every five minutes — the same trap as tightening STATUS_STALE_MS, one layer up, just swapping "dead agents read available" for "live agents read stale". The third commit reverses that: the age is true regardless of writer, and it is precisely what was missing when st agents said available and convoy said DEAD (status stale 3m ago). A verdict still exists where it can be justified — live stays in --json, and readIdentityLiveness takes a livenessMs override — so a consumer that knows its agents run a ding keeps a definite answer while inheriting one shared definition.

On the issue's question 3 — yes, the distinction matters and is preserved. recorded survives both windows, so an identity that was busy and went quiet stays distinguishable from one that cleanly went offline. That is exactly the signal you want when something died mid-work.

⚠️ Semantics change reviewers must weigh

• Unchanged: the derived status field, and what --status filters on. A stale-but-trusted available still matches --status available. There is a test pinning this.
• Changed: the rendered text cell of st agents now carries an age once past the liveness window. Anything parsing column 2 of the text output will see something it did not see before.
• Additive: --json gains live / statusMtimeMs / recorded.

statusMtimeMs is an absolute mtime rather than an age so two back-to-back reads of an unchanged bus still compare equal. An ageMs field broke that — caught by bus-reader's existing repeatability test, which is a good test.

---

Follow-up: a stall must not outlive its cause

An adversarial review found — and reproduced with a test — that the #101 fix as first written was worse than the defect it fixed. Fixed in 440b1f4.

The blocker

clearDeliveryStall() was called from exactly two places: normalDeliver on success, and preserveDeliver. Both are delivery paths. But the held message's most likely fate is never being delivered at all:

1. A busy agent's message is held past the cap → stall set, status mtime freezes.
2. The agent reads and archives the message itself. The documented boot ritual is literally "drain your inbox", so this is what agents are told to do.
3. The buffered poke is correctly dropped as a stale poke (continue) — but the stall is never cleared.

The heartbeat then never resumes, not even after the pane goes idle with an empty inbox: there is nothing left to deliver, so nothing can ever clear it. A transient "unreachable while busy" became a permanent "reads dead" for a perfectly healthy agent.

Precondition, stated so it is not waved off: the cap must trip before the agent archives. Both are ordinary.

The fix

The stall is cleared on the drained checkpoints — the two buffer.length === 0 && readPending.length === 0 → disarmTimer() points — which express exactly the condition "no message remains undeliverable".

The tempting one-line fix (clear on the archived-while-held continue) is wrong, and there is a test pinning that: deliveryStalled is a single per-daemon flag, so clearing it the moment any message is archived resumes the heartbeat while a second message is still genuinely stuck on the busy pane — re-introducing precisely the unearned liveness #101 exists to prevent. A single-message test passes straight over that regression.

Archived-while-buffered events are additionally pruned before the busy/dnd suppress-return, not only inside the drain loop. The drain never runs while the identity is busy/dnd, so a busy agent archiving its own mail would otherwise leave the event parked in the buffer forever — the buffer would never drain and the stall would outlive its cause by that path even with the checkpoint clear.

Evidence

Three new tests, each failing before the fix and passing after (verified by reverting the fix, re-running, and restoring):

test	without fix	with fix
agent archives the held message itself → stall clears, heartbeat resumes	❌ mtime frozen forever	✅
one of two held messages archived → stall persists (over-clearing guard)	❌ never recovers even after both archived	✅
archived while held on a busy identity → stall still clears	❌ mtime frozen forever	✅

All three failed identically without the fix — expected <mtime> to be greater than <mtime>, i.e. the heartbeat never resumed. The three pre-existing #101 tests pass in both states, so the fix does not weaken the original guarantee.

Can the stall still outlive its cause?

One narrow residual, pre-existing and not introduced here: an entry stuck in readPending keeps the drained checkpoint from firing. It requires a message that fails buildEvent permanently. In practice cmdRead falls back from inbox to archive, so the common archived-before-read case still reads fine, lands in the buffer, and is pruned normally — the residual needs a file present in neither folder, or genuinely unparseable. Deliberately not "fixed" by pruning readPending on !stillInInbox: an arrival is renamed into the inbox, so that predicate is briefly false mid-arrival and pruning on it would drop real messages. Flagging rather than silently trading a rare stuck heartbeat for a message-loss bug.

readIdentityLiveness: the comment was lying

Same review, lower severity. The doc comment claimed "single stat + single read". It performed 4 stats + 2 reads — it delegated to readState, which does its own existsSync + statSync + readFileSync, and then repeated the whole sequence itself.

Worse than the cost: status and recorded came from two different reads, so a status write landing between them could return a pair that never existed on disk (e.g. status: busy alongside recorded: 'available').

Now derived from one stat + one read, with every field consistent by construction. The collapsed existsSync+statSync pair was itself a stat-then-stat TOCTOU. The caller's injected now now governs the whole verdict rather than only ageMs/live — previously status silently used a second Date.now().

---

The two defects are coupled

Worth flagging: #101's fix depends on #102's window to bite. Suspending the heartbeat freezes the mtime, but with only the 15-minute trust window a sender still reads available for 15 more minutes. With STATUS_LIVENESS_MS, the roster tells the truth in ~2. Verified end to end:

t+0     roster: available
stall   -> DELIVERY STALLED on stderr, mtime frozen
t+195s  roster: available (2m ago)          <- age becomes visible
pane goes idle -> poke delivered, mtime resumes, "delivery recovered"

That is the argument for shipping them together.

Verification

• nix flake check green as of the first two commits. Not re-run after the follow-up commit — that commit was gated on tsc -p tsconfig.build.json --noEmit (clean over src/), which is the same typecheck flake check enforces, plus the full unit suite
• Integration tests were not run for the follow-up: the @compoundingtech/pty file: devDependency is absent from the dependency closure available here. The readIdentityLiveness change is still directly covered by the status and status-staleness unit files; what is unexercised is only the end-to-end st agents render surface
• 1347 unit tests pass across all 49 unit files (vitest run tests/unit); 16 new tests — 3 for Ding refreshes status for a dead provider — an unarmed sidecar forges liveness #101, 10 for st agents has no freshness window — reports stale status as live #102, and 3 for the stall-outliving-its-cause follow-up. tsc -p tsconfig.build.json --noEmit clean over src/
• The three follow-up tests were verified to fail on revert and pass on restore, not merely to pass
• The MCP tool tests are the surface most directly downstream of the AgentSummary change (three added keys, and those tests assert output shape), so I ran them explicitly in the nix sandbox where devDeps exist — tests/unit/mcp/, plus bus-reader / exports / overview / status: all green
• Heads-up on CI coverage: nix flake check gates help, completions, and typecheck only — vitest is not a CI gate (see the comment in flake.nix about Adopt the shared build-identity contract: st --version has no build identity, MCP serverInfo reports a different version #103). So the test evidence here is local, not enforced. Worth deciding separately whether that should change.
• Pre-existing, not from this PR: tests/unit/common.test.ts fails to parse on main — stConfig / stConfigFrom are each imported twice. Worth a separate one-line fix. The other collection failures in my tree are just a missing node_modules, not a code problem.

Reproduced vs inferred

Reproduced: the unbounded silent hold and its full signature; that an unregistered target still delivers; that a held message recovers when the pane goes static; that st agents already windows at 15 min and reports available at 3 min; the stall → frozen heartbeat → aged roster → recovery lifecycle.

Inferred: that this specific mechanism caused the production incident. The evidence fits tightly (available + silent + undelivered + alive is a signature nothing else in this code produces), and the production launch is exactly st ding <host>.cos --identity <host>.cos --root <net>/smalltalk with pane-guard defaults on — but the incident itself predates the diagnosis and I could not replay it.

Corrected: "never arms" (it arms), "unreachable for the rest of its life" (only while the pane keeps changing), and "no freshness window" (there is one, sized for a different question).

Corrected in this PR's own earlier state: that the stall "resumes on the next successful delivery" was an inadequate clear condition, and that readIdentityLiveness did a "single stat + single read" when it did four and two. Both found by adversarial review, both reproduced with failing tests before being fixed.

Testing safety

All testing ran against isolated temp roots with a stub pty binary, and the unit suite pins PTY_SESSION_DIR to a fresh temp dir via tests/setup/pty-isolation.ts. The live hosted agent, its bus, and all 45+ production pty sessions were untouched — pty ls count 91 before and 91 after, across both the original work and the follow-up. No process was pattern-killed; only processes I created were killed. The one read-only touch of production was pty peek --plain to characterize whether real panes are byte-static.

Closes #101

Refs #102 — deliberately not Closes. The issue was raised as a consensus question because it changes what a command means, so the semantics note above is a proposal, not a settled call. Close it if the shape is agreed.